OKFN Labs: The Data Wrangling Blog

data.okfn.org - update no. 1

2013-05-28T00:00:00-07:00

This is the first of regular updates on Labs project http://data.okfn.org/ and summarizes some of the changes and improvements over the last few weeks.

1. Refactor of site layout and focus.

We've done a refactor of the site to have stronger focus on the data. Front page tagline is now:

We're providing key datasets in high quality, easy-to-use and open form

Tools and standards are there in a clear supporting role. Thanks to all the suggestions and feedback on this and welcome more - we're still iterating.

2. Pull request data workflow

There was a nice example of the pull request data workflow being used (by a complete stranger!): https://github.com/datasets/house-prices-uk/pull/1

3. New datasets

For example:

US house prices http://data.okfn.org/data/house-prices-us
Annual consumer price index http://data.okfn.org/data/cpi

Looking to contribute data check out the instructions http://data.okfn.org/about/contribute#data and the outstanding requests: https://github.com/datasets/registry/issues

4. Tooling

We have a DataPackage.JSON creator tool in progress at http://data.okfn.org/tools/dp/create (here's the relevant github issue)
We have a new data package viewer created by James Smith

5. Feedback on standards

There's been a lot of valuable feedback on the data package and json table schema standards including some quite major suggestions (e.g. substantial change to JSON Table Schema to align more closely with JSON Schema - thx to jpmckinney)

Next steps

There's plenty more coming up soon in terms of data and the site and tools.

Complete the datapackage.json generator (support for gdocs especially)
Complete the datapackage.json validator
More datasets especially key indices

Get Involved

Anyone can contribute and its easy -- if you can use a spreadsheet you can help!

Instructions for getting involved here: http://data.okfn.org/about/contribute

Open Humanities Hangout - Open Correspondence and the Letter Net

2013-05-21T00:00:00-07:00

Our next Open Humanities Hangout will take place next Tuesday, 28th May. This is the latest in the series of regular hangouts we've been organizing over the past few months with people interested in tapping in to the growing amount of open cultural data and content.

What: Open Humanities Hangout looking at opening up historical correspondence and mapping the "letter net" – e.g. did Dickens write to George Eliot and did she write back? Come help us find out! Read more below
When: Tuesday 28th May 2013 at 1700 BST, 12:00 EDT, 1800 CET
Where: Online via Google Hangout and IRC – we'll publish the hangout url nearer the time
Who: anyone who loves the humanities and wants to see the great works of our past accessible and re-usable by anyone regardless of background or location.
Signup: please sign up here or email sam.leon@okfn.org. Note you can always just drop in on the day but it helps us if we have a sense of numbers!

About the Hangouts

The Humanities Hangouts are an informal virtual get together to build apps and insights using open cultural material. Among other things participants have put together an app that helps you to get to know Shakespeare better called Bardomatic, hacked on an annotation tool for public domain texts called TEXTUS and created interactive timelines of the great Western medieval philosophers (helping to improve and de-bug the Timeliner tool in the process).

The Challenge: Mapping Networks of Correspondence

We want to construct a workflow that will enable anyone to take a published set of letters and turn it into open data and content that we can explorer and visualize. Ultimately we want the network of correspondence – the "letter net".

Suggested process

This is something to discusson the hangout, but we think the effort will involve at least 3 steps:

Locate published collection of letters
- Great if these are already digitized on gutenberg
Extract structured data like author, recipient, date, location
- Geo-code all those locations
- If the texts are not digitized start thinking about that!
Visualise the results

We've already done work on steps 1 and even 2 in the case of Dickens. For geocoding there's already a simple geocoding guide on the School of Data. For visualization there are plenty of options that we'll explore on the hangout. (And if we want to start scanning and OCRing there are manuals on how to build your own scanner).

Our Goal

Our basic goal is a set of beautiful and insightful set of visualisations about the correspondence of key cultural figures.

Longer term we would love to see a database of correspondence that is open to everyone to use and add to.

Nomenklatura - Data Matching and Reconciliation Made Easy

2013-05-16T00:00:00-07:00

Nomenklatura is a simple service that makes it easy to maintain a canonical list of entities such as persons, companies or event streets and to match messy input, such as their names against that canonical list – for example, matching Acme Widgets, Acme Widgets Inc and Acme Widgets Incorporated to the canonical "Acme Widgets".

With Nomenklatura its a matters of minutes to set up your own set of master data to match against and it provides a simple user interface and API which you can then use do matching (the API is compatible with Open Refine's reconciliation function).

Nomenklatura can not only store the master set of entities you want to match against but also will learn and record the various aliases for a given entity - such as a person, organisation or place - may have in various datasets.

As such Nomenklatura chooses a design half way between an entity database (such as OpenCorporates, PopIt or similar services) and a automated de-duplication software (such as dedupe or SILK).

Nomenklatura has been battle-tested with real-world usage, for example to de-duplicate the names of German parliamentarians, UK government departments and spending data schemas and EU lobbyists.

Typically, a data extraction process will check all the entity names it discovers in the source data against nomenklaturas API. If Nomenklatura does not recognize a name, a new alias record is stored as a placeholder. This alias can then be matched to an entity by the user through a simple-to-use reconciliation user interface.

To kickstart such a process, data can be uploaded via CSV - but new entities can be created dynamically as well. The advantage of a manual approach is that it minimizes the risk of false matches -- this level of quality assurance can be crucial, if, for example, the output will be displayed in an application that is intended to hold government to account.

This Release

This latest release of Nomenklatura includes a number of important changes:

The domain model was refactored to use a clearer naming scheme, canonical values are now called "entities", and their alternative spellings are now "aliases".
CSV upload support allows users to submit a list of entities, aliases or fully executed mappings.
Support for the Open Refine API was added, so that each Nomenklatura dataset can be added as a reconciliation service and used to clean data from inside Refine.
Keyboard shortcuts were added to the reconciliation tool, so that matches can be identified without using a mouse - a fast user can now match a few hundred records an hour.
The Python client library has been refactored and submitted to PyPi, it can be installed via "pip install pynomenklatura".

Credits and Links

Nomenklatura was developed by Labs Member Friedrich Lindenberg with contributions from other folks including fellow Labs members Michael Bauer.

Nomenklatura source code on GitHub

Update on PublicBodies.org - a URL for every part of Government

2013-05-01T00:00:00-07:00

This is an update on PublicBodies.org - a Labs project whose aim is to provide a "URL for every part of Government": http://publicbodies.org/

PublicBodies.org is a database and website of "Public Bodies" – that is Government-run or controlled organizations (which may or may not have distinct corporate existence). Examples would include government ministries or departments, state-run organizations such as libraries, police and fire departments and more.

We run into public bodies all the time in projects like OpenSpending (either as spenders or recipients). Back in 2011 as part of the "Organizations" data workshop at OGD Camp 2011, Labs member Friedrich Lindenberg scraped together a first database and site of "public bodies" from various sources (primarily FoI sites like WhatDoTheyKnow, FragDenStaat and AskTheEU).

We've recently redone the site converting the sqlite DB to simple flat CSV files:

Main github repo: https://github.com/okfn/publicbodies
Example raw CSV: https://raw.github.com/okfn/publicbodies/master/data/gb.csv

The site itself is now super-simple flat-files hosted on s3 (build code here). Here's an example of the output:

European Parliament: http://publicbodies.org/eu/european-parliament.html
Associated JSON API (with CORS!) http://publicbodies.org/eu/european-parliament.json

The simplicity of CSV for data plus simple templating to flat-files is very attractive. There are some drawbacks such as changes to primary template resulting in a full rebuild and upload of ~6k files so, especially as the data grows, we may want to look into something a bit nicer but for the time being this works well.

Next Steps

There's plenty that could be improved e.g.

More data - other jurisdictions (we only cover EU, UK and Germany) + descriptions for the bodies (this could be a nice crowdcrafting app)
Search and Reconciliation (via nomenklatura)
Making it easier to submit corrections or additions

The full list of issues is on github here: https://github.com/okfn/publicbodies/issues

Help is most definitely wanted! Just grab one of the issues or get in touch ...

Quick and Dirty Analysis on Large CSVs

2013-04-11T00:00:00-07:00

I'm playing around with some large(ish) CSV files as part of a OpenSpending related data investigation to look at UK government spending last year -- example question: which companies were the top 10 recipients of government money? (More details can be found in this issue on OpenSpending's things-to-do repo).

The dataset I'm working with is the consolidated spending (over £25k) by all UK goverment departments. Thanks to the efforts of of OpenSpending folks (and specifically Friedrich Lindenberg) this data is already nicely ETL'd from thousands of individual CSV (and xls) files into one big 3.7 Gb file (see below for links and details).

My question is what is the best way to do quick and dirty analysis on this?

Examples of the kinds of options I was considering were:

Simple scripting (python, perl etc)
Postgresql - load, build indexes and then sum, avg etc
Elastic MapReduce (AWS Hadoop)
Google BigQuery

Love to hear what folks think and if there are tools or approaches they would specifically recommend.

The Data

Here's the 3.7 Gb CSV
A Data Package file for the data describing the fields: https://raw.github.com/openspending/dpkg-uk25k/master/datapackage.json

sqlaload, an ETL wrapper for SQLAlchemy

2013-03-30T00:00:00-07:00

sqlaload is a small library that I use to handle databases in Python data processing. In many projects, your process starts with very messy data (something you've scraped or loaded from a hand-prepared Excel sheet). In subsequent stages, you gradually add cleaned values in new columns or new tables. Managing a full SQL schema for such operations can be a hassle, you really want something close to MongoDB: a NoSQL data store you can throw fairly random data at and get it back later.

With sqlaload, the idea is to combine some of the schema flexibility, while still keeping things in a structured database in the background:

import sqlaload as sl
engine = sl.connect('sqlite:///test.db')

# add some data:  
sl.add_row(engine, 'mytable', {'name': 'Foo', 'has_this': True})
sl.add_row(engine, 'mytable', {'name': 'Bar', 'has_other': True})

# Look up a record
row = sl.find_one(engine, 'mytable', name='Foo')
assert row['has_this']==True

# Update a record:
sl.upsert(engine, 'mytable', {'name': 'Foo', 'location': 'Atlantis'}, ['name'])

# Or create one:
sl.upsert(engine, 'mytable', {'name': 'Qux', 'location': 'Elsewhere'}, ['name'])

I first saw this type of SQL schema generation implemented in ScraperWiki: they have a couple of high-level SQLite wrappers that expand your database as you feed them data. We later adopted that concept for the joint CKAN/ScraperWiki webstore, which neither project ended up using.

Still, webstore had become an essential part of many of my data projects as an operational data store. Eventually, I decided to kick out the networking aspect: data access via HTTP is terribly slow and I wanted to have my data in Postgres, not SQLite. The webstore code went into sqlaload, and became a thin wrapper on top of SQLAlchemy core (the non-ORM database abstraction part of SQLAlchemy).

Running on top of SQLAlchemy also means that all of its functionality - for example the query expression language - are available and can be used to call up more advanced functionality.

If you want to try it out, sqlaload is now on PyPI and the README has a lot of detailed documentation on the library.

Next Steps for Textus

2013-03-27T00:00:00-07:00

At the Culture Labs hangout yesterday we wrote up the plans for the next steps for Textus we have been discussing over the last few months.

The result is this slide deck overview. It both introduces Textus and outlines next steps (slide 12 onwards).

Key Points

We want to:

Maximize simplicity
Connect with a CMS (people always want other content than just the texts)

Implications are:

Componentize "Textus" and separate text preparation / import from presentation
Create a plugin to make "Textus" style functionality one-click install into Wordpress
Eliminate dependencies on ElasticSearch & NodeJS (texts & markup stored in plain files online or in WP ...)

Specifically, we plan to break Textus into 3 components:

textus-formatter - nodejs app/command line tool for formatting texts
textus-viewer - JS-only viewer
textus-wordpress - wordpress integration

Progress on the Data Explorer

2013-03-18T00:00:00-07:00

This is an update on progress with the Data Explorer (aka Data Transformer).

Progress is best seen from this demo which takes you on a tour of house prices and the difference between real and nominal values.

More information on recent developments can be found below. Feedback is very welcome - either here or the issues https://github.com/okfn/dataexplorer.

What is the Data Explorer

For those not familiar, the Data Explorer is a HTML+JS app to view, visualize and process data just in the browser (no backend!). It draws heavily on the Recline library and features now include:

Importing data from various sources (the UX of this could be much improved!)
Viewing and visualizing using Recline to create grids, graphs and maps
Cleaning and transforming data using a scripting component that allows you to write and run javascript
Saving and sharing: everything you create (scripts, graphs etc) can be saved and then shared via public URL.

Note, that persistence (for sharing) is to Gists (here's the gist for the House Prices demo linked above). This has some nice benefits such as versioning; offline editing (clone the gist, edit and push); and bl.ocks.org-style ability to create a gist and have it result in public viewable output (though with substantial differences vs blocks ...).

What's Next

There are many areas that could be worked on -- a full list of issues is in github. The most important I think at the moment are:

Storing the data "locally" in the data project. At present, data is always loaded from an "external" source. This probably involves extending the current Recline datastore to back on to IndexedDB.
A better project creation & data import process - I think we could learn a lot from Refine here
"Fork" support
More documentation and tutorials especially for scripting
Getting rid of the many rough edges especially on the UX side of things!

I'd very interested in people's thoughts on the app so far and what should be done next and code contributions are also very welcome (the app has already benefitted from the efforts of many people including the likes of Martin Keegan and Michael Aufreiter to the app itself; and from folks like Max Ogden, Friedrich Lindenberg, James Casbon, Gregor Aisch, Nigel Babu (and many more) in the form of ideas, feedback, work on Recline etc).

Recline JS - Componentization and a Smaller Core

2013-02-26T00:00:00-08:00

Over time Recline JS has grown. In particular, since the first public announce of Recline last summer we've had several people producing new backends and views (e.g. backends for Couch, a view for d3, a map view based on Ordnance Survey's tiles etc etc).

As I wrote to the labs list recently, continually adding these to core Recline runs the risk of bloat. Instead, we think it's better to keep the core lean and move more of these "extensions" out of core with a clear listing and curation process - the design of Recline means that new backends and views can extend the core easily and without any complex dependencies.

This approach is useful in other ways. For example, Recline backends are designed to support standalone use as well as use with Recline core (they have no dependency on any other part of Recline - including core) but this is not very obvious as it stands (where the backend is bundled with Recline). To take a concrete example, the Google Docs backend is a useful wrapper for the Google Spreadsheets API in its own right. While this is already true, when this code is in the main Recline repository it isn't very obvious but having the repo split out with its own README would make this much clearer.

So the plan is ...

Announce this approach of a leaner core and more "Extensions"
- Link to the specifications for Backends and Views
- Create an official Recline Extensions page
Identify first items to split out from core - see this issue
Identify what components should remain in core? (I'm thinking Dataset + Memory DataStore plus one Grid, Graph and Map)

So far I've already started the process of factoring out some backends (and soon views) into standalone repos, e.g. here's GDocs:

https://github.com/okfn/recline.backend.gdocs

Any thoughts very welcome and if you already have Recline extensions lurking in your repos please add them to the wiki page

Exporting PyBossa data to CSV or JSON with one click

2013-02-20T00:00:00-08:00

I'm really happy to announce that today we have finally added a feature that will allow to export your data into a CSV format with just one click (we also support the same for JSON).

For this purpose, all the applications in PyBossa now feature a new URI:

http://PYBOSSA-SERVER/app/slug/export

Where you will find several options to export the tasks or task runs (the answers) to different formats. In the case of the CSV format, you will get a CSV file that could be downloaded to your computer to load it later in any spreadsheet software :-)

NOTE: bear in mind that CSV is a flat format, so nested JSON objects will be "dumped" as they are, so for example if you are using GeoJSON for storing some location, you will get in the CSV file the JSON object as a string. You can see an example of this issue in the Urban Parks application, as this demo application uses the GeoJSON format for storing the location of the parks.

If you prefer JSON, just click in any of the buttons and save the generated file!

If you want to try the new feature, just go ahead and check it in CrowdCrafting.org

Mozilla FirefoxOS App Days & Crowdcrafting.org

2013-01-29T00:00:00-08:00

Last Saturday, the 26th of January, Mozilla held in parallel in 25 cities all over the world a hack day, the #FirefoxOSAppDay, about creating new web applications for their new FirefoxOS mobile OS and the desktop web browser (this stills in beta and alpha mode!).

One of the events was held in Madrid, Spain, organized by the Mozilla Hispano Community so I had the chance to expend some time with the Mozilla community and play with the new APIs and developer tools for their new platform.

In the morning we attend several talks by several experts on the new APIs that Mozilla are developing to integrate mobile actions like for example the battery API that will allow you to check the device battery status (right now integrated in the W3C standards) or the Alarm API that you can use to schedule a notification, or for an application to be started, at a specific time.

Mozilla is working really hard to standardize and integrate most of these APIs into the W3C in order to make them available in any web browser. Some of the APIs are actually now accepted in the W3C as for example the Battery Status API, Network Information API, Ambient light sensor or the Proximity sensor.

Mozilla also presented their efforts in making as easy as possible to create an application from scratch re-using several building-blocks they have created for their new platform. Basically, they have created a web page where you can copy and paste code snippets that you can later re-use in your own application, keeping the look and feel of the platform.

After the talks, all the participants had a better idea of what we could develop with the platform: a web application that could use the hardware of the new mobile phone devices, as well as Android phones out of the box!

As the goal of the day was to create an app for the FirefoxOS, my idea was to create an application that could help to track when a new scientific application has been added to crowdcrafting so you could help doing some tasks in the new application.

The web application basically lets you know which apps are new since the last time you check it out.

The application works in any web browser (even Chrome) but if you want to feel how it will be in the new OS you can try it in your phone if you have an Android device. You will need to install the latest Firefox nightly (note: this is an experimental build, so it may crash in your phone!) and then type this URL:

http://daniellombrana.es/crowdcrafting-app

You will be able to install it in your phone and run it whenever you want directly from your home screen. If you don't want to install the browser, just open the link with a modern web browser and you should see it running (the install button will only work in Firefox Beta and Aurora builds).

PyBossa.JS or how you can easily create new PyBossa applications

2013-01-28T00:00:00-08:00

In the last weeks we have been working hard in order to make easier to develop new PyBossa applications. For this reason, we are happy to announce a new version of PyBossa.JS. This new version introduces several improvements:

Creating an app is much easier! You only have to override two methods: pybossa.taskLoaded and pybossa.presentTask to fit your app, and call pybossa.run('your-app-slug').
Pre-loading tasks by default! Now your app could improve its performance, as the next task for the user will be loaded in the background for you while the user stills solving the first one!
Automatically update the task URL. The library will change the browser's URL to the current task automatically, so using services like Disqus for comments is really simple (check the updated version of Flickr Person Finder for more details!).

As a result of this new version, there are at least two applications using the new PyBossa.JS version:

Flickr Person Finder has been updating, using this new set of features. If you try the application you will see that loading the next task (in this case an image which is usually 1024x1024px big) is almost instantly. Additionally, the app shows how you can use the Disqus service to allow your users to add comments for each task, but only loading them when the user wants.
The Face We Make is a new application where you have to guess the emoticon that a person is representing in a photo. This app is a joint effort with the official The Face We Make project by Dexter Miranda and Daniel Lombraña González. The app has been updated for using the new pre-loading of tasks, and once you have completed all of them (only 10 photos!) show you your results, in other words, how many of your guesses are right/wrong.

Finally, we have also added the "missing features" that allow you to create an application without using the API. Right now, you can create an application using only the web forms for creating the application:

You can also add and work in the task presenter (we have included the CodeMirror plugin, so you will see how it looks your code as you type it!):

As well as importing the tasks via a CSV file importer (you can even import the CSV file from a Google Spreadsheet!):

The documentation has been updated in order to reflect this new features, and as a result you should be able to write an application really fast. However, we are far from perfect, so any feedback that you can give us will be really good! Thus, please, leave in the comments your feedback or send us an e-mail to info@pybossa.com. We will be more than happy to hear your thoughts on PyBossa!

Journoid, data notifications

2013-01-25T00:00:00-08:00

At the Open Interests hackday in November, a discussion with Martin Stabe from the FT's interactive desk led a prototype of Journoid. The idea is to monitor changing on-line datasets for remarkable information, like earthquakes, procurement in a particular industry or a close parliamentary vote. While we'd discussed alerting in the context of OpenSpending before, Martin had a pretty specific list of wishes that neither PANDA nor IFTTT can handle:

Search not just for a single keyword or query, but compare the incoming data to a table of matches, such as a list of famous people, well-known companies or any other set of items that you may be interested in.
Use Google Docs for configuration. The FT uses Google Apps internally and it's an interface that their reporters already understand - just add a "Config" sheet to your keyword document, and store all relevant settings - like the source URL and recipient email - in there.

The Journoid prototype from the hackday only fulfills the first of those requirements - and I'm still struggling with #2, as it's surprisingly hard to find a good Google Docs client library for Python.

Still, the hack was a nice demo: sift through a data dump from the UK departmental spending, check the supplier information against a list of companies of interest and finally send a message if there is a hit.

As a further experiment, I was able to use OpenCorporates to check the supplier's company status, answering a simple but interesting question: does the government do business with insolvent (or even dissolved) companies? It's interesting to think what other matches can be made when the comparison list is actually an API.

What's next? It's time to clean up the messy hackday code, to finish up GDocs configuration, some hosted solution and possibly a few other input formats.

This will also probably be my last post to OKFN Labs - early next month, I'll join Knight-Mozilla OpenNews at Spiegel Online to spend ten months working on tools like this, assisting journalists in telling more compelling stories on the web. I hope that by continuing to cooperate with my friends in the Spending Stories project on Journoid and similar efforts we can bring open (and some non-open) data into the media, making a difference.

Photo credit: Mike Tigas, If this then news demo (similar project we've started at OpenNews)

Web Scraping with CSS Selectors in Node using JSDOM or Cheerio

2013-01-15T00:00:00-08:00

I've traditionally used python for web scraping but I'd been increasingly thinking about using Node given that it is pure JS and therefore could be a more natural fit when getting info out of web pages.

In particular, when my first steps when looking to extract information from a website is to open up the Chrome Developer tools (or Firebug in Firefox) and try and extract information by inspecting the page and playing around in the console - the latter is especially attractive if jQuery is available.

What I often end up with from this is a few lines of jQuery selectors. My desire here was to find a way to directly reuse these same css selectors I use in my browser experimentation directly in the scraping script. Now, things like pyquery do exist in python (and there is some css selector support in the brilliant BeautifulSoup) but a connection with something like Node seems even more natural - it is after the JS engine from a browser!

UK Crime Data

My immediate motivation for this work was wanting to play around with the UK Crime data (all open data now!).

To do this I needed to:

Get the data in consolidated form by scraping the file list and data files from http://police.uk/data/ - while they commendably provide the data in bulk there is no single file to download, instead there is one file per force per month.
Do data cleaning and analysis - this included some fun geo-conversion and csv parsing

I'm just going to talk about the first part in what folllows - though I hope to cover the second part in a follow up post.

I should also note that all the code used for scraping and working with this data can be found in the UK Crime dataset data package on GitHub on Github - scrape.js file is here. You can also see some of the ongoing results of these data experiments in an experimental UK crime "dashboard" here.

Scraping using CSS Selectors in Node

Two options present themselves when doing simple scraping using css selectors in node.js:

Using jsdom (+ jquery)
Using cheerio (which provides jquery like access to html) + something to retrieve html (my preference is request but you can just uses node's built in http request)

For the UK crime work I used jsdom but I've subsequently used cheerio as it is substantially faster so I'll cover both here (I didn't discover cheerio until I'd started on the crime work!).

Here's an excerpted code example (full example in the source file):

var url = 'http://police.uk/data';
// holder for results
var out = {
  'streets': []
}
jsdom.env({
  html: url,
  scripts: [
    'http://code.jquery.com/jquery.js'
  ],
  done: function(errors, window) {
    var $ = window.$;
    // find all the html links to the street zip files
    $('#downloads .months table tr td:nth-child(2) a').each(function(idx, elem) {
      // push the url (href attribute) onto the list
      out['streets'].push( $(elem).attr('href') );
    });
  });
});

As an example of Cheerio scraping here's an example from work scraping info the EU's TED database (sample html file):

var url = 'http://files.opented.org.s3.amazonaws.com/scraped/100120-2011/summary.html';
// place to store results
var data = {};
// do the request using the request library
request(url, function(err, resp, body){
  $ = cheerio.load(body);

  data.winnerDetails = $('.txtmark .addr').html();

  $('.mlioccur .txtmark').each(function(i, html) {
    var spans = $(html).find('span');
    var span0 = $(spans[0]);
    if (span0.text() == 'Initial estimated total value of the contract ') {
      var amount = $(spans[4]).text()
      data.finalamount = cleanAmount(amount);
      data.initialamount = cleanAmount($(spans[1]).text());
    }
  });
});

Archiving Twitter the Hacky Way

2013-01-08T00:00:00-08:00

There are many circumstances where you want to archive a tweets - maybe just from your own account or perhaps for a hashtag for an event or topic.

Unfortunately Twitter search queries do not give data more than 7 days old and for a given account you can only get approximately the last 3200 of your tweets and 800 items from your timeline. [Update: People have pointed out that Twitter released a feature to download an archive of your personal tweets at the end of December - this, of course, still doesn't help with queries or hashtags]

Thus, if you want to archive twitter you'll need to come up with another solution (or pay them, or a reseller, a bunch of money - see Appendix below!). Sadly, most of the online solutions have tended to disappear or be acquired over time (e.g. twapperkeeper). So a DIY solution would be attractive. After reading various proposals on the web I've found the following to work pretty well (but see also this excellent google spreadsheet based solution).

The proposed process involves 3 steps:

Locate the Twitter Atom Feed for your Search
Use Google Reader as your Archiver
Get your data out of Google Reader (a 1000 items at a time!)

One current drawback of this solution is that each stage has to be done by hand. It could be possible to automate more of this, and especially the important third step, if I could work out how to do more with the Google Reader API. Contributions or suggestions here would be very welcome!

Note that the above method will become obsolete as of March 5 2013 when Twitter close down RSS and Atom feeds - continuing their long march to becoming a ~~fully~~ more closed and controlled ecosystem.

As you struggle, like me, to get precious archival information out of Twitter it may be worth reflecting on just how much information you've given to Twitter that you are now unable to retrieve (at least without paying) ...

Twitter Atom Feed

Twitter still have Atom feeds for their search queries:

http://search.twitter.com/search.atom?q=my_search

Note that if you want to search for a hash tag like #OpenData or a user e.g. @someone you'll need to escape the symbols:

http://search.twitter.com/search.atom?q=%23OpenData

Unfortunately twitter atom queries are limited to only a few items (around 20) so we'll need to continuously archive that feed to get full coverage.

Archiving in Google Reader

Just add the previous feed URL in your Google Reader account. It will then start archiving.

Aside: because the twitter atom feed is limited to a small number of items and the check in google reader only happens every 3 hours (1h if someone else is archiving the same feed) you can miss a lot of tweets. One option could be to use Topsy's RSS feeds http://otter.topsy.com/searchdate.rss?q=%23okfn (though not clear how to get more items from this feed either!)

Gettting Data out of Google Reader

Google Reader offers a decent (though still beta) API. Unoffical docs for it can be found here: http://undoc.in/

The key URL we need is:

http://www.google.com/reader/atom/feed/[feed_address]?n=1000

Note that the feed is limited to a maximum of 1000 items and you can only access it for your account if you are logged in. This means:

If you have more than a 1000 items you need to find the continuation token in each set of results and then at &c={continuation-token} to your query.
Because you need to be logged in your browser you need to do this by hand :-( (it may be possible to automate via the API but I couldn't get anything work - any tips much appreciated!)

Here's a concrete example (note, as you need to be logged in this won't work for you):

http://www.google.com/reader/atom/feed/http://search.twitter.com/search.atom%3Fq%3D%2523OpenData?n=1000

And that's it! You should now have a local archive of all your tweets!

Appendix

Increasing Twitter is selling access to the full Twitter archive and there are a variety of 3rd services (such as Gnip, DataSift, Topsy and possibly more) who are offering full or partial access for a fee.

Bundes-Git – German Laws on GitHub

2012-12-13T00:00:00-08:00

If you compare software code and legislation you can find many similarities: both are big bodies of text spread over multiple units (laws/files). The total amount of text inevitably grows bigger over time with many small changes to existing parts while most of the corpus stays the same.

However, the tooling and editing process for these domains is very different: while developers are in the fortunate position that they can build and improve their own tools, legislators are stuck with proprietary tools like MS Word that are simply not built to collaboratively work on a big corpus of text.

But if source code and laws have a similar information structure, why not apply the tools used in software development to the legislative process? That is what Bundes-Git ("Federal Git") is currently trying out in Germany.

Bundes-Git is a Git version control repository of all German Federal Laws and Regulations as Markdown. The goal was to come up with the simplest solution to handle laws that could possibly work and integrate it well into the existing developer ecosystem.

The idea has been well received with an article on Wired.com and articles on German IT news sites Heise and Golem.

The popularity can surely also be attributed to our marvelous Bundes-Git mascot, dubbed octo eagle, thought up by myself and designed by Konstantin Käfer released under CC0 (please go this way if you are interested in a t-shirt or hoodie).

Design decisions explained

All other law storage formats use XML. But to me XML is neither human readable nor human writable. Let me get into the details of some of the design decisions:

Git because it's the most popular distributed version control system right now.
GitHub because it's the most popular Git host right now and comes with some nice perks like Pull Request and GitHub Pages.
Markdown because any more structure like XML or JSON would make it harder for humans to read or write the format and diffs would be difficult to read.
Naming files index.md because it works nicely with Jekyll and GitHub Pages renders all laws into a currently very simple page.
YAML Front Matter is necessary for Jekyll but also serves as nice a meta data store on laws.
Committing from branches with non-fast-forward merges because... uhmm. This is really up for discussion. I want to keep track of where changes originate and branches are created for each law publication but this heavily diverts from the clean commit history philosophy that e.g. the Linux kernel lives by.

There are some more software development concepts that can be applied to the legislation process. Here are some fun things I'd like to try:

A prose.io-like editor to easily create law proposals and make a pull request.
Measuring the complexity of corpus/laws/paragraphs and using Travis CI to test pull requests if they make the complexity worse. Pattern is a Python NLP library and they recently released a German module which I want to try on our laws.
Testing foreign key integrity: are all referenced paragraphs still available?
Create an informative visualization out of the Git log automatically like Gregor Aisch did by hand for the German political party law.
Let the German president sign off on commits to master.

The design decisions around Bundes-Git fit nicely into the Git/GitHub eco system but they are not set in stone. They also create some problems and annoyances that need to be fixed or circumvented. While I believe the general philosophy and the freshness of the approach is the right direction, we clearly need more discussion.

Future happenings around Bundes-Git:

We applied for funding at Testing 123 Global Integrity Innovation Fund. Bundes-Git definitely fits their criteria of brand new, innovative and high-risk. The decision will be made later this month, fingers crossed!
I will talk at the 29th Chaos Communication Congress about Bundes-Git.
There will be Bundes-Git Hacker Meetup in mid January. If you are interested, sign up here.

We decided that the language of discussion on GitHub will be German, but feel free to start a conversation on the OKF Open Legislation mailing list.

Also be sure to follow @bundesgit on Twitter!

Speeding Up Your PyBossa App

2012-12-12T00:00:00-08:00

Thanks to the free crowd-crafting tool PyBossa, nowadays the biggest challenge for successful crowd-sourcing is engaging users for participating in tasks, and to keep that motivation at a high level over time. Therefor the user experience of crowd-sourcing apps plays a crucial role.

After participating in quite a few tasks myself, I found that the loading time in between two tasks was the most annoying thing. Doing crowd-sourcing tasks often feels like doing something stupid, and you really want to get things done as fast as possible. Sometimes it needs just a single click to solve a task, but then it takes seconds to load the next one.

This is because all existing apps where designed in a synchronous fashion. The client requests a new task and presents it to the user as soon as it has been loaded. After the user has solved the task, the result is submitted and after the result has been stored a new task is requested and so on.

(click to enlarge)

Some apps even need to load additional information, such as images or data coming from external APIs. This loading time accumulates quickly, and will most probably lower the motivation of your users!

Pre-loading subsequent tasks == magic

The idea for reducing the loading time is actually pretty simple: We let the app load the next task while the user is solving the current one. This results in a parallel process as described in the following chart:

To implement this in PyBossa, we needed to change the PyBossa API a little bit (thanks @teleyinex). Before that change consecutive calls to the newtask endpoint would return the same task again and again, until the user has solved it. Now with the newly introduced parameter offset you can request the next tasks in line.

Another requirement for pre-loading of tasks is to keep the entire app on one page as otherwise the cached task would be lost. The rest of this post describes a smart way to implement this using jQuery.Deferred.

Smart implementation using jQuery.Deferred

Looking from our PyBossa app, the pre-loading of the next task and the user solving the current one are two asynchronous actions running in parallel. We have to wait until both are completed before we can proceed to the next task.

This article reminded me of a smart way to implement this using jQuery.Deferred. The following function shows everything we need for our main loop.

function run(task) {
    var nextLoaded = loadTask(1),
        taskSolved = presentTask(task);
    $.when(nextLoaded, taskSolved).done(run);
}

To start the loop, we need to load the first task and pass it to run.

loadTask().done(run);

Now let's take a look at loadTask(). The parameter offset is passed to the API. After the task and everything else we might need is loaded we mark the deferred as resolved and pass the task over the done handler. Finally we return a 'locked' version of the deferred object.

function loadTask(offset) {
    offset = offset || 0;
    var taskLoaded = $.Deferred();
    $.getJSON('/api/app/'+appid+'/newtask?offset=' + offset, function(task) {
        // load more data if you need
        // and then, resolve Deferred
        taskLoaded.resolve(task);
    });
    return taskLoaded.promise();
}

We can use exactly the same method to model the user action. Therefor presentTask() will returned a deferred object, too. It gets resolved as soon as the user has solved the task and the answer is correctly submitted to PyBossa.

function presentTask(task) {
    var taskSolved = $.Deferred();
    // update presenter html
    $('.question').html(task.question);
    // wait for user action
    $('button.submit').off('click').on('click', function() {
        var answer = { foo: "Bar" }; // fetch answer from UI
        pybossa.saveTask(task.id, answer).done(function() {
            taskSolved.resolve();            
        });
    });
    return taskSolved.promise();
}

And that's it.

This method will significantly speed up your PyBossa app, especially if you need to fetch data from third party APIs. Remind yourself that even a speedup of a few seconds is a huge benefit for your voluntary users, as they are likely to go through this process quite often. And you really don't want to waste their time, do you?

Update: Why not try the FlickrPerson demo app the speedy way?

Javascript Timeline Libaries - A Review

2012-12-04T00:00:00-08:00

This post is a rough and ready overview of various javascript timeline libraries that arose from research in creating a timeline view for Recline JS. Note this material hung around on my hard disk for a few months so some of it may already be a little bit out of date!

I want to start with a general comment. Timeline libraries consist of various components:

Data loading
- Date parsing
Band (timeline) rendering
Showing render info on individual items

For me a timeline visualization library need only be the second of these but most that I've come across do more.

In fact a major issue in my opinion with most libraries is that they are under-componentized - they don't separate cleanly into these different components and end up doing everything.

To take one example, the Verite timeline (in my view is one of the best libraries out there) has a whole bunch of its own custom date parsing built in inside an internal utility library which are hard to override or replace and also has a large chunk of code just for loading from google docs and other data sources. (You can of course somewhat solve this somewhat -- as I do in Recline by parsing the dates directly and then submitting in a standardized form).

In my view, even if library authors do want to include these sorts of things, it would be good to do it in a way that allowed for a clean separation so that you could just use the parts you wanted (and/or over-ride parts more cleanly).

Propublica Timeline Setter

http://propublica.github.com/timeline-setter/
HTML + JS
- But Requires a build step (using ruby)
Very simple and compact design (nice!)

Verite Timeline

http://timeline.verite.co/
Very elegant frontend design
2 bands in timeline segment and tight integration of item display
Includes much more than Timeline (e.g. sourcing data from google docs etc)
Mozilla Public License (was GPL)

Simile Timeline

http://www.simile-widgets.org/timeline/
The original open-source JS timeline but less regularly update and maintained today: "As of Spring 2012, Exhibit is the only Simile widget seeing active development." and the timeline control has not been updated since 2009 (see this stackoverflow question for more

Chronoline

http://stoicloofah.github.com/chronoline.js/
Recently developed and updated
MIT licensed

Timeglider

https://github.com/timeglider/jquery_widget
Non-open license (but was MIT licensed earlier on

CHAPS Timeline

http://almende.github.com/chap-links-library/timeline.html
Looks pretty nice though CSS is not quite as elegant (probably fixable!)
Not clear whether it supports multiple bands

Following Money and Influence in the EU - the Open Interests Hackathon

2012-11-29T00:00:00-08:00

Making sense of massive datasets that document the processes of lobbying and public procurement at European Union level is not an easy task. Yet a group of 25 journalists, developers, graphic designers and activists worked together at the Open Interests Europe hackathon last weekend to create tools and maps that make it easier for citizens and journalists to see how lobbyists try to influence European policies and to understand how governments award contracts for public services. The hackathon was organised by the European Journalism Centre and the Open Knowledge Foundation with support from Knight-Mozilla OpenNews.

At the Google Campus Cafe in Londonndon, one group dived into European lobbying data made available via an API: api.lobbyfacts.eu. Created by a group of five NGOs: Corporate Europe Observatory, Friends of the Earth Europe, Lobby Control, Tactical Tech and the Open Knowledge Foundation, the API gives access to up-to-date, structured information about persons and organisations registered as lobbyists in the EU Transparency Register. The API is part of lobbyfacts.eu, a website that aims to make it easy for anyone to track lobbyists and their influence at European Union level, due to launch in January 2013.

One of the projects Createdd with the lobby register data is a map showing the locations of the offices of lobby firms based on their turnover. The size of the bubbles on the map corresponds to the turnover of the firm. Built by Friedrich Lindenberg, the map is an overlay of a Stamen Design map with Leafletjs.

Screenshot of api.lobbyfacts.eu/map showing locations of lobbying firms across Europe

Other teams focused on data analysis, comparing the data from the EU Transparency Register with that of the Register of Expert Groups. Interesting leads for possible further investigative work resulted from the comparison of the figures reported by lobby firms in the Transparency Register with those collected by the National Bank of Belgium. “Some companies underreported massively to the National Bank of Belgium and some of them were making themselves look bigger in the Transparency Register,” said Eric Wesselius, leader of the lobby transparency challenge and co-founder of Corporate Europe Observatory. Wesselius’ organisation will continue investigations in this area.

A second group of journalists and graphic designers led by Jack Thurston, an activist involved in Fishsubsidy.org, discussed how fish subsidy data could be used for finding journalistic stories and explored various ways in which the unintended consequences of the EU fish subsidies programme, such as overfishing, could be compellingly presented to the general public.

Sketch for interactive graphic showing fishing vessels, their trajectory and the subsidies they receive, made by graphic designer Helene Sears

A theyhird group looked into European public procurement data. “Public procurement is an area that is underreported by journalists,” said data journalist Anders Pedersen, founder of OpenTED. “9-25% of the GDP in the EU is procurement - highest in the Netherlands where it is around 35%. It’s a real issue in times of austerity who provides our services,” he added.

Several scrapers were built to access the data relating to winners of contracts and the values of these contracts from the EU publication TED (Tenders Electronic Daily). A map of public procurement contracts by awarding city was created using Google Fusion Tables by geocoding the original CSV file, enriched with OpenStreetMap.

Screenshot of map of public procurement contracts by Benjamin Simatos and Martin Stabe

Pedersen’s long term goal is to create an interface and an API for EU public procurement data and to publish some more visualisations. “A lot of the work that got done here [at the hackathon] we would not have gotten done in the next months maybe. It really helped us push far ahead in terms of ideas and in terms of getting stuff done.”

This blog post is cross-posted from the Data-driven Journalism Blog.

Photo of participants at the hackahon by Mehdi Guiraud.

Scraping Data Behind a CAPTCHA

2012-11-13T00:00:00-08:00

How much does the highest paid person in the Brazilian Federal Senate earns? That's the question I asked myself a few weeks ago, and one that should be easy to answer. In Brazil, every public body must publish its employees' salaries online, but some do so in a terrible way. The Federal Senate is one of these.

To access its data you have to not only fill in your personal info, but also solve a CAPTCHA for each salary you want to see. With no other tricks, it would take ages to answer my question. I needed a way to gather all salaries and compare them. But how to scrape a page that's "protected" behind a CAPTCHA?

Decaptcher is a company that sells CAPTCHA-solving services. They provide an API that you can send an image, and get the contained text. It's really cheap (US$ 1.38 per 1.000 CAPTCHAs), and works well, albeit a bit slow (30~40 secs). They promise a success rate of over 95%, but I got only 43% in my tests. Probably because the CAPTCHAs I'm sending are really hard to read.

Their API is simple to implement, with only 3 actions (upload, refund, and balance). There're examples in C# and PHP, and I've hacked together one in Ruby. For a bit more than US$ 5.92, I was able to access and publish the salaries of 4,487 public servants in http://senado.cc.

There're many other companies that offer the same service, like Death by CAPTCHA, Bypass CAPTCHA, Beat CAPTCHA, and Antigate. These services allow us to access public data that would be unreachable otherwise, but they might be considered illegal in some countries. As we're not breaking the CAPTCHA, but paying people to solve them, we should be fine. But don't take my word for it: ask a lawyer.

Recline JS Search Demo

2012-11-01T00:00:00-07:00

We've recently finished a demo for ReclineJS showing how it can be used to build JS-based (ajax-style) search interfaces in minutes (or even seconds!): http://reclinejs.com/demos/search/

Because of Recline's pluggable backends you get out of the box support for data sources such as SOLR, Google Spreadsheet, ElasticSearch, or plain old JSON or CSV – see examples below for live examples of using different backends.

Interested in using this yourself? The (prettified) source JS for the demo is available (plus the raw version) and it shows how simple it is to build an app like this using Recline – plus it has tips on how to customize and extend).

More Examples

In addition to the simple example with local data there are several other examples showing how one can use this with other data sources including Google Docs and SOLR:

A search example using a google docs listing Shell Oil spills in the Niger delta
A search example running of OpenSpending SOLR API – we suggest searching for something interesting like "Drugs" or "Nuclear power"!

Code

The full (prettified) source JS for the demo is available (plus the raw version) but here's a key code sample to give a flavour:

// ## Simple Search View
//
// This is a simple bespoke Backbone view for the Search. It Pulls together
// various Recline UI components and the central Dataset and Query (state)
// object
//
// It also provides simple support for customization e.g. of template for list of results
// 
//      var view = new SearchView({
//        el: $('some-element'),
//        model: dataset
//        // EITHER a mustache template (passed a JSON version of recline.Model.Record
//        // OR a function which receives a record in JSON form and returns html
//        template: mustache-template-or-function
//      });
var SearchView = Backbone.View.extend({
  initialize: function(options) {
    this.el = $(this.el);
    _.bindAll(this, 'render');
    this.recordTemplate = options.template;
    // Every time we do a search the recline.Dataset.records Backbone
    // collection will get reset. We want to re-render each time!
    this.model.records.bind('reset', this.render);
    this.templateResults = options.template;
  },

  // overall template for this view
  template: ' \
    <div class="controls"> \
      <div class="query-here"></div> \
    </div> \
    <div class="total"><h2><span></span> records found</h2></div> \
    <div class="body"> \
      <div class="sidebar"></div> \
      <div class="results"> \
        } \
      </div> \
    </div> \
    <div class="pager-here"></div> \
  ',
 
  // render the view
  render: function() {
    var results = '';
    if (_.isFunction(this.templateResults)) {
      var results = _.map(this.model.records.toJSON(), this.templateResults).join('\n');
    } else {
      // templateResults is just for one result ...
      var tmpl = '' + this.templateResults + ''; 
      var results = Mustache.render(tmpl, {
        records: this.model.records.toJSON()
      });
    }
    var html = Mustache.render(this.template, {
      results: results
    });
    this.el.html(html);

    // Set the total records found info
    this.el.find('.total span').text(this.model.recordCount);

    // ### Now setup all the extra mini-widgets
    // 
    // Facets, Pager, QueryEditor etc

    var view = new recline.View.FacetViewer({
      model: this.model
    });
    view.render();
    this.el.find('.sidebar').append(view.el);

    var pager = new recline.View.Pager({
      model: this.model.queryState
    });
    this.el.find('.pager-here').append(pager.el);

    var queryEditor = new recline.View.QueryEditor({
      model: this.model.queryState
    });
    this.el.find('.query-here').append(queryEditor.el);
  }
});

Labs Show and Tell - 26th October!

2012-10-23T00:00:00-07:00

We're having the next Show and Tell on Friday, 26 October at 2:30 pm BST via Google Hangout on Air. As usual, the URL will be posted on OKFN Labs' G+ Page.

If you'd like to present, add your name to the list. Remember, #okfn on irc.freenode.net will be the backchannel for discussion and questions, so don't forget to hang out there.

What's Show and Tell?

Have you built some cool tech you want to show everyone? Played around with some data? The Labs Show and Tell is your chance to share it with the OKFN Labs community! You get 2 to 5 minute to show us what you built!

Missed the last one?

On Oct 12, 2012, we had the first Labs Show and Tell. Here's what we talked about:

Scientific Promiscuity - Michael Bauer

Scientific papers are rarely written by a single person. Usually many authors come together to work on a specific issue. This visualization uses data obtained from Pubmed to show collaboration between authors.

Activity API - Code - Tom Rees

Activity API scrapes through multiple data sources and creates one single PostgreSQL database with all the data. It scrapes through GitHub, Twitter, mailing lists posts, and Twitter.

Dashboard - Code - Tom Rees

The OKFN Community Dashboard provides an overview of community activity. We have a flourishing and diverse set of activities and it can be hard, even for people 'inside' to see what is going on. The Dashboard helps us see quickly what is going on.

nomenklatura - Code - Friedrich

A lot of time in data wrangling is spent making mappings of variant names to a canonical form. This app provides an easy-to-use, web-based method for creating such mappings, to allow for a more managed data cleansing pipeline.

Messy Tables - Friedrich

A library for dealing with messy tabular data in several formats, guessing types and detecting headers.

froide - stefanw

Froide is a Freedom Of Information tracker. The name comes from Freedom of Information (de). Also Froide sounds like Freude which is German for joy.

PyBossa on Travis CI - Nigel

PyBossa now uses Travis CI for continuous integration. Makes reviewing pull requests easier since we can see test status right away.

Wrangling dirty data with messytables.

2012-10-22T00:00:00-07:00

One of the largest data collection projects we have done so far has been the consolidation of the UK's departmental expenditure. Over 370 different government entities have published a total of more than 7000 spreadsheets. Many of those have obviously been hand-crafted or at least manually processed. Our goal was to consolidate the contained information into a single spreadsheet, discarding all the eccentricities included by the individual publishers.

messytables is a simple Python library that tries to extract tabular contents from spreadsheet documents created by human editors. Often, even files released as CSV or Excel are still not easy to parse programmatically. Some people like to start off spreadsheets with a title column or some metadata, while others use inapproriate formats to represent numbers or dates.

The tool offers a set of functions that help to make parsing data easier:

A headers detector tries to determine which row in a spreadsheet contains the actual header definitions (as opposed to any trailing content).
type detection attempts to guess the data type for each column, including a wide range of commonly used date formats.
support for streaming data, so that extremely large tables can be processed without loading the entire data into memory.
and, of course, it supports a range of spreadsheet types - from trusty CSV to Excel and even OpenOffice formats.

We've since also started using messytables to load data into the data API of CKAN, where it serves as the ETL for the datastore and related ReclineJS previews.

If you're interested, check out the messytables documentation and the uk25k scripts which use it to gather UK government finance.

Of course, messytables is not a cure-all and only useful for reading data.

tablib, for example, has a fantastic API that makes writing, analyzing and converting data a breeze.
csvkit has a set of command line utilities that should be pre-installed on any computer.

But when it comes to tables that are a complete mess: give it a try!

Open Interests Hackathon in London, 24-25 November

2012-10-15T00:00:00-07:00

The European Journalism Centre and the Open Knowledge Foundation, sponsored by Knight-Mozilla OpenNews, invite you to the Open Interests Hackathon to track the the interests and money flows which shape European policy.

When: 24-25 November

Where: Google Campus Cafe, 4-5 Bonhill Street, EC2A 4BX London

How EU money is spent is an issue that concerns everyone who pays taxes to the EU. As the influence of Brussels lobbyists grows, it is increasingly important to draw the connections between lobbying, policy-making and funding. Journalists and activists need browsable databases, tools and platforms to investigate lobbyists’ influence and where the money goes in the EU. Join us and help build these tools!

Open Interests Europe brings together developers, designers, activists, journalists and other geeks for two days of collaboration, learning, fun, intense hacking and app building.

Visit the event page to learn more

Labs Show and Tell - All Welcome!

2012-10-10T00:00:00-07:00

Built an app or tool you want to show people? Played around with some interesting data? Know of a new development people should know about? Want to find out what others are doing?

Come to the Show and Tell this Friday and share what you are up to with the community!

Sign up

Want to participate? Just add your name to the list on the etherpad! If you want to present just add a brief title and/or short description.

Remember, #okfn on irc.freenode.net will be the backchannel for discussion and questions, so feel free jump in there if you have questions or queries or just want to shoot the breeze.

When?

Friday, 12 October at 2:30 pm BST - that's 10:30am EST, 3:30pm CET etc. The session will last 30m with presentation slots of 2-5m.

Where?

Google Hangout on Air and #okfn on irc.freenode.net. We'll post the on air URL on OKFN Labs' G+ Page, here and on OKFN Labs twitter.

Data Catalogues are People!

2012-09-25T00:00:00-07:00

Last week, Matej Kurian published a message on the okfn-labs mailing list, describing the various sources he had discovered for machine-readable excerpts of the EU's joint procurement system, TED. What struck me about this message was that, apparently, this polite and brilliant policy wonk had turned into something strange: into a data catalogue.

While not quite a Kafka-grade transformation, it's an odd turn to take for a researcher. But Matej is not the only one: the team of FarmSubsidies.org has experienced a similar re-definition, as did the ERDF researchers at the Bureau of Investigative Journalists.

The best data catalogues today are well-informed people.

When I talk to journalists about data acquisition, they seem to know this already: its often not just about where to look; it's even more important to know who to talk to. But why does this observation from a telephone-and-filofax world hold true even in digital space, where every bit of knowledge is supposed to be only a click away?

I believe that some blame goes to the simplistic model underlying our efforts to catalogue data: the question of where to find a dataset is certainly important, but for those actually working with the data it's just not enough. Once you dig into data, other questions rise to the foreground:

How do the different available datasets interact and integrate? Does the data I am looking for even make sense on its own - or do I need to combine several sources? Take, for example, the UKs Whole of Government Accounts: while data.gov.uk lists a few gigabytes worth of downloads for this dataset, it is completely impossible to interpret the data without also fetching Excel files (and PDF guidance) off the Treasury web site, the Department of Communities and Local Government site and - bonus points - emailing the Treasury for their internal toolkit.
How complete and up-to-date is the data? What technical and political constraints apply to the publication? Again, FarmSubsidies provide a nice example, as a 2010 European Court of Justice verdict has severely limited the availablity of the data - leading to an oddly limited dataset today.
Who else is working with this data and what are they doing? Are there derivative datasets that I should use instead of the source material? It may be worth knowing, for example, that as well as browsing the 6000-odd departmental spending spreadsheets, journalists can also search across a consolidated version of this data on OpenSpending.org

But why are current data portals so bad at capturing such information? Certainly, adding a few comment boxes and an app gallery can do a good job glossing over the problem, but the real problems seem to lie deeper in the technology:

Datasets are a useless unit. A while ago, Richard Cyganiak defined a dataset as "a set of data" - which I assume is a computer scientists way of telling you to get lost. And while I'm not normally a big fan of LOD-clouds, they got this right: all the interesting stuff is happening in between datasets. Whether it's about reconstructing a process across several datasets or finding out about geographical and temporal coverage - datasets are at best building blocks, more often they are just arbitrary. So maybe its time to think about other mechanisms to represent data sources: what about policy maps and government wiring plans?
Even worse, the metadata we keep about datasets is mostly based on a bureaucratic mindset: they're library-inspired, static index cards that hope to represent datasets, while data are really subject to complex processes both within and outside the institutions that produce them. For anyone using the data, activity metadata is the interesting part. We've already figured this out for software, where libraries like FreshMeat and SourceForge have been replaced by activity-driven platforms like GitHub. The key aspect here is that GitHub doesn't require me to explictly make metadata - the relevant narrative is simply summarized from my working pattern.

Of course, all of this is just a long way of saying that the best metadata is in the data itself. So unless you're working on the LHC stuff there really isn't much of a reason to separate the two any longer: let's make public, audit-trailed databases that report on themselves. This, of course, is easier said then done as it implies that all data will fit into one storage mechanism. In the real world (i.e. outside Linked Data land), this is unlikely to be true of structured data any time soon.

Still, even after fixing our model of how we talk about datasets on the web, I think we would still find that the best way to ensure that people collaborate around data is community-building: creating networks that garden the commons. Perhaps we should start cataloguing those.

WikipediaJS - accessing Wikipedia article data through Javascript

2012-09-10T00:00:00-07:00

WikipediaJS is a simple JS library for accessing information in Wikipedia articles such as dates, places, abstracts etc.

The library is the work of Labs member Rufus Pollock. In essence, it is a small wrapper around the data and APIs of the DBPedia project and it is they who have done all the heavy lifting of extracting structured data from Wikipedia - huge credit and thanks to DBPedia folks!

Demo and Examples

A demo is included and you can see some examples of the library in action at the following links:

Colophon

WikipediaJS source code is on github

One of the reasons for creating WikipediaJS is that we think it can be useful in Timeliner and other apps as a way to quickly add new items to your timeline.

Timeliner - Make Nice Timelines Fast

2012-08-08T00:00:00-07:00

As part of the Recline launch I put together quickly some very simple demo apps one of which was called Timeliner:

http://timeliner.reclinejs.com/

This uses the Recline timeline component (which itself is a relatively thin wrapper around the excellent Verite timeline) plus the Recline Google docs backend to provide an easy way for people to make timelines backed by a Google Docs spreadsheet.

As an example of use, I started work on a "spending stories" timeline about the bankruptcy of US cities (esp in California) as a result of the "Great Recession" (source spreadsheet). I've also created an example timeline of major wars, a screenshot of which I've inlined:

Code

Source code for the Timeliner is here: https://github.com/okfn/timeliner

If you have suggestions for improvements, want to see the ones that already exist, or, gasp, find a bug please see the issue tracker: https://github.com/okfn/timeliner/issues

The Data Transformer - Cleaning Up Data in the Browser

2012-07-31T00:00:00-07:00

This a brief post to announce an alpha prototype version of the Data Transformer, an app to let you clean up data in the browser using javascript:

http://transformer.datahub.io/

2m overview video:

What does this app do?

You load a CSV file from github (fixed at the moment but soon to be customizable)
You write simple javascript to edit this file (uses ReclineJS transform and grid views + CSV backends -- here's the original ReclineJS transform demo)
You save this updated file back to github (via oauth login - this utilizes Michael's great work in Prose!)

This prototype was hacked together in an afternoon a couple of weeks ago when I was fortunate enough to spend an an afternoon with Michael Aufreiter, Chris Herwig, Mike Morris and others at the Development Seed offices. It builds on ReclineJS + oauth / github connectors borrowed from Prose.

It's part of an ongoing plan to create a "Data Orchestra" of lightweight data services that can play nicely together with each other and connect to things like the DataHub (or GitHub ...): http://notebook.okfn.org/2012/06/22/datahub-small-pieces-loosely-joined/

Displaying PyBossa Urban Parks Data on a 3D Globe

2012-07-14T00:00:00-07:00

Labs member Daniel Lombraña González has built a 3-d globe showing the locatoins of urban parks around the world as located by volunteers using the Pybossa Urban Park geocoding app:

http://teleyinex.github.com/pybossa-urbanpark-globe/ — (Source code)

Background

The Urban Parks geo-coding application is a micro-tasking app running on PyBossa. In the app volunteers are asked to find an urban park for cities around the world. The volunteers use a web map to browse the city, and then submit an answer: the coordinates of the urban park by placing a marker in the map, or saying: I don't find any park.

More details about PyBossa can be found on the official site http://pybossa.com and also in the online documentation.

dataissues.org - public issue tracking for data defects

2012-07-10T00:00:00-07:00

On June 21st, the Knight News Challenge Round on Data ended. The day before, Rufus, Ross and I sat down to write out some ideas that we'd been discussing for a while. While we submitted proposals for Grano and DataProtocols, we decided to hold back on this idea for another round. Still, sharing is caring.

1. What do you propose to do? [20 words]

We’ll create a web service where data wranglers and consumers can log errors arising from processing, viewing or using data.

2. How will your project make data more useful? [50 words]

All data has errors. While data quality is often talked about, the best practice for data apps is often to have half a paragraph on the ‘about’ page. We want to build a service that is useful to data wranglers, but can also serve as documentation for end-users and basis for further discussion.

3. How is your project different from what already exists? [30 words]

Error reporting for software is either done as task tickets (e.g. github.com) or by capturing raw application output (e.g. exceptional.io). For data, we want to combine these two approaches to let users group recurring errors into issues that can then be discussed and fixed.

4. Why will it work? [100 words]

While all data processing workflows are different from dataset to dataset, the types of errors that occur are often quite similar and can be stored in a shared service. This is both immediately useful when doing data work - especially scheduled, unsupervised processes - but also as an activity log for other people to see.

We’ll create both an easy-to-use online validation tool to check spreadsheets against a certain schema and an API with client libraries that can be integrated into existing processing pipelines. The reported issues can be full-out errors, but also probes that highlight implausible values.

5. Who is working on it? [100 words]

The Open Knowledge Foundation is...

6. What part of the project have you already built? [100 words]

We’ve got extensive experience working with dataset metadata from DataHub.io and produced a number of complex data processing pipelines (e.g. for UK spending data, that merges over 5000 spreadsheets in different formats). These clearly show the need for better reporting, and we have built several ad-hoc solutions but know that is a major area that is inadequately addressed in our work and those of others. We have already got a basic prototype and can build a first increment quickly.

7. How would you use News Challenge funds? [50 words]

We’ll built it! We’ll develop a full version of this service iteratively, test and promote it. We plan to work together with civic data projects as early adopters to get quick feedback and adapt the service to suit their needs.

8. How would you sustain the project after the funding expires? [50 words]

This will be perfectly suited to SaaS freemium model in which heavy and/or professional users who need to report large amounts of errors and generate complex reports pay a subscription fee. In addition as open-source software the project can be re-used and extended by others.

If you think this is a good idea, help hacking and contribute patches to the dataissues repository!

Grano - social network analysis for advocates and journalists.

2012-07-09T00:00:00-07:00

On June 21st, the Knight News Challenge Round on Data ended. The day before, Rufus, Ross and I sat down to write out some ideas that we'd been discussing for a while. The first idea I want to repost here is a proposal for Grano, which I've discussed in this blog before.

1. What do you propose to do? [20 words]

We’ll make a powerful tool for journalists and advocates to keep track of actors and their relationships in complex environments.

2. How will your project make data more useful? [50 words]

It’ll enable users to manage research in a structured way, helping them to link raw data to the actors, events and organisations they’re already investigating and to find those that they may have missed before. We’ll help users do their job more thoroughly, while creating a structure that can be re-used later.

3. How is your project different from what already exists? [30 words]

Network analysis means many things to people: it’s graph algorithms to coders, network diagrams to designers and CRM to business. Journalists and advocates need evidence gathering and information linkage to be at the core of these things.

4. Why will it work? [100 words]

We want to focus on four functions that will make this a practical tool instead of a gimmick:

a) allowing users to easily integrate bulk data to complement manually entered information,

b) helping them to keep track of the source for each fact that is entered and keeping a full version history,

c) providing easy access control so that users can choose which information to keep private and which links to publish with others and

d) text snippets, so that researchers can combine structured analysis and narrative fragments in which the tool will detect references to the network’s entities.

5. Who is working on it? [100 words]

The Open Knowledge Foundation wants to cooperate with investigative networks around the world to develop this project. We’ve already been pioneering data collection and presentation tools, such as DataHub.io and OpenSpending, as well as efforts like the Data Journalism Handbook and the School of Data to widen data literacy.

Friedrich Lindenberg (OKFN) has worked on several data projects and data-journalism training and will lead this project.
Ross Jones (OKFN) will contribute as a software architect.
Stefan Candea (2011 Nieman Fellow at Harvard, Director of the Romanian Center for Investigative Journalism) has offered to advise us.

6. What part of the project have you already built? [100 words]

We’ve already built Grano, a REST backend that can store network information, generate custom reports about nodes and relations and run full-text search. Because we think that meaningful network analysis is hard, we are conservative in the choice of technology to focus on outcomes. To force that, we decided to base our tool on a concrete use case. The software is now first used in an unannounced project that tracks lobbying in the EU, powering a special-purpose, JavaScript-only site. Unfortunately, this means the current prototype does not have a stand-alone web interface and the serious data integration capabilities we think it needs.

7. How would you use News Challenge funds? [50 words]

We want to develop Grano to give investigative journalists and civic hackers a (re-usable) web interface to design their network structure, manually enter data, integrate bulk data sets and to explore the resulting network, make notes, calculate key metrics and to export reports, rankings and network visualizations.

8. How would you sustain the project after the funding expires? [50 words]

While the service is going to be of immediate use, we believe that advocacy groups and newsrooms will also deploy it as a backend to their features and campaign sites. We aim to make Grano into a thriving open source project, supported through custom services for power users.

If you like this idea, please vote for it on the proposal page.