A Data API for Data Packages in Seconds Using CKAN and its DataStore

Posted on 11 September 2014 by Rufus Pollock

dpm the command-line ‘data package manager’ now supports pushing (Tabular) Data Packages straight into a CKAN instance (including pushing all the data into the CKAN DataStore):

dpm ckan {ckan-instance-url}

This allows you, in seconds, to get a fully-featured web data API – including JSON and SQL-based query APIs:

dpm ckan demo

View fullsize

Once you have a nice web data API like this we can very easily create data-driven applications and visualizations. As a simple demonstration, there’s the CKAN Data Explorer (example with IMF data - see below).

Where Can I Find a CKAN instance to Upload to?

If you’re looking for a CKAN site to upload your Data Packages to we recommend the DataHub which is community-run and free. To upload to the DataHub you’ll want to.

Configure the DataHub CKAN instance in your .dpmrc

[ckan.datahub]
url = http://datahub.io/
apikey = your-api-key

Upload your Data Package
```
dpm ckan datahub --owner_org=your-organization
```
You have to set the owner organization as all datasts on the DataHub need an owner organization.

One I Did Earlier

Here’s a live example of one “I did earlier”:

Here’s the source Data Package: IMF World Economic Outlook in data.okfn.org registry (Data Package on github (source))
Get this on your local machine (dpm install or just clone the github repo)
Then I uploaded it: dpm ckan http://datahub.io/ --owner_org=rufuspollock
Now it’s live on the DataHub: http://datahub.io/dataset/imf-weo
- Indicators: http://datahub.io/dataset/imf-weo/resource/ea3926e3-43a8-46d0-832a-e53efd61ebb0
- Values: http://datahub.io/dataset/imf-weo/resource/24cd8ebe-fa3f-4353-9ad9-d53bd88751a6
- Note this is a normalized dataset in which there are 2 tables (the DataStore supports JOINS if we want to put them back together)
Here’s a sample API query to get all indicators related to GDP: http://datahub.io/api/action/datastore_search?resource_id=ea3926e3-43a8-46d0-832a-e53efd61ebb0&limit=5&q=GDP
Now the data has a nice web Data API you can easily build data-driven apps or visualizations. For example, the CKAN Explorer is a simple JS + HTML app which allows you to explore CKAN DataStore data. Here’s the app pre-loaded with the DataStore indicator data

Context: a big motivation (personally) for doing this is that I’d like to see a nice web data API available for the “Core” Data Packages we’re creating as part of the Frictionless Data effort. If you’re interested in helping, get in touch.