IDigBio Download API

Overview

Note: While the download API is currently used by the production portal, it should be considered highly unstable for non-iDigBio consumers. The next revision of the API will most likely be a total rewrite, backend and front.

The download API may not provide "friendly" error messages at this time.

The Download API works by performing the requested query and building a Darwin Core Archive. Once archive generation has begun, the status endpoint can be polled to determine if the generation has been completed. Once the archive generation is completed, the API provides a link to the file for download. If the optional email parameter is supplied on the query request, an email notification will be sent that includes a link to the downloadable file.

Large queries (and thus large archive file creation) can take multiple hours to complete.

A query submitted as a GET request must be URL-encoded.

A query submitted as a POST request must be supplied as JSON in the content body and specify the "Content-Type: application/json" request header.

Query Endpoint

The download service query url:

https://csv.idigbio.org/?query={Query in iDigBio query format}[&email={valid email address}]

See iDigBio query format for more information on writing queries.

The "email" parameter is optional. Specifying a valid email address will cause an email notification to be sent.

A successful request to the query endpoint will return a JSON document that includes a "complete" status flag and a "status_url" link. The "status_url" is a link to the Status Endpoint (see below) which can safely be polled until "complete" is "true".

Status Endpoint

The download service status url:

http://csv.idigbio.org/status/{status uuid}

Using the "status_url" that was returned in the JSON from the query endpoint, a successful request to the status endpoint will return a JSON document that includes a number of fields including "complete" which is a status flag and "download_url" which, once the generation is completed, is a link to the generated archive.

Query Example - genus acer with email

Consider the following query:

{ "genus" : "acer"}

We could request a download and notification sent to email address "donotreply@idigbio.org" via the following url:

https://csv.idigbio.org/?query=%7B%22genus%22%3A%22acer%22%7D&email=donotreply%40idigbio.org

When the download file is generated, the "complete" field will be set to "true" and the "download_url" field will include a link to the available file.

Using curl (output formatted with JSON "prettify"), we can see that a completed archive is available at the download_url:

$ curl "https://csv.idigbio.org/?query=%7B%22genus%22%3A%22acer%22%7D"
{
   "download_url" : "http://s.idigbio.org/idigbio-downloads/c5560080-23ce-42eb-a1d8-d663f31c77b8.zip",
   "query_hash" : "c91957c034f2a8c42796584c27ba3416acf81d94",
   "query" : {
      "genus" : "acer"
   },
   "status_url" : "http://csv.idigbio.org/status/c5560080-23ce-42eb-a1d8-d663f31c77b8",
   "task_status" : "SUCCESS",
   "complete" : true,
   "expires" : "2015-04-28T13:13:09.187258"
}

which is the same link that would be included in the "iDigBio Download Ready" email.

If we follow the "status_url" in the above we see similar information:

$ curl -s "http://csv.idigbio.org/status/c5560080-23ce-42eb-a1d8-d663f31c77b8"
{
   "status_url" : "http://csv.idigbio.org/status/c5560080-23ce-42eb-a1d8-d663f31c77b8",
   "complete" : true,
   "query" : {
      "genus" : "acer"
   },
   "expires" : "2015-04-28T12:28:46.393070",
   "task_status" : "SUCCESS",
   "query_hash" : "c91957c034f2a8c42796584c27ba3416acf81d94",
   "download_url" : "http://s.idigbio.org/idigbio-downloads/c5560080-23ce-42eb-a1d8-d663f31c77b8.zip"
}

Status Endpoint Example

Given the following query JSON:

{ "scientificname" : "puma concolor" }

becomes the following when url-encoded:

https://csv.idigbio.org/?query=%7B%22scientificname%22%3A%22puma+concolor%22%7D

Immediately after the query is run, the "complete" flag is still set to false. There is no archive available for download (yet).

{
  "complete": false,
  "status_url": "http://csv.idigbio.org/status/cba4ae0f-da2b-42ec-b763-132a209c3251",
  "expires": "2015-04-28T11:46:54.562842",
  "query_hash": "5921ce268fe0d911196a4564eea8ce9ffc2e2420",We
  "query": 
    {"scientificname": "puma concolor"},
  "task_status": "PENDING"
}

If we wait a while and try again, the status changes, and the file is available at the provided download_url:

{
  "complete": true,
  "status_url": "http://csv.idigbio.org/status/cba4ae0f-da2b-42ec-b763-132a209c3251",
  "expires": "2015-04-28T11:46:54.401498",
  "download_url": "http://s.idigbio.org/idigbio-downloads/cba4ae0f-da2b-42ec-b763-132a209c3251.zip",
  "query_hash": "5921ce268fe0d911196a4564eea8ce9ffc2e2420",
  "query": 
    {"scientificname": "puma concolor"},
  "task_status": "SUCCESS"
}