Data Ingestion Guidance

Guidance To Data Providers When First Considering iDigBio Data Ingestion

Audience: Data Providers, iDigBio data ingestion staff

This is a process description for the iDigBio staff to follow to assure that data are successfully and efficiently moved from data provider to the portal, available for searching. Some of the information contained here gives details of what the staff are looking for from the provider in terms of input, metadata standards, and file formats (e.g., .csv, .jpg, DwC-A).

Contact Info

If you find yourself in need assistance, contact data@idigbio.org

Below are what we ask of the data to make it fit for use (e.g., easily searchable) in the cyberinfrastructure we provide:

Process Terminology

Processing steps, each step has a start and an end, signifying that it has moved to the next step.

• negotiating - in the process of determining provider's interest in data ingestion
  • begins with email invitation to providers (in institutions, aggregators) to invite them to send their data to iDigbio specimen data portal
  • open a Redmine ticket in category=Data Mobilizing
  • ends with data exported by provider, ready for inspection and ingestion
• mobilizing - in the process of evaluating data being fit for ingestion
  • begins with provider exported data and cursory inspection
  • ends with data passing inspection and passing to ingesting state, Redmine ticket changes to category=Data
• ingesting - in the process of ingesting provider's data
  • begins with Redmine ticket change to category=Data
  • ends with
    • data successfully ingested, ready for consumption
    • report sent back to data mobilizing staff
    • Redmine ticket set to Status= Closed
• evaluating - in the process of evaluating a failure to be ingested
  • begins with ingestion failure
    • evaluate ingestion failure, if data error - send it back to mobilizing state for corrections or
    • evaluate ingestion failure, if ingestion error - make corrections
  • ends with data re-submission to ingesting state

Data Requirements

There are 3 kinds of data to submit for ingestion:

1. specimen data with metadata
2. media data related to and attached by reference to specimen records with metadata
3. media files - e.g., non-archival, .jpg (see acceptable format here: https://www.idigbio.org/content/idigbio-image-file-format-requirements-and-recommendations)

Packaging for specimen data

In order of preference:

• IPT or Symbiota feed
• DwC-A
• custom CSV (save the data in UTF-8 format to preserve diacritics in people and place names)

Sending data to us

• an RSS feed for ready access and update is recommended
• email the files to us

Specimen metadata

• each specimen record needs to have a unique (within the dataset) identifier in the dwc:occurrenceID field
• if using a custom CSV,name the fields as close to Darwin Core as possible, in XML style, e.g., 'dwc:continent, and additionally make use of the MISC field names (local iDigbio extensions to DarwinCore)

Permission to Ingest

• you need to have permission to submit the data
• fill in the dataset intellectual property and licensing fields, and be sure to fill in your official organization code (see and update http://GRBio.org)

Data recommendations for optimal searchability

• put dates in ISO 8601 format, i.e., YYYY-MM-DD, e.g., 2014-06-22
• put elevation in METERS units in the elevation field without the units (e.g., the fields minimumElevationInMeters and maximumElevationInMeters already assume the numeric values are in meters, so no need to include the units with the data)
• no unescaped newline characters
• no '0' in fields to represent no value, e.g., lat or lon
• lat and lon coordinates need to be in decimal, and no N, S, E, W
• parse genus, species, infraspecific epithet if already aggregated into a scientific name
• include parsed higher taxonomy

Packaging for images/media objects

• each media record needs to have a GUID: a persistent globally unique identifier or at least a unique (within the dataset) identifier in the occurrenceID field.
  • if submitting media records with specimen data records, put the specimen data record into the ??? field
• we need there to be Audubon Core metadata file, with one record to go with each media record, and we can provide coaching to help you create that file. The more you can flesh out the details of the image, the more likely it will be to be highly retrievable.
• just like the ownership of catalog records, the media records need to provided freely and with permission, and each record needs to have at least Creative Commons permission = "CC BY"

Use Case for Media and Data

2 CSV (one for image metadata, one for specimen data, plus uploaded images with > 1 image per specimen)

• Put specimen occurrenceID in the field associatedSpecimenReference in the image metadata csv (whose schema is in Audubon Core format)
• Generate a meta.xml file by hand and package up the files in a DwC-A like format.

(no eml.xml required).

Sample Scenarios of Data Transformations to Prepare Data for Ingestion

• Example preparing specimen data from Illinois Natural History survey (INHS) fish collection from FileMakerPro
• Example of transformations on InvertNet image metadata dataset

Additional References

If you want to learn about acceptable Creative Commons licenses in iDigBio:

• https://www.idigbio.org/content/idigbio-intellectual-property-policy