Data Carpentry
This wiki supports the Data Carpentry Workshop to be held at the University of Florida at iDigBio September 29-30, 2014. It is the first in a series of four biodiversity informatics workshops planned in the upcoming year (2014-2015).
General Information
Data Carpentry's aim is to teach researchers basic concepts, skills, and tools for working with data so that they can get more done in less time, and with less pain.
Our curriculum includes:
- Day 1 morning: Better spreadsheet skills and introduction to more powerful tools
- Day 1 afternoon: Introduction to databases, combining and querying data using SQL
- Day 2 morning: Introduction to the shell, Introduction to R and managing data in R
- Day 2 afternoon: Collaborative data management & publishing data
The concepts, skills, and tools we teach are domain-independent, but example problem cases and datasets will be taken from organismal and evolutionary biology, biodiversity science, ecology, and environmental science.
Data Carpentry's teaching is hands-on, so participants are required to bring their own laptops. (We will provide instructions on setting up the required software several days in advance) There are no pre-requisites, and we will assume no prior knowledge about the tools.
Updates will be posted to this website as they become available.
Software Installation Requirements and additional information is available at the github web site for the project:
http://datacarpentry.github.io/2014-09-29-iDigBio/
Planning Team
François Michonneau (FLMNH - iDigBio), Katja Seltmann (TTD-TCN, MNH), Matt Collins (ACIS - iDigBio), Dan Stoner (ACIS - iDigBio), Deborah Paul (FSU - iDigBio), Tracy K. Teal (BEACON), Pam Soltis (FLMNH - iDigBio PI), Derek Masaki (USGS - BISON), Shari Ellis (iDigBio), Kevin Love (iDigBio), Mike Smorul (SESYNC), Juliet Pulliam (UF), Ming Tang (Tommy) (UF), and assistance from Nirav Merchant at iPlant.
About
Instructors: François Michonneau (FLMNH - iDigBio), Tracy Teal (MSU - BEACON), Matt Collins (ACIS - iDigBio), Katja Seltmann (TTD-TCN, AMNH)
Assistants: Dan Stoner (ACIS - iDigBio), Deborah Paul (FSU - iDigBio), Pam Soltis (FLMNH - iDigBio PI), Derek Masaki (USGS), Shari Ellis (iDigBio), Kevin Love (iDigBio), Juliet Pulliam (UF), Tommy Tang (UF), Bernardo Santos (AMNH), Jonathan Foox (AMNH)
Who: The course is aimed at graduate students, postdocs, research staff, and other researchers.
Skill Level: Beginner. Data Carpentry courses are meant for novices.
Where: iDigBio in Gainesville, FL and AMNH (AMNH in New York City via teleconference)
Requirements: Participants must bring a laptop with a few specific software packages installed. If you will be travelling from out of town, you will need to make your own travel arrangements.
Contact: Please email data-carpentry@software-carpentry.org for questions and information not covered here.
Twitter: #datacarpentry
Tuition for the course is free, but prior registration is required for attending. You can register here.
Workshop Evaluation
- link to pre-workshop survey
- link here at end of workshop
Software Installation Details
Software needed for Data Carpentry Workshop at iDigBio
- Data Carpentry software installation instructions
- You must RSVP that the required software is installed, prior to the workshop. Instructors are available to help - see your email for their contact information.
We use Adobe Connect extensively in this workshop. Please perform the systems test using the link below. Also, you will also need to install the Adobe Connect Add-In to participate in the workshop.
Agenda
- AdobeConnect #datacarpentry Room
- Pre-workshop meeting and dinner at Piesano's, 6 PM Sunday September 28th. Piesano's is at NW 13th St. and 1250 W. University Ave. in Gainesville. All welcome.
| Course Overview - Day 1 | ||
|---|---|---|
| 8:30-8:45 | Introductions & Overview, Data Carpentry: Making data science more efficient | All, Deb Paul |
| 8:45-9:00 | Linking Heterogeneous Data in Biodiversity Studies: the need for data carpentry | Pam Soltis, iDigBio PI |
| 9:00-10:00 | Better use of spreadsheets, part I | Tracy Teal |
| 10:00-10:30 | Break | |
| 10:30-12:00 | Better use of spreadsheets part II | Tracy Teal |
| 12:00-1:30 | Lunch (with OpenRefine Demo) | Deb Paul |
| 1:30-3:00 | SQL Introduction | Matt Collins |
| 3:00-3:30 | Break | |
| 3:30-5:00 | SQL part II | Matt Collins |
| 5:00-5:30 | Review / Wrap up for tomorrow | |
| Course Overview - Day 2 | ||
| 8:30-10:00 | Introduction to the shell | Tracy Teal |
| 10:00-10:30 | Break | |
| 10:30-12:00 | Introduction to R | François Michonneau |
| 12:00-1:30 | Lunch | |
| 1:30-3:00 | Manipulating and plotting data in R | François Michonneau |
| 3:00-3:30 | Break | |
| 3:30-4:30 | Getting data in and out of R: How to integrate R in your workflow | François Michonneau |
| 4:30-5:00 | Sharing your data and your results: RMarkdown and Figshare | François Michonneau |
| 5:00-5:30 | Review / Wrap up / Evaluation and Feedback | |
Future plan: Scaling it up: Demo using the iPlant Discovery Environment (DE)
Link to Workshop Report
Data Carpentry - Please can we have some more?!
Logistics
- Logistics
- Hotel Information (for the out-of-towners)
- Where to find food
- Workshop Calendar Announcement
- Participant List
Adobe Connect Access
Adobe Connect will be used to provide access for a remote classroom at the American Museum of Natural History. Workshop participants will be encouraged to be logged in to the Adobe Connect room to facilitate sharing with this remote group: Already registered and accepted?
- Adobe Connect Room for Data Carpentry at iDigBio
- AMNH (remote) and iDigBio participants take notes together with this EtherPad
Presentation Documents and Links
Links to any presentations (like power points) here.
- Data Carpentry: Making data science more efficient
- Linking Heterogeneous Data in Biodiversity Studies: the need for data carpentry Presenter: Pam Soltis, iDigBio PI
- GitHub repository for Data Carpentry Workshop at iDigBio
- EtherPad for Data Carpentry at iDigBio: workshop notes
- SQL commands list
- SHELL commands list
- R lesson
- Getting started with Open Refine
Workshop Recordings
Day 1
- 8:30am-10:00am http://idigbio.adobeconnect.com/p7gqmvndl5a/
- 10:30am-12:00pm http://idigbio.adobeconnect.com/p6ipl39xnl3/
- 12-12:30 http://idigbio.adobeconnect.com/p4dsgo0y77y/
- 1:00pm-5:00pm http://idigbio.adobeconnect.com/p4rm50ukzv5/
Day2
- 8:30am-10:00am http://idigbio.adobeconnect.com/p2crpl68wr4/
- 10:30am-12:00pm http://idigbio.adobeconnect.com/p1e150mot75/
- 1:30pm-5:00pm http://idigbio.adobeconnect.com/p4t16mcnvvk/
Data Carpentry Resources and Links
- Data Carpentry Materials on GitHub
- Ten Simple Rules for the Care and Feeding of Scientific Data. Goodman et al
- Code and Data for the Social Sciences: A Practitioner's Guide. Matthew Gentzkow, Jesse M. Shapiro Chicago Booth and NBER March 10,2014
- Nine simple ways to make it easier to (re)use your data. White et al.
- You want to learn SQL independently? Try Head First SQL
- Head First Excel, O'Reilly
- Check out DataONE
- They've got a great Software Tools Catalog
- Put standard metatdata with your data. Wondering how to do that? Check out DataONE's Morpho Tool available under the tools menu at https://knb.ecoinformatics.org/.
- Why? Makes your data re-useable, and better still, makes your data discoverable. Get cited for your datasets in addition to your published papers!
- Using Open Refine? Want to compare your taxon names against a standard list? Try this reconciliation service.
- Read Gaurav ' Blog post first: http://gbif.blogspot.com/2013/07/validating-scientific-names-with.html
- Then, give it a try. The google plus Open Refine community will help you figure it out (it's not hard).
Links from You
- How about you? Got a favorite resource - a book?, a website? to share with your classmates?
- Data Science at the Command Line
- Free Training Resources for UF students, faculty, and staff UF provides free access to over 2600 online training courses through Lynda.com. Does your institution have similar free training opportunities?
Related Blog Posts
- Inaugural Data Carpentry Workshop by Tracy K. Teal
- Our First Data Carpentry Workshop by Karen Cranston
- Tales from the First Data Carpentry Workshop by Deb Paul