by: Deborah Paul, Shelley A James, David P Shorthouse, Christopher J Marshall, Teresa J Mayfield-Meyer, Carrie A Eaton
Symposium: Collecting Measures of Success
When and where: May at SPNHC 2019 in Chicago, Illinois
Organizers: (aka Instigators): Shelley James (RBGSyd), David Shorthouse (Ag-Canada, formerly CMN), Deborah Paul (iDigBio)
Thanks: Shari Ellis (iDigBio) who guided development of this anonymous metrics survey instrument.
Keywords: evaluation, metrics, metadata standards, small collections, publication, citation, attribution, curatorial credit, education, teaching collections, impact, annual report, individual effort, baseline, data quality, digitization dashboard, worldwide collections, interoperability, communication, capacity building
Talk about measuring success, over 100 people attended the SPNHC2019 “Collecting Measures of Success” 8-talk session that centered around metrics and measures for various scientific collections. Speakers shared insights from various points-of-view including collections in undergraduate education, collections in publications, individual collectors, standards developers, biodiversity data aggregators, industry, societies (like SPNHC itself), and from the collection institution perspective.
While the topics and vantage points for the individual talks differed, a number of shared themes emerged. First, there exists a wide range of metrics used and sought by various collections and biodiversity data providers given their particular circumstances, missions, and case-usage for the metrics. Second, several speakers discussed efforts to improve the visibility and metricable impact made by individual collections in providing digital specimen data. Under the current model, the vast majority of visible credit for specimen-data is given to data-aggregator and/or data portals, with no easy direct means for institutions to directly see the true contributions their specimens and digitizing efforts are having in scholarly products that depend on them. In order for an aggregated big bio-collection data model to be sustainable in the long run, benefit-sharing (in the form of true academic credit) needs to accurately reflect the true contribution and record it in a fashion that is easily retrieved, integrated into the academic culture and permanently recorded so that academics at large, not just administrators reading an annual report, can see who and where these important data come from.
From the response and participation, we can say that many in the collections and biodiversity data communities are interested in knowing more about metrics and measures of success currently in use by collections and as they relate to the use and care of collections items. At an iDigBio workshop eight years ago (see Object-to-Image-to-Data), held at the initiation of the NSF program known as ADBC (Advancing the Digitization of Biodiversity Collections), the collections community realized they needed to compare, contrast, and share digitization and imaging workflows in order to help standardize the data resulting from digitization efforts. It follows that the biocollections world also needs to gather, assess, and share measurement strategies to ensure continued funding, recognition and enhanced value of assets, both physical and human. If we want to meet the needs of collections to understand and communicate the value of collections and opportunities inherent therein we need to know collectively: where we are, where we want to go, and then decide how we want to get there (see the Talanoa Dialogue as an example). To know where we are and to develop meaningful benchmarks to measure progress, as a biocollections community, we need to begin by sharing our respective metrics and measurement strategies, both quantitative and qualitative.
Talanoa is a traditional word used in Fiji and across the Pacific to reflect a process of inclusive, participatory and transparent dialogue. The purpose of Talanoa is to share stories, build empathy and to make wise decisions for the collective good. The process of Talanoa involves the sharing of ideas, skills and experience through storytelling.
The Collecting Measures of Success symposium served as a story-telling starting point. The Symposium Wiki provides links to all talks given, and to resources shared (so far). We now encourage everyone to share their metrics/measurements strategies. What questions do you ask to assess success in your collections programs (whatever they may be)? Who are the stakeholders asking for metrics and measures? Do you have a plan for gathering impact data (quantitative and qualitative)? In asking the symposium audience questions about their reporting activities, it became clear that we all, in some way, report on biocollections progress and success, whether it be an annual performance review to annual institutional reports. In this anonymous metrics survey (if you have not already taken it), please add your voice to this discussion to help drive planning future events. More than 80 people have shared their contact details, thoughts and interest in this topic so far as of June 2019.
At the end of our SPNHC2019 symposium, a brief panel discussion session was held. At this point we were able to include news about efforts of the Global Biodiversity Information Facility (GBIF) to work with the broader collections community to re-envision a worldwide collections catalog (see example - but note link has known expiration date), and Jeroen Bloothoofd (Picturae), shared efforts of the DiSSCo project to build a Collections Digitization Dashboard to track metrics across more than 115 museums in Europe, starting in 2024.
We thank all of our speakers: Shelley James, David Shorthouse, Anne Thessen, Teresa Mayfield-Meyer, Carrie Eaton, Deborah Paul, Christopher Marshall, Anna Monfils, Kimberly Cook for their time, contribution and enthusiasm in helping the Co-Chairs deliver this symposium! As an indication of success, the following action items and discoveries resulting from this symposium:
- Look for a webinar on logic models for developing metrics to be presented by IMLS Grants Management Specialist, Connie Bodner. Connie also shares they have grant money to help collections with preservation and digitization. Logic Model Examples can be found on the IMLS Evaluation Resources page.
- Another goal of this symposium is to provide for the community a set of suggested - recommended - standardized metrics to attempt to collect and offer efficient means with which to collect them. Organizers plan to write a white paper on our findings via the survey and symposium and to summarize our recommendations and resources gathered.
- One of our speakers, Anne Thessen, is seeking funding to develop standards that support tracking the efforts of individuals doing curatorial work in the collections. Do you have feedback to give her? “The purpose of this survey is to get a general idea of how the professionals who work with natural history collections spend their time and how those decisions are influenced by metrics.”
- DiSSCo (Distributed System of Scientific Collections) folks found the model shared by Carrie Eaton to be intriguing and look forward to possibly using it.
- An increased awareness of Bloodhound-tracker.net and the need for individuals in our community to have an ORCID to support citation and attribution of curatorial work in collections. (Read more about ORCID on Wikipedia).
- Arctos has some built-in metrics tools to help identify quality issues with both shared resources such as taxonomy and locality as well as individual record data.
- ...and SPNHC is working on metrics it tracks for/about the organization and members. For more information on this, contact Sue Ryder, chair of the SPNHC International Relations Sessional Committee.
One of our speakers, Teresa Mayfield, also led a SPNHC Special Interest Group (SIG) titled Collection Management Software Providers to invite participants to discuss interoperability and data quality issues affecting all natural history collections. We have lots of follow up topics and you can see who was present as members of a new working group titled: Biodiversity Collection Management Solutions Working Group, led by Teresa. What follows is a summary of issues raised by those who attended this SIG:
- Difficult to compare the various software options for managing collections data
- Inconsistent or undocumented data standards and vocabularies
- Inability to connect data across collections and varied data management systems
- Lack of time/budget/IT expertise
- IT - nonIT communication gap
- Lack of awareness of existing resources (human and online): many participants in this SIG are not associated with any of the Thematic Collection Networks funded in the ADBC Program from NSF.
References.
What is Talanoa? United Nations Climate Change website. 2018 Talanoa Dialogue Platform https://talanoadialogue.com/background accessed 28 May 2019.
Evaluation Resources. Institute of Museum and Library Services (IMLS) website. https://www.imls.gov/research-evaluation/evaluation-resources accessed 28 May 2019.
