Dataset Errata

Errors noted in various files

New Errors 2/27/13

Gold Parsing Errors

Lichen Gold

verbatimLatitude and verbatimLongitude

Many of the Lichen Gold labels have verbatimLatitude and verbatimLongitude, but the Gold Parsed files do not have the calculated decimalLatitude and decimalLongitude. This seems especially true for the New York labels. (Daryl)
(Bryan: I think the decimal values were "bonus" I could be wrong. If we choose to do this later it might be easier to pre-fill as many fields as we can using your algorithm.)
(Ed: Verbatim field contain verbatim results. No lichen labels have DwC complaint decimal coordinates. Likewise, no labels has DwC compliant event dates, thus you probably only want to only use verbatim fields for stats) Check this with Alex --Dpaul 16:33, 12 June 2013 (EDT)

elevation verbatimElevation

This is open to debate, but I think Elevation should be a pure numeric field, assumed to be in meters. Therefore, it should not be expressed as "750 m", but rather as "750". verbatimElevation, of course, should retain the "m" if it was present on the label. (Note that Darwin Core apparently does not have a field called "elevation", but rather MinimumElevationInMeters, and MaximumElevationInMeters, both numeric fields.) Not sure if this is something to change on the labels, but worth being aware of. I think parsing programs should generate the Darwin Core fields. (Daryl)
(Bryan: Odd to not have "elevation" I agree with the use of verbatimElevation. If "elevation" is filled it is numeric.)
(Deb: What are the ramifications then? Does the lichen set need to be fixed in this regard? or just ignore derived columns and expect letters like m or mi or ft in verbatimElevation field?)--Dpaul 16:40, 12 June 2013 (EDT)

Gold Parsed Country errors

Inconsistency in the Gold Parsed labels for Country. If a US State is listed as the state, the label doesn't always say the name of the country, though it is obviously the USA. Some Gold parsed results leave it blank, some fill it in with "USA", or "United States", though neither of these are on the label. I think it is valid to fill it in, but it should be consistent. (Daryl)
(Bryan: I think for Gold the field should not be filled in if it is not on the label.)
(Deb: Note, to be clear!, the country is often in the DatasetName and from that, most (all?) of us put the country listed there in the dwc:country field). This seems fine and right to me.
(Deb: A different issue is putting in the country when it's not listed anywhere on the label -- that's indeed inference -- and we ought to try and clean that up where it happened. IMHO. --Dpaul 21:00, 13 June 2013 (EDT)).

Many Gold Parse Tennessee lichen labels have country errors.
Examples:

(Deb: if there are many country errors on these, where the error is one of putting country in by deducing it because the state is in the USA, ...removing the value will take some time to fix as every single record will need to be verified. --Dpaul 16:40, 12 June 2013 (EDT))

-- Gold Parsed TENN-L-0000001_lg.csv lists country as "USA", but on the .txt label, it is "U.S.A." (with periods).
Same with Gold Parsed TENN-L-0000035_lg.csv and others.(Daryl)
(Bryan: Agreed. Should be fixed to match the label.)
(Ed: fixed, note that TENN-L-0000035_lg.txt has "U. S. A. " with spaces, thus conserved format)

-- Gold Parsed TENN-L-0000005_lg.csv leaves country blank, but the label shows it as "USA". Again, maybe this is OK, but it should be consistent. (Daryl)
(Bryan: Agreed. Should be fixed to match the OCR label.)
(Ed: Fixed, country had county value)

Inconsistency and errors in TENN Lichen Gold Parsed dateIdentified. Examples:

dateIdentified or verbatimEventDate errors

-- TENN-L-0000015_lg.csv has dateIdentified in the wrong format, neither verbatim, nor standard DarwinCore format: Verbatim would be: Nov. 12, 1939, DarwinCore would be: 1939-11-12, Listed is: 1939-November-12.
(Bryan: I think excel may have imposed it's own format and changed the records. If the column type is set to "text" Excel will not transform to a new format. )
(Ed: The verbatimEventDate (not dateIdentified is in the correct format, but if you open file in excel it will convert the display to match program's defaults)).
Deb - verified verbatimEventDate="Nov. 12, 1939 in csv --Dpaul 21:21, 13 June 2013 (EDT)
-- TENN-L-0000017_lg.csv omits dateIdentified, though it is on the label as 3 Feb. 1963
(Bryan: Agreed. Should be fixed to match the label.)
(Ed: That is the collection date, not the dateIdentified. Format is correct in verbatimDate)
Deb: this is a confusing label! the line begins with

Det.:Barbara Moore, this is followed by Date: 3 Feb 1963.

(A person who knows this collector's label and habits would know if the date represents collected or identified date. An outsider may very well conclude eventDate (date collected) is missing.--Dpaul 21:21, 13 June 2013 (EDT)

-- TENN-L-0000019_lg.csv has 1954-Aug-8, but on the label it is "8 Aug 1954", again neither verbatim nor DarwinCore (1954-08-08). (Daryl)
(Bryan: Agreed. Should be fixed to match the label.)
Deb: in csv, dwc:verbatimEventDate=8 Aug 1954
dwc:eventDate=1954-08-08
dwc:dateIdentified=1954-Aug-8
it's this last field that's funky and I"ve changed it to be 1954-08-08 --Dpaul 21:32, 13 June 2013 (EDT)

comma, space, period, apostrophe in verbatimCoordinates errors

Gold Parsed NY01075760_lg.csv replaces the comma with a space, and replaces an apostrophe (') with a double quote (") in verbatimCoordinates: 38°42'20"N, 83°08'25'W is rendered as 38°42'20""N 83°08'25""W. (Note also that the double quote is replaced with two double quotes. This may be necessary to preserve the quote-delimited, comma separated fields, but could cause some problems when uploading to a database. Not presented here as an error, but we should be aware of possible implications.)
(Bryan: Agreed. Should be fixed to match the Label except, the double quoted double quote I think is needed for CSV readers to identify fileds. I am not sure)

Fixed --Dpaul 21:49, 13 June 2013 (EDT) See this for how csv looks now.

NY01075760_lg.csv:Richard C. Harris,52766,"38°42'20""N 83°08'25""W",21 May 2006,2006-05-21,,Scioto,Ohio,U.S.A.,Polycoccum minutulum Kocourková & 'F. Berger,"Shawnee State Forest, along Pond Lick Run N of Forest Road 1, 0.35 mi SW of SR 125",sandstone boulders along stream and adjacent oak woods.,on Trapeliopsis placodioides Coppins & James,ca. 190 m,,,"38°42'20""N",83°08'25'W,1075760,New York Botanical Garden,,Polycoccum minutulum Kocourková & F. Berger,,,,

Gold Parsed NY01075764_lg.csv has a similar problem where a single space is replaced with a double space in verbatimCoordinates.
(Bryan: Agreed. Should be fixed to match the label as best as possible. If it is not clear follow the OCR file.)

Inconsistencies in several Gold Parsed labels regarding whether to include the period at the end of a field as part of the field. Example: verbatimCoordinates in NY01075782_lg.csv includes the period at the end. NY01075780_lg.csv does not include the period. (Bryan: It could go either way but I think for consistancy throughout we should keep the period at the end of anything. Whenever there are sentences with a period we keep them as in "One mile east of Dodge City." we would not think of removing the period. In gold we should treat it as verbatim. If we do platinum it could be removed.)

Gold Parsed NY01075761_lg.txt corrects a Gold OCR error by adding the 1 to the end of 0107576. The field should be corrected in the Gold OCR, but until done so, the parsing should be verbatim (see below under Gold OCR Errors). (Bryan: hmmm. Gold shoudl be as-if the OCR engine read the label with no mistakes. Silver would leave it as 0107576 but in Gold we shoudl put what was on the label. That would include the "1".)

Gold Parsed NY01075766_lg.csv omits the catalogNumber, though it is present in the NY01075766_lg.txt file. (Bryan: Agreed. Should be fixed to match the label.)

Gold Parsed NY01075789_lg.csv adds "NY" as a prefix to the catalogNumber, though it is not present on the .txt file. (Bryan: Agreed. Should be fixed to match the Label.)

Gold Parsed NY01075760_lg.csv omits dataset, which should be "Lichens of Ohio" (Bryan: Agreed. Should be fixed to match the label.)

Gold Parsed NY01075775_lg.csv omits "Boulder" from municipality.(Bryan: Agreed. Should be fixed to match the Label.)

Gold Parsed NY01075761_lg.csv lists "Peru" as municipality, but NY01075779_lg.csv lists "Town of Peru", though they both appear identical ("Town of Peru") on the labels. Probably should be "Peru" on both...? (Bryan: This is complex but in this case I think it should be "Town of Peru". The bald "Peru" could be read as an error with the misplacment of "Country". Likely the Label author was worried about the same thing and included "Town of". In both cases include what is on the label.)

Gold Parsed TENN-L-0000029_lg.csv and TENN-L-0000035_lg.csv both list municipality as "NORTH AMERICA" (Bryan: Agreed. Should be assigned to "country".)

Gold Parsed TENN-L-0000083_lg.csv lists municipality as "Ontario", but this is not on the label. (Bryan: Agreed. Should be fixed to match the label.)

Gold Parsed WIS-L-0011732_lg.csv (and many other lichen gold parsed labels) removes a space from verbatimLatitude and from verbatimLongitude, changing this: 60° 33.579'N into this: 60°33.579'N. The space removal is inconsistent, on some labels, not on others. (Bryan: Agreed. Should be fixed to match the label. I think if the OCR had been perfect the space would not be n the OCR file do it is a tough call.)

Gold Parsed NY01075791_lg.csv converts the "u" in "Mull" to an umlaut yielding "Müll". This actually reflects the original label, but not the Gold OCR NY01075791_lg.txt file, which has "Mull". Same for NY01075792_lg.csv, and several other in the series. (Bryan: The OCR messed up. Gold should fix OCR errors so the umlaut shoudl saty.)

Gold Parsed CSV Files

Lichen NY

There are more errors in gold csv files. (Qianjin)
(Bryan: I agree with Qianjin's edits except as noted below)

NY01075759_lg verbatimEventDate (1998-04-19), it should be 19 April 1998

From Deb: path /home/aocr/webroot/datasets/lichens/gold/parsed the csv file, opened in Notepad++ has 19 April 1998, encoding is ANSI --Dpaul 21:20, 30 June 2013 (EDT)

here's what it looks like when opened in Notepad++

dwc:recordedBy, dwc:recordNumber, dwc:verbatimCoordinates, dwc:verbatimEventDate, dwc:eventDate, dwc:municipality, dwc:county, dwc:stateProvince, dwc:country, aocr:verbatimScientificName, dwc:verbatimLocality, dwc:habitat, dwc:substrate, dwc:verbatimElevation, dwc:identifiedBy, dwc:dateIdentified, dwc:verbatimLatitude, dwc:verbatimLongitude, dwc:catalogNumber, aocr:verbatimInstitution, dwc:datasetName, dwc:scientificName, dwc:decimalLatitude, dwc:decimalLongitude, dwc:fieldNotes, dwc:sex
Richard C. Harris,42164,"41°11'N, 74°08'W",19 April 1998,1998-04-19,,Rockland,NEW YORK,U.S.A.,,"Harriman State Park, along Woodtown Road West near dam at S end of Lake Sebago along Seven Lakes Drive",mixed hardwood-hemlock forest with granitic erratics.,on Trapelia placodioides Coppins & P. James,ca. 240 m,,,41°11'N,74°08'W,01075759,New York Botanical Garden,Lichens of New York State,Polycoccum minutulum Kocourkova & F. Berger,,,,

here's what it looks like when I open it at the command line with text editor (vi)

dwc:recordedBy, dwc:recordNumber, dwc:verbatimCoordinates, dwc:verbatimEventDate, dwc:eventDate, dwc:municipality, dwc:county, dwc:stateProvince, dwc:country, aocr:verbatimScientificName, dwc:verbatimLocality, dwc:habitat, dwc:substrate, dwc:verbatimElevation, dwc:identifiedBy, dwc:dateIdentified, dwc:verbatimLatitude, dwc:verbatimLongitude, dwc:catalogNumber, aocr:verbatimInstitution, dwc:datasetName, dwc:scientificName, dwc:decimalLatitude, dwc:decimalLongitude, dwc:fieldNotes, dwc:sex
Richard C. Harris,42164,"41Â°11'N, 74Â°08'W",1998-04-19,4/19/1998,,Rockland,NEW RK,U.S.A.,,"Harriman State Park, along Woodtown Road West near dam at S end of Lake Sebago along Seven Lakes Drive",mixed hardwood-hemlock forest with granitic erratics.,on Trapelia placodioides Coppins & P. James,ca. 240 m,,,41Â°11'N,74Â°0W,01075759,New York Botanical Garden,Lichens of New York State,Polycoccum minutulum Kocourkova & F. Berger,,,,

NY01075760_lg no datesetName

FIXED in /home/aocr/webroot/datasets/lichens/gold/parsed --Dpaul 23:18, 30 June 2013 (EDT)

NY01075765_lg verbatimEventDate (Feb. 1898), it should be verbatimEventDate ( Feb 1898.)

FIXED in /home/aocr/webroot/datasets/lichens/gold/parsed --Dpaul 23:18, 30 June 2013 (EDT)

NY01075766_lg decimalLatitude (White Horse Beach, between Manomet Pt. and Rocky Pt., Plymouth area), it should be locality or habitat; no catalogNumber

FIXED in /home/aocr/webroot/datasets/lichens/gold/parsed --Dpaul 23:18, 30 June 2013 (EDT)

NOTE since there's no host field, the shell of Balanus balanoides is in habitat which really is not right --Dpaul 23:18, 30 June 2013 (EDT)

NY01075767_lg verbatimEventDate format

FIXED in /home/aocr/webroot/datasets/lichens/gold/parsed --Dpaul 23:18, 30 June 2013 (EDT)

NY01075767_lg verbatimEventDate (July 1979), it should be (Jul-79)

See error above, Label reads July 1979 I fixed the verbatimEventDate to match.--Dpaul 23:18, 30 June 2013 (EDT)

NY01075768_lg country (canada), it should be (ca.)

Fixed verbatim locality to include the ca. (iow, about) --Dpaul 23:18, 30 June 2013 (EDT)

NY01075770_lg habitat (on Acmaea digitalis Eschsch. Host determined by A. R. Grant) and identifiedBy (A. R. Grant.)

fixed habitat to read: on Acmaea digitalis Eschsch. Host determined by A. R. Grant.

removed A. R. Grant. from identifiedBY (she or he identified the host, not the lichen)--Dpaul 23:18, 30 June 2013 (EDT)

NY01075770_lg habitat (Host determined by A. R. Grant)

see above --Dpaul 23:18, 30 June 2013 (EDT)

NY01075771_lg verbatimCoordinates mixed with verbatimLocality

verbatimLocality -includes all information in the locality -- looks correct to me.--Dpaul 23:18, 30 June 2013 (EDT)

verbatim coordinates on this record probably ought to be blank since the collector essentially provided 3 different sets of lat / lon values.--Dpaul 23:18, 30 June 2013 (EDT)

I did not change this record.--Dpaul 23:18, 30 June 2013 (EDT)

NY01075779_lg habitat concatenation (Bryan: "on Protoblastenia rupestris" appears before the location and habitate section of the label. However, Habitate says "dolomite rock along lake shore and adjacent Thuja forest; on Protoblastenia rupestris". There was a period after "forest". The period was removed and a ";" added. Then the "on Protoblastenia rupestris" from the earlier part of the label was concatinated.

changed dwc:habitat to:

on Protoblastenia rupestris dolomite rock along lake shore and adjacent Thuja forest.

--Dpaul 23:19, 30 June 2013 (EDT)

NY01075780_lg NEW YOUR BOTANICAL GARDEN (Bryan: the label said "GARDEN". The OCR said "CARDEN". SIlver should be "CARDEN" Gold should be "GARDEN"

the OCR I see says "Garden" in /home/aocr/webroot/datasets/lichens/silver/ocr --Dpaul 23:19, 30 June 2013 (EDT)

fixed the Gold (changed from Carden to Garden), left the silver alone.--Dpaul 23:19, 30 June 2013 (EDT)

NY01075789_lg catalogNumber (NY01075789) in the csv file; but it is (01075789) in the text file.

fixed in gold csv (path is /home/aocr/webroot/datasets/lichens/gold/parsed) --Dpaul 23:19, 30 June 2013 (EDT)

NY01075797_lg recordedBy ( William Russell Buck) in the csv file; but it is (William R. Buck) in the text file.

fixed in gold csv (path is /home/aocr/webroot/datasets/lichens/gold/parsed) --Dpaul 23:19, 30 June 2013 (EDT)

NY01075805_lg stateProvince (South Carolina) in the csv file; but it is (S.C.) in the text file.

changed to S.C. in the csv --Dpaul 23:19, 30 June 2013 (EDT)

NY01075812_lg recordedBy( William Russell Buck) in the csv file; but it is (William R. Buck) in the text file.