4,707
edits
Hackathon Challenge: Difference between revisions
Jump to navigation
Jump to search
m
→Metrics and Evaluation
| Line 61: | Line 61: | ||
*Graphics may be created | *Graphics may be created | ||
**For example, with an F-score for each dwc element entry, we can generate a graph / histogram across all participants | **For example, with an F-score for each dwc element entry, we can generate a graph / histogram across all participants | ||
:::; Evaluation : We will attempt to provide services that can validate the outcomes of hackathon deliverables. This hackathon is not structured as a competition, but we felt it would be beneficial for participants to have some baseline to evaluate the effectiveness of their methods. | |||
:::; OCR Text Evaluation : Evaluation of OCR Output will be based on a comparison to Gold Hand-Typed outputs, using confusion matrix like criteria for evaluating word presence, word correctness, and avoiding non-text garbage regions. We will attempt to avoid penalizing for attempts at text recognition in barcode and handwritten regions. | |||
:::; Parsed Field Evaluation : Evaluation of the effectiveness of parsing will be calculated based on a confusion matrix. Rows are named with each of the possible element names for parts of a label. Columns are also these same names. Counts along the diagonal represent the number of items that were tagged correctly. For example, a count that is correctly labeled as a county will add one to the diagonal. If a county is incorrectly marked as a stateProvince, a 1 is added to the “county” row under the stateProvince column. This format therefore provides a count of correct classifications and count of false positives and false negatives. We will calculate, precision, recall, f-score and potentially others. | |||
Back to the [[2013 AOCR Hackathon Wiki| Hackathon Wiki]] | Back to the [[2013 AOCR Hackathon Wiki| Hackathon Wiki]] | ||