ElementaryDependencyMatch

Elementary Dependency Match (EDM) is a granular evaluation metric based on the so-called 'ltriples' format that can be exported from a [incr tsdb()] profile. These triples are derived from the variable-free reduction of MRSs known as Elementary Dependencies, described by Oepen & Lønning, 2006. Elementary Dependencies describe almost all the semantics contained in an MRS (excepting scopal information), and can be divided into three types:

• NAMES: predicate name to char span ARGS: ARG-type relations between char spans PROPS: features of predicates, such as TENSE and GENDER

An EDM evaluation is measured over all three types of ED, but other combinations are possible. EDM_NA evaluates predicate names and arguments, and is closest to other metrics such as GR, CCG dependencies etc. The default ouput for the evaluation script shows precision, recall and f-score over each relation separately, as well as typical aggregations.

To use this evaluation, you first need to set the data up as described below.

Set up

1. Export gold:

• $LOGONROOT/lingo/lkb/src/tsdb/home/export --binary --format ltriples <gold profile>

2. Export test:

• $LOGONROOT/lingo/lkb/src/tsdb/home/export --binary --format ltriples --active=all <test profile>

This should produce directories containing one gzipped file per item parsed. The ltriples should look like:

• _treat_v_1<10:17> ARG2 _user_n_of<23:27>

The links (eg. <10:17>) are necessary to the evaluation, and if your output doesn't have them, ask StephanOepen why.

Evaluate

Usage: cat <goldfilelist>|./edm_eval.pl [-i] [-v] [-p <num>] [-s] <export directory>

• -i: ignore gold where parse failed -v: verbose output

  -p <num>: parse number -s: raw figures for statistical significance calculations

To evaluate a profile:

• ls -1 jhk.gold/*|edm_eval.pl jhk.test

To evaluate a profile, only over files that received a parse:

• ls -1 jhk.gold/*|edm_eval.pl -i jhk.test

To evaluate a single item:

• echo jhk.gold/3025231.gz |edm_eval.pl jhk.test

To evaluate a specific analysis of a single item:

• echo jhk.gold/3025231.gz |edm_eval.pl -p 100 jhk.test

To examine the errors in a single item:

• echo jhk.gold/3025231.gz |edm_eval.pl -v jhk.test

To produce the files needed for statistical significance testing:

• for file in jhk.gold/*;

  • do echo $file|edm_eval.pl -s jhk.test;

  done > jhk.test.stats

Significance Testing

An implementation of the computationally-intensive randomisation test described in:

• Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In Proceedings of the 18th International Conference on Computational Linguistics (COLING 2000), pages 947–953, Saarbruecken, Germany.

Usage:

• statsig_shuffle.pl <first stats file> <second stats file> [iterations] statsig_shuffle.pl jhk.gold.stats jhk.test.stats 10000