Results of the shared task on automatic identification of verbal multiword expressions
Last updated: February 10, 2017
Participants
Seven teams submitted results to the shared task: 6 to the closed track and 1 to the open track (no system contributed to both tracks); 5 systems were multilingual. All 18 languages were covered, by a variable subset of systems each.
Authors of the systems and their affiliations will be made public after reviewing system description papers.
Results
The evaluation results (disregarding VMWE categories) are given below, for each language separately. Each ranking is based on the
F-measure and is specific to three parameters:
Categorizing the identified VMWEs was not required but 4 out of the 7 systems submitted results in which VMWE are categorized. Detailed results per category (with no ranking) are available below.
Recall, that the aim of this shared task is a cross-language discussion rather than a true competition. The participation policy may have introduced bias to the annotated corpora and the participating systems. Therefore, the ranking results should be considered with caution.
BG
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
TRANSITION | closed | 0.6887 | 0.5518 | 0.6127 | 1 | 0.7898 | 0.5691 | 0.6615 | 1 |
MUMULS | closed | 0.3581 | 0.3362 | 0.3468 | 2 | 0.7686 | 0.4809 | 0.5916 | 2 |
CS
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
TRANSITION | closed | 0.7897 | 0.6560 | 0.7167 | 1 | 0.8246 | 0.6655 | 0.7365 | 1 |
ADAPT | closed | 0.5931 | 0.5621 | 0.5772 | 3 | 0.8191 | 0.6561 | 0.7286 | 2 |
RACAI | closed | 0.7009 | 0.5918 | 0.6418 | 2 | 0.8190 | 0.6228 | 0.7076 | 3 |
MUMULS | closed | 0.4413 | 0.1028 | 0.1667 | 4 | 0.7747 | 0.1387 | 0.2352 | 4 |
DE
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
SZEGED | closed | 0.5154 | 0.3340 | 0.4053 | 2 | 0.6592 | 0.3468 | 0.4545 | 1 |
TRANSITION | closed | 0.5503 | 0.3280 | 0.4110 | 1 | 0.5966 | 0.3133 | 0.4109 | 2 |
ADAPT | closed | 0.3308 | 0.1740 | 0.2280 | 3 | 0.7059 | 0.2837 | 0.4048 | 3 |
MUMULS | closed | 0.3277 | 0.1560 | 0.2114 | 4 | 0.6988 | 0.2286 | 0.3445 | 4 |
RACAI | closed | 0.3652 | 0.1300 | 0.1917 | 5 | 0.6716 | 0.1793 | 0.2830 | 5 |
EL
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
TRANSITION | closed | 0.3612 | 0.4500 | 0.4007 | 1 | 0.4635 | 0.4742 | 0.4688 | 1 |
ADAPT | closed | 0.3437 | 0.2880 | 0.3134 | 4 | 0.5380 | 0.3601 | 0.4314 | 2 |
MUMULS | closed | 0.2087 | 0.2580 | 0.2308 | 5 | 0.4294 | 0.4143 | 0.4217 | 3 |
SZEGED | closed | 0.3084 | 0.3300 | 0.3188 | 2 | 0.4451 | 0.3757 | 0.4075 | 4 |
RACAI | closed | 0.4286 | 0.2520 | 0.3174 | 3 | 0.5616 | 0.2953 | 0.3871 | 5 |
ES
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
TRANSITION | closed | 0.6122 | 0.5400 | 0.5739 | 1 | 0.6574 | 0.5252 | 0.5839 | 1 |
ADAPT | closed | 0.6105 | 0.3480 | 0.4433 | 2 | 0.7448 | 0.3670 | 0.4917 | 2 |
MUMULS | closed | 0.3673 | 0.3100 | 0.3362 | 4 | 0.6252 | 0.3995 | 0.4875 | 3 |
SZEGED | closed | 0.2575 | 0.5000 | 0.3399 | 3 | 0.3635 | 0.5629 | 0.4418 | 4 |
RACAI | closed | 0.6447 | 0.1960 | 0.3006 | 5 | 0.7233 | 0.1967 | 0.3093 | 5 |
FA
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
TRANSITION | closed | 0.8770 | 0.8560 | 0.8664 | 1 | 0.9159 | 0.8885 | 0.9020 | 1 |
ADAPT | closed | 0.7976 | 0.8040 | 0.8008 | 2 | 0.8660 | 0.8416 | 0.8536 | 2 |
FR
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
ADAPT | closed | 0.6147 | 0.4340 | 0.5088 | 2 | 0.8088 | 0.4964 | 0.6152 | 1 |
TRANSITION | closed | 0.7484 | 0.4700 | 0.5774 | 1 | 0.7947 | 0.4856 | 0.6028 | 2 |
RACAI | closed | 0.7415 | 0.3500 | 0.4755 | 3 | 0.7872 | 0.3673 | 0.5009 | 3 |
SZEGED | closed | 0.0639 | 0.0520 | 0.0573 | 6 | 0.5218 | 0.2482 | 0.3364 | 4 |
MUMULS | closed | 0.1466 | 0.0680 | 0.0929 | 5 | 0.5089 | 0.2067 | 0.2940 | 5 |
LIF | closed | 0.8056 | 0.0580 | 0.1082 | 4 | 0.8194 | 0.0532 | 0.1000 | 6 |
LATL | open | 0.4815 | 0.4680 | 0.4746 | 1 | 0.5865 | 0.5108 | 0.5461 | 1 |
HE
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
TRANSITION | closed | 0.7397 | 0.2160 | 0.3344 | 1 | 0.7537 | 0.1975 | 0.3130 | 1 |
MUMULS | closed | 0.0000 | 0.0000 | 0.0000 | n/a | 0.0000 | 0.0000 | 0.0000 | n/a |
HU
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
SZEGED | closed | 0.7936 | 0.6934 | 0.7401 | 1 | 0.8057 | 0.6317 | 0.7081 | 1 |
MUMULS | closed | 0.6291 | 0.6152 | 0.6221 | 5 | 0.7132 | 0.6657 | 0.6886 | 2 |
TRANSITION | closed | 0.6484 | 0.7575 | 0.6987 | 2 | 0.6502 | 0.7012 | 0.6747 | 3 |
ADAPT | closed | 0.7570 | 0.5992 | 0.6689 | 3 | 0.7846 | 0.5710 | 0.6610 | 4 |
RACAI | closed | 0.8029 | 0.5471 | 0.6508 | 4 | 0.8208 | 0.5015 | 0.6226 | 5 |
IT
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
TRANSITION | closed | 0.5354 | 0.3180 | 0.3990 | 1 | 0.6134 | 0.3378 | 0.4357 | 1 |
SZEGED | closed | 0.1503 | 0.1560 | 0.1531 | 4 | 0.4054 | 0.3064 | 0.3490 | 2 |
ADAPT | closed | 0.6174 | 0.1420 | 0.2309 | 2 | 0.6964 | 0.1532 | 0.2511 | 3 |
RACAI | closed | 0.6125 | 0.0980 | 0.1690 | 3 | 0.6837 | 0.1053 | 0.1824 | 4 |
LT
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
TRANSITION | closed | 0.6667 | 0.1800 | 0.2835 | 1 | 0.6786 | 0.1557 | 0.2533 | 1 |
MUMULS | closed | 0.0000 | 0.0000 | 0.0000 | n/a | 0.0000 | 0.0000 | 0.0000 | n/a |
MT
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
TRANSITION | closed | 0.1565 | 0.1340 | 0.1444 | 1 | 0.1843 | 0.1460 | 0.1629 | 1 |
ADAPT | closed | 0.2043 | 0.0380 | 0.0641 | 2 | 0.3084 | 0.0518 | 0.0887 | 2 |
RACAI | closed | 0.2333 | 0.0280 | 0.0500 | 3 | 0.2481 | 0.0259 | 0.0469 | 3 |
MUMULS | closed | 0.0000 | 0.0000 | 0.0000 | n/a | 0.0000 | 0.0000 | 0.0000 | n/a |
PL
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
ADAPT | closed | 0.7798 | 0.6020 | 0.6795 | 2 | 0.8742 | 0.6228 | 0.7274 | 1 |
TRANSITION | closed | 0.7709 | 0.6260 | 0.6909 | 1 | 0.8000 | 0.6312 | 0.7056 | 2 |
MUMULS | closed | 0.6562 | 0.5460 | 0.5961 | 3 | 0.8310 | 0.6013 | 0.6977 | 3 |
SZEGED | closed | 0.0000 | 0.0000 | 0.0000 | n/a | 0.0000 | 0.0000 | 0.0000 | n/a |
PT
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
TRANSITION | closed | 0.7543 | 0.6080 | 0.6733 | 1 | 0.8005 | 0.6370 | 0.7094 | 1 |
ADAPT | closed | 0.6410 | 0.5320 | 0.5814 | 2 | 0.8348 | 0.6054 | 0.7018 | 2 |
MUMULS | closed | 0.5358 | 0.3740 | 0.4405 | 3 | 0.8247 | 0.4717 | 0.6001 | 3 |
SZEGED | closed | 0.0129 | 0.0080 | 0.0099 | 4 | 0.6837 | 0.1987 | 0.3079 | 4 |
RO
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
MUMULS | closed | 0.7683 | 0.7760 | 0.7721 | 2 | 0.8620 | 0.8112 | 0.8358 | 1 |
ADAPT | closed | 0.7548 | 0.7140 | 0.7338 | 4 | 0.8832 | 0.7636 | 0.8190 | 2 |
TRANSITION | closed | 0.7097 | 0.8020 | 0.7531 | 3 | 0.7440 | 0.8449 | 0.7912 | 3 |
RACAI | closed | 0.8652 | 0.7060 | 0.7775 | 1 | 0.8773 | 0.7019 | 0.7799 | 4 |
SL
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
TRANSITION | closed | 0.4343 | 0.4300 | 0.4322 | 1 | 0.4796 | 0.4522 | 0.4655 | 1 |
MUMULS | closed | 0.3557 | 0.2760 | 0.3108 | 3 | 0.6142 | 0.3628 | 0.4562 | 2 |
ADAPT | closed | 0.5142 | 0.2900 | 0.3708 | 2 | 0.7285 | 0.3262 | 0.4506 | 3 |
RACAI | closed | 0.5503 | 0.2080 | 0.3019 | 4 | 0.7339 | 0.2145 | 0.3320 | 4 |
SV
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
ADAPT | closed | 0.4860 | 0.2203 | 0.3032 | 2 | 0.5253 | 0.2249 | 0.3149 | 1 |
SZEGED | closed | 0.2482 | 0.2966 | 0.2703 | 3 | 0.2961 | 0.3294 | 0.3119 | 2 |
TRANSITION | closed | 0.5100 | 0.2161 | 0.3036 | 1 | 0.5369 | 0.2150 | 0.3070 | 3 |
RACAI | closed | 0.5758 | 0.1610 | 0.2517 | 4 | 0.6538 | 0.1677 | 0.2669 | 4 |
TR
System | Track | Per-MWE results | Per-token results |
Precision | Recall | F-measure | Rank | Precision | Recall | F-measure | Rank |
TRANSITION | closed | 0.6106 | 0.5070 | 0.5540 | 1 | 0.6123 | 0.5039 | 0.5528 | 1 |
ADAPT | closed | 0.4541 | 0.4052 | 0.4283 | 3 | 0.5993 | 0.4728 | 0.5285 | 2 |
RACAI | closed | 0.6304 | 0.4391 | 0.5176 | 2 | 0.6340 | 0.4348 | 0.5159 | 3 |
MUMULS | closed | 0.4557 | 0.2774 | 0.3449 | 4 | 0.6452 | 0.3502 | 0.4540 | 4 |
Results per VMWE category (unranked)
BG
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
MUMULS | closed |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.4147 |
0.5320 |
0.4661 |
0.7250 |
0.7114 |
0.7181 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
CS
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
MUMULS | closed |
0.3182 |
0.0408 |
0.0724 |
0.5641 |
0.0566 |
0.1029 |
0.6395 |
0.1297 |
0.2156 |
0.8789 |
0.1573 |
0.2668 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.3043 |
0.0365 |
0.0651 |
0.5094 |
0.0467 |
0.0856 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
RACAI | closed |
0.7460 |
0.2741 |
0.4009 |
0.7970 |
0.2728 |
0.4065 |
0.7109 |
0.7554 |
0.7325 |
0.8160 |
0.8206 |
0.8183 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.5909 |
0.1354 |
0.2203 |
0.7273 |
0.1522 |
0.2518 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
DE
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
MUMULS | closed |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.5638 |
0.2345 |
0.3312 |
0.6803 |
0.2660 |
0.3825 |
0.2766 |
0.0607 |
0.0996 |
0.6196 |
0.1005 |
0.1730 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
RACAI | closed |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.4000 |
0.1000 |
0.1600 |
0.3333 |
0.0816 |
0.1311 |
0.6667 |
0.1593 |
0.2571 |
0.7027 |
0.1667 |
0.2694 |
0.3433 |
0.1075 |
0.1637 |
0.5154 |
0.1182 |
0.1923 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
SZEGED | closed |
0.0133 |
0.0750 |
0.0226 |
0.0275 |
0.1379 |
0.0459 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.7216 |
0.3097 |
0.4334 |
0.7216 |
0.2244 |
0.3423 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
EL
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
MUMULS | closed |
0.2553 |
0.2500 |
0.2526 |
0.4209 |
0.3588 |
0.3874 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.1392 |
0.1732 |
0.1544 |
0.2549 |
0.2759 |
0.2650 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
RACAI | closed |
0.4069 |
0.2798 |
0.3316 |
0.5117 |
0.3338 |
0.4040 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.6667 |
0.2500 |
0.3636 |
0.6667 |
0.2500 |
0.3636 |
0.2321 |
0.1024 |
0.1421 |
0.3145 |
0.1326 |
0.1866 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
SZEGED | closed |
0.2495 |
0.3958 |
0.3061 |
0.3165 |
0.4520 |
0.3723 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
ES
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
MUMULS | closed |
0.2800 |
0.2569 |
0.2679 |
0.3702 |
0.3333 |
0.3508 |
0.3646 |
0.4529 |
0.4040 |
0.5359 |
0.5852 |
0.5595 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.0000 |
0.0000 |
0.0000 |
0.1429 |
0.0020 |
0.0040 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
RACAI | closed |
0.6111 |
0.2018 |
0.3034 |
0.6623 |
0.2208 |
0.3312 |
0.6559 |
0.2735 |
0.3861 |
0.6758 |
0.2758 |
0.3917 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.5909 |
0.0783 |
0.1383 |
0.6786 |
0.0779 |
0.1397 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
SZEGED | closed |
0.0729 |
0.5872 |
0.1297 |
0.0856 |
0.6364 |
0.1509 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
FA
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
FR
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
LATL | open |
0.4486 |
0.3542 |
0.3959 |
0.5146 |
0.3657 |
0.4276 |
0.5398 |
0.9048 |
0.6762 |
0.5568 |
0.9333 |
0.6975 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.5000 |
0.1176 |
0.1905 |
0.8214 |
0.1456 |
0.2473 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
MUMULS | closed |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.2000 |
0.0095 |
0.0182 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.1677 |
0.2353 |
0.1958 |
0.3739 |
0.4177 |
0.3946 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
RACAI | closed |
0.7255 |
0.1365 |
0.2298 |
0.7959 |
0.1352 |
0.2311 |
0.7000 |
0.6667 |
0.6829 |
0.7400 |
0.7048 |
0.7220 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.7294 |
0.5210 |
0.6078 |
0.7580 |
0.5253 |
0.6206 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
SZEGED | closed |
0.1000 |
0.0443 |
0.0614 |
0.1125 |
0.0468 |
0.0661 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
HE
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
MUMULS | closed |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
HU
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
MUMULS | closed |
0.4348 |
0.2740 |
0.3361 |
0.6667 |
0.3843 |
0.4876 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.7258 |
0.7627 |
0.7438 |
0.7332 |
0.7960 |
0.7633 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
RACAI | closed |
0.6383 |
0.2055 |
0.3109 |
0.7978 |
0.2527 |
0.3838 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.8294 |
0.6864 |
0.7512 |
0.8241 |
0.6725 |
0.7406 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
SZEGED | closed |
0.5263 |
0.3425 |
0.4149 |
0.5344 |
0.3594 |
0.4298 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.7882 |
0.7571 |
0.7723 |
0.7882 |
0.6751 |
0.7273 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
IT
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
RACAI | closed |
0.7000 |
0.0805 |
0.1443 |
0.7727 |
0.0876 |
0.1574 |
0.3636 |
0.0533 |
0.0930 |
0.3636 |
0.0528 |
0.0922 |
0.3333 |
0.0909 |
0.1429 |
0.6000 |
0.1364 |
0.2222 |
0.6667 |
0.1200 |
0.2034 |
0.7200 |
0.1200 |
0.2057 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
SZEGED | closed |
0.0751 |
0.4483 |
0.1287 |
0.1175 |
0.5825 |
0.1955 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
LT
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
MUMULS | closed |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
MT
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
MUMULS | closed |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
RACAI | closed |
0.1837 |
0.0347 |
0.0584 |
0.1731 |
0.0333 |
0.0559 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.2000 |
0.0108 |
0.0205 |
0.1923 |
0.0088 |
0.0169 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
PL
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
MUMULS | closed |
0.5077 |
0.3905 |
0.4415 |
0.5977 |
0.4409 |
0.5075 |
0.8615 |
0.7509 |
0.8024 |
0.9013 |
0.7755 |
0.8337 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
SZEGED | closed |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
PT
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
MUMULS | closed |
0.5823 |
0.4407 |
0.5017 |
0.7461 |
0.5022 |
0.6004 |
0.8125 |
0.1605 |
0.2680 |
0.8788 |
0.1883 |
0.3102 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.5172 |
0.1667 |
0.2521 |
0.7746 |
0.2200 |
0.3427 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
SZEGED | closed |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
RO
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
MUMULS | closed |
0.7692 |
0.8148 |
0.7914 |
0.7774 |
0.8097 |
0.7932 |
0.7748 |
0.8069 |
0.7905 |
0.8359 |
0.8302 |
0.8330 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.6667 |
0.2933 |
0.4074 |
0.6915 |
0.3283 |
0.4452 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
RACAI | closed |
0.9167 |
0.8148 |
0.8627 |
0.9237 |
0.7958 |
0.8550 |
0.8197 |
0.6897 |
0.7491 |
0.8292 |
0.6913 |
0.7540 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.8864 |
0.5200 |
0.6555 |
0.8843 |
0.5404 |
0.6708 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
SL
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
MUMULS | closed |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.3899 |
0.4269 |
0.4075 |
0.5699 |
0.5406 |
0.5549 |
0.4474 |
0.1574 |
0.2329 |
0.5694 |
0.1775 |
0.2706 |
0.1667 |
0.0217 |
0.0385 |
0.3000 |
0.0330 |
0.0594 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
RACAI | closed |
0.6667 |
0.0444 |
0.0833 |
0.6667 |
0.0385 |
0.0727 |
0.5390 |
0.3004 |
0.3858 |
0.7241 |
0.3327 |
0.4559 |
0.6757 |
0.2315 |
0.3448 |
0.7368 |
0.2424 |
0.3648 |
0.5000 |
0.0109 |
0.0213 |
0.4286 |
0.0110 |
0.0214 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
SV
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
RACAI | closed |
0.4000 |
0.1429 |
0.2105 |
0.4000 |
0.1333 |
0.2000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.5614 |
0.2065 |
0.3019 |
0.5614 |
0.2065 |
0.3019 |
0.5000 |
0.0196 |
0.0377 |
0.7500 |
0.0226 |
0.0438 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
SZEGED | closed |
0.0106 |
0.2143 |
0.0203 |
0.0142 |
0.2667 |
0.0269 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
TR
System | Track | LVC | IReflV |
VPC | ID | OTH |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Per-MWE results | Per-token results |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
Precision | Recall | F-measure | Precision | Recall | F-measure |
MUMULS | closed |
0.5766 |
0.3970 |
0.4702 |
0.6187 |
0.4289 |
0.5066 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.4513 |
0.2048 |
0.2818 |
0.5714 |
0.2303 |
0.3283 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
0.0000 |
RACAI | closed |
0.6797 |
0.5226 |
0.5909 |
0.6818 |
0.5237 |
0.5924 |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
n/a |
0.5921 |
0.3614 |
0.4489 |
0.5987 |
0.3551 |
0.4458 |
0.5455 |
0.4528 |
0.4948 |
0.5455 |
0.4528 |
0.4948 |