Skip to main content.

Results of the shared task on automatic identification of verbal multiword expressions

Last updated: February 10, 2017

Participants

Seven teams submitted results to the shared task: 6 to the closed track and 1 to the open track (no system contributed to both tracks); 5 systems were multilingual. All 18 languages were covered, by a variable subset of systems each.

Authors of the systems and their affiliations will be made public after reviewing system description papers.

Results

The evaluation results (disregarding VMWE categories) are given below, for each language separately. Each ranking is based on the F-measure and is specific to three parameters:

Categorizing the identified VMWEs was not required but 4 out of the 7 systems submitted results in which VMWE are categorized. Detailed results per category (with no ranking) are available below.

Recall, that the aim of this shared task is a cross-language discussion rather than a true competition. The participation policy may have introduced bias to the annotated corpora and the participating systems. Therefore, the ranking results should be considered with caution.

BG

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
TRANSITIONclosed0.68870.55180.612710.78980.56910.66151
MUMULSclosed0.35810.33620.346820.76860.48090.59162

CS

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
TRANSITIONclosed0.78970.65600.716710.82460.66550.73651
ADAPTclosed0.59310.56210.577230.81910.65610.72862
RACAIclosed0.70090.59180.641820.81900.62280.70763
MUMULSclosed0.44130.10280.166740.77470.13870.23524

DE

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
SZEGEDclosed0.51540.33400.405320.65920.34680.45451
TRANSITIONclosed0.55030.32800.411010.59660.31330.41092
ADAPTclosed0.33080.17400.228030.70590.28370.40483
MUMULSclosed0.32770.15600.211440.69880.22860.34454
RACAIclosed0.36520.13000.191750.67160.17930.28305

EL

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
TRANSITIONclosed0.36120.45000.400710.46350.47420.46881
ADAPTclosed0.34370.28800.313440.53800.36010.43142
MUMULSclosed0.20870.25800.230850.42940.41430.42173
SZEGEDclosed0.30840.33000.318820.44510.37570.40754
RACAIclosed0.42860.25200.317430.56160.29530.38715

ES

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
TRANSITIONclosed0.61220.54000.573910.65740.52520.58391
ADAPTclosed0.61050.34800.443320.74480.36700.49172
MUMULSclosed0.36730.31000.336240.62520.39950.48753
SZEGEDclosed0.25750.50000.339930.36350.56290.44184
RACAIclosed0.64470.19600.300650.72330.19670.30935

FA

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
TRANSITIONclosed0.87700.85600.866410.91590.88850.90201
ADAPTclosed0.79760.80400.800820.86600.84160.85362

FR

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
ADAPTclosed0.61470.43400.508820.80880.49640.61521
TRANSITIONclosed0.74840.47000.577410.79470.48560.60282
RACAIclosed0.74150.35000.475530.78720.36730.50093
SZEGEDclosed0.06390.05200.057360.52180.24820.33644
MUMULSclosed0.14660.06800.092950.50890.20670.29405
LIFclosed0.80560.05800.108240.81940.05320.10006
LATLopen0.48150.46800.474610.58650.51080.54611

HE

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
TRANSITIONclosed0.73970.21600.334410.75370.19750.31301
MUMULSclosed0.00000.00000.0000n/a0.00000.00000.0000n/a

HU

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
SZEGEDclosed0.79360.69340.740110.80570.63170.70811
MUMULSclosed0.62910.61520.622150.71320.66570.68862
TRANSITIONclosed0.64840.75750.698720.65020.70120.67473
ADAPTclosed0.75700.59920.668930.78460.57100.66104
RACAIclosed0.80290.54710.650840.82080.50150.62265

IT

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
TRANSITIONclosed0.53540.31800.399010.61340.33780.43571
SZEGEDclosed0.15030.15600.153140.40540.30640.34902
ADAPTclosed0.61740.14200.230920.69640.15320.25113
RACAIclosed0.61250.09800.169030.68370.10530.18244

LT

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
TRANSITIONclosed0.66670.18000.283510.67860.15570.25331
MUMULSclosed0.00000.00000.0000n/a0.00000.00000.0000n/a

MT

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
TRANSITIONclosed0.15650.13400.144410.18430.14600.16291
ADAPTclosed0.20430.03800.064120.30840.05180.08872
RACAIclosed0.23330.02800.050030.24810.02590.04693
MUMULSclosed0.00000.00000.0000n/a0.00000.00000.0000n/a

PL

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
ADAPTclosed0.77980.60200.679520.87420.62280.72741
TRANSITIONclosed0.77090.62600.690910.80000.63120.70562
MUMULSclosed0.65620.54600.596130.83100.60130.69773
SZEGEDclosed0.00000.00000.0000n/a0.00000.00000.0000n/a

PT

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
TRANSITIONclosed0.75430.60800.673310.80050.63700.70941
ADAPTclosed0.64100.53200.581420.83480.60540.70182
MUMULSclosed0.53580.37400.440530.82470.47170.60013
SZEGEDclosed0.01290.00800.009940.68370.19870.30794

RO

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
MUMULSclosed0.76830.77600.772120.86200.81120.83581
ADAPTclosed0.75480.71400.733840.88320.76360.81902
TRANSITIONclosed0.70970.80200.753130.74400.84490.79123
RACAIclosed0.86520.70600.777510.87730.70190.77994

SL

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
TRANSITIONclosed0.43430.43000.432210.47960.45220.46551
MUMULSclosed0.35570.27600.310830.61420.36280.45622
ADAPTclosed0.51420.29000.370820.72850.32620.45063
RACAIclosed0.55030.20800.301940.73390.21450.33204

SV

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
ADAPTclosed0.48600.22030.303220.52530.22490.31491
SZEGEDclosed0.24820.29660.270330.29610.32940.31192
TRANSITIONclosed0.51000.21610.303610.53690.21500.30703
RACAIclosed0.57580.16100.251740.65380.16770.26694

TR

SystemTrackPer-MWE resultsPer-token results
PrecisionRecallF-measureRankPrecisionRecallF-measureRank
TRANSITIONclosed0.61060.50700.554010.61230.50390.55281
ADAPTclosed0.45410.40520.428330.59930.47280.52852
RACAIclosed0.63040.43910.517620.63400.43480.51593
MUMULSclosed0.45570.27740.344940.64520.35020.45404

Results per VMWE category (unranked)

BG

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
MUMULSclosed 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4147 0.5320 0.4661 0.7250 0.7114 0.7181 n/a n/a n/a n/a n/a n/a 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 n/a n/a n/a n/a n/a n/a

CS

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
MUMULSclosed 0.3182 0.0408 0.0724 0.5641 0.0566 0.1029 0.6395 0.1297 0.2156 0.8789 0.1573 0.2668 n/a n/a n/a n/a n/a n/a 0.3043 0.0365 0.0651 0.5094 0.0467 0.0856 n/a n/a n/a n/a n/a n/a
RACAIclosed 0.7460 0.2741 0.4009 0.7970 0.2728 0.4065 0.7109 0.7554 0.7325 0.8160 0.8206 0.8183 n/a n/a n/a n/a n/a n/a 0.5909 0.1354 0.2203 0.7273 0.1522 0.2518 n/a n/a n/a n/a n/a n/a

DE

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
MUMULSclosed 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5638 0.2345 0.3312 0.6803 0.2660 0.3825 0.2766 0.0607 0.0996 0.6196 0.1005 0.1730 n/a n/a n/a n/a n/a n/a
RACAIclosed 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4000 0.1000 0.1600 0.3333 0.0816 0.1311 0.6667 0.1593 0.2571 0.7027 0.1667 0.2694 0.3433 0.1075 0.1637 0.5154 0.1182 0.1923 n/a n/a n/a n/a n/a n/a
SZEGEDclosed 0.0133 0.0750 0.0226 0.0275 0.1379 0.0459 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7216 0.3097 0.4334 0.7216 0.2244 0.3423 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 n/a n/a n/a n/a n/a n/a

EL

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
MUMULSclosed 0.2553 0.2500 0.2526 0.4209 0.3588 0.3874 n/a n/a n/a n/a n/a n/a 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1392 0.1732 0.1544 0.2549 0.2759 0.2650 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
RACAIclosed 0.4069 0.2798 0.3316 0.5117 0.3338 0.4040 n/a n/a n/a n/a n/a n/a 0.6667 0.2500 0.3636 0.6667 0.2500 0.3636 0.2321 0.1024 0.1421 0.3145 0.1326 0.1866 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
SZEGEDclosed 0.2495 0.3958 0.3061 0.3165 0.4520 0.3723 n/a n/a n/a n/a n/a n/a 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

ES

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
MUMULSclosed 0.2800 0.2569 0.2679 0.3702 0.3333 0.3508 0.3646 0.4529 0.4040 0.5359 0.5852 0.5595 n/a n/a n/a n/a n/a n/a 0.0000 0.0000 0.0000 0.1429 0.0020 0.0040 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
RACAIclosed 0.6111 0.2018 0.3034 0.6623 0.2208 0.3312 0.6559 0.2735 0.3861 0.6758 0.2758 0.3917 n/a n/a n/a n/a n/a n/a 0.5909 0.0783 0.1383 0.6786 0.0779 0.1397 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
SZEGEDclosed 0.0729 0.5872 0.1297 0.0856 0.6364 0.1509 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 n/a n/a n/a n/a n/a n/a 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

FA

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure

FR

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
LATLopen 0.4486 0.3542 0.3959 0.5146 0.3657 0.4276 0.5398 0.9048 0.6762 0.5568 0.9333 0.6975 n/a n/a n/a n/a n/a n/a 0.5000 0.1176 0.1905 0.8214 0.1456 0.2473 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
MUMULSclosed 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2000 0.0095 0.0182 n/a n/a n/a n/a n/a n/a 0.1677 0.2353 0.1958 0.3739 0.4177 0.3946 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
RACAIclosed 0.7255 0.1365 0.2298 0.7959 0.1352 0.2311 0.7000 0.6667 0.6829 0.7400 0.7048 0.7220 n/a n/a n/a n/a n/a n/a 0.7294 0.5210 0.6078 0.7580 0.5253 0.6206 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
SZEGEDclosed 0.1000 0.0443 0.0614 0.1125 0.0468 0.0661 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 n/a n/a n/a n/a n/a n/a 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

HE

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
MUMULSclosed 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 n/a n/a n/a n/a n/a n/a 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

HU

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
MUMULSclosed 0.4348 0.2740 0.3361 0.6667 0.3843 0.4876 n/a n/a n/a n/a n/a n/a 0.7258 0.7627 0.7438 0.7332 0.7960 0.7633 n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a
RACAIclosed 0.6383 0.2055 0.3109 0.7978 0.2527 0.3838 n/a n/a n/a n/a n/a n/a 0.8294 0.6864 0.7512 0.8241 0.6725 0.7406 n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a
SZEGEDclosed 0.5263 0.3425 0.4149 0.5344 0.3594 0.4298 n/a n/a n/a n/a n/a n/a 0.7882 0.7571 0.7723 0.7882 0.6751 0.7273 n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a

IT

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
RACAIclosed 0.7000 0.0805 0.1443 0.7727 0.0876 0.1574 0.3636 0.0533 0.0930 0.3636 0.0528 0.0922 0.3333 0.0909 0.1429 0.6000 0.1364 0.2222 0.6667 0.1200 0.2034 0.7200 0.1200 0.2057 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
SZEGEDclosed 0.0751 0.4483 0.1287 0.1175 0.5825 0.1955 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

LT

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
MUMULSclosed 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 n/a n/a n/a n/a n/a n/a

MT

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
MUMULSclosed 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
RACAIclosed 0.1837 0.0347 0.0584 0.1731 0.0333 0.0559 n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a 0.2000 0.0108 0.0205 0.1923 0.0088 0.0169 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

PL

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
MUMULSclosed 0.5077 0.3905 0.4415 0.5977 0.4409 0.5075 0.8615 0.7509 0.8024 0.9013 0.7755 0.8337 n/a n/a n/a n/a n/a n/a 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 n/a n/a n/a n/a n/a n/a
SZEGEDclosed 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 n/a n/a n/a n/a n/a n/a 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 n/a n/a n/a n/a n/a n/a

PT

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
MUMULSclosed 0.5823 0.4407 0.5017 0.7461 0.5022 0.6004 0.8125 0.1605 0.2680 0.8788 0.1883 0.3102 n/a n/a n/a n/a n/a n/a 0.5172 0.1667 0.2521 0.7746 0.2200 0.3427 n/a n/a n/a n/a n/a n/a
SZEGEDclosed 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 n/a n/a n/a n/a n/a n/a 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 n/a n/a n/a n/a n/a n/a

RO

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
MUMULSclosed 0.7692 0.8148 0.7914 0.7774 0.8097 0.7932 0.7748 0.8069 0.7905 0.8359 0.8302 0.8330 n/a n/a n/a n/a n/a n/a 0.6667 0.2933 0.4074 0.6915 0.3283 0.4452 n/a n/a n/a n/a n/a n/a
RACAIclosed 0.9167 0.8148 0.8627 0.9237 0.7958 0.8550 0.8197 0.6897 0.7491 0.8292 0.6913 0.7540 n/a n/a n/a n/a n/a n/a 0.8864 0.5200 0.6555 0.8843 0.5404 0.6708 n/a n/a n/a n/a n/a n/a

SL

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
MUMULSclosed 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.3899 0.4269 0.4075 0.5699 0.5406 0.5549 0.4474 0.1574 0.2329 0.5694 0.1775 0.2706 0.1667 0.0217 0.0385 0.3000 0.0330 0.0594 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
RACAIclosed 0.6667 0.0444 0.0833 0.6667 0.0385 0.0727 0.5390 0.3004 0.3858 0.7241 0.3327 0.4559 0.6757 0.2315 0.3448 0.7368 0.2424 0.3648 0.5000 0.0109 0.0213 0.4286 0.0110 0.0214 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

SV

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
RACAIclosed 0.4000 0.1429 0.2105 0.4000 0.1333 0.2000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5614 0.2065 0.3019 0.5614 0.2065 0.3019 0.5000 0.0196 0.0377 0.7500 0.0226 0.0438 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
SZEGEDclosed 0.0106 0.2143 0.0203 0.0142 0.2667 0.0269 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

TR

SystemTrackLVCIReflV VPCIDOTH
Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results Per-MWE resultsPer-token results
PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure PrecisionRecallF-measurePrecisionRecallF-measure
MUMULSclosed 0.5766 0.3970 0.4702 0.6187 0.4289 0.5066 n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a 0.4513 0.2048 0.2818 0.5714 0.2303 0.3283 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
RACAIclosed 0.6797 0.5226 0.5909 0.6818 0.5237 0.5924 n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a 0.5921 0.3614 0.4489 0.5987 0.3551 0.4458 0.5455 0.4528 0.4948 0.5455 0.4528 0.4948