command line (run on 2004 Oct 29 at 14:17:47): ../src/md-eval-v19a.pl -af -e -D -d -W -w -t 1.0 -l 2 -u md_test12.uem -r md_test12.ref.rttm -s md_test12.sys.rttm Word-based metadata alignment, max gap between matching words = 1.0 sec Metadata evaluation parameters: word-optimized metadata mapping max gap between matching metadata events = 0.1 words max extent to match for SU's = 2 words Speaker Diarization evaluation parameters: The max time to extend no-score zones for NON-LEX exclusions is 0.5 sec The no-score collar at SPEAKER boundaries is 0 sec Exclusion zones for evaluation and scoring are: -----MetaData----- -----SpkrData----- exclusion set name: DEFAULT DEFAULT DEFAULT DEFAULT token type/subtype no-eval no-score no-eval no-score (UEM) X X LEXEME/un-lex X NON-LEX/breath X NON-LEX/cough X NON-LEX/laugh X NON-LEX/lipsmack X NON-LEX/other X NON-LEX/sneeze X NOSCORE/ X X X X NO_RT_METADATA/ X SU/unannotated X Word alignment and scoring details for channel 1 of file md_test12 ref del ins sub REF: token type tbeg tend speaker SYS: token type Rtbeg Rtend tbeg tend sys-speaker 1 - - 0 wordzero lex 9.00 10.00 spkr1r wordzero lex ( 9.00 10.00) 9.00 10.00 spkr1s 1 - - 0 firstword lex 10.00 11.00 spkr1r firstword lex ( 10.00 11.00) 10.00 11.00 spkr1s 1 - - 0 secondword lex 11.00 12.00 spkr1r secondword lex ( 11.00 12.00) 11.00 12.00 spkr1s 1 - - 0 thirdword lex 12.00 13.00 spkr1r thirdword lex ( 12.00 13.00) 12.00 13.00 spkr1s 1 - - 0 fourthword lex 13.00 14.00 spkr1r fourthword lex ( 13.00 14.00) 13.00 14.00 spkr1s 1 - - 0 fifthword lex 14.00 15.00 spkr1r fifthword lex ( 14.00 15.00) 14.00 15.00 spkr1s 1 - - 0 sixthword lex 15.00 16.00 spkr1r sixthword lex ( 15.00 16.00) 15.00 16.00 spkr1s 1 - - 0 seventhword lex 16.00 17.00 spkr1r seventhword lex ( 16.00 17.00) 16.00 17.00 spkr1s 1 - - 0 eighthword lex 17.00 18.00 spkr1r eighthword lex ( 17.00 18.00) 17.00 18.00 spkr1s 1 - - 0 ninthword lex 18.00 19.00 spkr1r ninthword lex ( 18.00 19.00) 18.00 19.00 spkr1s 1 - - 0 tenthword lex 19.00 20.00 spkr1r tenthword lex ( 19.00 20.00) 19.00 20.00 spkr1s 1 - - 0 eleventhword lex 20.00 21.00 spkr1r eleventhword lex ( 20.00 21.00) 20.00 21.00 spkr1s 1 - - 0 twelfthword lex 21.00 22.00 spkr1r twelfthword lex ( 21.00 22.00) 21.00 22.00 spkr1s 1 - - 0 thirteenthword lex 22.00 23.00 spkr1r thirteenthword lex ( 22.00 23.00) 22.00 23.00 spkr1s 1 - - 0 fourteenthword lex 23.00 24.00 spkr1r fourteenthword lex ( 23.00 24.00) 23.00 24.00 spkr1s 1 - - 0 fifteenthword lex 24.00 25.00 spkr1r fifteenthword lex ( 24.00 25.00) 24.00 25.00 spkr1s 1 - - 0 sixteenthword lex 25.00 26.00 spkr1r sixteenthword lex ( 25.00 26.00) 25.00 26.00 spkr1s 1 - - 0 seventeenthword lex 26.00 27.00 spkr1r seventeenthword lex ( 26.00 27.00) 26.00 27.00 spkr1s 1 - - 0 eighteenthword lex 27.00 28.00 spkr1r eighteenthword lex ( 27.00 28.00) 27.00 28.00 spkr1s 1 - - 0 nineteenthword lex 28.00 29.00 spkr1r nineteenthword lex ( 28.00 29.00) 28.00 29.00 spkr1s 1 - - 0 twentiethword lex 29.00 30.00 spkr1r twentiethword lex ( 29.00 30.00) 29.00 30.00 spkr1s EDIT alignment and scoring details for channel 1 of file md_test12 ref del ins sub REF: token type tbeg tend speaker SYS: token type Rtbeg Rtend tbeg tend sys-speaker 1 - - 0 revision EDIT 12.00 15.00 spkr1r revision EDIT ( 11.00 14.00) 11.00 14.00 spkr1s 1 - - 0 revision EDIT 18.00 21.00 spkr1r revision EDIT ( 16.00 19.00) 16.00 19.00 spkr1s 0 - 1 - --- --- --- --- --- restart EDIT ( 21.00 22.00) 21.00 22.00 spkr1s 1 - - 0 revision EDIT 24.00 27.00 spkr1r revision EDIT ( 25.00 26.00) 25.00 26.00 spkr1s IP alignment and scoring details for channel 1 of file md_test12 ref del ins sub REF: token type tbeg tend speaker SYS: token type Rtbeg Rtend tbeg tend sys-speaker 1 1 - - edit IP 15.00 15.00 spkr1r --- --- ( --- --- ) --- --- --- 1 1 - - edit IP 21.00 21.00 spkr1r --- --- ( --- --- ) --- --- --- 1 1 - - edit IP 27.00 27.00 spkr1r --- --- ( --- --- ) --- --- --- 0 - 1 - --- --- --- --- --- edit IP ( 14.00 14.00) 14.00 14.00 spkr1s 0 - 1 - --- --- --- --- --- edit IP ( 19.00 19.00) 19.00 19.00 spkr1s 0 - 1 - --- --- --- --- --- edit IP ( 22.00 22.00) 22.00 22.00 spkr1s 0 - 1 - --- --- --- --- --- edit IP ( 26.00 26.00) 26.00 26.00 spkr1s SU alignment and scoring details for channel 1 of file md_test12 ref del ins sub REF: token type tbeg tend speaker SYS: token type Rtbeg Rtend tbeg tend sys-speaker 1 - - 0 statement SU 9.00 30.00 spkr1r statement SU ( 9.00 30.00) 9.00 30.00 spkr1s Chronological display of sys data aligned with ref data for file 'md_test12', channel '1' ----------------------- reference ----------------------- | mapped | --------------------- system output --------------------- --type-- -subtyp- -----word/spkr----- -tbeg- -tend- | ref_ID | --type-- -subtyp- -----word/spkr----- -tbeg- -tend- beg SEGMENT spkr1r 9.00 | | beg SPEAKER adult_fe spkr1r 9.00 | | 9.00 | |beg SPEAKER adult_fe spkr1s=>spkr1r 9.00 beg SU statemen spkr1r 9.00 | SU1 |beg SU statemen spkr1s=>spkr1r 9.00 LEXEME lex WORDZERO 9.00 10.00 | LX1 | LEXEME lex WORDZERO 9.00 10.00 LEXEME lex FIRSTWORD 10.00 11.00 | LX2 | LEXEME lex FIRSTWORD 10.00 11.00 11.00 | ED1 |beg EDIT revision spkr1s=>spkr1r 11.00 dw=-1 LEXEME lex SECONDWORD 11.00 12.00 | LX3 | LEXEME lex SECONDWORD 11.00 12.00 beg EDIT revision spkr1r 12.00 | ED1 | LEXEME lex THIRDWORD 12.00 13.00 | LX4 | LEXEME lex THIRDWORD 12.00 13.00 LEXEME lex FOURTHWORD 13.00 14.00 | LX5 | LEXEME lex FOURTHWORD 13.00 14.00 14.00 | ED1 |end EDIT revision spkr1s=>spkr1r 14.00 dw=-1 14.00 | **FA** | IP edit spkr1s=>spkr1r 14.00 LEXEME lex FIFTHWORD 14.00 15.00 | LX6 | LEXEME lex FIFTHWORD 14.00 15.00 end EDIT revision spkr1r 15.00 | ED1 | IP edit spkr1r 15.00 | *Miss* | LEXEME lex SIXTHWORD 15.00 16.00 | LX7 | LEXEME lex SIXTHWORD 15.00 16.00 16.00 | ED2 |beg EDIT revision spkr1s=>spkr1r 16.00 dw=-2 LEXEME lex SEVENTHWORD 16.00 17.00 | LX8 | LEXEME lex SEVENTHWORD 16.00 17.00 LEXEME lex EIGHTHWORD 17.00 18.00 | LX9 | LEXEME lex EIGHTHWORD 17.00 18.00 beg EDIT revision spkr1r 18.00 | ED2 | LEXEME lex NINTHWORD 18.00 19.00 | LX10 | LEXEME lex NINTHWORD 18.00 19.00 19.00 | ED2 |end EDIT revision spkr1s=>spkr1r 19.00 dw=-2 19.00 | **FA** | IP edit spkr1s=>spkr1r 19.00 LEXEME lex TENTHWORD 19.00 20.00 | LX11 | LEXEME lex TENTHWORD 19.00 20.00 LEXEME lex ELEVENTHWORD 20.00 21.00 | LX12 | LEXEME lex ELEVENTHWORD 20.00 21.00 end EDIT revision spkr1r 21.00 | ED2 | 21.00 | **FA** |beg EDIT restart spkr1s=>spkr1r 21.00 IP edit spkr1r 21.00 | *Miss* | LEXEME lex TWELFTHWORD 21.00 22.00 | LX13 | LEXEME lex TWELFTHWORD 21.00 22.00 22.00 | **FA** |end EDIT restart spkr1s=>spkr1r 22.00 22.00 | **FA** | IP edit spkr1s=>spkr1r 22.00 LEXEME lex THIRTEENTHWORD 22.00 23.00 | LX14 | LEXEME lex THIRTEENTHWORD 22.00 23.00 LEXEME lex FOURTEENTHWORD 23.00 24.00 | LX15 | LEXEME lex FOURTEENTHWORD 23.00 24.00 beg EDIT revision spkr1r 24.00 | ED3 | LEXEME lex FIFTEENTHWORD 24.00 25.00 | LX16 | LEXEME lex FIFTEENTHWORD 24.00 25.00 25.00 | ED3 |beg EDIT revision spkr1s=>spkr1r 25.00 dw=1 LEXEME lex SIXTEENTHWORD 25.00 26.00 | LX17 | LEXEME lex SIXTEENTHWORD 25.00 26.00 26.00 | ED3 |end EDIT revision spkr1s=>spkr1r 26.00 dw=-1 26.00 | **FA** | IP edit spkr1s=>spkr1r 26.00 LEXEME lex SEVENTEENTHWORD 26.00 27.00 | LX18 | LEXEME lex SEVENTEENTHWORD 26.00 27.00 end EDIT revision spkr1r 27.00 | ED3 | IP edit spkr1r 27.00 | *Miss* | LEXEME lex EIGHTEENTHWORD 27.00 28.00 | LX19 | LEXEME lex EIGHTEENTHWORD 27.00 28.00 end SPEAKER adult_fe spkr1r 28.00 | | 28.00 | |end SPEAKER adult_fe spkr1s=>spkr1r 28.00 end SEGMENT spkr1r 28.00 | | LEXEME lex NINETEENTHWORD 28.00 29.00 | LX20 | LEXEME lex NINETEENTHWORD 28.00 29.00 LEXEME lex TWENTIETHWORD 29.00 30.00 | LX21 | LEXEME lex TWENTIETHWORD 29.00 30.00 end SU statemen spkr1r 30.00 | SU1 |end SU statemen spkr1s=>spkr1r 30.00 *** Performance analysis for EDITs *** overall error SCORE = 100.00% EDIT word coverage statistics -- in terms of reference words Nref Ndel Nins Nsub %Del %Ins %Sub %D+I %Tot ALL 9 5 4 0 55.56 44.44 0.00 100.00 100.00 EDIT detection statistics -- in terms of # of EDITs Nref Ndel Nins Nsub %Del %Ins %Sub %D+I %Tot ALL 3 0 1 0 0.00 33.33 0.00 33.33 33.33 f=md_test12 3 0 1 0 0.00 33.33 0.00 33.33 33.33 EDIT detection confusion matrix -- in terms of # of EDITs ALL - ref\sys restart revision {Miss} restart 0 0 0 revision 0 3 0 {FA} 1 0 EDIT word offset statistics for ALL data word offsets: <-3 -3 -2 -1 0 1 2 3 >3 BEG: 0 - 1 1 - 1 - - 0 END: 0 - 1 2 - - - - 0 *** Performance analysis for IPs *** overall error SCORE = 233.33% IP (exact) detection statistics -- in terms of reference words Nref Ndel Nins Nsub %Del %Ins %Sub %D+I %Tot ALL 3 3 4 0 100.00 133.33 0.00 233.33 233.33 IP detection statistics -- in terms of # of IPs Nref Ndel Nins Nsub %Del %Ins %Sub %D+I %Tot ALL 3 3 4 0 100.00 133.33 0.00 233.33 233.33 f=md_test12 3 3 4 0 100.00 133.33 0.00 233.33 233.33 IP detection confusion matrix -- in terms of # of IPs ALL - ref\sys edit {Miss} edit 0 3 {FA} 4 *** Performance analysis for SUs *** overall error SCORE = 0.00% SU (exact) end detection statistics -- in terms of reference words Nref Ndel Nins Nsub %Del %Ins %Sub %D+I %Tot ALL 1 0 0 0 0.00 0.00 0.00 0.00 0.00 SU detection statistics -- in terms of # of SUs Nref Ndel Nins Nsub %Del %Ins %Sub %D+I %Tot ALL 1 0 0 0 0.00 0.00 0.00 0.00 0.00 f=md_test12 1 0 0 0 0.00 0.00 0.00 0.00 0.00 SU detection confusion matrix -- in terms of # of SUs ALL - ref\sys statemen {Miss} statement 1 0 {FA} 0 SU word offset statistics for ALL data word offsets: <-3 -3 -2 -1 0 1 2 3 >3 BEG: 0 - - - 1 - - - 0 END: 0 - - - 1 - - - 0 *** Performance analysis for Speaker Diarization for f=md_test12 *** EVAL TIME = 21.00 secs EVAL SPEECH = 19.00 secs ( 90.5 percent of evaluated time) SCORED TIME = 21.00 secs (100.0 percent of evaluated time) SCORED SPEECH = 19.00 secs ( 90.5 percent of scored time) EVAL WORDS = 21 SCORED WORDS = 21 (100.0 percent of evaluated words) --------------------------------------------- MISSED SPEECH = 0.00 secs ( 0.0 percent of scored time) FALARM SPEECH = 0.00 secs ( 0.0 percent of scored time) MISSED WORDS = 2 ( 9.5 percent of scored words) --------------------------------------------- SCORED SPEAKER TIME = 19.00 secs (100.0 percent of scored speech) MISSED SPEAKER TIME = 0.00 secs ( 0.0 percent of scored speaker time) FALARM SPEAKER TIME = 0.00 secs ( 0.0 percent of scored speaker time) SPEAKER ERROR TIME = 0.00 secs ( 0.0 percent of scored speaker time) SPEAKER ERROR WORDS = 0 ( 0.0 percent of scored speaker words) --------------------------------------------- OVERALL SPEAKER DIARIZATION ERROR = 0.00 percent of scored speaker time `(f=md_test12) --------------------------------------------- Speaker type confusion matrix -- speaker weighted REF\SYS (count) adult_female MISS adult_female 1 / 100.0% 0 / 0.0% FALSE ALARM 0 / 0.0% --------------------------------------------- Speaker type confusion matrix -- time weighted REF\SYS (seconds) adult_female MISS adult_female 19.00 / 100.0% 0.00 / 0.0% FALSE ALARM 0.00 / 0.0% --------------------------------------------- *** Performance analysis for Speaker Diarization for ALL *** EVAL TIME = 21.00 secs EVAL SPEECH = 19.00 secs ( 90.5 percent of evaluated time) SCORED TIME = 21.00 secs (100.0 percent of evaluated time) SCORED SPEECH = 19.00 secs ( 90.5 percent of scored time) EVAL WORDS = 21 SCORED WORDS = 21 (100.0 percent of evaluated words) --------------------------------------------- MISSED SPEECH = 0.00 secs ( 0.0 percent of scored time) FALARM SPEECH = 0.00 secs ( 0.0 percent of scored time) MISSED WORDS = 2 ( 9.5 percent of scored words) --------------------------------------------- SCORED SPEAKER TIME = 19.00 secs (100.0 percent of scored speech) MISSED SPEAKER TIME = 0.00 secs ( 0.0 percent of scored speaker time) FALARM SPEAKER TIME = 0.00 secs ( 0.0 percent of scored speaker time) SPEAKER ERROR TIME = 0.00 secs ( 0.0 percent of scored speaker time) SPEAKER ERROR WORDS = 0 ( 0.0 percent of scored speaker words) --------------------------------------------- OVERALL SPEAKER DIARIZATION ERROR = 0.00 percent of scored speaker time `(ALL) --------------------------------------------- Speaker type confusion matrix -- speaker weighted REF\SYS (count) adult_female MISS adult_female 1 / 100.0% 0 / 0.0% FALSE ALARM 0 / 0.0% --------------------------------------------- Speaker type confusion matrix -- time weighted REF\SYS (seconds) adult_female MISS adult_female 19.00 / 100.0% 0.00 / 0.0% FALSE ALARM 0.00 / 0.0% ---------------------------------------------