Each test is described by comments in the .ref.rttm file.

STRUCTURAL METADATA   Tests to be run with options -w and -W
---------------------
md_test1  There should be no errors: three speakers (of different types),
          four SU's, and instances of all scoreable lexeme subtypes.

md_test2  Has an SU starting at  7.0 that should be completely non-scored
          has an SU starting at 10.0 that should be scored and without error
          has an SU starting at 13.0: type incomplete in ref and question in sys
          has an SU starting at 16.0 that should be scored and without error
          has an SU starting at 19.0 that in the ref ends just after the end
                                  of scored time (sys is entirely within
                                  scored time) -- illustrating some program
                                  logic for an earlier scoring rule.
        
          In the output:
             The SU starting at 7.0 should not count because it ends before
                scored time begins.
             The SU starting at 13.0 should have a type mismatch.
                 and should be an SU substitution error "in terms of # of SUs".
             There should be four scored SU's.
     
md_test3   Like test2, but the SU starting at 19.0 ends exactly at the end
           of scored time.  Contrasts with the md_test2 SU starting at 19.0.

md_test4   Has some oddball but syntactically legal speaker names.
           The program handles them OK.  SYSTEMS SHOULD NOT EMULATE THESE NAMES!!!
           Md_test4 also tests that Nref in terms of # of SU's is the same as
           Nref for exact SU boundaries in terms of words. 

md_test5   Tests subtype other.  Exercises some dark corners of the RTTM spec.

md_test6   This resembles tests 1 - 4
           The SU starting at 7.0 ends exactly at the start of
           scored time.  Thus, a system processing this SU would look at exactly
           zero point zero seconds of audio data.  The SU starting at 7.0 should
           not get scored.

md_test7   Has SU's with systematically varying number of words of overlap
           between ref SU and sys SU.  But all the words are "blah".
           This will often reveal changes to the word alignment algorithm
           (see comment at top of md_test7.ref.rttm for expected output)
 
           See md_test24 for a related variation. 

md_test8   Has EDIT of subtype REVISION, with explicit_editing_term.
           Has EDIT with edit subtype mismatch and filler subtype mismatch.
      
md_test9   Has ref=sys match on a filler and a revision.  Output should be "no errors".

           Because the words are hyphenated, each counts as two words.

md_test10  Has fillers and edits with time mismatches.  The word-warping should
           be apparent in the output, and it is.

           Because the words are hyphenated, each counts as two words.

md_test11  See the comment at the top of the ref file.  Output OK.
           (It may be useful to check this with time-based command-line options
            as well as word-based.)

           Because the words are hyphenated, each counts as two words.

md_test12  See the comment at the top of the ref file.

md_test13  The SU starting at 16.0 is out of order, but md-eval should sort the lines.
           The sys.rttm file has *negative* durations for the SEGMENT, SPEAKER, and SU
           records that start at time 10.0    Will md-eval do something appropriate???

md_test14  This has the ref and sys that will be used in md_test15  md_test16 and
           md_test17, but here (in md_test14) the UEM file has the whole thing as
           eval'd time

           Because the words are all hyphenated, each counts as two words.

md_test15 and Test16
           The UEM file has some of the filler and edit words in no-eval'd time.

           Because the words are all hyphenated, each counts as two words.

md_test17  Here, the UEM file has the entire FILLER and EDIT in no-eval'd time.

           Because the words are all hyphenated, each counts as two words.

md_test18  The ref carves up the time interval from 16.0 to 28.0 into three four-second SU's.
           The sys carves up that time interval differently.  Thus, the ref and sys carve up
           the time interval from 16.0 through 28.0 into different numbers of SU's, with
           different boundaries. 

           The expected scoring output is:
           ;;    SU (exact) end detection statistics -- in terms of reference words
           ;;                Nref  Ndel  Nins  Nsub  Nerr     %Del   %Ins   %Sub   %Err
           ;;         ALL       5     2     4  <ns>     6    40.00  80.00   <ns> 120.00
           ;;
           ;;    SU detection statistics -- in terms of # of SUs
           ;;                Nref  Ndel  Nins  Nsub  Nerr     %Del   %Ins   %Sub   %Err
           ;;         ALL       5     0     2     0     2     0.00  40.00   0.00  40.00

md_test19  Vaguely the same idea as test18.  The alignment of SU's is ambiguous, but
           the legal SU alignment giving minimum SU detection error will give
           ;;    SU (exact) end detection statistics -- in terms of reference words
           ;;                  Nref  Ndel  Nins  Nsub  Nerr     %Del   %Ins   %Sub   %Err
           ;;           ALL       5     2     3  <ns>     5    40.00  60.00   <ns> 100.00
           ;;
           ;;    SU detection statistics -- in terms of # of SUs
           ;;                  Nref  Ndel  Nins  Nsub  Nerr     %Del   %Ins   %Sub   %Err
           ;;           ALL       5     0     1     0     5    00.00  20.00   0.00  20.00

           However, with a different SU alignment, the output could be
           ;;    SU (exact) end detection statistics -- in terms of reference words
           ;;                  Nref  Ndel  Nins  Nsub  Nerr     %Del   %Ins   %Sub   %Err
           ;;           ALL       5     2     3  <ns>     5    40.00  60.00   <ns> 100.00
           ;;
           ;;    SU detection statistics -- in terms of # of SUs
           ;;                  Nref  Ndel  Nins  Nsub  Nerr     %Del   %Ins   %Sub   %Err
           ;;           ALL       5     2     3     0     5    40.00  60.00   0.00 100.00
           

md_test20  Like test1, except that there is an SU from 7.0 to 10.0, which ends exactly
           at the beginning of scored time: that SU should not count in the scoring.

           There should be 4 scored SU's with no errors.

md_test21  The ref and sys files are like test1, but the UEM file differs.
           For the SU from 13.0 to 16.0, only the time from 13.0 to 14.0 is eval'd time,
           ... which means its midpoint is non-eval'd, which means that SU should not count.
           For the  SU from 19.0 to 21.0, the time from 19.0 to 20.9 is UEM-scored time.
           There should be 3 scored SU's with no errors.

md_test22a Has 11.0 through 13.0 covered by a NOSCORE record.

           Has a SPEAKER record starting at 14.0 and an SU starting at 15.0 that have
           their midpoints during a NON-LEX.  Both should count in the scoring.

           Has an SU starting at 20.0 whose midpoint is after the end of UEM eval'd time.
           It should not count.

md_test23  Has 11.0 through 13.0 no-eval'd via the UEM file.
           Compare with md_test22a

md_test24  Has SU's with systematically varying number of words of overlap
           between ref SU and sys SU.  Unlike md_test7,  the words are distinct
           enough that ref and sys words with the same times should align in
           all SU's except the one from 12.0 through 20.0

           The SU from 12.0 through 20.0 tests the linear interpolation in the word-time warping.
           If you run it with options to generate the detailed output, you will see the following.

              Both ref and sys have a word from time 9.0 to 10.0, which get aligned
              to each other, so ref vs. sys times are identical at time 10.0

              The next word that gets matched up is from 14.0 to 15.0 in the ref
              and from 13.0 to 14.0 in the sys.  So sys time 13.0 is warped to
              the corresponding ref time (14.0)

              In between those matched-up words, there lie 4.0 seconds in the ref
              and 3.0 seconds in the sys, and (via straight line linear interpolation)
              each 1.0 of those three sys seconds equals 1.33 ref seconds.  The
              last of those three sys seconds has an inserted word (from sys time
              12.0 to 13.0).  The end time of that inserted sys word will be warped
              to 14.0 (see the end of the preceding paragraph).  So, the beginning of
              that word should be warped to 1.33 seconds earlier, which is 12.67

md_test25  This is an excerpt that blew up a previous version of md-eval
             (Sue Tranter provided this example)
              
md_test26  This has a region that is excluded by the UEM file and is NOSCORE in the RTTM
md_test27  This is md_test26 without the NOSCORE in the RTTM
md_test28  This is md_test26 without the exclusion in the UEM

md_test29  This is sd_test6 to show its output with the structural MDE command line



Speaker Diarization     Tests to be run with option -1 but neither -w nor -W
---------------------
sd_test1  spkr2 is in no-eval'd time and spkr4 is in no-eval'd time.
          There should be three scored speakers: two adult_females and one child
          with no misses and no false-alarms.

sd_test2  tests extension of no-score zone surrounding non-lex token
          see the details in the comments in the ref.rttm file
          sd_test2 and sd_test3 look at opposite directions

sd_test3  tests extension of no-score zone surrounding non-lex token
          see the details in the comments in the ref.rttm file
          sd_test2 and sd_test3 look at opposite directions

sd_test4  tests the error that was fixed in md-eval-v15.pl
          In the output of this test, the amount of MISSED SPEECH and
          of MISSED SPEAKER TIME should be 0.21 seconds.  There should
          be one missed word.   The amount of FALARM SPEECH and of
          FALARM SPEAKER TIME should be 26.73 seconds.
          
sd_test5  <distribution suppressed>

sd_test6  shows behavior when a system speaker pathologically overlaps with itself
          (this relates to the changes introduced in md-eval-v17)
          Note: this is duplicated as md_test29