.TH sclite 1 "" "" "" ""

NAME
sclite - score speech recognition system output
.PP
.PP


NOTE: This manual page was created automatically from
HTMl pages in the sclite/doc directory.  This manual page does not
include output file examples.  The author suggests using a HTML browser
for reading the sclite documentation.
.PP
SYNOPSIS
sclite 
\*L-r\*O reffile [ fmt ] 
\*L-h\*O hypfile [ fmt [ title ] ] 
\*LOPTIONS\*O
.PP
DESCRIPTION
.PP
The program \*Lsclite\*O is a tool
for scoring and evaluating the output of speech recognition systems.
Sclite is part of the \*LNIST SCTK\*O Scoring Tookit.
The program compares the hypothesized text (HYP) output by the speech
recognizer to the correct, or reference (REF) text.  After comparing
REF to HYP, (a process called 
\*Lalignment\*O),
statistics are gathered during the \*Lscoring process\*O
and a variety of 
\*Lreports\*O can be produced to
summarize the performance of the recognition system.

THE ALIGNMENT PROCESS
.PP
The Alignment process consists of two steps: 1) selecting matching
REF and HYP texts, and 2) performing an alignment of the reference 
and hypothesis texts.   
.PP
Step 1: Selection of matching REF and HYP texts 
.RS
\*LSclite\*O
accepts as input a wide variety of file formats.  The type of input
formats define the algorithm for selecting matching REF and 
HYP texts.  Currently sclite uses four algorithms:
.RS
.PP
Utterance ID Matching: 
.RS
Input reference and hypothesis files in 
"\*Ltrn\*O" transcript format can be
aligned by either dynamic programming 
(\*LDP\*O) or 
\*LGNU's "diff"\*O.
.PP
 
When alignments are performed via DP, corresponding REF and HYP
records with the same utterance id's are located in the REF and HYP
files.  DP Alignment and scoring are then performed on each pair of
records.  Only the utterance ID's present in the HYP file are aligned and
scored.  This means the REF file may contain more utterance 
records than the HYP.
.PP
  
When "diff" is used for alignment, corresponding REF and HYP records
with the same utterance id's are located in the REF and HYP files.
Rather than execute "diff" for each pair of records, all matching REF
and HYP pairs are re-formatted to be newline separated words and
written to a temporary files.  Using the two temporary files, "diff"
is then called to perform a global alignment.  The output of "diff" is
re-chunking into REF/HYP records by applying the rule: include all
words in the output stream up to and including the last word in the
reference record.
.PP
  
The reference file can contain extra transcripts, only needed
transcripts are loaded.
.RE
.PP
Word Time Mark Matching: 
.RS
When both the REF and HYP files are in the 
"\*Lctm\*O" format,
The first step in the alignment process is to segment both the
reference and hypothesis word lists by locating common areas of
silence, (i.e.  the absence of a word time mark).  Once completed, the
resulting "segments" are aligned via dynamic programming and scored as
usual.
.PP
By default, the DP alignment is performed using word-to-word distances
measures of: 0, 3, 3, 4 for correct, insertions, deletions and
substitutions respectively.
.PP
 
Optionally, the command line flag 
'\*L-T\*O' forces the alignments to be
performed using \*Ltime-mediated\*O alignments.
.RE
.PP
Reference Segment Time Mark to Hypothesis Word Time Mark 
.RS
When the reference file format is "\*Lstm\*O"
and the hypothesis file format is
"\*Lctm\*O", sclite chops up the hypothesis file into regions matching
the reference segments.  Currently, there a two methods of chopping
the hypothesis file.  The method is dependent on the text alignment algorithm. 
.PP
When DP alignments are performed, the hypothesis file is segmented to
match the reference segments by selecting the string of hypothesized
words whose times occur before the end of each reference segment.  The
midpoint time of a word is used to determine if the word falls within
a segment.  DP alignments are then performed on the selected
hypothesis words and the reference segment.
.PP
 
If the alignments are performed via "diff", pre-process the input
reference and hypothesis texts, creating temporary reference and
hypothesis files with one word per line.  Then use GNU's "diff"
program to perform a global alignment on the word lists.  The output
of "diff" is re-chunked into segments for scoring.  Alternate
reference transcripts can not be used with "diff" alignments.
.RE
.PP
Reference Segment Time Mark to Hypothesis Text file 
.RS
When the reference file format 
"\*Lstm\*O" and the hypothesis file
format "\*Ltxt\*O" are used as
inputs, the same alignment and scoring algorithm is used as describe
above under the label "Reference Segment Time Mark to Hypothesis Word
Time Mark" by GNU diff alignments.
.RE
.RE
.RE
.PP
Step 2: Text Alignments 
.RS
\*LSclite\*O
can use either of two algorithms for finding alignments
between reference and hypothesis word strings.  The first, and most
widely accepted, uses dynamic programming (DP) and the second uses
GNU's "diff", a FSF (Free Software Foundation) program for comparing
text files.
.PP
Dynamic Programming string alignment:
.RS
.PP
The DP string alignment algorithm performs a global minimization of a
Levenshtein distance function which weights the cost of correct words,
insertions, deletions and substitutions as 0, 3, 3 and 4 respectively.
The computational complexity of DP is 0(NN).
.PP
When evaluating the output of speech recognition systems, the
precision of generated statistics is directly correlated to the
reference text accuracy.  But uttered words can be coarticulated or
mumbled to where they have ambiguous transcriptions, (e.i., "what are"
or "what're").  In order to more accurately represent ambiguous
transcriptions, and not penalize recognition systems, the ARPA
community agreed upon a format for specifying alternative reference
transcriptions.  The convention, when used on the case above, allows
the recognition system to output either transcripts, "what are" or
"what're", and still be correct.
.PP
The case above handles ambiguously spoken words which are loud enough
for the transcriber to think something should be recognized.  For
mumbled or quietly spoken words, the ARPA community agreed to neither
penalize systems which correctly recognized the word, nor penalize
systems which did not.  To accommodate this, a NULL word, "@", can be
added to an alternative reference transcript.  For example, "the" is
often spoken quickly with little acoustic evidence.  If "the" and "@"
are alternates, the recognition system will be given credit for
outputting "the" but not penalized if it does not.
.PP
The presence of alternate transcriptions represents added
computational complexity to the DP algorithm.  Rather than align all
alternate reference texts to the hypothesis text, then choose the
lowest error rate alignment, this implementation of DP aligns two word
networks, thus reducing the computational complexity from 2^(ref_alts +
hyp_alts) * O(N_ref * N_hyp) to O((N_ref+ref_alts) *
(N_hyp+hyp_alts)).
.PP
.RS
For a detailed explanation of DP alignment, see TIME WARPS, STRING
EDITS, AND MACROMOLECULES: THE THEORY AND PRACTICE OF SEQUENCE
COMPARISON, by Sankoff and Kruskal, ISBN 0-201-07809-0.
.RE
.PP
As noted above, DP alignment minimizes a distance function that is applied
to word pairs.  In addition to the "word" alignments which uses
a distance function defined by static weights, the sclite DP alignment module can
use two other distance functions.  The first, called \*L Time-Mediated\*O alignment
and the second called \*L Word-Weight-Mediated\*O alignment.
.PP
\*LTime-Mediated Alignment\*O
.RS
Time-Mediated alignment is a variation of DP alignment where
word-to-word distances are based on the time of occurence for
individual words.  Time-mediated alignments are performed when the '<A
href="options.htm#option_T_name_0">-T' option is exercised and the
input formats for both the reference and hypothesis files are in "<A
href="infmts.htm#ctm_fmt_name_0">ctm" format.
.PP
Time-mediated alignments are computed by replacing the
standard word-to-word distance weights of 0, 3, 3, and 4 with measures
based on beginning and ending word times.  The formulas for
time-mediated word-to-word distances are:
.PP
.RS
D(correct) = | T1(ref) - T1(hyp) | + | T2(ref) - T2(hyp) |
.br
D(insertion)  = T2(hyp) - T1(hyp)
.br
D(deletion)  = T2(ref) - T1(ref)
.br
D(substitution) = | T1(ref) - T1(hyp) | + | T2(ref) - T2(hyp) | + 0.001
.br
Distance for an Insertion or Deletion of the NULL Token '@' = 0.001
.PP
Where,
.RS
T1(x) is the beginning time mark of word x 
.br

T2(x) is the ending time mark of word x
.RE
.RE
.PP
.RE
\*LWord-Weight-Mediated Alignment\*O
.RS
Word-weight-mediated alignment is a variation of DP alignments where word-to-word distances 
are based on pre-defined word-weights.  Each word has a unique weight assigned
to it, via either a word-weight-list file, using the <A
href="options.htm#option_w_name_0">-w option, or through a
language model file, using the \*L
-L option.
The formulas for word-weight-mediated word-to-word distances are:
.PP
.RS
D(correct) = 0.0
.br
D(insertion)  = W(hyp)
.br
D(deletion)  = W(ref)
.br
D(substitution) = W(hyp) + W(ref)
.br
Distance for and Insertion or Deletion of the NULL Token '@' = 0.001
.PP
.RS
Where W(x) is the weight assigned to word 'x'.
.RE
.RE
.RE
.RE
.PP
String alignments via GNU's "diff":
.RS
.PP
While the DP algorithm has the advantage of flexibility, it is slow
for aligning large chunks of text.  To address the speed concerns, an
alternative string alignment module, which utilizes GNU's "diff", has
been added to sclite.  The sclite program pre-processes the input
reference and hypothesis texts, creating temporary reference and
hypothesis files with one word per line.  Then GNU's "diff" program is
used to perform a global alignment on the word lists and the output is
re-chunked into utterances or text segments for scoring.
.PP
Alignments can be performed with "diff" in about half the time taken
for DP alignments on the standard 300 Utterance ARPA CSRNAB test set.
However, in the opinion of the author, "diff" has the following bad
effects:
.RS
.PP
1. it can not accommodate transcription alternations,
.PP
2. "diff" does not produce the same alignments as the DP alignments,
.PP
3. there is an increase measured error rates.
.RE
.RE
.RE
THE SCORING PROCESS 
.RS
After reference and hypothesis texts have been aligned, scores are
tallied for each speaker and each ref/hyp pair.  After the tallies
are made, a variety if output reports are generated by using the 
'\*L-o\*O' option.  
Here is a set of \*Lexamples\*O.
.PP
The categories tallied are: 
Percent of correct words 
= 
# Correct words  * 100
Percent of substituted words
= 
# Substituted words  * 100
Percent of inserted words 
= 
# Inserted words  * 100
Percent of deleted words 
= 
# Deleted words  * 100
Percent of sentence errors 
= 
# incorrect ref and hyp pairs  * 100
.PP
 A variation in scoring called  \*L
Weighted-Word Scoring \*O  can also be implemented by sclite.
After \*L Word-Weight-Mediated
Alignment, the word weights can be tallied to produce
weighted-word scores.  The formulas for weighted-word scoring are very
simliar to word scoring described above.  The difference is rather than
assume each word has the same weight, 1 in the case of word scoring, 
each individual word has a different weight.  The word scoring formulas become:
Weighted Percent of correct words
= 
Sum of W(hyp) if correct  * 100
Weighted Percent of substituted words
= 
Sum of W(hyp) + W(ref) if substituted  * 100
Weighted Percent of inserted words 
= 
Sum of W(hyp) if inserted  * 100
Weighted Percent of deleted words 
= 
Sum of W(ref) if deleted  * 100
.RS
W(hyp) is the weight assigned to a hypothesis word, and W(ref) is the weight
assigned to a reference word.  Optionally deletable words have the default
weight of 0.0.
.RE
.RE

WORD CONFIDENCE MEASURE EVALUATION
.RS
Confidence scores for each hypothesized word were requested of the
LVCSR (Large Vocabulary Speech Recognition) participants beginning
with the April 1996 evaluation.  Each site was asked to do its analysis
of these scores which were not processed by NIST.  A review meeting
was held at NIST in August 1996 which resulted in a decision to apply
an agreed upon standard metric.
.PP
Confidence scores as they have been implemented are associated with
each hypothesized word.  (The issue has been raised whether for
languages such as Mandarin, where character error rate is considered
the primary measure of performance, the confidence ought to be
associated with characters.)  The confidence score pc,
associated with a word must be in the closed interval [0,1] and
presumably, given the entropy related metric defined below, in the
open interval (0,1).  It should represent the system's best estimate
of the a posterior probability that the hypothesized word is
correct. (Correct here necessarily is with respect to an alignment
procedure of the reference and hypothesis word strings.)
.PP
A single metric to use in the evaluartion of confidence scores was
adopted at the August meeting.  This is a normalized version of the
cross entropy or mutual information.  Specifically, the metric is
defined as:
.PP
.PP
Sclite will automatically detect the presence of confidence measures 
when reading in a hypothesis "\*Lctm\*O"
file.  When sclite detects the confidence scores, the report genererated
by the options "\*L-o sum\*O" has 
an additional column containing the Normalized Cross Entropy (NCE).
.PP
Output graphs concerning confidence estimates are generated by using the 
'\*L-C\*O'
option.  A variety of graphs can be created:
.AL
.LI
 DET Curve \*L Example \*O
.LI
 Binned Histogram\*L Example \*O
.LI
 Word Confidence Score Histogram
\*L Example \*O
.LE
.RE
REVISION HISTORY 
.RS
See \*Lrevision.txt\*O in
the main directory of the sclite source code directory
package.  
.RE
EXAMPLE USES OF \*LSCLITE\*O 
.RS
The \*Lsclite\*O scoring utility was
written to be used as a standard scoring tool for the ARPA speech
recognition benchmark tests.  Since evaluation paradigms have changed 
over the past several years, file formats and scoring proceedures have
changed as well.  This utility supports the following speech recognition
benchmark tests:
.RS
Utterance based evaluations:
.RS
Resource Management

.br

ATIS (Airline Travel Information Systems): 
.RE
Found speech evaluations:
.RS
Hub 4 - Marketplace and Broadcast News

.br

Hub 5 - LVCSR Switchboard 
.RE
.RE
.RE
BUGS/COMMENTS 
.RS
Please contact Jon Fiscus at NIST with any bug reports or comments at
the email address 
\*Ljonathan.fiscus@nist.gov \*O or
by phone, (301)-975-3182.  Please include the version number of rover,
.RE
.RE
.\"  $Id: sclite.1,v 1.7 2013/04/29 20:58:48 jfiscus Exp $ 

\*LSclite\*O Commandline Options
.PP
The commandline options for \*Lsclite\*O
can be broken into four categories:
.LI
\*L Input File Options: \*O
.RS
\*L-e\*O,
\*L-h\*O,
\*L-i\*O,
\*L-P\*O,
\*L-r\*O,
\*L-R\*O
.RE
.LI
\*L Alignment Options: \*O
.RS
\*L-c\*O,
\*L-d\*O,
\*L-F\*O,
\*L-L\*O
\*L-m\*O,
\*L-s\*O,
\*L-S\*O,
\*L-T\*O
\*L-w\*O
.RE
.LI
\*L Output Options: \*O
.RS
\*L-f\*O,
\*L-l\*O,
\*L-O\*O,
\*L-p\*O
.RE
.LI
\*L Scoring Report Options: \*O
.RS
\*L-C\*O,
\*L-n\*O,
\*L-o\*O
.RE
.LE
.PP
Input File Options: 
.RS
These options control/define the input to 
\*Lsclite\*O.  Input can come from either
reference and hypothesis files, or piped input from previously aligned REF and
HYP files.
.br
.br
-e  gb|euc	
.RS
Define the character encoding used for the text portion
input  ref  and hyp files.  The flag "gb" stands for GB
encoded  Chinese  and  "euc"  stands  for  EUC  encoded
Japanese.   Both  encodings  are  2-byte  per character
encodings.  The default, is extended ASCII.

.RE
 
.br

-h
hypfile [  
\*Ltrn\*O |
\*Ltxt\*O |
\*Lctm\*O ] title
.RS
The '-h' option is a required argument which  specifies
the   input   hypothesis  file.   The  optional format field,
"[  \*Ltrn\*O |
\*Ltxt\*O |
\*Lctm\*O] "
specifies the input file format from  the  set
of  input  formats  described above.  The default input
format is "\*Ltrn\*O".  When reports are generated, the "hypfile"  name will be used to identify the origins of the
results.  If the "title" option is  used,  that  string
will be used instead.
.PP
 The -h option may be used more than once to align multiple files.

.RE
 
.br

-i [ wsj | atis | rm | swb | spu_id ] 
.RS
The '-i' option defines how to interpret the  utterance
id's  used in the transcription input file format "\*Ltrn\*O"
described above. This argument identifies the corpus of
the utterance id:

.br


.br

.VL 4m

.LI "wsj -
for Wall Street Journal and CSRNAB
.LI "atis -
for ATIS3
.LI "rm | swb | spu_id -
are synonyms which refer to generic  utterance  id
formats  whereby  the utterance id is made up of a
speaker code, followed by a hyphen or  underscore,
followed by an utterance number.
.LE
.PP

This option is only required  for  aligning  transcript
inputs (\*Ltrn\*O).
TBD 

.RE
 
.br

-P
.RS
Alignments are read from 'stdin' as  input  to  sclite.
The  format  of  the input must be in the "sgml" output
format, created either by '-o sgml' or by  piped  input
from another sclite utility.  No re-alignments are performed on the read in alignments, only scoring  reports
can be generated.

.RE
 
.br

-r reffile [
\*Ltrn\*O |
\*Lstm\*O |
\*Lctm\*O ]
.RS
The '-r' option, a  required  argument,  specifies  the
input  reference  file which the hypothesis file(s) are
compared to.  The optional format field
"[  \*Ltrn\*O |
\*Lstm\*O |
\*Lctm\*O ] "
field specifies the
input  file  format  from  the  set  of  input  formats
described above.  The default input format is "\*Ltrn\*O".

.RE
 
.br

-R	
.RS
Interpret the text symbols as a right-to-left language such as
Arabic.  The default is to interpret text in a left-to-right fashion
as in English.

.RE
 
.br

.RE
Alignment Options: 
.RS
-c [ NOASCII DH ]
.RS
Chop up the words into separate characters before doing
the alignment.  It is generally not the practice of the
ARPA community to score at the  character  level.   The
intent  of  this option is to be able to score Mandarin
Chinese at the character level.  The  option  "NOASCII"
does  not  separate  characters if they are ASCII.  The
option "DH"  deletes  hyphens  from  the  ref  and  hyp
strings before alignment.  This option only works using
the DP alignment algorithm.  (-c & -d are exclusive)

.RE
 
.br

-d
.RS
Use \*LGNU diff\*O
for alignments  rather  than  the  default
dynamic programming.  (-c & -d are exclusive)

.RE
 
.br

-F
.RS
Perform the  alignment  using  a  cost  function  which
counts  fragments,  words  ending  or  beginning with a
hyphen, as correct if the spelling  up  to  the  hyphen
matches the spelling of the hypothesized word.
Options -F and -d are exclusive.

.RE
 
.br

-L LM
.RS
Define the \*L
CMU-Cambridge Statistical Language Modeling Toolkit v2 language
mode file to be 'LM'.  The LM file must be created using the
\*Lidngram2lm\*O program.
(See the toolkit documentation details of how
to make the language model.)  Currently, SCTK supports 1, 2 and 3-grams.
.PP
 The language model is used to compute an individual weight for each
word in the reference and hypothesis strings.  The weight is defined
to be \*WLog2(P(word|context))\*O.  Each pair of aligned
strings is considered to be independent, so therefore, there is
no context for initial words in each pair.
.PP
The word-weights are used in two ways, first as a method to define word-to-word distances
for \*L word-weight-mediated alignment \*O
and second to perform \*L 
weighted word scoring .
.PP
 Out-of-Vocabulary words get the default weight of 20.0, and optionally
deletable words get a default weight of 0.0.

.RE
 
.br

-m [ ref | hyp ]
.RS
When scoring a hypothesis ctm file against a  reference
stm  file,  the  time  spans  of the two may not match,
(i.e. the start time of the first word/segment may  not
match  or the end time of the last word/segment may not
match).
.PP

When this option is used, the alignment phase of  scoring
ignores  any  segment  or  word  (depending on the
option(s) used) which is not in the time  span  of  the
opposite  file.   The time span of a file is defined to
be start time of the first time mark, to the  end  time
of the last time mark.
.PP

The "ref" option  reduces  the  reference  segments  to
those which are within the hypothesis file time span.
.PP

The "hyp" option reduces the hypothesis words to  those
which are within the reference file tiem span.
.PP

Both "ref" and "hyp" may be used simultaneously.
.PP

The  argument  -m  by  itself  defaults  to  '-m  ref'.
Exclusive with -d.

.RE
 
.br
 
-s
.RS
Do Case-sensitive alignments.  Otherwise all input is mapped to 
a single case before scoring.  Of course, GB and UEX encode text
data is never case-converted.

.RE
 
.br

-S algo1 lexicon [ ASCIITOO ] 
.RS
The '-S' option performs an inferred word  segmentation
alignment algorithm.  This  option
is intended to be used for the LVCSR evaluation of Mandarin
Chinese.  A problem with scoring Mandarin at  the
word level is the lack of clearly defined words in Mandari
text.  This option implements an algorithm which,
given  a word segmentation for the reference string and
a "lexicon" of legal words, computes  a  minimal  error
rate word alignment.  The algorithm is as follows:

.br
 
.LI
  Convert  the  previously  word-segmented  reference
string into a word network.
.LI
 Covert the hypothesis text to a string  of  characters,
each  character  representing  a word.  The data
represented is then convert to a network.
\*C
.DS
ex.    * --- A --- * --- T --- * --- 0 --- *
.DE
\*O
.LI
 Consider all possible sequences of letters  through
the  network.   If  a  sequence creates a word which is
represented in the lexicon, add an arc to  the  network
representing the word.  The maximum characters per word
is limited to the maximum word length in the lexicon.
\*C
.DS
                     ,-------- TO -------.
                    /                     \
ex.    * --- A --- * --- T --- * --- 0 --- *
        \                     /
         `------- AT --------'
.DE
\*O
.LI
 DP Align the reference and hypothesis networks, and
extract a minimal cost path.
.LE
.PP

The supplied "lexicon" must be a sorted  list  of  word
records,  each  separated by a newline.  Only the first
column, separated by whitespace, is read  in  and  used
for  the  lexicon.   By  default,  the  algorithm  only
separates hypothesis characters  that  are  GB  or  EUC
encoded.   If  the  option  "ASCIITOO"  is  used, ASCII
hypothesis words are also converted  to  characters  in
step 2.
.PP
Exclusive with -d.	  

.RE

.br

-S algo2 lexicon [ ASCIITOO ] 
.RS
Perform a similar algorithm as described in '-S alog1' except
the roles of the reference and hypothesis transcripts are reversed.
In this algorithm, the segmentation of the hypothesis text is held
constant, while the reference transcript undergoes the process of 
of coversion to characters and arcs added to the network for words
found in the lexicon.  Both "lexicon" and "ASCIITOO" have the same
usage as in algo1.  
.PP
Exclusive with -d.	  

.RE
 
.br

-T
.RS
The '-T' option performs  time-mediated  string  alignments  rather  than  the  traditional  word alignments.
Currently, only alignments involving  two  "ctm"  files
can be aligned in this manner.  The \*L main SCLITE\*O
page describes time-mediated alignments.
.PP

Options -F and -d are exlcusive.

.RE
 
.br

-w wwl_file
.RS
Define the word-weight list (WWL) file to be 'wwl_file'.  The WWL file 
defines an arbitrary weight for each word in the lexicon.  The weights are
used in two ways, first as a method to define word-to-word distances
for \*L word-weight-mediated alignment \*O
and second to perform \*L 
weighted word scoring .
.PP
 If the supplied WWL filename is "unity", then no file of weights is read in.
Instead, this is  a shorthand notation to use a weight of 1.0 for all words.
.PP
 Optionally deletable words get a default weight of 0.0, (even if "unity"
is supplied as the WWL filename).
.PP
 The format of the WWL file is as follows. 
.br

.RS
Comment lines begin with
double semi-colons.  The are two forms of "special" comment lines.  The
first defines heading labels each column in the table.  The format for this
line is: 
.br
 

.RS
 ;; 'Headings' '<COL1>' '<COL2>' '<COL3>' .... 
.RE
 
.br

The label for column 1 should be "Word Spelling" since this column is the
word's text.  The labels for columns 2 though 10 are defined by the user.
.PP

The second "special" comment line defines the default weight applied to
out-of-vocabulary words if any exist.  The format for this line is:  
.br
 

.RS
 ;; Default missing weight '<number>' 
.RE
 
.br

'number' must be a floating point number. 
.PP
 
The remainder of the file consists of word records, each word record separated by
a newline.  The format of each record is: 
.br
 

.RS
 <WORD_TEXT> <WEIGHT_1> <WEIGHT_2> . . . 
.RE
 
.br

There should be no whitespace at the beginning if the line, and the word
texts can not include whitespace.  The remainder of the line are whitespace
separated floating point weights, up to a maximum of 10 weights can 
be assigned per word.
.PP

\*LNOTE: The current version of SCTK only utilizes the first weight.\*O

.RE
	

.RE
 
.br

.RE
Output Options: 
.RS
-f level
.RS
As a well behaved program, reassure the user  that  the
program is continuing to perform it's task by providing
the user with  some  feedback.   The  feedback  levels,
defined by this option are: 0) no feedback, 1) processing feedback (i.e.  status of text loading  and  alignments);   2)  processing  feedback  plus  printing  out
aligned strings.  The feedback level defaults to  0  if
no  output options are specified using the '-o' option,
otherwise it defaults to 1.
.RE
.br
-l width
.RS
When printing the text alignments for the output option
"pralign"   wrap   the  lines  at  "width"  characters.
Default is 1000 characters.
.RE
.br
-O output_dir
.RS
Instead of writing the output files  to  the  directory
containing the hyp file, write the files into the directory
"output_dir".  If the output directory does not  exist,
all reports will be written to stdout.
.RE
.br
-p
.RS
Write to standard out the resulting alignments so  they
can  be piped to another sclite utility.  The format of
the output is the same as '-o sgml'.  The options  sets
the feedback level, with '-f' to 0.
.RE
.RE
Scoring Report Options: 
.RS
-C [ det | bhist | hist | none ] 
.RS
Defines the output formats for analysis of confidence scores.
Currently, the only way to assign confidence estimates to 
each hyp word is through the \*Lctm\*O hypothesis file.
Default: 'none'  
\*L Examples. \*O
.RE
.br
-n name
.RS
Writes all outputs using 'name' as a root filename instead of
'hypfile'.  For multiple hypothesis files, the root filename
is 'name'.'hypfile'
.RE
.br
-o [ sum | rsum | wws | pralign | all | sgml | stdout | lur | snt | spk | dtl | prf | none ]
.RS
Defines the output scoring  reports  generated  by  the
sclite.  The possible reports are:

.br


.br

.VL 4m

.LI " sum -
Produce a summary of speaker performance in  terms
of  Percents:  Correct,  Substitutions, Deletions,
Insertions, Word Errors and  Sentence  (or  Utterance)  errors.  System averages and speaker means,
medians and standard deviations are  computed  for
each  percentage.   If  the report is not going to
stdout, the output is  placed  in  a  file  called
"<hypfile>.sys".   The  options  '-O'  and  '-n' can
change the destination of the output file.
\*LExample\*O

.LI " rsum -
Produce a summary similar to 'sum'  except  output
word counts instead of percentages.  If the report
is not going to stdout, the output is placed in  a
file  called  .raw.  The options '-O' and
'-n' can change  the  destination  of  the  output
file.
\*LExample\*O

.LI " wws -
Produce a summary similar to 'sum'  except  output
\*Lweighted word error\*O instead of word error.  If the report
is not going to stdout, the output is placed in  a
file  called  .wws.  The options '-O' and
'-n' can change  the  destination  of  the  output
file.
\*LExample\*O

.LI " pralign  - 
.br

pra  -
Produce a text copy of all the string  alignments.
If  the  report is not going to stdout, the output
is placed in a  file  called  .pra.   The
options  '-O'  and '-n' can change the destination
of the output file.  "pralign" and "pra" are synonyms.
\*LExample\*O

.LI " prf  -
Produce a text copy of all the string  alignments similar
to that produced by "pralign" except, include all relevant
information concerning the alignments.  That is, include
in the output things like: word beginning and ending times,
reference 
segment beginning and ending times, and hypothesis word 
confidence scores.
\*LExample\*O

.LI " all -
Produces  the  three   reports:   
"\*Lsum\*O",
"\*Lrsum\*O", and
\*Lpralign\*O"

.LI " stdout -
Write all selected scoring reports to stdout.   If
the feedback level is not specified using the '-f'
option, the feedback level is set to 0.

.LI " sgml -
Produce a dump of the text alignments in  an  sgml
notation.  The output consists of tags at the system, speaker, and sentence level.   Text  information  is  only  present  at the sentence level and
consists a comma separated  list  of  word  alignments.   The  word alignments can be either of the
following: C:"word" or  I:"word"  or  D:"word"  or
S:"word1","word2" for correct, insertion, deletion
and substitution respectively.  If the  report  is
not  going  to  stdout,  the output is placed in a
file called .sgml.  The options '-O'  and
'-n'  can  change  the  destination  of the output
file.
\*LExample\*O

.LI " lur -
Produce a Labeled  Utterance  Report  (LUR)  based
information in the reference STM file. (Note: only
reference  files  in  STM  format   support   this
option.)   The  LUR report is a report which tabulates overall error rate statistics and statistics
over arbitrary subsets of the reference data, e.g.
speaker's  sex,  audio  characteristics.   If  the
report  is  not  going  to  stdout,  the output is
placed  in  a  file  called  .lur.    The
options  '-O'  and '-n' can change the destination
of the output file.
\*LExample\*O

.LI " snt -
Produce   a   scoring   report   file   for    all
utterance/segments  of  a  speaker.   Within  each
file, one per speaker,  is  a  by-utterance  error
analysis  which  contains: the aligned text, error
classification percentages and  other  statistics.
If  the  report is not going to stdout, the output
is     placed      in      a      file      called
.   The options '-O' and
'-n' can change the destination and  name  of  the
output file.	 
\*LExample\*O

.LI " spk -
Produce a  scoring  report  file  summarizing  the
errors  made  on the speaker's utterances.  Within
each file, one per speaker id, is a  summarization
of  utterance and word errors along with confusion
pair,  insertion,   deletion,   substitution   and
falsely recognized word lists.
If the report is not going to stdout, the output
is      placed      in      a      file     called
.  The options '-O'  and
'-n'  can  change  the destination and name of the
output file.
\*LExample\*O

.LI " dtl -
Produce a scoring report in the same format as the
"spk"  report  using  statistics gathered over the
entire test set.  If the report is  not  going  to
stdout,  the  output  is  placed  in a file called
.dtl.  The  options  -'O'  and  '-n'  can
change  the  destination  and  name  of the output
file.
\*LExample\*O

.LI " none -
Produce no output reports.
.LE
.PP

If this option is not specified,  the  default  options
are  "sum"  and  "stdout".   If the user wishes to have
reports other than "sum" to be written to stdout,  then
the  "stdout"  flag  must be used in the argument list.
Options that are duplicated, have the effect of nullification.   So  for  instance  using  the  options  "all
pralign" is equivalent to "sum rsum".
Defines the output reports. Default: 'sum stdout'
.RE
.br
.RS
.\"  $Id: sclite.1,v 1.7 2013/04/29 20:58:48 jfiscus Exp $ 

\*LSclite\*O Input file formats: 
\*Ltrn\*O,
\*Ltxt\*O,
\*Lstm\*O,
\*Lctm\*O
The inputs to "\*Lsclite\*O" are the
reference file and a hypothesis file(s), the text portions of which
may be either ASCII characters or GB encoded Chinese characters.
There are a number of different input formats permitted: 
"\*Ltrn\*O",
"\*Ltxt\*O",
"\*Lstm\*O", and
"\*Lctm\*O".
As new scoring paradigms were created for the ARPA
tests, accompanying formats were created to support the evaluations.
.PP
trn - Definition of a transcript input file 
.RS
.PP
The transcript  format  is  a  file  of  word  sequence
records  separated by newlines.  Each record contains a
word sequence, follow by the an utterance  ID  enclosed
in  parenthesis.   See  the  '\*L-i\*O'  option for a list of
accepted utterance id types.
.PP
example.
.RS
she had your dark suit in greasy  wash  water  all
year (cmh_sa01)
.RE
.PP
Transcript alternations, described above, can  be  used
in the word sequence by using this BNF format:
.PP
.RS
ALTERNATE :== "{" TEXT ALT+ "}"

.br
 
ALT       :== "/" TEXT
.br
TEXT      :== 1 or more whitespace separated words |
"@" | ALTERNATE
.RE
.PP
The "@" represents a  NULL  word  in  the  transcript.   For
scoring  purposes,  an  error  is  not counted if the "@" is
aligned as an insertion.
.PP
example
.RS
i've { um / uh / @ } as far as i'm concerned
.RE
.RE
.PP
txt - Definition of a text input file 
.RS
This format is simply  free-form  text  with  no  page,
paragraphs, sentence or speaker breaks.
.RE
stm - Definition of segment time mark input file 
.RS
.PP
This describes the segment time marked files to be used  for
scoring  the  output  of  speech  recognizers  via  the NIST
sclite() program.  This is a reference file format.
.PP
The segment time mark file consists of  a  concatenation  of
text  segment  records from a waveform file.  Each record is
separated by a newline and contains: the waveform's filename
and  channel  identifier  [A | B], the talkers id, begin and
end times (in seconds), optional subset label and  the  text
for the segment.  Each record follows this BNF format:
.PP
STM :== <F> <C> <S> <BT> <ET> [ <LABEL> ] transcript . . .
.RS
Where :
.RS
<F>
.RS
The waveform  filename.   NOTE:  no  pathnames  or
extensions are expected.
.RE
<C>
.RS
The waveform channel.   The text of the waveform channel
is not restricted by sclite.  The text can be any text string without
witespace so long as the matching string is found in both the reference
and hypothesis input files.
.RE
<S>
.RS
The speaker id,  no  restrictions  apply  to  this
name.
.RE
<BT>
.RS
The begin time (seconds) of the segment.
.RE
<ET>
.RS
The end time (seconds) of the segment.
.RE
<LABEL>
.RS
A  comma  separated  list  of  subset  identifiers
enclosed  in angle brackets.  Ex. "<O,F,00>".  See
"USING STM FORMAT FOR  LABELED  UTTERANCE  REPORTS
(LUR)" below.
.RE
transcript
.RS
The transcript can take on two forms: 1) a whitespace separated list of
words, or 2) the string "IGNORE_TIME_SEGMENT_IN_SCORING".
.PP
The list of
words can contain an transcript alternation using the following
BNF format:
.PP
.RS
ALTERNATE :== "{" <text> ALT+ "}"

.br

ALT       :== "/" <text>

.br

TEXT      :== 1 thru n words | "@" | ALTERNATE
.RE
.PP
The "@" represents a NULL word in the transcript.  For scoring
purposes, an error is not counted if the "@" is aligned as an
insertion.  
.PP
.RS
Example:     "i've { um / uh / @ } as far  as i'm concerned"
.RE
.PP
When the string "IGNORE_TIME_SEGMENT_IN_SCORING" is used as the transcript,
the process which chops the hypothesis file to matching reference segments
ignores all hypothesis words whose time-midpoints occur within the reference
segments beginning
and ending time.  The effect is to declare this segments regions as 
"out-of-bounds" for scoring, thus generation no errors from that time 
region. 
.PP
.RS
NOTE: this only works with DP alignment of a referenc stm file
and hypothesis ctm file.
.RE
.PP
.RE
.RE
Example STM file:
.RS
;; comment

.br

2345 A 2345-a 0.10 2.03 uh huh yes i thought

.br

2345 A 2345-b 2.10 3.04 dog walking is a very

.br

2345 A 2345-a 3.50 4.59 yes but it's worth it
.RE
.PP
The file must be sorted by the first and second  columns  in
ASCII order, and the fourth in numeric order.  The UNIX sort
command:  "sort +0 -1 +1 -2 +3nb -4"  will  sort  the  words
into appropriate order.
.PP
Lines beginning with ';;' are considered  comments  and  are
ignored.  Blank lines are also ignored.
.PP
.RE
USING STM FORMAT FOR LABELED UTTERANCE REPORTS (LUR):
.RS
Motivation:
.RS
For the Fall '95 ARPA CSR Evaluation, it was  desirable
to  not  only  report overall error-rate statistics but
also error-rate  statistics  for  arbitrary  partitions
and/or  groups  of  partitions within the test set.  To
this end, the STM file format was  extended  to  encode
arbitrary subset information for each segment.
.RE
Usage:
.RS
The subset information is encoded by adding  two  types
of  information  into the STM file.  The first information
type, is a special comment line, the subset information line, (SIL).
The SIL defines the subset's label
id, a short column heading and a description.  The special
comment line format is:
.RS
;; LABEL "<ID>" "<COL_HD>" "<DESC>"
.RS
where:
.RS
<ID>
.RS
The subset id.  Used to  label  each  segment
that  belongs  to  the subset.  The format is
arbitrary, but without spaces.
.RE
<COL_HD> 
.RS
Used as column headings in generated reports.
Format is arbitrary.
.RE
<DESC> 
.RS
Used for  subset  descriptions  in  generated
reports.  May be of arbitrary length and for-
mat.  Double  backslashes  '\\'  add  a  line
feed.
.RE
.RE
.RE
The order of the SIL lines in the STM file defines  the
order of subset presentation the generated reports.
The second type of information  incorporated  into  the
STM file is an optional sixth field to the text segment
record.  The field consists of a comma  separated  list
of  subset ids enclosed in angle brackets.  Each unique
id must have a special comment line,  specified  above,
to  be  properly interpreted.  Otherwise the id will be
ignored.
.PP
Each position within the label field,  separated  by  a
commas,  defines  a group of subsets that are presented
separately in the generated reports.  So for  instance,
the  first  group might be all segments, and the second
might be either male or female, and the third might  be
the story.  The example below shows an STM file encoded
with this information.
.RS
;; LABEL "M" "Male" "Male Talkers"
.br
;; LABEL "F" "Female" "Female Talkers"
.br
;; LABEL "01" "Story 1" "Business news"
.br
;; LABEL "00" "Not in Story" "Words or Phrases not
contained in a story"
.br
940328 1 A 4.00 18.10 <O,F,00> FROM LOS ANGELES
.br
940328 1 B 18.10 25.55 <O,M,01> MEXICO IN TURMOIL
.RE
.RE
.RE
.RE
.RE
.PP
ctm - Definition of time marked conversation scoring input 
.RS
.PP
This describes the time marked conversation input  files  to
be used for scoring the output of speech recognizers via the
NIST sclite() program.  Both the  reference  and  hypothesis
input files can share this format.
.PP
The ctm file format is a concatenation of time mark  records
for  each  word  in each channel of a waveform.  The records
are separated with a newline.  Each word token must  have  a
waveform  id,  channel identifier [A | B], start time, dura-
tion, and word text.  Optionally a confidence score  can  be
appended  for  each word.  Each record follows this BNF for-
mat:
.PP
CTM :== <F> <C> <BT> <DUR> word [ <CONF> ]
.RS
Where :
.RS
<F>  ->
.RS
The waveform  filename.   NOTE:  no  pathnames  or
extensions are expected.
.RE
<C>  ->
.RS
The waveform channel.  Either "A" or "B".  The text of the waveform channel
is not restricted by sclite.  The text can be any text string without
witespace so long as the matching string is found in both the reference
and hypothesis input files.
.RE
<BT> ->
.RS
The begin time (seconds)  of  the  word,  measured
from the start time of the file.
.RE
<DUR>  ->
.RS
The duration (seconds) of the word.
.RE
<CONF>  ->
.RS
Optional confidence score.  It  is  proposed  that
this score will be used in the future.
.RE
.RE
.RE
.PP
The file must be sorted by  the  first  three  columns:  the
first  and  the  second  in  ASCII order, and the third by a
numeric order.  The UNIX sort command: "sort  +0  -1  +1  -2
+2nb -3" will sort the words into appropriate order.
.PP
Lines beginning with ';;' are considered  comments  and  are
ignored.  Blank lines are also ignored.
.PP
Included below is an example:
.RS
;;

.br

;;  Comments follow ';;' 

.br

;;

.br

;;  The Blank lines are ignored

.br


.br

;;

.br

7654 A 11.34 0.2  YES -6.763

.br

7654 A 12.00 0.34 YOU -12.384530

.br

7654 A 13.30 0.5  CAN 2.806418

.br

7654 A 17.50 0.2  AS 0.537922

.br

:

.br

7654 B 1.34 0.2  I -6.763

.br

7654 B 2.00 0.34 CAN -12.384530

.br

7654 B 3.40 0.5  ADD 2.806418

.br

7654 B 7.00 0.2  AS 0.537922

.br

:
.RE
.PP
For CTM reference files, a format extension exists to permit
marking  alternate  transcripts.   The  alternation uses the
same file format  as  described  above,  except  three  word
strings, "<ALT_BEGIN>", "<ALT>" and "<ALT_END>", are used to
delimit the alternation.  Each tag is  treated  as  a  word,
with  a conversation id, channel and "*"'s for the begin and
duration time.
.PP
The alternation is begun using the word  "<ALT_BEGIN>",  and
terminated using the word "<ALT_END>".  In between the start
and  end,  are  at  least  2  alternative  time-marked  word
sequences separated by the word "<ALT>".  Each word sequence
can contain any number of words.  An empty alternative  sig-
nifies a null word.
.PP
Below is and example alternate reference transcript for  the
words "uh" and "um".
.PP
.RS
;;

.br

7654 A   *    *   <ALT_BEGIN>

.br

7654 A 12.00 0.34 UM

.br

7654 A   *    *   <ALT>

.br

7654 A 12.00 0.34 UH

.br

7654 A   *    *   <ALT_END>
.RS
.RE
.RE