.TH sc_stats 1 "" "" "" ""
WORK IN PROGRESS 
.br
 RELEASED FOR Hub-5NE '97 Eval

NAME
sc_stats - SCLITE's Statistical System Comparison Program
.PP
.PP

NOTE: This manual page was created automatically from
HTMl pages in the sclite/doc directory.  This manual page does not
include output file examples.  The author suggests using a HTML browser
for reading the sclite documentation.
.PP
SYNOPSIS
sc_stats \*LOPTIONS\*O
.PP
 
DESCRIPTION .PP

sc_stats is a program which statistically compares the performance of
two or more Automatic Speech Recognition (ASR) systems which have been
run on identical test data.  Sc_stats is part of the \*LNIST SCTK\*O Scoring Tookit. The program reads alignments generated by
sclite, and produces \*L summary reports
\*O, \*Lgraphs and/or compares the
systems using any number tests of significant differences.
.PP
.PP
INPUT ALIGNMENTS 
.RS
.RE
OUTPUT REPORTS 
.RS
.RE
STATISTICAL TESTS 
.RS
.RE
REVISION HISTORY 
.RS
.RE
BUGS/COMMENTS 
.RS
Please contact Jon Fiscus at NIST with any bug reports or comments at
the email address 
\*Ljonathan.fiscus@nist.gov \*O or
by phone, (301)-975-3182.  Please include the version number of rover,
.RE
.RE
.\"  $Id: sc_stats.1,v 1.6 2004/08/30 15:10:38 jfiscus Exp $ 

\*LSc_stats\*O Commandline Options
.PP
The commandline options for sc_stats can be broken into four categories:
.LI
\*L Input File Options: \*O
.RS
\*L-p\*O,
.RE
.LI
\*L Output Options: \*O
.RS
\*L-e\*O,
\*L-n\*O,
\*L-O\*O,
.RE
.LI
\*L Report Generation Options: \*O
.RS
\*L-r\*O
.RE
.LI
\*L Statistical Test Options: \*O
.RS
\*L-t\*O
\*L-v\*O
\*L-u\*O
\*L-g\*O
.RE
.LE
.PP
Input File Options: 
.RS
These options control/define the input to sc_stats.  Input must
come from stdin and the -p option must be used.  (Forcing the user to
use the -p option enables future expandability while maintaining backward
compatability.)
.br
.br
-p	
.RS
Alignments are read from 'stdin' as  input  to  sc_stats.
The  format  of  the input must be in the "sgml" output
format, created either by '-o sgml' or by  piped  input
from another sctk utility.  
.RE
.RE
Output Options: 
.RS
-e desc
.RS
Description of the ensemble of hyp files.
.RE
-O output_dir 
.RS
Writes all output files into output_dir.  Defaults to the
hypfile's directory
.RE
-n name 
.RS
Writes all multiple hypothesis file reports to files beginning
with 'name'.  Using '-' writes to stdout. Default: 'Ensemble'
.RE
.RE
Report Generation Options: 
.RS
-g
.RS
Generate per speaker range graphs, based on the formula defined
by '-f'.  The reports are written to files whose root name
begins with the values defined by '-n'.  There are two graphs
produced, one showing speaker performance variability across
systems and the
second showing system performance variablity for across speakers.
.PP
- The 'range' graphs are an ASCII
representation of the
of the variablity in error rates for a given speaker.  The
graph is sorted be the mean of statistic computed for each speaker.
\*LEXAMPLE\*O
.PP
 - The 'grange' graph is a gnuplot version of the same data
ploted in 'range.  There are two sets of files created. 
The first set, which is called '*.grange.spk.plt' and
'*.grange.spk.dat', contains the gnuplot command files and
data files respectively for the speaker performance variability 
across systems graph.
The second set, which is called '*.grange.sys.plt' and
'*.grange.sys.dat', contains the gnuplot command files and
data files respectively for the system
performance variability across speakers graph.
\*LEXAMPLE\*O
.PP
 - The 'grange2' graph is similar to the 'grange'
graph except that each systems speaker word error scores are 
identified by a unique symbol.
\*LEXAMPLE\*O
.RE

.br

-r
.RS
.VL 4m

.LI " prn -
\*LExample\*O

.LI " sum -
\*LExample\*O

.LI " rsum -
\*LExample\*O

.LI " lur -
\*LExample\*O

.LI " es -
\*LExample\*O

.LI " res -
\*LExample\*O

.LI " none -
Produce no output reports, Default.
.LE
.RE
.RE
Statistical Test Options: 
.RS
-t [ mcn | mapsswe | sign | wilc | anovar | std4 ]
.RS
.VL 4m

.LI " mcn -
Perform the McNemar Test.

.LI " mapsswe -
Perform the Matched Pairs Sentence Segment Word Error Test

.LI " sign -
Perform the Sign Test

.LI " wilc -
Perform the Wilcoxon Signed Rank Test

.LI " anovar -
Perform the Analysis of Variance by Rank Test

.LI " std -
This is a shorthand notation to do the 'standard' four tests:
mcn, mapsswe, wilc and sign.
.LE
.RE

.br

-v 
.RS
For each test performed on a pair of systems files, output a
detailed analysis.
.RE

.br

-u 
.RS
Rather than creating a comparison matrix for each test, unify
statistical test results into a single comparision matrix
.RE

.br

-f [ E | R | W ]  
.RS
Use the identified formula for statistical tests: sign,
wilcoxon and anovar tests.  The formulas are:
.AL
.LI
 E -> Percentage Word Error
.LI
 R -> Percentage Words Correctly Recognized
.LI
 E -> Percentage Word Accuracy
.LE
By default 'E'
.RE
.RE