// doc/data_prep.dox
  
  // Copyright 2012  Johns Hopkins University (author: Daniel Povey)
  
  // See ../../COPYING for clarification regarding multiple authors
  //
  // Licensed under the Apache License, Version 2.0 (the "License");
  // you may not use this file except in compliance with the License.
  // You may obtain a copy of the License at
  
  //  http://www.apache.org/licenses/LICENSE-2.0
  
  // THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  // KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
  // WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
  // MERCHANTABLITY OR NON-INFRINGEMENT.
  // See the Apache 2 License for the specific language governing permissions and
  // limitations under the License.
  
  /**
   \page data_prep  Data preparation
  
    \section data_prep_intro Introduction
  
    After running the example scripts (see \ref tutorial), you may want to set up
    Kaldi to run with your own data.  This section explains how to prepare the data.
    This page will assume that you are using the latest version of the example scripts
    (typically named "s5" in the example directories, e.g. egs/rm/s5/).
    In addition to this page, you can refer to the data preparation scripts in those
   directories.  The top-level run.sh scripts (e.g. egs/rm/s5/run.sh) have a few commands at
   the top of them that relate to various phases of data preparation.  The parts in
   the sub-directory named local/ are always specific to the database.  For example,
   in the Resource Management (RM) setup it is local/rm_data_prep.sh.  In the case of
   RM these commands are:
  \verbatim
  local/rm_data_prep.sh /export/corpora5/LDC/LDC93S3A/rm_comp || exit 1;
  
  utils/prepare_lang.sh data/local/dict '!SIL' data/local/lang data/lang || exit 1;
  
  local/rm_prepare_grammar.sh || exit 1;
  \endverbatim
  
  In the WSJ case the commands are:
  \verbatim
  
  wsj0=/export/corpora5/LDC/LDC93S6B
  wsj1=/export/corpora5/LDC/LDC94S13B
  
  local/wsj_data_prep.sh $wsj0/??-{?,??}.? $wsj1/??-{?,??}.?  || exit 1;
  
  local/wsj_prepare_dict.sh || exit 1;
  
  utils/prepare_lang.sh data/local/dict "<SPOKEN_NOISE>" data/local/lang_tmp data/lang || exit 1;
  
  local/wsj_format_data.sh || exit 1;
  \endverbatim
  There are more commands after these in the WSJ script that relate to training language
  models locally (rather than using the ones supplied by LDC), but the ones above are the most
  important ones.
  
  
  The output of the data preparation stage consists of two sets of things.  One relates
  to "the data" (directories like data/train/) and one relates to "the language"
  (directories like data/lang/).  The "data" part relates to the specific recordings you
  have, and the "lang" part contains things that relate more to the language itself,
  such as the lexicon, the phone set, and various extra information about the phone set
  that Kaldi needs.  If you want to prepare data which you will decode with an
  already existing system and an already existing language model, the "data" part is
  all you need to touch.
  
  \section data_prep_data Data preparation-- the "data" part.
  
  As an example of the "data" part of the data preparation, look at the directory
  "data/train" in one of the example directories (assuming you have already run
  the scripts there).  Note: there is nothing special about the directory name
  "data/train".  There are other directories such as "data/eval2000" (for a test set)
  that have essentially the same format ("essentially" because we may have an "stm" and
  "glm" file in the test directory, to enable sclite scoring).
  The specific example we'll look at the Switchboard recipe
  in egs/swbd/s5.
  \verbatim
  s5# ls data/train
  cmvn.scp  feats.scp  reco2file_and_channel  segments  spk2utt  text  utt2spk  wav.scp
  \endverbatim
  Not all of the files are equally important.  For a simple setup where there is no
  "segmentation" information (i.e. each utterance corresponds to a single file), the only
  files you have to create yourself are "utt2spk", "text" and "wav.scp" and possibly
  "segments" and "reco2file_and_channel", and the rest will be created by standard scripts.
  
  We will describe the files in this directory, starting with the files you need to create
  yourself.
  
  \subsection data_prep_data_yourself Files you need to create yourself
  
   The file "text" contains the transcriptions of each utterance.
  \verbatim
  s5# head -3 data/train/text
  sw02001-A_000098-001156 HI UM YEAH I'D LIKE TO TALK ABOUT HOW YOU DRESS FOR WORK AND
  sw02001-A_001980-002131 UM-HUM
  sw02001-A_002736-002893 AND IS
  \endverbatim
  The first element on each line is the utterance-id, which is an arbitrary text string,
  but if you have speaker information in your setup, you should make the speaker-id a
  prefix of the utterance id; this is important for reasons relating to the sorting of
  these files.  The rest of the line is the transcription of each sentence.  You don't
  have to make sure that all words in this file are in your vocabulary; out of vocabulary words will
  get mapped to a word specified in the file data/lang/oov.txt.
  
  It needs to be the case that when you sort both the utt2spk and spk2utt files,
  the orders "agree", e.g. the list of speaker-ids extracted from the utt2spk file
  is the same as the string sorted order.  The easiest way to make this happen is
  to make the speaker-ids a prefix of the utter Although, in this particular
  example we have used an underscore to separate the "speaker" and "utterance"
  parts of the utterance-id, in general it is probably safer to use a dash ("-").
  This is because it has a lower ASCII value; if the speaker-ids vary in length,
  in certain cases the speaker-ids and their corresponding utterance ids can end
  up being sorted in different orders when using the standard "C"-style ordering
  on strings, which will lead to a crash.
  \endverbatim
  Another important file is <DFN>wav.scp</DFN>.  In the Switchboard example,
  \verbatim
  s5# head -3 data/train/wav.scp
  sw02001-A /home/dpovey/kaldi-trunk/tools/sph2pipe_v2.5/sph2pipe -f wav -p -c 1 /export/corpora3/LDC/LDC97S62/swb1/sw02001.sph |
  sw02001-B /home/dpovey/kaldi-trunk/tools/sph2pipe_v2.5/sph2pipe -f wav -p -c 2 /export/corpora3/LDC/LDC97S62/swb1/sw02001.sph |
  \endverbatim
  The format of this file is
  \verbatim
  <recording-id> <extended-filename>
  \endverbatim
  where the "extended-filename" may be
  an actual filename, or as in this case, a command that extracts a wav-format file.  The pipe symbol
  on the end of the extended-filename specifies that it is to be interpreted as a pipe.  We will
  explain what "recording-id" is below, but we would first like to point out that if the "segments" file
  does not exist, the first token on each line of "wav.scp" file is just the utterance id.
  The files in wav.scp must be single-channel (mono); if the underlying wav files have multiple
  channels, then a sox command must be used in the wav.scp to extract a particular channel.
  
  In the Switchboard setup we have the "segments" file, so we'll discuss this next.
  \verbatim
  s5# head -3 data/train/segments
  sw02001-A_000098-001156 sw02001-A 0.98 11.56
  sw02001-A_001980-002131 sw02001-A 19.8 21.31
  sw02001-A_002736-002893 sw02001-A 27.36 28.93
  \endverbatim
  The format of the "segments" file is:
  \verbatim
  <utterance-id> <recording-id> <segment-begin> <segment-end>
  \endverbatim
  where the segment-begin and segment-end are measured in seconds.
  These specify time offsets into a recording.  The "recording-id"
  is the same identifier as is used in the "wav.scp" file-- again, this is
  an arbitrary identifier that you can choose.
  The file "reco2file_and_channel" is only used when scoring (measuring
  error rates) with NIST's "sclite" tool:
  \verbatim
  s5# head -3 data/train/reco2file_and_channel
  sw02001-A sw02001 A
  sw02001-B sw02001 B
  sw02005-A sw02005 A
  \endverbatim
  The format is:
  \verbatim
  <recording-id> <filename> <recording-side (A or B)>
  \endverbatim
  The filename is typically the name of the .sph file, without the suffix, but in
  general it's whatever identifier you have in your "stm" file.
  The recording side is a concept that relates to telephone conversations where there are
  two channels, and if not, it's probably safe to use "A". If you don't have
  an "stm" file or you have no idea what this is all about, then you don't need
  the "reco2file_and_channel" file.
  
  The last file you need to create yourself is the "utt2spk" file.  This says, for each
  utterance, which speaker spoke it.
  \verbatim
  s5# head -3 data/train/utt2spk
  sw02001-A_000098-001156 2001-A
  sw02001-A_001980-002131 2001-A
  sw02001-A_002736-002893 2001-A
  \endverbatim
  The format is
  \verbatim
  <utterance-id> <speaker-id>
  \endverbatim
  Note that the speaker-ids don't need to correspond in any very accurate sense
  to the names of actual speakers-- they simply need to represent a reasonable guess.
  In this case we assume each conversation side (each side of the telephone conversation)
  corresponds to a single speaker.  This is not entirely true -- sometimes one person
  may hand the phone to another person, or the same person may be speaking in multiple
  calls -- but it's good enough for our purposes.  <b> If you have no information at all about
  the speaker identities, you can just make the speaker-ids the same as the utterance-ids </b>,
  so the format of the file would be just <DFN>\<utterance-id\> \<utterance-id\></DFN>.
  We have made the previous sentence bold because we have encountered people creating
  a "global" speaker-id.  This is a bad idea because it makes cepstral mean normalization
  ineffective in training (since it's applied globally), and because it will create problems
  when you use utils/split_data_dir.sh to split your data into pieces.
  
  There is another file that exists in some setups; it is used only occasionally and
  not in the Kaldi system build.  We show what it looks like in the Resource Management
  (RM) setup:
  \verbatim
  s5# head -3 ../../rm/s5/data/train/spk2gender
  adg0 f
  ahh0 m
  ajp0 m
  \endverbatim
  This file maps from speaker-id to either "m" or "f" depending on the speaker gender.
  
  All of these files should be sorted.  If they are not sorted, you will get errors
  when you run the scripts.  In \ref io_sec_tables we explain why this is needed.
  It has to do with the I/O framework; the ultimate reason for the sorting is to
  enable something equivalent to random-access lookup on a stream that doesn't support
  fseek(), such as a piped command.  Many Kaldi programs are reading multiple pipes
  from other Kaldi commands, reading different types of object, and are doing something
  roughly comparable to merge-sort
  on the different inputs; merge-sort, of course, requires that the inputs be sorted.
  Be careful when you sort that you have the shell variable LC_ALL defined as "C",
  for example (in bash),
  \verbatim
  export LC_ALL=C
  \endverbatim
  If you don't do this, the files will be sorted in an order that's different from how
  C++ sorts strings, and Kaldi will crash.  You have been warned!
  
  If your data consists of a test set from NIST that has an "stm" and a "glm" file
  provided so that you can measure WER, then you can put these files in the data
  directory with the names "stm" and "glm".  Note that we put the scoring
  script (which measures WER) in <DFN>local/score.sh</DFN>, which means it is
  specific to the setup; not all of the scoring scripts in all of the setups will
  recognize the stm and glm file.  An example of a scoring script that uses those files is
  the one the Switchboard setup, i.e. <DFN>egs/swbd/s5/local/score_sclite.sh</DFN>,
  which is invoked by the top-level scoring script
  <DFN>egs/swbd/s5/local/score.sh</DFN> if it notices that your test set has the
  stm and glm files.
  
  \subsection data_prep_data_noneed Files you don't need to create yourself
  
  The other files in this directory can be generated from the files you provide.
  You can create the "spk2utt" file by a command like the following
  (this one is extracted from egs/rm/s5/local/rm_data_prep.sh)
  \verbatim
  utils/utt2spk_to_spk2utt.pl data/train/utt2spk > data/train/spk2utt
  \endverbatim
  This is possible because the utt2spk and spk2utt files contain exactly
  the same information; the format of the spk2utt file is
  <DFN>\<speaker-id\> \<utterance-id1\> \<utterance-id2\> ...</DFN>.
  
  Next we come to the <DFN>feats.scp</DFN> file.
  \verbatim
  s5# head -3 data/train/feats.scp
  sw02001-A_000098-001156 /home/dpovey/kaldi-trunk/egs/swbd/s5/mfcc/raw_mfcc_train.1.ark:24
  sw02001-A_001980-002131 /home/dpovey/kaldi-trunk/egs/swbd/s5/mfcc/raw_mfcc_train.1.ark:54975
  sw02001-A_002736-002893 /home/dpovey/kaldi-trunk/egs/swbd/s5/mfcc/raw_mfcc_train.1.ark:62762
  \endverbatim
  This points to the extracted features-- MFCC features in this case, because
  that is what we use in this particular script.  The format is:
  \verbatim
  <utterance-id> <extended-filename-of-features>
  \endverbatim
  Each of the feature files contains a matrix, in Kaldi format.
  In this case the dimension of the matrix would be (the length of the file in 10ms intervals) by 13.
  The "extended filename" <DFN>/home/dpovey/kaldi-trunk/egs/swbd/s5/mfcc/raw_mfcc_train.1.ark:24</DFN>
  means, open the "archive" file /home/dpovey/kaldi-trunk/egs/swbd/s5/mfcc/raw_mfcc_train.1.ark, fseek()
  to position 24, and read the data that's there.
  
  This feats.scp file is created by the command
  \verbatim
  steps/make_mfcc.sh --nj 20 --cmd "$train_cmd" data/train exp/make_mfcc/train $mfccdir
  \endverbatim
  which is invoked by the top-level "run.sh" script.  For the definitions of the
  shell variables, see that script.  <DFN>\$mfccdir</DFN> is a user-specified directory where the
  .ark files will be written.
  
  The last file in the directory data/train is "cmvn.scp".  This contains statistics
  for cepstral mean and variance normalization, indexed by speaker.  Each set of
  statistics is a matrix, of dimension 2 by 14 in this case.  In our example, we have:
  \verbatim
  s5# head -3 data/train/cmvn.scp
  2001-A /home/dpovey/kaldi-trunk/egs/swbd/s5/mfcc/cmvn_train.ark:7
  2001-B /home/dpovey/kaldi-trunk/egs/swbd/s5/mfcc/cmvn_train.ark:253
  2005-A /home/dpovey/kaldi-trunk/egs/swbd/s5/mfcc/cmvn_train.ark:499
  \endverbatim
  Unlike feats.scp, this scp file is indexed by speaker-id, not utterance-id.
  This file is created by a command such as this:
  \verbatim
  steps/compute_cmvn_stats.sh data/train exp/make_mfcc/train $mfccdir
  \endverbatim
  (this example is from <DFN>egs/swbd/s5/run.sh</DFN>).
  
  Because errors in data preparation can cause problems later on, we have a script to
  check that the data directory is correctly formatted.  Run e.g.
  \verbatim
  utils/validate_data_dir.sh data/train
  \endverbatim
  You may also find the following command useful:
  \verbatim
  utils/fix_data_dir.sh data/train
  \endverbatim
  (of course the command will work for any data directory, not just data/train).  This
  script will fix sorting errors and will remove any utterances for which some required
  data, such as feature data or transcripts, is missing.
  
  \section data_prep_lang Data preparation-- the "lang" directory.
  
  Now we turn our attention to the "lang" directory.
  \verbatim
  s5# ls data/lang
  L.fst  L_disambig.fst  oov.int	oov.txt  phones  phones.txt  topo  words.txt
  \endverbatim
  There may other directories with a very similar format: in the case we have
  a directory "data/lang_test" that contains the same information but also a file
  G.fst that is a Finite State Transducer form of the language model:
  \verbatim
  s5# ls data/lang_test
  G.fst  L.fst  L_disambig.fst  oov.int  oov.txt	phones	phones.txt  topo  words.txt
  \endverbatim
  Note that lang_test/ was created by copying lang/ and adding G.fst.
  Each of these directories seems to contain only a few files.
  It's not quite as simple as this though, because "phones" is a directory:
  \verbatim
  s5# ls data/lang/phones
  context_indep.csl  disambig.txt         nonsilence.txt        roots.txt    silence.txt
  context_indep.int  extra_questions.int  optional_silence.csl  sets.int     word_boundary.int
  context_indep.txt  extra_questions.txt  optional_silence.int  sets.txt     word_boundary.txt
  disambig.csl       nonsilence.csl       optional_silence.txt  silence.csl
  \endverbatim
  The phones directory contains various bits of information about the phone set; there
  are three versions of some of the files, with extensions .csl, .int and .txt, that contain
  the same information in three formats.  Fortunately you, as a Kaldi user, don't have
  to create all of these files because we have a script "utils/prepare_lang.sh" that
  creates it all for you based on simpler inputs.  Before we describe that script
  and the simpler inputs it takes, we feel obligated to explain what is in the "lang" directory.
  After that we will explain the easy way to create it.  The user who is simply
  aiming to quickly build a system without needing to understand how Kaldi works
  may skip to \ref data_prep_lang_creating below.
  
  \section data_prep_lang_contents Contents of the "lang" directory
  
  First there are the files <DFN>phones.txt</DFN> and <DFN>words.txt</DFN>.  These
  are both symbol-table files, in the OpenFst format, where each line is
  the text form and then the integer form:
  \verbatim
  s5# head -3 data/lang/phones.txt
  <eps> 0
  SIL 1
  SIL_B 2
  s5# head -3 data/lang/words.txt
  <eps> 0
  !SIL 1
  -'S 2
  \endverbatim
  These files are used by Kaldi to map back and forth between the integer and
  text forms of these symbols.  They are mostly only accessed by the scripts
  utils/int2sym.pl and utils/sym2int.pl, and by the OpenFst programs fstcompile and
  fstprint.
  
  The file <DFN>L.fst</DFN> is the Finite State Transducer form of the lexicon (L,
  see  <a href=http://www.cs.nyu.edu/~mohri/pub/hbka.pdf> "Speech Recognition
  with Weighted Finite-State Transducers" </a> by Mohri, Pereira and
  Riley, in Springer Handbook on SpeechProcessing and Speech Communication, 2008).
  with phone symbols on the input and word symbols on the output.  The file
  <DFN>L_disambig.fst</DFN> is the lexicon, as above but including the disambiguation
  symbols \#1, \#2, and so on, as well as the self-loop with \#0 on it to "pass through"
  the disambiguation symbol from the grammar.  See \ref graph_disambig for more
  explanation.  Anyway, you won't have to deal with this directly.
  
  The file <DFN>data/lang/oov.txt</DFN> contains just a single line:
  \verbatim
  s5# cat data/lang/oov.txt
  <UNK>
  \endverbatim
  This is the word that we will map all out-of-vocabulary words to during
  training.  There is nothing special about "<UNK>" here, and it does not have
  to be this particular word; what is important is that this word should have a pronunciation
  containing just a phone that we designate as a "garbage phone"; this phone will
  align with various kinds of spoken noise.  In our particular setup, this phone
  is called <DFN>\<SPN\></DFN> (short for "spoken noise"):
  \verbatim
  s5# grep -w UNK data/local/dict/lexicon.txt
  <UNK> SPN
  \endverbatim
  The file <DFN>oov.int</DFN> contains the integer form of this (extracted from <DFN>words.txt</DFN>),
  which happens to be 221 in this setup.  You might notice that in the Resource Management
  setup, oov.txt contains the silence word, which in that setup happens to be called "!SIL".
  In that case we simply chose an arbitrary word from the vocabulary-- there are no out of vocabulary
  words in the training set, so the word we choose has no effect.
  
  The file data/lang/topo contains the following data:
  \verbatim
  s5# cat data/lang/topo
  <Topology>
  <TopologyEntry>
  <ForPhones>
  21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188
  </ForPhones>
  <State> 0 <PdfClass> 0 <Transition> 0 0.75 <Transition> 1 0.25 </State>
  <State> 1 <PdfClass> 1 <Transition> 1 0.75 <Transition> 2 0.25 </State>
  <State> 2 <PdfClass> 2 <Transition> 2 0.75 <Transition> 3 0.25 </State>
  <State> 3 </State>
  </TopologyEntry>
  <TopologyEntry>
  <ForPhones>
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
  </ForPhones>
  <State> 0 <PdfClass> 0 <Transition> 0 0.25 <Transition> 1 0.25 <Transition> 2 0.25 <Transition> 3 0.25 </State>
  <State> 1 <PdfClass> 1 <Transition> 1 0.25 <Transition> 2 0.25 <Transition> 3 0.25 <Transition> 4 0.25 </State>
  <State> 2 <PdfClass> 2 <Transition> 1 0.25 <Transition> 2 0.25 <Transition> 3 0.25 <Transition> 4 0.25 </State>
  <State> 3 <PdfClass> 3 <Transition> 1 0.25 <Transition> 2 0.25 <Transition> 3 0.25 <Transition> 4 0.25 </State>
  <State> 4 <PdfClass> 4 <Transition> 4 0.75 <Transition> 5 0.25 </State>
  <State> 5 </State>
  </TopologyEntry>
  </Topology>
  \endverbatim
  This specifies the topology of the HMMs we use.  In this case, the "real" phones contain
  three emitting states
  with the standard 3-state left-to-right topology-- the "Bakis model".
  (Emitting states are states that "emit" feature vectors, as distinct from the "fake"
  non-emitting states that are just used to glue other states together).
  Phones 1 to 20 are various kinds of silence and noise; we have a lot because of word-position-dependency,
  and in fact most of these will never be used; the real number excluding word position
  dependency is more like five.  The "silence phones" have a more complex topology with an
  initial emitting state and an end emitting state, but then three emitting states in the middle.
  You don't have to create this file by hand.
  
  There are a number of files in <DFN>data/lang/phones/</DFN> that specify various things about
  the phone set.  Most of these files exist in three separate versions: a ".txt" form, e.g.:
  \verbatim
  s5# head -3 data/lang/phones/context_indep.txt
  SIL
  SIL_B
  SIL_E
  \endverbatim
  a ".int" form, e.g:
  \verbatim
  s5# head -3 data/lang/phones/context_indep.int
  1
  2
  3
  \endverbatim
  and a ".csl" form, which in a slight abuse of notation, denotes a colon-separated list,
  not a comma-separated list:
  \verbatim
  s5# cat data/lang/phones/context_indep.csl
  1:2:3:4:5:6:7:8:9:10:11:12:13:14:15:16:17:18:19:20
  \endverbatim
  These files always contain the same information, so let's focus on the ".txt" form which
  is more human-readable.  The file "context_indep.txt" contains a list of those phones
  for which we build context-independent models: that is, for those phones, we do not build a decision tree
  that gets to ask questions about the left and right phonetic context.  In fact, we do build
  smaller trees where we get to ask questions about the central phone and the HMM-state;
  this depends on the "roots.txt" file which we'll describe below.  See \ref tree_externals
  for more in-depth discussion of tree issues.
  
  The file <DFN>context_indep.txt</DFN> contains all the phones which are not "real phones":
  i.e. silence (SIL), spoken noise (SPN), non-spoken noise (NSN), and laughter (LAU):
  \verbatim
  # cat data/lang/phones/context_indep.txt
  SIL
  SIL_B
  SIL_E
  SIL_I
  SIL_S
  SPN
  SPN_B
  SPN_E
  SPN_I
  SPN_S
  NSN
  NSN_B
  NSN_E
  NSN_I
  NSN_S
  LAU
  LAU_B
  LAU_E
  LAU_I
  LAU_S
  \endverbatim
  There are a lot of variants of these phones because of word-position dependency; not all of these variants
  will ever be used.  Here, <DFN>SIL</DFN> would be the silence that gets optionally inserted by the
  lexicon (not part of a word), <DFN>SIL_B</DFN> would be a silence phone at the beginning of a word
  (which should never exist), <DFN>SIL_I</DFN> word-internal silence (unlikely to exist), <DFN>SIL_E</DFN>
  word-ending silence (should never exist), and <DFN>SIL_S</DFN> would be silence as a "singleton
  word", i.e. a phone with only one word-- this might be used if you had a "silence word" in your
  lexicon and explicit silences appear in your transcriptions.
  
  The files <DFN>silence.txt</DFN> and <DFN>nonsilence.txt</DFN> contains lists of the silence
  phones and nonsilence phones respectively.  These should be mutually exclusive and together,
  should contain all the phones.  In this particular setup, <DFN>silence.txt</DFN> is identical
  to <DFN>context_indep.txt</DFN>.
  What we mean by "nonsilence" phones is, phones that we
  intend to estimate various kinds of linear transforms on: that is, global transforms
  such as LDA and MLLT, and speaker adaptation transforms such as fMLLR.  Our belief based
  on prior experience is that it does not pay to include silence in the estimation of
  such transforms.  Our practice is
  to designate all silence, noise and vocalized-noise phones as "silence" phones, and all
  phones representing traditional phonemes as "nonsilence" phones.  We haven't experimented
  in Kaldi with the best way to do this.
  \verbatim
  s5# head -3 data/lang/phones/silence.txt
  SIL
  SIL_B
  SIL_E
  s5# head -3 data/lang/phones/nonsilence.txt
  IY_B
  IY_E
  IY_I
  \endverbatim
  
  The file <DFN>disambig.txt</DFN> contains a list of the "disambiguation symbols"
  (see \ref graph_disambig):
  \verbatim
  s5# head -3 data/lang/phones/disambig.txt
  #0
  #1
  #2
  \endverbatim
  These symbols appear in the file <DFN>phones.txt</DFN> as if they were phones.
  
  The file <DFN>optional_silence.txt</DFN> contains a single phone which can optionally
  appear between words:
  \verbatim
  s5# cat data/lang/phones/optional_silence.txt
  SIL
  \endverbatim
  The mechanism by which it appears optionally between words is that it appears
  optionally in the lexicon FST at the end of every word (and also the beginning of the
  utterance).  The reason it has to be specified in <DFN>phones/</DFN> instead of just appearing
  in <DFN>L.fst</DFN> is obscure and we won't go into it here.
  
  The file <DFN>sets.txt</DFN> contains sets of phones that we group together (consider as
  the same phone) while clustering the phones in order to create the context-dependency questions
  (in Kaldi we use automatically generated questions when building decision trees,
  rather than linguistically meaningful ones).
  In this particular setup, <DFN>sets.txt</DFN> groups together all the word-position-dependent
  versions of each phone:
  \verbatim
  s5# head -3 data/lang/phones/sets.txt
  SIL SIL_B SIL_E SIL_I SIL_S
  SPN SPN_B SPN_E SPN_I SPN_S
  NSN NSN_B NSN_E NSN_I NSN_S
  \endverbatim
  
  The file <DFN>extra_questions.txt</DFN> contains some extra questions which we'll include
  in addition to the automatically generated questions:
  \verbatim
  s5# cat data/lang/phones/extra_questions.txt
  IY_B B_B D_B F_B G_B K_B SH_B L_B M_B N_B OW_B AA_B TH_B P_B OY_B R_B UH_B AE_B S_B T_B AH_B V_B W_B Y_B Z_B CH_B AO_B DH_B UW_B ZH_B EH_B AW_B AX_B EL_B AY_B EN_B HH_B ER_B IH_B JH_B EY_B NG_B
  IY_E B_E D_E F_E G_E K_E SH_E L_E M_E N_E OW_E AA_E TH_E P_E OY_E R_E UH_E AE_E S_E T_E AH_E V_E W_E Y_E Z_E CH_E AO_E DH_E UW_E ZH_E EH_E AW_E AX_E EL_E AY_E EN_E HH_E ER_E IH_E JH_E EY_E NG_E
  IY_I B_I D_I F_I G_I K_I SH_I L_I M_I N_I OW_I AA_I TH_I P_I OY_I R_I UH_I AE_I S_I T_I AH_I V_I W_I Y_I Z_I CH_I AO_I DH_I UW_I ZH_I EH_I AW_I AX_I EL_I AY_I EN_I HH_I ER_I IH_I JH_I EY_I NG_I
  IY_S B_S D_S F_S G_S K_S SH_S L_S M_S N_S OW_S AA_S TH_S P_S OY_S R_S UH_S AE_S S_S T_S AH_S V_S W_S Y_S Z_S CH_S AO_S DH_S UW_S ZH_S EH_S AW_S AX_S EL_S AY_S EN_S HH_S ER_S IH_S JH_S EY_S NG_S
  SIL SPN NSN LAU
  SIL_B SPN_B NSN_B LAU_B
  SIL_E SPN_E NSN_E LAU_E
  SIL_I SPN_I NSN_I LAU_I
  SIL_S SPN_S NSN_S LAU_S
  \endverbatim
  You will observe that a question is simply a set of phones.
  The first four questions are asking about the word-position, for regular phones; and the last five do the same for
  the "silence phones".  The "silence" phones also come in a variety without a suffix like <DFN>_B</DFN>,
  for example <DFN>SIL</DFN>.  These may appear as optional silence in the lexicon, i.e. not inside an
  actual word.  In setups with things like tone dependency or stress markings, <DFN>extra_questions.txt</DFN>
  may contain questions that relate to those features.
  
  The file <DFN>word_boundary.txt</DFN> explains how the phones relate to word positions:
  \verbatim
  s5# head  data/lang/phones/word_boundary.txt
  SIL nonword
  SIL_B begin
  SIL_E end
  SIL_I internal
  SIL_S singleton
  SPN nonword
  SPN_B begin
  \endverbatim
  This is the same information as is in the suffixes of the phones (<DFN>_B</DFN> and so on), but
  we don't like to hardcode this in the text form of the phones-- for one thing, Kaldi executables
  never see the text form of the phones, but only an integerized form.  So it is specified
  by this file <DFN>word_boundary.txt</DFN>.  The main reason we need this information is
  in order to recover the word boundaries within lattices (for example, the program
  lattice-align-words reads the integer versin of this file, <DFN>word_boundaray.int</DFN>).
  Finding the word boundaries is useful for reasons including NIST sclite scoring, which requires
  the time markings for words, and for other downstream processing.
  
  The file <DFN>roots.txt</DFN> contains information that relates to how we build the phonetic-context
  decision tree:
  \verbatim
  head data/lang/phones/roots.txt
  shared split SIL SIL_B SIL_E SIL_I SIL_S
  shared split SPN SPN_B SPN_E SPN_I SPN_S
  shared split NSN NSN_B NSN_E NSN_I NSN_S
  shared split LAU LAU_B LAU_E LAU_I LAU_S
  ...
  shared split B_B B_E B_I B_S
  \endverbatim
  For now you can ignore the words "shared" and "split"-- these relate to certain options
  in how we build the decision tree (see \ref tree_externals for more information).
  The significance of having a number of phones on a single line, for
  example <DFN>SIL SIL_B SIL_E SIL_I SIL_S</DFN>, is that all of these phones
  have a single "shared root" in the decision tree, so states may be shared
  between those phones.  For stress and tone-dependent systems, typically
  all the stress or tone-dependent versions of a particular phone will appear on
  the same line.  In addition, all three states of a HMM (or all five states, for
  silences) share the root, and the decision-tree building process gets to
  ask about the state.  This sharing of the decision-tree root
  between the HMM-states is what we mean by "shared" in the roots file.
  
  \section data_prep_lang_creating Creating the "lang" directory
  
  The <DFN>data/lang/</DFN> directory contains a lot of different files, so we have
  provided a script that creates it for you starting from a relatively simple
  input:
  \verbatim
  utils/prepare_lang.sh data/local/dict "<UNK>" data/local/lang data/lang
  \endverbatim
  Here, the inputs are the directory <DFN>data/local/dict/</DFN>, and the label <DFN>\<UNK\></DFN>
  which is the dictionary word we will map OOV words to when appear in transcripts
  (this becomes data/lang/oov.txt).  The location <DFN>data/local/lang/</DFN> is simply a
  temporary directory which the script will use; <DFN>data/lang/</DFN> is where
  it actually puts its output.
  
  The thing which you, as the data-preparer, need to create, is the directory
  <DFN>data/local/dict/</DFN>.  The directory contains the following contents:
  \verbatim
  s5# ls data/local/dict
  extra_questions.txt  lexicon.txt nonsilence_phones.txt  optional_silence.txt  silence_phones.txt
  \endverbatim
  (in fact there are a few more files there which we haven't listed, but they are just temporary files that
  were put there while creating that directory, and we can ignore them).  The commands below give
  you an idea what is in these files:
  \verbatim
  s5# head -3 data/local/dict/nonsilence_phones.txt
  IY
  B
  D
  s5# cat data/local/dict/silence_phones.txt
  SIL
  SPN
  NSN
  LAU
  s5# cat data/local/dict/extra_questions.txt
  s5# head -5 data/local/dict/lexicon.txt
  !SIL SIL
  -'S S
  -'S Z
  -'T K UH D EN T
  -1K W AH N K EY
  \endverbatim
  As you can see, the contents of this directory are very simple in this
  setup (the Switchboard setup).  We just have lists of the "real" phones and of the
  "silence" phones respectively, an empty file called <DFN>extra_questions.txt</DFN>, and
  a file called <DFN>lexicon.txt</DFN> which has the format
  \verbatim
  <word> <phone1> <phone2> ...
  \endverbatim
  Note: <DFN>lexicon.txt</DFN> will contain repeated entries for the same word,
  on separate lines,
  if we have multiple pronunciations for it.  If you want to use pronunciation
  probabilities, instead of creating the file <DFN>lexicon.txt</DFN>, create a file
  called <DFN>lexiconp.txt</DFN> that has the probability as the second field.
  Note that it is a common practice to normalize the pronunciations probabilities so that
  instead of summing to one, the most probable pronunciation for each word is one.  This
  tends to give better results.  For a top-level script that runs with
  pronunciation probabilities, search for <DFN>pp</DFN> in <DFN>egs/wsj/s5/run.sh</DFN>.
  
  Notice that in this input there is no notion of word-position dependency,
  i.e. no suffixes like <DFN>_B</DFN> and <DFN>_E</DFN>.  This is because it is the
  scripts <DFN>prepare_lang.sh</DFN> that adds those suffixes.
  
  You can see from the empty <DFN>extra_questions.txt</DFN> file that there
  is some kind of potential here that is not being fully exploited.  This relates
  to things like stress markings or tone markings.  You may want to have different
  versions of a particular phone that have different stress or tone.  In order
  to demonstrate what this looks like, we'll view the same files as above,
  but in the <DFN>egs/wsj/s5/</DFN> setup.  The result is below:
  \verbatim
  s5# cat data/local/dict/silence_phones.txt
  SIL
  SPN
  NSN
  s5# head data/local/dict/nonsilence_phones.txt
  S
  UW UW0 UW1 UW2
  T
  N
  K
  Y
  Z
  AO AO0 AO1 AO2
  AY AY0 AY1 AY2
  SH
  s5# head -6 data/local/dict/lexicon.txt
  !SIL SIL
  <SPOKEN_NOISE> SPN
  <UNK> SPN
  <NOISE> NSN
  !EXCLAMATION-POINT  EH2 K S K L AH0 M EY1 SH AH0 N P OY2 N T
  "CLOSE-QUOTE  K L OW1 Z K W OW1 T
  s5# cat data/local/dict/extra_questions.txt
  SIL SPN NSN
  S UW T N K Y Z AO AY SH W NG EY B CH OY JH D ZH G UH F V ER AA IH M DH L AH P OW AW HH AE R TH IY EH
  UW1 AO1 AY1 EY1 OY1 UH1 ER1 AA1 IH1 AH1 OW1 AW1 AE1 IY1 EH1
  UW0 AO0 AY0 EY0 OY0 UH0 ER0 AA0 IH0 AH0 OW0 AW0 AE0 IY0 EH0
  UW2 AO2 AY2 EY2 OY2 UH2 ER2 AA2 IH2 AH2 OW2 AW2 AE2 IY2 EH2
  s5#
  \endverbatim
  You may notice that some of the lines in <DFN>nonsilence_phones.txt</DFN> contain
  multiple phones on a single line.  These are the different stress-dependent
  versions of the vowels.  Note that four different versions of each phone
  appear in the CMU dictionary: for example, <DFN>UW UW0 UW1 UW2</DFN>;
  for some reason, one of these versions does not have a numeric suffix.
  The order of the phones on the line does not matter, but the grouping into
  different lines does matter; in general, we advise users to group all forms of
  each "real phone" on a separate line.
  We use the stress markings present in the CMU
  dictionary.  The file extra_questions.txt contains a single question
  containing all of the "silence" phones (in fact this is unnecessary as
  it appears that the script <DFN>prepare_lang.sh</DFN> adds such a question anyway),
  and also a question corresponding to each of the different stress markers.
  These questions are necessary in order to get any benefit from the
  stress markers, because the fact that the different stress-dependent versions
  of each phone are together on the lines of <DFN>nonsilence_phones.txt</DFN>,
  ensures that they stay together in <DFN>data/lang/phones/roots.txt</DFN> and
  <DFN>data/lang/phones/sets.txt</DFN>, which in turn ensures that they
  share the same tree root and can never be distinguished by a question.  Thus,
  we have to provide a special question that affords the decision-tree building
  process a way to distinguish between the phones.  Note: the reason we put the
  phones together in the <DFN>sets.txt</DFN> and <DFN>roots.txt</DFN> is that some
  of the stress-dependent versions of phones may have too little data to
  robustly estimate either a separate decision tree or the phone clustering
  information that's used in producing the questions.  By grouping them together
  like this, we ensure that in the absence of enough data to estimate them
  separately, these different versions of the phone all "stay together" throughout
  the decision-tree building process.
  
  We should mention at this point that the script <DFN>utils/prepare_lang.sh</DFN>
  supports a number of options.  To give you an idea of what they are, here is
  the usage messages of that script:
  \verbatim
  usage: utils/prepare_lang.sh <dict-src-dir> <oov-dict-entry> <tmp-dir> <lang-dir>
  e.g.: utils/prepare_lang.sh data/local/dict <SPOKEN_NOISE> data/local/lang data/lang
  options:
       --num-sil-states <number of states>             # default: 5, #states in silence models.
       --num-nonsil-states <number of states>          # default: 3, #states in non-silence models.
       --position-dependent-phones (true|false)        # default: true; if true, use _B, _E, _S & _I
                                                       # markers on phones to indicate word-internal positions.
       --share-silence-phones (true|false)             # default: false; if true, share pdfs of
                                                       # all non-silence phones.
       --sil-prob <probability of silence>             # default: 0.5 [must have 0 < silprob < 1]
  \endverbatim
  A potentially important option is the <DFN>--share-silence-phones</DFN> option.
  The default is false.  If this option is true, all the pdf's (the Gaussian
  mixture models) of all the silence phones such as silence, vocalized-noise,
  noise and laughter, will be shared and only the transition probabilities will
  differ between those models.  It's not clear why this should help, but we found
  that it was extremely helpful for the Cantonese data of IARPA's BABEL project.
  That data is very messy and has long untranscribed portions that we try to
  align to a special phone which we designate for that purpose.  We suspect
  that the training data was somehow failing to align correctly, and for some reason
  setting this option to true changed that.
  
  Another potentially important option is the "--sil-prob" option.   In general, we have
  not experimented much with any of these options so we cannot give very detailed advice.
  
  \section data_prep_grammar Creating the language model or grammar
  
  Our tutorial above on how to create the <DFN>lang/</DFN> directory did not address how
  to create the file <DFN>G.fst</DFN>, which is the finite state transducer form of
  the language model or grammar that we'll decode with.  In fact, in some setups
  we may have many "lang" directories for testing purposes, with different
  language models and dictionaries.  The Wall Street Journal (WSJ) setup is an example:
  \verbatim
  s5# echo data/lang*
  data/lang data/lang_test_bd_fg data/lang_test_bd_tg data/lang_test_bd_tgpr data/lang_test_bg \
   data/lang_test_bg_5k data/lang_test_tg data/lang_test_tg_5k data/lang_test_tgpr data/lang_test_tgpr_5k
  \endverbatim
  
  The process for creating <DFN>G.fst</DFN> is different depending on whether we're using
  a statistical language model or some kind of grammar.  In the RM setup there is
  a bigram grammar, which only allows certain pairs of words.  We make this sum to
  one within each grammar state by assigning a probability of 1 over the number of
  outgoing arcs.  There is a statement in <DFN>local/rm_data_prep.sh</DFN> that does:
  \verbatim
  local/make_rm_lm.pl $RMROOT/rm1_audio1/rm1/doc/wp_gram.txt  > $tmpdir/G.txt || exit 1;
  \endverbatim
  This script <DFN>local/make_rm_lm.pl</DFN> creates a grammar in FST format (text format,
  not binary format).  It contains lines like the following:
  \verbatim
  s5# head data/local/tmp/G.txt
  0    1    ADD    ADD    5.19849703126583
  0    2    AJAX+S    AJAX+S    5.19849703126583
  0    3    APALACHICOLA+S    APALACHICOLA+S    5.19849703126583
  \endverbatim
  See <a href=www.openfst.org> www.openfst.org </a> for more information on OpenFst (they
  have a useful tutorial).  The script <DFN>local/rm_prepare_grammar.sh</DFN> will turn this into
  the binary-format file <DFN>G.fst</DFN> using the following statement:
  \verbatim
  fstcompile --isymbols=data/lang/words.txt --osymbols=data/lang/words.txt --keep_isymbols=false \
      --keep_osymbols=false $tmpdir/G.txt > data/lang/G.fst
  \endverbatim
  If you want to create your own grammar, you will probably want to do something similar.
  Note: this type of procedure only applies to grammars of a certain class: it won't
  allow you to compile a complete Context Free Grammar, because it can't be represented
  in OpenFst format.  There are ways to do this in the WFST framework
  (e.g. see recent work by Mike Riley with push down transducers), but we have not yet
  worked with those ideas in Kaldi.
  
  Please, before asking any questions on the list about language models or about making
  grammar FSTs, read "A Bit of Progress in Language Modeling" by Joshua Goodman; and go to
  www.openfst.org and do the FST tutorial so that you understand the basics of finite
  state transducers.  (Note that language models would be represented as finite state
  acceptors, or FSAs, which can be considered as a special case of finite state transducers).
  
  The script <DFN>utils/format_lm.sh</DFN> deals with converting the ARPA-format language
  models into an OpenFst format. Here is the usage messages of that script:
  \verbatim
  Usage: utils/format_lm.sh <lang_dir> <arpa-LM> <lexicon> <out_dir>
  E.g.: utils/format_lm.sh data/lang data/local/lm/foo.kn.gz data/local/dict/lexicon.txt data/lang_test
  Convert ARPA-format language models to FSTs.
  \endverbatim
  Some of the key commands from that script are:
  \verbatim
  gunzip -c $lm \
    | arpa2fst --disambig-symbol=#0 \
               --read-symbol-table=$out_dir/words.txt - $out_dir/G.fst
  \endverbatim
  This Kaldi program, <DFN>arpa2fst</DFN>, turns the ARPA-format language model
  into a Weight Finite State Transducer (actually, an acceptor).
  
  A popular toolkit for building language models is SRILM.  Various language
  modeling toolkits are used in the Kaldi example scripts.  SRILM is the best
  documented and most fully featured, and we generally recommend it (its only
  drawback is that it don't have the most free licence). Here is the usage
  messages of <DFN>utils/format_lm_sri.sh</DFN>
  
  \verbatim
  Usage: utils/format_lm_sri.sh [options] <lang-dir> <arpa-LM> <out-dir>
  E.g.: utils/format_lm_sri.sh data/lang data/local/lm/foo.kn.gz data/lang_test
  Converts ARPA-format language models to FSTs. Change the LM vocabulary using SRILM.
  \endverbatim
  
  
  \section data_prep_unknown Note on unknown words
  
  This is an explanation of how Kaldi deals with unknown words (words not in the
  vocabulary); we are putting it on the "data preparation" page for lack of a more obvious
  location.
  
  In many setups, <DFN>\<unk\></DFN> or something similar will be present in the
  LM as long as the data that you used to train the LM had words that were not
  in the vocabulary you used to train the LM,
  because language modeling toolkits tend to map those all to a
  single special world, usually called <DFN>\<unk\></DFN> or
  <DFN>\<UNK\></DFN>.  You can look at the arpa file to figure out what it's called; it
  will usually be one of those two.
  
  
  During training, if there are words in the <DFN>text</DFN> file in your data
  directory that are not in the <DFN>words.txt</DFN> in the lang directory that
  you are using, Kaldi will map them to a special word that's specified in the
  lang directory in the file <DFN>data/lang/oov.txt</DFN>; it will usually be
  either <DFN>\<unk\></DFN>, <DFN>\<UNK\></DFN> or maybe
  <DFN>\<SPOKEN_NOISE\></DFN>.  This word will have been chosen by the user
  (i.e., you), and supplied to <DFN>prepare_lang.sh</DFN> as a command-line argument.
  If this word has nonzero probability in the language model (which you can test
  by looking at the arpa file), then it will be possible for Kaldi to recognize
  this word in test time.  This will often be the case if you call this word
  <DFN>\<unk\></DFN>, because as we mentioned above, language modeling toolkits
  will often use this spelling for ``unknown word'' (which is a special word that
  all out-of-vocabulary words get mapped to).  Decoding output will always be limited to the
  intersection of the words in the language model with the words in the lexicon.txt (or whatever file format you supplied the
  lexicon in, e.g. lexicop.txt); these words will all be present in the <DFN>words.txt</DFN>
  in your <DFN>lang</DFN> directory.
  So if Kaldi's "unknown word" doesn't match the LM's "unknown word", you will
  simply never decode this word.  In any
  case, even when allowed to be decoded, this word typically won't be output very
  often and in practice it doesn't tend to have much impact on WERs.
  
  Of course a single phone isn't a very good, or accurate, model of OOV words.  In
  some Kaldi setups we have example scripts with names
  <DFN>local/run_unk_model.sh</DFN>: e.g., see the file
  <DFN>tedlium/s5_r2/local/run_unk_model.sh</DFN>.  These scripts replace the unk
  phone with a phone-level LM on phones.  They make it possible to get access to
  the sequence of phones in a hypothesized unknown word.  Note: unknown words
  should be considered an "advanced topic" in speech recognition and we discourage
  beginners from looking into this topic too closely.
  
  
  
  */