// doc/io_tut.dox

// Copyright 2013  Johns Hopkins University (author: Daniel Povey)

// See ../../COPYING for clarification regarding multiple authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at

//  http://www.apache.org/licenses/LICENSE-2.0

// THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
// WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
// MERCHANTABLITY OR NON-INFRINGEMENT.
// See the Apache 2 License for the specific language governing permissions and
// limitations under the License.


namespace kaldi {
/** \page io_tut  Kaldi I/O from a command-line perspective.

 This page describes the I/O mechanisms in Kaldi from the perspective of
 a user of the command line tools.  See \ref io for a more code-level overview.

 \section Overview

 \subsection io_tut_nontable Non-table I/O

 We first describe "non-table" I/O.  This refers to files or streams containing just
 one or two objects (e.g. acoustic model files; transformation matrices), rather than a
 collection of objects indexed by strings.

   - Kaldi file formats are binary by default but programs will output non-binary
     if you supply the flag --binary=false.
   - Many objects have corresponding "copy" programs, e.g. copy-matrix or gmm-copy,
      which can be used with the --binary=false flag to convert to text form, e.g.
     "copy-matrix --binary=false foo.mat -".
   - There is typically a one-to-one correspondence between an file on disk and a C++ object
     in memory, e.g. a matrix of floats, although some files contain more than one object
     (Case in point: for acoustic model files, typically a TransitionModel object and then
      an acoustic model).
   - Kaldi programs typically know which type of object they are expecting to read, rather
     than working it out from the stream.
   - Similarly to perl, a filename can be replaced with - (for standard input/output) or
     a string such as "|gzip -c >foo.gz" or "gunzip -c foo.gz|"
   - For reading files, we also support things like foo:1045, meaning character-offset
     1045 within file foo.
   - In order to refer to our concept of an extended filename, we generally use the special terms 'rxfilename' for
     a string describing a stream to be read (i.e. a file, stream or the standard input),
     and 'wxfilename' for a string describing an output stream.  See \ref io_sec_xfilename for more details.

 To illustrate the concepts above, make sure $KALDI_ROOT/src/bin is on your path,
  where $KALDI_ROOT is the top of the repository, and type the following:
\verbatim
  echo '[ 0 1 ]' | copy-matrix - -
\endverbatim
It will print out a log message and some binary data corresponding to that matrix.  Now try:
\verbatim
  echo '[ 0 1 ]' | copy-matrix --binary=false - -
\endverbatim
The output will look like this:
\verbatim
# copy-matrix --binary=false - -
copy-matrix --binary=false - -
 [
  0 1 ]
LOG (copy-matrix:main():copy-matrix.cc:68) Copied matrix to -
\endverbatim
Although it looks like the matrix and log messages are mixed up, the log messages
are on the standard error and would not be passed into a pipe; to avoid seeing
the log messages you could redirect stderr to /dev/null by adding 2>/dev/null to the
command line.

Kaldi programs may be connected using pipes or by using the
stream-as-a-file mechanism of Kaldi I/O.  Here is a pipe example:
\verbatim
 echo '[ 0 1 ]' | copy-matrix - - | copy-matrix --binary=false - -
\endverbatim
This outputs the matrix in text form (the first copy-matrix command converts
to binary form and the second to text form, which is of course pointless).
You could accomplish the same thing in a more convoluted way by doing this:
\verbatim
  copy-matrix 'echo [ 0 1 ]|' '|copy-matrix --binary=false - -'
\endverbatim
There is no reason to do this here, but it can sometimes be useful when
programs have multiple inputs or outputs so the stdin or stdout is
already being used.  It is particularly useful with tables (see next section).


 \subsection io_tut_table Table I/O

 Kaldi has special I/O mechanisms for dealing with collections of objects
 indexed by strings.  Examples of this are feature matrices indexed by
 utterance-ids, or speaker-adaptation transformation matrices indexed
 by speaker-ids.  The strings that index the collection must be nonempty
 and whitespace free.   See \ref io_sec_tables for a more in-depth
 discussion.

 A Table may exist in two forms: an "archive" or a "script file".  The
 difference is that the archive actually contains the data, while
 a script file points to the location of the data.

 Programs that read from Tables expect a string we call an "rspecifier" that
 says how to read the indexed data, and programs that write to Tables expect
 a string we call a "wspecifier" to write it.  These are strings that specify
 whether to expect script file or an archive, and the file location, along
 with various options.  Common types of
 rspecifiers include "ark:-", meaning read the data as an archive
 from the standard input, or "scp:foo.scp", meaning the script file
 foo.scp says where to read the data from.  Points to bear in
 mind are:

   - The part after the colon is interpreted as a wxfilename or rxfilename (as
     in \ref io_tut_nontable), meaning that things like pipes and standard
     input/output are supported.
   - A Table always contains just one type of object (e.g., a matrix of floats).
   - You may see options on rspecifiers and wspecifiers, principally:
      - In rspecifiers, ark,s,cs:- means that when we read (from the standard input in this case)
        we expect the keys to be in sorted order (,s) and we assert that they will be accessed
        in sorted order (,cs) meaning that we know the program will
        access them in sorted order (the program will crash if these conditions do not hold).
        This allows Kaldi to emulate random access without using up a lot of memory.
      - For data that isn't too large and for which it's inconvenient to ensure sorted order
        (e.g. transforms for speaker adaptation), there is little harm in omitting the ,s,cs.
      - Typically programs that take multiple rspecifiers will iterate over the objects in the
        first one (sequential access) and do random access on the later ones, so ",s,cs" is
        generally not needed for the first rspecifier.
      - In scp,p:foo.scp, the ,p means we should not crash if some of the
        referenced files do not exist (for archives, ,p will prevent a crash if
        the archive is corrupted or truncated.)
      - For writing, the option ,t means text mode, e.g. in ark,t:-.
        The --binary command-line option has no effect for archives.
   - The script-file format is, on each line, "<key> <rspecifier|wspecifier>", e.g.
      "utt1 /foo/bar/utt1.mat".  It is OK for the rspecifier or wspecifier to contain
      spaces, e.g. "utt1 gunzip -c /foo/bar/utt1.mat.gz|".
   - The archive format is: <key1> <object1> <newline> <key2> <object2> <newline> ...
   - Archives may be concatenated and they will still be valid archives, but be careful about
     the order of concatenation, e.g. avoid \verbatim "cat a/b/*.ark"\endverbatim
     if you need the sorted order.
   - Although not often used, script files may be used for output, e.g. if we write to
     the wspecifier scp:foo.scp, and the program tries to write to key utt1,
     it looks for a line like "utt1 some_file" in foo.scp, and will write to "some_file".  It will crash
     if there is no such line.
   - It is possible to write to both an archive and script at the same time,
     e.g. ark,scp:foo.ark,foo.scp.  The script file will be written to, with lines
     like "utt1 foo.ark:1016" (i.e. it points to byte offsets in the archive). This is useful when data is to be accessed in random order
     or in parts, but you don't want to produce lots of small files.
   - It is possible to trick the archive mechanism into operating on single files.  For instance,
\verbatim
     echo '[ 0 1 ]' | copy-matrix 'scp:echo foo -|' 'scp,t:echo foo -|'
\endverbatim
     This deserves a little explanation.  Firstly, the rspecifier "scp:echo foo -|" is equivalent
     to scp:bar.scp if the file bar.scp contained just the line "foo -".  This
     tells it to read the object indexed by "foo" from the standard input.  Similarly, for
     the wspecifier "scp,t:echo foo -|", it writes the data for "foo" to the standard
     output.  This trick should not be overused.  In this particular case, it is unnecessary
     because we have made the copy-matrix program support non-table I/O directly,
     so you could have written just "copy-matrix - -".  If you have to use
     this trick too much, it's better to modify the program concerned.
   - If you want to extract just one member from an archive, you can use the ",p" option for the "scp:" wspecifier to
     cause it to write out just that element, and ignore the other missing elements in the scp file.
     Suppose the key you want is "foo_bar" and an archive named some_archive.ark contains matrices, then
     you could extract that element as follows:
\verbatim
     copy-matrix 'ark:some_archive.ark' 'scp,t,p:echo foo_bar -|'
\endverbatim
   - In certain cases the archive-reading code allows for limited type conversion, e.g.
     between float and double for matrices, or Lattice and CompactLattice for lattices.


  \subsubsection io_tut_table_ranges Table I/O (with ranges)

   It is now possible to specify row ranges and column ranges of matrices from scp files.
   It is typical, when you dump feature files, to have them represented as an scp file
   that looks something like:
\verbatim
 utt-00001  /some/dir/feats.scp:0
 utt-00002  /some/dir/feats.scp:16402
 ...
\endverbatim
 You can modify this scp file by adding row and column ranges, with a format similar to
 MATLAB (except with zero-based indexes).  So, for instance, if you modify it to:
\verbatim
 utt-00001  /some/dir/feats.scp:0[0:9]
 utt-00001  /some/dir/feats.scp:0[10:19]
 ...
\endverbatim
 the first two lines would represent rows 0 through 9, and rows 10 through 19,
 of utt-00001.  You can represent column indexes in a similar way:
\verbatim
 utt-00001  /some/dir/feats.scp:0[:,0:12]
 utt-00001  /some/dir/feats.scp:0[:,13:25]
 ...
\endverbatim
 would be columns 0 through 12, and columns 13 through 25, of that file.
 You can also have combinations of row and column indexes: for instance,
 \verbatim
 utt-00001 /some/dir/feats.scp:0[10:19,0:12]
\endverbatim




  \subsection io_tut_maps Utterance-to-speaker and speaker-to-utterance maps.

  Many Kaldi programs take utterance-to-speaker and speaker-to-utterances maps-- files
  called "utt2spk" or "spk2utt".  These are generally specified by command-line options
   --utt2spk and --spk2utt.  The utt2spk map has the format
\verbatim
utt1 spk_of_utt1
utt2 spk_of_utt2
...
\endverbatim
   and the spk2utt map has the format
\verbatim
spk1 utt1_of_spk1 utt2_of_spk1 utt3_of_spk1
spk2 utt1_of_spk2 utt2_of_spk2
...
\endverbatim
 These files are used for speaker adaptation, e.g. for finding which speaker corresponds
 to an utterance, or to iterate over speakers.
 For reasons that relate mostly to the way the Kaldi example scripts are set up
 and the way we split data up into multiple pieces, it's important to ensure
 that the speakers in the utterance-to-speaker map are in sorted order (see \ref data_prep).
 Anyway, these files are actually treated as archives, and for this reason
 you will see command-line options like --utt2spk=ark:data/train/utt2spk.
 You will see that these files fit the generic archive format of: "<key1> <data> <newline> <key2> <data> <newline>",
 where in this case the data is in text form.
 At the code level, the utt2spk file is treated as a table containing a string, and the spk2utt
 file is treated as a table containing a list of strings.


*/
}