// doc/tutorial_looking.dox

// Copyright 2009-2011 Microsoft Corporation

// See ../../COPYING for clarification regarding multiple authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at

//  http://www.apache.org/licenses/LICENSE-2.0

// THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
// WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
// MERCHANTABLITY OR NON-INFRINGEMENT.
// See the Apache 2 License for the specific language governing permissions and
// limitations under the License.

/**
 \page tutorial_looking Kaldi tutorial: Overview of the distribution (20 minutes)

   \ref tutorial "Up: Kaldi tutorial" <BR>
   \ref tutorial_git "Previous: Version control with Git" <BR>
   \ref tutorial_running "Next: Running the example scripts" <BR>

 Before we jump into the example scripts, let us take a few minutes to look at what
 else is included in the Kaldi distribution.  Go to the kaldi-1 directory and list it.
 There are a few files and subdirectories.
 The important subdirectories are "tools/", "src/", and "egs/" which we will
 look at in the next section.
 We will give an overview of "tools/" and "src/".

 \section tutorial_looking_tools The tools/ directory (10 minutes)

 The directory "tools/' is where we install things that Kaldi depends on in
 various ways.  Change directory to tools/ and list it.  You will see various
 files and subdirectories, mostly things that have been installed by the make command. 
 Look very quickly at the file INSTALL. This file gives instructions on how to install the tools.

 The most important subdirectory is the one for OpenFst.  cd to openfst/.  This is a soft link
 to the actual directory which has a version number.  List the openfst directory.
 If the installation succeeded, there will be a bin/ directory with the installed
 binaries, and a lib/ directory with the library (we require both of these).
 The most important code is in the directory include/fst/.  If you ever want to
 understand Kaldi deeply you will need to understand OpenFst.  For this,
 the best starting point is http://www.openfst.org/.

 For now, just view the file include/fst/fst.h.  This consists of some declarations
 of an abstract FST type.  You can see that there are a lot of templates involved.
 If templates are not your thing, you will probably have trouble understanding this code.

 Change directory to bin/, or add it to your path.
 We will be executing some simple example instructions from
 <a href=http://www.openfst.org/twiki/bin/view/FST/FstQuickTour#CreatingFsts>here</a>.


 Paste the following command into the shell:
\verbatim
# arc format: src dest ilabel olabel [weight]
# final state format: state [weight]
# lines may occur in any order except initial state must be first line
# unspecified weights default to 0.0 (for the library-default Weight type)
cat >text.fst <<EOF
0 1 a x .5
0 1 b y 1.5
1 2 c z 2.5
2 3.5
EOF
\endverbatim

The following commands create the symbol tables; paste them into the shell too.
\verbatim
cat >isyms.txt <<EOF
<eps> 0
a 1
b 2
c 3
EOF

cat >osyms.txt <<EOF
<eps> 0
x 1
y 2
z 3
EOF
\endverbatim
Note: for the following steps to work, if you don't have the current directory on your path
you may have to type:
\verbatim
export PATH=.:$PATH
\endverbatim
Next create a binary-format FST:
\verbatim
fstcompile --isymbols=isyms.txt --osymbols=osyms.txt text.fst binary.fst
\endverbatim
Let's execute an example command:
\verbatim
fstinvert binary.fst | fstcompose - binary.fst > binary2.fst
\endverbatim
The resulting WFST, binary2.fst, should be similar to binary.fst
but with twice the weights.  You can print them both out to see:
\verbatim
fstprint --isymbols=isyms.txt --osymbols=osyms.txt binary.fst
fstprint --isymbols=isyms.txt --osymbols=osyms.txt binary2.fst
\endverbatim
This example was modified from a longer tutorial available at
<a href=www.openfst.org> www.openfst.org </a>.  After you have done
this, clean up by typing:
\verbatim
rm *.fst *.txt
\endverbatim


 \section tutorial_looking_src The src/ directory (10 minutes)

 Change directory back up to the top level (kaldi-1) and into src/.
 List the directory.  You will see a few files and a large number of
 subdirectories.  Look at the Makefile.  At the top it sets the variable
 SUBDIRS.  This is a list of the subdirectories containing code.
 Notice that some of them end in "bin".  These are the ones that contain
 executables (the code and executables are in the same directory).  The
 other directories contain internal code.

 You can see that one of the targets in the Makefile is "test".
 Type "make test".  This command goes into the various subdirectories and
 runs test programs in there.  All the tests should succeed.  If you are
 feeling lucky you can also type "make valgrind".  This runs the same
 tests with a memory checker, and takes longer, but will find more
 errors.  If this doesn't work, forget about it; it's not important
 for now.  If it is taking too long, stop it with ctrl-c.

 Change directory to base/.  Look at the Makefile.  Notice the line
\verbatim
include ../kaldi.mk
\endverbatim
 This lines includes the file ../kaldi.mk verbatim whenever a Makefile in a
 subdirectory is invoked (just like a C \#include directive).
 Look at the file ../kaldi.mk.  It will contain
 some rules related to valgrind (for memory debugging), and then some
 system-specific configuration in the form of variables such as CXXFLAGS.
 See if there are any -O options (e.g. -O0).  The flags
 -O0 and -DKALDI_PARANOID are disabled by default as they slow things
 down (you might want to enable them for better debugging).
 Look again at base/Makefile.  The statement "all:" at the top tells Make
 that "all" is the top-level target (because there are targets in kaldi.mk
 and we don't want these to become the top-level target).  Because the
 dependencies of "all" depend on variables defined later, we have another
 statement (the target is defined in default_rules.mk) in which we define what "all" depends on.
 Look for it.  Several other targets are defined, starting with "clean".
 Look for them.  To make "clean" you would type "make clean".
 The target .valgrind is not something you would invoke from the command line;
 you would type "make valgrind" (the target is defined in kaldi.mk).
 Invoke all of these targets, i.e. type "make clean" and the same for the others,
 and notice what commands are issued when you do this.

 In the Makefile in the base/ directory: choose one of the binaries
listed in TESTFILES, and run it.  Then briefly view the corresponding .cc file.
The math one is a good example (note: this excludes the majority of math functions
in Kaldi, which are matrix-vector related functions, and are located in
../matrix/).  Notice that there are a lot of assertions, with the macro
KALDI_ASSERT.  These test programs are designed to exit with error status if
there is a problem (they are not supposed to rely on human inspection of the
output).

Look at the header kaldi-math.h.  You will see some elements of our coding practices.
Notice that all our local \#includes are relative to the src/ directory (so we \#include
base/kaldi-types.h even though we are already in the base/ directory).
Notice that all macros we \#define, except for standard ones that we are just
making sure have their normal values, begin with KALDI_.  This is a precaution
to avoid future conflicts with other codebases (since \#defines don't limit themselves
to the kaldi namespace).  Notice the style of the function names: LikeThis().
Our style is generally based on
<a href=https://google.github.io/styleguide/cppguide.html> this one </a>,
to conform with OpenFst, but there are some differences.

To see other elements of the style, which will help you to understand Kaldi
code, cd to ../util, and view text-utils.h.  Notice that the inputs of these
functions are always first, and are generally const references, while the
outputs (or inputs that are modified) are always last, and are pointer arguments.  Non-const references
as function arguments are not allowed.  You can read more about the Kaldi-specific
elements of the coding style \ref style "here" later if you are interested.
For now, just be aware that there is a coding style with quite specific rules.

Change directory to ../gmmbin and type
\verbatim
./gmm-init-model
\endverbatim
It prints out the usage, which should give you a generic idea of how Kaldi programs
are called.  Note that while there is a --config option that can be used to
pass a configuration file, in general Kaldi is not as config-driven as HTK and these
files are not widely used.  You will see a --binary option.  In general, Kaldi file
formats come in both binary and text forms, and the --binary option controls how
they are written.  However, this only controls how single objects (e.g. acoustic models)
are written.  For whole collections of objects (e.g. collections of feature files),
there is a different mechanism that we will come to later.
Type
\verbatim
./gmm-init-model >/dev/null
\endverbatim
What do you see, and what does this tell you about what Kaldi does with logging-type
output?  The place that the usage message goes is the same place that all error and
logging messages go, and there is a reason for this, which should become apparent
when you start looking at the scripts.

To get a little insight into the build process,  cd to ../matrix, and type
\verbatim
rm *.o
make
\endverbatim
Look at the options that are passed to the compiler.  These are ultimately
controlled by the variables that are set in ../kaldi.mk, which in turn is
determined by ../configure.  Also look at the linking options, passed in when it
creates matrix-lib-test.  You will get some idea what math libraries it is
linking against (this is somewhat system dependent).  For more information on how
we make use of external matrix libraries, you can read \ref matrixwrap.

Change directory to one level up (to src/), and look at the "configure" file.  If you
are familiar with the "configure" files generated by automake, you will notice that it
is not one of those.  It is hand generated.  Search within it for "makefiles/"
and quickly scan all the places where that string occurs (e.g. type into the shell
"less configure", type "/makefiles[enter]" and then type "n" to see later instances).
You will see that it makes use of some files with the suffix .mk in the subdirectory "makefiles/".
These are essentially "prototype" versions of kaldi.mk.  Look at one of the prototypes,
e.g. makefiles/cygwin.mk, to see the kinds of things they contain.  For systems that are more
predictable, it just concatenates the system specific makefile together with makefiles/kaldi.mk.common
and writes it to kaldi.mk.  For Linux, it has to
do a little more sleuthing because there are so many distributions.  Mostly this relates
to finding where the math libraries are installed.  If you are having problems with
a build process, one solution is to try modifying kaldi.mk by hand.  In order to do this you should
probably understand how Kaldi makes use of external math libraries (see \ref matrixwrap).

   \ref tutorial "Up: Kaldi tutorial" <BR>
   \ref tutorial_git "Previous: Version control with Git" <BR>
   \ref tutorial_running "Next: Running the example scripts" <BR>
<P>
*/