// doc/kaldi_for_dummies.dox

// Copyright 2016 Wit Zieliński  Pavel Denisov

// See ../../COPYING for clarification regarding multiple authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at

//  http://www.apache.org/licenses/LICENSE-2.0

// THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
// WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
// MERCHANTABLITY OR NON-INFRINGEMENT.
// See the Apache 2 License for the specific language governing permissions and
// limitations under the License.

/**
 \page kaldi_for_dummies Kaldi for Dummies tutorial

\tableofcontents

\section kaldi_for_dummies_introduction Introduction

This is a step by step tutorial for absolute beginners on how to create a
simple ASR (Automatic Speech Recognition) system in Kaldi toolkit using your
own set of data. I really would have liked to read something like this when I
was starting to deal with Kaldi. This is all based on my experience as an
amateur in case of speech recognition subject and script programming as well.
If you have ever delved through Kaldi tutorial on the official project site and
felt a little bit lost, well, my piece of art might be the choice for you. You
will learn how to install Kaldi, how to make it work and how to run an ASR
system using your own audio data. As an effect you will get your first speech
decoding results.  It was created by Wit Zielinski.

First of all - get to know what Kaldi actually is and why you should use it
instead of something else. In my opinion Kaldi requires solid knowledge about
speech recognition and ASR systems in general. It is also good to know the
basics of script programming languages (bash, perl, python). C++ migh be useful
in the future (probably you will want to make some modifications in the source
code).

To read: <br>
\ref about <br>
\ref tutorial_prereqs

\section kaldi_for_dummies_environment Environment

Rule number 1 - use Linux. Although it is possible to use Kaldi on Windows,
most people I find trustworthy convinced me that Linux will do the job with the
less amount of problems. I have chosen Ubuntu 14.10. This was (in 2014/15) a
rich and stable Linux representation which I honestly recommend. When you
finally have your Linux running properly, please open a terminal and install
some necessary stuff (if you do not already have it):

(has to be installed)
 - \c atlas – automation and optimization of calculations in the
field of linear algebra,
 - \c autoconf – automatic software compilation on different operating systems,
 - \c automake – creating portable Makefile files,
 - \c git – distributed revision control system,
 - \c libtool – creating static and dynamic libraries,
 - \c svn – revision control system (Subversion), necessary for Kaldi download
and installation,
 - \c wget – data transfer using HTTP, HTTPS and FTP protocols,
 - \c zlib – data compression,

(probably has to be installed)
 - \c awk – programming language, used for searching and processing patterns
in files and data streams,
 - \c bash – Unix shell and script programming language,
 - \c grep – command-line utility for searching plain-text datasets for lines
matching a regular expression,
 - \c make – automatically builds executable programs and libraries from
source code,
 - \c perl – dynamic programming language, perfect for text files processing.

Done. Operating system and all the necessary Linux tools are ready to go.

\section kaldi_for_dummies_download Download Kaldi

Just follow the instruction carefully:
\ref install.
If you do not have much idea about how to use GIT, please read about it:
\ref tutorial_git.

I installed Kaldi in this directory (called 'Kaldi root path'):
\c /home/{user}/kaldi

\section kaldi_for_dummies_directories Kaldi directories structure

Try to acknowledge where particular Kaldi components are placed. Also it would
be nice if you read any \c README files you find.

\c kaldi - main Kaldi directory which contains:
 - \c egs – example scripts allowing you to quickly build ASR
systems for over 30 popular speech corpora (documentation is attached for each
project),
 - \c misc – additional tools and supplies, not needed for proper
Kaldi functionality,
 - \c src – Kaldi source code,
 - \c tools – useful components and external tools,
 - \c windows – tools for running Kaldi using Windows.

The most important directory for you is obviously \c egs. Here you
will create your own ASR system.

\section kaldi_for_dummies_project Your exemplary project

For the purpose of this tutorial, imagine that you have the same simple set of
data as me (described below, in \ref kaldi_for_dummies_audio "Audio data" section).
Then try to 'transpose' every action I do straight into your own project. If
you completely do not have any audio data or you want to follow my tutorial in
an identical way, feel free to record your own tracks - it will be even bigger
experience to play with ASR. Here we go.

<h2>Your precondition</h2>
You have some amount of audio data that contain only spoken digits by at least
several different speakers. Each audio file is an entire spoken sentence
(e.g. 'one, nine, five').

<h2>Your purpose</h2>
You want to divide your data into train and test sets, set up an ASR system,
train it, test it and get some decoding results.

<h2>Your first task</h2>
Something to begin with - create a folder \c digits in
\c kaldi/egs/ directory. This is a place where you will put all
the stuff related to your project.

\section kaldi_for_dummies_data Data preparation

\subsection kaldi_for_dummies_audio Audio data

I assume that you want to set up an ASR system, basing on your own audio data.
For example - let it be a set of 100 files. File format is WAV. Each file
contains 3 spoken digits recorded in English language, one by one. Each of
these audio files is named in a recognizable way (e.g. \c 1_5_6.wav,
which in my pattern means that the spoken sentence is 'one, five, six') and
placed in the recognizable folder representing particular speaker during a
particular recording session (there may be a situation that you have recordings
of the same person but in two different quality/noise environments - put these
in separate folders). So to sum up, my exemplary dataset looks like this:
 - 10 different speakers (ASR systems must be trained and tested on different
speakers, the more speakers you have the better),
 - each speaker says 10 sentences,
 - 100 sentences/utterances (in 100 *.wav files placed in 10 folders related to
particular speakers - 10 *.wav files in each folder),
 - 300 words (digits from zero to nine),
 - each sentence/utterance consist of 3 words.

Whatever your first dataset is, adjust my example to your particular case. Be
careful with big datasets and complex grammars - start with something simple.
Sentences that contain only digits are perfect in this case.

<h2>Task</h2>
Go to \c kaldi/egs/digits directory and create
\c digits_audio folder. In \c kaldi/egs/digits/digits_audio
create two folders: \c train and \c test. Select one speaker
of your choice to represent testing dataset. Use this speaker's 'speakerID' as
a name for an another new folder in \c kaldi/egs/digits/digits_audio/test
directory. Then put there all the audio files related to that person. Put the
rest (9 speakers) into \c train folder - this will be your training
dataset. Also create subfolders for each speaker.

\subsection kaldi_for_dummies_acoustic Acoustic data

Now you have to create some text files that will allow Kaldi to communicate with
your audio data. Consider these files as 'must be done'. Each file that you will
create in this section (and in \ref kaldi_for_dummies_language "Language data"
section as well) can be considered as a text file with some number of strings
(each string in a new line). These strings need to be sorted. If you will
encounter any sorting issues you can use Kaldi scripts for checking
(\c utils/validate_data_dir.sh) and fixing (\c utils/fix_data_dir.sh) data order.
And for your information - \c utils directory will be attached to your project in
\ref kaldi_for_dummies_tools "Tools attachment" section.

<h2>Task</h2>
In \c kaldi/egs/digits directory, create a folder \c data. Then create
\c test and \c train subfolders inside. Create in each subfolder following files
(so you have files named in <b>the same way in \c test and \c train subfolders
but they relate to two different datasets</b> that you created before):

a.) \c spk2gender <br>
This file informs about speakers gender. As we assumed, 'speakerID' is a unique
name of each speaker (in this case it is also a 'recordingID' - every speaker
has only one audio data folder from one recording session). In my example there
are 5 female and 5 male speakers (f = female, m = male).

<b>Pattern:</b> <speakerID> <gender>
\verbatim
cristine f
dad m
josh m
july f
# and so on...
\endverbatim

b.) \c wav.scp <br>
This file connects every utterance (sentence said by one person during
particular recording session) with an audio file related to this utterance. If
you stick to my naming approach, 'utteranceID' is nothing more than 'speakerID'
(speaker's folder name) glued with *.wav file name without '.wav' ending (look
for examples below).

<b>Pattern:</b> <uterranceID> <full_path_to_audio_file>
\verbatim
dad_4_4_2 /home/{user}/kaldi/egs/digits/digits_audio/train/dad/4_4_2.wav
july_1_2_5 /home/{user}/kaldi/egs/digits/digits_audio/train/july/1_2_5.wav
july_6_8_3 /home/{user}/kaldi/egs/digits/digits_audio/train/july/6_8_3.wav
# and so on...
\endverbatim

c.) \c text <br>
This file contains every utterance matched with its text transcription.

<b>Pattern:</b> <uterranceID> <text_transcription>
\verbatim
dad_4_4_2 four four two
july_1_2_5 one two five
july_6_8_3 six eight three
# and so on...
\endverbatim

d.) \c utt2spk <br>
This file tells the ASR system which utterance belongs to particular speaker.

<b>Pattern:</b> <uterranceID> <speakerID>
\verbatim
dad_4_4_2 dad
july_1_2_5 july
july_6_8_3 july
# and so on...
\endverbatim

e.) \c corpus.txt <br>
This file has a slightly different directory. In \c kaldi/egs/digits/data
create another folder \c local. In \c kaldi/egs/digits/data/local create a
file \c corpus.txt which should contain every single utterance transcription
that can occur in your ASR system (in our case it will be 100 lines from 100
audio files).

<b>Pattern:</b> <text_transcription>
\verbatim
one two five
six eight three
four four two
# and so on...
\endverbatim

\subsection kaldi_for_dummies_language Language data

This section relates to language modeling files that also need to be considered
as 'must be done'. Look for the syntax details here: \ref data_prep (each file
is precisely described). Also feel free to read some examples in other \c egs
scripts. Now is the perfect time.

<h2>Task</h2>
In \c kaldi/egs/digits/data/local directory, create a folder \c dict. In
\c kaldi/egs/digits/data/local/dict create following files:

a.) \c lexicon.txt <br>
This file contains every word from your dictionary with its 'phone
transcriptions' (taken from \c /egs/voxforge).

<b>Pattern:</b> <word> <phone 1> <phone 2> ...
\verbatim
!SIL sil
<UNK> spn
eight ey t
five f ay v
four f ao r
nine n ay n
one hh w ah n
one w ah n
seven s eh v ah n
six s ih k s
three th r iy
two t uw
zero z ih r ow
zero z iy r ow
\endverbatim

b.) \c nonsilence_phones.txt <br>
This file lists nonsilence phones that are present in your project.

<b>Pattern:</b> <phone>
\verbatim
ah
ao
ay
eh
ey
f
hh
ih
iy
k
n
ow
r
s
t
th
uw
w
v
z
\endverbatim

c.) \c silence_phones.txt <br>
This file lists silence phones.

<b>Pattern:</b> <phone>
\verbatim
sil
spn
\endverbatim

d.) \c optional_silence.txt <br>
This file lists optional silence phones.

<b>Pattern:</b> <phone>
\verbatim
sil
\endverbatim

\section kaldi_for_dummies_finalization Project finalization

Last chapter before runnig scripts creation. Your project structure will become
complete.

\subsection kaldi_for_dummies_tools Tools attachment

You need to add necessary Kaldi tools that are widely used in exemplary scripts.

<h2>Task</h2>
From \c kaldi/egs/wsj/s5 copy two folders (with the whole content) -
\c utils and \c steps - and put them in your
\c kaldi/egs/digits directory. You can also create links to these
directories. You may find such links in, for example,
\c kaldi/egs/voxforge/s5.

\subsection kaldi_for_dummies_scoring Scoring script

This script will help you to get decoding results.

<h2>Task</h2>
From \c kaldi/egs/voxforge/s5/local copy the script \c score.sh into
similar location in your project (\c kaldi/egs/digits/local).

\subsection kaldi_for_dummies_srilm SRILM installation

You also need to install language modelling toolkit that is used in my
example - SRI Language Modeling Toolkit (SRILM).

<h2>Task</h2>
For detailed installation instructions go to
\c kaldi/tools/install_srilm.sh (read all comments inside).

\subsection kaldi_for_dummies_configuration Configuration files

It is not necessary to create configuration files but it can be a good habit
for future.

<h2>Task</h2>
In \c kaldi/egs/digits create a folder \c conf. Inside
\c kaldi/egs/digits/conf create two files (for some configuration
modifications in decoding and mfcc feature extraction processes - taken from
\c /egs/voxforge):

a.) \c decode.config <br>
\verbatim
first_beam=10.0
beam=13.0
lattice_beam=6.0
\endverbatim

b.) \c mfcc.conf <br>
\verbatim
--use-energy=false
\endverbatim

\section kaldi_for_dummies_running Running scripts creation

Your first ASR system written in Kaldi environment is almost ready. Your last
job is to prepare running scripts to create ASR system of your choice. I put
some comments in prepared scripts for ease of understanding.

These scripts are based on solution used in \c /egs/voxforge directory. I
decided to use two different training methods:
- MONO - monophone training,
- TRI1 - simple triphone training (first triphone pass).

These two methods are enough to show noticable differences in decoding results
using only digits lexicon and small training dataset.

<h2>Task</h2>
In \c kaldi/egs/digits directory create 3 scripts:

a.) \c cmd.sh <br>
\code{.sh}
# Setting local system jobs (local CPU - no external clusters)
export train_cmd=run.pl
export decode_cmd=run.pl
\endcode

b.) \c path.sh <br>
\code{.sh}
# Defining Kaldi root directory
export KALDI_ROOT=`pwd`/../..

# Setting paths to useful tools
export PATH=$PWD/utils/:$KALDI_ROOT/src/bin:$KALDI_ROOT/tools/openfst/bin:$KALDI_ROOT/src/fstbin/:$KALDI_ROOT/src/gmmbin/:$KALDI_ROOT/src/featbin/:$KALDI_ROOT/src/lmbin/:$KALDI_ROOT/src/sgmm2bin/:$KALDI_ROOT/src/fgmmbin/:$KALDI_ROOT/src/latbin/:$PWD:$PATH

# Defining audio data directory (modify it for your installation directory!)
export DATA_ROOT="/home/{user}/kaldi/egs/digits/digits_audio"

# Enable SRILM
. $KALDI_ROOT/tools/env.sh

# Variable needed for proper data sorting
export LC_ALL=C
\endcode

c.) \c run.sh
\code{.sh}
#!/bin/bash

. ./path.sh || exit 1
. ./cmd.sh || exit 1

nj=1       # number of parallel jobs - 1 is perfect for such a small dataset
lm_order=1 # language model order (n-gram quantity) - 1 is enough for digits grammar

# Safety mechanism (possible running this script with modified arguments)
. utils/parse_options.sh || exit 1
[[ $# -ge 1 ]] && { echo "Wrong arguments!"; exit 1; }

# Removing previously created data (from last run.sh execution)
rm -rf exp mfcc data/train/spk2utt data/train/cmvn.scp data/train/feats.scp data/train/split1 data/test/spk2utt data/test/cmvn.scp data/test/feats.scp data/test/split1 data/local/lang data/lang data/local/tmp data/local/dict/lexiconp.txt

echo
echo "===== PREPARING ACOUSTIC DATA ====="
echo

# Needs to be prepared by hand (or using self written scripts):
#
# spk2gender  [<speaker-id> <gender>]
# wav.scp     [<uterranceID> <full_path_to_audio_file>]
# text	      [<uterranceID> <text_transcription>]
# utt2spk     [<uterranceID> <speakerID>]
# corpus.txt  [<text_transcription>]

# Making spk2utt files
utils/utt2spk_to_spk2utt.pl data/train/utt2spk > data/train/spk2utt
utils/utt2spk_to_spk2utt.pl data/test/utt2spk > data/test/spk2utt

echo
echo "===== FEATURES EXTRACTION ====="
echo

# Making feats.scp files
mfccdir=mfcc
# Uncomment and modify arguments in scripts below if you have any problems with data sorting
# utils/validate_data_dir.sh data/train     # script for checking prepared data - here: for data/train directory
# utils/fix_data_dir.sh data/train          # tool for data proper sorting if needed - here: for data/train directory
steps/make_mfcc.sh --nj $nj --cmd "$train_cmd" data/train exp/make_mfcc/train $mfccdir
steps/make_mfcc.sh --nj $nj --cmd "$train_cmd" data/test exp/make_mfcc/test $mfccdir

# Making cmvn.scp files
steps/compute_cmvn_stats.sh data/train exp/make_mfcc/train $mfccdir
steps/compute_cmvn_stats.sh data/test exp/make_mfcc/test $mfccdir

echo
echo "===== PREPARING LANGUAGE DATA ====="
echo

# Needs to be prepared by hand (or using self written scripts):
#
# lexicon.txt           [<word> <phone 1> <phone 2> ...]
# nonsilence_phones.txt	[<phone>]
# silence_phones.txt    [<phone>]
# optional_silence.txt  [<phone>]

# Preparing language data
utils/prepare_lang.sh data/local/dict "<UNK>" data/local/lang data/lang

echo
echo "===== LANGUAGE MODEL CREATION ====="
echo "===== MAKING lm.arpa ====="
echo

loc=`which ngram-count`;
if [ -z $loc ]; then
	if uname -a | grep 64 >/dev/null; then
		sdir=$KALDI_ROOT/tools/srilm/bin/i686-m64
	else
			sdir=$KALDI_ROOT/tools/srilm/bin/i686
	fi
	if [ -f $sdir/ngram-count ]; then
			echo "Using SRILM language modelling tool from $sdir"
			export PATH=$PATH:$sdir
	else
			echo "SRILM toolkit is probably not installed.
				Instructions: tools/install_srilm.sh"
			exit 1
	fi
fi

local=data/local
mkdir $local/tmp
ngram-count -order $lm_order -write-vocab $local/tmp/vocab-full.txt -wbdiscount -text $local/corpus.txt -lm $local/tmp/lm.arpa

echo
echo "===== MAKING G.fst ====="
echo

lang=data/lang
arpa2fst --disambig-symbol=#0 --read-symbol-table=$lang/words.txt $local/tmp/lm.arpa $lang/G.fst

echo
echo "===== MONO TRAINING ====="
echo

steps/train_mono.sh --nj $nj --cmd "$train_cmd" data/train data/lang exp/mono  || exit 1

echo
echo "===== MONO DECODING ====="
echo

utils/mkgraph.sh --mono data/lang exp/mono exp/mono/graph || exit 1
steps/decode.sh --config conf/decode.config --nj $nj --cmd "$decode_cmd" exp/mono/graph data/test exp/mono/decode

echo
echo "===== MONO ALIGNMENT ====="
echo

steps/align_si.sh --nj $nj --cmd "$train_cmd" data/train data/lang exp/mono exp/mono_ali || exit 1

echo
echo "===== TRI1 (first triphone pass) TRAINING ====="
echo

steps/train_deltas.sh --cmd "$train_cmd" 2000 11000 data/train data/lang exp/mono_ali exp/tri1 || exit 1

echo
echo "===== TRI1 (first triphone pass) DECODING ====="
echo

utils/mkgraph.sh data/lang exp/tri1 exp/tri1/graph || exit 1
steps/decode.sh --config conf/decode.config --nj $nj --cmd "$decode_cmd" exp/tri1/graph data/test exp/tri1/decode

echo
echo "===== run.sh script is finished ====="
echo
\endcode

\section kaldi_for_dummies_results Getting results

Now all you have to do is to run \c run.sh script. If I have made any mistakes
in this tutorial, logs from the terminal should guide you how to deal with it.

Besides the fact that you will notice some decoding results in the terminal
window, go to newly made \c kaldi/egs/digits/exp. You may notice there
folders with \c mono and \c tri1 results as well - directories structure are the
same. Go to \c mono/decode directory. Here you may find result files (named in
a <c>wer_{number}</c> way). Logs for decoding process may be found in \c log
folder (same directory).

\section kaldi_for_dummies_summary Summary

This is just an example. The point of this short tutorial is to show you how to
create 'anything' in Kaldi and to get a better understanding of how to think
while using this toolkit. Personally I started with looking for tutorials made
by the Kaldi authors/developers. After successful Kaldi installation I launched
some example scripts (Yesno, Voxforge, LibriSpeech - they are relatively easy
and have free acoustic/language data to download - I used these three as a base
for my own scripts).

Make sure you follow <a href="http://kaldi-asr.org/">
http://kaldi-asr.org/</a>- official project website.
There are two very useful sections for beginners inside: <br>
a.) \ref tutorial - almost 'step by step' tutorial on how to set up an ASR
system; up to some point this can be done without RM dataset. It is good to
read it, <br>
b.) \ref data_prep - very detailed explanation of how to use your own data
in Kaldi.

More useful links about Kaldi I found: <br>
<a href="https://sites.google.com/site/dpovey/kaldi-lectures">
https://sites.google.com/site/dpovey/kaldi-lectures
</a> - Kaldi lectures created by the main author <br>
<a href="http://www.superlectures.com/icassp2011/category.php?lang=en&id=131">
http://www.superlectures.com/icassp2011/category.php?lang=en&id=131
</a> - similar; video version <br>
<a href="http://www.diplomovaprace.cz/133/thesis_oplatek.pdf">
http://www.diplomovaprace.cz/133/thesis_oplatek.pdf
</a> - some master diploma thesis about speech recognition using Kaldi

This is all from my side. Good luck!

*/