// doc/online_programs.dox
  
  
  // Copyright 2013 Polish-Japanese Institute of Information Technology (author: Danijel Korzinek)
  
  // See ../../COPYING for clarification regarding multiple authors
  //
  // Licensed under the Apache License, Version 2.0 (the "License");
  // you may not use this file except in compliance with the License.
  // You may obtain a copy of the License at
  
  //  http://www.apache.org/licenses/LICENSE-2.0
  
  // THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  // KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
  // WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
  // MERCHANTABLITY OR NON-INFRINGEMENT.
  // See the Apache 2 License for the specific language governing permissions and
  // limitations under the License.
  
  namespace kaldi {
  
  /**
  
  \page online_programs Online Recognizers
  
  Warning, this page is deprecated as it refers to the older online-decoding setup.
  The page for the new setup is \ref online_decoding.
  
  There are several programs in the Kaldi toolkit that can be used for online recognition. 
  They are all located in the src/onlinebin folder and require the files from
  the src/online folder to be compiled as well (you can currently compile these with
  "make ext"). Many of these programs will also require the
  Portaudio library present in the tools folder, which can be downloaded using the appropriate
  script found there. The programs are as follows:
    - online-gmm-decode-faster - records audio from the microphone and outputs the result on stdout 
    - online-wav-gmm-decode-faster - reads a list of WAV files and outputs the result into special output format
    - online-server-gmm-decode-faster - reads vectors of MFCC features from a UDP socket and prints the result to stdout
    - online-net-client - records audio from the microphone, converts it to vectors of features and sends them over UDP to online-server-gmm-decode-faster   
    - online-audio-server-decode-faster - reads RAW audio data from a TCP socket and outputs aligned words to the same socket
    - online-audio-client - reads a list of WAV files, sends their audio over TCP to online-audio-server-decode-faster, reads the output and saves the result to the chosen file format
  
  There is also a Java equivalent of the online-audio-client which contains slightly more features and has a GUI.
  
  In addition, there is a GStreamer 1.0 compatible plugin that acts as a filter, taking raw audio as input and producing 
  recognized word as output. The plugin is based on  \ref OnlineFasterDecoder, as other online recognition programs.   
  
  \section audio_server Online Audio Server
  
  The main difference between the online-server-gmm-decode-faster and
  online-audio-server-decode-faster programs is the input: the former accepts
  feature vectors, while the latter accepts RAW audio.  The advantage of the
  latter is that it can be deployed directly as a back-end for any client: whether
  it is another computer on the Internet or a mobile device.  Main thing here is
  that the client doesn't need to know anything about the feature set used to
  train the models and provided it can record standard audio at the predetermined
  sampling frequency and bit-depth, it will always be compatible with the
  server. An advantage of the server that accepts feature vectors, instead of
  audio, is a lower cost of data transfer between the client and the server, but
  this can be easily outperformed by simply using a state-of-the-art codec for
  audio (which is something that may be done in the future).
  
  The communication between the online-audio-client and
  online-audio-server-decode-faster consists of two phases: first the client sends
  packets of raw audio to the server, second the server replies with the output of
  the recognition. The two phases may happen asynchronously, meaning that the
  decoder can output results online, as fast as it is certain of their outcome and
  not wait for the end of the data to arrive. This opens up more possibilities for
  creating applications in the future.
  
  \subsection audio_data Audio Data
  
  The audio data format is currently hardcoded to be RAW 16 kHz, 16-bit, signed, little-endian (server native), linear PCM. The protocol works by splitting data into 
  chunks and prepending each chunk with a 4-byte code containing its exact size. The size is also (server native) little-endian and can contain any value as long as it is positive
  and even (because of 16-bit sampling). The last packet of size 0 is treated as the end of stream and forces the decoder to dump the rest of its results and finish the recognition process.
  
  \subsection results Recognition Results
  
   There are two types of packets sent by the server. Time-aligned results and partial results. Partial results are sent as fast as the decoder recognizes each word and time-alignment is performed
  every time the decoder recognizes the end of an utterance (which may or may not correspond to silences and/or sentance boundaries).
  
   Each partial result packet is prepended by the characters "PARTIAL:" follwed by exactly one word. Each word is sent in a different partial packet.
  
   Time-aligned results are prepended by a header starting with the characters "RESULT:".  Following that is a comma separated list of key=value parameters containing some useful information:
    - NUM - number of words to follow
    - FORMAT - the format of the result to follow; currently only WSEC (word-start-end-confidence), but it will allow modification in the future
    - RECO-DUR - time it took to recognize the sequence (as a floating point value in seconds)
    - INPUT-DUR - length of the input audio sequence (as a floating point value in seconds)
  
   You can divide INPUT-DUR by RECO-DUR to get real-time recognition speed of the decoder.  
  
   The header "RESULT:DONE" is sent when there are no more results that can be returned by the server. In this case, the server simply waits for either more data by the client, or for a disconnect.
    
   The data underneath the header consists of exactly NUM lines of words formatted in the way determined by the FORMAT parameter. In the case of WSE format, this is simply a comma separated list
  containing 4 tokens: the word (as present in the dictionary), start time and end time (as floating point values in seconds). Beware that the words 
  are going to be encoded exactly as they are in the dictionary provided to the server and therefore, the client must make sure to perform the appropriate character conversion if necessary. The 
  online-audio-client, for example, doesn't perform any character conversion while generating WebVTT files (which require UTF8), so you need to convert the resulting files to UTF8 using iconv (or 
  a similar program).
   
   An example of the results of the server is as follows:
   \verbatim
   RESULT:NUM=3,FORMAT=WSE,RECO_DUR=1.7,INPUT_DUR=3.22
   one,0.4,1.2
   two,1.4,1.9
   three,2.2,3.4
   RESULT:DONE 
  \endverbatim
  
  \subsection usage Example Usage
  
  Command line to start the server:
  \verbatim
  online-audio-server-decode-faster --verbose=1 --rt-min=0.5 --rt-max=3.0 --max-active=6000 --beam=72.0 --acoustic-scale=0.0769 
  final.mdl graph/HCLG.fst graph/words.txt '1:2:3:4:5' graph/word_boundary.int 5010 final.mat
  \endverbatim
  
  Arguments are as follow:
    - final.mdl - the acoustic model
    - HCLG.fst - the complete FST
    - words.txt - word dictionary (mapping word ids to their textual representation)
    - '1:2:3:4:5' - list of silence phoneme ids
    - 5010 - port the server is listening on
    - word_boundary.int - a list of phoneme boundary information required for word alignemnt
    - final.mat - feature LDA matrix
    
   Command line to start the client:
   \verbatim
   online-audio-client --htk --vtt localhost 5010 scp:test.scp
   \endverbatim
  
  Arguments are as follow:
    - --htk - save results as an HTK label file
    - --vtt - save results as a WebVTT file
    - localhost - server to connect to
    - 5010 - port to connect to
    - scp:test.scp - list of WAV files to send
    
  Command line to start the Java client:
  \verbatim
  java -jar online-audio-client.jar
  \endverbatim
  
  Or simply double-click the JAR file in the graphical interface.
  
  \section gst_plugin GStreamer plugin
  
  Kaldi toolkit comes with a plugin for the <a href="http://gstreamer.freedesktop.org/">GStreamer</a> media streaming framework (version 1.0 or compatible).
  The plugin acts as a filter that accepts raw audio as input and produces recognized words as output.
  
  The main benefit of the plugin is the fact that it makes Kaldi's online speech recognition functionality available to all
  programming languages that support GStreamer 1.0 (that includes Python, Ruby, Java, Vala and many more). It also simplifies the integration
  of the Kaldi online decoder in applications since communicating with the decoder follows GStreamer standards. 
  
  \subsection gst_plugin_installation Installation
  
  The source of the GStreamer plugin is located in the `src/gst-plugin` directory. To compile the plugin, rest of the Kaldi
  toolkit has to be compiled to use the shared libraries. To do this, invoke `configure` with the `--shared` flag.
  Also compile the online extensions (`make ext`). 
   
  Make sure the package that provides  GStreamer 1.0 development headers is installed on your system. On Debian Jessie (current 'testing' 
  distribution), the needed package is called `libgstreamer1.0-dev`. On Debian Wheezy (current stable), GStreamer 1.0 is available from the backports
  repository (http://backports.debian.org). Install the 'good' GStreamer plugins (package gstreamer1.0-plugins-good)
  and GStreamer 1.0 tools (package `gstreamer1.0-tools`). A demo program also requires the PulseAudio Gstreamer plugins 
  (package gstreamer1.0-pulseaudio).
  
  Finally, run `make depend` and `make` in the `src/gst-plugin` directory. This should result in a file `src/gst-plugin/libgstonlinegmmdecodefaster.so`
  which contains the GStreamer plugin.
  
  To make GStreamer able to find the Kaldi plugin, you have to add the `src/gst-plugin` directory to its plugin search path. To do this,
  add the directory to the GST_PLUGIN_PATH environment variable:
  \verbatim
  export GST_PLUGIN_PATH=$KALDI_ROOT/src/gst-plugin
  \endverbatim
  Of course, replace `$KALDI_ROOT` with the actual location of the Kaldi root folder on your file system. 
  
  Now, running `gst-inspect-1.0 onlinegmmdecodefaster` should provide info about the plugin:
  \verbatim
  # gst-inspect-1.0 onlinegmmdecodefaster
  Factory Details:
    Rank:     none (0)
    Long-name:        OnlineGmmDecodeFaster
    Klass:            Speech/Audio
    Description:      Convert speech to text
    Author:           Tanel Alumae <tanel.alumae@phon.ioc.ee>
  [..]
  Element Properties:
    name                : The name of the object
                          flags: readable, writable
                          String. Default: "onlinegmmdecodefaster0"
    parent              : The parent of the object
                          flags: readable, writable
                          Object of type "GstObject"
    silent              : Determines whether incoming audio is sent to the decoder or not
                          flags: readable, writable
                          Boolean. Default: false
    model               : Filename of the acoustic model
                          flags: readable, writable
                          String. Default: "final.mdl"
    fst                 : Filename of the HCLG FST
                          flags: readable, writable
                          String. Default: "HCLG.fst"
  [..]
    min-cmn-window      : Minumum CMN window used at start of decoding (adds latency only at start)
                          flags: readable, writable
                          Integer. Range: -2147483648 - 2147483647 Default: 100 
  
  Element Signals:
    "hyp-word" :  void user_function (GstElement* object,
                                      gchararray arg0,
                                      gpointer user_data);
  \endverbatim
  
  \subsection usage_cli Usage through the command-line
  
  The most simple way to use the GStreamer plugin is via the command line. You have to specify the model files used for decoding
  when lauching the plugin. To do this, set the `model`, `fst`, `word-syms`, `silence-phones` and optionally the `lda-mat`
  plugin properties (similarly to Kaldi's command-line online decoders). The decoder accepts only 16KHz 16-bit mono audio. Any audio stream can be automatically  converted to the
  required format using GStreamer's `audioresample` and `audioconvert` plugins.  
  
  For example, to decode the file `test1.wav` using a model files in `tri2b_mmi`, and have the recognized stream of words printed to stdout, execute:
  \verbatim
  gst-launch-1.0 -q filesrc location=test1.wav \
      ! decodebin ! audioconvert ! audioresample \
      ! onlinegmmdecodefaster model=tri2b_mmi/model fst=tri2b_mmi/HCLG.fst \
                              word-syms=tri2b_mmi/words.txt silence-phones="1:2:3:4:5" lda-mat=tri2b_mmi/matrix \
      ! filesink location=/dev/stdout buffer-mode=2
  \endverbatim
  Note that the audio stream is segmented on the fly, with "<#s>" denoting silence.
  
  You can easily try live decoding of microphone input by replacing `filesrc location=test1.wav` with `pulsesrc` (given that
  your OS uses the PulseAudio framework).
  
  An example stript that uses the plugin via the command-line to process a buch of audio files is located in `egs/voxforge/gst_demo/run-simulated.sh`.
  
  \subsection usage_gst Usage through GStreamer bindings
  
  An example of a Python GUI program that uses the plugin via the GStreamer bindings is located in `egs/voxforge/gst_demo/run-live.py`.
  
  The program constructs in the `init_gst(self)` method a similar pipeline of GStreamer elements as in the command-line example.
  The model files and some decoding parameters are communicated to the `onlinegmmdecodefaster` element through the standard `set_property()`
  method. More interesting is this part of the code:
  \verbatim
          self.asr.connect('hyp-word', self._on_word)
  \endverbatim
  This expression orders our decoding plugin to call the GUI's `_on_word` method whenever it produces a new recognized word. 
  The `_on_word()` method looks like this:
  \verbatim
      def _on_word(self, asr, word):
          Gdk.threads_enter()
          if word == "<#s>":
            self.textbuf.insert_at_cursor("
  ")
          else:
            self.textbuf.insert_at_cursor(word)
          self.textbuf.insert_at_cursor(" ")
          Gdk.threads_leave()
  \endverbatim
  What it does (apart from some GUI-related chemistry), is that it inserts the recognized word into the text buffer that is connected
  to the GUI's main text box. If a segmentation symbol is recognized, it inserts a line break instead.
  
  Recognition start and stop are controlled by setting the `silent` property of the decoder plugin to `False` or `True`. Setting the
  property to `False` orders the plugin not to process any incoming audio (although the audio that is already being processed might
  produce some new recognized words).
   
  
  \subsection code_structure Overview of the online decoders' code
  
  The main components used in the implemention of the online decoders are as follows:
      - \ref OnlineAudioSource - represents a source of raw audio samples. There are different 
        implementations of this "interface", that can for example acquire samples from microphone
        (e.g. \ref OnlinePaSource, using PortAudio), or read them from existing \ref Vector
        (\ref OnlineVectorSource). Technically OnlineAudioSource is currently not a C++ interface, using 
        virtual methods - instead the classes "implementing" it are passed as template parameters
        to their users.
      - \ref OnlineFeatInputItf - used to read feature vectors that are computed in some manner.
        The computation in this case can mean feature extraction(\ref OnlineFeInput), cepstral mean
        normalization (\ref OnlineCmnInput), calculation of higher order features
        (\ref OnlineDeltaInput) and so on.
      - \ref OnlineDecodableDiagGmmScaled - an implementation of \ref DecodableInterface, reading
        its input features from an implementation of OnlineFeatInputItf(through OnlineFeatureMatrix proxy).
      - \ref OnlineFasterDecoder - extends \ref FasterDecoder (described in the \ref decoders section),
        to make it work with real time input.
  
  The process of feature extraction and transformation is implemented by forming a pipeline of several
  objects, each of which pulling data from the previous object in the pipeline, performing a specific type 
  of computation and passing the result to the next one. Usually, at the start of the pipeline there is an
  OnlineAudioSource object, which is used to acquire the raw audio in some way. Then comes a component
  of type \ref OnlineFeInput, that reads the raw samples and extracts features. OnlineFeInput can be used to produce 
  either MFCC or PLP features, by passing reference to the object doing the actual job(\ref Mfcc or \ref Plp) as
  parameter to its constructor. After that the pipeline continues with other OnlineFeatInputItf objects. For
  example if we want the decoder to receive audio from a microphone using PortAudio and to use a vector of 
  MFCC, delta and delta-delta features the pipeline looks like:
  
  \verbatim
  OnlinePaSource -> OnlineFeInput<OnlinePaSource, Mfcc> -> OnlineCmnInput -> OnlineDeltaInput
  \endverbatim 
  
  <B>NOTE:</B> The pipeline shown above has nothing to do with GStreamer pipeline described in the previous
  sections - the both structures are completely orthogonal to each other. The GStreamer plug-in makes it 
  possible to grab audio samples from GStreamer's pipeline, by implementing a new OnlineAudioSource 
  (\ref GstBufferSource), and feed them into Kaldi online decoder's pipeline.
  
  \ref OnlineFasterDecoder itself is completely oblivious to the details of feature computation. As is the case with
  the other Kaldi decoders, the decoder only "knows" that it can get scores for a given feature frame using a Decodable 
  object. The actual type of this object, for the currently implemented example programs, is OnlineDecodableDiagGmmScaled. 
  OnlineDecodableDiagGmmScaled doesn't directly access the final OnlineFeatInputItf either. 
  Instead, the responsibility for bookkeeping and requesting batches of features from the pipeline, as well as
  coping with potential timeouts is delegated to an object of type \ref OnlineFeatureMatrix.
  
  The chain of function calls, that results in pulling new features along the pipeline, looks like the following:
  
  -# in \ref OnlineFasterDecoder::ProcessEmitting(), the decoder invokes 
     \ref OnlineDecodableDiagGmmScaled::LogLikelihood()
  -# this triggers a call to \ref OnlineFeatureMatrix::IsValidFrame(). If the requested feature frame is not
     already available inside the cache maintained by OnlineFeatureMatrix, a new batch of features is requested,
     by calling the \ref OnlineFeatInputItf::Compute() method of the last object in the pipeline.
  -# the Compute() method of the last object (e.g. OnlineDeltaInput in the example above) then calls the Compute()
     method of the previous object in the pipeline and so on to the first OnlineFeatInputItf object, which invokes
     OnlineAudioSource::Read().
  
  OnlineFasterDecoder inherits most of its implementation from FasterDecoder, but its Decode() method processes the 
  feature vectors in batches of configurable size and returns a code corresponding to the current decoder state.
  There are three different codes: 
  
  - <I>kEndFeats</I> means there are no more features left in the pipeline
  - <I>kEndUtt</I> signals end of the current utterance, which can be used for example to separate the utterances by
    new line characters, as in our demos, or other application-dependent feadback. An utterance is considered 
    to be finished when sufficiently long silence is detected in the traceback. The number of the silence frames 
    that are interpreted as utterance delimiter is controlled by \ref OnlineDecoderOpts::inter_utt_sil.
  - <I>kEndBatch</I> - nothing extraordinary has happened - the decoder just finished processing the current batch of
    feature vectors. 
  
  OnlineFasterDecoder makes it possible to get recognition results as soon as possible, by implementing partial traceback.
  The back-pointers of all active \ref FasterDecoder::Token objects are followed until a point is reached, back in time,
  where there is only one active token. That means that this (immortal) token is a common ancestor of all currently 
  active tokens. Moreover by keeping track of the previous such token(for the previously processed batches), 
  the decoder can produce a partial result, by taking the output symbols along the traceback from the current immortal
  token to the previous one.
  
  */
  
  
  }