// doc/dnn3_scripts_context.dox
  
  
  // Copyright 2015   Johns Hopkins University (author: Daniel Povey)
  
  // See ../../COPYING for clarification regarding multiple authors
  //
  // Licensed under the Apache License, Version 2.0 (the "License");
  // you may not use this file except in compliance with the License.
  // You may obtain a copy of the License at
  
  //  http://www.apache.org/licenses/LICENSE-2.0
  
  // THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  // KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
  // WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
  // MERCHANTABLITY OR NON-INFRINGEMENT.
  // See the Apache 2 License for the specific language governing permissions and
  // limitations under the License.
  
  namespace kaldi {
  namespace nnet3 {
  
  /**
    \page dnn3_scripts_context  Context and chunk-size in the "nnet3" setup
  
    \section dnn3_scripts_context_intro Introduction
  
    This page discusses certain issues of terminology in the nnet3 setup
    about chunk sizes for decoding and training, and left and right context.
    This will be helpful in understanding some of the scripts.  At the current
    time don't have any 'overview' documentation of nnet3 from a scripting perspective,
    so this will have to stand as an isolated piece of documentation.
  
   \section dnn3_scripts_context_basics The basics
  
   If you have read the previous documentation available for \ref dnn3, you will
   realize that the "nnet3" setup supports setups other than simple feedforward
   DNNs.  It can be used for time delay neural networks (TDNNs) where temporal
   splicing (frame splicing) is done at internal layers of the network; and also
   for recurrent topologies (RNNs, LSTMs, BLSTMs, etc.).  So nnet3
   "knows about" the time axis.  Below we estabilish some terminology.
  
     \subsection dnn3_scripts_context_basics_context Left and right context
  
     Suppose we want a network to compute an output for a specific time index;
     to be concrete, say time t = 154.  If the network does frame splicing
     internally (or anything else nontrivial with the 't' indexes), it may not be able to
     compute this output without seeing a range of input frames.  For example,
     it may be impossible to compute the output without seeing the range of
     't' values from t = 150 through t = 157.  In this case (glossing over details),
     we'd say that the network has a \b left-context of 4 and a \b right-context of 3.
     The actual computation of the context is a bit more complex as it has to
     take into account special cases like where, say, the behavior for odd and
     even 't' values is different (c.f. Round() descriptors in
     \ref dnn3_dt_nnet_descriptor_config).
  
     There are cases with recurrent topologies where, in addition to the
     "required" left and right context, we want to give the training or the
     decoding "extra" context.  For such topologies, the network can make use
     of context beyond the required context.
     In the scripts you'll generally see variables called
     \b extra-left-context and \b extra-right-context, which mean
     "the amount of context that we're going to provide in addition to what is required".
  
     In some circumstances the names \b left-context and
     \b right-context simply mean the total left and right context that we're
     adding to the chunks, i.e. the sums of the model left/right context and the
     extra left/right context.  So in some circumstances you may have to work out
     from the context whether a variable refers to the <em>model</em> left/right context
     of the left/right context of the chunks of data.
  
     In Kaldi version 5.0 and earlier the left and right context in the chunks
     of data is not affected by whether the chunks were at the
     beginning or end of the utterance; at the ends we pad the input with copies of the
     first or last frame.  This means that for recurrent topologies, we might end up
     padding the start or end of the utterance with a lot of frames (up to 40 or so).
     This is wasteful and rather strange.
     In versions 5.1 and later, you can specify configuration values \b extra-left-context-initial and
     \b extra-right-context-final that allow the start/end of the utterance to have a different
     amount of context.  If you specify these values, you would normally specify them both to be 0
     (i.e. no extra context).  However, for back compatibility to older setups, they
     generally default to -1 (meaning, just copy the default extra-left-context and extra-right-context).
  
  
     \subsection dnn3_scripts_context_basics_chunk Chunk size
  
     The \b chunk-size is the number of (output) frames for each chunk of data
     that we evaluate in training or decoding.  In the get_egs.sh script
     and train_dnn.py it is also referred to as \b frames-per-eg (in some contexts,
     this is not the same as the chunk size; see below).  In decoding we call this
     the \b frames-per-chunk.
  
     \subsubsection dnn3_scripts_context_basics_chunk_dnn Non-recurrent, non-chain case
  
     For the very simplest types of networks, such as feedforward networks or TDNNs
     trained with the cross-entropy objective function, we randomize the entire
     dataset at the frame level and we just train on one frame at a time.   In order
     for the training jobs to mostly do sequential I/O, we aim pre-randomize the
     data at the frame level.  However, when you consider that we might easily
     require 10 frames each of left and right context, and we have to write this out,
     we could easily be increasing the amount of data by a factor of 20 or so when we
     generate the training examples.  To solve this problem we include labels for
     a range of time values, controlled by \b frames-per-eg (normally 8), and include
     enough left/right context that we can train on any of those 8 frames.  Then
     when we train the model, any given training job will pick one of those 8 frames to
     train on.
  
     \subsubsection dnn3_scripts_context_basics_chunk_rnn  Recurrent or chain case
  
    In models that are RNNs or LSTMs or are \ref chain, we always train on fairly large
    chunks (generally in the range 40 to 150 frames).  This is referred to as the
    \b chunk-size.  When we decode, we also generally evaluate the neural net on fairly
    large chunks of data (like, 30, 50 or 100 frames).  This is usually referred to
    as the \b frames-per-chunk.  For recurrent networks we tend to
    make sure that the \b chunk-size/\b frames-per-chunk
    and the \b extra-left-context and \b extra-right-context are about the same in
    training and decoding, because this generally gives the best results (although
    sometimes it's best to make the extra-context values slightly larger in decoding).
    One might expect that in decoding time longer context would always be better, but
    this does not always seem to be the case (however, see \ref dnn3_scripts_context_looped
    below, where we mention a way around this).
  
  
     \subsubsection dnn3_scripts_context_basics_chunk_subsampling Interaction of chunk size with frame-subsampling-factor
  
     In cases where there is frame-subsampling at the output (like the chain model),
     the chunk-size is still measured in multiples of 't', and we make sure (via
     rounding up in the code) that it's  a multiple of the frame-subsampling factor.
     Bear in mind that if the \b chunk-size is 90 and the \b frame-subsampling-factor
     is 3, then we're only evaluating 30 distinct output indexes for each chunk of
     90 frames (e.g. t=0, t=3 ... t=87).
  
    \subsection dnn3_scripts_context_basics_variable Variable chunk size
  
    Variable chunk size is something used in training that is only available in Kaldi version
    5.1 or later.  This is a mechanism to allow fairly large chunks while avoiding
    the loss of data due to files that are not exact multiples of the chunk size.
    Instead of specifying the chunk size as (say) 150, we might specify the chunk
    size as a comma-separated list like 150,120,90,75, and the commands that generate the
    training examples are allowed to create chunks of any of those sizes.  The
    first chunk size specified is referred to as the primary chunk size, and is
    "special" in that for any given utterance, we are allowed pick at most two of the
    non-primary chunk size; the remaining chunks must be of the primary chunk size.
    This restriction makes it easier to work out the optimal split of a file of
    a given length into chunks, and allows us to bias the chunk-generation to
    chunks of a certain length.
  
  
    \subsection dnn3_scripts_context_basics_minibatch Minibatch size
  
    The program nnet3-merge-egs merges individual training examples into
    minibatches containing many different examples (each original example
    gets a different 'n' index).  The \b minibatch-size is the desired
    size of minibatch, by which we mean the number of examples (frames or
    sequences) that we combine into one(for example, minibatch-size=128).
    When the chunk sizes
    are variable (and taking into account that the context may be different
    at the start/end of utterances if we set the \b extra-left-context-initial
    and \b extra-right-context-final), it's important to ensure that only
    ``similar'' examples are merged into minibatches; this prevents expensive
    recompilation from happening on every single minibatch.
  
    In Kaldi version
    5.1 and later, nnet3-merge-egs only merges together chunks of the same
    structure (i.e. the same chunk-size and left and right context).
    It keeps reading chunks from the input until it finds that
    for some structure of input, there are \b minibatch-size examples ready
    to merge into one.  In Kaldi versions prior to 5.1 we generally discarded
    the "odd-numbered" examples that couldn't be fit into a normal-sized
    minibatch, but this becomes problematic now that there are many different
    chunk-sizes (we'd discard too much data).
  
    \subsubsection dnn3_scripts_context_basics_minibatch_variable Variable minibatch size
  
    From Kaldi 5.1 and later,
    the --minibatch-size is a more general string that allows the user more
    control than just having a fixed minibatch size.  For example, you can specify --minibatch-size=64,128 and
    for each type of example it will try to accumulate batches of the
    largest specified size (128) and output
    them, until it reaches the end of the input; then it will output
    a minibatch of size 64 if there are >= 64 egs left.  Ranges are also
    supported, e.g. --minibatch-size=1:64 means to output minibatches of size 64
    until the end of the input, then output all remaining examples as a single
    minibatch.  You may also specify different rules for examples of different
    sizes (run nnet3-merge-egs without arguments for details of this); this can be useful
    to stay within GPU memory limits.
  
    \section dnn3_scripts_context_looped  Looped decoding
  
    Looped decoding in nnet3 is another feature that is new in Kaldi version 5.1.
    It is applicable to forward-recurrent neural networks such as RNNs and LSTMs
    (but not to BLSTMs).  It allows us to re-use hidden-state activations from
    previously-computed chunks.  This allows us to have effectively unlimited left
    context.  The reason why it's called ``looped decoding'' relates to the way
    it's implemented: we create a computation whose last statement is a 'goto'
    that jumps to somewhere in the middle, so effectively it has a loop like
    'while(1)'.  (Note: the computations have statements that request user input or
    provide output, so the loop doesn't cause the computation to run indefinitely when called;
    it will stop when an I/O operation is reached).  Looped computation is intended to solve two problems: wasteful
    computation, and latency.  Suppose we trained our LSTMs with 40 frames of left
    context and a chunk-size of 100.  Without looped computation, we'd probably
    want to decode with chunks of size about 100 and we'd left-pad the input with around 40
    frames.  But this takes about 40\% extra computation; and the chunk size of 1
    second would be a problem for latency/responsiveness in a real-time
    application.  With looped computation, we can choose any chunk size that's
    convenient, because the effective left context is infinite; and the chunk size
    doesn't affect the computed output any more.
  
    However, there is a slight problem with what we sketched out above.  In
    practice, we've found for LSTMs that decoding works best with about the same
    chunk sizes and context as we trained with.  That is, adding more context than
    we trained on is not helpful.  Our theory about why this happens is that
    as the context gets longer we reach parts of activation space that were unreachable
    before.  The maximum value of the cells \f$c_t\f$ in LSTMs rises linearly with
    the number of frames we've seen.  Following this theory, we made a modification
    to LSTMs that seems to fix the problem.  We scale the \f$c_t\f$ in the LSTM equations
    by a value slightly less than one in the recurrence (for example, like 0.9).
    This puts a bound on the maximum hidden activation activations and makes them
    increase less dramatically with increasing recurrence time.  It's specified
    as a configuration value in the LSTM components in the "xconfig" configuration files
    with the "decay-time" value, e.g. "decay-time=20".  This doesn't seem to
    degrade the Word Error Rates, and it removes the discrepancy between regular
    and looped decoding (i.e. it makes the networks tolerant to longer context than
    was seen in training).
  
    The script steps/nnet3/decode_looped.sh (only available from Kaldi version 5.1)
    takes only two chunk- or context-related configuration values:
    \b frames-per-chunk (which only affects the speed/latency tradeoff and not
    results), and \b extra-left-context-initial, which should be set to
    match the training condition (generally this will be zero, in up-to-date
    scripts).
  
  
    At the time of writing, we have not yet created a program similar to
    online2-wav-nnet3-latgen-faster that uses the looped decoder; that is
    on our TODO list (it's not inherently difficult).
  
  
   - Up: \ref dnn3
   - Previous: \ref dnn3_code_optimization
  
  */
  
  }
  }