// doc/transform.dox
  
  
  // Copyright 2009-2011 Microsoft Corporation
  
  // See ../../COPYING for clarification regarding multiple authors
  //
  // Licensed under the Apache License, Version 2.0 (the "License");
  // you may not use this file except in compliance with the License.
  // You may obtain a copy of the License at
  
  //  http://www.apache.org/licenses/LICENSE-2.0
  
  // THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  // KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
  // WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
  // MERCHANTABLITY OR NON-INFRINGEMENT.
  // See the Apache 2 License for the specific language governing permissions and
  // limitations under the License.
  
  namespace kaldi {
  
  /**
    \page transform Feature and model-space transforms in Kaldi
  
    \section transform_intro Introduction
  
    Kaldi code currently supports a number of feature and model-space transformations
    and projections.  Feature-space transforms and projections are treated in a consistent
    way by the tools (they are essientially just matrices), and the following sections
    relate to the commonalities:
     - \ref transform_apply
     - \ref transform_perspk
     - \ref transform_utt2spk
     - \ref transform_compose
     - \ref transform_weight
  
    Transforms, projections and other feature operations that are typically not speaker specific include:
     - \ref transform_lda
     - \ref transform_splice and \ref transform_delta
     - \ref transform_hlda
     - \ref transform_mllt
  
    Global transforms that are typically applied in a speaker adaptive way are:
      - \ref transform_cmllr_global
      - \ref transform_lvtln
      - \ref transform_et
      - \ref transform_cmvn
  
    We next discuss regression class trees and transforms that use them:
      - \ref transform_regtree
  
  
    \section transform_apply Applying global linear or affine feature transforms
  
    In the case of feature-space transforms and projections that are global,
    and not associated with classes (e.g. speech/silence or regression classes), we
    represent them as matrices.  A linear transform or
    projection is represented as a matrix by which we will left-multiply a feature vector,
    so the transformed feature is \f$ A x \f$.  An affine transform or projection
    is represented the same way, but we imagine a 1 has been appended to the
    feature vector, so the transformed feature is
    \f$ W \left[ \begin{array}{c} x \\ 1 \end{array} \right] \f$ where
     \f$ W = \left[ A ; b \right] \f$, with A and b being the linear transform
    and the constant offset.
    Note that this convention differs from some of the literature, where the 1 may appear as
    the first dimension rather than the last.
    Global transforms and projections are generally written
    as a type Matrix<BaseFloat> to a single file, and speaker or utterance-specific
    transforms or projections are stored in a table of such matrices (see \ref io_sec_tables)
    indexed by speaker-id or utterance-id.
  
    Transforms may be applied to features
    using the program transform-feats.  Its syntax is
  \verbatim
   transform-feats <transform> <input-feats> <output-feats>
  \endverbatim
    where <input-feats> is an rspecifier, <output-feats> is an wspecifier, and <transform>
    may be an rxfilename or an rspecifier (see \ref io_sec_specifiers and \ref io_sec_xfilename).
    The program will work out whether the transform
    is linear or affine based on whether or not the matrix's number of columns equals the
    feature dimension, or the feature dimension plus one.
    This program is typically used as part of a pipe.
    A typical example is:
  \verbatim
   feats="ark:splice-feats scp:data/train.scp ark:- |
            transform-feats $dir/0.mat ark:- ark:-|"
   some-program some-args "$feats" some-other-args ...
  \endverbatim
   Here, the file 0.mat contains a single matrix.  An example of applying
   speaker-specific transforms is:
  \verbatim
   feats="ark:add-deltas scp:data/train.scp ark:- |
     transform-feats --utt2spk=ark:data/train.utt2spk ark:$dir/0.trans ark:- ark:-|"
   some-program some-args "$feats" some-other-args ...
  \endverbatim
  A per-utterance example would be as above but removing the --utt2spk option.
  In this example, the archive file 0.trans would contain transforms (e.g. CMLLR transforms)
  indexed by speaker-id, and the file data/train.utt2spk would have
  lines of the form "utt-id spk-id" (see next section for more explanation).
  The program transform-feats does not care how the transformation matrix was
  estimated, it just applies it to the
  features.  After it has been through all the features it prints out the average
  per-frame log determinant.  This can be useful when comparing objective functions
  (this log determinant would have to be added to the per-frame likelihood printed
  out by programs like gmm-align, gmm-acc-stats, or gmm-decode-kaldi).  If the
  linear part A of the transformation (i.e. ignoring the offset term) is not square,
  then the program will instead print out the per-frame average of
  \f$ \frac{1}{2} \mathbf{logdet} (A A^T) \f$.  It refers to this as the pseudo-log-determinant.
  This is useful in checking convergence of MLLT estimation where the transformation matrix
  being applied is the MLLT matrix times an LDA matrix.
  
  \section transform_perspk Speaker-independent versus per-speaker versus per-utterance adaptation
  
  Programs that estimate transforms are generally set up to do a particular kind of
  adaptation, i.e. speaker-independent versus (speaker- or utterance-specific).  For example, LDA
  and MLLT/STC transforms are speaker-independent but fMLLR transforms are speaker- or
  utterance-specific.  Programs that estimate speaker- or utterance-specific transforms
  will work in per-utterance mode by default, but in per-speaker mode if the --spk2utt
  option is supplied (see below).
  
  One program that can accept either speaker-independent or speaker- or utterance-specific
  transforms is transform-feats.  This program detects whether the first argument (the transform)
  is an rxfilename (see \ref io_sec_xfilename)
  or an rspecifier (see \ref io_sec_specifiers).  If the former, it treats it as a speaker-independent
  transform (e.g. a file containing a single matrix).
  If the latter, there are two choices.  If no --utt2spk option is provided,
  it treats the transform as a table of matrices indexed by utterance id.  If an --utt2spk option is provided
  (utt2spk is a table of strings indexed by utterance that contains the string-valued speaker id),
  then the transforms are assumed to be indexed by speaker id, and the table
  provided to the --utt2spk option is used to map each utterance to a speaker id.
  
  \section transform_utt2spk Utterance-to-speaker and speaker-to-utterance maps
  
   At this point we give a general overview of the --utt2spk and --spk2utt options.
   These options are accepted by programs that deal with transformations; they are used when
   you are doing per-speaker (as opposed to per-utterance) adaptation.
   Typically programs that process already-created transforms will need the --utt2spk
   option and programs that create the transforms will need the --spk2utt option.
   A typical case is that there will be a file called some-directory/utt2spk
   that looks like:
  \verbatim
  spk1utt1  spk1
  spk1utt2  spk1
  spk2utt1  spk2
  spk2utt2  spk2
  ...
  \endverbatim
  where these strings are just examples, they stand for generic speaker and
  utterance identifiers; and there will be a file called some-directory/spk2utt that looks like:
  \verbatim
  spk1 spk1utt1 spk1utt2
  spk2 spk2utt1 spk2utt2
  ...
  \endverbatim
   and you will supply options that look like --utt2spk=ark:some-directory/utt2spk
   or --spk2utt=ark:some-directory/spk2utt.  The 'ark:' prefix is necessary because
   these files are given as rspecifiers by the Table code, and are interpreted as archives
   that contain strings (or vectors of strings, in the spk2utt case).  Note that
   the utt2spk archive is generally accessed in a random-access manner, so if you
   are processing subsets of data it is safe to provide the whole file, but the
   spk2utt archive is accessed in a sequential manner so if you are using subsets
   of data you would have to split up the spk2utt archive.
  
   Programs that accept the spk2utt option will normally iterate over the
   speaker-ids in the spk2utt file, and for each speaker-id they will iterate over
   the utterances for each speaker, accumulating statistics for each utterance.  Access to the feature
   files will then be in random-access mode, rather than the normal sequential
   access.  This requires some care to set up, because feature files are quite large
   and fully-processed features are normally read from an archive, which does not
   allow the most memory-efficient random access unless carefully set up.  To avoid memory
   bloat when accessing the feature files in this case, it may be advisable to
   ensure that all archives are sorted on utterance-id, that the utterances in the
   file given to the --spk2utt option appear in sorted order, and that the
   appropriate options are given on the rspecifiers that specify the feature input
   to such programs (e.g. "ark,s,cs:-" if it is the standard input).  See \ref io_sec_bloat
   for more discussion of this issue.
  
   \section transform_compose Composing transforms
  
   Another program that accepts generic transforms is the program compose-transforms.
   The general syntax is "compose-transforms a b c", and it performs the multiplication
   c = a b (although this involves a little more than matrix multiplication if a is affine).
   An example modified from a script is as follows:
  \verbatim
   feats="ark:splice-feats scp:data/train.scp ark:- |
           transform-feats
             \"ark:compose-transforms ark:1.trans 0.mat ark:- |\"
             ark:- ark:- |"
   some-program some-args "$feats" ...
  \endverbatim
   This example also illustrates using two levels of commands invoked from a program.
   Here, 0.mat is a global matrix (e.g. LDA) and 1.trans is a set of fMLLR/CMLLR matrices
   indexed by utterance id.   The program compose-transforms composes the transforms
   together.  The same features could be computed more simply,  but less efficiently, as follows:
  \verbatim
   feats="ark:splice-feats scp:data/train.scp ark:- |
           transform-feats 0.mat ark:- ark:- |
           transform-feats ark:1.trans ark:- ark:- |"
   ...
  \endverbatim
   In general, the transforms a and b that are the inputs to compose-transforms
   may be either speaker-independent transforms or speaker- or utterance-specific
   transforms.  If a is utterance-specific and b is speaker-specific then you have to supply
   the --utt2spk option.  However, the combination of a being speaker-specific and b being utterance-specific
   (which does not make much sense) is not supported.  The output of compose-transforms
   will be a table if either a or b are tables.  The three arguments a, b and c may all
   represent either tables or normal files (i.e. either {r,w}specifiers or {r,w}xfilenames),
   subject to consistency requirements.
  
   If a is an affine transform, in order to perform the composition correctly, compose-transforms
   needs to know whether b is affine or linear (it does not know this because it does not have access
   to the dimension of the features
   that are transformed by b).  This is controlled by the option --b-is-affine (bool, default false).
   If b is affine but you forget to set this option and a is affine, compose-transforms
   will treat b as a linear transform from dimension (the real input feature dimension) plus one,
   and will output a transform whose input dimension is (the real input feature dimension) plus two.  There
   is no way for "transform-feats" to interpret this when it is to be applied to features,
   so the error should become obvious as a dimension mismatch at this point.
  
  
  \section transform_weight Silence weighting when estimating transforms
  
  Eliminating silence frames can be helpful when estimating speaker adaptive
  transforms such as CMLLR.  This even appears to be true when using
  a multi-class approach with a regression tree (for which, see \ref transform_regtree).
  The way we implement this is by weighting down the posteriors associated with
  silence phones.  This takes place as a modification to the \ref hmm_post
  "state-level posteriors".  An extract of a bash shell script that does this
  is below (this script is discussed in more detail in \ref transform_cmllr_global):
  \verbatim
  ali-to-post ark:$srcdir/test.ali ark:- | \
    weight-silence-post 0.0 $silphones $model ark:- ark:- | \
    gmm-est-fmllr --fmllr-min-count=$mincount \
      --spk2utt=ark:data/test.spk2utt $model "$sifeats" \
     ark,o:- ark:$dir/test.fmllr 2>$dir/fmllr.log
  \endverbatim
  Here, the shell variable "silphones" would be set to a colon-separated
  list of the integer id's of the silence phones.
  
  \section transform_lda Linear Discriminant Analysis (LDA) transforms
  
  Kaldi supports LDA estimation via class LdaEstimate.  This class does not interact
  directly with any particular type of model; it needs to be initialized with the
  number of classes, and the accumulation function is declared as:
  \verbatim
  class LdaEstimate {
    ...
    void Accumulate(const VectorBase<BaseFloat> &data, int32 class_id,
                    BaseFloat weight=1.0);
  };
  \endverbatim
  The program acc-lda accumulates LDA statistics using the acoustic states (i.e. pdf-ids) as the
  classes.  It requires the transition model in order to map the alignments (expressed in terms
  of transition-ids) to pdf-ids.  However, it is not limited to a particular type of acoustic model.
  
  The program est-lda does the LDA estimation (it reads in the statistics from acc-lda).  The features you get from the transform will
  have unit variance, but not necessarily zero mean.  The program est-lda outputs the LDA transformation matrix,
  and using the option --write-full-matrix you can write out the full matrix without dimensionality
  reduction (its first rows will be equivalent to the LDA projection matrix).  This can be useful
  when using LDA as an initialization for HLDA.
  
  \section transform_splice Frame splicing
  
  Frame splicing (e.g. splicing nine consecutive frames together) is typically done
  to the raw MFCC features prior to LDA.  The program splice-feats does this.  A typical
  line from a script that uses this is the following:
  \verbatim
  feats="ark:splice-feats scp:data/train.scp ark:- |
          transform-feats $dir/0.mat ark:- ark:-|"
  \endverbatim
  and the "feats" variable would later be used as an rspecifier (c.f. \ref io_sec_specifiers)
  by some program that needs to read features.  In this example we don't specify the number of frames to splice
  together because we are using the defaults (--left-context=4, --right-context=4, or
  9 frames in total).
  
  \section transform_delta Delta feature computation
  
  Computation of delta features is done by the program add-deltas, which uses the
  function ComputeDeltas.  The delta feature computation has the same default setup
  as HTK's, i.e. to compute the first delta feature we multiply by the features
  by a sliding window of values [ -2, -1, 0, 1, 2 ], and then normalize by
  dividing by (2^2 + 1^2 + 0^2 + 1^2 + 2^2 = 10).  The second delta feature
  is computed by applying the same approach to the first delta feature.  The
  number of frames of context on each side is controlled by --delta-window (default: 2)
  and the number of delta features to add is controlled by --delta-order (default: 2).
  A typical script line that uses this is:
  \verbatim
  feats="ark:add-deltas --print-args=false scp:data/train.scp ark:- |"
  \endverbatim
  
  \section transform_hlda Heteroscedastic Linear Discriminant Analysis (HLDA)
  
   HLDA is a dimension-reducing linear feature projection, estimated using
   Maximum Likelihood, where "rejected" dimensions are modeled using a global
   mean and variance, and "accepted" dimensions are modeled with a particular
   model whose means and variances are estimated via Maximum Likelihood.
   The form of HLDA
   that is currently integrated with the tools is as implemented in
   HldaAccsDiagGmm.  It estimates HLDA for GMMs, using a relatively compact
   form of the statistics.  The classes correspond to the Gaussians in the model.
   Since it does not use a standard estimation method, we will explain the idea
   here.  Firstly, because of memory limitations we do not want to store
   the largest form of HLDA statistics which is mean and full-covariance statistics
   for each class.  We observe that if during the HLDA update phase we leave
   the variances fixed, then the problem of HLDA estimation reduces to MLLT
   (or global STC) estimation.  See "Semi-tied Covariance Matrices for Hidden
   Markov Models", by Mark Gales, IEEE Transactions on Speech and Audio Processing,
   vol. 7, 1999, pages 272-281, e.g. Equations (22) and (23).  The statistics
   that are \f$ \mathbf{G}^{(ri)} \f$ there, are also used here, but in the HLDA
   case they need to be defined
   slightly differently for the accepted and rejected dimensions.
   Suppose the original feature dimension is D and the
   reduced feature dimension is K.
   Let us forget the iteration superscript r, and use subscript j for state and
   m for Gaussian mixture.
   For accepted dimensions (\f$0 \leq i < K\f$), the statistics are:
   \f[
     \mathbf{G}^{(i)} = \sum_{t,j,m} \gamma_{jm}(t) \frac{1}{ \sigma^2_{jm}(i) } (\mu_{jm} - \mathbf{x}(t)) (\mu_{jm} - \mathbf{x}(t))^T
   \f]
   where \f$\mu_{jm} \in \Re^{K}\f$ is the Gaussian mean in the original D-dimensional space,
   and \f$\mathbf{x}(t)\f$ is the feature in the original K-dimensional space, but
   \f$\sigma^2_{jm}(i)\f$ is the i'th dimension of the variance within the the K-dimensional model.
  
   For rejected dimensions (\f$ K \leq d < D\f$), we use a unit variance Gaussian, and
   the statistics are as follows:
   \f[
     \mathbf{G}^{(i)} = \sum_{t,j,m} \gamma_{jm}(t)  (\mu - \mathbf{x}(t)) (\mu - \mathbf{x}(t))^T ,
   \f]
   where \f$\mu\f$ is the global feature mean in the K-dimensional space.  Once we have
   these statistics, HLDA estimation is the same as MLLT/STC estimation in dimension D.
   Note here that all the \f$\mathbf{G}\f$ statistics for rejected dimensions are the
   same, so in the code we only store statistics for K+1 rather than D dimensions.
  
   Also, it is convenient for the program that accumulates the statistics to only have
   access to the K-dimensional model, so during HLDA accumulation we accumulate
   statistics sufficient to estimate the K-dimensional means \f$\mu_{jm}\f$, and insead of
   G we accumulate the following statistics: for accepted dimensions (\f$0 \leq i < K\f$),
   \f[
     \mathbf{S}^{(i)} = \sum_{t,j,m} \gamma_{jm}(t) \frac{1}{ \sigma^2_{jm}(i) }  \mathbf{x}(t) \mathbf{x}(t)^T
   \f]
   and for rejected dimensions \f$K \leq i < D\f$
   \f[
     \mathbf{S}^{(i)} = \sum_{t,j,m} \gamma_{jm}(t)  \mathbf{x}(t) \mathbf{x}(t)^T ,
   \f]
   and of course we only need to store one of these (e.g. for i = K) because they are all the same.
   Then in the update time we can compute the G statistics for \f$0 \leq i < K\f$ as:
   \f[
    \mathbf{G}^{(i)} = \mathbf{S}^{(i)}  - \sum_{j,m} \gamma_{jm}  \mu_{jm} \mu_{jm}^T ,
   \f]
   and for \f$K \leq i < D\f$,
   \f[
    \mathbf{G}^{(i)} = \mathbf{S}^{(i)} - \beta \mu \mu^T,
   \f]
   where \f$ \beta = \sum_{j,m} \gamma_{jm} \f$ is the total count and \f$\mu = \frac{1}{\beta} \sum_{j,m} \mu_{j,m}\f$
   is the global feature mean.   After computing the transform from the G statistics using the same computation as MLLT,
   we output the transform, and we also use the first K rows of the transform to project the means
   into dimension K and write out the transformed model.
  
   The computation described here is fairly slow; it is \f$ O(K^3) \f$ on each frame,
   and K is fairly large (e.g. 117).  This is the price we pay for compact statistics;
   if we stored full mean and variance statistics, the per-frame computation would be \f$O(K^2)\f$.
   To speed it up, we have an optional parameter ("speedup" in the code) which
   selects a random subset of frames to actually compute the HLDA statistics on.
   For instance, if speedup=0.1 we would only accumulate HLDA statistics on 1/10 of
   the frames.  If this option is activated, we need to store two separate
   versions of the sufficient statistics for the means.  One version of the mean
   statistics, accumulated on the subset, is only used in the HLDA computation, and
   corresponds to the quantities \f$\gamma_{jm}\f$ and \f$\mu_{jm}\f$ in the formulas above.
   The other version of the mean statistics is accumulated on all the training data
   and is used to write out the transformed model.
  
   The overall HLDA estimation process is as follows (see rm_recipe_2/scripts/train_tri2j.sh):
      - First initialize it with LDA (we store both the reduced dimension matrix
        and the full matrix).
      - Start model-building and training process.  On certain (non-consecutive)
        iterations where we have decided to do the HLDA update, do the following:
        - Accumulate HLDA statistics (S, plus statistics for the full-dimensional means).
          The program that accumulates these (gmm-acc-hlda) needs the model, the un-transformed features,
          and the current transform (which it needs to transform the features in order
          to compute Gaussian posteriors)
        - Update the HLDA transform.  The program that does this (gmm-est-hlda)
          needs the model; the statistics; and the previous full (square)
          transformation matrix which it needs to start the optimization and to correctly
          report auxiliary function changes.  It outputs the new transform (both full and
          reduced dimension), and the model with newly estimated and transformed means.
  
   \section transform_mllt Global Semi-tied Covariance (STC) / Maximum Likelihood Linear Transform (MLLT) estimation
  
    Global STC/MLLT is a square feature-transformation matrix.  For more details,
    see "Semi-tied Covariance Matrices for Hidden Markov Models", by Mark Gales,
    IEEE Transactions on Speech and Audio Processing, vol. 7, 1999, pages 272-281.
    Viewing it as a feature-space transform, the objective function is the average
    per-frame log-likelihood of the transformed features given the model, plus the
    log determinant of the transform.  The means of the model are also rotated by
    transform in the update phase.  The sufficient statistics are the following,
    for \f$ 0 \leq i < D \f$ where D is the feature dimension:
   \f[
     \mathbf{G}^{(i)} = \sum_{t,j,m} \gamma_{jm}(t) \frac{1}{ \sigma^2_{jm}(i) } (\mu_{jm} - \mathbf{x}(t)) (\mu_{jm} - \mathbf{x}(t))^T
   \f]
    See the reference, Equations (22) and (23) for the update equations.  These are
    basically a simplified form of the diagonal row-by-row Constrained MLLR/fMLLR update
    equations, where the first-order term of the quadratic equation disappears.  Note that
    our implementation differs from that reference by using a column of the inverse of the matrix
    rather than the cofactor, since multiplying by the determinant does not make a difference to the
    result and could potentially cause problems with floating-point underflow or overflow.
  
    We describe the overall process as if we are doing MLLT on top of LDA features,
    but it is also applicable on top of traditional delta features.  See the script
    rm_recipe_2/steps/train_tri2f for an example.  The process is as follows:
  
    - Estimate the LDA transformation matrix (we only need the first rows of this, not the full matrix).
      Call this matrix \f$\mathbf{M}\f$.
    - Start a normal model building process, always using features transformed with \f$\mathbf{M}\f$.
      At certain selected iterations (where we will update the MLLT matrix), we do the following:
        - Accumulate MLLT statistics in the current fully-transformed space
          (i.e., on top of features transformed with \f$\mathbf{M}\f$).  For efficiency we do this using
          a subset of the training data.
        - Do the MLLT update; let this produce a square matrix \f$\mathbf{T}\f$.
        - Transform the model means by setting \f$ \mu_{jm} \leftarrow \mathbf{T} \mu_{jm} \f$.
        - Update the current transform by setting \f$ \mathbf{M} \leftarrow \mathbf{T} \mathbf{M} \f$
  
    The programs involved in MLLT estimation are gmm-acc-mllt and est-mllt.  We also need the
    programs gmm-transform-means (to transform the Gaussian means using \f$\mathbf{T}\f$), and
    compose-transforms (to do the multiplication \f$\mathbf{M} \leftarrow \mathbf{T} \mathbf{M} \f$).
  
  
   \section transform_cmllr_global Global CMLLR/fMLLR transforms
  
    Constrained Maximum Likelihood Linear Regression (CMLLR), also known as feature-space MLLR (fMLLR),
    is an affine feature transform of the form \f$ \mathbf{x} \rightarrow \mathbf{A} \mathbf{x}  + \mathbf{b} \f$,
    which we write in the form  \f$ \mathbf{x} \rightarrow \mathbf{W} \mathbf{x}^+ \f$, where
    \f$\mathbf{x}^+ = \left[\begin{array}{c} \mathbf{x} \\ 1 \end{array} \right]\f$ is the feature with
    a 1 appended.  Note that this differs from some of the literature where the 1 comes first.
  
    For a review paper that explains CMLLR and the estimation techniques we use, see
   "Maximum likelihood linear transformations for HMM-based speech recognition" by Mark Gales,
    Computer Speech and Language Vol. 12, pages 75-98.
  
    The sufficient statistics we store are:
    \f[ \mathbf{K} = \sum_{t,j,m} \gamma_{j,m}(t) \Sigma_{jm}^{-1} \mu_{jm} \mathbf{x}(t)^+ \f]
    where \f$\Sigma_{jm}^{-1}\f$ is the inverse covariance matrix,
    and for \f$0 \leq i < D \f$ where D is the feature dimension,
    \f[ \mathbf{G}^{(i)} = \sum_{t,j,m} \gamma_{j,m}(t) \frac{1}{\sigma^2_{j,m}(i)} \mathbf{x}(t)^+  \left.\mathbf{x}(t)^+\right.^T \f]
  
    Our estimation scheme is the standard one, see Appendix B of the reference (in particular section B.1,
    "Direct method over rows").  We differ by using a column of the inverse in place of the cofactor row,
    i.e. ignoring the factor of the determinant, as it does not affect the result and causes danger of
    numerical underflow or overflow.
  
    Estimation of global Constrained MLLR (CMLLR) transforms is done by the
    class FmllrDiagGmmAccs,
    and by the program gmm-est-fmllr (also see gmm-est-fmllr-gpost).  The syntax
    of gmm-est-fmllr is:
  \verbatim
  gmm-est-fmllr [options] <model-in> <feature-rspecifier> \
     <post-rspecifier> <transform-wspecifier>
  \endverbatim
   The "<post-rspecifier>" item corresponds to posteriors at the transition-id level
   (see \ref hmm_post).  The program writes out a table of CMLLR transforms
    indexed by utterance by default, or if the --spk2utt option is given, indexed by speaker.
  
    Below is a simplified extract of a script
    (rm_recipe_2/steps/decode_tri_fmllr.sh) that estimates and uses CMLLR transforms based
    on alignments from a previous, unadapted decoding.  The previous decoding is assumed
    to be with the same model (otherwise we would have to convert the alignments with
   the program "convert-ali").
  \verbatim
  ...
  silphones=48 # colon-separated list with one phone-id in it.
  mincount=500 # min-count to estimate an fMLLR transform
  sifeats="ark:add-deltas --print-args=false scp:data/test.scp ark:- |"
  
  # The next comand computes the fMLLR transforms.
  ali-to-post ark:$srcdir/test.ali ark:- | \
    weight-silence-post 0.0 $silphones $model ark:- ark:- | \
    gmm-est-fmllr --fmllr-min-count=$mincount \
      --spk2utt=ark:data/test.spk2utt $model "$sifeats" \
     ark,o:- ark:$dir/test.fmllr 2>$dir/fmllr.log
  
  feats="ark:add-deltas --print-args=false scp:data/test.scp ark:- |
    transform-feats --utt2spk=ark:data/test.utt2spk ark:$dir/test.fmllr
         ark:- ark:- |"
  
  # The next command decodes the data.
  gmm-decode-faster --beam=30.0 --acoustic-scale=0.08333 \
    --word-symbol-table=data/words.txt $model $graphdir/HCLG.fst \
   "$feats" ark,t:$dir/test.tra ark,t:$dir/test.ali 2>$dir/decode.log
  \endverbatim
  
   \section transform_lvtln Linear VTLN (LVTLN)
  
   In recent years, there have been a number of papers that describe
   implementations of Vocal Tract Length Normalization (VTLN) that
   work out a linear feature transform corresponding to each VTLN
   warp factor.  See, for example, ``Using VTLN for broadcast news transcription'',
   by D. Y. Kim, S. Umesh, M. J. F. Gales, T. Hain and P. C. Woodland, ICSLP 2004.
  
   We implement a method in this general category using the class LinearVtln, and programs
   such as gmm-init-lvtln, gmm-train-lvtln-special, and gmm-est-lvtln-trans.
   The LinearVtln object essentially stores a set of linear feature transforms,
   one for each warp factor.  Let these linear feature transform matrices
   be
     \f[\mathbf{A}^{(i)},  0\leq i < N,  \f]
   where for instance we might have \f$N\f$=31, corresponding to 31 different warp
   factors. We will describe below how we obtain these matrices below.
   The way the speaker-specific transform is estimated is as follows.
   First, we require some kind of model and a corresponding alignment.  In the
   example scripts we do this either with a small monophone model, or with
   a full triphone model.  From this model and alignment, and using the original,
   unwarped features, we compute the conventional statistics for estimating
   CMLLR.  When computing the LVTLN transform, what we do is take each matrix
   \f$\mathbf{A}^{(i)}\f$, and compute the  offset vector \f$\mathbf{b}\f$ that
   maximizes the CMLLR auxiliary function for the transform
    \f$\mathbf{W} = \left[  \mathbf{A}^{(i)} \, ; \, \mathbf{b} \right]\f$.
   This value of \f$\mathbf{W}\f$ that gave the best auxiliary function value
   (i.e. maximizing over i) becomes the transform for that speaker.  Since we
   are estimating a mean offset here,
   we are essentially combining a kind of model-based cepstral mean normalization
   (or alternatively an offset-only form of CMLLR) with VTLN warping implemented
   as a linear transform.  This avoids us having to implement mean normalization
   as a separate step.
  
   We next describe how we estimate the matrices \f$\mathbf{A}^{(i)}\f$.  We
   don't do this in the same way as described in the referenced paper; our method
   is simpler (and easier to justify).  Here we describe our computation for a
   particular warp factor; in the current scripts we have 31 distinct warp
   factors ranging from 0.85, 0.86, ..., 1.15.
   We take a subset of feature data (e.g. several tens of utterances),
   and for this subset we compute both the original and transformed features,
   where the transformed features are computed using a conventional VLTN computation
   (see \ref feat_vtln).
   Call the original and transformed features \f$\mathbf{x}(t)\f$ and \f$\mathbf{y}(t)\f$ respectively,
   where \f$t\f$ will range over the frames of the selected utterances.
   We compute the affine transform that maps \f$\mathbf{x}\f$ to \f$\mathbf{y}\f$ in a least-squares
   sense, i.e. if \f$\mathbf{y}' = \mathbf{A} \mathbf{x} + \mathbf{b}\f$,
   we compute \f$\mathbf{A}\f$ and \f$\mathbf{b}\f$ that minimizes the sum-of-squares
   difference \f$\sum_t (\mathbf{y}'(t) - \mathbf{y}(t) )^T (\mathbf{y}'(t) - \mathbf{y}(t) )\f$.
   Then we normalize the diagonal variance as follows: we compute the
   variance of the original features as \f$\mathbf{\Sigma}^{(x)}\f$ and of the linearly transformed
   features as \f$\mathbf{\Sigma}^{(y')}\f$, and for each dimension index d we multiply the
   d'th row of \f$\mathbf{A}\f$ by
    \f$\sqrt{ \frac{\mathbf{\Sigma}^{(x)}_{d,d}}{\mathbf{\Sigma}^{(y')}_{d,d}}}\f$.
   The resulting matrix will become \f$\mathbf{A}^{(i)}\f$ for some value of i.
  
   The command-line tools support the option to ignore the log determinant term
   when evaluating which of the transform matrices to use (e.g., you can set
   --logdet-scale=0.0).  Under certain circumstances this appears to improve
   results; ignoring the log determinant it always makes the distribution of warp
   factors more bimodal because the log determinant is never positive and is zero
   for a warp factor of 1.0, so the log determinant essentially acts as a penalty
   on warp factors that are far away from 1.  However, for certain types of
   features (in particular, features derived from LDA), ignoring the log
   determinant makes results a lot worse and leads to very odd distributions of
   warp factors, so our example scripts always use the log-determinant.  This is
   anyway the "right" thing to do.
  
   The internal C++ code supports accumluating statistics for Maximum Likelihood
   re-estimation of the transform matrices \f$\mathbf{A}^{(i)}\f$.  Our expectation
   was that this would improve results.  However, it led to a degradation in
   performance so we do not include example scripts for doing this.
  
  
   \section transform_et Exponential Transform (ET)
  
   The Exponential Transform (ET) is another approach to computing a VTLN-like
   transform, but unlike Linear VTLN we completely sever the connection
   to frequency warping, and learn it in a data-driven way.  For normal
   training data, we find that it does learn something very similar to
   conventional VTLN.
  
   ET is a transform of the form:
  \f[
    \mathbf{W}_s = \mathbf{D}_s \exp ( t_s \mathbf{A} ) \mathbf{B} ,
  \f]
   where exp is the matrix exponential function, defined via a Taylor
   series in \f$\mathbf{A}\f$ that is the same as the Taylor series for
   the scalar exponential function.  Quantities with a subscript "s"
   are speaker-specific; other quantities (i.e. \f$\mathbf{A}\f$ and
   \f$\mathbf{B}\f$) are global and shared across all speakers.
  
   The most important factor in this equation is the middle one,
   with the exponential function in it.
   The factor \f$\mathbf{D}_s\f$ gives us the ability to combine
   model-based mean and optionally variance normalization (i.e. offset-only
   or diagonal-only CMLLR)
   with this technique, and the factor \f$\mathbf{B}\f$ allows the transform to include
   MLLT (a.k.a. global STC), and is also a byproduct of the process
   of renormalizing the \f$t_s\f$ quantities on each iteration of
   re-estimation.  The dimensions of these quantities are as follows,
    where D is the feature dimension:
  \f[
     \mathbf{D}_s \in \Re^{D \times (D+1)}, \ t_s \in \Re, \  \mathbf{A} \in \Re^{(D+1)\times(D+1)}, \ \mathbf{B} \in \Re^{(D+1)\times (D+1)}  .
  \f]
   Note that if \f$\mathbf{D}_s\f$ were a completely unconstrained CMLLR matrix,
   there would be no point to this technique as the other quantities in the
   equation would add no degrees of freedom.  The tools support three kinds of
   constraints on \f$\mathbf{D}_s\f$: it may be of the form
   \f$[ {\mathbf I} \, \;\, {\mathbf 0} ]\f$ (no adaptation), or
   \f$[ {\mathbf I} \, \;\, {\mathbf m} ]\f$ (offset only), or
   \f$[ {\mathrm{diag}}( {\mathbf d} ) \, \;\, {\mathbf m} ]\f$ (diagonal CMLLR);
   this is controlled by the --normalize-type options to the command-line tools.
   The last rows of \f$\mathbf{A}\f$ and \f$\mathbf{B}\f$ are
   fixed at particular values (these rows are involved in propagating the
   last vector element with value 1.0, which is appended to the feature in order
   to express an affine transform as a matrix).  The last row
   of \f$\mathbf{A}\f$ is fixed at zero and the last row
   of \f$\mathbf{B}\f$ is fixed at \f$[ 0\ 0\ 0 \ \ldots\ 0 \ 1]\f$.
  
   The speaker-specific quantity \f$t_s\f$ may be interpreted
   very loosely as the log of the speaker-specific warp factor.
   The basic intuition behind the use of the exponential function is that
   if we were to warp by a factor f and then a factor g,
   this should be the same as warping by the combined factor
   fg.  Let l = log(f) and m = log(g).  Then we achieve this
   property via the identity
    \f[ \exp( l \mathbf{A} ) \exp( m \mathbf{A}) = \exp( (l+m) \mathbf{A} ) . \f]
  
   The ET computation for a particular speaker is as follows; this assumes we
   are given \f$\mathbf{A}\f$ and \f$\mathbf{B}\f$.  We accumulate conventional
   CMLLR sufficient statistics for the speaker.  In the update phase we iteratively optimize
   \f$t_s\f$ and \f$\mathbf{D}_s\f$ to maximize the auxiliary function.
    The update for \f$t_s\f$ is an iterative procedure based on Newton's method;
   the update for \f$\mathbf{D}_s\f$ is based on the conventional CMLLR
   update,  specialized for the diagonal or offset-only case, depending on
   the exact constraints we are putting \f$\mathbf{D}_s\f$.
  
   The overall training-time computation is as follows:
    - First, initialize \f$\mathbf{B}\f$ to the identity and \f$\mathbf{A}\f$ to
      a random matrix with zero final row.
  
   Then, starting with some known model, start an iterative E-M process.
   On each iteration, we first estimate the speaker-specific parameters
   \f$t_s\f$ and \f$\mathbf{D}_s\f$, and compute the transforms \f$\mathbf{W}_s\f$
   that result from them.  Then we choose to update either \f$\mathbf{A}\f$, or
   \f$\mathbf{B}\f$, or the model.
     - If updating \f$\mathbf{A}\f$, we do this given fixed values of
       \f$t_s\f$ and \f$\mathbf{D}_s\f$.  The update is not guaranteed to
       converge, but converges rapidly in practice; it's based on a
       quadratic "weak-sense auxiliary function"
       where the quadratic term is obtained using a first-order truncation
       of the Taylor series expansion of the matrix exponential function.
       After updating \f$\mathbf{A}\f$, we modify \f$\mathbf{B}\f$ in order
       to renormalize the \f$t_s\f$ to zero; this involves premultiplying
       \f$\mathbf{B}\f$  by \f$\exp(t \mathbf{A})\f$, where t is the average
       value of \f$t_s\f$.
  
     - If updating \f$\mathbf{B}\f$, this is also done using fixed values of
       \f$t_s\f$ and \f$\mathbf{D}_s\f$, and the update is similar to MLLT
       (a.k.a. global STC).
       For purposes of the accumulation and update, we imagine we are estimating
       an MLLT matrix just to the left of \f$\mathbf{A}\f$, i.e. some matrix
       \f$\mathbf{C} \in \Re^{D\times D}\f$; let us define
       \f$\mathbf{C}^+ = \left[ \begin{array}{cc} \mathbf{C} & 0 \\ 0 & 1 \end{array} \right]\f$.
       The transform will be
       \f$\mathbf{W}_s = \mathbf{D}_s \mathbf{C}^+ \exp ( t_s \mathbf{A} ) \mathbf{B}\f$.
       Conceptually, while estimating \f$\mathbf{C}\f$ we view \f$\mathbf{D}_s\f$ as
       a model-space transform creating speaker-specific models, which this is only possible
       due to the diagonal structure of \f$\mathbf{D}_s\f$; and we view
       \f$\exp ( t_s \mathbf{A} ) \mathbf{B}\f$ as a feature-space transform (i.e.
       as part of the features).  After estimating \f$\mathbf{C}\f$, we will use the identity
  \f[
     \mathbf{C}^+ \exp ( t_s \mathbf{A} ) =  \exp ( t_s \mathbf{C}^+ \mathbf{A}  \left.\mathbf{C}^+\right.^{-1} ) \mathbf{C}^+
  \f]
    so the update becomes:
  \f[
          \mathbf{A} \leftarrow \mathbf{C}^+ \mathbf{A}  \left.\mathbf{C}^+\right.^{-1} , \ \ \mathbf{B} \leftarrow \mathbf{C}^+ \mathbf{B} .
  \f]
       At this point we need to transform the model means with the matrix
       \f$\mathbf{C}\f$.  The reader might question how this interacts with the
       fact that for estimating \f$\mathbf{C}\f$, we viewed the quantity
       \f$\mathbf{D}_s\f$ as a model-space transform.  If \f$\mathbf{D}_s\f$ only
       contains a mean offset, we can still prove that the auxiliary function
       would increase, except we would have to change the offsets appropriately
       (this is not necessary to do explicitly, as we will re-estimate them on
       the next iteration anyway).  However, if \f$\mathbf{D}_s\f$ has non-unit
       diagonal (i.e. is diagonal not offset CMLLR),  this re-estimation process
       is not guaranteed to improve the likelihood; the tools will print a warning
       in this case.  In order to avoid encountering this case, our scripts
       train in a mode where \f$\mathbf{D}_s\f$ is an offset-only transform; but
       in test time we allow \f$\mathbf{D}_s\f$ to be a diagonal CMLLR transform, which seems
       to give slightly better results than the offset-only case.
  
     - Updating the model is straightforward; it just involves training on the adapted
       features.
  
    Important programs related to the use of exponential transforms are as follows:
     - gmm-init-et initializes the exponential transform object (that contains A and B) and writes it to disk; the initialization of A is random.
     - gmm-est-et estimates the exponential transforms for a set of speakers; it reads the exponential transform object, the model, the features and \ref hmm_gpost "Gaussian-level posteriors", and it writes out the transforms \f$\mathbf{W}_s\f$ and optionally the "warp factors" \f$t_s\f$.
     - gmm-et-acc-a accumulates statistics for updating \f$\mathbf{A}\f$, and and gmm-et-est-a does the corresponding update.
     - gmm-et-acc-b accumulates statistics for updating \f$\mathbf{B}\f$, and and gmm-et-est-b does the corresponding update.
  
  \section transform_cmvn Cepstral mean and variance normalization
  
  Cepstral mean and variance normalization consists of normalizing the mean
  and variance of the raw cepstra, usually to give zero-mean, unit-variance
  cepstra, either on a per-utterance or per-speaker basis.  We provide code
  to support this, and some example scripts, but we do not particularly recommend its use.
  In general we prefer model-based approaches to mean and variance normalization;
  e.g., our code for \ref transform_lvtln also learns a mean offset and the code
  for \ref transform_et does a diagonal CMLLR transform that has the same power as
  cepstral mean and variance normalization (except usually applied to the fully
  expanded features).  For very fast operation, it is possible to apply these
  approaches using a very tiny model with a phone-based language model, and some of
  our example scripts demonstrate this.  There is also the capability in the
  feature extraction code to subtract the mean on a per-utterance basis (the
  --subtract-mean option to compute-mfcc-feats and compute-plp-feats).
  
  In order to support per-utterance and per-speaker mean and variance normalization
  we provide the programs compute-cmvn-stats and apply-cmvn.  The program
  compute-cmvn-stats will, by default, compute the sufficient statistics for mean
  and variance normalization, as a matrix (the format is not very important; see
  the code for details), and will write out a table of these statistics indexed by
  utterance-id.  If it is given the --spk2utt option, it will write out the
  statistics on a per-speaker basis instead (warning: before using this option,
  read \ref io_sec_bloat, as this option causes the input features to be read in
  random-access mode).  The program "apply-cmvn" reads in features and cepstral
  mean and variance statistics; the statistics are expected to be indexed per
  utterance by default, or per speaker if the --utt2spk option is applied.  It
  writes out the features after mean and variance normalization.  These programs,
  despite the names, do not care whether the features in question consist of
  cepstra or anything else; it simply regards them as matrices.  Of course, the
  features supplied to compute-cmvn-stats and apply-cmvn must have the same
  dimension.
  
  We note that it would probably be more consistent with the overall design of the
  feature transformation code, to supply a version of compute-cmvn-stats that would
  write write out the mean and variance normalizing transforms as generic affine
  transforms (in the same format as CMLLR transforms), so that they could be
  applied by the program transform-feats, and composed as needed with other
  transforms using compose-transforms.  If needed we may supply such a program, but
  because we don't regard mean and variance normalization as an important part of
  any recipes, we have not done so yet.
  
  
  \section transform_regtree Building regression trees for adaptation
  
    Kaldi supports regression-tree MLLR and CMLLR (also known as fMLLR).  For
    an overview of regression trees, see "The generation and use of regression class trees for MLLR
    adaptation" by M. J. F. Gales, CUED technical report, 1996.
  
  
  
  
  */
  
  }