// doc/cudamatrix.dox
  
  
  // Copyright 2012  Karel Vesely
  //           2015  Johns Hopkins University (author: Daniel Povey)
  
  // See ../../COPYING for clarification regarding multiple authors
  //
  // Licensed under the Apache License, Version 2.0 (the "License");
  // you may not use this file except in compliance with the License.
  // You may obtain a copy of the License at
  
  //  http://www.apache.org/licenses/LICENSE-2.0
  
  // THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  // KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
  // WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
  // MERCHANTABLITY OR NON-INFRINGEMENT.
  // See the Apache 2 License for the specific language governing permissions and
  // limitations under the License.
  
  namespace kaldi {
  /**
    \page cudamatrix The CUDA Matrix library
  
   The CUDA matrix library provides access to GPU-based matrix operations with
   an interface similar to \ref matrix.
   The general principle is that if you want to be able to run a particular part
   of the computation the GPU, you would declare the relevant quantities as
   type CuMatrix or CuVector instead of Matrix or Vector.  Then, if you have
   configured Kaldi to use the GPU and if the Kaldi program you are running
   has initialized access to the GPU, those operations will run on the GPU.
   Otherwise, they will run on the CPU.  CuMatrix and CuVector quantities
   store their contents in GPU memory space, if you have configured for GPU
   and your program has initialized the GPU device.
  
   You can't mix CuMatrix and CuVector with Matrix and Vector in matrix operations,
   because they live in different memory spaces, but you can copy from one to
   the other.  Kaldi does not try to automatically decide which operations are
   best done on GPU: it is all under the control of the programmer.
  
  
    \subsection cudamatrix_configuration Configuring the CUDA matrix library
  
   If the <tt>configure</tt> script sees that the NVidia compilation tool <tt>nvcc</tt> is on
   the path when it is run, it assumes you want to compile for GPU, and will define
   <tt>HAVE_CUDA=1</tt> and set other Makefile variables to enable GPU compilation.
   You can disable this if you don't want it by calling <tt>configure</tt> with
   <tt>--use-cuda=no</tt>.  If the script doesn't find the location where you installed
   the CUDA toolkit but you want to use it, you can use an option like
   <tt>--cudatk-dir=/opt/cuda-4.2</tt>.
   If you want to tell whether Kaldi has been configured to use CUDA, you can
   grep for <tt>CUDATKDIR</tt> in <tt>kaldi.mk</tt>; if the string appears, then it has
   been configured to use CUDA.  In scripts, you can check the return status of
   the program <tt>cuda-compiled</tt>: it returns success (0) if you compiled for CUDA.
  
   You can also tell from the logs whether a program is using the GPU.  If it is using
   the GPU, you'll see lines like this near the top of the program's output:
  \verbatim
  LOG (nnet-train-simple:IsComputeExclusive():cu-device.cc:229) CUDA setup operating under Compute Exclusive Mode.
  LOG (nnet-train-simple:FinalizeActiveGpu():cu-device.cc:194) The active GPU is [1]: Tesla K10.G2.8GB  \
      free:3519M, used:64M, total:3583M, free/total:0.982121 version 3.0
  \endverbatim
   In addition to configuring at the Makefile to use CUDA, if any individual
   program wants to use GPU operations it needs to have code like the following:
  \verbatim
  #if HAVE_CUDA==1
      CuDevice::Instantiate().SelectGpuId(use_gpu);
  #endif
  \endverbatim
  where <tt>use_gpu</tt> is a string, typically a command-line option, that can take
  the following values:
  
    - <tt>"yes"</tt>: use the GPU (or crash if one is not available).
    - <tt>"no"</tt> don't use the GPU.
    - <tt>"optional"</tt> use the GPU if the machine it's running on has GPUs attached.
    - <tt>"wait"</tt>: like <tt>"yes"</tt> but if the GPUs are running other processes,
         the program will wait indefinitely until one becomes free.
  
  If a program doesn't take the <tt>--use-gpu</tt> command line option, that generally
  means that it hasn't been programmed to support the use of GPU operations, even if the code
  it runs contains the CuVector and CuMatrix types.  Usually we only run specific tasks
  on the GPU- mainly neural net training.
  
  \subsection cudamatrix_modes GPU compute modes
  
   NVidia GPUs (which is the only kind Kaldi supports) have various "compute modes":
   "default", "process exclusive", "thread exclusive".  This controls whether or not the
   GPU is configured to run multiple processes at the same time.  Kaldi is intended
   to be run in "exclusive mode"; whether it's process exclusive or thread exclusive doesn't
   matter.  You can find out what mode your GPU is running in as follows:
  \verbatim
  # nvidia-smi  --query | grep 'Compute Mode'
      Compute Mode                    : Exclusive_Process
  \endverbatim
   You can set the correct mode by typing <tt>nvidia-smi -c 3</tt>.  You might want to
   do this in a startup script so it happens each time you reboot.
  
   Rather than calling the malloc and free functions that NVidia provides, Kaldi
   does caching of previously released memory so that we don't have to incur the
   overhead of NVidia's malloc.  This was done because at one point we were running
   in Amazon's cloud and found that NVidia's malloc was very slow.  This was probably
   caused by the virtualization, and we're not sure whether that problem still exists.
   Anyway, the memory caching can cause a problem if for some reason you run using
   the default (non-exclusive) compute mode, because it can cause allocation
   failures.  You can disable it at the code level by calling
   <tt>CuDevice::Instantiate().DisableCaching()</tt>, if needed.
  
  
  
  */
  
  }