// doc/queue.dox
  
  
  // Copyright 2009-2011 Microsoft Corporation
  //                2013 Johns Hopkins University (author: Daniel Povey)
  
  // See ../../COPYING for clarification regarding multiple authors
  //
  // Licensed under the Apache License, Version 2.0 (the "License");
  // you may not use this file except in compliance with the License.
  // You may obtain a copy of the License at
  
  //  http://www.apache.org/licenses/LICENSE-2.0
  
  // THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  // KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
  // WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
  // MERCHANTABLITY OR NON-INFRINGEMENT.
  // See the Apache 2 License for the specific language governing permissions and
  // limitations under the License.
  
  namespace kaldi {
  
  /**
    \page queue Parallelization in Kaldi
  
   \section parallelization_intro Introduction
  
   Kaldi is designed to work best with software such as Sun GridEngine or other software
   that works on a similar principle; and if multiple machines are to work together
   in a cluster then they need access to a shared file system such as one based on NFS.
   However, Kaldi can easily be configured to run on a single machine.
  
   If you look at a top-level example script like <tt>egs/wsj/s5/run.sh</tt>, you'll see commands like
  \verbatim
   steps/train_sat.sh  --cmd "$train_cmd" \
     4200 40000 data/train_si284 data/lang exp/tri3b_ali_si284 exp/tri4a
  \endverbatim
   At the top of the <tt>run.sh</tt> script you'll see it sourcing a file called <tt>cmd.sh</tt>:
  \verbatim
  . ./cmd.sh
  \endverbatim
  and in <tt>cmd.sh</tt> you'll see the following variable being set:
  \verbatim
  export train_cmd="queue.pl -l arch=*64"
  \endverbatim
  You'll change this variable if you don't have GridEngine or if your queue is configured
  differently from CLSP\@JHU.   To run everything locally on a single machine you can
  set <tt>export train_cmd=run.pl</tt>.
  
  In <tt>steps/train_sat.sh</tt> the varible <tt>cmd</tt> is set to the argument
  to the <tt>--cmd</tt> option, i.e. to <tt>queue.pl -l arch=*64</tt> in this case,
  and in the script you'll see commands like the following:
  \verbatim
        $cmd JOB=1:$nj $dir/log/fmllr.$x.JOB.log \
          ali-to-post "ark:gunzip -c $dir/ali.JOB.gz|" ark:-  \| \
          weight-silence-post $silence_weight $silphonelist $dir/$x.mdl ark:- ark:- \| \
          gmm-est-fmllr --fmllr-update-type=$fmllr_update_type \
          --spk2utt=ark:$sdata/JOB/spk2utt $dir/$x.mdl \
          "$feats" ark:- ark:$dir/tmp_trans.JOB || exit 1;
  \endverbatim
  What's going on is that the command <tt>$cmd</tt> (e.g. <tt>queue.pl</tt> or <tt>run.pl</tt>)
  is being executed; it is responsible for spawning the jobs and waiting until they are done,
  and returning with nonzero status if something went wrong.  The basic usage of these
  commands (and there are others called <tt>slurm.pl</tt> and <tt>ssh.pl</tt>) is like this:
  \verbatim
   queue.pl <options> <log-file> <command>
  \endverbatim
  and the simplest possible example of using one of these scripts is:
  \verbatim
   run.pl foo.log echo hello world
  \endverbatim
  (we're using <tt>run.pl</tt> for the example because it will run on any system, it doesn't
  require GridEngine).  It's possible to run a one-dimensional array of jobs, and an example
  is:
  \verbatim
   run.pl JOB=1:10 foo.JOB.log echo hello world number JOB
  \endverbatim
  and these programs will replace any instance of JOB in the command line with a number
  within that range, so make sure that your working directory doesn't contain the string
  JOB, or bad things may happen.  You can even submit jobs with pipes and redirection by
  suitable use of quoting or escaping:
  \verbatim
   run.pl JOB=1:10 foo.JOB.log echo "hello world number JOB" \| head -n 1 \> output.JOB
  \endverbatim
  In this case, the command that actually gets executed will be something like:
  \verbatim
  echo "hello world number JOB" | head -n 1 > output.JOB
  \endverbatim
  If you want to see what's actually getting executed, you can look in a file like
  <tt>foo.1.log</tt>, where you'll see the following:
  \verbatim
  # echo "hello world number 1" | head -n 1 > output.1
  # Started at Sat Jan  3 17:44:20 PST 2015
  #
  # Accounting: time=0 threads=1
  # Ended (code 0) at Sat Jan  3 17:44:20 PST 2015, elapsed time 0 seconds
  \endverbatim
  
  
  \section parallelization_common Common interface of parallelization tools
  
  In this section we discuss the commonalities of the parallelization tools.  They
  are designed to all be interchangeable on the command line, so that a script
  that is tested for one of these parallelization tools will work for any; you can
  switch over to using another by setting the <tt>$cmd</tt> variable to a
  different value.
  
  The basic usage of these tools is, as we said,
  \verbatim
   queue.pl <options> <log-file> <command>
  \endverbatim
  and what we are about to say also holds for <tt>run.pl</tt>, <tt>ssh.pl</tt> and <tt>slurm.pl</tt>.
  
  <tt><options></tt> may include some or all of the following:
  <ul>
    <li> A job range specifier (e.g. JOB=1:10).  The name is uppercase by convention only, and may include underscores.
         The starting index must be 1 or more; this is a GridEngine limitation.
    <li> Anything that looks as if it would be accepted by GridEngine as an option to <tt>qsub</tt>.
         For example, <tt>-l arch=*64*</tt>, or <tt>-l mem_free=6G,ram_free=6G</tt>, or <tt>-pe smp 6</tt>.
         For compatibility, scripts other than <tt>queue.pl</tt> will ignore such options.
    <li> New-style options like <tt>--mem 10G</tt> (see below).
  </ul>
  
  <tt><log-file></tt> is just a filename, which for array jobs must contain the identifier of
   the array (e.g. <tt>exp/foo/log/process_data.JOB.log</tt>).
  
  <tt><command></tt> can basically be anything, including symbols that would
  be interpreted by the shell, but of course <tt>queue.pl</tt> can't process
  something if it gets interpreted by <tt>bash</tt> first.   For instance, this is WRONG:
  \verbatim
   queue.pl test.log  echo foo | awk 's/f/F/';
  \endverbatim
  because <tt>queue.pl</tt> won't even see the arguments after the pipe symbol, but will get its
  standard output piped into the awk command.  Instead you should write
  \verbatim
   queue.pl test.log  echo foo \| awk 's/f/F/';
  \endverbatim
  You need to escape or quote the pipe symbol, and also things like ";" and ">".
  If one of the arguments in the <tt><command></tt> contains a space, then
  queue.pl will assume you quoted it for a reason, and will quote it for you when
  it gets passed to bash.  It quotes using single quotes by default, but if the
  string itself contains single quotes then it uses double quotes instead.  This
  usually does what we want.  The <tt>PATH</tt> variable from the shell that
  you executed </tt>queue.pl</tt> from will be passed through to the scripts
  that get executed, and just to be certain you get everything you need,
  the file <tt>./path.sh</tt> will also be sourced.  The commands will be executed
  with bash.
  
  \subsection parallelization_common_new New-style options (unified interface)
  
  When we originally wrote Kaldi, we made the example scripts pass in options like
  <tt>-l ram_free=6G,mem_free=6G</tt> to <tt>queue.pl</tt>, when we needed to
  specify things like memory requirements.  Because the scripts
  like <tt>steps/train_sat.sh</tt> can't make assumptions about how GridEngine is
  configured or whether we are using GridEngine at all, such options had to be
  passed in from the very outer level of the scripts, which is awkward.  We have more recently
  defined a "new-style interface" to the parallelization scripts, such that they
  all accept the following types of options (examples shown):
  \verbatim
     --config conf/queue_mod.conf
     --mem 10G
     --num-threads 6
     --max-jobs-run 10
     --gpu 1
  \endverbatim
  The config file specifies how to convert new-style options into a form that
  GridEngine (or your grid software of choice) can interpret.  Currently
  only <tt>queue.pl</tt> actually interprets these options; the other scripts
  ignore them.  Our plan is to gradually modify the scripts in <tt>steps/</tt> to
  make use of the new-style options, where necessary, and to use <tt>queue.pl</tt>
  for other varieties of grid software through use of the config files (where possible).
  
  <tt>queue.pl</tt> will read <tt>conf/queue.conf</tt> if it exists; otherwise it
  will default to a particular config file that we define in the code.  The config
  file specifies how to convert the "new-style" options into options that
  GridEngine or similar software can interpret.  The following example show the
  behavior that the default config file specifies:
  <table>
   <tr> <th> New-style option </th>  <th>  Converted form (for GridEngine) </th> <th> Comment </th> </tr>
   <tr> <td> <tt>--mem 10G</tt></td>  <td> <tt>-l mem_free=10G,ram_free=10G</tt></td> <td></td> </tr>
   <tr> <td> <tt>--max-jobs-run 10</tt></td> <td> <tt>-tc 10</tt></td> <td> (We use this for jobs that cause too much I/O). </td></tr>
   <tr> <td> <tt>--num-threads 6</tt></td> <td> <tt>-pe smp 6</tt></td> <td></td>(general case) </tr>
   <tr> <td> <tt>--num-threads 1</tt></td> <td> <tt>(no extra options)</tt></td> <td>(special case)</td> </tr>
   <tr> <td> <tt>--gpu 1</tt></td> <td> <tt>-q g.q -l gpu=1 </tt></td> <td>(general case)</td> </tr>
   <tr> <td> <tt>--gpu 0</tt></td> <td> (no extra options)  </td> <td>(special case for gpu=0)</td> </tr>
  </table>
  It's also possible to add extra options with this general format, i.e. options
  that look like
  <tt>--foo-bar</tt> and take one argument.  The default configuration tabulated above works for the CLSP grid
  but may not work everywhere, because GridEngine is very configurable.  Thefore you may
  have to create a config file <tt>conf/queue.conf</tt> and edit it to work with your grid.
  The following configuration file is the one that <tt>queue.pl</tt> defaults to if <tt>conf/queue.conf</tt>
  does not exist and the <tt>--config</tt> option is not specified, and may be used as a starting
  point for your own config file:
  \verbatim
  # Default configuration
  command qsub -v PATH -cwd -S /bin/bash -j y -l arch=*64*
  option mem=* -l mem_free=$0,ram_free=$0
  option mem=0          # Do not add anything to qsub_opts
  option num_threads=* -pe smp $0
  option num_threads=1  # Do not add anything to qsub_opts
  option max_jobs_run=* -tc $0
  default gpu=0
  option gpu=0
  option gpu=* -l gpu=$0 -q g.q
  \endverbatim
  The line beginning with
  <tt>command</tt> specifies the unchanging part of the command line, and you can
  modify this to get it to use grid software other than GridEngine, or to specify
  options that you always want.  The lines beginning with
  <tt>option</tt> specify how to transform the input options such as <tt>--mem</tt>.
  Lines beginning with something like <tt>"option mem=*"</tt> handle the general case
  (the <tt>$0</tt> gets replaced with the actual argument to the option), while
  lines like <tt>"option gpu=0"</tt> allow you to specify special behavior for special
  cases of the argument, so in this case the option <tt>--gpu 0</tt> is configured
  to produce no extra options to <tt>qsub</tt> at all.  The line <tt>"default gpu=0"</tt> specifies
  that if you don't give the <tt>--gpu</tt> option at all, <tt>queue.pl</tt> should act like
  you specified <tt>--gpu 0</tt>.  In this particular configuration we could
  have omitted the line <tt>default gpu=0</tt>, because in any case the
  effect is to produce no extra command line options. We previously had it
  configured with a line: <tt>"option gpu=0 -q all.q"</tt>, so there was a time when the line
  <tt>"default gpu=0"</tt> used to make a difference.
  
  (Note: most of the time if you are configuring GridEngine yourself you will want a queue.conf
  that is the same as the one above missing the "-q g.q" option).
  
  The mapping from what the config-file specifies to what appears on the command-line
  of qsub sometimes has to be tweaked slightly in the perl code: for instance, we made it
  so that the <tt>--max-jobs-run</tt> option is ignored for non-array jobs.
  
   \subsection parallelization_common_new_example Example of configuring grid software with new-style options
  
  
  We'd like to give an example of how the config file can be used in a real
  situation.  We had a problem where, due to a bug in an outdated version of the
  CUDA toolkit that we had installed on the grid, some of our neural net training
  runs were crashing on our K20 GPU cards  but not the K10s.  We created a config
  file  <tt>conf/no_k20.conf</tt> which was as the configuration file above (search in
  the text above for <tt># Default configuration</tt>, but with the following
  lines added:
  \verbatim
  default allow_k20=true
  option allow_k20=true
  option allow_k20=false -l 'hostname=!g01*&!g02*&!b06*'
  \endverbatim
  We then set the relevant <tt>$cmd</tt> variable to the value
  <tt>queue.pl -config conf/no_k20.conf --allow-k20 false</tt>.
  Note that a simpler way to have done this would have been
  to simply edit the <tt>command</tt> line in the config file to read
  \verbatim
  command qsub -v PATH -cwd -S /bin/bash -j y -l arch=*64* -l 'hostname=!g01*&!g02*&!b06*'
  \endverbatim
  and if we had done that, it would not have been necessary to add <tt>--allow-k20 false</tt>.
  
  
  \section parallelization_specific Parallelization using specific scripts
  
  In this section we explain the things that are specific to individual
  parallelization scripts.
  
   \subsection parallelization_specific_queue Parallelization using queue.pl
  
   <tt>queue.pl</tt> is the normal, recommended way to parallelize.  It was originally
   designed for use with GridEngine, but now that we have introduced the "new-style options"
   we believe it can be configured for use with other parallelization engines, such as Tork
   or slurm.  If you develop config files that work for such engines, please
   contact the maintainers of Kaldi.  It may also be necessary to make further changes to <tt>queue.pl</tt>
   to properly support other engines, since some parts of the command line are currently
   constructed in <tt>queue.pl</tt> in a way that is not configurable from the command line:
   e.g., adding <tt>-o foo/bar/q/jobname.log</tt> to direct output from <tt>qsub</tt> itself
   to a separate log-file; and for array jobs, adding options like <tt>-t 1:40</tt> to the command
   line.  The scripts that we ask <tt>qsub</tt> to run also make use of the variable
   <tt>$SGE_TASK_ID</tt>, which SGE sets to the job index for array jobs.  Our plan is to extend
   the config-file mechanism as necessary to accommodate whatever changes are needed to support
   other grid software, within reason.
  
   Since we have explained the behavior of <tt>queue.pl</tt> at length above, we aren't going
   to provide many further details in this section, but please see below the section
   \ref parallelization_gridengine.
  
  \subsection parallelization_specific_run Parallelization using run.pl
  
   <tt>run.pl</tt> is a fall-back option in case the user
   does not have GridEngine installed.  This script is very simple; it runs all
   the jobs you request on the local machine, and it does so in parallel if you
   use a job range specifier like <tt>JOB=1:10</tt>.  in parallel on the local
   machine.  It doesn't try to keep track of how many CPUs are
   available or how much memory your machine has.  Therefore if you
   use <tt>run.pl</tt> to run scripts that were designed to run
   with <tt>queue.pl</tt> on a larger grid, you may end up exhausting the memory
   of your machine or overloading it with jobs.  We recommend to study the script
   you are running, and being particularly careful with decoding scripts that run
   in the background (with <tt>&</tt>) and with scripts that use a large number of
   jobs, e.g. <tt>--nj 50</tt>.  Generally speaking you can reduce the value of the <tt>--nj</tt>
   option without affecting the outcome, but there are some situations where
   the <tt>--nj</tt> options given to multiple scripts must match, or
   a later stage will crash.
  
   <tt>run.pl</tt> will ignore all options given to it except for the job-range specifier.
  
  \subsection parallelization_specific_ssh Parallelization using ssh.pl
  
   <tt>ssh.pl</tt> is a poor man's <tt>queue.pl</tt>, for use in case
   you have a small cluster of several machines but don't want the trouble of setting
   up GridEngine.  Like <tt>run.pl</tt>, it doesn't attempt to keep track of
   CPUs or memory; it works like <tt>run.pl</tt> except that it distributes the
   jobs across multiple machines.
   You have to create a file <tt>.queue/machines</tt>
   (where <tt>.queue</tt> is a subdirectory of the directory you are running the script from),
   where each line contains the name of a machine.  It needs to be possible to ssh to each
   of these machines without a password, i.e. you have to set up your ssh keys.
  
  
   \subsection parallelization_specific_slurm Parallelization using slurm.pl
  
   <tt>slurm.pl</tt> was written to accomodate the <tt>slurm</tt> grid management tool,
   which operates on similar principles to GridEngine.  It has not been tested very
   recently.  Probably it is now possible to set up <tt>queue.pl</tt> to use <tt>slurm</tt>
   using a suitable configuration file, which would make <tt>slurm.pl</tt> unnecessary.
  
  
  \section parallelization_gridengine Setting up GridEngine for use with Kaldi
  
   Sun GridEngine (SGE) is the open-source grid management tool that we (the
   maintainers of Kaldi) have most experience with.  Oracle now maintains SGE and
   has started calling it Oracle GridEngine.  The version that is in use at CLSP\@JHU is
   6.2u5; SGE is old and fairly stable so the precise version
   number is not too critical.  There are various open-source alternatives to SGE
   and various forks of it, but our instructions here relate to the main-line version
   which is currently maintained by Oracle.
  
   In this section we explain how to install and configure GridEngine on an
   arbitrary cluster.  If you have a cluster in Amazon's EC2 cloud and you want
   something that can take care of spinning up new nodes, you might want to look
   at MIT's StarCluster project, although we (the maintainers of Kaldi) have also
   created a project called "kluster" on Sourceforge that provides some scripts
   and documentation for the same purpose.  StarCluster was not very stable at the
   time we developed kluster, but we believe it's improved since then.
  
  
    \subsection parallelization_gridengine_installing Installing GridEngine
  
    To start with, you probably want to get a basic installation of GridEngine
    working.  In GridEngine, the queue management software runs on the "master",
    and a different set of programs runs on all the nodes.  The master can also be
    a node in the queue.  There's also a concept of a "shadow master" which is
    like a backup for the master, in case the master dies, but we won't address
    that here (probably it's just a question of installing the <tt>gridengine-master</tt>
    package on another node and setting the master to be another node, but we're not sure).
  
    Last time we checked, installing GridEngine from source was a huge pain.  Your
    life will be much easier if your distribution of Linux has a GridEngine
    package in its repositories, and choose your distribution wisely because not
    all distributions have such a package.  We'll discuss how to do this with
    Debian, because that's what we're most experienced with.
  
    To install GridEngine on the master, you'll run (on your chosen master node):
  \verbatim
    sudo apt-get install gridengine-master gridengine-client
  \endverbatim
   Select "yes" for automatic configuration.
   It will ask you for the "cell name", which you can leave as "default", and it
   will ask for the name of the "master", which you should set to the hostname of
   your chosen master.  Typically this should be the fully qualified domain name (FQDN)
   of the master, but I believe anything that resolves via hostname lookup to the
   master node should work.  Note that GridEngine is sometimes picky about about
   hostname lookup and reverse DNS lookup matching up, and GridEngine problems can
   sometimes be traced to this.  Also be aware that doing "apt-get remove" of these
   packages and reinstalling them won't give you a blank slate because Debian
   sometimes remembers your selections; this can be a pain.
  
   It will make your life easier if you add yourself as manager, so do:
  \verbatim
   sudo qconf -am <your-user-id>
  \endverbatim
   Here "am" means add manager; "dm" would mean delete manager and "sm" would mean
   show all managers.  To see the available options, do <tt>qconf -help</tt>.
  
   To install GridEngine on the normal nodes, you'll run
  \verbatim
    sudo apt-get install gridengine-client gridengine-exec
  \endverbatim
   The "cell name" should be left as "default", and the "master" should be the name of
   the master node that you previously installed.
   You can run this on the master too if the master is to run jobs also.
  
   Typing <tt>qstat</tt> and <tt>qhost -q</tt> will let you know whether things are working.
   The following is what it looks like when things are working fine (we tested this
   in the Google cloud):
  \verbatim
  dpovey_gmail_com@instance-1:~$ qstat
  dpovey_gmail_com@instance-1:~$ qhost -q
  HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS
  -------------------------------------------------------------------------------
  global                  -               -     -       -       -       -       -
  instance-1.c.analytical-rig-638.internal lx26-amd64      1  0.07    3.6G  133.9M     0.0     0.0
  \endverbatim
   We don't have a fully working setup yet, we still need to configure it; we're just checking
   that the client can reach the master.  At this point, any errors likely relate to
   DNS lookup, reverse DNS lookup, or your /etc/hostname or /etc/hosts files; GridEngine
   doesn't like it when these things are inconsistent.  If you need to change the name of
   the master from what you told the installer, you may be able to do so by editing the file
  \verbatim
  /var/lib/gridengine/default/common/act_qmaster
  \endverbatim
   (at least, this is where it's located in Debian Wheezy).
  
    \subsection parallelization_gridengine_configuring Configuring GridEngine
  
   First let's make sure that a queue is defined.  GridEngine doesn't define any queues by
   default.  We'll set up a queue called <tt>all.q</tt>.  Make sure the shell variable <tt>EDITOR</tt>
   is set to your favorite shell (e.g. <tt>vim</tt> or <tt>emacs</tt>), and type as follows; and this should
   work from master or client.
  \verbatim
   qconf -aq
  \endverbatim
    This will bring up an editor.  Edit the line
  \verbatim
  qname               template
  \endverbatim
  so it says
  \verbatim
  qname               all.q
  \endverbatim
  Also change the field <tt>shell</tt> from <tt>/bin/csh</tt>
  to <tt>/bin/bash</tt>; this is a better default, although it shouldn't affect
  Kaldi.  Quitting the editor will save the changes,  although if you made syntax
  errors, <tt>qconf</tt> will reject your edits.  Later on we'll make more
  changes to this queue by typing <tt>qconf -mq all.q</tt>.
  
   GridEngine stores some global configuration values, not connected with any queue,
   which can be viewed with <tt>qconf -sconf</tt>.  We'll edit them using <tt>qconf -mconf</tt>.
   There is a line that reads
  \verbatim
  administrator_mail           root
  \endverbatim
  and if you have sending emails working from your machine (i.e. you can
  type <tt>mail foobar\@gmail.com</tt> and the mail gets delivered), then you can
  change <tt>root</tt> to an email address where you want to receive
  notifications if things go wrong.  Be advised that due to anti-spam measures,
  sending emails from the cloud is painful from EC2 and close to impossible
  from Google's cloud offering, so it may be best just to leave this field the
  way it is and make do without email notifications.
  You may also want to run <tt>qconf -msconf</tt> and edit it so that
  the schedule_interval is:
  \verbatim
  schedule_interval                 0:0:5
  \endverbatim (the default is <tt>00:00:15</tt>), which will
  give a slightly faster turnaround time for submitting jobs.
  
  GridEngine has the concept of "resources" which can be requested or specified by
  your jobs, and these can be viewed using <tt>qconf -sc</tt>.  Modify them
  using <tt>qconf -mc</tt>.  Modify the <tt>mem_free</tt> line to change the default
  memory requirement from 0 to 1G, and to make it consumable, i.e.:
  \verbatim
  #name               shortcut    type        relop requestable consumable default  urgency
  #------------------------------------------------------------------------------------------
  <snip>
  mem_free            mf         MEMORY      <=    YES         YES         1G        0
  \endverbatim
   and also add the following two new lines; it doesn't matter where in the file you add them.
  \verbatim
  #name               shortcut    type        relop requestable consumable default  urgency
  #------------------------------------------------------------------------------------------
  <snip>
  gpu                 g           INT         <=    YES         YES        0        10000
  ram_free            ram_free    MEMORY      <=    YES         JOB        1G       0
  \endverbatim
   You'll only need the "gpu" field if you add GPUs to your grid; the ram_free is a field
   that we find useful in managing the memory of the machines, as the inbuilt field
   <tt>mem_free</tt> doesn't seem to work quite right for our purposes.  Later on
   when we add hosts to the grid, we'll use the command <tt>qconf -me <some-hostname></tt> to
   edit the <tt>complex_values</tt> field to read something like:
  \verbatim
   complex_values        ram_free=112G,gpu=2
  \endverbatim
   (for a machine with 112G of physical memory and 2 GPUs).  If we want to submit
   a job that needs 10G of memory, we'll specify <tt>-l mem_free=10G,ram_free=10G</tt> as an
   option to <tt>qsub</tt>; the <tt>mem_free</tt> requirement makes sure the machine has that much free
   memory at the time the job starts, and the <tt>ram_free</tt> requirement makes sure we
   don't submit a lot of jobs requiring a lot of memory, all to the same host.
   We tried, as an alternative to adding the <tt>ram_free</tt> resource,
   using <tt>qconf -mc</tt> to edit the <tt>consumable</tt> field of the inbuilt <tt>mem_free</tt> resource to say
   <tt>YES</tt>, to make GridEngine keep track of memory requests; but this did not
   seem to work as desired.  Note that
   both <tt>ram_free</tt> and <tt>gpu</tt> are names that we chose ourselves; they have no
   special meaning to GridEngine, while some of the inbuilt resources such as <tt>mem_free</tt>
   do have special meanings.  The string <tt>JOB</tt> in the <tt>consumable</tt> entry for
   <tt>ram_free</tt> means that the <tt>ram_free</tt> resource is specified per job rather than
   per thread; this is more convenient for Kaldi scripts.
  
   We are not saying anything here about the queue "g.q" which is required for GPUs
   in the default queue.pl behavior.  That was introduced at Hopkins in order to avoid
   the situation where GPUs are blocked because all CPU slots are used.  It's really not
   necessary in general; the easiest way to get things working without it is to create a file
   <tt>conf/queue.conf</tt> that's the same as the default one (search above) except
   without the "-q g.q" option.
  
   Next you have to have to add the parallel environment called <tt>smp</tt> to GridEngine.
   This is a kind of tradition in GridEngine setups, but it's not built-in.  It's a
   simple parallel environment where GridEngine doesn't really do anything, it just reserves you
   a certain number of slots, so if you do <tt>qsub -pe smp 10 <your-script></tt> you will
   get 10 CPU slots reserved; this can be useful for multi-threaded or multi-process jobs.
   Do <tt>qconf -ap smp</tt>, and edit the <tt>slots</tt> field to say 9999, so it reads:
  \verbatim
  pe_name            smp
  slots              9999
  ...
  \endverbatim
   Then do <tt>qconf -mq all.q</tt>, and edit the <tt>pe_list</tt> field by adding <tt>smp</tt>, so
   it reads:
  \verbatim
  pe_list               make smp
  \endverbatim
   This enables the <tt>smp</tt> parallelization method in the queue <tt>all.q</tt>.
  
  
   \subsection parallelization_gridengine_configuring_advanced Configuring GridEngine (advanced)
  
   In this section we just want to make a note of some things that might be helpful, but which
   aren't necessary just to get things running.
   In the CLSP cluster, we edited the <tt>prolog</tt> field in <tt>qconf -mq all.q</tt>
   so that it says
  \verbatim
  prolog                /var/lib/gridengine/default/common/prolog.sh
  \endverbatim
   (the default was <tt>NONE</tt>),
   and the script <tt>/var/lib/gridengine/default/common/prolog.sh</tt>,
   which we copied to that location on each individual node in the cluster,
   reads as follows.  Its only purpose is to wait a short time if the job script can't be
   accessed, to give NFS some time to sync in case the scripts were written very
   recently and haven't yet propagated across the grid:
  \verbatim
  #!/bin/bash
  
  function test_ok {
    if [ ! -z "$JOB_SCRIPT" ] && [ "$JOB_SCRIPT" != QLOGIN ] && [ "$JOB_SCRIPT" != QRLOGIN ]; then
      if [ ! -f "$JOB_SCRIPT" ]; then
         echo "$0: warning: no such file $JOB_SCRIPT, will wait" 1>&2
         return 1;
      fi
    fi
    if [ ! -z "$SGE_STDERR_PATH" ]; then
      if [ ! -d "`dirname $SGE_STDERR_PATH`" ]; then
        echo "$0: warning: no such directory $JOB_SCRIPT, will wait." 1>&2
        return 1;
      fi
    fi
    return 0;
  }
  
  if ! test_ok; then
    sleep 2;
    if ! test_ok; then
       sleep 4;
       if ! test_ok; then
          sleep 8;
       fi
    fi
  fi
  
  exit 0;
  \endverbatim
  This script waits at most 14 seconds, which is enough because we configured <tt>acdirmax=8</tt>
  in our NFS options as the maximum wait before refreshing a cached directory (see \ref parallelization_grid_stable_nfs below).
  
  We also edited the queue with <tt>qconf -mq all.q</tt> to change
  <tt>rerun</tt> from FALSE to TRUE, i.e. to say:
  \verbatim
  rerun                 TRUE
  \endverbatim
  This means that when jobs fail, they get in a status that shows up in the output of
  <tt>qstat</tt> as <tt>Eqw</tt>, with the <tt>E</tt> indicating error, and you can ask the
  queue to reschedule them by clearing the error status with <tt>qmod -cj <numeric-job-id></tt>
  (or if you don't want to rerun them, you can delete them with <tt>qmod -dj <numeric-job-id></tt>).
  Setting the queue to allow reruns can avoid the hassle of rerunning scripts from the
  start when things break due to NFS problems.
  
  Something else we did in the CLSP queue is to edit the following fields, which by default read:
  \verbatim
  rlogin_daemon                /usr/sbin/sshd -i
  rlogin_command               /usr/bin/ssh
  qlogin_daemon                /usr/sbin/sshd -i
  qlogin_command               /usr/share/gridengine/qlogin-wrapper
  rsh_daemon                   /usr/sbin/sshd -i
  rsh_command                  /usr/bin/ssh
  \endverbatim
  to read instead:
  \verbatim
  qlogin_command               builtin
  qlogin_daemon                builtin
  rlogin_command               builtin
  rlogin_daemon                builtin
  rsh_command                  builtin
  rsh_daemon                   builtin
  \endverbatim
  This was to solve a problem whose nature we can no longer recall, but it's something you might want to try it if
  commands like <tt>qlogin</tt> and <tt>qrsh</tt> don't work.
  
  \subsection parallelization_gridengine_configuring_adding Configuring GridEngine (adding nodes)
  
   In this section we address what you do when you add nodes to the queue.
   As mentioned above, you can install GridEngine on nodes by doing
  \verbatim
    sudo apt-get install gridengine-client gridengine-exec
  \endverbatim
   and you need to specify <tt>default</tt> as the cluster name, and the name of your master
   node as the master (probably using the FQDN of the master is safest here, but if you are on
   a local network, just the last part of the name may also work).
  
   But that doesn't mean your machine is fully in the queue.  GridEngine has separate notions
   of hosts being administrative hosts, execution hosts and submit hosts.  All your machines
   should be all three.  You can view which machines have these three roles using the commands
   <tt>qconf -sh</tt>, <tt>qconf -sel</tt>, and <tt>qconf -ss</tt> respectively.   You can
   add your machine as an administrative or submit host with the commands:
  \verbatim
   qconf -ah <your-fqdn>
   qconf -as <your-fqdn>
  \endverbatim
   and you can add your host as an execution host with the command
  \verbatim
   qconf -ae <your-fqdn>
  \endverbatim
   which brings up an editor; you can put in the ram_free and possibly GPU fields in here, e.g.
  \verbatim
  complex_values    ram_free=112G,gpu=1
  \endverbatim
  You'll notice is a slight asymmetry between the commands <tt>qconf -sh</tt>
  and <tt>qconf -ss</tt> on the one hand, and <tt>qconf -sel</tt> on the other.
  The <tt>"l"</tt> in the latter command means show the list.   The difference is that
  administrative and submit host lists are just lists of hosts, whereas
  qconf stores a bunch of information about the execution hosts, so it views it as a different
  type of data structure.
  You can view the information about a particular host with <tt>qconf -se <some-hostname></tt>,
  add a new host with <tt>qconf -ae <some-hostname</tt>, and
  modify with <tt>qconf -me <some-hostname></tt>.  This is a general pattern in GridEngine:
  for things like queues that have a bunch of information in them, you can show the full list
  by typing a command ending in "l" like <tt>qconf -sql</tt>, and the corresponding "add"
  (<tt>"a"</tt>) and "modify" (<tt>"m"</tt>) commands accept arguments.
  
  It's not enough to tell GridEngine that a node is an execution host; you have to also add it to the queue,
  and tell the queue how many slots to allocate for that node.  First figure out how many CPUs (or virtual CPUs)
  your machine has, by doing:
  \verbatim
  grep proc /proc/cpuinfo | wc -l
  \endverbatim
  Suppose this is 48.  You can choose a number a little smaller than this, say 40, and use that for the
  number of slots.  Edit the queue using <tt>qconf -mq all.q</tt>, add your machine
  to the hostlist, and set the number of slots.  It should look like this:
  \verbatim
  qname                 all.q
  hostlist              gridnode1.research.acme.com,gridnode2.research.acme.com
  <snip>
  slots                 30,[gridnode1.research.acme.com=48],[gridnode1.research.acme.com=48]
  <snip>
  \endverbatim
  In the <tt>slots</tt> field, the <tt>30</tt> at the beginning is a default value; for any
  nodes with that number of slots you can save yourself some time and avoid adding the node's
  name to the <tt>slots</tt> field.
  There is an alternative way to set up the <tt>hostlist</tt> field.  GridEngine has the concept of host groups,
  so you could do <tt>qconf -ahgrp \@allhosts</tt> to add a group of hosts, and edit it using
  <tt>qconf -mhgrp \@allhosts</tt> to add your new nodes.  The configuration of
  <tt>all.q</tt> could then just read:
  \verbatim
  hostlist            @allhosts
  \endverbatim
  It's your choice.  For simple queues, it's probably fine to just put the host list in the <tt>all.q</tt>
  configuration.
  
  A useful command to list the hosts that GridEngine knows about, and what queues they
  are in, is <tt>qhost -q</tt>.  For example:
  \verbatim
  # qhost -q
  HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS
  -------------------------------------------------------------------------------
  global                  -               -     -       -       -       -       -
  a01.clsp.jhu.edu        lx26-amd64     24 12.46  126.2G   11.3G   86.6G  213.7M
     all.q                BIP   0/6/20
  a02.clsp.jhu.edu        lx26-amd64     24 16.84  126.2G   12.4G   51.3G  164.5M
     all.q                BIP   0/18/20
  <snip>
  \endverbatim
  If you see the letter <tt>"E"</tt> in the place where the example above shows <tt>"BIP"</tt>,
  it means the node is in the error state.  Other letters you don't want to see in that position are
  <tt>"a"</tt> for alarm (a generic indicator of badness) and <tt>"u"</tt> for unreachable.
  <tt>"d"</tt> means a node has been disabled by an administrator.
  Nodes sometimes get in the error (<tt>"E"</tt>) state when GridEngine had
  trouble running a job, which is often due to NFS or automount problems.
  You can clear the error by doing something like
  \verbatim
  qmod -c all.q@a01
  \endverbatim
  but of course if the node has serious problems, it might be wise to fix them first.
  It's sometimes also useful to enable and disable nodes in the queue by doing
  \verbatim
  qmod -d all.q@a01
  \endverbatim
  to disable a node, and <tt>qmod -e all.q\@a01</tt> to enable it again.
  
  A common symptom of GridEngine problems is jobs waiting when you think nodes
  are free.  The easiest way to debug this is to look for the job-id in the output
  of <tt>qstat</tt>, and then to do <tt>qstat -j <job-id></tt> and look for
  the reasons why the job is not running.
  
  You can view all jobs from all users by running
  \verbatim
  qstat -u '*'
  \endverbatim
  
  \section parallelization_grid_stable Keeping your grid stable
  
  In this section we have some general notes on how to ensure stability in
  a compute cluster of the kind useful for Kaldi.
  
  \subsection parallelization_grid_stable_oom Keeping your grid stable (OOM)
  
  One of the major causes of crashes in compute clusters is memory exhaustion.
  The default OOM-killer in Linux is not very good, so if you exhaust memory, it
  may end up killing an important system process, which tends to cause
  hard-to-diagnose instability.  Even if nothing is killed, <tt>malloc()</tt> may
  start failing when called from processes on the system; and very few programs
  deal with this gracefully.  In the CLSP grid we wrote our own version of an OOM
  killer, which we run as root, and we wrote the corresponding init scripts.  When
  our OOM killer detects memory overload, it kills the largest process of whichever
  non-system user is using the most memory.  This is usually the right thing to
  do.  These scripts have been made public as part of the <tt>kluster</tt>
  project, and you can get them as shown below if you want to add them to your
  system.  The following commands will only work as-is if you have LSB-style init
  scripts, which is the case in Debian
  <tt>wheezy</tt>.  The next Debian release, <tt>jessie</tt>, won't have init scripts at all and
  will use <tt>systemd</tt> instead (the so-called "systemd controversy").  If someone can figure out how to do the
  following in <tt>systemd</tt>, please let us know.
  Type
  \verbatim
  sudo bash
  \endverbatim
  and then as root, do:
  \verbatim
  apt-get install -y subversion
  svn cat https://svn.code.sf.net/p/kluster/code/trunk/scripts/sbin/mem-killer.pl > /sbin/mem-killer.pl
  chmod +x /sbin/mem-killer.pl
  cd /etc/init.d
  svn cat https://svn.code.sf.net/p/kluster/code/trunk/scripts/etc/init.d/mem-killer >mem-killer
  chmod +x mem-killer
  update-rc.d mem-killer defaults
  service mem-killer start
  \endverbatim
  <tt>mem-killer.pl</tt> is capable of sending email to the administrators and to the person
  whose jobs was killed if something is wrong, but this will only work if your
  system can send mail, and if the user has put their email somewhere in the
  "office" field of their gecos information, using chfn (or ypchfn if using NIS).
  Again, if you're running in the cloud, it's best to forget about anything email-related,
  as it's too hard to get working.
  
  \subsection parallelization_grid_stable_nfs Keeping your grid stable (NFS)
  
  We aren't going to give a complete run-down of how to install NFS here, but we
  want to mention some potential issues, and explain some options that work well.
  Firstly, NFS is not the only option for a shared filesystem; there are some newer
  distributed file systems available too, but we don't have much experience with them.
  
  NFS can perform quite badly if the options are wrong.  Below are the options we use.
  We show it as if we're grepping it from /etc/fstab; this isn't actually how we do it
  (we actually use NIS and automount), but the following way is simpler:
  \verbatim
  # grep a05 /etc/fstab
  a05:/mnt/data /export/a05 nfs rw,vers=3,rsize=8192,wsize=8192,acdirmin=5,acdirmax=8,hard,proto=tcp 0 0
  \endverbatim
  The option "vers=3" means we use NFS version 3, which is stateless.  We tried using version 4,
  a supposedly more advanced "stateful" protocol, but we got a lot of crashes.
  
  The <tt>acdirmin=5</tt> and <tt>acdirmin=8</tt> options are the minimum and maximum times that NFS
  waits before re-reading cached directory information; the defaults are 30 and 60 seconds respectively.
  This is important for Kaldi scripts, because the files that we execute on GridEngine are written
  only shortly before we run the scripts, so with default NFS options they may not yet be visible
  on the execution host at the time they are needed.  Above we showed our script <tt>/var/lib/gridengine/default/common/prolog.sh</tt>
  which waits up to 14 seconds for the script to appear.  It's significant that 14 > 8, i.e. that the
  number of seconds the prolog script will wait for is greater than the maximum directory caching period for NFS.
  
  The <tt>hard</tt> option is also important; it means that if the server is busy, the client will wait
  for it to succeed rather than reporting an error (e.g. <tt>fopen</tt> returning error status).  If you
  specify <tt>soft</tt>, Kaldi may crash.  <tt>hard</tt> is the default so could be omitted.
  
  The <tt>proto=tcp</tt> option is also the default on Debian currently; the
  alternative is <tt>proto=udp</tt>.  The TCP protocol is important for stability
  when local networks may get congested; we have found this through experience.
  
  The <tt>rsize=8192,wsize=8192</tt> are packet sizes; they are supposedly
  important in the performance of NFS.  Kaldi reads and writes are generally contiguous and
  don't tend to seek much within files, so large packet size is probably
  suitable.
  
  Another thing you might want to tune that's NFS related is the number of threads in the server.
  You can see this as follows:
  \verbatim
  /$ head /etc/default/nfs-kernel-server
  # Number of servers to start up
  RPCNFSDCOUNT=64
  \endverbatim
  To change it you edit that file and then restart the service (<tt>service nfs-kernel-server restart</tt>, on Debian).
  Apparently it is not good for this to be less than the number of clients that might simultaneously access your
  server (although 64 is the upper limit of what I've seen people recommend to set this to).  Apparently one way to
  tell if you have too few threads is too look at the  <tt>retrans</tt> count in your NFS stats:
  \verbatim
  nfsstat -rc
  Client rpc stats:
  calls      retrans    authrefrsh
  434831612   6807461    434979729
  \endverbatim
  The stats above have a largish number of <tt>retrans</tt>, and apparently in the ideal case it should be zero.
  We had the number of NFS threads set to a lower number (24) for most of the time that machine was up, which
  was less than the potential number of clients, so it's not surprising that we had a high amount of <tt>retrans</tt>.
  What seems to happen is that if a large number of clients are actively using the server, especially writing
  large amounts of data simultaneously, they can tie up all the server threads and then when another client tries
  to do something, it fails and in the client logs you'll see something like <tt>nfs: server a02 not responding, still trying</tt>.
  This can sometimes be associated with crashed jobs, and if you use automount on your setup, it can sometimes
  cause jobs that are not even accessing that server to crash or be delayed (automount has a brain-dead, single-threaded
  design, so failure of one mount request can hang up all other automount reqeuests).
  
  \subsection parallelization_grid_stable_misc Keeping your grid stable (general issues)
  
  In this section we have some general observations about how to configure
  and manage a compute grid.
  
  In CLSP we use a lot of NFS hosts, not just one or two; in fact, most of our
  nodes also export data via NFS.  If you do this you should use
  our <tt>mem-killer.pl</tt> or a similar script, or you will get instability due
  to memory exhaustion when users make mistakes.
  Having a large number of file
  servers is a particularly good idea for queues that are shared by many people,
  because it's inevitable that people will overload file servers, and if there are
  only one or two file servers, the queue will end up being in a degraded state
  for much of the time.  The network bandwidth of each individual file server at
  CLSP@JHU is quite slow: for cost reasons we still use 1G ethernet, but all
  connected to each other with an expensive Cisco router so that there is no
  global bottleneck.  This means that each file server gets overloaded quite
  easily; but because there are so many individual file servers, this generally
  only inconveniences the person who is overloading it and maybe one or two
  others.
  
  When we do detect that a particular file server is loaded (generally either by
  noticing slowness, or by seeing errors like <tt>nfs: server a15 not
  responding</tt> in the output of <tt>dmesg</tt>), we try to track down why this
  is happening.  Without exception this is due to bad user behavior, i.e. users
  running too many I/O-heavy jobs.  Usually through a combination of the output
  of <tt>iftop</tt> and the output of <tt>qstat</tt>, we can figure out which user
  is causing the problem.  In order to keep the queue stable it's necessary to get
  users to correct their behavior and to limit the number of I/O heavy jobs they
  run, by sending them emails and asking them to modify their setups.  If we
  didn't contact users in this way the queue would be unusable, because users
  would persist in their bad habits.
  
  Many other groups have a lot of file servers, like us, but still have all their
  traffic going to one file server because they buy the file servers one at a time
  and they always allocate space on the most recently bought one.  This is just
  stupid.  In CLSP we avoid the administrator hassle of having to allocate space,
  by having all the NFS servers world-writable at the top level, and by instructing
  people to make directories with their userid on them, on a server of their choice.
  We set up scripts to notify the administrators by email when any of the file
  servers gets 95% full, and to let us know which directories contain a lot of
  data.  We can then ask the users concerned to delete some data, or we delete it
  ourselves if the users are gone and if we feel it's appropriate.  We also set up
  scripts to work out, queue-wide, which users are using the most space, and to
  send them emails informing them where they are consuming the most space.  The
  administrators will also ask the users to clean up, particularly in extreme cases
  (e.g. when a junior student is using a huge amount of space for no obvious
  reason).  Space should never really be a problem, because it's almost always the
  case that the disk is 95% full of junk that no-one cares about or even
  remembers.  It's simply a matter of finding a way to ask those responsible to
  clean up, and making their life easy by telling them where the bulk of their
  data is.
  
  Another useful thing is to locate home directories on a server that is not also
  used for experiments; this ensures that users can always get a prompt, even when
  other users are being stupid.  It also makes the backup policy easier: we back
  up the home directories but not the NFS volumes used for experimental work, and
  we make clear to users that those volumes are not backed up (of course, students
  will still fail to back up their important data, and will sometimes lose it as a
  result).  The ban on running experiments in home directories needs to be
  enforced; we frequently have to tell users to stop parallel jobs with data in
  their home directories.  This is the most frequent cause of grid-wide problems.
  
  
  */
  
  }