Jean-François Rey / otmedia

Browse Code »

Commit 5a14b86787e34200360c3f5d4ef4616d7fceccf3

Authored by Jean-François Rey 2013-08-07 17:13:35 +0200

1 parent c00c2f1895

Exists in master

update doc

Showing 5 changed files with 60 additions and 14 deletions Inline Diff

HOWTO
INSTALL
README
install.sh
main_tools/ConfidenceMeasure.sh

HOWTO

Diff comments View file @ 5a14b86

 #---------------#
 # OTMEDIA LIA   #
 # HOWTO         #
 # version 1.0   #
 #---------------#
 1\ Main scripts options
 -----------------------
 There are five main options for otmedia scripts.
 -h : for help
 -D : Debug mode
 -v n : Verbose mode 1 low to 3 high
 -c : Check results
 -r : force to rerun a script, without deleting work already done
 2\ Main scripts
 ---------------
     2.1\ FirstPass.sh
     -----------------
     FirstPass.sh do speaker diarization and transcription of an audio file. Convert it into wav format if not already done (16000Hz, 16 bits, mono).
     If a .SRT file is present in the same directory of the audio file it will copy it.
     $> FisrtPass.sh [options] 110624FR2_20002100.wav result_directory
     Options:
     -f n : number of forks for speeral
     Output : result_directory/110624FR2_20002100/res_p1/
+            and .ctm, .trs and .txt files.
     2.2\ SecondPass.sh
     ------------------
     SecondPass.sh do speaker adaptation and transcriptions base on the first pass.
     $> SecondPass.sh [options] result_directory/110624FR2_20002100/
     Options:
     -f n : number of forks for speeral
     Output : result_directory/110624FR2_20002100/res_p2/
+            and .ctm, .trs and .txt files.
     2.3\ ConfPass.sh
     ----------------
     ConfPass.sh do confidence measure using the second or third pass.
     $> Confpass.sh [options] result_directory/110624FR2_20002100/ <res_p2|res_p3>
     Output : result_directory/110624FR2_20002100/conf/res_p2/scored_ctm/
             and result_directory/110624FR2_20002100.usf file
     2.4\ ExploitConfidencePass.sh
     -----------------------------
     It exploits confidence pass measure to :
     - boost confidente zone
     - find alternative in non confidente zone (using SOLR DB)
     - extend the lexicon
     $> ExploitConfidencePass.sh [options] result_directory/110624FR2_20002100
     Output :   result_directory/110624FR2_20002100/trigg/speeral
                result_directory/110624FR2_20002100/LEX/speeral/_ext
     2.5\ ThirstPass.sh
     ------------------
     ThirdPass.sh do transcriptions using SecondPass speaker adaptation and ExploitConfidencePass trigg files and new lexicon.
     $> ThirdPass.sh [options] result_directory/110624FR2_20002100/
     Options :
     -f n : number of forks for speeral
     Output : result_directory/110624FR2_20002100/conf/res_p3
+            and .ctm, .trs and .txt files.
     2.6\ RecomposePass.sh
     --------------------
     RecomposePass.sh copy results that missing in ThirsPass from the Second and First Pass.
     $> RecomposePass.sh [options] result_directory/110624FR2_20002100/
     Output : result_directory/110624FR2_20002100/res_all
+            and .ctm, .trs and .txt files.
     2.7\ ScoringRes.sh
     ------------------
     ScoringRes.sh run differents scoring tools to score the results using SRT file if exists.
     $> ScoringRes.sh [options] result_directory/110624FR2_20002100/
     Output : result_directory/110624FR2_20002100/scoring
     2.8\ CheckResults.sh
     --------------------
     CheckResults.sh parse results directories to synthesize works already done.
     $> CheckResults.sh [options] result_directory
     Output : "Directory name      #plp    #res_p1 #treil_p2   #treil_p3   usf_p2  usf_p3"
             #plp number of plp files
             #res_p1 number of .res files at first pass
             #treil_p2 number of .treil files at second pass
             #treil_p3 number of .treil files at third pass
             usf_p2 usf file from confidence pass result on second pass (OK|ERR|NAN)
             usf_p3 usf file from confidence pass result on third pass (OK|ERR|NAN)
 3\ OneScriptToRuleThemAll.sh
 ----------------------------
     The script to do all OTMEDIA LIA pass in one call.
     $> OneScriptToRuleThemAll.sh [options] 110624FR2_20002100.wav result_directory
     Options : (default options are availables)
     -a Do every pass
     -1 Do First pass
     -2 Do Second pass
     -3 Do Third pass
     -C Do Confidence pass
     -e Do Exploit Confidence pass
     -R Do Recompose pass
     -s Do Scoring pass
 4\ Modify configuration
+-----------------------
+    Most of the main scripts got a configuration file (cfg/ directory).
+    You can change script behaviour and data used.
+    Speeral configuration file can be also change (tools/Speeral/CFG/ directory)
     4.1\ Scripts configurations
+        In scripts configuration files (OTMEDIA_HOME/cfg/) you can change default options as architecture, verbose ...
+        Scripts using Speeral got information on binaries, models path and name, and the configuration file for speeral.
     4.2\ Speeral configurations
+        Speeral configuration files are in OTMEDIA_HOME/tools/Speeral/CFG directory.
+        The .tmp files are use to generate .xml file from install.sh.
+        You can modify .xml files for your needs, but most of data informations are pass through arguments at speeral call in scripts.
 5\ Modify binaries
+------------------
+    Binaries can be find in bin and tools directory.
+    Some binaries are compiled in 32 and 64 bits. By default all binaries are compiled in 32 bits.
+    You can update binaries as you need.
+    To modify tools binaries, you need to download a compatible version.
+        lia_ltbox can be found in /labo/Tools/
+        Speeral (binaries) can be compiled from the git remote git@gitlia.univ-avignon.fr:vaudriguard/libspeeral.git . Do not modify Speeral data from OTMEDIA (unless you know what you do).
+        In PACKAGES_MESURES_V1.0 you can update icsiboost binary (in bin) from the projet page : https://code.google.com/p/icsiboost/
+        For QUOTE_FINDER and SIGMUND please contact support.
+Good Luck ! Luke !
+And the force be with you !

INSTALL

Diff comments View file @ 5a14b86

 #---------------#
 # OTMEDIA LIA   #
 # INSTALL       #
 # version : 1.0 #
 #---------------#
 OTMEDIA LIA ready to use ? Really ?
 No ! You have to do manualy configuartion for some features.
 Let see...
 SUMMARY
 -------
 1\ Before installation
 2\ install.sh script
 3\ SOLR install
+4\ Install descriptions
 1\ Before installation
 ----------------------
 - Check and install dependencies.
 - In 64 bits architcture be sure you can run 32 bits programs.
 - Have 300 Go of free space.
 - Have acces to the network and the nyx server.
-2/ install.sh script
+2\ install.sh script
 --------------------
 install.sh script will do most of the work.
 It will check dependencies and configure pass tools.
 By default it will do a complet install (300 Go).
 You can modifiy behavior by editing install.sh :
 To disable lexicon adaption using SOLR DB put EXPLOITCONFPASS to 0 (mainly the 290 Go).
 To disable confidence measure put CONFPASS to 0.
 To disable second and third pass put PASS2 to 0.
+If your login name differ from your nyx login name, edit install.sh and change the username variable to your nyx login name.
 run install.sh and follow the white rabbit.
 3\ SOLR install
 ---------------
 The install.sh script download otmedia-2013-04.tar.gz and untar it in OTMEDIA_HOME/tools/SOLR/ .
 See SOLR.INSTALL file to install OTMEDIA SOLR DB.
 4\ Install descriptions
+-----------------------
 OTMEDIA_HOME
     |-> bin/
            |-> aff_mat
            |-> aff_mat.64
            |-> lia_plp_mt
            |-> lia_plp_mt.64
            |-> LIUM_SpkDiarization-4.2.jar
            |-> sclite
-    |-> cfg/
+    |-> cfg/                                    # Main scripts configurations files
             |-> ConfidenceMeasure.cfg
             |-> ConfPass.cfg
             |-> ExploitConfidencePass.cfg
             |-> FirstPass.cfg
             |-> main_cfg.cfg
             |-> RecomposePass.cfg
             |-> Scoring.cfg
             |-> Secondass.cfg
             |-> ThirdPass.cfg
-    |-> data/
+    |-> data/                                   # Some data
             |-> rules/
                     |-> asupp
                     |-> basic
                     |-> lastprocess.regex
                     |-> muRules.tab
                     |-> numeric_rules
                     |-> postprocess.regex
                     |-> preprocess.regex
                     |-> random_regex.tab
-    |-> main_tools/
+    |-> main_tools/                             # Main scripts
                 |-> CheckResults.sh
                 |-> ConfidenceMeasure.sh
                 |-> ConfPass.sh
                 |-> ExploitConfidencePass.sh
                 |-> FirstPass.sh
                 |-> OneScriptToRuleThemAll.sh
                 |-> RecomposePass.sh
                 |-> ScoringRes.sh
                 |-> SecondPass.sh
                 |-> ThirdPass.sh
-    |-> tools/
+    |-> tools/                                  # Tools
             |-> lia_ltbox/
             |-> PACKAGE_MESURES_V1.0/
             |-> QUOTE_FINDER/
-            |-> scripts/
+            |-> scripts/                        # Secondary scripts (but useful)
                        |-> ApplyCorrectionRules.pl
                        |-> BdlexUC.pl
                        |-> CheckConfPass.sh
                        |-> CheckExploitConfPass.sh
                        |-> CheckFirstPass.sh
                        |-> CheckSecondPass.sh
                        |-> CheckThirdPass.sh
                        |-> CleanFilter.sh
                        |-> CoverageReportMaker.pl
                        |-> ctm2show.pl
                        |-> Date2txt.pl
                        |-> daybefore2after.sh
                        |-> ExtractAudioFromTV.sh
                        |-> FindNormRules.pl
                        |-> formatRES.pl
                        |-> GenerateSOLRQueries.pl
                        |-> intersec.pl
                        |-> KeepConfZone.pl
                        |-> LexPhonFilter.pl
                        |-> MergeLexicon.pl
                        |-> NbMaxWordsFilter.pl
                        |-> Number2txt.pl
                        |-> perlmod/
                                   |-> Utils.pm
                        |-> PhonFormatter.pl
                        |-> ProcessSOLRQueries.py
                        |-> RandomRegex.pl
                        |-> RemoveLineContaining.pl
                        |-> res2out.pl
                        |-> ScoreCtm2trigg.pl
                        |-> scoredCtmAndTaggedLem2All.pl
                        |-> Sentencer.pl
                        |-> srt2stm.pl
                        |-> Tools.sh
                        |-> UrlConverter.pl
             |-> SIGMUND/
+            |-> SOLR/
+            |-> Speeral/
     |-> COPYING
     |-> CorpusOTMedia.txt
     |-> HOWTO
     |-> INSTALL
     |-> README
     |-> SOLR.INSTALL
     |-> TODO

README

Diff comments View file @ 5a14b86

              ___ _____ __  __ _____ ____ ___    _      _     ___    _
             / _ \_   _|  \/  | ____|  _ \_ _|  / \    | |   |_ _|  / \
            | | | || | | |\/| |  _| | | | | |  / _ \   | |    | |  / _ \
            | |_| || | | |  | | |___| |_| | | / ___ \  | |___ | | / ___ \
             \___/ |_| |_|  |_|_____|____/___/_/   \_\ |_____|___/_/   \_\
 #---------------#
 #  OTMEDIA LIA  #
 #    README     #
 #  version 1.0  #
 #---------------#
 DESCRIPTION
 -----------
     OTMEDIA means "Observatoire Transmedia", its main objective is to study the evolution and transformation of the media world.
     The scientific objective of the project is the creation of a new generation of media observatory
     based on an interactive automatic analysis system (semi-automatic) transmedia to understand
     the world of information and developments.
     Web Site : http://www.otmedia.fr
     OTMEDIA LIA project is a set of tools to transcribe radio and TV shows.
     It does multiple things :
     - First pass : default transcription with speeral and speaker diarization.
     - Second pass : speaker adaptation and a second transcription pass with speeral.
     - Confidence pass : calcul confidence measure from transcription output.
     - Exploit Confidence Measure : use SOLR DB data to extend the lexicon on low confidence measure and create trigg files.
     - Third pass : second pass using the new lexicon and trigg files.
+    From GIT : git@gitlia.univ-avignon.fr/otmedia.git
 DEPENDENCIES
 ------------
 GNU Toolchain
         Available from : http://www.gnu.org
         and debian packages
         Compiling, linking, and building applications.
 avconv (libav-tools >= 0.8)
     Available from : http://libav.org
     and debian package
     avconv is a very fast video and audio converter.
 JAVA JDK and JRE ( >= 6)
     Available from : http://www.oralce.com
     and debian packages
     JAVA Developpment kit and JAVA runtime environment.
 Python ( >= 2.7.0)
     Available from : http://http://www.python.org/
     and debian packages
     Python is a programming language.
 Perl ( >= 5.0.0)
     Available from : http://www.perl.org/
     and debian packages
     Perl is a programming language.
 iconv ( >= 2.0.0)
     Available from : http://www.gnu.org
     and debian package
     Character set conversion.
 csh shell (csh)
     Available on debian packages.
     The C shell was originally written at UCB to overcome limitations in the
     Bourne shell.  Its flexibility and comfort (at that time) quickly made it
     the shell of choice until more advanced shells like ksh, bash, zsh or
     tcsh appeared.  Most of the latter incorporate features original to csh
 The SRI Language Modeling Toolkit (SRILM >= 1.6.0)
     Available from : http://www.speech.sri.com/projects/srilm/
     SRILM is a toolkit for building and applying statistical language models.
 Tomcat ( >= 7.0.0)
     Available from : http://tomcat.apache.org/
     and debian packages
     Apache Tomcat is an open source software implementation of the Java Servlet and JavaServer Pages technologies.
 INSTALL
 -------
     See the INSTALL file for the installation procedure.
     Quick install below.
     Before launching installation :
       Be certain that all dependencies are satisfied.
       Have 300 Go of free space for complet install.
     Issue the following commands to the shell :
     $> ./install.sh
     $> export OTMEDIA_HOME=path/to/OTMEDIA/directory
     Read SOLR.INSTALL part 3 to install SOLRDB.
 RUNNING
 -------
     See HOWTO file.
 ACKNOWLEDGEMENTS
 ----------------
     Many thanks to Jean-François Rey for useful help and work done.
 KNOWN BUGS
 ----------
     Many.
     For Bug report, please contact Pascal Nocera at pascal.nocera@univ-avignon.fr
 COPYRIGHT
 ---------
     See the COPYING file.
 AUTHORS
 -------
     Jean-François Rey <jean-francois.rey@univ-avignon.fr>
     Hugo Mauchrétien <hugo.mauchretien@univ-avignon.fr>
     Emmanuel Ferreira <emmanuel.ferreira@univ-avignon.fr>

install.sh

Diff comments View file @ 5a14b86

 #!/bin/bash
 #-------------------#
 # OTMEDIA LIA       #
 # Install script    #
-# version : 1.0.0   #
+# version : 1.1.0   #
 #-------------------#
+# nyx login name
 username=${LOGNAME}
 # Color variables
 txtred=$(tput setaf 1) # red
 txtgrn=$(tput setaf 2) # Green
 txtylw=$(tput setaf 3) # Yellow
 txtblu=$(tput setaf 4) # Blue
 txtred=$(tput setaf 5) # Purple
 txtcyn=$(tput setaf 6) # Cyan
 txtwht=$(tput setaf 7) # White
 txtrst=$(tput sgr0) # Text reset.
 #/color
 #
 ### Global Variables
 #
 PWD=$(pwd)
 OTMEDIA_HOME=$PWD
 test=$(arch)
 if [ "$test" == "x86_64" ]; then ARCH=".64"; else ARCH=""; fi
 #/Global
 #
 # Put to 0 to disable dependencies of a pass
 # and 1 to enable
 #
 PASS1=1     # First Pass
 PASS2=1     # Second and Third Pass
 CONFPASS=1  # Confidence Pass
 EXPLOITCONFPASS=1   # SOLR query and trigg
 echo -e "\nWill do install for :"
 if [ $PASS1 -eq 1 ];then echo "- Pass 1";fi
 if [ $PASS2 -eq 1 ];then echo "- Pass 2";fi
 if [ $CONFPASS -eq 1 ];then echo "- Confidence Pass";fi
 if [ $EXPLOITCONFPASS -eq 1 ];then echo "- Exploit Confidence Pass";fi
 #
 ### CHECK Dependencies ###
 #
 echo -e "\n\t${txtblu}Check Dependencies${txtrst}\n"
 ## make
 test=$(whereis make)
 if [ "$test" == "make:" ]
 then
     echo -e "${txtred}ERROR${txtrst} make not found\n You have to install make\n sudo apt-get install make"
     exit 1;
 fi
 echo -e "make \t ${txtgrn}OK${txtrst}"
 ## CC
 test=$(whereis cc)
 if [ "$test" == "cc:" ]
 then
     echo -e "${txtred}ERROR${txtrst} cc not found\n You have to install cc\n sudo apt-get install gcc"
     exit 1;
 fi
 echo -e "cc \t ${txtgrn}OK${txtrst}"
 ## AVCONV
 test=$(whereis avconv)
 if [ "$test" == "avconv:" ]
 then
     echo -e "${txtred}ERROR${txtrst} avconv not found\n You have to install avconv\n sudo apt-get install libav-tools"
     exit 1;
 fi
 echo -e "libav-tools : avconv \t ${txtgrn}OK${txtrst}"
 ## JAVA
 test=$(whereis java)
 if [ "$test" == "java:" ]
 then
     echo -e "${txtred}ERROR${txtrst} java not found\n You have to install java JRE\n sudo apt-get install openjdk-7-jre"
     exit 1;
 fi
 echo -e "Java : JRE \t ${txtgrn}OK${txtrst}"
 test=$(whereis javac)
 if [ "$test" == "javac:" ]
 then
     echo -e "${txtred}ERROR${txtrst} javac not found\n You have to install java JDK\n sudo apt-get install openjdk-7-jdk"
     exit 1;
 fi
 echo -e "Java : JDK \t ${txtgrn}OK${txtrst}"
 if [ $EXPLOITCONFPASS -eq 1 ]
 then
     ## Python
     test=$(whereis python)
     if [ "$test" == "python:" ]
     then
         echo -e "${txtred}ERROR${txtrst} python not found\n You have to install python\n sudo apt-get install python"
         exit 1;
     fi
     echo -e "python : \t ${txtgrn}OK${txtrst}"
     ## csh shell
     test=$(whereis csh)
     if [ "$test" == "csh:" ]
     then
         echo -e "${txtred}ERROR${txtrst} csh shell not found\n You have to install csh shell\n sudo apt-get install csh"
         exit 1;
     fi
     echo -e "csh shell : \t ${txtgrn}OK${txtrst}"
 fi
 ## Perl
 test=$(whereis perl)
 if [ "$test" == "perl:" ]
 then
     echo -e "${txtred}ERROR${txtrst} perl not found\n You have to install perl\n sudo apt-get install perl"
     exit 1;
 fi
 echo -e "perl : \t ${txtgrn}OK${txtrst}"
 ## iconv
 test=$(whereis iconv)
 if [ "$test" == "iconv:" ]
 then
     echo -e "${txtred}ERROR${txtrst} iconv not found\n You have to install iconv\n sudo apt-cache search iconv"
     exit 1;
 fi
 echo -e "iconv : \t ${txtgrn}OK${txtrst}"
 ## SRI LM
 if [ -z "$SRILM" ] && [ -z "$MACHINE_TYPE" ]
 then
     echo -e "${txtred}ERROR${txtrst} SRILM toolkit variables are not defined (SRILM and MACHINE_TYPE)\n You have to install SRILM Toolkit\n"
     exit 1;
 fi
 export SRILM_BIN=$SRILM/bin/$MACHINE_TYPE
 echo -e "SRILM toolkit : \t ${txtgrn}OK${txtrst}"
 ### Speeral Configuration ###
 echo -e "\n\t${txtblu}Speeral configuration${txtrst}\n"
-echo -e "Download Speeral bin and data :"
+read -e -p "Download Speeral bin and data ? (y/n) " speeral
-scp -r ${username}@nyx:~/OTMEDIA_DATA/Speeral $OTMEDIA_HOME/tools/
+if [ "$speeral" == "y" ]
+then
+    echo -e "Download Speeral bin and data :"
+    scp -r ${username}@nyx:/local/OTMEDIA/OTMEDIA_DATA/Speeral $OTMEDIA_HOME/tools/
+fi
 echo -e "\n\t${txtblu}Generating Speeral configuration files :${txtrst}\n"
 cat $PWD/tools/Speeral/CFG/SpeeralFirstPass.xml.tmp | sed -e "s|<nom>[^<]*</nom>|<nom>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer</nom>|g" \
     | sed -e "s|<ngramme>[^<]*</ngramme>|<ngramme>$PWD/tools/Speeral/LM/ML_4gOTMEDIA_LEXIQUE_V6</ngramme>|g" \
     | sed -e "s|<binode>[^<]*</binode>|<binode>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer.bin</binode>|g" \
     > $PWD/tools/Speeral/CFG/SpeeralFirstPass.xml
 echo $PWD/tools/Speeral/CFG/SpeeralFirstPass.xml
 cat $PWD/tools/Speeral/CFG/SpeeralSecondPass.xml.tmp | sed -e "s|<nom>[^<]*</nom>|<nom>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer</nom>|g" \
     | sed -e "s|<ngramme>[^<]*</ngramme>|<ngramme>$PWD/tools/Speeral/LM/ML_4gOTMEDIA_LEXIQUE_V6</ngramme>|g" \
     | sed -e "s|<binode>[^<]*</binode>|<binode>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer.bin</binode>|g" \
     > $PWD/tools/Speeral/CFG/SpeeralSecondPass.xml
 echo $PWD/tools/Speeral/CFG/SpeeralSecondPass.xml
 cat $PWD/tools/Speeral/CFG/SpeeralThirdPass.xml.tmp | sed -e "s|<nom>[^<]*</nom>|<nom>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer</nom>|g" \
     | sed -e "s|<ngramme>[^<]*</ngramme>|<ngramme>$PWD/tools/Speeral/LM/ML_4gOTMEDIA_LEXIQUE_V6</ngramme>|g" \
     | sed -e "s|<binode>[^<]*</binode>|<binode>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer.bin</binode>|g" \
     > $PWD/tools/Speeral/CFG/SpeeralThirdPass.xml
 echo $PWD/tools/Speeral/CFG/SpeeralThirdPass.xml
 if [ $EXPLOITCONFPASS -eq 1 ]
 then
     ### LIA ltbox ###
     echo -e "\t${txtblu}Install lia_ltbox${txtrst}\n"
     export LIA_TAGG_LANG="french"
     export LIA_TAGG="$OTMEDIA_HOME/tools/lia_ltbox/lia_tagg/"
     export LIA_PHON_REP="$OTMEDIA_HOME/tools/lia_ltbox/lia_phon/"
     export LIA_BIGLEX="$OTMEDIA_HOME/tools/lia_ltbox/lia_biglex/"
     ### config lia_phon
     cd $LIA_PHON_REP
     make all > /dev/null
     make ressource > /dev/null
     ### config lia_tagg
     cd $LIA_TAGG
     make all > /dev/null
     make ressource.french > /dev/null
     ### config lia_biglex
     cd $LIA_BIGLEX
     make -f makefile.biglex > /dev/null
     cd $OTMEDIA_HOME
     ### SOLR DB ###
     # Tomcat fisrtly
     test=$(dpkg -l | grep "^ii" | grep tomcat)
     if [ "$test" == "" ]
     then
         echo -e "${txtred}ERROR${txtrst} TOMCAT seems to not be installed)\n You have to install TOMCAT\n"
         #exit 1;
     fi
     echo -e "\nTOMCAT : \t ${txtgrn}OK${txtrst}\n"
     # SOLR secondly
     echo -e "\t${txtblu}Install SOLR DB${txtrst}\n"
     echo -e "You will need 300 Go of free space to install SOLR DB"
     read -p "Continue ? (y/n) " solr
     if [ "$solr" == "y" ]
     then
         echo -e "Download SOLR DB\r"
         mkdir -p $OTMEDIA_HOME/tools/SOLR 2> /dev/null
-        scp -r ${username}@nyx:~/OTMEDIA_DATA/SOLR/otmedia-2013-04.tar.gz $OTMEDIA_HOME/tools/SOLR
+        scp -r ${username}@nyx:/local/OTMEDIA/OTMEDIA_DATA/SOLR/otmedia-2013-04.tar.gz $OTMEDIA_HOME/tools/SOLR
         echo -e "Unzip SOLR DB\r"
         res=0
         #res = $(tar -xvzf "$OTMEDIA_HOME/tools/SOLR/otmedia-2013-04.tar.gz" "$OTMEDIA_HOME/tools/SOLR/")
         if [ $res -eq 2 ]; then echo " ${txtred}NOT OK${txtrst}";
         else echo " ${txtgrn}OK${txtrst}"; fi
     else
         echo "Skipping SOLR install"
     fi
     read -e -p "Configure SOLR DB server ? (y/n) " solr
     if [ "$solr" == "y" ]
     then
         read -p "Enter SOLR server IP :" ip
         if [ "${ip}" == "" ];then ip="localhost";fi
         echo "machine = \"${ip}\"" > $OTMEDIA_HOME/tools/scripts/solrinfo.py
         read -p "Enter SOLR server port :" port
         if [ "${port}" == "" ]; then port="8080";fi
         echo -e "\n\tSOLR server IP ${ip}"
         echo -e "\tSOLR server port ${port}"
         echo "port = \"${port}\"" >> $OTMEDIA_HOME/tools/scripts/solrinfo.py
     else
         echo "Skipping SOLR DB Configuration"
     fi
     echo -e "\nSee SOLR.INSTALL file for more information\n"
 fi
 ### Set Variables in bashrc ###
 cat ~/.bashrc | grep -v "OTMEDIA_HOME" | grep -v "SRILM_BIN" > ~/.bashrc.org
 #cat ~/.bashrc | grep -v "OTMEDIA_HOME" | grep -v "SRILM_BIN" | grep -v "LIA_TAGG" | grep -v "LIA_PHON" | grep -v "LIA_BIGLEX" > ~/.bashrc.org
 cp ~/.bashrc.org ~/.bashrc
 export OTMEDIA_HOME=$PWD
 echo "export OTMEDIA_HOME=$PWD" >> ~/.bashrc
 echo "export $PATH=$PATH:$PWD/main_tools" >> ~/.bashrc
 echo "export SRILM_BIN=$SRILM/bin/$MACHINE_TYPE" >> ~/.bashrc
 #echo "export LIA_TAGG_LANG=french" >> ~/.bashrc
 #echo "export LIA_TAGG=$OTMEDIA_HOME/tools/lia_ltbox/lia_tagg/" >> ~/.bashrc
 #echo "export LIA_PHON_REP=$OTMEDIA_HOME/tools/lia_ltbox/lia_phon/" >> ~/.bashrc
 #echo "export LIA_BIGLEX=$OTMEDIA_HOME/tools/lia_ltbox/lia_biglex/" >> ~/.bashrc
 # set global configuration file
 echo "OTMEDIA_HOME=$PWD" > $OTMEDIA_HOME/cfg/main_cfg.cfg
 echo "ARCH=$ARCH" >> $OTMEDIA_HOME/cfg/main_cfg.cfg
 echo "VERBOSE=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg
 echo "DEBUG=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg
 echo "CHECK=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg
 echo "RERUN=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg
 echo -e "\n\t${txtgrn}### Install completed ###${txtrst}\n"
 echo -e "do : source ~/.bashrc"
 echo -e "or set variable :\n"
 echo "export OTMEDIA_HOME=$PWD"
 echo "export PATH=\$PATH:$OTMEDIA_HOME/main_tools"
 echo "export SRILM_BIN=$SRILM/bin/$MACHINE_TYPE"
-echo "             \\\\ "
+echo "${txtwht}             \\\\ "
 echo "        ,-~~~-\\\\_"
 echo "       (        .\ "
-echo "       @\___(__--'"
+echo "       @\___(__--'${txtrst}"
 echo "${txtgrn}Yes${txtylw}I${txtred}Rastafari${txtrst}"

main_tools/ConfidenceMeasure.sh

Diff comments View file @ 5a14b86

 #!/bin/bash
 #-----------------------------------------------------------------------------------------
 # Author : Benjamin Lecouteux & Emmanuel FERREIRA (contact emmanuel.ferreira0194@gmail.com)
 # Brief: Determine les mesures de confiance d'une transcription (res de speeral)
 #-----------------------------------------------------------------------------------------
 # where is ConfidenceMeasure.sh
 if [ -z $MAIN_SCRIPT_PATH ]; then MAIN_SCRIPT_PATH=$(dirname $(readlink -e $0)); fi
 # where is ConfidenceMeasure.cfg
 CONFIDENCEMEASURE_CONFIG_FILE=$OTMEDIA_HOME"/cfg/ConfidenceMeasure.cfg"
 if [ -e $CONFIDENCEMEASURE_CONFIG_FILE ]
 then
     . $CONFIDENCEMEASURE_CONFIG_FILE
 else
     echo "ERROR : Can't find configuration file $CONFIDENCEMEASURE_CONFIG_FILE" >&2
     exit 1
 fi
 PACKAGE_CONF_MEASURE=$CONFIDENCEMEASURE_CONFIG_FILE
 #------------------
 # Parser les options
 #-------------------
 while getopts ":c:s:h" OPTION
 do
 	case $OPTION in
 	h) 	#Display help
 		echo -e "$0 :"
 		echo -e "\tAuthor : Benjamin Lecouteux & Emmanuel FERREIRA (contact: emmanuel.ferreira0194@gmail.com)"
 		echo -e "\tVersion : 2.0"
 		echo -e "\tBrief : Determine confidence measure of a transcription"
 		echo -e "\tUsage : $0 [OPTIONS] <(i) REP_IN> <REP_NAME>"
 		echo -e "\tOptions:"
 		echo -e "\t\tc) specify the path of the configuration file (default $PACKAGE_CONF_MEASURE)"
 		echo -e "\t\ts) specify PORT@HOST of a SRILM server"
 		exit 1
 	;;
 	c) 	#Change the configuration file
 	   	PACKAGE_CONF_MEASURE=$OPTARG
 	;;
 	s)	#use an SRILM server (avoid loading arpa model in memory)
 		SERVER=$OPTARG
 	;;
 	:)
 		echo "BAD USAGE : OPTION $OPTARG need a value"
 	   	exit 1
 	;;
 	\?)
 		echo "BAD USAGE : unknow option '$OPTARG'"
 	   	exit 1
 	;;
 	esac
 done
 #-------------------------------------------
 # Shift options pour recuperation arguments
 #-------------------------------------------
 shift $((OPTIND-1))
 if [ -z "$1" ]
 then
     echo "BAD USAGE: $0 [OPTIONS] <(i) repertoire (ex:20041006_0800_0900_CULTURE)> <REP_NAME (ex:res_p2)>"
 	exit 1
 fi
 if [ -z "$2" ]
 then
     echo "BAD USAGE: $0 [OPTIONS] <(i) repertoire (ex:20041006_0800_0900_CULTURE)> <REP_NAME (ex:res_p2)>"
 	exit 1
 fi
 . $PACKAGE_CONF_MEASURE
 #------------------------------------
 # INIT - Creation du workspace
 #------------------------------------
 NAME=`basename $1`
 CONF_DIR=$1/conf/$2
 FICHIER_RES=$2
 REF=$CONF_DIR/ref
 POS=$CONF_DIR/pos
 MLCLASS=$CONF_DIR/mlclass
 GVALIGN=$CONF_DIR/gvalign
 HTK_POST=$CONF_DIR/htk_post
 HTK_LM=$CONF_DIR/htk_lm
 WLAT=$CONF_DIR/wlat
 LIKELIHOOD=$CONF_DIR/likelihood
 GVCTM=$CONF_DIR/gvctm
 SEGCTM=$CONF_DIR/segctm
 SUPER_CTM=$CONF_DIR/super_ctm
 SCORED_CTM=$CONF_DIR/scored_ctm
 mkdir -p $CONF_DIR > /dev/null 2>&1
 mkdir -p $REF > /dev/null 2>&1
 mkdir -p $POS > /dev/null 2>&1
 mkdir -p $MLCLASS > /dev/null 2>&1
 mkdir -p $GVALIGN > /dev/null 2>&1
 mkdir -p $HTK_POST > /dev/null 2>&1
 #mkdir -p $HTK_LM ==> generer auto par SRILM si besoin
 mkdir -p $WLAT > /dev/null 2>&1
 mkdir -p $LIKELIHOOD > /dev/null 2>&1
 mkdir -p $GVCTM > /dev/null 2>&1
 mkdir -p $SEGCTM > /dev/null 2>&1
 mkdir -p $SUPER_CTM > /dev/null 2>&1
 mkdir -p $SCORED_CTM > /dev/null 2>&1
-if [ -z $BOOST_BIN ];then
+if [ -z $BOOST_BIN ] && [ $ARCH == ".64"] ;then
 	BOOST_BIN=$ROOT/bin/icsiboost-64bit-static-r160
+fi
+if [ -z $BOOST_BIN ] ;then
+	BOOST_BIN=$ROOT/bin/icsiboost-32bit-static-r176
 fi
 #-----------------------------------------------------------------
 # STEP 1 - Extension des treillis + ajout posteriors (format htk)
 #-----------------------------------------------------------------
 if [ $EXTEND  == 1 ]
 then
 	echo "EXTEND step..."
 	rm -r $HTK_LM > /dev/null 2>&1
 	rm $HTK_POST/* > /dev/null 2>&1
 	#
 	# --> Ajout des scores linguistiques dans le HTK
 	#
 	ls $1/$FICHIER_RES/*.treil > $CONF_DIR/Liste_treil_${NAME}.lst
 	LM_ACCESS="-lm $ML"
 	if [ ! -z $SERVER ]; then
 		LM_ACCESS="-use-server $SERVER -cache-served-ngrams"
 	fi
 	echo "$SRILM_BIN/lattice-tool -read-htk -in-lattice-list $CONF_DIR/Liste_treil_${NAME}.lst $LM_ACCESS -order $ORDER -htk-logbase 10 -htk-lmscale $FUDGE -htk-wdpenalty $PENALITE -write-htk -out-lattice-dir $HTK_LM";
 	$SRILM_BIN/lattice-tool -read-htk -in-lattice-list $CONF_DIR/Liste_treil_${NAME}.lst $LM_ACCESS -order $ORDER -htk-logbase 10 -htk-lmscale $FUDGE -htk-wdpenalty $PENALITE -write-htk -out-lattice-dir $HTK_LM
 	#
 	# --> Calcul des posteriors a partir des scores acoustiques et linguistiques present dans le HTK
 	#
 	for file in `ls $HTK_LM/*.treil`
 	do
 		base=`basename $file .treil`;
 		#echo "lattice-tool -read-htk -in-lattice $file -compute-posteriors -write-htk -out-lattice $HTK_POST/${base}.htk"
 		$SRILM_BIN/lattice-tool -read-htk -in-lattice $file -compute-posteriors -write-htk -out-lattice $HTK_POST/${base}.htk
 	done
 fi
 #---------------------------------------------------------------------------------------------------------------
 # STEP 2 - alignement res et wlat pour creer res avec scores + infos (utilise un fastnc modifie)
 # Exemple :
 #  ok amendement 0.814885 ( time=36 nodes=3 min=0.0016862 max=0.814885 mean=0.333896 var=0.363849 svar=0.603199 )
 #----------------------------------------------------------------------------------------------------------------
 if [ $FASTNC == 1 ]
 then
 	echo "FASTNC step..."
 	rm -f $POS/* $WLAT/* > /dev/null 2>&1
 	for file in `ls $HTK_LM/*.treil`
 	do
 		base=`basename $file .treil`;
 		#echo "$ROOT/bin/fastnc_v1.4 $HTK_POST/${base}.htk $WLAT/${base}.wlat  $1/$FICHIER_RES/${base}.res rien -dtw2 > $POS/$base.pos2&"
 		$ROOT/bin/fastnc_v1.4 $HTK_POST/${base}.htk $WLAT/${base}.wlat  $1/$FICHIER_RES/${base}.res rien -dtw2 > $POS/$base.pos2
 	done
 fi
 #------------------------------------------------------------------------------------------------------------
 # STEP 3 - recuperation de la probabilite pour chaque mot + info relatives au modele de langue (backoff, ...)
 #------------------------------------------------------------------------------------------------------------
 if [ $PPL == 1 ]
 then
 	echo "PPL step..."
 	rm -f $REF/* $CONF_DIR/${NAME}_ALLREF.* $MLCLASS/* > /dev/null 2>&1
 	#
 	# --> Creation des references a partir des .res (uniquement si .treil present)
 	#
 	for file in `ls $1/$FICHIER_RES/*.res`
 	do
 		base=`basename $file .res`;
 		if [ -f $1/$FICHIER_RES/$base.treil ];then
 			cat $file | cut -f5 -d' ' | tr "\n" " " > $REF/${base}.ref
 		fi
 	done
 	#
 	# --> creation d'un fichier contenant l'ensemble des transcriptions du show
 	#
 	compteur=0
 	for file in `du -sh $REF/*.ref | grep -v "^0" | cut -f2`
 	do
 		base=`basename $file .ref`;
 		cat $file >> $CONF_DIR/${NAME}_ALLREF.txt
 		echo "" >> $CONF_DIR/${NAME}_ALLREF.txt
 		ListeFichiers[$compteur]=$base.mlclass
 		compteur=$(( $compteur + 1 ))
 	done
 	#
 	# --> recuperation de la probabilite pour chaque mot provenant des resultats de l'ASR + informations linguistiques (backoff used, ngram,...)
 	#
 	$SRILM_BIN/ngram -lm $ML -order $ORDER -ppl $CONF_DIR/${NAME}_ALLREF.txt -debug 2 > $CONF_DIR/${NAME}_ALLREF.mlclass
 	#
 	# --> creation d'un fichier par fichier .ref
 	#
 	compteur=0
 	cat $CONF_DIR/${NAME}_ALLREF.mlclass | while read line
 	do
 		echo $line | grep "^$" > /dev/null
 		if [ $? == 0 ];then
 			compteur=$(( $compteur + 1 ))
 		else
 			echo "$line" | grep "p(" > /dev/null
 			if [ $? == 0 ];then
 				echo "$line" >> $MLCLASS/${ListeFichiers[${compteur}]};
 			fi
 		fi
 	done
 fi
 #----------------------------------------------------------
 # STEP 4 - recuperation du score acoustique de chaque mot
 #----------------------------------------------------------
 if [ $ACOUST == 1 ]
 then
 	echo "ACOUST step..."
 	rm -f $GVALIGN/* > /dev/null 2>&1
 	rm -f $GVCTM/* > /dev/null 2>&1
 	rm -f $SEGCTM/* > /dev/null 2>&1
 	rm -f $LIKELIHOOD/* > /dev/null 2>&1
 	for file in `ls $1/$FICHIER_RES/*.res`
 	do
 		base=`basename $file .res`
 		if [ -f $1/$FICHIER_RES/$base.treil ];then
 			#echo "$ROOT/script/MakeListForGVAlign.pl $file $GVALIGN";
 			$ROOT/script/MakeListForGVAlign.pl $file $GVALIGN
 		fi
 	done
 	for file in `ls $GVALIGN/*.gvalign`
 	do
 		base=`basename $file .gvalign`;
 		type=`echo $base | cut -f2 -d: | cut -f2- -d\# | sed -e "s/[0-9]\+//"`
 		case "$type" in
 			"M#S")
 			HMM=$mod_ms
 			;;
 			"F#S")
 			HMM=$mod_fs
 			;;
 			"M#T")
 			HMM=$mod_mt
 			;;
 			"F#T")
 			HMM=$mod_ft
 			;;
 		esac
 		#echo "$ROOT/bin/gvalign.old $HMM $PHON $file -e $1/${REP_PLP}/ -f .plp -r $GVALIGN -g .gv -C FAST -W $GVCTM -O CTM -s $SEGCTM > $LIKELIHOOD/${base}.likelihood | sed -e 's/Decoding/\\nDecoding/g' > $LIKELIHOOD/${base}.likelihood";
 		#$ROOT/bin/gvalign.old $HMM $PHON $file -e $1/${REP_PLP}/ -f .plp -r $GVALIGN -g .gv -C FAST -W $GVCTM  -O CTM  -s $SEGCTM | sed -e 's/Decoding/\nDecoding/g' > $LIKELIHOOD/${base}.likelihood
                 touch $LIKELIHOOD/${base}.likelihood
 	done
 fi
 #--------------------------------------------------------------------------------------------------------------------------------
 # STEP 5 - Merge de tous les scores caclules => res (ctm) avec scores/params utiliser dans la classif
 # Format :
 #   mot    NbNode MinNode MaxNode MeanNode VarNode SVarNode Posterior AcousticLogLikelihood AcousticLogLikelihood/Frame ...
 #	   AcousticConfidenceLikelihood AcousticConstraintLikeLihood AcousticNoConstraint Likelihood ClasseRepliLinguistique ...
 #          RepliLinguistique LogLinguistique LogUnigramme NbMotsFenetre NbNulNode NbTrame
 #---------------------------------------------------------------------------------------------------------------------------------
 if [ $EXTRACT == 1 ]
 then
 	echo "EXTRACT step..."
 	rm -f $SUPER_CTM/* > /dev/null 2>&1
 	for file in `ls $1/$FICHIER_RES/*.res`
 	do
 		base=`basename $file .res`;
 		like=`echo "$base" | sed -e 's/\..*//'`;
 		if [ -f $1/$FICHIER_RES/$base.treil ]; then
 			echo "$ROOT/scripts/ExtractData.pl $pathML $nameML $POS/${base}.pos2 $file $LIKELIHOOD/${like}.likelihood $MLCLASS/${base}.mlclass $TYPE_ML >  $SUPER_CTM/${base}.ctm";
 			$ROOT/script/ExtractData.pl $pathML $nameML $POS/${base}.pos2 $file $LIKELIHOOD/${like}.likelihood $MLCLASS/${base}.mlclass $TYPE_ML >  $SUPER_CTM/${base}.ctm
 #			$ROOT/script/ExtractData.pl $pathML $nameML $POS/${base}.pos2 $file $LIKELIHOOD/${like}.likelihood $MLCLASS/${base}.mlclass $TYPE_ML >  $SUPER_CTM/${base}.ctm
 		fi
 	done
 fi
 #----------------------------------------------------------------
 # STEP 6 - Calcul effectif du score de confiance pour chaque mot
 #----------------------------------------------------------------
 if [ $BOOST == 1 ]
 then
 	echo "BOOST step..."
 	rm -f $SCORED_CTM/* $CONF_DIR/${NAME}.sctm $CONF_DIR/${NAME}.boost* $CONF_DIR/${NAME}.resboost* $CONF_DIR/${NAME}.corres* > /dev/null 2>&1
 	# utilise pour le test sans etiquette
 	$ROOT/script/DissociateErroneousFromDecoded.pl $SUPER_CTM 2 equilibre > $CONF_DIR/${NAME}.sctm
 	$ROOT/script/ConvertSuperCTMtoDataSVM.pl $CONF_DIR/${NAME}.sctm boost 2 0 0 > $CONF_DIR/${NAME}.boost
 	$ROOT/script/ConvertSuperCTMtoDataSVM.pl $CONF_DIR/${NAME}.sctm boost 2 0 1 > $CONF_DIR/${NAME}.boost_refs
 	$BOOST_BIN -S $ROOT/TRAIN -C --posteriors < $CONF_DIR/${NAME}.boost  > $CONF_DIR/${NAME}.resboost
 	cat $CONF_DIR/${NAME}.resboost | cut -f4 -d" " > $CONF_DIR/${NAME}.resboost2
 	cat $CONF_DIR/${NAME}.boost_refs |  sed -e 's/.*ref=//' > $CONF_DIR/${NAME}.corres
 	paste $CONF_DIR/${NAME}.corres $CONF_DIR/${NAME}.resboost2 | sed -e 's/\.ctm/\.res/' > $CONF_DIR/${NAME}.corres2
 	$ROOT/script/AssociateScoreToCtm.pl $CONF_DIR/${NAME}.corres2 $1/$FICHIER_RES/ $SCORED_CTM/
 fi
 echo "END"