Jean-François Rey / otmedia

Browse Code »

Commit 665a8dac322f0a4232d39c379136a945f4d76081

Authored by Jean-François Rey 2013-08-05 16:54:37 +0200

1 parent b9a54507e8

Exists in master

! follow the white rabbit !

Showing 6 changed files with 232 additions and 28 deletions Inline Diff

HOWTO
INSTALL
README
SOLR.INSTALL
install.sh
main_tools/ExploitConfidencePass.sh

HOWTO

Diff comments View file @ 665a8da

		1	#---------------#
		2	# OTMEDIA LIA #
		3	# HOWTO #
		4	# version 1.0 #
		5	#---------------#
		6
		7	1\ Main options
		8	---------------
		9
		10	There are five main options for otmedia scripts.
		11	-h : for help
		12	-D : Debug mode
		13	-v n : Verbose mode 1 low to 3 high
		14	-c : Check results
		15	-r : force to rerun a script, without deleting work already done
		16
		17	2\ Main scripts
		18	---------------
		19	2.1\ FirstPass.sh
		20	-----------------
		21
		22	FirstPass.sh do speaker diarization and transcription of an audio file. Convert it into wav format if not already done (16000Hz, 16 bits, mono).
		23	If a .SRT file is present in the same directory of the audio file it will copy it.
		24
		25	$> FisrtPass.sh [options] 110624FR2_20002100.wav result_directory
		26
		27	Options:
		28	-f n : number of forks for speeral
		29
		30	Output : result_directory/110624FR2_20002100/res_p1/
		31
		32	2.2\ SecondPass.sh
		33	------------------
		34
		35	SecondPass.sh do speaker adaptation and transcriptions base on the first pass.
		36
		37	$> SecondPass.sh [options] result_directory/110624FR2_20002100/
		38
		39	Options:
		40	-f n : number of forks for speeral
		41
		42	Output : result_directory/110624FR2_20002100/res_p2/
		43
		44	2.3\ ConfPass.sh
		45	----------------
		46
		47	ConfPass.sh do confidence measure using the second or third pass.
		48
		49	$> Confpass.sh [options] result_directory/110624FR2_20002100/ <res_p2\|res_p3>
		50
		51	Output : result_directory/110624FR2_20002100/conf/res_p2/scored_ctm/
		52	and result_directory/110624FR2_20002100.usf file
		53
		54	2.4\ ExploitConfidencePass.sh
		55	-----------------------------
		56
		57	It exploits confidence pass measure to :
		58	- boost confidente zone
		59	- find alternative in non confidente zone (using SOLR DB)
		60	- extend the lexicon
		61
		62	$> ExploitConfidencePass.sh [options] result_directory/110624FR2_20002100
		63
		64	Output : result_directory/110624FR2_20002100/trigg/speeral
		65	result_directory/110624FR2_20002100/LEX/speeral/_ext
		66
		67	2.5\ ThirstPass.sh
		68	------------------
		69
		70	ThirdPass.sh do transcriptions using SecondPass speaker adaptation and ExploitConfidencePass trigg files and new lexicon.
		71
		72	$> ThirdPass.sh [options] result_directory/110624FR2_20002100/
		73
		74	Options :
		75	-f n : number of forks for speeral
		76
		77	Output : result_directory/110624FR2_20002100/conf/res_p3
		78
		79	2.6\ RecomposePass.sh
		80	--------------------
		81
		82	RecomposePass.sh copy results that missing in ThirsPass from the Second and First Pass.
		83
		84	$> RecomposePass.sh [options] result_directory/110624FR2_20002100/
		85
		86	Output : result_directory/110624FR2_20002100/res_all
		87
		88	2.7\ ScoringRes.sh
		89	------------------
		90
		91	ScoringRes.sh run differents scoring tools to score the results using SRT file if exists.
		92
		93	$> ScoringRes.sh [options] result_directory/110624FR2_20002100/
		94
		95	Output : result_directory/110624FR2_20002100/scoring
		96
		97	2.8\ CheckResults.sh
		98	--------------------
		99
		100	CheckResults.sh parse results directories to synthesize works already done.
		101
		102	$> CheckResults.sh [options] result_directory
		103
		104	Output : "Directory name #plp #res_p1 #treil_p2 #treil_p3 usf_p2 usf_p3"
		105	#plp number of plp files
		106	#res_p1 number of .res files at first pass
		107	#treil_p2 number of .treil files at second pass
		108	#treil_p3 number of .treil files at third pass
		109	usf_p2 usf file from confidence pass result on second pass (OK\|ERR\|NAN)
		110	usf_p3 usf file from confidence pass result on third pass (OK\|ERR\|NAN)
		111
		112	3\ OneScriptToRuleThemAll.sh
		113	----------------------------
		114
		115	The script to do all OTMEDIA LIA pass in one call.
		116
		117	$> OneScriptToRuleThemAll.sh [options] 110624FR2_20002100.wav result_directory
		118
		119	Options : (default options are availables)
		120	-a Do every pass
		121	-1 Do First pass
		122	-2 Do Second pass
		123	-3 Do Third pass
		124	-C Do Confidence pass
		125	-e Do Exploit Confidence pass
		126	-R Do Recompose pass
		127	-s Do Scoring pass
		128
		129

INSTALL

Diff comments View file @ 665a8da

File was created	1	#---------------#
	2	# OTMEDIA LIA #
	3	# INSTALL #
	4	# version : 1.0 #
	5	#---------------#
	6
	7	OTMEDIA LIA ready to use ? Really ?
	8	No ! You have to do manualy configuartion for some features.
	9	Let see...
	10
	11	SUMMARY
	12	-------
	13
	14	1\ Before installation
	15	2\ install.sh script
	16	3\ SOLR install
	17
	18
	19	1\ Before installation
	20	----------------------
	21
	22	- Check and install dependencies.
	23	- In 64 bits architcture be sure you can run 32 bits programs.
	24	- Have 300 Go of free space.
	25	- Have acces to the network and the nyx server.
	26
	27	2/ install.sh script
	28	--------------------
	29
	30	install.sh script will do most of the work.
	31	It will check dependencies and configure pass tools.
	32	By default it will do a complet install (300 Go).
	33
	34	You can modifiy behavior by editing install.sh :
	35
	36	To disable lexicon adaption using SOLR DB put EXPLOITCONFPASS to 0 (mainly the 290 Go).
	37	To disable confidence measure put CONFPASS to 0.
	38	To disable second and third pass put PASS2 to 0.
	39
	40	run install.sh and follow the white rabbit.
	41
	42	3\ SOLR install
	43	---------------
	44
	45	The install.sh script download otmedia-2013-04.tar.gz and untar it in OTMEDIA_HOME/tools/SOLR/ .
	46	See SOLR.INSTALL file to install OTMEDIA SOLR DB.
	47
	48
	49
	50
	51
	52
	53
	54
	55
	56
	57
	58
	59
	60
	61
	62
	63
	64

README

Diff comments View file @ 665a8da

              ___ _____ __  __ _____ ____ ___    _      _     ___    _
             / _ \_   _|  \/  | ____|  _ \_ _|  / \    | |   |_ _|  / \
            | | | || | | |\/| |  _| | | | | |  / _ \   | |    | |  / _ \
            | |_| || | | |  | | |___| |_| | | / ___ \  | |___ | | / ___ \
             \___/ |_| |_|  |_|_____|____/___/_/   \_\ |_____|___/_/   \_\
-#-------------------#
+#---------------#
-# OTMEDIA LIA       #
+#  OTMEDIA LIA  #
-#   README          #
+#    README     #
-# version 1.0       #
+#  version 1.0  #
-#-------------------#
+#---------------#
 DESCRIPTION
 -----------
     OTMEDIA means "Observatoire Transmedia", its main objective is to study the evolution and transformation of the media world.
     The scientific objective of the project is the creation of a new generation of media observatory
     based on an interactive automatic analysis system (semi-automatic) transmedia to understand
     the world of information and developments.
     Web Site : http://www.otmedia.fr
     OTMEDIA LIA project is a set of tools to transcribe radio and TV shows.
+    It does multiple things :
+    - First pass : default transcription with speeral and speaker diarization.
+    - Second pass : speaker adaptation and a second transcription pass with speeral.
+    - Confidence pass : calcul confidence measure from transcription output.
+    - Exploit Confidence Measure : use SOLR DB data to extend the lexicon on low confidence measure and create trigg files.
+    - Third pass : second pass using the new lexicon and trigg files.
 DEPENDENCIES
 ------------
 GNU Toolchain
         Available from : http://www.gnu.org
         and debian packages
         Compiling, linking, and building applications.
 avconv (libav-tools >= 0.8)
     Available from : http://libav.org
     and debian package
     avconv is a very fast video and audio converter.
 JAVA JDK and JRE ( >= 6)
     Available from : http://www.oralce.com
     and debian packages
     JAVA Developpment kit and JAVA runtime environment.
 Python ( >= 2.7.0)
     Available from : http://http://www.python.org/
     and debian packages
     Python is a programming language.
 Perl ( >= 5.0.0)
     Available from : http://www.perl.org/
     and debian packages
     Perl is a programming language.
-iconvi ( >= 2.0.0)
+iconv ( >= 2.0.0)
     Available from : http://www.gnu.org
     and debian package
     Character set conversion.
 csh shell (csh)
     Available on debian packages.
     The C shell was originally written at UCB to overcome limitations in the
     Bourne shell.  Its flexibility and comfort (at that time) quickly made it
     the shell of choice until more advanced shells like ksh, bash, zsh or
     tcsh appeared.  Most of the latter incorporate features original to csh
 The SRI Language Modeling Toolkit (SRILM >= 1.6.0)
     Available from : http://www.speech.sri.com/projects/srilm/
     SRILM is a toolkit for building and applying statistical language models.
 Tomcat ( >= 7.0.0)
     Available from : http://tomcat.apache.org/
     and debian packages
     Apache Tomcat is an open source software implementation of the Java Servlet and JavaServer Pages technologies.
 INSTALLATION
 ------------
     See the INSTALL file for the installation procedure.
     Quick install below.
-    Before launch installation :
+    Before launching installation :
       Be certain that all dependencies are satisfied.
+      Have 300 Go of free space for complet install.
     Issue the following commands to the shell :
     $> ./install.sh
     $> export OTMEDIA_HOME=path/to/OTMEDIA/directory
-    Read SOLR.INSTALL part 3/ to install SOLRDB.
+    Read SOLR.INSTALL part 3 to install SOLRDB.
 RUNNING
 -------
     See HOWTO file.
 ACKNOWLEDGEMENTS
 ----------------
     Many thanks to Jean-François Rey for useful help and work done.
 KNOWN BUGS
 ----------
     Many.
+    For Bug report, please contact Pascal Nocera at pascal.nocera@univ-avignon.fr
 COPYRIGHT
 ---------
     See the COPYING file.
 AUTHORS
 -------
     Jean-François Rey <jean-francois.rey@univ-avignon.fr>
     Hugo Mauchrétien <hugo.mauchretien@univ-avignon.fr>
     Emmanuel Ferreira <emmanuel.ferreira@univ-avignon.fr>

SOLR.INSTALL

Diff comments View file @ 665a8da

 ################
 # SOLR INSTALL #
 ################
 #
 # Author Jean-François Rey
 # Version : 1.0
 # Date : 18/07/2013
 #
 1/ Edit install.sh and put CONFPASS=1
 2/ Run install.sh, this will check tomcat is installed, download and untar otmedia SOLR DB and ask for solr service info.
 3/ Configure Tomcat and SOLR
-SOLR_OTMEDIA_PATH=OTMEDIA_PATH/tools/SOLR/otemdia-2013-04
+otmedia-2013-04 SOLR DB is untar in :
+SOLR_OTMEDIA_PATH=OTMEDIA_HOME/tools/SOLR/otemdia-2013-04
     3.1/ Set context file
          ----------------
     - in SOLR_OTMEDIA_PATH/solr/otmedia-document/solr-tomcat-deploy/solr-otmedia-document.xml
     change DocBase to  DocBase="SOLR_OTMEDIA_PATH/solr/otmedia-document/apache-solr-3.5.0.war"
     and value to value="SOLR_OTMEDIA_PATH/solr/otmedia-document/"
     - in SOLR_OTMEDIA_PATH/solr/otmedia-multimedia/solr-tomcat-deploy/solr-otmedia-multimedia.xml
     change DocBase to  DocBase="SOLR_OTMEDIA_PATH/solr/otmedia-multimedia/apache-solr-3.5.0.war"
     and value to value="SOLR_OTMEDIA_PATH/solr/otmedia-multimedia/"
     3.2/ SOLR data configuration
          -----------------------
     - in SOLR_OTMEDIA_PATH/solr/otmedia-document/conf/solrconfig.xml
     change datadir (solr.data.dir) to SOLR_OTMEDIA_PATH/index/otmedia-document/
     - in SOLR_OTMEDIA_PATH/solr/otmedia-multimedia/conf/solrconfig.xml
     change datadir (solr.data.dir) to SOLR_OTMEDIA_PATH/index/otmedia-multimedia/
     3.3/ Add SOLR DB to Tomcat
          ---------------------
     - in tomcat/Catalina/localhost/ (mainly in /etc/tomcat/Catalina/localhost or /var/lib/tomcat/conf/Catalina/localhost)
     run : $> ln -s SOLR_OTMEDIA_PATH/solr/otmedia-document/solr-tomcat-deploy/solr-otmedia-document.xml solr-otmedia-document.xml
     run : $> ln -s SOLR_OTMEDIA_PATH/solr/otmedia-multimedia/solr-tomcat-deploy/solr-otmedia-multimedia.xml solr-otmedia-document.xml
 4/ Tomcat trouble
     4.1/ SOLR use a lot of memory, you need to increase java heap space !
          -------------------------
     - in catalina.sh (/usr/share/tomcat/bin)
     add CATALINA_OPTS="$CATALINA_OPTS -Xms256 -Xmx512m"
     4.2/ Directory permissions
          ---------------------
     - SOLR_OTMEDIA_PATH and subdirectory (and files) need to belong to tomcat group (and tomcat user if the default user don't belong to tomcat group).
     chgrp -r tomcat7 otmedia-2013-04
     chmod g+rx otmedia-2013-04
 5/ Test
 You can test those requests (change ip and port):
 http://localhost:8080/solr-otmedia-multimedia/select?q=test+bonus+&fq=docDate:[2011-12-30T00\:00\:01Z+TO+2012-01-01T23\:59\:59Z]
 http://localhost:8080/solr-otmedia-document/select?q=test+bonus+&fq=docDate:[2011-12-30T00\:00\:01Z+TO+2012-01-01T23\:59\:59Z]

install.sh

Diff comments View file @ 665a8da

 #!/bin/bash
 #-------------------#
+# OTMEDIA LIA       #
 # Install script    #
-# OTMEDIA           #
+# version : 1.0.0   #
 #-------------------#
 # Color variables
 txtgrn=$(tput setaf 2) # Green
 txtylw=$(tput setaf 3) # Yellow
 txtblu=$(tput setaf 4) # Blue
 txtpur=$(tput setaf 5) # Purple
 txtcyn=$(tput setaf 6) # Cyan
 txtwht=$(tput setaf 7) # White
 txtrst=$(tput sgr0) # Text reset.
 #/color
 #
 ### Global Variables
 #
 PWD=$(pwd)
 OTMEDIA_HOME=$PWD
 test=$(arch)
 if [ "$test" == "x86_64" ]; then ARCH=".64"; else ARCH=""; fi
 #/Global
 #
 # Put to 0 to disable dependencies of a pass
 # and 1 to enable
 #
 PASS1=1     # First Pass
-PASS2=1     # Second Pass
+PASS2=1     # Second and Third Pass
 CONFPASS=1  # Confidence Pass
 EXPLOITCONFPASS=1   # SOLR query and trigg
 echo -e "\nWill do install for :"
 if [ $PASS1 -eq 1 ];then echo "- Pass 1";fi
 if [ $PASS2 -eq 1 ];then echo "- Pass 2";fi
 if [ $CONFPASS -eq 1 ];then echo "- Confidence Pass";fi
 if [ $EXPLOITCONFPASS -eq 1 ];then echo "- Exploit Confidence Pass";fi
 #
 ### CHECK Dependencies ###
 #
 echo -e "\n\t${txtblu}Check Dependencies${txtrst}\n"
 ## make
 test=$(whereis make)
 if [ "$test" == "make:" ]
 then
     echo -e "${txtpur}ERROR${txtrst} make not found\n You have to install make\n sudo apt-get install make"
     exit 1;
 fi
 echo -e "make \t ${txtgrn}OK${txtrst}"
 ## CC
 test=$(whereis cc)
 if [ "$test" == "cc:" ]
 then
     echo -e "${txtpur}ERROR${txtrst} cc not found\n You have to install cc\n sudo apt-get install gcc"
     exit 1;
 fi
 echo -e "cc \t ${txtgrn}OK${txtrst}"
 ## AVCONV
 test=$(whereis avconv)
 if [ "$test" == "avconv:" ]
 then
     echo -e "${txtpur}ERROR${txtrst} avconv not found\n You have to install avconv\n sudo apt-get install libav-tools"
     exit 1;
 fi
 echo -e "libav-tools : avconv \t ${txtgrn}OK${txtrst}"
 ## JAVA
 test=$(whereis java)
 if [ "$test" == "java:" ]
 then
     echo -e "${txtpur}ERROR${txtrst} java not found\n You have to install java JRE\n sudo apt-get install openjdk-7-jre"
     exit 1;
 fi
 echo -e "Java : JRE \t ${txtgrn}OK${txtrst}"
 test=$(whereis javac)
 if [ "$test" == "javac:" ]
 then
     echo -e "${txtpur}ERROR${txtrst} javac not found\n You have to install java JDK\n sudo apt-get install openjdk-7-jdk"
     exit 1;
 fi
 echo -e "Java : JDK \t ${txtgrn}OK${txtrst}"
 if [ $EXPLOITCONFPASS -eq 1 ]
 then
     ## Python
     test=$(whereis python)
     if [ "$test" == "python:" ]
     then
         echo -e "${txtpur}ERROR${txtrst} python not found\n You have to install python\n sudo apt-get install python"
         exit 1;
     fi
     echo -e "python : \t ${txtgrn}OK${txtrst}"
+    ## csh shell
+    test=$(whereis csh)
+    if [ "$test" == "csh:" ]
+    then
+        echo -e "${txtpur}ERROR${txtrst} csh shell not found\n You have to install csh shell\n sudo apt-get install csh"
+        exit 1;
+    fi
+    echo -e "csh shell : \t ${txtgrn}OK${txtrst}"
 fi
 ## Perl
 test=$(whereis perl)
 if [ "$test" == "perl:" ]
 then
     echo -e "${txtpur}ERROR${txtrst} perl not found\n You have to install perl\n sudo apt-get install perl"
     exit 1;
 fi
 echo -e "perl : \t ${txtgrn}OK${txtrst}"
 ## iconv
 test=$(whereis iconv)
 if [ "$test" == "iconv:" ]
 then
     echo -e "${txtpur}ERROR${txtrst} iconv not found\n You have to install iconv\n sudo apt-cache search iconv"
     exit 1;
 fi
 echo -e "iconv : \t ${txtgrn}OK${txtrst}"
-## csh shell
-test=$(whereis csh)
-if [ "$test" == "csh:" ]
-then
-    echo -e "${txtpur}ERROR${txtrst} csh shell not found\n You have to install csh shell\n sudo apt-get install csh"
-    exit 1;
-fi
-echo -e "csh shell : \t ${txtgrn}OK${txtrst}"
 ## SRI LM
 if [ -z "$SRILM" ] && [ -z "$MACHINE_TYPE" ]
 then
     echo -e "${txtpur}ERROR${txtrst} SRILM toolkit variables are not defined (SRILM and MACHINE_TYPE)\n You have to install SRILM Toolkit\n"
     exit 1;
 fi
 export SRILM_BIN=$SRILM/bin/$MACHINE_TYPE
 echo -e "SRILM toolkit : \t ${txtgrn}OK${txtrst}"
 ### Speeral Configuration ###
 echo -e "\n\t${txtblu}Speeral configuration${txtrst}\n"
 echo -e "Download Speeral bin and data :"
 scp -r rey@nyx:~/OTMEDIA_DATA/Speeral $OTMEDIA_HOME/tools/
 echo -e "\n\t${txtblu}Generating Speeral configuration files :${txtrst}\n"
 cat $PWD/tools/Speeral/CFG/SpeeralFirstPass.xml.tmp | sed -e "s|<nom>[^<]*</nom>|<nom>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer</nom>|g" \
     | sed -e "s|<ngramme>[^<]*</ngramme>|<ngramme>$PWD/tools/Speeral/LM/ML_4gOTMEDIA_LEXIQUE_V6</ngramme>|g" \
     | sed -e "s|<binode>[^<]*</binode>|<binode>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer.bin</binode>|g" \
     > $PWD/tools/Speeral/CFG/SpeeralFirstPass.xml
 echo $PWD/tools/Speeral/CFG/SpeeralFirstPass.xml
 cat $PWD/tools/Speeral/CFG/SpeeralSecondPass.xml.tmp | sed -e "s|<nom>[^<]*</nom>|<nom>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer</nom>|g" \
     | sed -e "s|<ngramme>[^<]*</ngramme>|<ngramme>$PWD/tools/Speeral/LM/ML_4gOTMEDIA_LEXIQUE_V6</ngramme>|g" \
     | sed -e "s|<binode>[^<]*</binode>|<binode>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer.bin</binode>|g" \
     > $PWD/tools/Speeral/CFG/SpeeralSecondPass.xml
 echo $PWD/tools/Speeral/CFG/SpeeralSecondPass.xml
 cat $PWD/tools/Speeral/CFG/SpeeralThirdPass.xml.tmp | sed -e "s|<nom>[^<]*</nom>|<nom>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer</nom>|g" \
     | sed -e "s|<ngramme>[^<]*</ngramme>|<ngramme>$PWD/tools/Speeral/LM/ML_4gOTMEDIA_LEXIQUE_V6</ngramme>|g" \
     | sed -e "s|<binode>[^<]*</binode>|<binode>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer.bin</binode>|g" \
     > $PWD/tools/Speeral/CFG/SpeeralThirdPass.xml
 echo $PWD/tools/Speeral/CFG/SpeeralThirdPass.xml
 if [ $EXPLOITCONFPASS -eq 1 ]
 then
     ### LIA ltbox ###
     echo -e "\t${txtblu}Install lia_ltbox${txtrst}\n"
     export LIA_TAGG_LANG="french"
     export LIA_TAGG="$OTMEDIA_HOME/tools/lia_ltbox/lia_tagg/"
     export LIA_PHON_REP="$OTMEDIA_HOME/tools/lia_ltbox/lia_phon/"
     export LIA_BIGLEX="$OTMEDIA_HOME/tools/lia_ltbox/lia_biglex/"
     ### config lia_phon
     cd $LIA_PHON_REP
     make all > /dev/null
     make ressource > /dev/null
     ### config lia_tagg
     cd $LIA_TAGG
     make all > /dev/null
     make ressource.french > /dev/null
     ### config lia_biglex
     cd $LIA_BIGLEX
     make -f makefile.biglex > /dev/null
     cd $OTMEDIA_HOME
     ### SOLR DB ###
     # Tomcat fisrtly
     test=$(dpkg -l | grep "^ii" | grep tomcat)
     if [ "$test" == "" ]
     then
         echo -e "${txtpur}ERROR${txtrst} TOMCAT seems to not be installed)\n You have to install TOMCAT\n"
-        exit 1;
+        #exit 1;
     fi
     echo -e "\nTOMCAT : \t ${txtgrn}OK${txtrst}\n"
     # SOLR secondly
     echo -e "\t${txtblu}Install SOLR DB${txtrst}\n"
     echo -e "You will need 300 Go of free space to install SOLR DB"
     read -p "Continue ? (y/n) " solr
     if [ "$solr" == "y" ]
     then
         echo -e "Download SOLR DB\r"
         mkdir -p $OTMEDIA_HOME/tools/SOLR 2> /dev/null
         scp -r rey@nyx:~/OTMEDIA_DATA/SOLR/otmedia-2013-04.tar.gz $OTMEDIA_HOME/tools/SOLR
         echo -e "Unzip SOLR DB\r"
         res=0
         #res = $(tar -xvzf "$OTMEDIA_HOME/tools/SOLR/otmedia-2013-04.tar.gz" "$OTMEDIA_HOME/tools/SOLR/")
         if [ $res -eq 2 ]; then echo " ${txtpur}NOT OK${txtrst}";
         else echo " ${txtgrn}OK${txtrst}"; fi
     else
         echo "Skipping SOLR install"
     fi
     read -e -p "Configure SOLR DB server ? (y/n) " solr
     if [ "$solr" == "y" ]
     then
         read -p "Enter SOLR server IP :" ip
         if [ "${ip}" == "" ];then ip="localhost";fi
         echo "machine = \"${ip}\"" > $OTMEDIA_HOME/tools/scripts/solrinfo.py
         read -p "Enter SOLR server port :" port
         if [ "${port}" == "" ]; then port="8080";fi
         echo -e "\n\tSOLR server IP ${ip}"
         echo -e "\tSOLR server port ${port}"
         echo "port = \"${port}\"" >> $OTMEDIA_HOME/tools/scripts/solrinfo.py
     else
         echo "Skipping SOLR DB Configuration"
     fi
     echo -e "\nSee SOLR.INSTALL file for more information\n"
 fi
 ### Set Variables in bashrc ###
 cat ~/.bashrc | grep -v "OTMEDIA_HOME" | grep -v "SRILM_BIN" > ~/.bashrc.org
 #cat ~/.bashrc | grep -v "OTMEDIA_HOME" | grep -v "SRILM_BIN" | grep -v "LIA_TAGG" | grep -v "LIA_PHON" | grep -v "LIA_BIGLEX" > ~/.bashrc.org
 cp ~/.bashrc.org ~/.bashrc
 export OTMEDIA_HOME=$PWD
 echo "export OTMEDIA_HOME=$PWD" >> ~/.bashrc
+echo "export $PATH=$PATH:$PWD/main_tools" >> ~/.bashrc
 echo "export SRILM_BIN=$SRILM/bin/$MACHINE_TYPE" >> ~/.bashrc
 #echo "export LIA_TAGG_LANG=french" >> ~/.bashrc
 #echo "export LIA_TAGG=$OTMEDIA_HOME/tools/lia_ltbox/lia_tagg/" >> ~/.bashrc
 #echo "export LIA_PHON_REP=$OTMEDIA_HOME/tools/lia_ltbox/lia_phon/" >> ~/.bashrc
 #echo "export LIA_BIGLEX=$OTMEDIA_HOME/tools/lia_ltbox/lia_biglex/" >> ~/.bashrc
 # set global configuration file
 echo "OTMEDIA_HOME=$PWD" > $OTMEDIA_HOME/cfg/main_cfg.cfg
 echo "ARCH=$ARCH" >> $OTMEDIA_HOME/cfg/main_cfg.cfg
 echo "VERBOSE=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg
 echo "DEBUG=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg
 echo "CHECK=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg
 echo "RERUN=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg
 echo -e "\n\t${txtgrn}### Install completed ###${txtrst}\n"
 echo -e "do : source ~/.bashrc"
 echo -e "or set variable :\n"
 echo "export OTMEDIA_HOME=$PWD"

main_tools/ExploitConfidencePass.sh

Diff comments View file @ 665a8da

 #!/bin/bash
 #####################################################
 # File :    ExploitConfidencePass.sh                #
 # Brief :   Exploit the ASR confidence pass to :    #
 #           -> boost the confident zone             #
 #           -> find alternative in non confident zone
 #           -> dynamicly extend the lexicon         #
 # Author :  Jean-François Rey                       #
 #	        (base on Emmanuel Ferreira              #
 #	        and Hugo Mauchrétien works)             #
 # Version : 1.0                                     #
 # Date :    25/06/13                                #
 #####################################################
 echo "### ExploitConfidencePass.sh ###"
 # Check OTMEDIA_HOME env var
 if [ -z ${OTMEDIA_HOME} ]
 then
     OTMEDIA_HOME=$(dirname $(dirname $(readlink -e $0)))
     export OTMEDIA_HOME=$OTMEDIA_HOME
 fi
 # where is ExploitConfidencePass.sh
 MAIN_SCRIPT_PATH=$(dirname $(readlink -e $0))
 if [ -z ${SCRIPT_PATH} ]
 then
     SCRIPT_PATH=$OTMEDIA_HOME/tools/scripts
 fi
 # Include scripts
 . $SCRIPT_PATH"/Tools.sh"
 . $SCRIPT_PATH"/CheckExploitConfPass.sh"
 # where is ExploitConfidencePass.cfg
 EXPLOITCONFIDENCEPASS_CONFIG_FILE=$OTMEDIA_HOME"/cfg/ExploitConfidencePass.cfg"
 if [ -e $EXPLOITCONFIDENCEPASS_CONFIG_FILE ]
 then
 	. $EXPLOITCONFIDENCEPASS_CONFIG_FILE
 else
 	echo "ERROR : Can't find configuration file $EXPLOITCONFIDENCEPASS_CONFIG_FILE" >&2
 	exit 1
 fi
 #---------------#
 # Parse Options #
 #---------------#
-while getopts ":hDv:cf:r" opt
+while getopts ":hDv:cr" opt
 do
 	case $opt in
 		h)
 			echo -e "$0 [OPTIONS] <INPUT_DIRECTORY>\n"
             echo -e "\t Options:"
             echo -e "\t\t-h :\tprint this message"
             echo -e "\t\t-D :\tDEBUG mode on"
             echo -e "\t\t-v l :\tVerbose mode, l=(1|2|3) level mode"
             echo -e "\t\t-c :\tCheck process, stop if error detected"
-            echo -e "\t\t-f n :\tspecify a speeral forks number (default 1)"
             echo -e "\t\t-r n :\tforce rerun without deleting files"
 			exit 1
 			;;
 		D)
 			DEBUG=1
 			;;
         v)
             VERBOSE=$OPTARG
             ;;
         c)
             CHECK=1
-            ;;
-        f)
-            FORKS="--forks $OPTARG"
             ;;
         r)
             RERUN=1
             ;;
 		:)
 			echo "Option -$OPTARG requires an argument." >&2
 			exit 1
 			;;
 		\?)
 			echo "BAD USAGE : unknow opton -$OPTARG"
 			#exit 1
 			;;
 	esac
 done
 # mode debug enable
 if [ $DEBUG -eq 1 ]
 then
        set -x
        echo -e "## Mode DEBUG ON ##"
 fi
 # mode verbose enable
 if [ $VERBOSE -gt 0 ]; then echo -e "## Verbose level : $VERBOSE ##" ;fi
 # Check USAGE by arguments number
 if [ $(($#-($OPTIND-1))) -ne 1 ]
 then
 	echo "BAD USAGE : ExploitConfidencePass.sh [OPTIONS] <INPUT_DIRECTORY>"
 	echo "$0 -h for more info"
 	exit 1
 fi
 shift $((OPTIND-1))
 # check input directory - first argument
 if [ ! -e $1 ]
 then
     print_error "can't open $1"
     exit 1
 fi
 print_info "[${BASENAME}] => ExploitConfPass start | $(date +'%d/%m/%y %H:%M:%S')" 1
 #-------------#
 # GLOBAL VARS #
 #-------------#
 INPUT_DIR=$(readlink -e $1)
 OUTPUT_DIR=$INPUT_DIR
 BASENAME=$(basename $OUTPUT_DIR)
 SHOW_DIR="$OUTPUT_DIR/shows/"
 SOLR_RES="$OUTPUT_DIR/solr/"
 EXT_LEX="$OUTPUT_DIR/LEX/"
 TRIGGER_CONFZONE="$OUTPUT_DIR/trigg/"
 LOGFILE="$OUTPUT_DIR/info_exploitconf.log"
 ERRORFILE="$OUTPUT_DIR/error_exploitconf.log"
 CONFPASS_CONFIG_FILE="$(readlink -e $1)/ConfPass.cfg"
 if [ -e $CONFPASS_CONFIG_FILE ]
 then
 {
     RES_CONF_DIR=$(cat $CONFPASS_CONFIG_FILE | grep "^RES_CONF_DIR=" | cut -f2 -d"=")
     RES_CONF=$(cat $CONFPASS_CONFIG_FILE | grep "^CONF_DIR=" | cut -f2 -d"=")
     print_info "[${BASENAME}] Use confidence measure from : $RES_CONF" 2
 }
 else
 {
     print_error "[${BASENAME}] Can't find $CONFPASS_CONFIG_FILE"
     print_error "[${BASENAME}] -> use res_p2"
     RES_CONF_DIR="$INPUT_DIR/conf/res_p2/scored_ctm"
     RES_CONF="$INPUT_DIR/conf/res_p2"
 }
 fi
 mkdir -p $SHOW_DIR > /dev/null 2>&1
 mkdir -p $SOLR_RES > /dev/null 2>&1
 mkdir -p $EXT_LEX > /dev/null 2>&1
 mkdir -p $TRIGGER_CONFZONE > /dev/null 2>&1
 #------------------#
 # Create Workspace #
 #------------------#
 # Lock directory
 if [ -e "$OUTPUT_DIR_BASENAME/EXPLOITCONFPASS.lock" ] && [ $RERUN -eq 0 ]
 then
     print_warn "[${BASENAME}] ExploitConfidencePass is locked -> exit" 2
     exit 1
 fi
 rm "$OUTPUT_DIR/EXPLOITCONFPASS.unlock" > /dev/null 2>&1
 touch "$OUTPUT_DIR/EXPLOITCONFPASS.lock" > /dev/null 2>&1
 #------#
 # Save #
 #------#
 cp $EXPLOITCONFIDENCEPASS_CONFIG_FILE $OUTPUT_DIR/ExploitConfPass.cfg
 echo "TRIGGER_DIR=$TRIGGER_CONFZONE" >> $OUTPUT_DIR/ExploitConfPass.cfg
 echo "TRIGGER_SPEERAL=$TRIGGER_CONFZONE/speeral/" >> $OUTPUT_DIR/ExploitConfPass.cfg
 echo "LEX_SPEERAL=$EXT_LEX/speeral/${lexname}_ext" >> $OUTPUT_DIR/ExploitConfPass.cfg
 echo "LEX_BINODE_SPEERAL=$EXT_LEX/speeral/${lexname}_ext.bin" >> $OUTPUT_DIR/ExploitConfPass.cfg
 print_info "[${BASENAME}] Save config in $OUTPUT_DIR_BASENAME/ExploitConfPass.cfg" 1
 #---------------#
 # Check Pass    #
 #---------------#
 if [ $( ls ${RES_CONF_DIR}/*.res 2> /dev/null | wc -l) -eq 0 ]
 then
     print_error "[${BASENAME}] No Conf Pass res -> exit ExploitConfPass"
     if [ $CHECK -eq 1 ]; then print_log_file $ERRORFILE "No ConfPass res in ${RES_CONF_DIR}" ;fi
     exit 1
 fi
 #-----------------------#
 # Segmentation by show  #
 #-----------------------#
 # create txt file from scored res
 # tag pos and lemmatization of the txt file
 # merge the scored res and taglem file
 # segment using the last generated file
 # and create a ctm file by show
 print_info "[${BASENAME}] Segmentation by show" 1
 # -> to txt
 print_info "[${BASENAME}] Create txt from scored res" 3
 cat ${RES_CONF_DIR}/*.res > $INPUT_DIR/$BASENAME.sctm
 cat $INPUT_DIR/$BASENAME.seg | $SIGMUND_BIN/myConvert.pl $INPUT_DIR/$BASENAME.sctm $INPUT_DIR/$BASENAME.tmp
 cat $INPUT_DIR/$BASENAME.tmp | $SCRIPT_PATH/BdlexUC.pl $RULES/basic -f | sed -e "s/_/ /g" | sort -nt 'n' -k '2' > $INPUT_DIR/$BASENAME.txt
 # -> to tagger + lemme
 print_info "[${BASENAME}] Tag pos and lem in txt file" 3
 iconv -t ISO_8859-1 $INPUT_DIR/$BASENAME.txt > $INPUT_DIR/$BASENAME.tmp
 $SIGMUND_BIN/txt2lem.sh $INPUT_DIR/$BASENAME.tmp $INPUT_DIR/$BASENAME.taglem
 # merge sctm and taglem
 print_info "[${BASENAME}] Merge scored ctm with tag pos and lem file" 3
 cat $INPUT_DIR/$BASENAME.sctm | $SCRIPT_PATH/BdlexUC.pl ${RULES}/basic -f | iconv -t ISO_8859-1 | $SCRIPT_PATH/scoredCtmAndTaggedLem2All.pl $INPUT_DIR/$BASENAME.taglem > $INPUT_DIR/$BASENAME.ctl
 # -> new seg
 print_info "[${BASENAME}] Create xml file and run Topic Seg" 3
 $SIGMUND_BIN/tagLem2xml.pl $INPUT_DIR/$BASENAME.taglem $INPUT_DIR/$BASENAME.doc.xml
 rm $INPUT_DIR/$BASENAME.tmp #$INPUT_DIR/$BASENAME.taglem
 # Lia_topic_seg : bring together sentences into show
 cp $INPUT_DIR/$BASENAME.doc.xml 0.xml
 java -cp $LIATOPICSEG/bin Test > $INPUT_DIR/show.seg
 cat $INPUT_DIR/show.seg | $SIGMUND_BIN/toSegEmiss.pl $INPUT_DIR/$BASENAME.show.seg
 rm 0.xml $INPUT_DIR/show.seg
 if [ $CHECK -eq 1 ]
 then
     if [ ! -s $INPUT_DIR/$BASENAME.show.seg ]
     then
         print_error "[${BASENAME}] No Topic segmentation ! "
         print_error "[${BASENAME}] Check $ERRORFILE "
         print_log_file "$ERRORFILE" "No Topic segmentation in ${BASENAME}.show.seg"
     fi
 fi
 # Segment ctm into several show files and create a seg list by show
 print_info "[${BASENAME}] Segment ctm into show files and a seg list by show" 1
 $SCRIPT_PATH/ctm2show.pl $INPUT_DIR/$BASENAME.ctl $INPUT_DIR/$BASENAME.show.seg $SHOW_DIR
 #-----------------------------------------------------------#
 # SOLR QUERIES                                              #
 # -> Create Confidente Word                                 #
 #   Keep conf words and use Tags                            #
 # -> Query SOLR (document & multimedia)                     #
 #   concat word + add date 2 day before and after the show  #
 #   query document & multimedia                             #
 #-----------------------------------------------------------#
 print_info "[${BASENAME}] Create SOLR queries and ask SOLR" 1
 for show in $(ls $SHOW_DIR/*.ctm)
 do
     bn=$(basename $show .ctm)
     # Remove words with low confidence and keep useful tagger words
     cat $show | $SCRIPT_PATH/KeepConfZone.pl | grep -e "MOTINC\|NMS\|NMP\|NFS\|NFP\|X[A-Z]{3,5}" | cut -f3 -d' ' > "$SHOW_DIR/$bn.confzone"
     # Get date 2 day before and after the show
     datePattern=`$SCRIPT_PATH/daybefore2after.sh $(echo $BASENAME | cut -c1-6)`
     # Create SOLR queries
     cat $SHOW_DIR/$bn".confzone" | $SCRIPT_PATH/GenerateSOLRQueries.pl | iconv -f ISO_8859-1 -t UTF-8 > "$SHOW_DIR/$bn.queries"
     # Ask SOLR DB
     if [ $(wc -w "$SHOW_DIR/$bn.queries" | cut -f1 -d' ') -gt 0 ]; then
         query=$(cat $SHOW_DIR/$bn.queries)"&fq=docDate:[$datePattern]"
         echo $query > $SHOW_DIR/$bn.queries
         print_info "python $SCRIPT_PATH/ProcessSOLRQueries.py $SHOW_DIR/$bn.queries $SOLR_RES/$bn.keywords.tmp $SOLR_RES/$bn.txt.tmp" 3
         python $SCRIPT_PATH/ProcessSOLRQueries.py $SHOW_DIR/$bn.queries $SOLR_RES/$bn.keywords.tmp $SOLR_RES/$bn.txt.tmp
         cat $SOLR_RES/$bn.keywords.tmp | sort -u > $SOLR_RES/$bn.keywords
         cat $SOLR_RES/$bn.txt.tmp | sort -u > $SOLR_RES/$bn.txt
         rm $SOLR_RES/*.tmp > /dev/null 2>&1
     fi
     if [ $CHECK -eq 1 ]
     then
         if [ ! -e $SOLR_RES/$bn.keywords ] || [ ! -e $SOLR_RES/$bn.txt ]
         then
             print_warn "$bn.keywords and $bn.txt are empty !\nMaybe SOLR server is down !" 2
             print_log_file "$LOGFILE" "$bn.keywords and $bn.txt are empty !\nMaybe SOLR server is down !"
         fi
     fi
 done
 #-----------------------------------------------------------------------------------------------
 #  Build trigger file
 #       1) keywords are automatically boosted in the non confident zone of the current res
 #          confident zone are boosted
 #          previous words in sensible zone are penalized
 #       2) OOVs are extracted + phonetized
 #       3) Try to find OOVs acousticly in the current segment
 #       4) Generate the .trigg file
 #------------------------------------------------------------------------------------------------
 print_info "[${BASENAME}] Build trigger files" 1
 for i in `ls $SOLR_RES/*.keywords`
 do
     basename=`basename $i .keywords`
     #
     # Tokenize & produce coverage report
     # Use filter you need
     #
     print_info "[${BASENAME}] keywords filtering and produce coverage report" 3
     # Default filter
     cat $i | $SCRIPT_PATH/CleanFilter.sh | ${SCRIPT_PATH}/ApplyCorrectionRules.pl ${LEXICON}.regex | $SCRIPT_PATH/BdlexUC.pl $RULES/basic -t |\
         $SCRIPT_PATH/CoverageReportMaker.pl --out $SOLR_RES/${basename}_tmp_report $LEXICON.bdlex_tok
     # do less filter
     #cat $i | $SCRIPT_PATH/BdlexUC.pl $RULES/basic -t | sed -f $RULES/preprocess.regex | sed -f $RULES/lastprocess.regex | $SCRIPT_PATH/CoverageReportMaker.pl --out $SOLR_RES/${basename}_tmp_report $LEXICON.bdlex_tok
     #
     # Extract "real" OOV and phonetize them
     # -> petit filtrage persoo pour eviter d'avoir trop de bruits
     #
     print_info "[${BASENAME}] Extract OOV and phonetize them" 3
     ${SCRIPT_PATH}/FindNormRules.pl $SOLR_RES/${basename}_tmp_report/report.oov $LEXICON.bdlex_tok | cut -f3 | grep -v "#" | grep -v "^[A-Z]\+$" | grep -v "^[0-9]" | grep --perl-regex -v "^([a-z']){1,3}$" | $SCRIPT_PATH/BdlexUC.pl $RULES/basic -f | iconv -t ISO_8859-1 -f UTF-8 | ${LIA_LTBOX}/lia_phon/script/lia_lex2phon_variante | grep -v "core dumped" | cut -d"[" -f1 | sort -u | ${SCRIPT_PATH}/PhonFormatter.pl | iconv -f ISO_8859-1 -t UTF-8 | $SCRIPT_PATH/BdlexUC.pl $RULES/basic -t > $SOLR_RES/${basename}.phon_oov
     #
     # Search INVOC & OOV in the current lattice
     #
     print_info "[${BASENAME}] Search INVOC and OOV in the current lattice" 3
     cat $SOLR_RES/${basename}_tmp_report/report.invoc | grep -v "\b0" | cut -f1 | grep -v --perl-regex -v "^[a-zA-Z']{1,3}$" | grep -v --perl-regex "^[a-zA-Z0-9]{1,3}$" | grep -v "<s>" | grep -v "</s>" | $SCRIPT_PATH/BdlexUC.pl $RULES/basic -t > $TRIGGER_CONFZONE/$basename.tosearch
     cat $SOLR_RES/${basename}.phon_oov | cut -f1 >>  $TRIGGER_CONFZONE/$basename.tosearch
     # For each treil
     for baseseg in $(cat "$SHOW_DIR/$basename.lst")
     do
         $OTMEDIA_HOME/tools/QUOTE_FINDER/bin/acousticFinder ${LEXICON}.speer_phon $RES_CONF/wlat/$baseseg.wlat $TRIGGER_CONFZONE/${basename}.tosearch $SOLR_RES/$basename.phon_oov > $TRIGGER_CONFZONE/$baseseg.acousticlyfound $OUTPUT_REDIRECTION
         #
         # Produce the boost file for the next decoding pass
         #
         print_info "[${BASENAME}] Produce trigg file : $baseseg " 3
         cat $RES_CONF_DIR/$baseseg.res | $SCRIPT_PATH/ScoreCtm2trigg.pl $TRIGGER_CONFZONE/$baseseg.acousticlyfound > $TRIGGER_CONFZONE/$baseseg.trigg
     done
 done
 #-----------------------------------------------------------------------------------------------
 # Build the extended SPEERAL Lexicon
 #   1) Merge OOVs + LEXICON
 #   1) Related text are collected in order to find the invoc word with maximizing the ppl (LM proba)
 #   2) The current lexicon is extended with all the valid OOVs
 #-----------------------------------------------------------------------------------------------
 print_info "[${BASENAME}] Build extended Speeral Lexicon" 1
 mkdir -p $EXT_LEX/final
 mkdir -p $EXT_LEX/tmp
 mkdir -p $EXT_LEX/tmp/txt
 #
 # Collect the acousticly found oov and their phonetisation
 #
 print_info "[${BASENAME}] Get all OOV and retrieve all phonetisation" 3
 for i in `ls $SOLR_RES/*.phon_oov`
 do
     basename=`basename $i .phon_oov`
     rm $EXT_LEX/$basename.acousticlyfound 2> /dev/null
     # list acousticly found for the show
     for baseseg in $(cat "$SHOW_DIR/$basename.lst")
     do
         cat $TRIGGER_CONFZONE/$baseseg.acousticlyfound | cut -f1 | cut -f2 -d"=" >> $EXT_LEX/$basename.acousticlyfound
     done
     cat $EXT_LEX/$basename.acousticlyfound | sort -u > $EXT_LEX/.tmp
     mv $EXT_LEX/.tmp $EXT_LEX/$basename.acousticlyfound
     #
     # Extract OOV really added
     #
     cat $SOLR_RES/$basename.phon_oov | cut -f1 | sort -u > $EXT_LEX/$basename.oov
     $SCRIPT_PATH/intersec.pl $EXT_LEX/$basename.oov $EXT_LEX/$basename.acousticlyfound > $EXT_LEX/$basename.oov_acousticlyfound
     #
     # Retrieve all phonetisation
     #
     cat $SOLR_RES/${basename}.phon_oov | $SCRIPT_PATH/LexPhonFilter.pl $EXT_LEX/$basename.oov_acousticlyfound > $EXT_LEX/$basename.oov_acousticlyfound_phon
 done
 #
 # Merge OOVs and their phonetisation
 #
 print_info "[${BASENAME}] Merge OOV and their phonetisation" 3
 lexname=$(basename $LEXICON)
 cat $EXT_LEX/*.oov_acousticlyfound_phon | sort -u > $EXT_LEX/final/all.oov_acousticlyfound_phon
 cat $EXT_LEX/*.oov_acousticlyfound | sort -u | grep --perl-regex -v "^([a-z']){3}$"  > $EXT_LEX/final/all.oov_acousticlyfound
 $SCRIPT_PATH/MergeLexicon.pl $EXT_LEX/final/all.oov_acousticlyfound_phon > $EXT_LEX/final/${lexname}_ext.phon
 #
 # Collect + clean retrieved txt
 #
 print_info "[${BASENAME}] Collect and clean SOLR txt answers" 2
 # choose filter
 # default
 cat $SOLR_RES/*.txt | $SCRIPT_PATH/CleanFilter.sh | $SCRIPT_PATH/ApplyCorrectionRules.pl ${LEXICON}.regex | $SCRIPT_PATH/BdlexUC.pl $RULES/basic -t > $EXT_LEX/final/all.bdlex_txt
 # low filter
 #cat $SOLR_RES/*.txt | $SCRIPT_PATH/BdlexUC.pl $RULES/basic -t | sed -f $RULES/preprocess.regex | sed -f $RULES/lastprocess.regex > $EXT_LEX/final/all.bdlex_txt
 #
 # Construct the map file
 #
 # Notes:
 # - Expected format :
 #   <WORD1_STRING> <CANDIDATE1_STRING> <PHON_1>
 #
 print_info "[${BASENAME}] Construct map file" 3
 rm -f $EXT_LEX/final/${lexname}_ext.map 2>/dev/null
 rm -f $EXT_LEX/final/${lexname}.unvalid_oov 2>/dev/null
 while read oov
 do
     oov=`echo $oov | sed "s/\n//g"`
     #
     # Obtain the oov's tag
     #
     #oov_tag=`grep --perl-regex "^$oov\t" $DYNAMIC_TAGSTATS/all.tags | cut -f2`
     #
     # Try to collect text containing the oov word
     #
     print_info "[${BASENAME}] Collect text containing the oov" 3
     cat $EXT_LEX/final/all.bdlex_txt | grep --perl-regex " $oov " | $SCRIPT_PATH/NbMaxWordsFilter.pl 40 |uniq > $EXT_LEX/tmp/txt/$oov.bdlex_txt
     if [ -f $EXT_LEX/tmp/txt/$oov.bdlex_txt ]; then
         nbWords=`wc -l $EXT_LEX/tmp/txt/$oov.bdlex_txt | cut -f1 -d" "`
         if [ $nbWords -eq 0 ]; then
             print_warn "[${BASENAME}] UNVALID OOV: $oov => $nbWords occurrences" 2
             echo "$oov" >> $EXT_LEX/final/${lexname}.unvalid_oov
         else
             #
             # Find a candidate in a filtred invoc lexicon => a candidate which maximize the ppl in the overall txt collected
             #
             #echo "$/getCandidate $SPEER_LM_PATH $SPEER_LM_BASENAME $oov $LEXICON.bdlex_tok $EXT_LEX/tmp/txt/$oov.bdlex_txt"
             print_info `$SPEERAL_PATH/bin/getCandidate $SPEER_LM_PATH $SPEER_LM_BASENAME $oov $CANDIDATE_LEXICON $EXT_LEX/tmp/txt/$oov.bdlex_txt | cut -f1 -d" "` 3
             candidate=`$SPEERAL_PATH/bin/getCandidate $SPEER_LM_PATH $SPEER_LM_BASENAME $oov $CANDIDATE_LEXICON $EXT_LEX/tmp/txt/$oov.bdlex_txt | cut -f1 -d" "`
             if [ ! "$candidate" == "" ]; then
                 grep --perl-regex "^$oov\t" $EXT_LEX/final/all.oov_acousticlyfound_phon > $EXT_LEX/tmp/$oov.phon
                 while read phonLine
                 do
                     #<word> <phon> => <word> <candidate> <phon>
                     echo "$phonLine" | sed "s|\t|\t$candidate\t|"  >> $EXT_LEX/final/${lexname}_ext.map
                 done < $EXT_LEX/tmp/$oov.phon
             else
                 print_warn "[${BASENAME}] UNVALID OOV: $oov => no availaible Candidate word in LM" 2
                 echo "$oov" >> $EXT_LEX/final/${lexname}.unvalid_oov
             fi
         fi
     else
         print_warn "[${BASENAME}] UNVALID OOV: $oov" 2
         echo "$oov" >> $EXT_LEX/final/${lexname}.unvalid_oov
     fi
 done < $EXT_LEX/final/all.oov_acousticlyfound
 #
 ### Speeral
 #
 lexname=`basename $LEXICON`
 #
 # Build the final trigger file
 #
 print_info "[${BASENAME}] Clean trigg files" 3
 mkdir -p $TRIGGER_CONFZONE/speeral/ 2> /dev/null
 mkdir -p $EXT_LEX/speeral/ 2> /dev/null
 for i in `ls $TRIGGER_CONFZONE/*.trigg`
 do
     basename=`basename $i .trigg`
     cat $i | $SCRIPT_PATH/RemoveLineContaining.pl $EXT_LEX/$lexname.unvalid_oov > $TRIGGER_CONFZONE/speeral/$basename.trigg
 done
 #
 # Compile the speeral extended lexicon
 #
 print_info "[${BASENAME}] Compile Speeral extended lexicon" 3
 print_info "$SPEERAL_PATH/bin/buildmappedbinode $LEXICON.bdlex_phon $EXT_LEX/final/${lexname}_ext.map $AM_SKL $EXT_LEX/speeral/${lexname}_ext" 3
 $SPEERAL_PATH/bin/buildmappedbinode $LEXICON.bdlex_phon $EXT_LEX/final/${lexname}_ext.map $AM_SKL $EXT_LEX/speeral/${lexname}_ext
 if [ $CHECK -eq 1 ]
 then
     check_exploitconfpass_lex_check "${EXT_LEX}/speeral/${lexname}_ext"
     if [ $? -eq 1 ]
     then
         print_error "[${BASENAME}] Building Speeral Lexicon $INPUT_DIR -> exit"
         print_error "[${BASENAME}] Check $ERRORFILE"
         print_log_file $ERRORFILE "ERROR : Building Speeral Lexicon $INPUT_DIR"
         print_log_file $ERRORFILE "ERROR : ${EXT_LEX}/speeral/${lexname}_ext Empty after buildmappedbinode ?"
         exit 1;
     fi
 fi
 #-------#
 # CLOSE #
 #-------#
 # Seem OK
 print_info "[${BASENAME}] <= ExploitConfidencePass End | $(date +'%d/%m/%y %H:%M:%S')" 1
 # unlok directory
 mv "$OUTPUT_DIR/EXPLOITCONFPASS.lock" "$OUTPUT_DIR/EXPLOITCONFPASS.unlock"