Jean-François Rey / otmedia

Download as
Browse Code »

Commit 0052714e741ee8d5bc334cd9b41a4430f0621ce6

Authored by Jean-François Rey
1 parent 665a8dac32
Exists in master

bugfix 3pass

Showing 5 changed files with 27 additions and 8 deletions Inline Diff

1 #---------------# 1 #---------------#
2 # OTMEDIA LIA # 2 # OTMEDIA LIA #
3 # HOWTO # 3 # HOWTO #
4 # version 1.0 # 4 # version 1.0 #
5 #---------------# 5 #---------------#
6 6
7 1\ Main options 7 1\ Main scripts options
8 --------------- 8 -----------------------
9 9
10 There are five main options for otmedia scripts. 10 There are five main options for otmedia scripts.
11 -h : for help 11 -h : for help
12 -D : Debug mode 12 -D : Debug mode
13 -v n : Verbose mode 1 low to 3 high 13 -v n : Verbose mode 1 low to 3 high
14 -c : Check results 14 -c : Check results
15 -r : force to rerun a script, without deleting work already done 15 -r : force to rerun a script, without deleting work already done
16 16
17 2\ Main scripts 17 2\ Main scripts
18 --------------- 18 ---------------
19 2.1\ FirstPass.sh 19 2.1\ FirstPass.sh
20 ----------------- 20 -----------------
21 21
22 FirstPass.sh do speaker diarization and transcription of an audio file. Convert it into wav format if not already done (16000Hz, 16 bits, mono). 22 FirstPass.sh do speaker diarization and transcription of an audio file. Convert it into wav format if not already done (16000Hz, 16 bits, mono).
23 If a .SRT file is present in the same directory of the audio file it will copy it. 23 If a .SRT file is present in the same directory of the audio file it will copy it.
24 24
25 $> FisrtPass.sh [options] 110624FR2_20002100.wav result_directory 25 $> FisrtPass.sh [options] 110624FR2_20002100.wav result_directory
26 26
27 Options: 27 Options:
28 -f n : number of forks for speeral 28 -f n : number of forks for speeral
29 29
30 Output : result_directory/110624FR2_20002100/res_p1/ 30 Output : result_directory/110624FR2_20002100/res_p1/
31 31
32 2.2\ SecondPass.sh 32 2.2\ SecondPass.sh
33 ------------------ 33 ------------------
34 34
35 SecondPass.sh do speaker adaptation and transcriptions base on the first pass. 35 SecondPass.sh do speaker adaptation and transcriptions base on the first pass.
36 36
37 $> SecondPass.sh [options] result_directory/110624FR2_20002100/ 37 $> SecondPass.sh [options] result_directory/110624FR2_20002100/
38 38
39 Options: 39 Options:
40 -f n : number of forks for speeral 40 -f n : number of forks for speeral
41 41
42 Output : result_directory/110624FR2_20002100/res_p2/ 42 Output : result_directory/110624FR2_20002100/res_p2/
43 43
44 2.3\ ConfPass.sh 44 2.3\ ConfPass.sh
45 ---------------- 45 ----------------
46 46
47 ConfPass.sh do confidence measure using the second or third pass. 47 ConfPass.sh do confidence measure using the second or third pass.
48 48
49 $> Confpass.sh [options] result_directory/110624FR2_20002100/ <res_p2|res_p3> 49 $> Confpass.sh [options] result_directory/110624FR2_20002100/ <res_p2|res_p3>
50 50
51 Output : result_directory/110624FR2_20002100/conf/res_p2/scored_ctm/ 51 Output : result_directory/110624FR2_20002100/conf/res_p2/scored_ctm/
52 and result_directory/110624FR2_20002100.usf file 52 and result_directory/110624FR2_20002100.usf file
53 53
54 2.4\ ExploitConfidencePass.sh 54 2.4\ ExploitConfidencePass.sh
55 ----------------------------- 55 -----------------------------
56 56
57 It exploits confidence pass measure to : 57 It exploits confidence pass measure to :
58 - boost confidente zone 58 - boost confidente zone
59 - find alternative in non confidente zone (using SOLR DB) 59 - find alternative in non confidente zone (using SOLR DB)
60 - extend the lexicon 60 - extend the lexicon
61 61
62 $> ExploitConfidencePass.sh [options] result_directory/110624FR2_20002100 62 $> ExploitConfidencePass.sh [options] result_directory/110624FR2_20002100
63 63
64 Output : result_directory/110624FR2_20002100/trigg/speeral 64 Output : result_directory/110624FR2_20002100/trigg/speeral
65 result_directory/110624FR2_20002100/LEX/speeral/_ext 65 result_directory/110624FR2_20002100/LEX/speeral/_ext
66 66
67 2.5\ ThirstPass.sh 67 2.5\ ThirstPass.sh
68 ------------------ 68 ------------------
69 69
70 ThirdPass.sh do transcriptions using SecondPass speaker adaptation and ExploitConfidencePass trigg files and new lexicon. 70 ThirdPass.sh do transcriptions using SecondPass speaker adaptation and ExploitConfidencePass trigg files and new lexicon.
71 71
72 $> ThirdPass.sh [options] result_directory/110624FR2_20002100/ 72 $> ThirdPass.sh [options] result_directory/110624FR2_20002100/
73 73
74 Options : 74 Options :
75 -f n : number of forks for speeral 75 -f n : number of forks for speeral
76 76
77 Output : result_directory/110624FR2_20002100/conf/res_p3 77 Output : result_directory/110624FR2_20002100/conf/res_p3
78 78
79 2.6\ RecomposePass.sh 79 2.6\ RecomposePass.sh
80 -------------------- 80 --------------------
81 81
82 RecomposePass.sh copy results that missing in ThirsPass from the Second and First Pass. 82 RecomposePass.sh copy results that missing in ThirsPass from the Second and First Pass.
83 83
84 $> RecomposePass.sh [options] result_directory/110624FR2_20002100/ 84 $> RecomposePass.sh [options] result_directory/110624FR2_20002100/
85 85
86 Output : result_directory/110624FR2_20002100/res_all 86 Output : result_directory/110624FR2_20002100/res_all
87 87
88 2.7\ ScoringRes.sh 88 2.7\ ScoringRes.sh
89 ------------------ 89 ------------------
90 90
91 ScoringRes.sh run differents scoring tools to score the results using SRT file if exists. 91 ScoringRes.sh run differents scoring tools to score the results using SRT file if exists.
92 92
93 $> ScoringRes.sh [options] result_directory/110624FR2_20002100/ 93 $> ScoringRes.sh [options] result_directory/110624FR2_20002100/
94 94
95 Output : result_directory/110624FR2_20002100/scoring 95 Output : result_directory/110624FR2_20002100/scoring
96 96
97 2.8\ CheckResults.sh 97 2.8\ CheckResults.sh
98 -------------------- 98 --------------------
99 99
100 CheckResults.sh parse results directories to synthesize works already done. 100 CheckResults.sh parse results directories to synthesize works already done.
101 101
102 $> CheckResults.sh [options] result_directory 102 $> CheckResults.sh [options] result_directory
103 103
104 Output : "Directory name #plp #res_p1 #treil_p2 #treil_p3 usf_p2 usf_p3" 104 Output : "Directory name #plp #res_p1 #treil_p2 #treil_p3 usf_p2 usf_p3"
105 #plp number of plp files 105 #plp number of plp files
106 #res_p1 number of .res files at first pass 106 #res_p1 number of .res files at first pass
107 #treil_p2 number of .treil files at second pass 107 #treil_p2 number of .treil files at second pass
108 #treil_p3 number of .treil files at third pass 108 #treil_p3 number of .treil files at third pass
109 usf_p2 usf file from confidence pass result on second pass (OK|ERR|NAN) 109 usf_p2 usf file from confidence pass result on second pass (OK|ERR|NAN)
110 usf_p3 usf file from confidence pass result on third pass (OK|ERR|NAN) 110 usf_p3 usf file from confidence pass result on third pass (OK|ERR|NAN)
111 111
112 3\ OneScriptToRuleThemAll.sh 112 3\ OneScriptToRuleThemAll.sh
113 ---------------------------- 113 ----------------------------
114 114
115 The script to do all OTMEDIA LIA pass in one call. 115 The script to do all OTMEDIA LIA pass in one call.
116 116
117 $> OneScriptToRuleThemAll.sh [options] 110624FR2_20002100.wav result_directory 117 $> OneScriptToRuleThemAll.sh [options] 110624FR2_20002100.wav result_directory
118 118
119 Options : (default options are availables) 119 Options : (default options are availables)
120 -a Do every pass 120 -a Do every pass
121 -1 Do First pass 121 -1 Do First pass
122 -2 Do Second pass 122 -2 Do Second pass
123 -3 Do Third pass 123 -3 Do Third pass
124 -C Do Confidence pass 124 -C Do Confidence pass
125 -e Do Exploit Confidence pass 125 -e Do Exploit Confidence pass
126 -R Do Recompose pass 126 -R Do Recompose pass
127 -s Do Scoring pass 127 -s Do Scoring pass
128
129 4\ Modify configuration
130
131 4.1\ Scripts configurations
132 4.2\ Speeral configurations
133
134 5\ Modify binaries
128 135
1 #---------------# 1 #---------------#
2 # OTMEDIA LIA # 2 # OTMEDIA LIA #
3 # INSTALL # 3 # INSTALL #
4 # version : 1.0 # 4 # version : 1.0 #
5 #---------------# 5 #---------------#
6 6
7 OTMEDIA LIA ready to use ? Really ? 7 OTMEDIA LIA ready to use ? Really ?
8 No ! You have to do manualy configuartion for some features. 8 No ! You have to do manualy configuartion for some features.
9 Let see... 9 Let see...
10 10
11 SUMMARY 11 SUMMARY
12 ------- 12 -------
13 13
14 1\ Before installation 14 1\ Before installation
15 2\ install.sh script 15 2\ install.sh script
16 3\ SOLR install 16 3\ SOLR install
17 17
18 18
19 1\ Before installation 19 1\ Before installation
20 ---------------------- 20 ----------------------
21 21
22 - Check and install dependencies. 22 - Check and install dependencies.
23 - In 64 bits architcture be sure you can run 32 bits programs. 23 - In 64 bits architcture be sure you can run 32 bits programs.
24 - Have 300 Go of free space. 24 - Have 300 Go of free space.
25 - Have acces to the network and the nyx server. 25 - Have acces to the network and the nyx server.
26 26
27 2/ install.sh script 27 2/ install.sh script
28 -------------------- 28 --------------------
29 29
30 install.sh script will do most of the work. 30 install.sh script will do most of the work.
31 It will check dependencies and configure pass tools. 31 It will check dependencies and configure pass tools.
32 By default it will do a complet install (300 Go). 32 By default it will do a complet install (300 Go).
33 33
34 You can modifiy behavior by editing install.sh : 34 You can modifiy behavior by editing install.sh :
35 35
36 To disable lexicon adaption using SOLR DB put EXPLOITCONFPASS to 0 (mainly the 290 Go). 36 To disable lexicon adaption using SOLR DB put EXPLOITCONFPASS to 0 (mainly the 290 Go).
37 To disable confidence measure put CONFPASS to 0. 37 To disable confidence measure put CONFPASS to 0.
38 To disable second and third pass put PASS2 to 0. 38 To disable second and third pass put PASS2 to 0.
39 39
40 run install.sh and follow the white rabbit. 40 run install.sh and follow the white rabbit.
41 41
42 3\ SOLR install 42 3\ SOLR install
43 --------------- 43 ---------------
44 44
45 The install.sh script download otmedia-2013-04.tar.gz and untar it in OTMEDIA_HOME/tools/SOLR/ . 45 The install.sh script download otmedia-2013-04.tar.gz and untar it in OTMEDIA_HOME/tools/SOLR/ .
46 See SOLR.INSTALL file to install OTMEDIA SOLR DB. 46 See SOLR.INSTALL file to install OTMEDIA SOLR DB.
47
48 4\ Install descriptions
49
50 OTMEDIA_HOME
51 |-> bin/
52 |-> cfg/
53 |-> data/
54 |-> main_tools/
55 |-> tools/
56 |-> COPYING
57 |-> CorpusOTMedia.txt
58 |-> HOWTO
59 |-> INSTALL
60 |-> README
61 |-> SOLR.INSTALL
62 |-> TODO
47 63
48 64
49 65
50 66
51 67
52 68
53 69
54 70
55 71
56 72
57 73
58 74
59 75
1 - Check and add Verbose messages 1 - Check and add Verbose messages
2 - Modify option -r behaviour 2 - Modify option -r behaviour
3 - Modifiy SOLR request 3 - Modifiy SOLR request
4 - Create Doc in latex
4 5
main_tools/ThirdPass.sh
Diff comments   View file @ 0052714
1 #!/bin/bash 1 #!/bin/bash
2 2
3 ##################################################### 3 #####################################################
4 # File : ThirdPass.sh # 4 # File : ThirdPass.sh #
5 # Brief : ASR third pass using trigg files # 5 # Brief : ASR third pass using trigg files #
6 # Author : Jean-François Rey # 6 # Author : Jean-François Rey #
7 # Version : 1.0 # 7 # Version : 1.0 #
8 # Date : 18/07/13 # 8 # Date : 18/07/13 #
9 ##################################################### 9 #####################################################
10 10
11 echo "### ThirdPass.sh ###" 11 echo "### ThirdPass.sh ###"
12 12
13 # Check OTMEDIA_HOME env var 13 # Check OTMEDIA_HOME env var
14 if [ -z ${OTMEDIA_HOME} ] 14 if [ -z ${OTMEDIA_HOME} ]
15 then 15 then
16 OTMEDIA_HOME=$(dirname $(dirname $(readlink -e $0))) 16 OTMEDIA_HOME=$(dirname $(dirname $(readlink -e $0)))
17 export OTMEDIA_HOME=$OTMEDIA_HOME 17 export OTMEDIA_HOME=$OTMEDIA_HOME
18 fi 18 fi
19 19
20 # where is SecondPass.sh 20 # where is SecondPass.sh
21 MAIN_SCRIPT_PATH=$(dirname $(readlink -e $0)) 21 MAIN_SCRIPT_PATH=$(dirname $(readlink -e $0))
22 22
23 # Scripts Path 23 # Scripts Path
24 SCRIPT_PATH=$OTMEDIA_HOME/tools/scripts 24 SCRIPT_PATH=$OTMEDIA_HOME/tools/scripts
25 25
26 # Include scripts 26 # Include scripts
27 . $SCRIPT_PATH"/Tools.sh" 27 . $SCRIPT_PATH"/Tools.sh"
28 . $SCRIPT_PATH"/CheckThirdPass.sh" 28 . $SCRIPT_PATH"/CheckThirdPass.sh"
29 29
30 # where is ThirdPass.cfg 30 # where is ThirdPass.cfg
31 THIRDPASS_CONFIG_FILE=$OTMEDIA_HOME"/cfg/ThirdPass.cfg" 31 THIRDPASS_CONFIG_FILE=$OTMEDIA_HOME"/cfg/ThirdPass.cfg"
32 if [ -e $THIRDPASS_CONFIG_FILE ] 32 if [ -e $THIRDPASS_CONFIG_FILE ]
33 then 33 then
34 . $THIRDPASS_CONFIG_FILE 34 . $THIRDPASS_CONFIG_FILE
35 else 35 else
36 echo "ERROR : Can't find configuration file $THIRDPASS_CONFIG_FILE" >&2 36 echo "ERROR : Can't find configuration file $THIRDPASS_CONFIG_FILE" >&2
37 exit 1 37 exit 1
38 fi 38 fi
39 39
40 #---------------# 40 #---------------#
41 # Parse Options # 41 # Parse Options #
42 #---------------# 42 #---------------#
43 while getopts ":hDv:crf:" opt 43 while getopts ":hDv:crf:" opt
44 do 44 do
45 case $opt in 45 case $opt in
46 h) 46 h)
47 echo -e "$0 [OPTIONS] <PASS_DIRECTORY>\n" 47 echo -e "$0 [OPTIONS] <PASS_DIRECTORY>\n"
48 echo -e "\t Options:" 48 echo -e "\t Options:"
49 echo -e "\t\t-h :\tprint this message" 49 echo -e "\t\t-h :\tprint this message"
50 echo -e "\t\t-D :\tDEBUG mode on" 50 echo -e "\t\t-D :\tDEBUG mode on"
51 echo -e "\t\t-v l :\tVerbose mode, l=(1|2|3) level mode" 51 echo -e "\t\t-v l :\tVerbose mode, l=(1|2|3) level mode"
52 echo -e "\t\t-c :\t Check process, stop if error detected" 52 echo -e "\t\t-c :\t Check process, stop if error detected"
53 echo -e "\t\t-f n :\tSpecify speeral forks number (default 1)" 53 echo -e "\t\t-f n :\tSpecify speeral forks number (default 1)"
54 echo -e "\t\t-r n :\tforce rerun the show, without deleting works already done" 54 echo -e "\t\t-r n :\tforce rerun the show, without deleting works already done"
55 exit 1 55 exit 1
56 ;; 56 ;;
57 D) 57 D)
58 DEBUG=1 58 DEBUG=1
59 ;; 59 ;;
60 v) 60 v)
61 VERBOSE=$OPTARG 61 VERBOSE=$OPTARG
62 ;; 62 ;;
63 c) 63 c)
64 CHECK=1 64 CHECK=1
65 ;; 65 ;;
66 f) 66 f)
67 FORKS="--forks $OPTARG" 67 FORKS="--forks $OPTARG"
68 ;; 68 ;;
69 r) 69 r)
70 RERUN=1 70 RERUN=1
71 ;; 71 ;;
72 :) 72 :)
73 echo "Option -$OPTARG requires an argument." >&2 73 echo "Option -$OPTARG requires an argument." >&2
74 exit 1 74 exit 1
75 ;; 75 ;;
76 \?) 76 \?)
77 echo "BAD USAGE : unknow opton -$OPTARG" 77 echo "BAD USAGE : unknow opton -$OPTARG"
78 #exit 1 78 #exit 1
79 ;; 79 ;;
80 esac 80 esac
81 done 81 done
82 82
83 # mode debug enable 83 # mode debug enable
84 if [ $DEBUG -eq 1 ] 84 if [ $DEBUG -eq 1 ]
85 then 85 then
86 set -x 86 set -x
87 echo -e "## Mode DEBUG ON ##" 87 echo -e "## Mode DEBUG ON ##"
88 fi 88 fi
89 89
90 # mode verbose enable 90 # mode verbose enable
91 if [ $VERBOSE -gt 0 ]; then echo -e "## Verbose level : $VERBOSE ##" ; fi 91 if [ $VERBOSE -gt 0 ]; then echo -e "## Verbose level : $VERBOSE ##" ; fi
92 92
93 # Check USAGE by arguments number 93 # Check USAGE by arguments number
94 if [ $(($#-($OPTIND-1))) -ne 1 ] 94 if [ $(($#-($OPTIND-1))) -ne 1 ]
95 then 95 then
96 echo "BAD USAGE : ThirdPass.sh [OPTIONS] <PASS_DIR>" 96 echo "BAD USAGE : ThirdPass.sh [OPTIONS] <PASS_DIR>"
97 echo "$0 -h for more info" 97 echo "$0 -h for more info"
98 exit 1 98 exit 1
99 fi 99 fi
100 100
101 shift $((OPTIND-1)) 101 shift $((OPTIND-1))
102 # check Pass directory - First argument 102 # check Pass directory - First argument
103 if [ -e $1 ] && [ -d $1 ] 103 if [ -e $1 ] && [ -d $1 ]
104 then 104 then
105 PASS_DIR=$(readlink -e $1) 105 PASS_DIR=$(readlink -e $1)
106 BASENAME=$(basename $PASS_DIR) 106 BASENAME=$(basename $PASS_DIR)
107 else 107 else
108 print_error "can't find $1 directory" 108 print_error "can't find $1 directory"
109 exit 1 109 exit 1
110 fi 110 fi
111 111
112 print_info "[${BASENAME}] => P3 start | $(date +'%d/%m/%y %H:%M:%S')" 1 112 print_info "[${BASENAME}] => P3 start | $(date +'%d/%m/%y %H:%M:%S')" 1
113 113
114 #-------------# 114 #-------------#
115 # GLOBAL VARS # 115 # GLOBAL VARS #
116 #-------------# 116 #-------------#
117 EXPLOITCONFPASS_CONFIG_FILE="$PASS_DIR/ExploitConfPass.cfg" 117 EXPLOITCONFPASS_CONFIG_FILE="$PASS_DIR/ExploitConfPass.cfg"
118 if [ -e $EXPLOITCONFPASS_CONFIG_FILE ] 118 if [ -e $EXPLOITCONFPASS_CONFIG_FILE ]
119 then 119 then
120 TRIGGER_SPEERAL=$(cat $EXPLOITCONFPASS_CONFIG_FILE | grep "TRIGGER_SPEERAL=" | cut -f2 -d"=") 120 TRIGGER_SPEERAL=$(cat $EXPLOITCONFPASS_CONFIG_FILE | grep "TRIGGER_SPEERAL=" | cut -f2 -d"=")
121 LEX_SPEERAL=$(cat $EXPLOITCONFPASS_CONFIG_FILE | grep "LEX_SPEERAL=" | cut -f2 -d"=") 121 LEX_SPEERAL=$(cat $EXPLOITCONFPASS_CONFIG_FILE | grep "LEX_SPEERAL=" | cut -f2 -d"=")
122 LEX_BINODE_SPEERAL=$(cat $EXPLOITCONFPASS_CONFIG_FILE | grep "LEX_BINODE_SPEERAL=" | cut -f2 -d"=") 122 LEX_BINODE_SPEERAL=$(cat $EXPLOITCONFPASS_CONFIG_FILE | grep "LEX_BINODE_SPEERAL=" | cut -f2 -d"=")
123 LST="" 123 LST=""
124 PLP_DIR_P1="" 124 PLP_DIR_P1=""
125 HMM="" 125 HMM=""
126 else 126 else
127 print_error "[${BASENAME}] Can't find $EXPLOITCONFPASS_CONFIG_FILE file" 127 print_error "[${BASENAME}] Can't find $EXPLOITCONFPASS_CONFIG_FILE file"
128 print_warn "[${BASENAME}] Will use default directories and files" 2 128 print_warn "[${BASENAME}] Will use default directories and files" 2
129 #exit 1 129 #exit 1
130 TRIGGER_SPEERAL=$PASS_DIR/trigg/speeral/ 130 TRIGGER_SPEERAL=$PASS_DIR/trigg/speeral/
131 LEX_SPEERAL=$PASS_DIR/LEX/speeral/LEXIQUE_V6_ext 131 LEX_SPEERAL=$PASS_DIR/LEX/speeral/LEXIQUE_V6_ext
132 LEX_BINODE_SPEERAL=$PASS_DIR/LEX/speeral/LEXIQUE_V6_ext.bin 132 LEX_BINODE_SPEERAL=$PASS_DIR/LEX/speeral/LEXIQUE_V6_ext.bin
133 fi 133 fi
134 134
135 SECONDPASS_CONFIG_FILE="$PASS_DIR/SecondPass.cfg" 135 SECONDPASS_CONFIG_FILE="$PASS_DIR/SecondPass.cfg"
136 if [ -e $SECONDPASS_CONFIG_FILE ] 136 if [ -e $SECONDPASS_CONFIG_FILE ]
137 then 137 then
138 LST=$(cat $SECONDPASS_CONFIG_FILE | grep "^LST=" | cut -f2 -d"=") 138 LST=$(cat $SECONDPASS_CONFIG_FILE | grep "^LST=" | cut -f2 -d"=")
139 HMM=$(cat $SECONDPASS_CONFIG_FILE | grep "^HMM=" | cut -f2 -d"=") 139 HMM=$(cat $SECONDPASS_CONFIG_FILE | grep "^HMM=" | cut -f2 -d"=")
140 PLP_DIR_P1=$(cat $SECONDPASS_CONFIG_FILE | grep "^PLP_DIR_P1=" | cut -f2 -d"=") 140 PLP_DIR_P1=$(cat $SECONDPASS_CONFIG_FILE | grep "^PLP_DIR_P1=" | cut -f2 -d"=")
141 PLP_LIST_P1=$(cat $SECONDPASS_CONFIG_FILE | grep "^PLP_LIST_P1=" | cut -f2 -d"=") 141 PLP_LIST_P1=$(cat $SECONDPASS_CONFIG_FILE | grep "^PLP_LIST_P1=" | cut -f2 -d"=")
142 else 142 else
143 print_error "[${BASENAME}] Can't find $SECONDPASS_CONFIG_FILE file" 143 print_error "[${BASENAME}] Can't find $SECONDPASS_CONFIG_FILE file"
144 print_warn "[${BASENAME}] Will use default directories and files" 2 144 print_warn "[${BASENAME}] Will use default directories and files" 2
145 #exit 1 145 #exit 1
146 LST=$PASS_DIR/lists 146 LST=$PASS_DIR/lists
147 HMM=$PASS_DIR/hmm 147 HMM=$PASS_DIR/hmm
148 PLP_DIR_P1=$PASS_DIR/PLP 148 PLP_DIR_P1=$PASS_DIR/PLP
149 PLP_LIST_P1=$PASS_DIR/plp.lst 149 PLP_LIST_P1=$PASS_DIR/plp.lst
150 fi 150 fi
151 151
152 OUTPUT_DIR_BASENAME=$PASS_DIR 152 OUTPUT_DIR_BASENAME=$PASS_DIR
153 RES_DIR="$PASS_DIR/res_p3" 153 RES_DIR="$PASS_DIR/res_p3"
154 LOGFILE=$PASS_DIR"/info_p3.log" 154 LOGFILE=$PASS_DIR"/info_p3.log"
155 ERRORFILE=$PASS_DIR"/error_p3.log" 155 ERRORFILE=$PASS_DIR"/error_p3.log"
156 156
157 #------------------# 157 #------------------#
158 # Create WORKSPACE # 158 # Create WORKSPACE #
159 #------------------# 159 #------------------#
160 160
161 # Lock directory 161 # Lock directory
162 if [ -e $OUTPUT_DIR_BASENAME/THIRDPASS.lock ] && [ $RERUN -eq 0 ]; then print_warn "[${BASENAME}] THIRDPASS is locked -> exit" 2; exit 1; fi 162 if [ -e $OUTPUT_DIR_BASENAME/THIRDPASS.lock ] && [ $RERUN -eq 0 ]; then print_warn "[${BASENAME}] THIRDPASS is locked -> exit" 2; exit 1; fi
163 rm "$OUTPUT_DIR_BASENAME/THIRDPASS.unlock" > /dev/null 2>&1 163 rm "$OUTPUT_DIR_BASENAME/THIRDPASS.unlock" > /dev/null 2>&1
164 touch "$OUTPUT_DIR_BASENAME/THIRDPASS.lock" > /dev/null 2>&1 164 touch "$OUTPUT_DIR_BASENAME/THIRDPASS.lock" > /dev/null 2>&1
165 165
166 if [ $RERUN -eq 0 ]; then rm -r $RES_DIR > /dev/null 2>&1; fi 166 if [ $RERUN -eq 0 ]; then rm -r $RES_DIR > /dev/null 2>&1; fi
167 mkdir -p $RES_DIR 167 mkdir -p $RES_DIR
168 print_info "[${BASENAME}] Make directory $RES_DIR" 3 168 print_info "[${BASENAME}] Make directory $RES_DIR" 3
169 169
170 rm $LOGFILE $ERRORFILE 2>/dev/null
170 #--------------------# 171 #--------------------#
171 # Save configuration # 172 # Save configuration #
172 #--------------------# 173 #--------------------#
173 cp $THIRDPASS_CONFIG_FILE $OUTPUT_DIR_BASENAME/ThirdPass.cfg 174 cp $THIRDPASS_CONFIG_FILE $OUTPUT_DIR_BASENAME/ThirdPass.cfg
174 echo "RES_DIR=$RES_DIR" >> $OUTPUT_DIR_BASENAME/ThirdPass.cfg 175 echo "RES_DIR=$RES_DIR" >> $OUTPUT_DIR_BASENAME/ThirdPass.cfg
175 print_info "[${BASENAME}] Save config in $OUTPUT_DIR_BASENAME/ThirdPass.cfg" 1 176 print_info "[${BASENAME}] Save config in $OUTPUT_DIR_BASENAME/ThirdPass.cfg" 1
176 177
177 #--------------------------------------------------# 178 #--------------------------------------------------#
178 # Third Pass using trigger file (DECODING) # 179 # Third Pass using trigger file (DECODING) #
179 #--------------------------------------------------# 180 #--------------------------------------------------#
180 print_info "[${BASENAME}] Launch Third Pass" 1 181 print_info "[${BASENAME}] Launch Third Pass" 1
181 182
182 ## Generate speeral config file adding trigger rep 183 ## Generate speeral config file adding trigger rep
183 cat $SPEERAL_CFG_PATH/$SPEERAL_CFG_FILE | sed -e "s|<nom>[^<]*</nom>|<nom>$OUTPUT_DIR_BASENAME/LEX/speeral/LEXIQUE_V6_ext</nom>|g" \ 184 cat $SPEERAL_CFG_PATH/$SPEERAL_CFG_FILE | sed -e "s|<nom>[^<]*</nom>|<nom>$OUTPUT_DIR_BASENAME/LEX/speeral/LEXIQUE_V6_ext</nom>|g" \
184 | sed -e "s|<binode>[^<]*</binode>|<binode>$OUTPUT_DIR_BASENAME/LEX/speeral/LEXIQUE_V6_ext.bin</binode>|g" \ 185 | sed -e "s|<binode>[^<]*</binode>|<binode>$OUTPUT_DIR_BASENAME/LEX/speeral/LEXIQUE_V6_ext.bin</binode>|g" \
185 | sed -e "s|<trigger><dir>[^<]*</dir></trigger>|<trigger><dir>$TRIGGER_SPEERAL</dir></trigger>|g" > $OUTPUT_DIR_BASENAME/SpeeralThirdPass.xml 186 | sed -e "s|<trigger><dir>[^<]*</dir></trigger>|<trigger><dir>$TRIGGER_SPEERAL</dir></trigger>|g" > $OUTPUT_DIR_BASENAME/SpeeralThirdPass.xml
186 SPEERAL_THIRD_CFG=$OUTPUT_DIR_BASENAME/SpeeralThirdPass.xml 187 SPEERAL_THIRD_CFG=$OUTPUT_DIR_BASENAME/SpeeralThirdPass.xml
187 print_info "[${BASENAME}] Third Pass Speeral Config file ${SPEERAL_THIRD_CFG}" 3 188 print_info "[${BASENAME}] Third Pass Speeral Config file ${SPEERAL_THIRD_CFG}" 3
188 189
189 # for all speaker 190 # for all speaker
190 for lspeaker in $(ls $LST/*.lst) 191 for lspeaker in $(ls $LST/*.lst)
191 do 192 do
192 speaker=$(basename $lspeaker ".lst") 193 speaker=$(basename $lspeaker ".lst")
193 # for all AM 194 # for all AM
194 for (( i=0; $i<${#MTAG[@]} ; i++ )) 195 for (( i=0; $i<${#MTAG[@]} ; i++ ))
195 do 196 do
196 if [ -e $OUTPUT_DIR_BASENAME/plp_${MODS[$i]}.lst ]; then 197 if [ -e $OUTPUT_DIR_BASENAME/plp_${MODS[$i]}.lst ]; then
197 type=$(grep -e "${speaker}$" "$OUTPUT_DIR_BASENAME/plp_${MODS[$i]}.lst") 198 type=$(grep -e "${speaker}$" "$OUTPUT_DIR_BASENAME/plp_${MODS[$i]}.lst")
198 ## if is the good AM for the speaker 199 ## if is the good AM for the speaker
199 if [ -n "$type" ] 200 if [ -n "$type" ]
200 then 201 then
201 ## Speeral decoding 202 ## Speeral decoding
202 if [ -s $HMM/$speaker.hmm ] && [ -s $HMM/$speaker.cls ] 203 if [ -s $HMM/$speaker.hmm ] && [ -s $HMM/$speaker.cls ]
203 then 204 then
204 print_info "[${BASENAME}] $SPEERAL_BIN $LST/$speaker.lst $RES_DIR $SPEERAL_THIRD_CFG -r $PLP_DIR_P1 -m $HMM/$speaker.hmm -c $HMM/$speaker.cls $FORKS --lock" 3 205 print_info "[${BASENAME}] $SPEERAL_BIN $LST/$speaker.lst $RES_DIR $SPEERAL_THIRD_CFG -r $PLP_DIR_P1 -m $HMM/$speaker.hmm -c $HMM/$speaker.cls $FORKS --lock" 3
205 $SPEERAL_BIN $LST/$speaker.lst $RES_DIR ${SPEERAL_THIRD_CFG} -r $PLP_DIR_P1 -m $HMM/$speaker.hmm -c $HMM/$speaker.cls $FORKS --lock 206 $SPEERAL_BIN $LST/$speaker.lst $RES_DIR ${SPEERAL_THIRD_CFG} -r $PLP_DIR_P1 -m $HMM/$speaker.hmm -c $HMM/$speaker.cls $FORKS --lock
206 else 207 else
207 print_warn "[${BASENAME}] $HMM/$speaker.hmm and $speaker.cls empty, do default decoding..." 2 208 print_warn "[${BASENAME}] $HMM/$speaker.hmm and $speaker.cls empty, do default decoding..." 2
208 print_info "$SPEERAL_BIN $LST/$speaker.lst $RES_DIR ${SPEERAL_THIRD_CFG} -r $PLP_DIR_P1 -m $SPEERAL_AM/${MODS[$i]}.hmm -c $SPEERAL_AM/${MODS[$i]}.cls $FORKS --lock" 3 209 print_info "$SPEERAL_BIN $LST/$speaker.lst $RES_DIR ${SPEERAL_THIRD_CFG} -r $PLP_DIR_P1 -m $SPEERAL_AM/${MODS[$i]}.hmm -c $SPEERAL_AM/${MODS[$i]}.cls $FORKS --lock" 3
209 $SPEERAL_BIN $LST/$speaker.lst $RES_DIR ${SPEERAL_THIRD_CFG} -r $PLP_DIR_P1 -m $SPEERAL_AM/${MODS[$i]}.hmm -c $SPEERAL_AM/${MODS[$i]}.cls $FORKS --lock 210 $SPEERAL_BIN $LST/$speaker.lst $RES_DIR ${SPEERAL_THIRD_CFG} -r $PLP_DIR_P1 -m $SPEERAL_AM/${MODS[$i]}.hmm -c $SPEERAL_AM/${MODS[$i]}.cls $FORKS --lock
210 fi 211 fi
211 212
212 if [ $CHECK -eq 1 ] 213 if [ $CHECK -eq 1 ]
213 then 214 then
214 check_third_pass_output_speeral "$LST/$speaker.lst" "$RES_DIR" 215 check_third_pass_output_speeral "$LST/$speaker.lst" "$RES_DIR"
215 if [ $? -eq 1 ] 216 if [ $? -eq 1 ]
216 then 217 then
217 218
218 print_warn "[${BASENAME}] Speeral output error : check $LOGFILE" 2 219 print_warn "[${BASENAME}] Speeral output error : check $LOGFILE" 2
219 print_log_file $LOGFILE "WARN : Speeral number of output ERROR $LST/$speaker.lst" 220 print_log_file $LOGFILE "WARN : Speeral number of output ERROR $LST/$speaker.lst"
220 ls $RES_DIR/*.seg 2>/dev/null | grep -e "$speaker" | sed -e "s|$RES_DIR\/||" | sed -e 's/\.seg//' | sort > ${OUTPUT_DIR_BASENAME}/.tmp 221 ls $RES_DIR/*.seg 2>/dev/null | grep -e "$speaker" | sed -e "s|$RES_DIR\/||" | sed -e 's/\.seg//' | sort > ${OUTPUT_DIR_BASENAME}/.tmp
221 print_log_file $LOGFILE "Segs (and treil) not done :\n[" 222 print_log_file $LOGFILE "Segs (and treil) not done :\n["
222 diff $LST/$speaker.lst ${OUTPUT_DIR_BASENAME}/.tmp | grep -e "^< " | sed -e "s/< //" >> $LOGFILE 223 diff $LST/$speaker.lst ${OUTPUT_DIR_BASENAME}/.tmp | grep -e "^< " | sed -e "s/< //" >> $LOGFILE
223 print_log_file $LOGFILE "] [$(date +'%d/%m/%y %H:%M:%S')]" 224 print_log_file $LOGFILE "] [$(date +'%d/%m/%y %H:%M:%S')]"
224 rm ${OUTPUT_DIR_BASENAME}/.tmp 225 rm ${OUTPUT_DIR_BASENAME}/.tmp
225 fi 226 fi
226 fi 227 fi
227 break 228 break
228 fi 229 fi
229 fi 230 fi
230 done 231 done
231 done 232 done
232 233
233 ## Check missing seg and log it 234 ## Check missing seg and log it
234 235
235 ## Check missing seg and log it 236 ## Check missing seg and log it
236 if [ $CHECK -eq 1 ] 237 if [ $CHECK -eq 1 ]
237 then 238 then
238 ls $RES_DIR/*.treil 2>/deV/null | sed -e "s|$RES_DIR\/||g" | sed -e 's/\.treil//' | sort > ${OUTPUT_DIR_BASENAME}/.tmp 239 ls $RES_DIR/*.treil 2>/dev/null | sed -e "s|$RES_DIR\/||g" | sed -e 's/\.treil//' | sort > ${OUTPUT_DIR_BASENAME}/.tmp
239 todo=$(cat ${PLP_LIST_P1} 2>/dev/null | wc -l) 240 todo=$(cat ${PLP_LIST_P1} 2>/dev/null | wc -l)
240 if [ $todo -eq 0 ]; then todo=1;fi 241 if [ $todo -eq 0 ]; then todo=1;fi
241 notdone=$(($todo - $(cat ${OUTPUT_DIR_BASENAME}/.tmp | wc -l))) 242 notdone=$(($todo - $(cat ${OUTPUT_DIR_BASENAME}/.tmp | wc -l)))
242 pourcentage=$((($notdone*100)/$todo)) 243 pourcentage=$((($notdone*100)/$todo))
243 if [ $notdone -ne 0 ] 244 if [ $notdone -ne 0 ]
244 then 245 then
245 print_error "[${BASENAME}] Check $ERRORFILE" 246 print_error "[${BASENAME}] Check $ERRORFILE"
246 print_log_file "$ERRORFILE" "ERROR : Treil not done [" 247 print_log_file "$ERRORFILE" "ERROR : Treil not done ["
247 diff ${PLP_LIST_P1} ${OUTPUT_DIR_BASENAME}/.tmp | grep -e "^< " | sed -e "s/< //" >> $ERRORFILE 248 diff ${PLP_LIST_P1} ${OUTPUT_DIR_BASENAME}/.tmp | grep -e "^< " | sed -e "s/< //" >> $ERRORFILE
248 print_log_file "$ERRORFILE" "] $pourcentage% $BASENAME" 249 print_log_file "$ERRORFILE" "] $pourcentage% $BASENAME"
249 else 250 else
250 print_log_file "$LOGFILE" "P3 OK $BASENAME | $(date +'%d/%m/%y %H:%M:%S')" 251 print_log_file "$LOGFILE" "P3 OK $BASENAME | $(date +'%d/%m/%y %H:%M:%S')"
251 fi 252 fi
252 rm ${OUTPUT_DIR_BASENAME}/.tmp > /dev/null 2>&1 253 rm ${OUTPUT_DIR_BASENAME}/.tmp > /dev/null 2>&1
253 fi 254 fi
254 255
255 #---------------# 256 #---------------#
256 # Convert res # 257 # Convert res #
257 #---------------# 258 #---------------#
258 259
259 print_info "[${BASENAME}] Convert .res into .ctm" 1 260 print_info "[${BASENAME}] Convert .res into .ctm" 1
260 # .res => .ctm 261 # .res => .ctm
261 $SCRIPT_PATH/res2out.pl --dir $RES_DIR --format CTM --ignore $RULES/asupp --out ${OUTPUT_DIR_BASENAME}/${BASENAME}.3pass.ctm 262 $SCRIPT_PATH/res2out.pl --dir $RES_DIR --format CTM --ignore $RULES/asupp --out ${OUTPUT_DIR_BASENAME}/${BASENAME}.3pass.ctm
262 print_info "[${BASENAME}] Convert .res into .trs" 1 263 print_info "[${BASENAME}] Convert .res into .trs" 1
263 # .res => .trs 264 # .res => .trs
264 echo -e "name $AUTHOR\nfileName ${BASENAME}\nfileExt wav\nsegFile ${OUTPUT_DIR_BASENAME}/${BASENAME}.seg" > ${OUTPUT_DIR_BASENAME}/${BASENAME}.trs_cfg 265 echo -e "name $AUTHOR\nfileName ${BASENAME}\nfileExt wav\nsegFile ${OUTPUT_DIR_BASENAME}/${BASENAME}.seg" > ${OUTPUT_DIR_BASENAME}/${BASENAME}.trs_cfg
265 $SCRIPT_PATH/res2out.pl --dir $RES_DIR --format TRS --ignore $RULES/asupp --out ${OUTPUT_DIR_BASENAME}/${BASENAME}.3pass.trs --trs_config ${OUTPUT_DIR_BASENAME}/${BASENAME}.trs_cfg 266 $SCRIPT_PATH/res2out.pl --dir $RES_DIR --format TRS --ignore $RULES/asupp --out ${OUTPUT_DIR_BASENAME}/${BASENAME}.3pass.trs --trs_config ${OUTPUT_DIR_BASENAME}/${BASENAME}.trs_cfg
266 rm ${OUTPUT_DIR_BASENAME}/${BASENAME}.trs_cfg > /dev/null 2>&1 267 rm ${OUTPUT_DIR_BASENAME}/${BASENAME}.trs_cfg > /dev/null 2>&1
267 print_info "[${BASENAME}] Convert .res into .txt" 1 268 print_info "[${BASENAME}] Convert .res into .txt" 1
268 # .res => .txt 269 # .res => .txt
269 $SCRIPT_PATH/res2out.pl --dir $RES_DIR --format TXT --ignore $RULES/asupp --out ${OUTPUT_DIR_BASENAME}/${BASENAME}.3pass.txt 270 $SCRIPT_PATH/res2out.pl --dir $RES_DIR --format TXT --ignore $RULES/asupp --out ${OUTPUT_DIR_BASENAME}/${BASENAME}.3pass.txt
270 271
271 272
272 print_info "[${BASENAME}] <= P3 End | $(date +'%d/%m/%y %H:%M:%S')" 1 273 print_info "[${BASENAME}] <= P3 End | $(date +'%d/%m/%y %H:%M:%S')" 1
273 274
274 # unlock directory 275 # unlock directory
275 mv "$OUTPUT_DIR_BASENAME/THIRDPASS.lock" "$OUTPUT_DIR_BASENAME/THIRDPASS.unlock" 276 mv "$OUTPUT_DIR_BASENAME/THIRDPASS.lock" "$OUTPUT_DIR_BASENAME/THIRDPASS.unlock"
276 277
277 278
tools/scripts/CheckThirdPass.sh
Diff comments   View file @ 0052714
1 #!/bin/bash 1 #!/bin/bash
2 2
3 ################################################# 3 #################################################
4 # File : CheckThirdPass.sh # 4 # File : CheckThirdPass.sh #
5 # Brief : Check Second Pass # 5 # Brief : Check Second Pass #
6 # Author : Jean-François Rey # 6 # Author : Jean-François Rey #
7 # Version : 1.0 # 7 # Version : 1.0 #
8 # Date : 25/06/13 # 8 # Date : 25/06/13 #
9 ################################################# 9 #################################################
10 10
11 # configuration file 11 # configuration file
12 . "$OTMEDIA_HOME/cfg/ThirdPass.cfg" 12 . "$OTMEDIA_HOME/cfg/ThirdPass.cfg"
13 13
14 14
15 #check_third_pass_output_speeral(plplist,resdir) 15 #check_third_pass_output_speeral(plplist,resdir)
16 # - plplist : filename containing plp list 16 # - plplist : filename containing plp list
17 # - resdir : output Speeral directory 17 # - resdir : output Speeral directory
18 function check_third_pass_output_speeral() 18 function check_third_pass_output_speeral()
19 { 19 {
20 if [ $# -eq 2 ] 20 if [ $# -eq 2 ]
21 then 21 then
22 if [ ! -e "$1" ] || [ -z "$1" ]; then return 1; fi 22 if [ ! -e "$1" ] || [ -z "$1" ]; then return 1; fi
23 FILENBSEG=$(ls $2/*.seg 2>/dev/null | grep -f "$1" | wc -l) 23 FILENBSEG=$(ls $2/*.seg 2>/dev/null | grep -f "$1" | wc -l)
24 FILENBRES=$(ls $2/*.res 2>/dev/null | grep -f "$1" | wc -l) 24 FILENBRES=$(ls $2/*.res 2>/dev/null | grep -f "$1" | wc -l)
25 FILENBTREIL=$(ls $2/*.treil 2>/deV/null | grep -f "$1" | wc -l) 25 FILENBTREIL=$(ls $2/*.treil 2>/dev/null | grep -f "$1" | wc -l)
26 FILENB=$(cat "$1" | wc -l) 26 FILENB=$(cat "$1" | wc -l)
27 if [ $FILENB -ne $FILENBSEG ] || [ $FILENB -ne $FILENBRES ] || [ $FILENB -ne $FILENBTREIL ] ; then return 1;fi 27 if [ $FILENB -ne $FILENBSEG ] || [ $FILENB -ne $FILENBRES ] || [ $FILENB -ne $FILENBTREIL ] ; then return 1;fi
28 fi 28 fi
29 return 0 29 return 0
30 } 30 }
31 31
32 32