Jean-François Rey / otmedia

Download as
Browse Code »

Commit d93dab09a0b95d54fa03aed8a2b5c5de69c13c7e

Authored by Jean-François Rey
1 parent b50ebdf386
Exists in master

update

Showing 3 changed files with 21 additions and 12 deletions Inline Diff

1 Main corpus : 1 Main corpus :
2 2
3 /local_disk/hermes2/OTMEDIA/INA-2011-16-01/radio-7h-10h/ 3 /local_disk/hermes2/OTMEDIA/INA-2011-16-01/radio-7h-10h/
4 /local_disk/hermes2/OTMEDIA/INA-TV1 4 /local_disk/hermes2/OTMEDIA/INA-TV1
5 /local_disk/zeus2/OTMEDIA/INA-TV2 5 /local_disk/zeus2/OTMEDIA/INA-TV2
6 6
7 Il y a des trucs de partout... 7 Il y a des trucs de partout...
8 mais le plus interressent est dans hermes2, zeus2, et gaia 8 mais le plus intéressant est dans hermes2, zeus2, et gaia
9 9
10 10
11 NYX ================================================================ 11 NYX ================================================================
12 12
13 /local_disk/nyx/OTMEDIA/AFP-2011/ 04/2011 -> 12/2011 13 /local_disk/nyx/OTMEDIA/AFP-2011/ 04/2011 -> 12/2011
14 newsml (xml) -> 6153 fichiers (67 Mo) 14 newsml (xml) -> 6153 fichiers (67 Mo)
15 15
16 /local_disk/nyx/OTMEDIA/INA-XML (80 Mo) 16 /local_disk/nyx/OTMEDIA/INA-XML (80 Mo)
17 01/04/2009 -> 31/01/2010 17 01/04/2009 -> 31/01/2010
18 Arte.xml Info (8.2 Mo) 18 Arte.xml Info (8.2 Mo)
19 France2.xml 20h (24 Mo) 19 France2.xml 20h (24 Mo)
20 France3.xml outre-mer national (20 Mo) 20 France3.xml outre-mer national (20 Mo)
21 M6.xml le 19h45 (10 Mo) 21 M6.xml le 19h45 (10 Mo)
22 TF1.xml journal 20h (18 Mo) 22 TF1.xml journal 20h (18 Mo)
23 23
24 24
25 HERMES ============================================================== 25 HERMES ==============================================================
26 26
27 /local_disk/hermes2/OTMEDIA/INA-2011-16-01 10/06/2011 -> 31/12/2011 27 /local_disk/hermes2/OTMEDIA/INA-2011-16-01 10/06/2011 -> 31/12/2011
28 |-> radio-7h-10h/ (274 Go) 16605 fichiers .ENV || .HDR || .MPA 28 |-> radio-7h-10h/ (274 Go) 16605 fichiers .ENV || .HDR || .MPA
29 BFM 1845 fichiers (25 Go) 29 BFM 1845 fichiers (25 Go)
30 EU1 1845 fichiers (25 Go) 30 EU1 1845 fichiers (25 Go)
31 FCR 1845 fichiers (50 GO) 31 FCR 1845 fichiers (50 GO)
32 FIF 1845 fichiers (25 Go) 32 FIF 1845 fichiers (25 Go)
33 FIT 1845 fichiers (25 Go) 33 FIT 1845 fichiers (25 Go)
34 RCL 1845 fichiers (50 Go) 34 RCL 1845 fichiers (50 Go)
35 RFI 1845 fichiers (25 Go) 35 RFI 1845 fichiers (25 Go)
36 RMC 1845 fichiers (25 Go) 36 RMC 1845 fichiers (25 Go)
37 RTL 1845 fichiers (25 Go) 37 RTL 1845 fichiers (25 Go)
38 |-> tv-19h-21h00/ (867 Go) 12191 fichiers .HDR || .MP4 || .SRT 38 |-> tv-19h-21h00/ (867 Go) 12191 fichiers .HDR || .MP4 || .SRT
39 NAS_ART 1055 fichiers (73 Go) 39 NAS_ART 1055 fichiers (73 Go)
40 NAS_C+_ 1205 fichiers (72 Go) 40 NAS_C+_ 1205 fichiers (72 Go)
41 NAS_F24 820 f (72 Go) 41 NAS_F24 820 f (72 Go)
42 NAS_FR3 895 f (72 Go) 42 NAS_FR3 895 f (72 Go)
43 NAS_ITL 904 f (74 Go) 43 NAS_ITL 904 f (74 Go)
44 NAS_M6_ 1206 f (73 Go) 44 NAS_M6_ 1206 f (73 Go)
45 NAS_BFT 820 f (74 Go) 45 NAS_BFT 820 f (74 Go)
46 NAS_ENW 820 f (73 Go) 46 NAS_ENW 820 f (73 Go)
47 NAS_FR2 1210 f (71 Go) 47 NAS_FR2 1210 f (71 Go)
48 NAS_FR5 1205 f (73 Go) 48 NAS_FR5 1205 f (73 Go)
49 NAS_LCI 820 f (72 Go) 49 NAS_LCI 820 f (72 Go)
50 NAS_TF1 1230 f (73 Go) 50 NAS_TF1 1230 f (73 Go)
51 51
52 |-> web 52 |-> web
53 archive INA XML 109511 (12 Go) 53 archive INA XML 109511 (12 Go)
54 |-> document-tv (90 Mo) 18/06/2011 -> 22/11/2011 54 |-> document-tv (90 Mo) 18/06/2011 -> 22/11/2011
55 ARTE.xml (4.4 Mo) 55 ARTE.xml (4.4 Mo)
56 BFMTV.xml (18 Mo) 56 BFMTV.xml (18 Mo)
57 Canal+.xml (4.8 Mo) 57 Canal+.xml (4.8 Mo)
58 France24.xml (9.3 Mo) 58 France24.xml (9.3 Mo)
59 France2.xml (17 Mo) 59 France2.xml (17 Mo)
60 France3.xml (8 Mo) 60 France3.xml (8 Mo)
61 LCI.xml (15 Mo) 61 LCI.xml (15 Mo)
62 M6.xml (5.8 Mo) 62 M6.xml (5.8 Mo)
63 TF1.xml (9.2 Mo) 63 TF1.xml (9.2 Mo)
64 64
65 /local_disk/hermes2/OTMEDIA/INA-TV1/ TV1-SIZE 1.7T NbMP4 : 9403 Nb-SRT : 815 16/06/2011 -> 29/04/2012 65 /local_disk/hermes2/OTMEDIA/INA-TV1/ TV1-SIZE 1.7T NbMP4 : 9403 Nb-SRT : 815 16/06/2011 -> 29/04/2012
66 |-> LIA/tv-1/otstore/dlweb/ina-dl-tv/data 66 |-> LIA/tv-1/otstore/dlweb/ina-dl-tv/data
67 |-> NAS_ART NbMP4: 385 NbSRT: 177 67 |-> NAS_ART NbMP4: 385 NbSRT: 177
68 |-> NAS_BFT NbMP4: 2028 NbSRT: 45 68 |-> NAS_BFT NbMP4: 2028 NbSRT: 45
69 |-> NAS_C+_ NbMP4: 605 NbSRT: 593 69 |-> NAS_C+_ NbMP4: 605 NbSRT: 593
70 |-> NAS_F24 NbMP4: 6385 70 |-> NAS_F24 NbMP4: 6385
71 71
72 72
73 POPEYE ============================================================== 73 POPEYE ==============================================================
74 74
75 /local_disk/popeye/users/hugo/HD_INA-2009/ 75 /local_disk/popeye/users/hugo/HD_INA-2009/
76 |-> INA-JT-05052009_18012010 7068 fichiers MP4 (1.3 To) 05/05/2009 -> 18/01/2010 76 |-> INA-JT-05052009_18012010 7068 fichiers MP4 (1.3 To) 05/05/2009 -> 18/01/2010
77 |-> FR2-captation-01042009_31012010.xml 20H (24 Mo) 01/04/2009 -> 31/01/2010 77 |-> FR2-captation-01042009_31012010.xml 20H (24 Mo) 01/04/2009 -> 31/01/2010
78 78
79 79
80 APHRODITE =========================================================== 80 APHRODITE ===========================================================
81 81
82 /local_disk/aphrodite2 82 /local_disk/aphrodite2
83 |-> même fichier que dans popeye 83 |-> même fichier que dans popeye
84 |-> afp_corpus (368 Go) 84 |-> afp_corpus (368 Go)
85 |-> 2009-MMD jpg -> .bz2 (368 Go) 85 |-> 2009-MMD jpg -> .bz2 (368 Go)
86 |-> 2009-TXT .bz2 (109 Mo) 86 |-> 2009-TXT .bz2 (109 Mo)
87 87
88 88
89 ZEUS ================================================================ 89 ZEUS ================================================================
90 90
91 /local_disk/zeus2/OTMEDIA/ 91 /local_disk/zeus2/OTMEDIA/
92 |-> /AFP2009-2-corpus/2009 128587 fichiers (2 Go) 31/12/2008 -> 31/12/2009 92 |-> /AFP2009-2-corpus/2009 128587 fichiers (2 Go) 31/12/2008 -> 31/12/2009
93 |-> INA copie des mois 05 et 06 de INA-JT-05052009_18012010 93 |-> INA copie des mois 05 et 06 de INA-JT-05052009_18012010
94 |-> ina-dl-radio/ 24 fichiers (993 Mo) .HDR .MPA .ENV 10/06/2011 94 |-> ina-dl-radio/ 24 fichiers (993 Mo) .HDR .MPA .ENV 10/06/2011
95 |-> INA-TV2/ (1.7 To) .MP4 (9402) && .SRT (4727) 95 |-> INA-TV2/ (1.7 To) .MP4 (9402) && .SRT (4727)
96 |-> NAS_F24 57*24h fichiers (230 Go) 09/03/2012 -> 12/05/2012 96 |-> NAS_F24 57*24h fichiers (230 Go) 09/03/2012 -> 12/05/2012
97 |-> NAS_FR2 321*3h f (245 Go) 16/06/2011 -> 03/05/2012 97 |-> NAS_FR2 321*3h f (245 Go) 16/06/2011 -> 03/05/2012
98 |-> NAS_FR3 311*2h f (110 Go) 16/06/2011 -> 02/05/2012 98 |-> NAS_FR3 311*2h f (110 Go) 16/06/2011 -> 02/05/2012
99 |-> NAS_FR5 310*5h f (267 Go) 16/06/2011 -> 04/05/2012 99 |-> NAS_FR5 310*5h f (267 Go) 16/06/2011 -> 04/05/2012
100 |-> NAS_ITL 310*4h f (233 Go) 16/06/2011 -> 27/04/2012 100 |-> NAS_ITL 310*4h f (233 Go) 16/06/2011 -> 27/04/2012
101 |-> NAS_LCI 323*5h f (359 Go) 16/06/2011 -> 03/05/2012 101 |-> NAS_LCI 323*5h f (359 Go) 16/06/2011 -> 03/05/2012
102 |-> NAS_M6_ 291*2h f (102 Go) 16/06/2011 -> 08/04/2012 102 |-> NAS_M6_ 291*2h f (102 Go) 16/06/2011 -> 08/04/2012
103 |-> NAS_TF1 309*2h f (105 Go) 16/06/2011 -> 19/04/2012 103 |-> NAS_TF1 309*2h f (105 Go) 16/06/2011 -> 19/04/2012
104 104
105 |-> USF ? 589 .usf (950 Mo) 01/11/11 -> 03/12/11 105 |-> USF ? 589 .usf (950 Mo) 01/11/11 -> 03/12/11
106 |-> otmedia-2013-04/ (DDB SOLR) 106 |-> otmedia-2013-04/ (DDB SOLR)
107 |-> index/ (303 Go) 107 |-> index/ (303 Go)
108 |-> otmedia-multimedia/index (21 Go) 108 |-> otmedia-multimedia/index (21 Go)
109 |-> /otmedia-document/index (241 Go) 109 |-> /otmedia-document/index (241 Go)
110 110
111 111
112 GAIA ================================================================ 112 GAIA ================================================================
113 113
114 /local_disk/gaia/OTMEDIA/CORPUS 114 /local_disk/gaia/OTMEDIA/CORPUS
115 |-> AFP ? 115 |-> AFP ?
116 |-> afp-corpus-text ? 116 |-> afp-corpus-text ?
117 |-> AFP-Quotes-2011 ? 117 |-> AFP-Quotes-2011 ?
118 |-> extraction-lia ? 118 |-> extraction-lia ?
119 |-> fromSyllabs 119 |-> fromSyllabs
120 |-> INA-XML (cf nyx) 120 |-> INA-XML (cf nyx)
121 |-> REPERE ? 121 |-> REPERE ?
122 |-> similar-events-2009-France 122 |-> similar-events-2009-France
123 |-> similar-events-2009-Monde 123 |-> similar-events-2009-Monde
124 |-> SRT (cf INA-TV1 et INA-TV2) 124 |-> SRT (cf INA-TV1 et INA-TV2)
125 |-> toSyllabs ? 125 |-> toSyllabs ?
126 |-> tv .xml ? 126 |-> tv .xml ?
127 |-> visual ? 127 |-> visual ?
128 |-> web ? 128 |-> web ?
129 129
130 /local_disk/gaia/OTMEDIA/ALL_WAV/ 36586 .wav (3.9 To) # corpus INA -> WAV 130 /local_disk/gaia/OTMEDIA/ALL_WAV/ 36586 .wav (3.9 To) # corpus INA -> WAV
131 |-> RADIO/2011 5535 fichiers (594 Go) 10/06/11 -> 31/12/11 131 |-> RADIO/2011 5535 fichiers (594 Go) 10/06/11 -> 31/12/11
132 |-> RADIO/2012 11906 fichiers (1.3 To) 01/01/12 -> 26/11/12 132 |-> RADIO/2012 11906 fichiers (1.3 To) 01/01/12 -> 26/11/12
133 |-> TV/2011 7986 fichiers (857 Go) 10/06/11 -> 31/12/11 133 |-> TV/2011 7986 fichiers (857 Go) 10/06/11 -> 31/12/11
134 |-> TV/2012 11159 fichiers (1.2 To) 01/01/12 -> 21/11/12 134 |-> TV/2012 11159 fichiers (1.2 To) 01/01/12 -> 21/11/12
135 135
136 136
137 137
1 ################ 1 ################
2 # SOLR INSTALL # 2 # SOLR INSTALL #
3 ################ 3 ################
4 # 4 #
5 # Author Jean-François Rey 5 # Author Jean-François Rey
6 # Version : 1.0 6 # Version : 1.0
7 # Date : 18/07/2013 7 # Date : 18/07/2013
8 # 8 #
9 9
10 1/ Edit install.sh and put CONFPASS=1 10 1/ Edit install.sh and put CONFPASS=1
11 11
12 2/ Run install.sh, this will check tomcat is installed, untar otmedia SOLR DB and ask for solr service info. 12 2/ Run install.sh, this will check tomcat is installed, download and untar otmedia SOLR DB and ask for solr service info.
13 13
14 3/ Configure Tomcat and SOLR 14 3/ Configure Tomcat and SOLR
15 15
16 SOLR_OTMEDIA_PATH=OTMEDIA_PATH/tools/SOLR/otemdia-2013-04 16 SOLR_OTMEDIA_PATH=OTMEDIA_PATH/tools/SOLR/otemdia-2013-04
17 17
18 3.1/ Set context file 18 3.1/ Set context file
19 ---------------- 19 ----------------
20 20
21 - in SOLR_OTMEDIA_PATH/solr/otmedia-document/solr-tomcat-deploy/solr-otmedia-document.xml 21 - in SOLR_OTMEDIA_PATH/solr/otmedia-document/solr-tomcat-deploy/solr-otmedia-document.xml
22 change DocBase to DocBase="SOLR_OTMEDIA_PATH/solr/otmedia-document/apache-solr-3.5.0.war" 22 change DocBase to DocBase="SOLR_OTMEDIA_PATH/solr/otmedia-document/apache-solr-3.5.0.war"
23 and value to value="SOLR_OTMEDIA_PATH/solr/otmedia-document/" 23 and value to value="SOLR_OTMEDIA_PATH/solr/otmedia-document/"
24 24
25 - in SOLR_OTMEDIA_PATH/solr/otmedia-multimedia/solr-tomcat-deploy/solr-otmedia-multimedia.xml 25 - in SOLR_OTMEDIA_PATH/solr/otmedia-multimedia/solr-tomcat-deploy/solr-otmedia-multimedia.xml
26 change DocBase to DocBase="SOLR_OTMEDIA_PATH/solr/otmedia-multimedia/apache-solr-3.5.0.war" 26 change DocBase to DocBase="SOLR_OTMEDIA_PATH/solr/otmedia-multimedia/apache-solr-3.5.0.war"
27 and value to value="SOLR_OTMEDIA_PATH/solr/otmedia-multimedia/" 27 and value to value="SOLR_OTMEDIA_PATH/solr/otmedia-multimedia/"
28 28
29 3.2/ SOLR data configuration 29 3.2/ SOLR data configuration
30 ----------------------- 30 -----------------------
31 31
32 - in SOLR_OTMEDIA_PATH/solr/otmedia-document/conf/solrconfig.xml 32 - in SOLR_OTMEDIA_PATH/solr/otmedia-document/conf/solrconfig.xml
33 change datadir (solr.data.dir) to SOLR_OTMEDIA_PATH/index/otmedia-document/ 33 change datadir (solr.data.dir) to SOLR_OTMEDIA_PATH/index/otmedia-document/
34 34
35 - in SOLR_OTMEDIA_PATH/solr/otmedia-multimedia/conf/solrconfig.xml 35 - in SOLR_OTMEDIA_PATH/solr/otmedia-multimedia/conf/solrconfig.xml
36 change datadir (solr.data.dir) to SOLR_OTMEDIA_PATH/index/otmedia-multimedia/ 36 change datadir (solr.data.dir) to SOLR_OTMEDIA_PATH/index/otmedia-multimedia/
37 37
38 3.3/ Add SOLR DB to Tomcat 38 3.3/ Add SOLR DB to Tomcat
39 --------------------- 39 ---------------------
40 40
41 - in tomcat/Catalina/localhost/ (mainly in /etc/tomcat/Catalina/localhost or /var/lib/tomcat/conf/Catalina/localhost) 41 - in tomcat/Catalina/localhost/ (mainly in /etc/tomcat/Catalina/localhost or /var/lib/tomcat/conf/Catalina/localhost)
42 run : $> ln -s SOLR_OTMEDIA_PATH/solr/otmedia-document/solr-tomcat-deploy/solr-otmedia-document.xml solr-otmedia-document.xml 42 run : $> ln -s SOLR_OTMEDIA_PATH/solr/otmedia-document/solr-tomcat-deploy/solr-otmedia-document.xml solr-otmedia-document.xml
43 run : $> ln -s SOLR_OTMEDIA_PATH/solr/otmedia-multimedia/solr-tomcat-deploy/solr-otmedia-multimedia.xml solr-otmedia-document.xml 43 run : $> ln -s SOLR_OTMEDIA_PATH/solr/otmedia-multimedia/solr-tomcat-deploy/solr-otmedia-multimedia.xml solr-otmedia-document.xml
44 44
45 4/ Tomcat trouble 45 4/ Tomcat trouble
46 46
47 4.1/ SOLR use a lot of memory, you need to increase java heap space ! 47 4.1/ SOLR use a lot of memory, you need to increase java heap space !
48 ------------------------- 48 -------------------------
49 49
50 - in catalina.sh (/usr/share/tomcat/bin) 50 - in catalina.sh (/usr/share/tomcat/bin)
51 add CATALINA_OPTS="$CATALINA_OPTS -Xms256 -Xmx512m" 51 add CATALINA_OPTS="$CATALINA_OPTS -Xms256 -Xmx512m"
52 52
53 4.2/ Directory permissions 53 4.2/ Directory permissions
54 --------------------- 54 ---------------------
55 55
56 - SOLR_OTMEDIA_PATH and subdirectory (and files) need to belong to tomcat group (and tomcat user if the default user don't belong to tomcat group). 56 - SOLR_OTMEDIA_PATH and subdirectory (and files) need to belong to tomcat group (and tomcat user if the default user don't belong to tomcat group).
57 chgrp -r tomcat7 otmedia-2013-04 57 chgrp -r tomcat7 otmedia-2013-04
58 chmod g+rx otmedia-2013-04 58 chmod g+rx otmedia-2013-04
59 59
60 5/ Test 60 5/ Test
61 61
62 You can test those requests (change ip and port): 62 You can test those requests (change ip and port):
63 http://localhost:8080/solr-otmedia-multimedia/select?q=test+bonus+&fq=docDate:[2011-12-30T00\:00\:01Z+TO+2012-01-01T23\:59\:59Z] 63 http://localhost:8080/solr-otmedia-multimedia/select?q=test+bonus+&fq=docDate:[2011-12-30T00\:00\:01Z+TO+2012-01-01T23\:59\:59Z]
64 http://localhost:8080/solr-otmedia-document/select?q=test+bonus+&fq=docDate:[2011-12-30T00\:00\:01Z+TO+2012-01-01T23\:59\:59Z] 64 http://localhost:8080/solr-otmedia-document/select?q=test+bonus+&fq=docDate:[2011-12-30T00\:00\:01Z+TO+2012-01-01T23\:59\:59Z]
65 65
66 66
1 #!/bin/bash 1 #!/bin/bash
2 2
3 #-------------------# 3 #-------------------#
4 # Install script # 4 # Install script #
5 # OTMEDIA # 5 # OTMEDIA #
6 #-------------------# 6 #-------------------#
7 7
8 # Color variables 8 # Color variables
9 txtgrn=$(tput setaf 2) # Green 9 txtgrn=$(tput setaf 2) # Green
10 txtylw=$(tput setaf 3) # Yellow 10 txtylw=$(tput setaf 3) # Yellow
11 txtblu=$(tput setaf 4) # Blue 11 txtblu=$(tput setaf 4) # Blue
12 txtpur=$(tput setaf 5) # Purple 12 txtpur=$(tput setaf 5) # Purple
13 txtcyn=$(tput setaf 6) # Cyan 13 txtcyn=$(tput setaf 6) # Cyan
14 txtwht=$(tput setaf 7) # White 14 txtwht=$(tput setaf 7) # White
15 txtrst=$(tput sgr0) # Text reset. 15 txtrst=$(tput sgr0) # Text reset.
16 #/color 16 #/color
17 17
18 # 18 #
19 ### Global Variables 19 ### Global Variables
20 # 20 #
21 PWD=$(pwd) 21 PWD=$(pwd)
22 OTMEDIA_HOME=$PWD 22 OTMEDIA_HOME=$PWD
23 test=$(arch) 23 test=$(arch)
24 if [ "$test" == "x86_64" ]; then ARCH=".64"; else ARCH=""; fi 24 if [ "$test" == "x86_64" ]; then ARCH=".64"; else ARCH=""; fi
25 #/Global 25 #/Global
26 26
27 27
28 # 28 #
29 # Put to 0 to disable dependencies of a pass 29 # Put to 0 to disable dependencies of a pass
30 # and 1 to enable 30 # and 1 to enable
31 # 31 #
32 PASS1=1 # First Pass 32 PASS1=1 # First Pass
33 PASS2=1 # Second Pass 33 PASS2=1 # Second Pass
34 CONFPASS=1 # Confidence Pass 34 CONFPASS=1 # Confidence Pass
35 EXPLOITCONFPASS=1 # SOLR query and trigg 35 EXPLOITCONFPASS=1 # SOLR query and trigg
36 36
37 echo -e "\nWill do install for :" 37 echo -e "\nWill do install for :"
38 if [ $PASS1 -eq 1 ];then echo "- Pass 1";fi 38 if [ $PASS1 -eq 1 ];then echo "- Pass 1";fi
39 if [ $PASS2 -eq 1 ];then echo "- Pass 2";fi 39 if [ $PASS2 -eq 1 ];then echo "- Pass 2";fi
40 if [ $CONFPASS -eq 1 ];then echo "- Confidence Pass";fi 40 if [ $CONFPASS -eq 1 ];then echo "- Confidence Pass";fi
41 if [ $EXPLOITCONFPASS -eq 1 ];then echo "- Exploit Confidence Pass";fi 41 if [ $EXPLOITCONFPASS -eq 1 ];then echo "- Exploit Confidence Pass";fi
42 42
43 # 43 #
44 ### CHECK Dependencies ### 44 ### CHECK Dependencies ###
45 # 45 #
46 echo -e "\n\t${txtblu}Check Dependencies${txtrst}\n" 46 echo -e "\n\t${txtblu}Check Dependencies${txtrst}\n"
47 47
48 ## make 48 ## make
49 test=$(whereis make) 49 test=$(whereis make)
50 if [ "$test" == "make:" ] 50 if [ "$test" == "make:" ]
51 then 51 then
52 echo -e "${txtpur}ERROR${txtrst} make not found\n You have to install make\n sudo apt-get install make" 52 echo -e "${txtpur}ERROR${txtrst} make not found\n You have to install make\n sudo apt-get install make"
53 exit 1; 53 exit 1;
54 fi 54 fi
55 echo -e "make \t ${txtgrn}OK${txtrst}" 55 echo -e "make \t ${txtgrn}OK${txtrst}"
56 56
57 ## CC 57 ## CC
58 test=$(whereis cc) 58 test=$(whereis cc)
59 if [ "$test" == "cc:" ] 59 if [ "$test" == "cc:" ]
60 then 60 then
61 echo -e "${txtpur}ERROR${txtrst} cc not found\n You have to install cc\n sudo apt-get install gcc" 61 echo -e "${txtpur}ERROR${txtrst} cc not found\n You have to install cc\n sudo apt-get install gcc"
62 exit 1; 62 exit 1;
63 fi 63 fi
64 echo -e "cc \t ${txtgrn}OK${txtrst}" 64 echo -e "cc \t ${txtgrn}OK${txtrst}"
65 65
66 ## AVCONV 66 ## AVCONV
67 test=$(whereis avconv) 67 test=$(whereis avconv)
68 if [ "$test" == "avconv:" ] 68 if [ "$test" == "avconv:" ]
69 then 69 then
70 echo -e "${txtpur}ERROR${txtrst} avconv not found\n You have to install avconv\n sudo apt-get install libav-tools" 70 echo -e "${txtpur}ERROR${txtrst} avconv not found\n You have to install avconv\n sudo apt-get install libav-tools"
71 exit 1; 71 exit 1;
72 fi 72 fi
73 echo -e "libav-tools : avconv \t ${txtgrn}OK${txtrst}" 73 echo -e "libav-tools : avconv \t ${txtgrn}OK${txtrst}"
74 74
75 ## JAVA 75 ## JAVA
76 test=$(whereis java) 76 test=$(whereis java)
77 if [ "$test" == "java:" ] 77 if [ "$test" == "java:" ]
78 then 78 then
79 echo -e "${txtpur}ERROR${txtrst} java not found\n You have to install java JRE\n sudo apt-get install openjdk-7-jre" 79 echo -e "${txtpur}ERROR${txtrst} java not found\n You have to install java JRE\n sudo apt-get install openjdk-7-jre"
80 exit 1; 80 exit 1;
81 fi 81 fi
82 echo -e "Java : JRE \t ${txtgrn}OK${txtrst}" 82 echo -e "Java : JRE \t ${txtgrn}OK${txtrst}"
83 test=$(whereis javac) 83 test=$(whereis javac)
84 if [ "$test" == "javac:" ] 84 if [ "$test" == "javac:" ]
85 then 85 then
86 echo -e "${txtpur}ERROR${txtrst} javac not found\n You have to install java JDK\n sudo apt-get install openjdk-7-jdk" 86 echo -e "${txtpur}ERROR${txtrst} javac not found\n You have to install java JDK\n sudo apt-get install openjdk-7-jdk"
87 exit 1; 87 exit 1;
88 fi 88 fi
89 echo -e "Java : JDK \t ${txtgrn}OK${txtrst}" 89 echo -e "Java : JDK \t ${txtgrn}OK${txtrst}"
90 90
91 if [ $EXPLOITCONFPASS -eq 1 ] 91 if [ $EXPLOITCONFPASS -eq 1 ]
92 then 92 then
93 ## Python 93 ## Python
94 test=$(whereis python) 94 test=$(whereis python)
95 if [ "$test" == "python:" ] 95 if [ "$test" == "python:" ]
96 then 96 then
97 echo -e "${txtpur}ERROR${txtrst} python not found\n You have to install python\n sudo apt-get install python" 97 echo -e "${txtpur}ERROR${txtrst} python not found\n You have to install python\n sudo apt-get install python"
98 exit 1; 98 exit 1;
99 fi 99 fi
100 echo -e "python : \t ${txtgrn}OK${txtrst}" 100 echo -e "python : \t ${txtgrn}OK${txtrst}"
101 fi 101 fi
102 102
103 ## Perl 103 ## Perl
104 test=$(whereis perl) 104 test=$(whereis perl)
105 if [ "$test" == "perl:" ] 105 if [ "$test" == "perl:" ]
106 then 106 then
107 echo -e "${txtpur}ERROR${txtrst} perl not found\n You have to install perl\n sudo apt-get install perl" 107 echo -e "${txtpur}ERROR${txtrst} perl not found\n You have to install perl\n sudo apt-get install perl"
108 exit 1; 108 exit 1;
109 fi 109 fi
110 echo -e "perl : \t ${txtgrn}OK${txtrst}" 110 echo -e "perl : \t ${txtgrn}OK${txtrst}"
111 111
112 ## iconv 112 ## iconv
113 test=$(whereis iconv) 113 test=$(whereis iconv)
114 if [ "$test" == "iconv:" ] 114 if [ "$test" == "iconv:" ]
115 then 115 then
116 echo -e "${txtpur}ERROR${txtrst} iconv not found\n You have to install iconv\n sudo apt-cache search iconv" 116 echo -e "${txtpur}ERROR${txtrst} iconv not found\n You have to install iconv\n sudo apt-cache search iconv"
117 exit 1; 117 exit 1;
118 fi 118 fi
119 echo -e "iconv : \t ${txtgrn}OK${txtrst}" 119 echo -e "iconv : \t ${txtgrn}OK${txtrst}"
120 120
121 ## csh shell 121 ## csh shell
122 test=$(whereis csh) 122 test=$(whereis csh)
123 if [ "$test" == "csh:" ] 123 if [ "$test" == "csh:" ]
124 then 124 then
125 echo -e "${txtpur}ERROR${txtrst} csh shell not found\n You have to install csh shell\n sudo apt-get install csh" 125 echo -e "${txtpur}ERROR${txtrst} csh shell not found\n You have to install csh shell\n sudo apt-get install csh"
126 exit 1; 126 exit 1;
127 fi 127 fi
128 echo -e "csh shell : \t ${txtgrn}OK${txtrst}" 128 echo -e "csh shell : \t ${txtgrn}OK${txtrst}"
129 129
130 ## SRI LM 130 ## SRI LM
131 if [ -z "$SRILM" ] && [ -z "$MACHINE_TYPE" ] 131 if [ -z "$SRILM" ] && [ -z "$MACHINE_TYPE" ]
132 then 132 then
133 echo -e "${txtpur}ERROR${txtrst} SRILM toolkit variables are not defined (SRILM and MACHINE_TYPE)\n You have to install SRILM Toolkit\n" 133 echo -e "${txtpur}ERROR${txtrst} SRILM toolkit variables are not defined (SRILM and MACHINE_TYPE)\n You have to install SRILM Toolkit\n"
134 exit 1; 134 exit 1;
135 fi 135 fi
136 export SRILM_BIN=$SRILM/bin/$MACHINE_TYPE 136 export SRILM_BIN=$SRILM/bin/$MACHINE_TYPE
137 echo -e "SRILM toolkit : \t ${txtgrn}OK${txtrst}" 137 echo -e "SRILM toolkit : \t ${txtgrn}OK${txtrst}"
138 138
139 139
140 140
141 ### Speeral Configuration ### 141 ### Speeral Configuration ###
142
143 echo -e "\t${txtblu}Speeral configuration${txtrst}\n"
142 echo -e "Download Speeral bin and data :" 144 echo -e "Download Speeral bin and data :"
143 scp -r rey@nyx:~/OTMEDIA_DATA/Speeral $OTMEDIA_HOME/tools/ 145 scp -r rey@nyx:~/OTMEDIA_DATA/Speeral $OTMEDIA_HOME/tools/
144 echo -e "\t${txtblu}Generating Speeral configuration files :${txtrst}\n" 146 echo -e "\n\t${txtblu}Generating Speeral configuration files :${txtrst}\n"
145 cat $PWD/tools/Speeral/CFG/SpeeralFirstPass.xml.tmp | sed -e "s|<nom>[^<]*</nom>|<nom>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer</nom>|g" \ 147 cat $PWD/tools/Speeral/CFG/SpeeralFirstPass.xml.tmp | sed -e "s|<nom>[^<]*</nom>|<nom>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer</nom>|g" \
146 | sed -e "s|<ngramme>[^<]*</ngramme>|<ngramme>$PWD/tools/Speeral/LM/ML_4gOTMEDIA_LEXIQUE_V6</ngramme>|g" \ 148 | sed -e "s|<ngramme>[^<]*</ngramme>|<ngramme>$PWD/tools/Speeral/LM/ML_4gOTMEDIA_LEXIQUE_V6</ngramme>|g" \
147 | sed -e "s|<binode>[^<]*</binode>|<binode>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer.bin</binode>|g" \ 149 | sed -e "s|<binode>[^<]*</binode>|<binode>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer.bin</binode>|g" \
148 > $PWD/tools/Speeral/CFG/SpeeralFirstPass.xml 150 > $PWD/tools/Speeral/CFG/SpeeralFirstPass.xml
149 echo $PWD/tools/Speeral/CFG/SpeeralFirstPass.xml 151 echo $PWD/tools/Speeral/CFG/SpeeralFirstPass.xml
150 cat $PWD/tools/Speeral/CFG/SpeeralSecondPass.xml.tmp | sed -e "s|<nom>[^<]*</nom>|<nom>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer</nom>|g" \ 152 cat $PWD/tools/Speeral/CFG/SpeeralSecondPass.xml.tmp | sed -e "s|<nom>[^<]*</nom>|<nom>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer</nom>|g" \
151 | sed -e "s|<ngramme>[^<]*</ngramme>|<ngramme>$PWD/tools/Speeral/LM/ML_4gOTMEDIA_LEXIQUE_V6</ngramme>|g" \ 153 | sed -e "s|<ngramme>[^<]*</ngramme>|<ngramme>$PWD/tools/Speeral/LM/ML_4gOTMEDIA_LEXIQUE_V6</ngramme>|g" \
152 | sed -e "s|<binode>[^<]*</binode>|<binode>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer.bin</binode>|g" \ 154 | sed -e "s|<binode>[^<]*</binode>|<binode>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer.bin</binode>|g" \
153 > $PWD/tools/Speeral/CFG/SpeeralSecondPass.xml 155 > $PWD/tools/Speeral/CFG/SpeeralSecondPass.xml
154 echo $PWD/tools/Speeral/CFG/SpeeralSecondPass.xml 156 echo $PWD/tools/Speeral/CFG/SpeeralSecondPass.xml
155 cat $PWD/tools/Speeral/CFG/SpeeralThirdPass.xml.tmp | sed -e "s|<nom>[^<]*</nom>|<nom>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer</nom>|g" \ 157 cat $PWD/tools/Speeral/CFG/SpeeralThirdPass.xml.tmp | sed -e "s|<nom>[^<]*</nom>|<nom>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer</nom>|g" \
156 | sed -e "s|<ngramme>[^<]*</ngramme>|<ngramme>$PWD/tools/Speeral/LM/ML_4gOTMEDIA_LEXIQUE_V6</ngramme>|g" \ 158 | sed -e "s|<ngramme>[^<]*</ngramme>|<ngramme>$PWD/tools/Speeral/LM/ML_4gOTMEDIA_LEXIQUE_V6</ngramme>|g" \
157 | sed -e "s|<binode>[^<]*</binode>|<binode>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer.bin</binode>|g" \ 159 | sed -e "s|<binode>[^<]*</binode>|<binode>$PWD/tools/Speeral/LEX/LEXIQUE_V6.speer.bin</binode>|g" \
158 > $PWD/tools/Speeral/CFG/SpeeralThirdPass.xml 160 > $PWD/tools/Speeral/CFG/SpeeralThirdPass.xml
159 echo $PWD/tools/Speeral/CFG/SpeeralThirdPass.xml 161 echo $PWD/tools/Speeral/CFG/SpeeralThirdPass.xml
160 162
161 163
162 if [ $EXPLOITCONFPASS -eq 1 ] 164 if [ $EXPLOITCONFPASS -eq 1 ]
163 then 165 then
164 ### LIA ltbox ### 166 ### LIA ltbox ###
165 echo -e "\t${txtblu}Install lia_ltbox${txtrst}\n" 167 echo -e "\t${txtblu}Install lia_ltbox${txtrst}\n"
166 export LIA_TAGG_LANG="french" 168 export LIA_TAGG_LANG="french"
167 export LIA_TAGG="$OTMEDIA_HOME/tools/lia_ltbox/lia_tagg/" 169 export LIA_TAGG="$OTMEDIA_HOME/tools/lia_ltbox/lia_tagg/"
168 export LIA_PHON_REP="$OTMEDIA_HOME/tools/lia_ltbox/lia_phon/" 170 export LIA_PHON_REP="$OTMEDIA_HOME/tools/lia_ltbox/lia_phon/"
169 export LIA_BIGLEX="$OTMEDIA_HOME/tools/lia_ltbox/lia_biglex/" 171 export LIA_BIGLEX="$OTMEDIA_HOME/tools/lia_ltbox/lia_biglex/"
170 172
171 ### config lia_phon 173 ### config lia_phon
172 cd $LIA_PHON_REP 174 cd $LIA_PHON_REP
173 make all > /dev/null 175 make all > /dev/null
174 make ressource > /dev/null 176 make ressource > /dev/null
175 ### config lia_tagg 177 ### config lia_tagg
176 cd $LIA_TAGG 178 cd $LIA_TAGG
177 make all > /dev/null 179 make all > /dev/null
178 make ressource.french > /dev/null 180 make ressource.french > /dev/null
179 ### config lia_biglex 181 ### config lia_biglex
180 cd $LIA_BIGLEX 182 cd $LIA_BIGLEX
181 make -f makefile.biglex > /dev/null 183 make -f makefile.biglex > /dev/null
182 cd $OTMEDIA_HOME 184 cd $OTMEDIA_HOME
183 185
184 186
185 ### SOLR DB ### 187 ### SOLR DB ###
186 # Tomcat fisrtly 188 # Tomcat fisrtly
187 test=$(dpkg -l | grep "^ii" | grep tomcat) 189 test=$(dpkg -l | grep "^ii" | grep tomcat)
188 if [ "$test" == "" ] 190 if [ "$test" == "" ]
189 then 191 then
190 echo -e "${txtpur}ERROR${txtrst} TOMCAT seems to not be installed)\n You have to install TOMCAT\n" 192 echo -e "${txtpur}ERROR${txtrst} TOMCAT seems to not be installed)\n You have to install TOMCAT\n"
191 exit 1; 193 exit 1;
192 fi 194 fi
193 echo -e "\nTOMCAT : \t ${txtgrn}OK${txtrst}" 195 echo -e "\nTOMCAT : \t ${txtgrn}OK${txtrst}\n"
194 # SOLR secondly 196 # SOLR secondly
195 echo -e "\t${txtblu}Install SOLR DB${txtrst}" 197 echo -e "\t${txtblu}Install SOLR DB${txtrst}"
196 read -p "You will need 300 Go of free space to install SOLR DB\nContinue ? (y/n) " solr 198 read -e -p "You will need 300 Go of free space to install SOLR DB\nContinue ? (y/n) " solr
197 if [ "$solr" == "y" ] 199 if [ "$solr" == "y" ]
198 then 200 then
199 201
200 echo -e "Download SOLR DB\r" 202 echo -e "Download SOLR DB\r"
201 mkdir -p $OTMEDIA_HOME/tools/SOLR 2> /dev/null 203 mkdir -p $OTMEDIA_HOME/tools/SOLR 2> /dev/null
202 scp -r rey@nyx:~/OTMEDIA_DATA/SOLR/otmedia-2013-04.tar.gz $OTMEDIA_HOME/tools/SOLR 204 scp -r rey@nyx:~/OTMEDIA_DATA/SOLR/otmedia-2013-04.tar.gz $OTMEDIA_HOME/tools/SOLR
203 echo -e "Unzip SOLR DB\r" 205 echo -e "Unzip SOLR DB\r"
206 res=0
204 #res = $(tar -xvzf "$OTMEDIA_HOME/tools/SOLR/otmedia-2013-04.tar.gz" "$OTMEDIA_HOME/tools/SOLR/") 207 #res = $(tar -xvzf "$OTMEDIA_HOME/tools/SOLR/otmedia-2013-04.tar.gz" "$OTMEDIA_HOME/tools/SOLR/")
205 if [ $res -eq 2 ]; then echo " ${txtpur}NOT OK${txtrst}"; 208 if [ $res -eq 2 ]; then echo " ${txtpur}NOT OK${txtrst}";
206 else echo " ${txtgrn}OK${txtrst}"; fi 209 else echo " ${txtgrn}OK${txtrst}"; fi
210 else
211 echo "Skipping SOLR install"
212 fi
213 read -e -p "Configure SOLR DB server ? (y/n) " solr
214 if [ "$solr" == "y" ]
207 read -p "Enter SOLR server IP :" ip 215 read -p "Enter SOLR server IP :" ip
208 if [ $ip == "" ];then ip="127.0.0.1";fi 216 if [ "${ip}" == "" ];then ip="127.0.0.1";fi
209 echo 'machine = "194.57.216.43"' > $OTMEDIA_HOME/tools/scripts/solrinfo.py 217 echo 'machine = "${ip}"' > $OTMEDIA_HOME/tools/scripts/solrinfo.py
210 read -p "Enter SOLR server port :" port 218 read -p "Enter SOLR server port :" port
211 if [ $port == "" ]; then port="8080";fi 219 if [ "${port}" == "" ]; then port="8080";fi
212 echo "\n\tSOLR server IP $ip" 220 echo -e "\n\tSOLR server IP ${ip}"
213 echo "\tSOLR server port $port" 221 echo -e "\tSOLR server port ${port}"
214 echo 'port = "8080"' >> $OTMEDIA_HOME/tools/scripts/solrinfo.py 222 echo 'port = "${port}"' >> $OTMEDIA_HOME/tools/scripts/solrinfo.py
215 else 223 else
216 echo "Skipping SOLR install" 224 echo "Skipping SOLR DB Configuration"
217 fi 225 fi
226 echo -e "See SOLR.INSTALL file for more information\n"
218 fi 227 fi
219 228
220 ### Set Variables in bashrc ### 229 ### Set Variables in bashrc ###
221 cat ~/.bashrc | grep -v "OTMEDIA_HOME" | grep -v "SRILM_BIN" > ~/.bashrc.org 230 cat ~/.bashrc | grep -v "OTMEDIA_HOME" | grep -v "SRILM_BIN" > ~/.bashrc.org
222 #cat ~/.bashrc | grep -v "OTMEDIA_HOME" | grep -v "SRILM_BIN" | grep -v "LIA_TAGG" | grep -v "LIA_PHON" | grep -v "LIA_BIGLEX" > ~/.bashrc.org 231 #cat ~/.bashrc | grep -v "OTMEDIA_HOME" | grep -v "SRILM_BIN" | grep -v "LIA_TAGG" | grep -v "LIA_PHON" | grep -v "LIA_BIGLEX" > ~/.bashrc.org
223 cp ~/.bashrc.org ~/.bashrc 232 cp ~/.bashrc.org ~/.bashrc
224 export OTMEDIA_HOME=$PWD 233 export OTMEDIA_HOME=$PWD
225 echo "export OTMEDIA_HOME=$PWD" >> ~/.bashrc 234 echo "export OTMEDIA_HOME=$PWD" >> ~/.bashrc
226 echo "export SRILM_BIN=$SRILM/bin/$MACHINE_TYPE" >> ~/.bashrc 235 echo "export SRILM_BIN=$SRILM/bin/$MACHINE_TYPE" >> ~/.bashrc
227 #echo "export LIA_TAGG_LANG=french" >> ~/.bashrc 236 #echo "export LIA_TAGG_LANG=french" >> ~/.bashrc
228 #echo "export LIA_TAGG=$OTMEDIA_HOME/tools/lia_ltbox/lia_tagg/" >> ~/.bashrc 237 #echo "export LIA_TAGG=$OTMEDIA_HOME/tools/lia_ltbox/lia_tagg/" >> ~/.bashrc
229 #echo "export LIA_PHON_REP=$OTMEDIA_HOME/tools/lia_ltbox/lia_phon/" >> ~/.bashrc 238 #echo "export LIA_PHON_REP=$OTMEDIA_HOME/tools/lia_ltbox/lia_phon/" >> ~/.bashrc
230 #echo "export LIA_BIGLEX=$OTMEDIA_HOME/tools/lia_ltbox/lia_biglex/" >> ~/.bashrc 239 #echo "export LIA_BIGLEX=$OTMEDIA_HOME/tools/lia_ltbox/lia_biglex/" >> ~/.bashrc
231 240
232 # set global configuration file 241 # set global configuration file
233 echo "OTMEDIA_HOME=$PWD" > $OTMEDIA_HOME/cfg/main_cfg.cfg 242 echo "OTMEDIA_HOME=$PWD" > $OTMEDIA_HOME/cfg/main_cfg.cfg
234 echo "ARCH=$ARCH" >> $OTMEDIA_HOME/cfg/main_cfg.cfg 243 echo "ARCH=$ARCH" >> $OTMEDIA_HOME/cfg/main_cfg.cfg
235 echo "VERBOSE=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg 244 echo "VERBOSE=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg
236 echo "DEBUG=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg 245 echo "DEBUG=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg
237 echo "CHECK=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg 246 echo "CHECK=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg
238 echo "RERUN=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg 247 echo "RERUN=0" >> $OTMEDIA_HOME/cfg/main_cfg.cfg
239 248
240 echo -e "\n\n\t${txtgrn}### Install completed ###${txtrst}\n" 249 echo -e "\n\n\t${txtgrn}### Install completed ###${txtrst}\n"
241 echo -e "do : source ~/.bashrc" 250 echo -e "do : source ~/.bashrc"
242 echo -e "or set variable :\n" 251 echo -e "or set variable :\n"
243 echo "export OTMEDIA_HOME=$PWD" 252 echo "export OTMEDIA_HOME=$PWD"
244 echo "export SRILM_BIN=$SRILM/bin/$MACHINE_TYPE" 253 echo "export SRILM_BIN=$SRILM/bin/$MACHINE_TYPE"
245 254
246 255