Hello:
I have analysis apache log from hive, but there is a problem
When I write hive command in Script file and use crontab for schedule it
The result is different with run in hive container
The attachment file is 2 way process detail
Hive_error.txt is run hive command in script
Hive_normal.txt is run hive command in hive container
Best Regard
Sunplus technology 日嘉 / Harold Chen
MSN: <mailto:chen0...@hotmail.com> chen0...@hotmail.com
SKYPE: chen0727
Mobil: 886-937545215
Tel: 886-2-8798-2988 #222
Fax:886-2-8751-5499
hadoop@hadoop-00:~$ uname
Linux
hadoop@hadoop-00:~$ cat /proc/version
Linux version 2.6.35-22-server (buildd@allspice) (gcc version 4.4.5
(Ubuntu/Linaro 4.4.4-14ubuntu4) ) #33-Ubuntu SMP Sun Sep 19 20:48:58 UTC 2010
/home/hadoop/hive-0.6.0/bin/hive -e "insert overwrite table varnish_data select
host,to_date(from_unixtime(unix_timestamp(regexp_extract(time, '^.([^:]*):.*',
1),'dd/MMM/yyyy'))),substr(regexp_extract(request, 'http://(\\S*)\/..', 1), 1,
instr(regexp_extract(request, 'http://(\\S*)\/..',
1),'/')-1),status,size,regexp_extract(referer, 'http://(\\S*)/', 1),agent FROM
varnish;"
hive> desc varnish_data;
OK
host string
time string
request string
status string
size int
referer string
agent string
Time taken: 0.073 seconds
hive> select * from varnish_data limit 5;
OK
115.87.233.142 2011-08-04 200 5012 "Mozilla/4.0
(compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET
CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2)"
202.76.19.134 2011-08-04 200 397 "Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322)"
171.243.127.174 2011-08-04 200 941 "Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB7.1; MS Internet Explorer; .NET
CLR 2.0.50727)"
121.33.94.45 2011-08-04 200 4273 "Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET4.0C; .NET4.0E)"
118.68.36.97 2011-08-04 200 941 "Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1; SV1)"
Time taken: 0.491 seconds
hadoop@hadoop-00:~$ uname
Linux
hadoop@hadoop-00:~$ cat /proc/version
Linux version 2.6.35-22-server (buildd@allspice) (gcc version 4.4.5
(Ubuntu/Linaro 4.4.4-14ubuntu4) ) #33-Ubuntu SMP Sun Sep 19 20:48:58 UTC 2010
/home/hadoop/hive-0.6.0/bin/hive -e "insert overwrite table varnish_data select
host,to_date(from_unixtime(unix_timestamp(regexp_extract(time, '^.([^:]*):.*',
1),'dd/MMM/yyyy'))),substr(regexp_extract(request, 'http://(\\S*)\/..', 1), 1,
instr(regexp_extract(request, 'http://(\\S*)\/..',
1),'/')-1),status,size,regexp_extract(referer, 'http://(\\S*)/', 1),agent FROM
varnish;"
hive> desc varnish_data;
OK
host string
time string
request string
status string
size int
referer string
agent string
Time taken: 0.073 seconds
hive> insert overwrite table varnish_data select
host,to_date(from_unixtime(unix_timestamp(regexp_extract(time, '^.([^:]*):.*',
1),'dd/MMM/yyyy'))),substr(regexp_extract(request, 'http://(\\S*)\/..', 1), 1,
instr(regexp_extract(request, 'http://(\\S*)\/..',
1),'/')-1),status,size,regexp_extract(referer, 'http://(\\S*)/', 1),agent FROM
varnish;
hive> select * from varnish_data limit 5;
OK
115.87.233.142 2011-08-04 image.155551.com 200
5012 7uwak.155551.com "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT
5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR
3.5.30729; InfoPath.2)"
202.76.19.134 2011-08-04 image.8899988.com 200 397
dxiqd.8899988.com "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2;
SV1; .NET CLR 1.1.4322)"
171.243.127.174 2011-08-04 image.333332.com 200 941
a5vpni7x.333332.com "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;
SV1; GTB7.1; MS Internet Explorer; .NET CLR 2.0.50727)"
121.33.94.45 2011-08-04 image.8899988.com 200 4273
u1u1u.8899988.com "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;
SV1; .NET4.0C; .NET4.0E)"
118.68.36.97 2011-08-04 image.333332.com 200 941
q9r8m.333332.com "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;
SV1)"
Time taken: 0.418 seconds
