John, just wanted to thank you for your help. I've also pasted in the file, a sample
data file and the output to this message, in case anyone's following along in the
archives.
You're suggestions to improve time were amazing. I went from analyzing an 84,000 line
log file in 30 minutes to just seconds.
Thank you again for all your help.
-Kevin Zembower
=================================================
[EMAIL PROTECTED]:/opt/analog/logdata/db$ cat listqueries3.pl
#!/usr/bin/perl
use warnings;
use strict;
my $debug = 1;
while (<>) {
next if /^#/; #Skip it if it's one of the comments at the beginning of the file.
my ($date, $time, $source, undef, undef, $host, $FQDN, $method, $file, $query,
undef) = split;
next unless $file =~ /dbtwpcgi\.exe/ and $query =~ /&TN=popline&/i and $query =~
/QF0/;
my $type = $query =~ /QI2/? 'A': 'B';
my %hash = (abstract =>'', author =>'', engtitle =>'', docno =>'', keywordsmajor
=>'', journaltitle =>'', languages =>'', popreporttopic =>'', refereed =>'', year
=>'');
#This is the more elaborate version, suggested by John Krahn
$hash{ lc $2 } = $3 while $query =~
/QF(\d+)=\b(abstract|author|engtitle|docno|keywordsmajor|journaltitle|languages|popreporttopic|refereed|year)\b(?=.*?&QI\1=([^&]*))/ig;
my $outstring = join("\t", $type, $date, $time, @hash{ qw( abstract author engtitle
docno keywordsmajor journaltitle languages popreporttopic refereed year)}) . "\n";
print translate($outstring);
}# while there are more lines in the input file
sub translate() {
$_ = $_[0];
s/%22/\"/g;
s/%2C/,/g;
s/%20/ /g;
s#%2F#/#g;
s/%3D/=/g;
s/%3B/;/g;
s/%26/&/g;
s/%0D//g;
s/%0A//g;
s/\+/ /g;
s/%29/)/g;
s/%28/(/g;
s/%27/'/g;
s/%2b/+/g;
s/%7C/|/g;
s/%3A/:/g;
#Debbie request all boolean logical words and sumbols be replaced with '|'
s/\band\b/|/ig;
s/\bor\b/|/ig;
tr!&/!|!;
$_;
}
[EMAIL PROTECTED]:/opt/analog/logdata/db$ cat x
2004-03-28 00:38:31 d7.facsmf.utexas.edu - W3SVC1 DB db.jhuccp.org GET
/dbtw-wpd/exec/dbtwpcgi.exe
XC=%2Fdbtw-wpd%2Fexec%2Fdbtwpcgi.exe&BU=http%3A%2F%2Fdb.jhuccp.org%2Fpopinform%2Fbasic.html&QB0=AND&QF0=Abstract+%7C+KeywordsMajor+%7C+KeywordsMinor+%7C+Notes+%7C+EngTitle+%7C+TT+%7C+FREAb+%7C+SPAAb&QI0=China%0D%0A&QB1=AND&QF1=Author+%7C+CN&QI1=&MR=10&TN=popline&RF=ShortRecordDisplay&DF=LongRecordDisplay&DL=1&RL=1&NP=0&AC=QBE_QUERY&x=37&y=4
200 0 21248 814 19391 80 HTTP/1.1
Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+.NET+CLR+1.0.3705) -
http://db.jhuccp.org/popinform/basic.html
2004-03-28 00:38:31 d7.facsmf.utexas.edu - W3SVC1 DB db.jhuccp.org GET
/dbtw-wpd/jscript/dbtw-r-main.js - 200 0 6256 647 172 80 HTTP/1.1
Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+.NET+CLR+1.0.3705) -
http://db.jhuccp.org/dbtw-wpd/exec/dbtwpcgi.exe?XC=%2Fdbtw-wpd%2Fexec%2Fdbtwpcgi.exe&BU=http%3A%2F%2Fdb.jhuccp.org%2Fpopinform%2Fbasic.html&QB0=AND&QF0=Abstract+%7C+KeywordsMajor+%7C+KeywordsMinor+%7C+Notes+%7C+EngTitle+%7C+TT+%7C+FREAb+%7C+SPAAb&QI0=China%0D%0A&QB1=AND&QF1=Author+%7C+CN&QI1=&MR=10&TN=popline&RF=ShortRecordDisplay&DF=LongRecordDisplay&DL=1&RL=1&NP=0&AC=QBE_QUERY&x=37&y=4
2004-03-28 00:38:31 d7.facsmf.utexas.edu - W3SVC1 DB db.jhuccp.org GET
/popinform/res-style.css - 200 0 2212 639 63 80 HTTP/1.1
Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+.NET+CLR+1.0.3705) -
http://db.jhuccp.org/dbtw-wpd/exec/dbtwpcgi.exe?XC=%2Fdbtw-wpd%2Fexec%2Fdbtwpcgi.exe&BU=http%3A%2F%2Fdb.jhuccp.org%2Fpopinform%2Fbasic.html&QB0=AND&QF0=Abstract+%7C+KeywordsMajor+%7C+KeywordsMinor+%7C+Notes+%7C+EngTitle+%7C+TT+%7C+FREAb+%7C+SPAAb&QI0=China%0D%0A&QB1=AND&QF1=Author+%7C+CN&QI1=&MR=10&TN=popline&RF=ShortRecordDisplay&DF=LongRecordDisplay&DL=1&RL=1&NP=0&AC=QBE_QUERY&x=37&y=4
2004-03-30 14:59:24 218.11.80.208 - W3SVC1 DB db.jhuccp.org GET
/dbtw-wpd/jscript/dbtw-r-main.js - 200 0 6256 444 2078 80 HTTP/1.1
Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+NT+5.0) -
http://db.jhuccp.org/dbtw-wpd/exec/dbtwpcgi.exe?BU=http%3A%2F%2Fdb.jhuccp.org%2Fdbtw-wpd%2Fexec%2Fdbtwpcgi.exe&QF0=Country&QI0=China&TN=Condoms&AC=QBE_QUERY&MR=30&RF=ShortRecordDisplay&NP=3&DF=results_long
2004-03-30 14:59:24 218.11.80.208 - W3SVC1 DB db.jhuccp.org GET
/dbtw-wpd/exec/dbtwpcgi.exe
BU=http%3A%2F%2Fdb.jhuccp.org%2Fdbtw-wpd%2Fexec%2Fdbtwpcgi.exe&QF0=Country&QI0=China&TN=Condoms&AC=QBE_QUERY&MR=30&RF=ShortRecordDisplay&NP=3&DF=results_long
200 0 33987 588 5109 80 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+NT+5.0)
- http://condoms.jhuccp.org/asia.html
2004-03-31 13:28:05 proxy.unam.na - W3SVC1 DB db.jhuccp.org GET
/dbtw-wpd/exec/dbtwpcgi.exe
XC=%2Fdbtw-wpd%2Fexec%2Fdbtwpcgi.exe&BU=http%3A%2F%2Fdb.jhuccp.org%2Fpopinform%2Fbasic.html&QB0=AND&QF0=Abstract+%7C+KeywordsMajor+%7C+KeywordsMinor+%7C+Notes+%7C+EngTitle+%7C+TT+%7C+FREAb+%7C+SPAAb&QI0=child+labour+AND++literacy+NOT+China+&QB1=AND&QF1=Author+%7C+CN&QI1=&MR=10&TN=popline&RF=ShortRecordDisplay&DF=LongRecordDisplay&DL=1&RL=1&NP=0&AC=QBE_QUERY&x=22&y=15
200 0 751 821 1484 80 HTTP/1.0 Mozilla/4.75+[en]+(Win98;+U) -
http://db.jhuccp.org/popinform/basic.html
2004-03-31 19:41:46 172.16.4.160 - W3SVC1 DB db.jhuccp.org GET
/dbtw-wpd/jscript/dbtw-r-main.js - 200 0 6256 874 15 80 HTTP/1.1
Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+NT+4.0) -
http://db.jhuccp.org/dbtw-wpd/exec/dbtwpcgi.exe?XC=%2Fdbtw-wpd%2Fexec%2Fdbtwpcgi.exe&BU=http%3A%2F%2Fdb.jhuccp.org%2Fpopinform%2Fexpert.html&MR=10&QB10=AND&QF10=OnlineDate&QI10=&QF0=Author+%7C+CN&QI0=&QB0=AND&QF1=EngTitle+%7C+TT&QI1=&QB1=AND&QF5=JournalTitle+%7C+MI+%7C+Imprint+%7C+Series&QI5=&QB5=AND&QF9=Refereed&QF3=Abstract%7CKeywordsMajor%7CKeywordsMinor%7CNotes%7CEngTitle%7CTT%7CFREAb%7CSPAAb&QI3=IUDs+%26+China&QB3=AND&QF6=Year&QI6=&QB6=AND&QF4=Languages&QI4=&QB4=AND&QF2=KeywordsMajor%7CKeywordsMinor&QI2=&QB2=AND&QF7=DocNo&QI7=&QB7=AND&TN=popline&RF=ShortRecordDisplay&DF=LongRecordDisplay&DL=1&RL=1&NP=0&AC=QBE_QUERY&x=21&y=6
2004-03-31 19:41:46 172.16.4.160 - W3SVC1 DB db.jhuccp.org GET
/dbtw-wpd/exec/dbtwpcgi.exe
XC=%2Fdbtw-wpd%2Fexec%2Fdbtwpcgi.exe&BU=http%3A%2F%2Fdb..jhuccp.org%2Fpopinform%2Fexpert.html&MR=10&QB10=AND&QF10=OnlineDate&QI10=&QF0=Author+%7C+CN&QI0=&QB0=AND&QF1=EngTitle+%7C+TT&QI1=&QB1=AND&QF5=JournalTitle+%7C+MI+%7C+Imprint+%7C+Series&QI5=&QB5=AND&QF9=Refereed&QF3=Abstract%7CKeywordsMajor%7CKeywordsMinor%7CNotes%7CEngTitle%7CTT%7CFREAb%7CSPAAb&QI3=IUDs+%26+China&QB3=AND&QF6=Year&QI6=&QB6=AND&QF4=Languages&QI4=&QB4=AND&QF2=KeywordsMajor%7CKeywordsMinor&QI2=&QB2=AND&QF7=DocNo&QI7=&QB7=AND&TN=popline&RF=ShortRecordDisplay&DF=LongRecordDisplay&DL=1&RL=1&NP=0&AC=QBE_QUERY&x=21&y=6
200 0 21063 1025 2891 80 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+NT+4.0)
- http://db.jhuccp.org/popinform/expert.html
2004-03-31 19:41:46 172.16.4.160 - W3SVC1 DB db.jhuccp.org GET /popinform/infoCart.js
- 304 0 142 953 0 80 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+NT+4.0) -
http://db.jhuccp.org/dbtw-wpd/exec/dbtwpcgi.exe?XC=%2Fdbtw-wpd%2Fexec%2Fdbtwpcgi.exe&BU=http%3A%2F%2Fdb.jhuccp.org%2Fpopinform%2Fexpert.html&MR=10&QB10=AND&QF10=OnlineDate&QI10=&QF0=Author+%7C+CN&QI0=&QB0=AND&QF1=EngTitle+%7C+TT&QI1=&QB1=AND&QF5=JournalTitle+%7C+MI+%7C+Imprint+%7C+Series&QI5=&QB5=AND&QF9=Refereed&QF3=Abstract%7CKeywordsMajor%7CKeywordsMinor%7CNotes%7CEngTitle%7CTT%7CFREAb%7CSPAAb&QI3=IUDs+%26+China&QB3=AND&QF6=Year&QI6=&QB6=AND&QF4=Languages&QI4=&QB4=AND&QF2=KeywordsMajor%7CKeywordsMinor&QI2=&QB2=AND&QF7=DocNo&QI7=&QB7=AND&TN=popline&RF=ShortRecordDisplay&DF=LongRecordDisplay&DL=1&RL=1&NP=0&AC=QBE_QUERY&x=21&y=6
2004-03-31 19:41:46 172.16.4.160 - W3SVC1 DB db.jhuccp.org GET
/popinform/res-style.css - 200 0 2212 866 0 80 HTTP/1.1
Mozilla/4.0+(compatible;+MSIE+5..5;+Windows+NT+4.0) -
http://db.jhuccp.org/dbtw-wpd/exec/dbtwpcgi.exe?XC=%2Fdbtw-wpd%2Fexec%2Fdbtwpcgi.exe&BU=http%3A%2F%2Fdb.jhuccp.org%2Fpopinform%2Fexpert.html&MR=10&QB10=AND&QF10=OnlineDate&QI10=&QF0=Author+%7C+CN&QI0=&QB0=AND&QF1=EngTitle+%7C+TT&QI1=&QB1=AND&QF5=JournalTitle+%7C+MI+%7C+Imprint+%7C+Series&QI5=&QB5=AND&QF9=Refereed&QF3=Abstract%7CKeywordsMajor%7CKeywordsMinor%7CNotes%7CEngTitle%7CTT%7CFREAb%7CSPAAb&QI3=IUDs+%26+China&QB3=AND&QF6=Year&QI6=&QB6=AND&QF4=Languages&QI4=&QB4=AND&QF2=KeywordsMajor%7CKeywordsMinor&QI2=&QB2=AND&QF7=DocNo&QI7=&QB7=AND&TN=popline&RF=ShortRecordDisplay&DF=LongRecordDisplay&DL=1&RL=1&NP=0&AC=QBE_QUERY&x=21&y=6
2004-04-30 23:56:26 dhcp-103-107.sphtm.tulane.edu - W3SVC1 DB db.jhuccp.org GET
/dbtw-wpd/exec/dbtwpcgi.exe
BU=http%3A%2F%2Fdb.jhuccp.org%2Fdbtw-wpd%2Fexec%2Fdbtwpcgi.exe&QF0=KeywordsMajor%&QF1=Year&QI0=HIV_Infections*%26women%&QI1=2001%3A2004&TN=Popline&AC=QBE_QUERY&MR=30&DL=1&&RL=1&&RF=ShortRecordDisplay&DF=LongRecordDisplay
200 0 42970 594 1610 80 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1) -
http://db.jhuccp.org/popinform/basic.html
[EMAIL PROTECTED]:/opt/analog/logdata/db$ ./listqueries3.pl x
main::translate() called too early to check prototype at ./listqueries3.pl line 19.
B 2004-03-28 00:38:31 China
B 2004-03-31 13:28:05 child labour | literacy NOT China
A 2004-03-31 19:41:46 IUDs | China
B 2004-04-30 23:56:26
HIV_Infections*|women% 2001:2004
[EMAIL PROTECTED]:/opt/analog/logdata/db$
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
<http://learn.perl.org/> <http://learn.perl.org/first-response>
