On 10/26/2013 06:11 PM, Nick the Gr33k wrote:
Στις 27/10/2013 2:52 πμ, ο/η Nick the Gr33k έγραψε:
Ah foun it had to change in you code this line:
key = host, city, useros, browser, ref
to this line:
key = host, city, useros, browser
so 'ref' wouldnt be calculated in the unique combination key.
I'am still trying to understand the logic of your code and trying to
create a history list column for the 'referrers'
I dont know how to write it though to produce the sam
Iam trying.
Ah foun it had to change in you code this line:
key = host, city, useros, browser, ref
to this line:
key = host, city, useros, browser
so 'ref' wouldnt be calculated in the unique combination key.
I'am still trying to understand the logic of your code and trying to
create a history list column for the 'referrers'
I dont know how to write it though to produce the same output for referrers.
The bast i came up with is:
[code]
def coalesce( data ):
newdata = []
seen = {}
for host, city, useros, browser, ref, hits, visit in data:
# Here i have to decide how to group the rows together.
# I want an html row for every unique combination of
(host, city,
useros, browser) and that hits should be summed together.
key = host, city, useros, browser
if key not in seen:
newdata.append( [host, city, useros, browser,
[ref], hits, [visit]] )
seen[key] = len( newdata ) - 1 # Save
index (for 'newdata') of this
row.
else: # This row is a duplicate row with a
different visit time.
rowindex = seen[key]
newdata[rowindex][4].append( ref )
newdata[rowindex][5] += hits
newdata[rowindex][6].append( visit )
return newdata
cur.execute( '''SELECT host, city, useros, browser, ref, hits,
lastvisit FROM visitors
WHERE counterID = (SELECT ID FROM
counters WHERE url = %s) ORDER BY
lastvisit DESC''', page )
data = cur.fetchall()
newdata = coalesce( data )
for row in newdata:
(host, city, useros, browser, refs, hits, visits) = row
# Note that 'ref' & 'visits' are now lists of visit times.
print( "<tr>" )
for item in (host, city, useros, browser):
print( "<td><center><b><font color=white> %s </td>" %
item )
print( "<td><select>" )
for n, ref in enumerate( refs ):
if n == 0:
op_selected = 'selected="selected"'
else:
op_selected = ''
print( "<option %s>%s</option>" % (op_selected, ref) )
print( "</select></td>" )
for item in (hits):
print( "<td><center><b><font color=white> %s </td>" %
item )
print( "<td><select>" )
for n, visit in enumerate( visits ):
visittime = visit.strftime('%A %e %b, %H:%M')
if n == 0:
op_selected = 'selected="selected"'
else:
op_selected = ''
print( "<option %s>%s</option>" % (op_selected,
visittime) )
print( "</select></td>" )
print( "</tr>" )
[/code]
But this doesnt work correctly for refs and also doenst not print for
some reason the hits and visit colums.
Without a traceback it is hard to figure out what is happening.
(Actually in this case there is one obvious error, but there are
also some unobvious ones.)
Here is what I did to find the problems, and what you can do
the next time. The main thing was to extract the code from
the cgi script so that I could run it outside of the web server
and without needing access to the database. Then you can add
print statements (or run with the pdb debugger) and see tracebacks
and other errors easily.
1. Copy and paste the code from your message into a .py file.
2. Put a "def main(): line at the top of your main code.
3. Add a line at the bottom to call main()
4. Copy and paste a part of the web page you gave that list all the visits.
5. Edit it (change TAB to "|", add comma's after each line, etc, to
create a statement that will create variable, DATA.
6. Add a few statements to turn DATA into variable 'data' which has a
format similar to the format returned by your cur.fetchall() call.
This all took just 10 or 15 minutes, and I ended up with the
following code:
--------------
def main():
data = [ln.split('|') for ln in DATA]
for r in data: r[5] = int(r[5]) # Change the 'hit' values from str to
int.
print ('<table border="1">')
newdata = coalesce( data )
for row in newdata:
(host, city, useros, browser, refs, hits, visits) = row
# Note that 'ref' & 'visits' are now lists of visit times.
print( "<tr>" )
for item in (host, city, useros, browser):
print( "<td><center><b><font color=white> %s </td>" %
item )
print( "<td><select>" )
for n, ref in enumerate( refs ):
if n == 0:
op_selected = 'selected="selected"'
else:
op_selected = ''
print( "<option %s>%s</option>" % (op_selected, ref) )
print( "</select></td>" )
for item in (hits):
print( "<td><center><b><font color=white> %s </td>" %
item )
print( "<td><select>" )
for n, visit in enumerate( visits ):
visittime = visit.strftime('%A %e %b, %H:%M')
if n == 0:
op_selected = 'selected="selected"'
else:
op_selected = ''
print( "<option %s>%s</option>" % (op_selected,
visittime) )
print( "</select></td>" )
print( "</tr>" )
def coalesce( data ):
newdata = []
seen = {}
for host, city, useros, browser, ref, hits, visit in data:
# Here i have to decide how to group the rows together.
# I want an html row for every unique combination of
(host, city, useros, browser) and that hits should be summed together.
key = host, city, useros, browser
if key not in seen:
newdata.append( [host, city, useros, browser,
[ref], hits, [visit]] )
seen[key] = len( newdata ) - 1 # Save
index (for 'newdata') of this row.
else: # This row is a duplicate row with a
different visit time.
rowindex = seen[key]
newdata[rowindex][4].append( ref )
newdata[rowindex][5] += hits
newdata[rowindex][6].append( visit )
return newdata
DATA = [
'209.133.77.165.T01713-01.above.net|Άγνωστη Πόλη|Windows|Explorer|Direct
Hit|1|Σάββατο 26 Οκτ, 18:59',
'mail14.ess.barracuda.com|Άγνωστη Πόλη|Windows|Explorer|Direct
Hit|1|Σάββατο 26 Οκτ, 18:49',
'mail14.ess.barracuda.com|Άγνωστη Πόλη|Windows|Explorer|Direct
Hit|1|Σάββατο 26 Οκτ, 18:48',
'mail0.ess.barracuda.com|Άγνωστη Πόλη|Windows|Explorer|Direct
Hit|1|Σάββατο 26 Οκτ, 18:48',
'mail0.ess.barracuda.com|Άγνωστη Πόλη|Windows|Explorer|Direct
Hit|1|Σάββατο 26 Οκτ, 18:47',
'209.133.77.164.T01713-01.above.net|Άγνωστη Πόλη|Windows|Explorer|Direct
Hit|1|Σάββατο 26 Οκτ, 18:47',
'89-145-108-206.as29017.net|Άγνωστη Πόλη|Windows|Explorer|Direct
Hit|1|Σάββατο 26 Οκτ, 18:47',
]
main()
-------------
I then ran it and it reported:
[...some html output...]
Traceback (most recent call last):
File "xx2.py", line 88, in <module>
if __name__ == '__main__': main()
File "xx2.py", line 26, in main
for item in (hits):
TypeError: 'int' object is not iterable
Line 88 is:
for item in (hits):
Remember that 'hits' is just an integer number, not a list.
So I changed:
for item in (hits):
print( "<td><center><b><font color=white> %s </td>" % item )
to:
print( "<td><center><b><font color=white> %s </td>" % hits )
and ran again. This time:
Traceback (most recent call last):
File "xx2.py", line 87, in <module>
if __name__ == '__main__': main()
File "xx2.py", line 30, in main
visittime = visit.strftime('%A %e %b, %H:%M')
AttributeError: 'str' object has no attribute 'strftime'
This is not a problem with the program but with the input data. When
you get data from your database, 'visit' is a datetime object. But
when it comes from the synthetic data we created, it is an already-
formatted string.
So I replaced
:
visittime = visit.strftime('%A %e %b, %H:%M')
with
visittime = visit #.strftime('%A %e %b, %H:%M')
I also realized you use white fonts so I changed the table to have
a blue background color:
print ('<table border="1">')
to
print ('<table border="1" bgcolor="blue">')
Now when I run the program it runs without errors and produces html
output. So run again but save the output to a file.
$ python3 test.py > test.html
And open the file with a browser. Looks ok but none of the 'ref'
buttons has more than one entry. So I edited the data to be:
DATA = [
'209.133.77.165.T01713-01.above.net|Άγνωστη Πόλη|Windows|Explorer|Direct
Hit|1|Σάββατο 26 Οκτ, 18:59',
'mail14.ess.barracuda.com|Άγνωστη Πόλη|Windows|Explorer|Direct
Hit|1|Σάββατο 26 Οκτ, 18:49',
'mail14.ess.barracuda.com|Άγνωστη
Πόλη|Windows|Explorer|http://superhost.gr/|1|Σάββατο 26 Οκτ, 18:48',
'mail0.ess.barracuda.com|Άγνωστη Πόλη|Windows|Explorer|Direct
Hit|1|Σάββατο 26 Οκτ, 18:48',
'mail0.ess.barracuda.com|Άγνωστη Πόλη|Windows|Explorer|Direct
Hit|1|Σάββατο 26 Οκτ, 18:47',
'209.133.77.164.T01713-01.above.net|Άγνωστη Πόλη|Windows|Explorer|Direct
Hit|1|Σάββατο 26 Οκτ, 18:47',
'mail14.ess.barracuda.com|Άγνωστη
Πόλη|Windows|Explorer|http://mythosweb.gr/|1|Σάββατο 26 Οκτ, 18:22',
]
And ran again. But still, the 'ref' button for mail14.ess.barracuda.com
showed on one item in the dropdown list. After looking at the generated
source code and adding some print statement to the python code I realized
that you had:
for n, ref in enumerate( refs ):
if n == 0:
op_selected = 'selected="selected"'
else:
op_selected = ''
print( "<option %s>%s</option>" % (op_selected, ref) )
but what you want is:
for n, ref in enumerate( refs ):
if n == 0:
op_selected = 'selected="selected"'
else:
op_selected = ''
print( "<option %s>%s</option>" % (op_selected, ref) )
I also realized after looking at the HTML spec for the OPTION element
(http://www.w3.org/TR/html401/interact/forms.html#h-17.6)
that 'selected="selected"' should be just 'selected'. So in
both places they occur, I changed
if n == 0:
op_selected = 'selected="selected"'
to
if n == 0:
op_selected = 'selected'
Now the generated page looks right and the only thing left to do
in to copy the fixed code back into your main cgi script, remember
to undo the temp change made for testing:
visittime = visit #.strftime('%A %e %b, %H:%M')
back to
visittime = visit.strftime('%A %e %b, %H:%M')
So here is the fixed code:
----------------
newdata = coalesce( data )
for row in newdata:
(host, city, useros, browser, refs, hits, visits) = row
# Note that 'ref' & 'visits' are now lists of visit times.
print( "<tr>" )
for item in (host, city, useros, browser):
print( "<td><center><b><font color=white> %s </td>" %
item )
print( "<td><select>" )
for n, ref in enumerate( refs ):
if n == 0:
op_selected = 'selected'
else:
op_selected = ''
print( "<option %s>%s</option>" % (op_selected, ref) )
print( "</select></td>" )
print( "<td><center><b><font color=white> %s </td>" % hits )
print( "<td><select>" )
for n, visit in enumerate( visits ):
visittime = visit.strftime('%A %e %b, %H:%M')
if n == 0:
op_selected = 'selected'
else:
op_selected = ''
print( "<option %s>%s</option>" % (op_selected,
visittime) )
print( "</select></td>" )
print( "</tr>" )
def coalesce( data ):
newdata = []
seen = {}
for host, city, useros, browser, ref, hits, visit in data:
# Here i have to decide how to group the rows together.
# I want an html row for every unique combination of
(host, city, useros, browser) and that hits should be summed together.
key = host, city, useros, browser
if key not in seen:
newdata.append( [host, city, useros, browser,
[ref], hits, [visit]] )
seen[key] = len( newdata ) - 1 # Save
index (for 'newdata') of this row.
else: # This row is a duplicate row with a
different visit time.
rowindex = seen[key]
newdata[rowindex][4].append( ref )
newdata[rowindex][5] += hits
newdata[rowindex][6].append( visit )
return newdata
----------------
