fastest method

[david.gar...@gmail.com]

I am looking for the fastest way to parse a log file.


currently I have this... Can I speed this up any? The script is written to
be a generic log file parser so I can't rely on some predictable pattern.


def check_data(data,keywords):
#get rid of duplicates
unique_list = list(set(data))
string_list=' '.join(unique_list)
#print string_list
for keyword in keywords:
if keyword in string_list:
return True


I am currently using file seek and maintaining a last byte count file:

with open(filename) as f:
print "Here is filename:%s" %filename
f.seek(0, 2)
eof = f.tell()
print "Here is eof:%s" %eof
if last is not None:
print "Here is last:%s" %last
# if last is less than current
last = int(last)
if (eof - last  > 0):
offset = eof - last
offset = offset * -1
print "Here is new offset:%s" %offset
f.seek(offset, 2)
mylist = f.readlines()
else:
# if last doesn't exist or is greater than current
f.seek(0)
bof = f.tell()
print "Here is bof:%s" %bof
mylist = f.readlines()



Thanks,
-- 
David Garvey
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: fastest method

[david.gar...@gmail.com]

I see one issue;)

# if last doesn't exist or is greater than current

This else doesn't catch the last greater than current:


This is a little messy.


with open(filename) as f:
print "Here is filename:%s" %filename
f.seek(0, 2)
eof = f.tell()
print "Here is eof:%s" %eof
if last is not None:
print "Here is last:%s" %last
# if last is less than current
last = int(last)
if (eof - last  > 0):
offset = eof - last
offset = offset * -1
print "Here is new offset:%s" %offset
f.seek(offset, 2)
mylist = f.readlines()
else:
f.seek(0)
bof = f.tell()
print "Here is bof:%s" %bof
mylist = f.readlines()
else:
f.seek(0)
bof = f.tell()
print "Here is bof:%s" %bof
    mylist = f.readlines()




On Wed, Jun 20, 2012 at 11:17 AM, david.gar...@gmail.com <
david.gar...@gmail.com> wrote:

> I am looking for the fastest way to parse a log file.
>
>
> currently I have this... Can I speed this up any? The script is written to
> be a generic log file parser so I can't rely on some predictable pattern.
>
>
> def check_data(data,keywords):
> #get rid of duplicates
> unique_list = list(set(data))
> string_list=' '.join(unique_list)
> #print string_list
> for keyword in keywords:
> if keyword in string_list:
> return True
>
>
> I am currently using file seek and maintaining a last byte count file:
>
> with open(filename) as f:
> print "Here is filename:%s" %filename
> f.seek(0, 2)
> eof = f.tell()
> print "Here is eof:%s" %eof
> if last is not None:
> print "Here is last:%s" %last
> # if last is less than current
> last = int(last)
> if (eof - last  > 0):
> offset = eof - last
> offset = offset * -1
> print "Here is new offset:%s" %offset
> f.seek(offset, 2)
> mylist = f.readlines()
> else:
> # if last doesn't exist or is greater than current
> f.seek(0)
> bof = f.tell()
> print "Here is bof:%s" %bof
> mylist = f.readlines()
>
>
>
> Thanks,
> --
> David Garvey
>



-- 
David Garvey
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Web Server for Python

[david.gar...@gmail.com]

Hi Mura,

apache is used as a proxy with python. You would fire up your python web
app and put apache in front of it. Apache does come these days with wsgi
https://code.google.com/p/modwsgi/ that can be added to apache as module,
something like mod_wsgi can be used for your python web app. PHP is another
web app language similar to python that would be sitting behind apache and
would be loaded with PHP code. http://dan.drydog.com/apache2php.html

Example of python web app frameworks would be:
django: https://docs.djangoproject.com/en/1.2/howto/deployment/modwsgi/
web.py http://webpy.org/

And a lot more...

I hope this helps,
Cheers, dgarvey


On Mon, Dec 16, 2013 at 6:21 PM, Mura Zalukhu  wrote:

> Hi all,
> I am a newbiew for python,
> i want to improve my programming for python.
> but i have searching any tutorial to configuration python for webserver
> but no one of tutorial solved my problem.
>
> could you give me how to configuration ptyhon on webserver like apache /
> php or
> Thanks so much
> --
> https://mail.python.org/mailman/listinfo/python-list
>



-- 
David Garvey
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: gdb python core dump file : not in executable format: File format not

[david.gar...@gmail.com]

Does this help?

http://plasmodic.github.io/ecto/ecto/usage/external/debugging.html


http://gnuradio.org/redmine/projects/gnuradio/wiki/TutorialsDebugging


http://downloads.conceptive.be/downloads/camelot/doc/sphinx/build/advanced/debug.html

http://forums.gentoo.org/viewtopic-p-7123814.html


On Mon, Apr 14, 2014 at 1:19 AM, Wesley  wrote:

> Hi guys,
>Today I am debugging an issue related to memory leak.
> I use gdb 7.7 and python 2.7.6 to generate one core dump file from
> production env.
>
> And then, just use gdb to debug the coredump upon the same machine.
> Got error that seems not support debug core file using pyton?
>
> Here is snippet:
> [root@localhost server]# gdb --core  memleak.core
> GNU gdb (GDB) 7.7
> Copyright (C) 2014 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> .
> Find the GDB manual and other documentation resources online at:
> .
> For help, type "help".
> Type "apropos word" to search for commands related to "word".
> [New LWP 25738]
> [New LWP 25739]
> [New LWP 25740]
> [New LWP 25745]
> [New LWP 25746]
> [New LWP 25747]
> [New LWP 25635]
> Core was generated by `python'.
> #0  0x0030016e15e3 in ?? ()
> (gdb) file /root/server/deviceserver.py
> "/root/server/deviceserver.py": not in executable format: File format not
> recognized
> (gdb) file /root/server/deviceserver
> /root/server/deviceserver: No such file or directory.
> (gdb) file /root/server/deviceserver.py
> "/root/server/deviceserver.py": not in executable format: File format not
> recognized
> (gdb)
> --
> https://mail.python.org/mailman/listinfo/python-list
>



-- 
David Garvey
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What use with Python

[david.gar...@gmail.com]

How about Web python?
https://wiki.python.org/moin/WebFrameworks
PC can run linux.
Perhaps a vagrant linux virtual host would be in order?
http://www.vagrantup.com/
Or windoz https://www.python.org/download/windows

David


On Tue, Jul 1, 2014 at 3:56 PM,  wrote:

> Hi,
>
> I am new to Python. Weeks ago, I was asked about Python questions on an
> interview.
> Now I want to learn Python, but I do not know what I can do with it on a
> PC.
> Especially I would like to do something interesting instead of some text
> search etc.
> Python may can do more than I realize now. Could you tell me something on
> a PC?
>
> Thanks,
> --
> https://mail.python.org/mailman/listinfo/python-list
>



-- 
David Garvey
-- 
https://mail.python.org/mailman/listinfo/python-list

classes and __iter__

[david.gar...@gmail.com]

Hello,

I have a class and i return both a key list and dictionary from the class.
Is it good form to do this? The print helo.__dict__ shows both the list and
dictionary.



>>> class Parse_Nagios_Header:
... def __init__(self):
... self.keylist = []
... self.d = {}
... f = open("common.h")
... lines = f.readlines()
... lines = filter(lambda x: not x.isspace(), lines)
... f.close()
... for line in lines:
... if re.search("CMD", line):
... line = line.lstrip('#define ')
... line = line.strip()
... line.replace('\t', ' ')
... line = line.split()
... if len(line) > 2:
... pass
... elif len(line) == 2:
... line[1] = int(line[1])
... self.d[line[1]] = line[0]
... self.keylist = sorted(self.d.iterkeys())
... def __iter__(self):
... return iter(self.keylist, self.d)
...
>>> helo = Parse_Nagios_Header()
>>> print helo
<__main__.Parse_Nagios_Header instance at 0x7fbf5b09b488>
>>> for key in helo.keylist:
... print "Key:%s Value:%s" %(key,helo.d[key])
...
Key:0 Value:CMD_NONE
Key:1 Value:CMD_ADD_HOST_COMMENT
Key:2 Value:CMD_DEL_HOST_COMMENT
Key:3 Value:CMD_ADD_SVC_COMMENT
Key:4 Value:CMD_DEL_SVC_COMMENT
Key:5 Value:CMD_ENABLE_SVC_CHECK

-- 
David Garvey
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: classes and __iter__

[david.gar...@gmail.com]

'CMD_SCHEDULE_FORCED_HOST_CHECK', 99: 'CMD_START_OBSESSING_OVER_SVC', 100:
'CMD_STOP_OBSESSING_OVER_SVC', 101: 'CMD_START_OBSESSING_OVER_HOST', 102:
'CMD_STOP_OBSESSING_OVER_HOST', 103: 'CMD_ENABLE_HOSTGROUP_HOST_CHECKS',
104: 'CMD_DISABLE_HOSTGROUP_HOST_CHECKS', 105:
'CMD_ENABLE_HOSTGROUP_PASSIVE_SVC_CHECKS', 106:
'CMD_DISABLE_HOSTGROUP_PASSIVE_SVC_CHECKS', 107:
'CMD_ENABLE_HOSTGROUP_PASSIVE_HOST_CHECKS', 108:
'CMD_DISABLE_HOSTGROUP_PASSIVE_HOST_CHECKS', 109:
'CMD_ENABLE_SERVICEGROUP_SVC_NOTIFICATIONS', 110:
'CMD_DISABLE_SERVICEGROUP_SVC_NOTIFICATIONS', 111:
'CMD_ENABLE_SERVICEGROUP_HOST_NOTIFICATIONS', 112:
'CMD_DISABLE_SERVICEGROUP_HOST_NOTIFICATIONS', 113:
'CMD_ENABLE_SERVICEGROUP_SVC_CHECKS', 114:
'CMD_DISABLE_SERVICEGROUP_SVC_CHECKS', 115:
'CMD_ENABLE_SERVICEGROUP_HOST_CHECKS', 116:
'CMD_DISABLE_SERVICEGROUP_HOST_CHECKS', 117:
'CMD_ENABLE_SERVICEGROUP_PASSIVE_SVC_CHECKS', 118:
'CMD_DISABLE_SERVICEGROUP_PASSIVE_SVC_CHECKS', 119:
'CMD_ENABLE_SERVICEGROUP_PASSIVE_HOST_CHECKS', 120:
'CMD_DISABLE_SERVICEGROUP_PASSIVE_HOST_CHECKS', 121:
'CMD_SCHEDULE_SERVICEGROUP_HOST_DOWNTIME', 122:
'CMD_SCHEDULE_SERVICEGROUP_SVC_DOWNTIME', 123:
'CMD_CHANGE_GLOBAL_HOST_EVENT_HANDLER', 124:
'CMD_CHANGE_GLOBAL_SVC_EVENT_HANDLER', 125:
'CMD_CHANGE_HOST_EVENT_HANDLER', 126: 'CMD_CHANGE_SVC_EVENT_HANDLER', 127:
'CMD_CHANGE_HOST_CHECK_COMMAND', 128: 'CMD_CHANGE_SVC_CHECK_COMMAND', 129:
'CMD_CHANGE_NORMAL_HOST_CHECK_INTERVAL', 130:
'CMD_CHANGE_NORMAL_SVC_CHECK_INTERVAL', 131:
'CMD_CHANGE_RETRY_SVC_CHECK_INTERVAL', 132:
'CMD_CHANGE_MAX_HOST_CHECK_ATTEMPTS', 133:
'CMD_CHANGE_MAX_SVC_CHECK_ATTEMPTS', 134:
'CMD_SCHEDULE_AND_PROPAGATE_TRIGGERED_HOST_DOWNTIME', 135:
'CMD_ENABLE_HOST_AND_CHILD_NOTIFICATIONS', 136:
'CMD_DISABLE_HOST_AND_CHILD_NOTIFICATIONS', 137:
'CMD_SCHEDULE_AND_PROPAGATE_HOST_DOWNTIME', 138:
'CMD_ENABLE_SERVICE_FRESHNESS_CHECKS', 139:
'CMD_DISABLE_SERVICE_FRESHNESS_CHECKS', 140:
'CMD_ENABLE_HOST_FRESHNESS_CHECKS', 141:
'CMD_DISABLE_HOST_FRESHNESS_CHECKS', 142:
'CMD_SET_HOST_NOTIFICATION_NUMBER', 143: 'CMD_SET_SVC_NOTIFICATION_NUMBER',
144: 'CMD_CHANGE_HOST_CHECK_TIMEPERIOD', 145:
'CMD_CHANGE_SVC_CHECK_TIMEPERIOD', 146: 'CMD_PROCESS_FILE', 147:
'CMD_CHANGE_CUSTOM_HOST_VAR', 148: 'CMD_CHANGE_CUSTOM_SVC_VAR', 149:
'CMD_CHANGE_CUSTOM_CONTACT_VAR', 150:
'CMD_ENABLE_CONTACT_HOST_NOTIFICATIONS', 151:
'CMD_DISABLE_CONTACT_HOST_NOTIFICATIONS', 152:
'CMD_ENABLE_CONTACT_SVC_NOTIFICATIONS', 153:
'CMD_DISABLE_CONTACT_SVC_NOTIFICATIONS', 154:
'CMD_ENABLE_CONTACTGROUP_HOST_NOTIFICATIONS', 155:
'CMD_DISABLE_CONTACTGROUP_HOST_NOTIFICATIONS', 156:
'CMD_ENABLE_CONTACTGROUP_SVC_NOTIFICATIONS', 157:
'CMD_DISABLE_CONTACTGROUP_SVC_NOTIFICATIONS', 158:
'CMD_CHANGE_RETRY_HOST_CHECK_INTERVAL', 159:
'CMD_SEND_CUSTOM_HOST_NOTIFICATION', 160:
'CMD_SEND_CUSTOM_SVC_NOTIFICATION', 161:
'CMD_CHANGE_HOST_NOTIFICATION_TIMEPERIOD', 162:
'CMD_CHANGE_SVC_NOTIFICATION_TIMEPERIOD', 163:
'CMD_CHANGE_CONTACT_HOST_NOTIFICATION_TIMEPERIOD', 164:
'CMD_CHANGE_CONTACT_SVC_NOTIFICATION_TIMEPERIOD', 165:
'CMD_CHANGE_HOST_MODATTR', 166: 'CMD_CHANGE_SVC_MODATTR', 167:
'CMD_CHANGE_CONTACT_MODATTR', 168: 'CMD_CHANGE_CONTACT_MODHATTR', 169:
'CMD_CHANGE_CONTACT_MODSATTR', 170: 'CMD_DEL_DOWNTIME_BY_HOST_NAME', 171:
'CMD_DEL_DOWNTIME_BY_HOSTGROUP_NAME', 172:
'CMD_DEL_DOWNTIME_BY_START_TIME_COMMENT', 999: 'CMD_CUSTOM_COMMAND'}}

On Mon, Jan 9, 2012 at 3:17 PM, Chris Rebert  wrote:

> On Mon, Jan 9, 2012 at 2:51 PM, david.gar...@gmail.com
>  wrote:
> 
> >>>> class Parse_Nagios_Header:
> > ... def __init__(self):
> > ... self.keylist = []
> > ... self.d = {}
> 
> > ... def __iter__(self):
> > ... return iter(self.keylist, self.d)
>
> No idea what you're expecting this __iter__() to do, but it doesn't work
> at all.
> Relevant docs:
> http://docs.python.org/library/functions.html#iter
> http://docs.python.org/reference/datamodel.html#object.__iter__
>
> Cheers,
> Chris
>



-- 
David Garvey
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: classes and __iter__

[david.gar...@gmail.com]

Thanks Ian & Chris for the conversation...



On Mon, Jan 9, 2012 at 4:15 PM, Chris Rebert  wrote:

> On Mon, Jan 9, 2012 at 3:30 PM, david.gar...@gmail.com
>  wrote:
> > Chris,
> >
> > Both a list and dict are both iterable.  I get a python dictionary
> object of
> > both iterables.;)
>
> No, you get a Python object with both iterables as instance variables.
> Instance variables happen to be stored using a dict (which is
> accessible as .__dict__), but that's to some extent an implementation
> detail whose relevance here I fail to see. My point was that, as Ian
> explained, your __iter__() method, as written, is horribly broken.
>
> > It is nice... but I don't know if this is good form?
>
> I'm confused as to why you wrote a class (particularly when its name
> is an *action* rather than a noun; big red flag right there!) for your
> task in the first place. I think you'd be best served by moving your
> parsing code into a function and using a
> http://pypi.python.org/pypi/sorteddict , which provides the
> sorted-keys property you seemed to be trying to accomplish.
>
> Also, please avoid top-posting in the future. (See
> http://en.wikipedia.org/wiki/Posting_style )
>
> Cheers,
> Chris
> --
> http://rebertia.com
>



-- 
David Garvey
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: classes and __iter__

[david.gar...@gmail.com]

I see your meaning for __iter__ method.;)

On Mon, Jan 9, 2012 at 4:57 PM, david.gar...@gmail.com <
david.gar...@gmail.com> wrote:

> Thanks Ian & Chris for the conversation...
>
>
>
>
> On Mon, Jan 9, 2012 at 4:15 PM, Chris Rebert  wrote:
>
>> On Mon, Jan 9, 2012 at 3:30 PM, david.gar...@gmail.com
>>  wrote:
>> > Chris,
>> >
>> > Both a list and dict are both iterable.  I get a python dictionary
>> object of
>> > both iterables.;)
>>
>> No, you get a Python object with both iterables as instance variables.
>> Instance variables happen to be stored using a dict (which is
>> accessible as .__dict__), but that's to some extent an implementation
>> detail whose relevance here I fail to see. My point was that, as Ian
>> explained, your __iter__() method, as written, is horribly broken.
>>
>> > It is nice... but I don't know if this is good form?
>>
>> I'm confused as to why you wrote a class (particularly when its name
>> is an *action* rather than a noun; big red flag right there!) for your
>> task in the first place. I think you'd be best served by moving your
>> parsing code into a function and using a
>> http://pypi.python.org/pypi/sorteddict , which provides the
>> sorted-keys property you seemed to be trying to accomplish.
>>
>> Also, please avoid top-posting in the future. (See
>> http://en.wikipedia.org/wiki/Posting_style )
>>
>> Cheers,
>> Chris
>> --
>> http://rebertia.com
>>
>
>
>
> --
> David Garvey
>



-- 
David Garvey
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: classes and __iter__

[david.gar...@gmail.com]

*Here is a good tutorial:

http://shutupandship.com/articles/iterators/index.html
*
On Mon, Jan 9, 2012 at 5:22 PM, david.gar...@gmail.com <
david.gar...@gmail.com> wrote:

> I see your meaning for __iter__ method.;)
>
>
> On Mon, Jan 9, 2012 at 4:57 PM, david.gar...@gmail.com <
> david.gar...@gmail.com> wrote:
>
>> Thanks Ian & Chris for the conversation...
>>
>>
>>
>>
>> On Mon, Jan 9, 2012 at 4:15 PM, Chris Rebert  wrote:
>>
>>> On Mon, Jan 9, 2012 at 3:30 PM, david.gar...@gmail.com
>>>  wrote:
>>> > Chris,
>>> >
>>> > Both a list and dict are both iterable.  I get a python dictionary
>>> object of
>>> > both iterables.;)
>>>
>>> No, you get a Python object with both iterables as instance variables.
>>> Instance variables happen to be stored using a dict (which is
>>> accessible as .__dict__), but that's to some extent an implementation
>>> detail whose relevance here I fail to see. My point was that, as Ian
>>> explained, your __iter__() method, as written, is horribly broken.
>>>
>>> > It is nice... but I don't know if this is good form?
>>>
>>> I'm confused as to why you wrote a class (particularly when its name
>>> is an *action* rather than a noun; big red flag right there!) for your
>>> task in the first place. I think you'd be best served by moving your
>>> parsing code into a function and using a
>>> http://pypi.python.org/pypi/sorteddict , which provides the
>>> sorted-keys property you seemed to be trying to accomplish.
>>>
>>> Also, please avoid top-posting in the future. (See
>>> http://en.wikipedia.org/wiki/Posting_style )
>>>
>>> Cheers,
>>> Chris
>>> --
>>> http://rebertia.com
>>>
>>
>>
>>
>> --
>> David Garvey
>>
>
>
>
> --
> David Garvey
>



-- 
David Garvey
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: classes and __iter__

[david.gar...@gmail.com]

Many thanks mrabarnett for the code critic...



On Mon, Jan 9, 2012 at 5:08 PM, MRAB  wrote:

> On 09/01/2012 22:51, david.gar...@gmail.com wrote:
>
>> Hello,
>>
>> I have a class and i return both a key list and dictionary from the
>> class. Is it good form to do this? The print helo.__dict__ shows both
>> the list and dictionary.
>>
>>
>>
>>  >>> class Parse_Nagios_Header:
>> ... def __init__(self):
>> ... self.keylist = []
>> ... self.d = {}
>> ... f = open("common.h")
>> ... lines = f.readlines()
>> ... lines = filter(lambda x: not x.isspace(), lines)
>> ... f.close()
>> ... for line in lines:
>> ... if re.search("CMD", line):
>>
>
> You don't need to use a regex for this. It's better to do this instead:
>
>if "CMD" in line:
>
>
>  ... line = line.lstrip('#define ')
>>
>
> The .lstrip, .rstrip and .strip methods treat the argument as a _set_
> of characters, so that line will strip the characters '#', 'd', 'e',
> 'f, 'i', 'n' and ' ' from the left end of the line however many times
> they occur.
>
>
>  ... line = line.strip()
>> ... line.replace('\t', ' ')
>> ... line = line.split()
>>
>
> The .split method, when called without an argument (or an argument of
> None), will split on a sequence of whitespace characters, but ignore
> leading and trailing whitespace.
>
> Therefore, there is no need to use the .strip method or the .replace
> (which will have no effect anyway because it _returns_ its result,
> which is then discarded) before the split.
>
>
>  ... if len(line) > 2:
>> ... pass
>>
>
> The preceding two lines are pointless. Just turn the next line into an
> 'if'; it'll have the same effect.
>
>
>  ... elif len(line) == 2:
>> ... line[1] = int(line[1])
>> ... self.d[line[1]] = line[0]
>> ... self.keylist = sorted(self.d.iterkeys())
>> ... def __iter__(self):
>> ... return iter(self.keylist, self.d)
>> ...
>>
> [snip]
> --
> http://mail.python.org/**mailman/listinfo/python-list<http://mail.python.org/mailman/listinfo/python-list>
>



-- 
David Garvey
-- 
http://mail.python.org/mailman/listinfo/python-list