Help me with a bytes decoding problem in python 3

[J]

I have a bit of a problem I hope you could help me sort out with some
code I'm porting from 2.x to 3.

I have a program with a wrapper for Popen that is called to run
certain linux shell commands and gather system info.  Essentially the
code looks something like this:


process = 
subprocess.Popen(command_str,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
result = process.communicate()

stdout,stderr = result

message = ['Output:\n'
  '- returncode:\n{0}'.format(self.process.returncode)]
if stdout:
   if type(stdout) is bytes:
   stdout = stdout.decode()
   message.append('- stdout:\n{0}'.format(stdout))
if stderr:
   if type(stderr) is bytes:
   stderr = stderr.decode()
   message.append('- stderr:\n{0}'.format(stderr))
logging.debug('\n'.join(message))

So what essentially happens is that command_str is passed to this
function and is a string like this:

'lspci | grep Network'
or
'xinput list'

using the first command string, I get output like this:

In [12]: command = 'lspci |grep Network'
In [13]: process =
subprocess.Popen(command,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
In [14]: result = process.communicate()
In [15]: stdout,stderr = result
In [16]: result
Out[16]:
(b'00:19.0 Ethernet controller: Intel Corporation 82577LC Gigabit
Network Connection (rev 06)\n07:00.0 Network controller: Intel
Corporation Ultimate N WiFi Link 5300\n',
 b'')

Which is a bytes object that is very easily converted to text by the
decode() call prior to sending "message" to the logger.

However, the program seems to now be hanging up when running the
second command_str because it's actually returning bytecode inside the
bytes object:

In [18]: command = 'xinput list'
In [19]: process =
subprocess.Popen(command,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
In [20]: result = process.communicate()
In [21]: stdout,stderr = result
In [22]: result
Out[22]:
(b'\xe2\x8e\xa1 Virtual core pointer
\tid=2\t[master pointer  (3)]\n\xe2\x8e\x9c   \xe2\x86\xb3 Virtual
core XTEST pointer  \tid=4\t[slave  pointer
(2)]\n\xe2\x8e\x9c   \xe2\x86\xb3 Logitech Unifying Device. Wireless
PID:1017\tid=9\t[slave  pointer  (2)]\n\xe2\x8e\x9c   \xe2\x86\xb3
SynPS/2 Synaptics TouchPad  \tid=12\t[slave  pointer
(2)]\n\xe2\x8e\x9c   \xe2\x86\xb3 MCE IR Keyboard/Mouse (ite-cir)
  \tid=14\t[slave  pointer  (2)]\n\xe2\x8e\xa3 Virtual core keyboard
\tid=3\t[master keyboard (2)]\n\xe2\x86\xb3
Virtual core XTEST keyboard \tid=5\t[slave  keyboard
(3)]\n\xe2\x86\xb3 Power Button
\tid=6\t[slave  keyboard (3)]\n\xe2\x86\xb3 Video Bus
\tid=7\t[slave  keyboard (3)]\n\xe2\x86\xb3
Laptop_Integrated_Webcam_2M \tid=8\t[slave  keyboard
(3)]\n\xe2\x86\xb3 HID 413c:8157
\tid=10\t[slave  keyboard (3)]\n\xe2\x86\xb3 AT Translated Set 2
keyboard\tid=11\t[slave  keyboard (3)]\n\xe2\x86\xb3
Dell WMI hotkeys\tid=13\t[slave  keyboard
(3)]\n\xe2\x86\xb3 ITE8708 CIR transceiver
\tid=15\t[slave  keyboard (3)]\n',
 b'')

Unfortunately, what's happening here is all that extra formatting
stuff that xinput injects is also converted when doing bytes.decode()
on result[0].


In [24]: result[0].decode()
Out[24]: '⎡ Virtual core pointer\tid=2\t[master
pointer  (3)]\n⎜   ↳ Virtual core XTEST pointer
\tid=4\t[slave  pointer  (2)]\n⎜   ↳ Logitech Unifying Device.
Wireless PID:1017\tid=9\t[slave  pointer  (2)]\n⎜   ↳ SynPS/2
Synaptics TouchPad  \tid=12\t[slave  pointer  (2)]\n⎜   ↳
MCE IR Keyboard/Mouse (ite-cir) \tid=14\t[slave  pointer
(2)]\n⎣ Virtual core keyboard   \tid=3\t[master
keyboard (2)]\n↳ Virtual core XTEST keyboard
\tid=5\t[slave  keyboard (3)]\n↳ Power Button
 \tid=6\t[slave  keyboard (3)]\n↳ Video Bus
   \tid=7\t[slave  keyboard (3)]\n↳
Laptop_Integrated_Webcam_2M \tid=8\t[slave  keyboard
(3)]\n↳ HID 413c:8157   \tid=10\t[slave
keyboard (3)]\n↳ AT Translated Set 2 keyboard
\tid=11\t[slave  keyboard (3)]\n↳ Dell WMI hotkeys
  \tid=13\t[slave  keyboard (3)]\n↳ ITE8708 CIR transceiver
 \tid=15\t[slave  keyboard (3)]\n'

And so I believe that when that decoded string is being pushed to the
logger, the logger is choking, causing the program to just hang
waiting on something... not sure which.

Though when doing this manually via ipython3, logger seems to handle
it just fine:

In [37]: message = ['Output:\n'
'- returncode:\n{0}'.format(process.returncode)]

In [38]: message.append('- stdout:\n{0}'.format(stdout.decode()))

In [39]: logging.debug('\n'.join(message))
DEBUG:root:Output:
- returncode:
0
- stdout:
⎡ Virtual core pointer  id=2[master pointer  (3)]
⎜   ↳ Virtual core XTEST pointerid=4[slave  pointer  (2)]
⎜   ↳ Logitech Unifying Device. Wireless PID:1017   id=9   

Re: Help me with a bytes decoding problem in python 3

[J]

On Thu, Jun 21, 2012 at 5:54 PM, Chris Angelico  wrote:
> On Fri, Jun 22, 2012 at 7:47 AM, J  wrote:
>> \xe2\x86\xb3
>
> This is the UTF-8 encoded form of U+21B3, which is the DOWNWARDS ARROW
> WITH TIP RIGHTWARDS character that you're showing. That's what the
> "bytecode" you're seeing is. You may be able to ask the underlying
> program to make its output in a cleaner format - try looking for a
> --porcelain option or similar - but as I see it, everything in Python
> is doing what it ought to do.
>
> Or am I misunderstanding your question?
>
> ChrisA
> --
> http://mail.python.org/mailman/listinfo/python-list

No, that's exactly what was choking my python program...

And you hit the nail on the head.  Apparently, if I run xinput using
default output (what we were doing originally) you get those
characters.  If you run it like so:

xinput list --name-only

you get a clean, text only list of only the names of the various input
devices found.  So doing that, after an afternoon of pulling my hair
out over this mess, makes the program work as it's supposed to.

What confuses me though, is that the program calls that function
twice, and this works fine the first time around, but fails the
second.  The function is called before rebooting the machine, then is
called again after rebooting the machine and compares the output of
various system info commands to make sure no hardware disappeared
after a reboot.

This was not an issue in Python 2.x, it's only on the Python 3
conversion this occurred. :/ In fact, if I change the shebang from
python3 back to python2.7, it works beautifully.

In the log file, this appears the first time that the xinput command is run:
2012-06-21 18:02:43,043 INFO Gathering hardware information...
2012-06-21 18:02:43,043 DEBUGExecuting: 'xinput list'...
2012-06-21 18:02:43,052 DEBUGOutput:
- returncode:
0
- stdout:
⎡ Virtual core pointer  id=2[master pointer  (3)]
⎜   ↳ Virtual core XTEST pointerid=4[slave  pointer  (2)]
⎜   ↳ Logitech Unifying Device. Wireless PID:1017   id=9[slave
 pointer  (2)]
⎜   ↳ SynPS/2 Synaptics TouchPadid=12   [slave  pointer  (2)]
⎜   ↳ MCE IR Keyboard/Mouse (ite-cir)   id=14   [slave  pointer  (2)]
⎣ Virtual core keyboard id=3[master keyboard (2)]
↳ Virtual core XTEST keyboard   id=5[slave  keyboard (3)]
↳ Power Button  id=6[slave  keyboard (3)]
↳ Video Bus id=7[slave  keyboard (3)]
↳ Laptop_Integrated_Webcam_2M   id=8[slave  keyboard (3)]
↳ HID 413c:8157 id=10   [slave  keyboard (3)]
↳ AT Translated Set 2 keyboard  id=11   [slave  keyboard (3)]
↳ Dell WMI hotkeys  id=13   [slave  keyboard (3)]
↳ ITE8708 CIR transceiver   id=15   [slave  keyboard (3)]

2012-06-21 18:02:43,053 DEBUGTouchpad and Keyboard:
⎡ Virtual core pointer  id=2[master pointer  (3)]
⎜   ↳ Virtual core XTEST pointerid=4[slave  pointer  (2)]
⎜   ↳ Logitech Unifying Device. Wireless PID:1017   id=9[slave
 pointer  (2)]
⎜   ↳ SynPS/2 Synaptics TouchPadid=12   [slave  pointer  (2)]
⎜   ↳ MCE IR Keyboard/Mouse (ite-cir)   id=14   [slave  pointer  (2)]
⎣ Virtual core keyboard id=3[master keyboard (2)]
↳ Virtual core XTEST keyboard   id=5[slave  keyboard (3)]
↳ Power Button  id=6[slave  keyboard (3)]
↳ Video Bus id=7[slave  keyboard (3)]
↳ Laptop_Integrated_Webcam_2M   id=8[slave  keyboard (3)]
↳ HID 413c:8157 id=10   [slave  keyboard (3)]
↳ AT Translated Set 2 keyboard  id=11   [slave  keyboard (3)]
↳ Dell WMI hotkeys  id=13   [slave  keyboard (3)]
↳ ITE8708 CIR transceiver   id=15   [slave  keyboard (3)]

(it appears twice because I ran it in debug mode)

but after the reboot, it chokes:

2012-06-21 18:03:56,396 INFO reboot operations remaining: 0
2012-06-21 18:03:56,396 INFO Reboot time: 0:01:13.396280
2012-06-21 18:04:27,201 INFO Gathering hardware information...
2012-06-21 18:04:27,201 DEBUGExecuting: 'xinput list'...
2012-06-21 18:07:22,985 DEBUGRemoving desktop file
('/home/bladernr/.config/autostart/pm_test.desktop')...
2012-06-21 18:07:22,986 DEBUGRestoring sudoers configuration...
2012-06-21 18:07:22,986 DEBUGExecuting: "sed -i -e '/#
Automatically added by pm.py/,+1d' /etc/sudoers"...
2012-06-21 18:07:22,993 DEBUGRestoring autologin configuration...
2012-06-21 18:07:23,013 INF

Frustrating circular bytes issue

[J]

This is driving me batty... more enjoyment with the Python3
"Everything must be bytes" thing... sigh...
I have a file that contains a class used by other scripts.  The class
is fed either a file, or a stream of output from another command, then
interprets that output and returns a set that the main program can
use...  confusing, perhaps, but not necessarily important.

The class is created and then called with the load_filename method:


 def load_filename(self, filename):
logging.info("Loading elements from filename: %s", filename)

file = open(filename, "rb", encoding="utf-8")
return self.load_file(file, filename)

As you can see, this calls the load_file method, by passing the
filehandle and filename (in common use, filename is actually an
IOStream object).

load_file starts out like this:


def load_file(self, file, filename=""):
elements = []
for string in self._reader(file):
if not string:
break

element = {}


Note that it now calls the private _reader() passing along the
filehandle further in.  THIS is where I'm failing:

This is the private _reader function:


def _reader(self, file, size=4096, delimiter=r"\n{2,}"):
buffer_old = ""
while True:
buffer_new = file.read()
print(type(buffer_new))
if not buffer_new:
break
lines = re.split(delimiter, buffer_old + buffer_new)
buffer_old = lines.pop(-1)

for line in lines:
yield line

yield buffer_old


(the print statement is something I put in to verify the problem.

So stepping through this, when _reader executes, it executes read() on
the opened filehandle.  Originally, it read in 4096 byte chunks, I
removed that to test a theory.  It creates buffer_new with the output
of the read.

Running type() on buffer_new tells me that it's a bytes object.

However no matter what I do:

file.read().decode()
buffer_new.decode() in the lines = re.split() statement
buffer_str = buffer_new.decode()

I always get a traceback telling me that the str object has no decoe() method.

If I remove the decode attempts, I get a traceback telling me that it
can't implicitly convert a bytes_object to a str object.

So I'm stuck in a vicious circle and can't see a way out.

here's sample error messages:
When using the decode() method to attempt to convert the bytes object:
Traceback (most recent call last):
  File "./filter_templates", line 134, in 
sys.exit(main(sys.argv[1:]))
  File "./filter_templates", line 126, in main
options.whitelist, options.blacklist)
  File "./filter_templates", line 77, in parse_file
matches = match_elements(template.load_file(file), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/checkbox/lib/template.py", line
73, in load_file
for string in self._reader(file):
  File "/usr/lib/python3/dist-packages/checkbox/lib/template.py", line
35, in _reader
lines = re.split(delimiter, buffer_old + buffer_new.decode())
AttributeError: 'str' object has no attribute 'decode'

It's telling me that buffer_new is a str object.

so if I remove the decode():

Traceback (most recent call last):
  File "./run_templates", line 142, in 
sys.exit(main(sys.argv[1:]))
  File "./run_templates", line 137, in main
runner.process(args, options.shell)
  File "./run_templates", line 39, in process
records = self.process_output(process.stdout)
  File "./run_templates", line 88, in process_output
return template.load_file(output)
  File "/usr/lib/python3/dist-packages/checkbox/lib/template.py", line
73, in load_file
for string in self._reader(file):
  File "/usr/lib/python3/dist-packages/checkbox/lib/template.py", line
35, in _reader
lines = re.split(delimiter, buffer_old + buffer_new)
TypeError: Can't convert 'bytes' object to str implicitly

now it's complaining that buffer_new is a bytes object and can't be
implicitly converted to str.

This is a bug introduced in our conversion from Python 2 to Python 3.
I am really, really starting to dislike some of the things Python3
does... or just am really, really frustrated.
-- 
http://mail.python.org/mailman/listinfo/python-list

Flushing buffer on file copy on linux

[J]

I was hoping someone could give me some ideas for a particular problem.

I've got a python program that is used for basic testing of removable
storage devices (usb, mmc, firewire, etc).

Essentially, it looks for a mounted device (either user specified, or
if not, the program loops through all removable disks), generates a
file of random data of a predetermined size, does a md5sum of the
parent, uses shutil copy() to copy the file to the device, then does a
md5sum of the copy on removable media, compares, cleans up and Bob's
your uncle.

Now, I'm working on enhancements and one enhancement is a threaded
"stress test" that will perform the exact same operations, but in
individual threads.

So if you the script to do 10 iterations using 10MB files, instead of
doing 10 loops, it does 10 concurrent threads (I'm new to doing
threads so this is a learning experience there as well).

Now, the problem I have is that linux tends to buffer data writes to a
device, and I want to work around that.  When run in normal non-stress
mode, the program is slow enough that the linux buffers flush and put
the file on disk before the hash occurs.  However, when run in stress
mode, what I'm finding is that it appears that the files are possibly
being hashed while still in the buffer, before being flushed to disk.

So ultimately, I wanted to see if there were some ideas for working
around this issue.

One idea I had was to do the following:

Generate the parent data file
hash parent
instead of copy, open parent and write to a new file object on disk
with a 0 size buffer
or flush() before close()
hash the copy.

Does that seem reasonable? or is there a cleaner way to copy a file
from one place to another and ensure the buffers are properly flushed
(maybe something in os or sys that forces file buffers to be flushed?)

Anyway, I'd appreciate any suggestions with that.  I'll try
implementing the idea mentioned above tomorrow, but if there's a
cleaner way I'd be interested in learning it.

Cheers,

jeff
-- 
http://mail.python.org/mailman/listinfo/python-list

Is there a way to create kernel log messages via Python?

[J]

Hi...

I have a bit of code that does the following:

uses the syslog module to inject a LOG_INFO message into the syslog on
my linux machine
runs a suspend/resume cycle
uses the syslog module to inkect a LOG_INFO message marking the end of test.

Then I parse everything between the start and stop markers for certain
items that the Linux kernel logs during a suspend and resume cycle.

But my  "resume complete" timing is not as accurate as it could be.
The problem with doing it this way is that while I can find definite
kernel messages that mark various points in the suspend/resume cycle,
the final message when the kernel is done resuming is NOT the point I
actually want to mark.

Instead, the end point I want is the time of the ending marker itself,
as this happens after certain other things are done such as resumption
of networking services.

Here's the problem.  I can't just use syslog timestamps.  The reason
is that the syslog timestamps are only indicative of when messages are
written to syslog via syslogd.  The kernel timestamps are different.
For example, the following bits of log are taken from the time the
test starts until the end of the "going to sleep" kernel messages.
First, note that there's a 5 second difference between the START
marker and the first kernel message.  Next, look at the kernel
timestamps.  The total real time to suspend starts at 421320.380947
and ends at 421322.386355, around 2 seconds later, where the log
messages themselves all state that the events occurred at the same
time.

Oct 15 10:24:19 klaatu sleep_test: ---SLEEP TEST START 1350296656---
Oct 15 10:25:24 klaatu kernel: [421320.380947] PM: Syncing filesystems ... done.
Oct 15 10:25:24 klaatu kernel: [421320.391282] PM: Preparing system
for mem sleep
[SNIP]
Oct 15 10:25:24 klaatu kernel: [421322.282943] Broke affinity for irq 23
Oct 15 10:25:24 klaatu kernel: [421322.386355] CPU 7 is now offline

So, what I REALLY want is to inject my start/stop markers into klogd
rather than syslogd.  This will, I hope, give my markers kernel
timestamps rather than syslog timestamps which are not as accurate.

So does anyone know of a way to do this?  Unfortunately, I've tried
some searching but google doesn't like the term klog, and most of the
hits involved injecting code or other things that are not related at
all.

Or, if there's a better way to get accurate timestamps, what would that be?
-- 
http://mail.python.org/mailman/listinfo/python-list

Is there a more elegant way to handle determing fail status?

[J]

Ok, so I have a diagnostic tool, written by someone else. That tool
runs a series of small tests defined by the user and can simplified
summary output that can be one of the following:

FAILED_CRITICAL
FAILED_HIGH
FAILED_MEDIUM
FAILED_LOW
PASSED

I also have a wrapper script I wrote to run these tests, summarize the
results of all tests aggregated and then fail based on a particular
fail level.

The idea is that if I run 3 tests with the diagnostic tool and it
tells me the following:

testA: PASSED
testB: FAILED_MEDIUM
testC: PASSED

AND I told the wrapper to only fail on HIGH or above, the wrapper will
tell me that I had a medium failure, but the wrapper will still exit
with a 0 (success)

if I get the same results as above, but tell the wrapper to fail on
LOW, then it will tell me I had that medium failure, but the wrapper
will exit with a 1 (failure).

The problem is that my exit determination looks like this:

if fail_priority == fail_levels['FAILED_CRITICAL']:
if critical_fails:
return 1
if fail_priority == fail_levels['FAILED_HIGH']:
if critical_fails or high_fails:
return 1
if fail_priority == fail_levels['FAILED_MEDIUM']:
if critical_fails or high_fails or medium_fails:
return 1
if fail_priority == fail_levels['FAILED_LOW']:
if critical_fails or high_fails or medium_fails or low_fails:
return 1

return 0

So, to explain the above... the fail level can be set by the user when
running the wrapper using -f (or it defaults to 'high')
the wrapper assigns a number to each level using this:

# Set correct fail level
args.fail_level = 'FAILED_%s' % args.fail_level.upper()

# Get our failure priority and create the priority values
fail_levels = {'FAILED_CRITICAL':4,
   'FAILED_HIGH':3,
   'FAILED_MEDIUM':2,
   'FAILED_LOW':1}
fail_priority = fail_levels[args.fail_level]

the variables critical_fails, high_fails, medium_fails, low_fails are
all counters that are etiher None, or the number of tests that were
failed.

So using this output from the diagnostic tool:

testA: PASSED
testB: FAILED_HIGH
testC: PASSED
testD: FAILED_MEDIUM
testE: PASSED

critical_fails would be None
high_fails would be 1
medium_fails would be 1
low_fails would be None.

The exit code determination above works, but it just feels inelegant.
It feels like there's a better way of implementing that, but I can't
come up with one that still honors the fail level properly (e.g. other
solutions will fail on medium, but won't fail properly on medium OR
higher).

I can provide the full script if necessary, if the above isn't enough
to point me in a direction that has a better way of doing this...

Thanks for looking,

Jeff
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The devolution of English language and slothful c.l.p behaviors exposed!

[J]

On Tue, Jan 24, 2012 at 09:05, Martin P. Hellwig
 wrote:
> On 24/01/2012 05:57, Rick Johnson wrote:
> 
> I would wish that pedantic citizens of the British colony in America stopped
> calling whatever misinterpreted waffle they produce, English.

I, sir, as a citizen of that FORMER British colony here on the
continent of North America, am offended by this baseless insult. I
know waffles, sir, I eat waffles and I can guarantee that no American
calls their waffles "English".  We have English Muffins and Belgian
Waffles, but no English Waffles.  Though I am particularly fond of
Blueberry Waffles, myself.

Which reminds me, time for breakfast.

Cheers,

Jeff
-- 
http://mail.python.org/mailman/listinfo/python-list

Help me with weird logging problem

[J]

Hi,

I'm trying to add a couple log handlers to a program.  The end goal is
to log things at INFO or above to console, and if a -v option is set
to ALSO log everything at DEBUG or above to a file.  However, while I
DO get the log file created, and log messages are appearing there,
Debug messages are not...

Here's some sample code that is essentially the guts of the logging
setup right now, based on what I read in the docs:


#!/usr/bin/python

import logging
import sys

verbose = True

# Set up the logger
console_format = '%(levelname)-8s %(message)s'
console_handler = logging.StreamHandler()
console_handler.setFormatter(logging.Formatter(console_format))
console_handler.setLevel(logging.INFO)
logger = logging.getLogger()
logger.addHandler(console_handler)
if verbose:
verbose_format = '%(asctime)s %(levelname)-8s %(message)s'
verbose_handler = logging.FileHandler('testlog.log')
verbose_handler.setLevel(logging.DEBUG)
verbose_handler.setFormatter(logging.Formatter(verbose_format))
logger.addHandler(verbose_handler)
logging.debug("Setting log level to DEBUG for verbosity")

logging.info("INFO event logged")
logging.warning("WARNING event logged")
logging.error("ERROR event logged")
logging.critical("CRITICAL event logged")
logging.debug("DEBUG event logged")

When I run this I get the following console and log file output:

bladernr@klaatu:~/development/cert-submit-script-improvements/bin$ ./logtest.py
WARNING  WARNING event logged
ERRORERROR event logged
CRITICAL CRITICAL event logged
bladernr@klaatu:~/development/cert-submit-script-improvements/bin$ cat
testlog.log
2012-03-06 11:05:40,297 WARNING  WARNING event logged
2012-03-06 11:05:40,297 ERRORERROR event logged
2012-03-06 11:05:40,298 CRITICAL CRITICAL event logged

So I AM getting logging, but I am not getting the DEBUG level logs in
the log file, nor am I getting INFO level logs on console.

Any idea what I'm doing wrong?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Help me with weird logging problem

[J]

On Tue, Mar 6, 2012 at 11:19, Vinay Sajip  wrote:
> On Mar 6, 4:09 pm, J  wrote:
>>
>> Any idea what I'm doing wrong?
>
> Levels can be set on loggers as well as handlers, and you're only
> setting levels on the handlers. The default level on the root logger
> is WARNING. A logger checks its level first, and only if the event
> passes that test will it be passed to the handlers (which will also
> perform level tests).
>
> So, a logger.setLevel(logging.DEBUG) should be all you need to add
> before logging anything.
>
> Regards,
>
> Vinay Sajip
> --
> http://mail.python.org/mailman/listinfo/python-list

Thank you very much, Vinay :)

I thought it was something simple like that I had overlooked or misunderstood.

Cheers,

Jeff
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: tee-like behavior in Python

[J]

On Wed, May 9, 2012 at 11:35 AM, Florian Lindner  wrote:
> Hello,
>
> how can I achieve a behavior like tee in Python?
>
> * execute an application
> * leave the output to stdout and stderr untouched
> * but capture both and save it to a file (resp. file-like object)
>
> I have this code
>
> proc = subprocess.Popen(shlex.split(cmd), stdout = subprocess.PIPE,
> stderr=subprocess.STDOUT)
> while True:
>    out = proc.stdout.readline()
>    if out == '' and proc.poll() != None:
>       break
>    sys.stdout.write(out)
>    logfile.write(out)
>
> This works so far but always buffers a couple of lines and outputs
> them en bloc. The final output is like it is desired but with a
> significant delay. Is there a way around that?

Perhaps this would help:

http://docs.python.org/library/subprocess.html#popen-constructor

specifically, the bits about setting bufsize to 0 indicating
unbuffered behaviour.
-- 
http://mail.python.org/mailman/listinfo/python-list

How to iterate through two dicts efficiently

[J]

Hello,

I am looking to improve the performance of the following piece of Python code:-

for cc in StatusContainer:
for srv in StatusContainer[cc]:
for id in StatusContainer[cc][srv]['RECV']:
if id in StageContainer['13']:
StatusContainer[cc][srv].setdefault('ACK', 
[]).append(id)
elif id in StageContainer['14']:
StatusContainer[cc][srv].setdefault('NACK', 
[]).append(id)
else:
if id in StageContainer['504'] or id in 
StageContainer['505']:

StatusContainer[cc][srv].setdefault('RETRY', []).append(id)

I am new to programming and I decided that I would use python as my first 
programming language.  I have written couple of (very basic but useful) scripts 
and now I am looking to optimise the performance of some of these scripts so I 
can make use of them.

The code above is an extract from a bigger script which I am using to count how 
many message IDs have been received, sent or rejected per service per country.  
The code is iterating through two dictionaries (a status dictionary which is a 
set of sub-dictionaries and a stages dictionary which again is a set of 
sub-dictionaries), if an entry (in this case an id) is found in the status 
dictionary, I am performing a search for that id in the stages dictionary.  
Depending where in the stages dictionary the id is found (which stage), then I 
am creating a new sub-dictionary inside the status dictionary and I append the 
id.

At the end, all I do is count how many IDs each sub-dictionary is holding (e.g: 
len(StatusContainer[cc][srv]['RECV'])).

My two dictionaries look like this:-

StatusContainer:-

defaultdict(, {'FR': {'VDMS': {'RECV': 
['cad1c651-a61e-11e0-9f43-00238bbdc9eb', 
'df25a1d1-a61e-11e0-9f43-00238bbdc9eb', '1a9452c0-a61f-11e0-9f43-00238bbdc9eb', 
'1a9b3090-a61f-11e0-9f43-00238bbdc9eb']}}, 'PT': {'VDMS': {'RECV': 
['2410e980-a61f-11e0-cffc-00238bbdc9e7', 
'24146bf0-a61f-11e0-cffc-00238bbdc9e7', '29c5f0f0-a61f-11e0-cffc-00238bbdc9e7', 
'2ebe5f20-a61f-11e0-cffc-00238bbdc9e7']}}, 'CL': {'VDMS': {'RECV': 
['379cfda0-a61e-11e0-9bd6-00238bbdc9e7', 
'ff60f990-a61e-11e0-9bd6-00238bbdc9e7', '1a7e80d0-a61f-11e0-9bd6-00238bbdc9e7', 
'7ab64f00-a61f-11e0-9bd6-00238bbdc9e7']}}})

StageContainer:-

defaultdict(, {'13': ['17897931-a61e-11e0-a764-00238bce423b', 
'd0fcb681-a61d-11e0-e693-00238bbdc9eb', '178819a1-a61e-11e0-a764-00238bce423b', 
'17161df1-a61e-11e0-b558-00238bbdc9e7', '15eb5991-a61e-11e0-8aeb-00238bbdc9b7', 
'1872eed0-a61e-11e0-de66-00238bce424c', 
'b3e470b1-a61d-11e0-b558-00238bbdc9e7'], '14': 
['1a98dc11-a61e-11e0-a9e7-00238bce423b', 
'da485231-a61d-11e0-ff8f-00238bce423b', '1c245e11-a61e-11e0-8aeb-00238bbdc9b7', 
'1eab5711-a61e-11e0-da5a-00238bce423b', '8db533c1-a61d-11e0-9393-00238bbdc9b7', 
'e0e066a1-a61d-11e0-b558-00238bbdc9e7']})

I ran this piece of code through the python profiler and I see that the code is 
taking on average around 1300 CPU secs to execute, depending on the size of the 
dicts.  The problem is that both dictionaries will always hold hundreds of 
thousands of entries, so I need to re-write this code to be more efficient or I 
need to find a different way to achieve my goal.

Someone in a different forum suggested that I use 'binary search' to iterate 
through the dictionaries but like I mentioned before, I am still very new to 
programming and 'binary search' sounds a bit to complicated to me.  I was 
wondering if someone here can have a look at my code above and maybe re-write 
it using 'binary search' or any other way you may see fit and re-post the code 
here with some comments on what each section is doing?

Your help would be very much appreciated.

Regards,

Junior
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to iterate through two dicts efficiently

[J]

Hi guys,

Thank you for your suggestions.  I have managed to get my whole script to 
execute in under 10 seconds by changing the 'for loop' I posted above to the 
following:-

for opco in Cn:
for service in Cn[opco]:
ack = set(Cn[opco][service]['RECV']) & set(Pr['13'])
for jarid in ack:
Cn[opco][service].setdefault('ACK', set()).add(jarid)
nack = set(Cn[opco][service]['RECV']) & set(Pr['14'])
for jarid in nack:
Cn[opco][service].setdefault('NACK', set()).add(jarid)
retry = set(Cn[opco][service]['RECV']) & set(Pr['504'])
for jarid in retry:
Cn[opco][service].setdefault('RETRY', set()).add(jarid)
retry1 = set(Cn[opco][service]['RECV']) & set(Pr['505'])
for jarid in retry1:
Cn[opco][service].setdefault('RETRY', set()).add(jarid)

The script is by no means complete and I will be looking at your suggestions 
more closely to adapt my script accordingly based on your suggestions.

But for now I am happy to have dramatically speed up the execution of my script 
from over 20 mins to under 10 secs!

Thanks again for your time.
-- 
http://mail.python.org/mailman/listinfo/python-list

UnicodeEncodeError -- 'character maps to '

[J]

Hi there,

I'm attempting to print a dictionary entry of some twitter data to screen but 
every now and then I get the following error:

(, UnicodeEncodeError('charmap', u'RT 
@ciaraluvsjb26: BIEBER FEVER \u2665', 32, 33, 'character maps to '), 
)

I have googled this but haven't really found any way to overcome the error. Any 
ideas?

J
-- 
http://mail.python.org/mailman/listinfo/python-list

Convert AWK regex to Python

[J]

Good morning all,
Wondering if you could please help me with the following query:-
I have just started learning Python last weekend after a colleague of mine 
showed me how to dramatically cut the time a Bash script takes to execute by 
re-writing it in Python.  I was amazed at how fast it ran.  I would now like to 
do the same thing with another script I have.

This other script reads a log file and using AWK it filters certain fields from 
the log and writes them to a new file.  See below the regex the script is 
executing.  I would like to re-write this regex in Python as my script is 
currently taking about 1 hour to execute on a log file with about 100,000 
lines.  I would like to cut this time down as much as possible.

cat logs/pdu_log_fe.log | awk -F\- '{print $1,$NF}' | awk -F\. '{print $1,$NF}' 
| awk '{print $1,$4,$5}' | sort | uniq | while read service command status; do 
echo "Service: $service, Command: $command, Status: $status, Occurrences: `grep 
$service logs/pdu_log_fe.log | grep $command | grep $status | wc -l | awk '{ 
print $1 }'`" >> logs/pdu_log_fe_clean.log; done

This AWK command gets lines which look like this:-

2011-05-16 09:46:22,361 [Thread-4847133] PDU D 

And outputs lines like this:-

CC_SMS_SERVICE_51408 submit_resp: 0

I have tried writing the Python script myself but I am getting stuck writing 
the regex.  So far I have the following:-

#!/usr/bin/python

# Import RegEx module
import re as regex
# Log file to work on
filetoread = open('/tmp/ pdu_log.log', "r")
# File to write output to
filetowrite =  file('/tmp/ pdu_log_clean.log', "w")
# Perform filtering in the log file
linetoread = filetoread.readlines()
for line in linetoread:
filter0 = regex.sub(r"http://mail.python.org/mailman/listinfo/python-list

Re: Convert AWK regex to Python

[J]

Good morning Angelico,
Do I understand correctly? Do you mean incorporating a Python dict inside the 
AWK command? How can I do this?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Convert AWK regex to Python

[J]

Hello Peter, Angelico,

Ok lets see, My aim is to filter out several fields from a log file and write 
them to a new log file.  The current log file, as I mentioned previously, has 
thousands of lines like this:-
2011-05-16 09:46:22,361 [Thread-4847133] PDU D 

All the lines in the log file are similar and they all have the same length 
(same amount of fields).  Most of the fields are separated by spaces except for 
couple of them which I am processing with AWK (removing "http://mail.python.org/mailman/listinfo/python-list

Re: Convert AWK regex to Python

[J]

Thanks for the sugestions Peter, I will give them a try

Peter Otten wrote:
> J wrote:
>
> > Hello Peter, Angelico,
> >
> > Ok lets see, My aim is to filter out several fields from a log file and
> > write them to a new log file.  The current log file, as I mentioned
> > previously, has thousands of lines like this:- 2011-05-16 09:46:22,361
> > [Thread-4847133] PDU D  > CC_SMS_SERVICE_51408_656-ServerThread-
> VASPSessionThread-7ee35fb0-7e87-11e0-a2da-00238bce423b-TRX
> > - 2011-05-16 09:46:22 - OUT - (submit_resp: (pdu: L: 53 ID: 8004
> > Status: 0 SN: 25866) 98053090-7f90-11e0-a2da-00238bce423b (opt: ) ) >
> >
> > All the lines in the log file are similar and they all have the same
> > length (same amount of fields).  Most of the fields are separated by
> > spaces except for couple of them which I am processing with AWK (removing
> > " > evaluate each line in the log file and break them down into fields which I
> > can call individually and write them to a new log file (for example
> > selecting only fields 1, 2 and 3).
> >
> > I hope this is clearer now
>
> Not much :(
>
> It doesn't really matter whether there are 100, 1000, or a million lines in
> the file; the important information is the structure of the file. You may be
> able to get away with a quick and dirty script consisting of just a few
> regular expressions, e. g.
>
> import re
>
> filename = ...
>
> def get_service(line):
> return re.compile(r"[(](\w+)").search(line).group(1)
>
> def get_command(line):
> return re.compile(r"
> def get_status(line):
> return re.compile(r"Status:\s+(\d+)").search(line).group(1)
>
> with open(filename) as infile:
> for line in infile:
> print get_service(line), get_command(line), get_status(line)
>
> but there is no guarantee that there isn't data in your file that breaks the
> implied assumptions. Also, from the shell hackery it looks like your
> ultimate goal seems to be a kind of frequency table which could be built
> along these lines:
>
> freq = {}
> with open(filename) as infile:
> for line in infile:
> service = get_service(line)
> command = get_command(line)
> status = get_status(line)
> key = command, service, status
> freq[key] = freq.get(key, 0) + 1
>
> for key, occurences in sorted(freq.iteritems()):
> print "Service: {}, Command: {}, Status: {}, Occurences: {}".format(*key
> + (occurences,))
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Convert AWK regex to Python

[J]

Hello,

I have managed to get my script finished in the end by taking bits from 
everyone who answered.  Thank you so much.  the finished query string looks 
like this (still not the best but it gets the job done.  Once I learn to code 
more with Python I will probably go back to it and re-write it):-

# Log file to work on
filetoread = open("/tmp/pdu.log", "r")
# Perform filtering in the log file
text = filetoread.read()
text = text.replace("http://mail.python.org/mailman/listinfo/python-list

packaging python

[J]

Hi everyone,

I managed to package python for distribution with my app. I
made 'dist' using py2exe and everything runs fine except there
is no more output from the python interpreter. I redirect the
output like so...


static PyMethodDef ioMethods[] =
{
   {"output", output, METH_VARARGS, "output"},
   {NULL, NULL, 0, NULL}
};

sModule = PyImport_ImportModule ("__main__");
sDictionary = PyModule_GetDict(sModule);

PyObject* lModule = Py_InitModule("gld", ioMethods);
char *code =  "class Sout:\n"
  "def write(self, s):\n"
  "output(s)\n"
  "\n"
  "import sys\n"
  "from code import *\n"
  "from gld import *\n"
  "sys.stdout = Sout()\n"
  "sys.stderr = Sout()\n"
  "sys.stdin  = None\n"
  "print 12\n";

Output never gets called, but everything else works fine. If I
switch back the installed verion of python output works again. Does
anyone have a quick solution ?

Cheers
Jochen

-- 
http://mail.python.org/mailman/listinfo/python-list

reference count

[J]

Hi,


I have a problem with the reference count after an error occurs
in a script which I execute as follows..

PyObject* lDict = PyDict_New();
PyDict_SetItemString(lDict, "item", (PyObject*)iItem->mPyObject);
PyObject* lResult = PyEval_EvalCode(mCode, ScnGlobal::sDictionary,
lDict);
if (! lResult)
{
PyErr_Print();
}
else
{
Py_DECREF(lResult);
}
Py_DECREF(lDict);


The script within the mCode object only contains the following line,
which generates an error.

print aasas


The problem is that after PyEval_EvalCode returns the reference count
for lDict is one larger than before. Py_DECREF(lDict) does
therefore not delete the Dictionary, which causes me problems. Is
Python keeping an exception around. I assume that I have to do some
more cleanup ??


Cheers
Jochen

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: reference count

[J]

I tend to answer my own questions a lot. But he, lets grow the
knowledge base :)... PyErr_Clear() after PyErr_Print did the trick

J

-- 
http://mail.python.org/mailman/listinfo/python-list

VC++ linking problem

[J]

Hi everyone,

I started embedding python into a 3D graphics App and I came
across this linking problem.


SO.lib(ScnGlobal.obj) : error LNK2001: unresolved external symbol
__imp__Py_InitModule4TraceRefs
SO.lib(ScnNode.obj) : error LNK2001: unresolved external symbol
__imp___Py_RefTotal
SO.lib(ScnLight.obj) : error LNK2001: unresolved external symbol
__imp___Py_RefTotal

I am linking against python24.lib using VC++ 6.0. Before I got to this
point it couldn't find python24_d.lib. After searching around
for a while I came across a solution that included changing
python24_d.lib to python24.lib in one of the header files. I hope that
didn't have a negative effect. I would really appreciate
some help


Thanks
Jochen

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: VC++ linking problem

[J]

Thank you very much Miki!

That was an easy solution. It links... I will put a hold on compiling
the latest version until I really have to. I will probably switch to VC
7 before that.

Cheers
Jochen


Miki Tebeka wrote:
> Hello Jochen,
>
> > I started embedding python into a 3D graphics App and I came
> > across this linking problem.
> >
> >
> > SO.lib(ScnGlobal.obj) : error LNK2001: unresolved external symbol
> > __imp__Py_InitModule4TraceRefs
> > SO.lib(ScnNode.obj) : error LNK2001: unresolved external symbol
> > __imp___Py_RefTotal
> > SO.lib(ScnLight.obj) : error LNK2001: unresolved external symbol
> > __imp___Py_RefTotal
> >
> > I am linking against python24.lib using VC++ 6.0. Before I got to this
> > point it couldn't find python24_d.lib. After searching around
> > for a while I came across a solution that included changing
> > python24_d.lib to python24.lib in one of the header files. I hope that
> > didn't have a negative effect. I would really appreciate
> > some help
> This is since Python header file is in debug mode and you link with the
> regular library.
>
> Just do:
> #undef _DEBUG /* Link with python24.lib and not python24_d.lib */
> #include 
>
> and you'll be fine.
>
> IMO python should try and link with python24.lib when compiling in debug
> mode since we want to debug our program and not Python.
>
> Bye.
> --
> 
> Miki Tebeka <[EMAIL PROTECTED]>
> http://tebeka.bizhat.com
> The only difference between children and adults is the price of the toys

-- 
http://mail.python.org/mailman/listinfo/python-list

publishing an already instantiated class

[J]

Hi everyone,

Thanks for your awsome replies. I started replacing Javascript
with python 2 days ago and I already have my first scripts running.
Mainly due to the wealth of information in this blog.

I have a C++ object that is already instantiated and now
I want to give python a reference to it... Basically I have

class X
{
typedef struct
{
PyObject_HEAD
ScnNode* mObject;
} PyStruct;

static PyTypeObject PyType;

static init()
{
   PyType_Ready(&PyType) < 0)
   Py_INCREF(&PyType);
   PyModule_AddObject(sModule, "Node", (PyObject*) &PyType);
}
};

I think that in the constructor of X I now have to make some calls in
order to provide access to this instance from a python script I
also think that I only want to get PyObject_New to be called, but not
PyObject_Init, because there I create a new instance of X when it is
instantiated through a script Am I on the right track here ?

Cheers
Jochen

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: publishing an already instantiated class

[J]

I think I just found my own solution, so I will record it for future
references

I did this in my Constructor of X
{
PyStruct* lObject;
lObject = PyObject_New(PyStruct, &X::PyType);
lObject->mObject = this;
}

What does everyone think about this. I am posting a little bit out of
the hip here,
and should probably investigate my solution a little more.
Nevertheless, it is
a good way to record my experiences...

Cheers
Jochen

-- 
http://mail.python.org/mailman/listinfo/python-list

embedding a new method

[J]

Hi everyone,


I am fairly new to python (3rd day), but i am fairly keen on
replacing javascript. I want to externalize some of the mehtod
in my App, an allow user to imp

char* lBuffer = "def handler(color):\n"
" print 12"

PyObject* lCode = Py_CompileString(lBuffer, "",
Py_file_input);
if (lCode)
{
mFunction = PyFunction_New(lCode, ScnGlobal::sDictionary);
Py_XDECREF(lCode);
}
else
{
PyErr_Print();
}

So far, there is no error, but I am wondering if that function object
knows about the signature of the function at this point.
Next I am defining an class like


PyType_Ready(&PyType);
Py_INCREF(&PyType);
PyModule_AddObject(ScnGlobal::sModule, "Place",
(PyObject*)&PyType);

where PyType is pretty much taken out of the "embedding tutorial",
and ScnGlobal::sModule is the result of the call to
PyImport_ImportModule ("__main__"). I am installing tp_init and tp_new
in PyType.

Next, I instantiate an object of PyType like

PyObject* new_args = PyTuple_New(0);
mPyObject = (PyStruct*) PyObject_Call((PyObject*)&PyType, new_args,
0);

This also works and tp_init and tp_new get called. Now, I want to add
the new mehtod to this instance, and not the class, so I call


PyObject* lParam = PyMethod_New(mFunction, (PyObject*)mPyObject,
(PyObject*)&PyType);
if (lParam)
{
Py_XDECREF(lParam);
}
else
{
PyErr_Print();
}

This call also returns an object. The problem starts when I try to
call the new method like so

PyObject* lObject = PyObject_CallMethod((PyObject*)mPyObject,
"handler", "", NULL);

if (lObject)
{
Py_DECREF(lObject);
}
else
{
PyErr_Print();
}

This when I get an error saying "AtrributeError:handler". I would
really appreciate some help on this. Here is also what the PyType
structure looks like


PyTypeObject CmdPlace::PyType =
{
PyObject_HEAD_INIT(NULL)
0,  /*ob_size*/
"Place",/*tp_name*/
sizeof(CmdPlace::PyStruct), /*tp_basicsize*/
0,  /*tp_itemsize*/
0,  /*tp_dealloc*/
0,  /*tp_print*/
0,  /*tp_getattr*/
0,  /*tp_setattr*/
0,  /*tp_compare*/
0,  /*tp_repr*/
0,  /*tp_as_number*/
0,  /*tp_as_sequence*/
0,  /*tp_as_mapping*/
0,  /*tp_hash */
0,  /*tp_call*/
0,  /*tp_str*/
0,  /*tp_getattro*/
0,  /*tp_setattro*/
0,  /*tp_as_buffer*/
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
"CmdPlace", /* tp_doc */
0,  /* tp_traverse */
0,  /* tp_clear */
0,  /* tp_richcompare */
0,  /* tp_weaklistoffset */
0,  /* tp_iter */
0,  /* tp_iternext */
CmdPlace::sPyMethods,   /* tp_methods */
0,  /* tp_members */
CmdPlace::sPyGetSeters, /* tp_getset */
0,  /* tp_base */
0,  /* tp_dict */
0,  /* tp_descr_get */
0,  /* tp_descr_set */
0,  /* tp_dictoffset */
(initproc)CmdPlace::sPyInit,/* tp_init */
0,  /* tp_alloc */
CmdPlace::sPyNew,   /* tp_new */
};


I hopesome has an answer. I have been stuck with this for a while.


Cheers
Jochen

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: embedding a new method

[J]

If your are reading this, I am sorry but I forgot to finish the first
paragraph :(. I am
trying to externalize some methods of a C++ object using python. User
will be
able to implement the function via the GUI and I then would like to add
it to an instance... So the class consits of both python and c++
methods... 

Cheers

-- 
http://mail.python.org/mailman/listinfo/python-list

linking problem

[J]

Hi everyone,

I thought that I would compile the python debug library and
step through it to figure out where things fail. So, I downloaded
python 2.4.1 and compile the python24_d.lib using the dsp files
in PCBuild. It compiles fine whitout a problem, but when I link agains
it i get a unresolved external:

error LNK2001: unresolved external symbol _Py_InitModule4

The strange thing is that this only function I have problems
with. I am using a lot of other PY.. funtions and they link fine.
I don't know what to do, but I will give my kingdom for an answer.

Cheers
Jochen

-- 
http://mail.python.org/mailman/listinfo/python-list

inheritance

[J]

Hi


I am trying to make a C++ class hierarchy accessible to python scripts.
Am I right to assume that the type structure of a derived class must
have all the members of its base class declared as well ?

For example,

typedef struct
{
PyObject_HEAD
int somebaseclassmember;
} PyStructBaseClass

is the structure for a base class. In this case, the structure for a
derived class would have to look like

typedef struct
{
PyObject_HEAD
ScnNode* mObject;
int somebaseclassmember;
float somederivedclassmember

} PyStructDerivedClass;


Is this true ? I have linked the derived class type to the base class
type via tp_base. I don't think that when I instantiate a derived
class, a PyStructBaseClass is being created. I believe that if
PyStructDerivedClass was missing the "int somebaseclassmember;" then
the offset for that memory would point into the float during base class
member lookups  ???

Any thoughts ?

Cheers
Jochen

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: inheritance

[J]

just to follow up. I managed to get it working by having a closer look
at the xxmodules. However, the example does the following

typedef struct
{
PyStructBaseClass  mBase;
int mValue;
} PyStructDerivedClass;

Does anything speak agains doing this ?

typedef struct : public PyStructBaseClass
{
int mValue;
} PyStructDerivedClass;

It seems to work for me. The advantage is that access to the members in
that structure is easier. The .mBase is not needed 

Cheers
Jochen

-- 
http://mail.python.org/mailman/listinfo/python-list

Adding and attribute to an instance

[J]

Hi,


I am trying to add new data attributes to my extension classes from
within a script. I am under the impression that python allows
that implicity


This is the definition of my class

PyTypeObject CmdPlace::PyType =
{
PyObject_HEAD_INIT(NULL)
0,  /*ob_size*/
"Place",/*tp_name*/
sizeof(CmdPlace::PyStruct), /*tp_basicsize*/
0,  /*tp_itemsize*/
0,  /*tp_dealloc*/
0,  /*tp_print*/
0,  /*tp_getattr*/
0,  /*tp_setattr*/
0,  /*tp_compare*/
0,  /*tp_repr*/
0,  /*tp_as_number*/
0,  /*tp_as_sequence*/
0,  /*tp_as_mapping*/
0,  /*tp_hash */
0,  /*tp_call*/
0,  /*tp_str*/
PyObject_GenericGetAttr,/*tp_getattro*/
PyObject_GenericSetAttr,/*tp_setattro*/
0,  /*tp_as_buffer*/
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
"CmdPlace", /* tp_doc */
0,  /* tp_traverse */
0,  /* tp_clear */
0,  /* tp_richcompare */
0,  /* tp_weaklistoffset */
0,  /* tp_iter */
0,  /* tp_iternext */
CmdPlace::sPyMethods,   /* tp_methods */
CmdPlace::sPyMembers,   /* tp_members */
CmdPlace::sPyGetSeters, /* tp_getset */
0,  /* tp_base */
0,  /* tp_dict */
0,  /* tp_descr_get */
0,  /* tp_descr_set */
0,  /* tp_dictoffset */
0,  /* tp_init */
0,  /* tp_alloc */
0,  /* tp_new */
};

I call

PyType_Ready(&PyType);
Py_INCREF(&PyType);

to initialize the type, and

PyObject_INIT((PyObject*)&mPyObject, &CmdPlace::PyType);

to initialize an object. Objects of this type are only ever
instantiated from C++. When I evaluate a sript I just add the object as
"MyObject" to the dicitonary passed to Py_Eval... All the
members and methods work fine, but when I do

MyObject.aNewAttribue = 12

I get at an error saying

object has no attribute "aNewAttribue".

I have looked at some of the source code in PyObject_GenericGetAttr and
it turns out that the object has no dictionary. It seens that the
address of the dictionary is computed somehow via tp_dictoffset in the
type object.

Basically my question is, how can I make this work.


Cheers
Jochen

-- 
http://mail.python.org/mailman/listinfo/python-list

packaging python for install.

[J]

Hi everyone,

I have created an App that embedds the python interpreter and I am
now in the process of creating an installer. I am currently linking
python24.lib, but it is only 184k and I suspect that it imports other
dlls... I am also using numarray. Does anyone have any experiences in
packaging python with an application in a single installer ? Has that
been done before ?


Cheers

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: packaging python for install.

[J]

Hi Miki

Thx for you reply. I have tried the procedure but I am stuck with

python setup.py py2exe --includes mymodule.

I get the following error: "ImportError: No module named mymodule"

I don't know what mymodule should contain. Basically I want all of
python and numarray
to be part of the distribution. The script creates a directory called
dist, but that's about it...

Am I right to assume that mymodule just contains an bunch of import
statements ?


Cheers
Jochen

-- 
http://mail.python.org/mailman/listinfo/python-list

packaging python for install.

[J]

Hi everyone,


I have posted a question on this topic before and already received some
useful advice. I am basically trying to package python with
an App that embedds the interpretor. I am also including numarray (i.e.
I have a #include  in my C++ code).  However it
looks like I have problems with numarray.

I have run the following:

>>python setup.py py2exe --includes numarray

where

>setup.py contains
>
>from distutils.core import setup
>import py2exe
>
>setup()

This produced the 'dist' directory including the library.zip file.
However there was also the following error message:

>The following modules appear to be missing
>['numarray._dotblas']

I renamed library.zip to python24.zip and changed the registry entry
PythonDir to c:\python24\dist... Then I started my app again. It seems
that native python (i.e. PyType_READY) worked fine, but a call to
numarray function PyArray_FromDimsAndData just crashes the App without
an error message.

I have also tried using

>>python setup.py py2exe -p numarray

without success. Has anyone packaged numarray before... any advice ?


Cheers
Jochen

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: packaging python for install.

[J]

Hi


I found the solution to problem and I just want to document it for the
next guy. I
did not copy the content of the 'dist' directory created by Py2exe to
the folder containing my executable.

Changing HKEY_LOCAL_MACHINE\SOFTWARE\PythonCore\. alone
did not work for me... maybe I should have found and changed PYTHONPATH
as
well :) Anyways, now I have a solution.

thx for all the help

-- 
http://mail.python.org/mailman/listinfo/python-list

A bit of a boggle about subprocess.poll() and the codes it receives from a process

[J]

Hi,

I need a bit of help sorting this out...

I have a memory test script that is a bit of compiled C.  The test itself
can only ever return a 0 or 1 exit code, this is explicitly coded and there
are no other options.

I also have a wrapper test script that calls the C program that should also
only return 0 or 1 on completion.

The problem i'm encountering, however, involves the return code when
subprocess.poll() is called against the running memory test process.  The
current code in my wrapper program looks like this:

def run_processes(self, number, command):
passed = True
pipe = []
for i in range(number):
pipe.append(self._command(command))
print "Started: process %u pid %u: %s" % (i, pipe[i].pid,
command)
sys.stdout.flush()
waiting = True
while waiting:
waiting = False
for i in range(number):
if pipe[i]:
line = pipe[i].communicate()[0]
if line and len(line) > 1:
print "process %u pid %u: %s" % (i, pipe[i].pid,
line)
sys.stdout.flush()
if pipe[i].poll() == -1:
waiting = True
else:
return_value = pipe[i].poll()
if return_value != 0:
print "Error: process  %u pid %u retuned %u" %
(i, pipe[i].pid, return_value)
passed = False
print "process %u pid %u returned success" % (i,
pipe[i].pid)
pipe[i] = None
sys.stdout.flush()
return passed

So what happens here is that in the waiting loop, if pipe[i].poll returns a
-1, we keep waiting, and then if it returns anything OTHER than -1, we exit
and return the return code.

BUT, I'm getting, in some cases, a return code of 127, which is impossible
to get from the memory test program.

The output from this bit of code looks like this in a failing situation:
Error: process 0 pid 2187 retuned 127
process 0 pid 2187 returned success
Error: process 1 pid 2188 retuned 127
process 1 pid 2188 returned success

I'm thinking that I'm hitting some sort of race here where the kernel is
reporting -1 while the process is running, then returns 127 or some other
status when the process is being killed and then finally 0 or 1 after the
process has completely closed out.  I "think" that the poll picks up this
intermediate exit status and immediately exits the loop, instead of waiting
for a 0 or 1.

I've got a modified version that I'm getting someone to test for me now that
changes

 if pipe[i].poll() == -1:
 waiting = True

to this

if pipe[i].poll() not in [0,1]:
waiting = True

So my real question is: am I on the right track here, and am I correct in my
guess that the kernel is reporting different status codes to
subprocess.poll() during the shutdown of the polled process?
-- 
http://mail.python.org/mailman/listinfo/python-list

Embedding a "frame" into a movie using python

[J]

Hi there,

I'm currently experimenting with python by putting together a little webapp 
running on appengine. 

My objective is to have the user put together some images html and embed the 
image as a frame into a video file (any embeddable format e.g. swf). So there 
are three steps:

(1) Convert the html to a "frame" 
(2) Insert the frame into the existing movie 
(3) save the new movie with the frame

For example, I have a film of a sunrise. I want to embed a "frame" which has a 
little poem about sunrises into the film much like windows movie maker and 
other such programs allow you to do. 

Is there a way I can do this with python?

Cheers,
J
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Embedding a "frame" into a movie using python

[J]

>> My objective is to have the user put together some images html and embed the 
>> image as a frame into a video file (any embeddable format e.g. swf). 

>SWF isn't the easiest format to work with; it's not necessarily a 
video, but might be a scripted object. Try working with real video 
formats such as avi, mpg, or ogv. 

>You may wish to look into PiTiVi, an open source video editor; it's 
written in Python. I'm not sure if you can achieve what you want by 
simply driving PiTiVi externally, but at very least, it's worth a 
peek. 

>ChrisA 

Hi Chris,

thank you for your response. I am format agnostic for processing so am happy to 
work with avi etc., however in displaying the final film to the user I think 
I'd need to convert whatever I've made to swf to make it a little less 
downloadable. I'll check out PiTiVi - perhaps I can hack it to work on the back 
of appengine...

Cheers,
J
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How do I pass a variable to os.popen?

[J]

On Mon, Oct 31, 2011 at 16:16, extraspecialbitter
 wrote:

>    if line[10:20] == "Link encap":
>       interface=line[:9]
>    cmd = 'ethtool %interface'
>    print cmd
>    gp = os.popen(cmd)

because you're saying that cmd is 'ethtool %interface' as you pointed
out later on...

how about:

cmd = 'ethtool %s' % interface

note the spaces there... that tells it to convert the contents of
interface to a string and insert them into the string you're assigning
to cmd...

assuming interface is things like eth0, you shoud now see "ethtool
eth0" when the print statement runs.
-- 
http://mail.python.org/mailman/listinfo/python-list

Is there a way to merge two XML files via Python?

[J]

This is more a theory exercise and something I'm trying to figure out,
and this is NOT a homework assignment...

I'm trying to make a tool I use at work more efficient :)

So this is at test tool that generates an XML file as it's output that
is eventually used by a web service to display test results and system
information.

The problem is that the testing is broken down into to different runs:
Functional and Automated where the Functional tests are all manual,
then the automated tests are run separately, usually overnight.

Each of those test runs generates essentially an identical XML file.
What I want to learn is a way to merge them.

In abstract terms, the idea is essentially to diff the two files
creating a patch and then use that patch to merge the two files into a
single XML file.

SO what I was hoping I could get pointers on from those of you who are
experienced in using Python with XML is what python libraries or means
are there for working with XML files specifically, and how easy or
difficult would this be?

I'm also doing research on my own in my spare time on this, but I also
wanted to ask here to get the opinion of developers who are more
experienced in working with XML than I am.

Thanks

Jeff
-- 
http://mail.python.org/mailman/listinfo/python-list

question about try/except blocks

[J]

I have this function in a class:

 def write_file(self, data, dest):
with open(dest, 'wb', 0) as outfile:
try:
print("IN WRITE_FILE")
outfile.write(self.data)
except IOError as exc:
logging.error("Unable to write data to %s: %s", dest, exc)
return False
else:
outfile.flush()
os.fsync(outfile.fileno())
return True


Which is simply called like this:
test.write_file(test.data, target_file)

However, when this is called on a read-only filesystem, it throws an OSError:

  File "./scripts/removable_storage_test", line 118, in write_file
with open(dest, 'wb', 0) as outfile:
OSError: [Errno 30] Read-only file system:
'/media/bladernr/5747-AD2E/tmpfbzsxk.0'

So what I am not sure of is how to handle this?

Would it be better to wrap the call and catch the OSError there, or
wrap the whole with open() block in the function itself?

My thought is to wrap the with open() call in the function so that I'm
not wrapping the function call every time I use the class somewhere,
but then I am not sure of that as it leads to nested try blocks like
so:

try:
with open(dest, 'wb', 0) as outfile:
try:
stuff
except IOError as exec:
more stuff
else:
other stuff
except OSError as exc:
 error handling stuff
 return False


I think, functionally, that should work, but should nested try/except
blocks be avoided?
-- 
http://mail.python.org/mailman/listinfo/python-list

Numarray, numeric, NumPy, scpy_core ??!!

[J]

Hi

I hope the title of this message indicates my question. I am looking
for basic
array functionality in Python and it turns out that there are all these
packages which
are somehow related. Some are allegedly discontinued but still seem to
get updated. Could we start a discussion about which package will or
may or should survive ?

I started to use numarray, but I have a bug that I just cannot find a
solution for, so I started
to look around again. Basically I want to provide scripting support to
a graphics engine. All the
matrices and vectors are in C++ and all I want to do is provide an
interface to python. As a matter of fact, all of the matrix
multiplication etc, could remain in the native C++ code, but I need a
way to extend the python language with the array data type and somehow
intercept operator calls such as *,+,- 

How is that done in the current libraries... Thx for any help.


Cheers
Jochen

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Numarray, numeric, NumPy, scpy_core ??!!

[J]

Ok, I will look at NumPy ...

I have another question about performance. Are the array operations
such as matrix multiplication implemented in
python or in C ? I was under the impression that the function calls in
numarray are only wrappers to C code, but now I suspect that matrix
multiplicaiton is implemented in python and executed in the
interpreter. Is that true ?

Can someone also please point me to examples of how I could make my own
array classes. Basically all I need is vectors and 3x3 matrices. I
already have all of the multiplication code in my C++ classes and only
need wrappers. But how can I create an array class and intercept
operators calls like * + -. This is not neccessarily a solution that I
want to follow, but it may be what
I have to do for now.

BTW, here is also the problem I have with a numarray. I have created a
vector like so...

mPyObject = PyArray_FromDimsAndData(GeoVec4f::sD,GeoVec4f::sDim,
PyArray_FLOAT,(char*)mValue.mValues);
Py_INCREF(mPyObject);

The PyObject is then used as a member called "gradient" in another
embedded class. When I execute the following
script

x.gradient*3

my app just crashes. But when I execute

print (x.gradient)
x.gradient*3

everything works fine... I assum there must be some kind of reference
count problem...


Cheers
Jochen

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Numarray, numeric, NumPy, scpy_core ??!!

[J]

I will just jump in an use NumPy. I hope this one will stick and evolve
into the mother of array packages.
How stable is it ? For now I really just need basic linear algebra.
i.e. matrix multiplication, dot, cross etc

Cheers

-- 
http://mail.python.org/mailman/listinfo/python-list

NewB: Glob Question

[J]

Greetings Group-

I'm trying to put together a pattern matching script that scans a
directory tree for tif images contained in similar folder names, but
running into a NewB problem already. Is it the way I'm trying to join
multiple paths? Any help would be greatly appericated. Thanks, J!

import glob, sys, os

topdir = sys.argv[1]
tifFilter = '*.tif'
dirFilter = '**'

tifList = glob.glob(os.path.join(topdir, tifFilter))
tifList = tifList + glob.glob(os.path.join(topdir, dirFilter,
tifFilter))

for tif in tifList:
   print os.basename(tif) + " is in " + os.dirname(tif)

-- 
http://mail.python.org/mailman/listinfo/python-list

Matching Directory Names and Grouping Them

[J]

Hello Group-

I have limited programming experience, but I'm looking for a generic
way to search through a root directory for subdirectories with similar
names, organize and group them by matching their subdirectory path, and
then output their full paths into a text file. For example, the
contents of the output text file may look like this:

\Input1\2001\01\
\Input2\2001\01\
\Input3\2001\01\

\Input1\2002\03\
\Input2\2002\03\
\Input3\2002\03\

\Input2\2005\05\
\Input3\2005\05\

\Input1\2005\12\
\Input3\2005\12\

I tried working with python regular expressions, but so far haven't
found code that can do the trick. Any help would be greatly
appreciated. Thanks!


J.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Matching Directory Names and Grouping Them

[J]

Steve-

Thanks for the reply. I think what I'm trying to say by similar is
pattern matching. Essentially, walking through a directory tree
starting at a specified root folder, and returning a list of all
folders that matches a pattern, in this case, a folder name containing
a four digit number representing year and a subdirectory name
containing a two digit number representing a month. The matches are
grouped together and written into a text file. I hope this helps.

Kind Regards,
J

Steve Holden wrote:
> J wrote:
> > Hello Group-
> >
> > I have limited programming experience, but I'm looking for a generic
> > way to search through a root directory for subdirectories with similar
> > names, organize and group them by matching their subdirectory path, and
> > then output their full paths into a text file. For example, the
> > contents of the output text file may look like this:
> >
> > \Input1\2001\01\
> > \Input2\2001\01\
> > \Input3\2001\01\
> >
> > \Input1\2002\03\
> > \Input2\2002\03\
> > \Input3\2002\03\
> >
> > \Input2\2005\05\
> > \Input3\2005\05\
> >
> > \Input1\2005\12\
> > \Input3\2005\12\
> >
> > I tried working with python regular expressions, but so far haven't
> > found code that can do the trick. Any help would be greatly
> > appreciated. Thanks!
> >
> Define "similar".
>
> regards
>   Steve
> --
> Steve Holden   +44 150 684 7255  +1 800 494 3119
> Holden Web LLC/Ltd  http://www.holdenweb.com
> Skype: holdenweb http://del.icio.us/steve.holden
> Blog of Note:  http://holdenweb.blogspot.com

-- 
http://mail.python.org/mailman/listinfo/python-list

Python Module: nift

[J]

What are your thoughts on this module I created?

'''
A Python Module created by a High School Student.
   Includes rev(), reet(), and leet().

Import nift

Function:   nift.rev()
Description:Reverses a string
Usage:  nift.rev('string')
Example:nift.rev('Testing function.')
Output: '.noitcnuf gnitseT'

Function:   nift.reet()
Description:Reverses a string
and transforms to leet text.
Usage:  nift.reet('string')
Example:nift.reet('Hello World!')
Output: 'dlr0W 0ll3H'

Function:   nift.leet()
Description:Transforms a string
into leet text.
Usage:  nift.leet('string')
Example:nift.leet('Hello there!')
Output: 'H3ll0 7h3r3!'

Setup:
* Save to Python Directory\Lib\ as nift.py
* Now you may import it by:
from nift import function
function()
OR
import nift
nift.function()

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
*
* This Module and its functions were created for my own fun, also
*
* contributing to Inter-Society. I don't mind if anyone modifies
*
*  this module or adds on to it, as long as you attribute it to
*
*   Myself.
*
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
*
'''

def rev(string):
list = []
final = ''
for letters in string:
list.append(letters)
list.reverse()
for letters in list:
final = final + letters
return final

def reet(string):
list = []
final = ''
for letters in string:
list.append(letters)
list.reverse()
for letters in list:
if letters == 'e' or letters == 'E':
letters = '3'
elif letters == 'a' or letters == 'A':
letters = '4'
elif letters == 'i' or letters == 'I':
letters = '1'
elif letters == 't' or letters == 'T':
letters = '7'
elif letters == 's' or letters == 'S':
letters = '5'
elif letters == 'o' or letters == 'O':
letters = '0'
elif letters == 'b' or letters == 'B':
letters = '8'
final = final + letters
return final

def leet(string):
list = []
final = ''
for letters in string:
if letters == 'e' or letters == 'E':
letters = '3'
elif letters == 'a' or letters == 'A':
letters = '4'
elif letters == 'i' or letters == 'I':
letters = '1'
elif letters == 't' or letters == 'T':
letters = '7'
elif letters == 's' or letters == 'S':
letters = '5'
elif letters == 'o' or letters == 'O':
letters = '0'
elif letters == 'b' or letters == 'B':
letters = '8'
list.append(letters)
for letters in list:
final = final + letters
return final
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Module: nift

[J]

Thanks for your answers, especially Chris Rebert and Paul McGuire's. I
have a question:
How far does Python go in the Game Development field? (Using Python
only, no extensions)
--
http://mail.python.org/mailman/listinfo/python-list

Email Program

[J]

Is it possible to make a GUI email program in Python that stores
emails, composes, ect?
--
http://mail.python.org/mailman/listinfo/python-list

Email Program

[J]

Is it possible to make a GUI email program in Python that stores
emails, composes, ect? Also, could I create my own programming
language in Python? What are Pythons limits, or is this just a waste
of my time to learn it.
--
http://mail.python.org/mailman/listinfo/python-list

Question about xml.com

[J]

Dear all, 

I have loaded an xml file into xmldoc.
I would have expected that print commandlet.childNodes[0].toxml() would
contain the  content but that's only at print
commandlet.childNodes[1].toxml() 

The same for print commandlet.childNodes[2].toxml() 

Why are commandlet.childNodes[0] and commandlet.childNodes[2] empty? 

Thanks in advance! 

J 

>>> print xmldoc.firstChild.toxml()

 0.1 Beta
 test
 just a test
 somebody
 linux2
 0

 3.19

>>> commandlet=xmldoc.firstChild
>>> print commandlet.firstChild.toxml()

>>> print commandlet.childNodes[0].toxml()

>>> print commandlet.childNodes[1].toxml()

 0.1 Beta
 test
 just a test
 somebody
 linux2
 0

 3.19

>>> print commandlet.childNodes[2].toxml()

>>> print commandlet.childNodes[3].toxml()

>>>--
http://mail.python.org/mailman/listinfo/python-list

SimpleXMLRPCServer and creating a new object on for each new client request.

[J]

Hi list,

My goals is to have concurrent and separated client sessions using xmlrpc.
Initially my though was that SimpleXMLRPCServer was able to create a new
object instance for each incoming request.
But this doesn't appear to be the case, unless I'm overlooking something,
if so please point me out.

Concider following simplified code


#!/usr/bin/python

from SimpleXMLRPCServer import SimpleXMLRPCServer
from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler
import random

# Restrict to a particular path.
class RequestHandler(SimpleXMLRPCRequestHandler):
rpc_paths = ('/RPC2',)

# Create a simple example class
class Randomizer:
def __init__(self):
self.random=random.randrange(0,10)
def show_random(self):
return self.random

# Create server
server = SimpleXMLRPCServer(("localhost",
8000),requestHandler=RequestHandler,allow_none=1)
server.register_introspection_functions()

server.register_instance(Randomizer())
server.serve_forever()



I start python interactively:
>>> import xmlrpclib
>>> session1=xmlrpclib.ServerProxy('http://localhost:8000')
>>> session2=xmlrpclib.ServerProxy('http://localhost:8000')
>>> print session1.show_random()
13930
>>> print session2.show_random()
13930
>>> 

I though that session1 and session2 would be 2 different Randomizer objects
each having a different result for self.random
But as the example shows this is not the case.
How can I solve this?

Thanks

Jelle

--
http://mail.python.org/mailman/listinfo/python-list

slightly OT: Python BootCamp

[J]

Ok... so I've been re-teaching myself python, as it's been several
years since I last really used it.  And in the midst of this, my
contracting company came up to me on Friday and asked if I'd be
interested in filling a last minute vacancy in this:

http://www.otg-nc.com/python-bootcamp

It's a week long Python Bootcamp.  I figured, free class, a week where
I'll not have to actually go to work, AND I'll get paid, sure!

So I was just wondering if any of you have attended this one, or one
like it, and what your impressions were.  I've attended a few before,
and found them widely different.  The RH300, for example, was
blisteringly fast and really just touches on things, so if you don't
already know Red Hat inside and out, you'll never survive.  The VMWare
VCP courses, on the other hand, were fairly languid by contrast and
seemed to flow in a less frantic state.

So I figured this would be the place to ask.

And if it matters, I do have an educational background in programming
(VB, C++, C, Java, etc) but my professional background has mostly been
limited to Bash, OCCASIONAL C, and now a touch of Python, so I am not
new to programming and OO programming, just not as proficient as I
would like to be...

Cheers
Jeff

-- 

Jonathan Swift  - "May you live every day of your life." -
http://www.brainyquote.com/quotes/authors/j/jonathan_swift.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: slightly OT: Python BootCamp

[J]

On Sun, Nov 29, 2009 at 03:57, Carl Banks  wrote:
> On Nov 28, 6:15 pm, J  wrote:

>> http://www.otg-nc.com/python-bootcamp
>>
>> It's a week long Python Bootcamp.
>
>
> I'm surprised they're able to fill out 5 days with intensive training
> on Python.
>
> :)

Well, if it's anything like other similar courses I've taken over the
years, the 10 minute break ever 2 hours will be more like 20 minutes,
the hour lunch more like 1.5 hours, and the 8:30 to 4:00 times will be
more like 8:45 to 3:15

;)

but on a serious note, your comment was why I was asking... I was
trying to figure out just how much python you could teach in an
assumed 40 hours... but it's also supposed to be probably 80%
(arbitrary semi-educated guess) hands on coding, and if that's the
case, each 30 minute project could well be 45 minutes to an hour
depending on how fast people type, how familiar they are with actually
properly formatting code, etc...



-- 

Pablo Picasso  - "Computers are useless. They can only give you
answers." - http://www.brainyquote.com/quotes/authors/p/pablo_picasso.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Question about file objects...

[J]

Something that came up in class...

when you are pulling data from a file using f.next(), the file is read
one line at a time.

What was explained to us is that Python iterates the file based on a
carriage return as the delimiter.
But what if you have a file that has one line of text, but that one
line has 16,000 items that are comma delimited?

Is there a way to read the file, one item at a time, delimited by
commas WITHOUT having to read all 16,000 items from that one line,
then split them out into a list or dictionary??

Cheers
Jeff

-- 

Ogden Nash  - "The trouble with a kitten is that when it grows up,
it's always a cat." -
http://www.brainyquote.com/quotes/authors/o/ogden_nash.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about file objects...

[J]

On Wed, Dec 2, 2009 at 09:27, nn  wrote:
>> Is there a way to read the file, one item at a time, delimited by
>> commas WITHOUT having to read all 16,000 items from that one line,
>> then split them out into a list or dictionary??

> File iteration is a convenience since it is the most common case. If
> everything is on one line, you will have to handle record separators
> manually by using the .read() method on the file
> object and searching for the comma. If everything fits in memory the
> straightforward way would be to read the whole file with .read() and
> use .split(",") on the returned string. That should give you a nice
> list of everything.

Agreed. The confusion came because the guy teaching said that
iterating the file is delimited by a carriage return character...
which to me sounds like it's an arbitrary thing that can be changed...

I was already thinking that I'd have to read it in small chunks and
search for the delimiter i want...  and reading the whole file into a
string and then splitting that would would be nice, until the file is
so large that it starts taking up significant amounts of memory.

Anyway, thanks both of you for the explanations... I appreciate the help!

Cheers
Jeff



-- 

Charles de Gaulle  - "The better I get to know men, the more I find
myself loving dogs." -
http://www.brainyquote.com/quotes/authors/c/charles_de_gaulle.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: slightly OT: Python BootCamp

[J]

On Sun, Nov 29, 2009 at 20:14, Terry Reedy  wrote:
> J wrote:
>>
>> Ok... so I've been re-teaching myself python, as it's been several
>> years since I last really used it.  And in the midst of this, my
>> contracting company came up to me on Friday and asked if I'd be
>> interested in filling a last minute vacancy in this:
>>
>> http://www.otg-nc.com/python-bootcamp

> The list of topics seems to cover basic + intermediate issues.
> If you do go, perhaps you could review it here.
> Same for people who have attended other Python training courses, of course.

The last day is tomorrow, so I'll actually try and write something
real then... I just wanted to say that this has been a pretty good
class.  Really good, compared to others I've taken (in college and so
forth)...

Anyway, just thought I'd share this:

The difference between me (learning a new language) and the guy
teaching (who knows it inside and out)

My code:

import os

startDir = '/home/student/testdirectory' #raw_input("Enter starting location: ")
searchList=['html','py','jpg']
filenames=[]
report={}

try:
fhtml=open('HTMLFiles','w')
fpy=open('PYFiles','w')
fjpg=open('JPGFiles','w')
except:
assert 0, 'Unable to open file'

for root, dirs, files in os.walk(startDir):
filenames += [os.path.normcase(os.path.join(root,f))
  for f in files if f[f.rfind('.')+1:] in searchList]

for file in filenames:
ext=file.split('.')[1]
if report.has_key(ext):
report[ext].extend([file])
else:
report.update({ext:[file]})

for k,v in report.iteritems():
if k == 'html':
for item in v:
fhtml.write("%s\n" % item)
if k == 'py':
for item in v:
fpy.write("%s\n" % item)
if k == 'jpg':
for item in v:
fjpg.write("%s\n" % item)

His code:

import os

def FindFiles(base_dir="/", exten=[]):

results={}

# Initialize results dictionary with blank arrays

for l in exten:

results[l]=[]

for root, dirs, files in os.walk(base_dir):

for e in exten:

results[e] += [os.path.join(root,fn) for fn in files if
fn.endswith(e)]

return results




files=FindFiles('C:/Python26',['exe','html'])



for (exten, list) in files.iteritems():

try:

f=open('extensions-%s.txt' % exten,'w')

f.write('\n'.join(list))

except:

assert 0, 'Unable to create output file extensions-%s.txt.' % exten

Different thought processes, plus experience makes a difference too...

That and while I have an educational background in programming, the
only programming I've been able to do in my professional (read daily)
life is BASH... I think it shows.

Cheers,

Jeff



-- 

Ted Turner  - "Sports is like a war without the killing." -
http://www.brainyquote.com/quotes/authors/t/ted_turner.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: slightly OT: Python BootCamp

[J]

On Fri, Dec 4, 2009 at 09:21, Steven D'Aprano
 wrote:
> On Thu, 03 Dec 2009 16:20:24 -0500, J wrote:
>
>> The difference between me (learning a new language) and the guy teaching
>> (who knows it inside and out)
>
> I don't know about that. He's teaching some pretty atrocious habits. See
> below.

Ok... I snipped it because I don't know enough really to comment on
any one thing...

I supposed in his defense, I could say that the point of the exercise
had nothing to do with exception handling, and the part that DID
handle exceptions covered a good bit on actually catching individual
exceptions for various things, as well as doing things like writing
extensions to handle things.

Also, consider that this is a "boot camp" that's geared toward getting
you up to speed and writing functional code as quickly as possible.
One week has gone from "This is a string variable and you can iterate
it" to This is file operation, OS and SYS usage, exception handling,
etc... really, just in 4 days.  The last day, I think just covers
database access and operations and a brief overview of python web
development and the cgi module.

It's more an intense introduction for people who already know how to
write code (not necessarily, though, ones who know how to write it
properly) than it is a formal "This is how you write code the right
way".

I'm not necessarily trying to make excuses, but, that's my take on it.
 All I wanted to get out of it was familiarity with Python, a refresh
of OOP practices, and the ability to get REALLY started on some
projects quickly, making the code pretty and better afterwards...

Thankfully, I've got places like comp.lang.python and various books
and online references to help me take what I know and make it better,
but as a starting point, at least, it's not terrible.

Now, that all being said...  how atrocious was my OWN code sample.
I'm a firm believer in constructive criticism, so I'd like to know how
I'm doing so far.  I can provide other samples if necessary, because I
DO want to learn to write this cleanly and properly.

Keep in mind that while I DO know how to write programs, the only
thing I've really written in in 3 years now is BASH.  It's been ages
since I did anything in any other languages to amount to anything, and
if I did, it was something quick and dirty that got the job done and
that was that, not something I'd really want to give to anyone else to
use... Hackery if you will...

Cheers,

Jeff
-- 

Ted Turner  - "Sports is like a war without the killing." -
http://www.brainyquote.com/quotes/authors/t/ted_turner.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Why Can't I Delete a File I Created with Win XP?

[J]

On Sat, Dec 5, 2009 at 17:42, John Machin  wrote:
> On Dec 6, 2:46 am, "W. eWatson"  wrote:
>> However, even at
>> that, why can't I delete this empty file called Analysis?
>
> Are you trying to delete the file from another command window while
> Python is paused at the interactive prompt? In any case describe how
> you are trying to delete the file. Fire up another command window, cd
> to whatever is the current working directory for your script, do a dir
> command, and copy/paste the RELEVANT parts (including all directory
> names). Please don't top-post, and before you hit the send button,
> please delete any text that is more appropriate to your memoirs than a
> problem description.

I'm no expert in Python, so YMMV and all that, but sounds like, as was
mentioned before, that the file isn't being properly closed.  I was
under the impression that the interpreter would close any open
filehandles as part of garbage collection as the script ends and the
interpreter closes out... but it's good practice to explicitly close
them in any case instead of just assuming that the interpreter or os
will do it for you... applicable to any language, not just python.

But that being said, this brings to mind a question about this...  in
*nix, when I can't do something like delete a file or directory, or
unmount a filesystem and cant find the cause, I can do an lsof and
grep for references to the file/directory in question, then work from
there to close anything that still has a lock.

Most all of the development work I've ever done (other than C++ and
VB.Net) has been done on Linux systems.  And honestly, most of my C++
stuff was written/compiled on Linux systems too... so I'm not that
experienced with developing applications on windows.  Does Windows (XP
or later) have an lsof or lsof type tool to show who/what has a lock
on a file or folder?  I've not found anything like that so far, but as
I said, I'm not much of a windows user anyway.

I'm at the point where I can actually write useful code in Python, and
not just hack my way through to a solution, so this is somethign that
would be useful for me to know.  I'm about to start work on writing a
test tool in Python 2.6 which is going to have to work with a few
filehandles, both for read and write actions, as well as socket
connections eventually.  So being able to find who has a lock on a
given file or directory if the program dies unexpectedly would be
useful.

Cheers
Jeff



-- 

Charles de Gaulle  - "The better I get to know men, the more I find
myself loving dogs." -
http://www.brainyquote.com/quotes/authors/c/charles_de_gaulle.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Why Can't I Delete a File I Created with Win XP?

[J]

On Sat, Dec 5, 2009 at 20:32, J  wrote:

> connections eventually.  So being able to find who has a lock on a
> given file or directory if the program dies unexpectedly would be
> useful.

Google tells me that the program Process Explorer from SysInternals
will provide most of the functionality of lsof... so I'll look into
that and see if it will work.  Certainly will help once I start
opening and writing output files with my python scripts on Windows
systems...


-- 

Stephen Leacock  - "I detest life-insurance agents: they always argue
that I shall some day die, which is not so." -
http://www.brainyquote.com/quotes/authors/s/stephen_leacock.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Why Can't I Delete a File I Created with Win XP?

[J]

On Sat, Dec 5, 2009 at 21:14, W. eWatson  wrote:

> What I'm trying to do is really simple. In the Win XP NG, I have two
> suggestions to get rid of the Analysis folder and the empty file in it. One
> is to use a program like you suggested, and the other is to delete it from
> DOS. I just tried cmd prompt, but was not successful. My DOS skills are long
> gone, so I have no idea if there is something I overlooked there. I bored
> down to Analysis and the into it. DIR showed an unnamed empty file, so I
> tried DEL *. I seriously doubt it was removed.
>
> Well, I'm going to reboot sometime later this evening, and knock it out as I
> described I was able to do once before after a reboot. Thne I'm going to fix
> the Python program and write a file correctly.

And those are your only options, really.  From what I've been able to
see, there is no native ability in Linux to actually see who has a
lock on a file that's been opened.  And I completely understand your
frustration.

I've not used the process explorer program, so I really can't say how
well it works for this kind of thing.  But every post I found
searching for this indicates that it will at least tell you what still
has a lock on the file and maybe then you can figure out what needs to
be closed from there.

Rebooting should always work in cases like this.  Rebooting the system
should clear all file locks and is a last resort for a persistent
stale file lock.  So yeah, by rebooting, you'll always be able to
release the lock on that file and then delete it once the system is
back and running.

However, that's not something you'd want to do on a production system
except as a last resort.  At least, that's not something that I'd want
to do.

So anyway, since you said the code is not yours, does the code
actually close the files anywhere?  I'm assuming that it does at some
point, but if not, that really is something that needs to be added in.
 As I said in my last post, I am just a novice in the Python stuff,
but I've written enough code in my life to know that you never assume
that a file will be closed properly by the system once the program is
finished running.  I'm not saying that YOU specifically are doing
this, but just making the suggestion as this is the kind of problem
that can happen.

Of course, a program that's dying for some reason is a different story... :-)
-- 

Mike Ditka  - "If God had wanted man to play soccer, he wouldn't have
given us arms." -
http://www.brainyquote.com/quotes/authors/m/mike_ditka.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Why Can't I Delete a File I Created with Win XP?

[J]

On Sat, Dec 5, 2009 at 23:17, W. eWatson  wrote:
> J wrote:

>> And those are your only options, really.  From what I've been able to
>> see, there is no native ability in Linux to actually see who has a
>> lock on a file that's been opened.  And I completely understand your
>> frustration.

Crud.. that should have said no native ability in Windows... not Linux...

> Well, I'm pretty well done now. I fixed the program so it would write the
> file and close it. It did this successfully. I'll worry about getting the
> oddball file deleted later.

Cool.  Glad you got it worked out... good luck!



-- 

Joan Crawford  - "I, Joan Crawford, I believe in the dollar.
Everything I earn, I spend." -
http://www.brainyquote.com/quotes/authors/j/joan_crawford.html
-- 
http://mail.python.org/mailman/listinfo/python-list

question about imports in a class

[J]

Just a little newbie confusion about OS imports...

Why does this give me an error:

class Windows:

def __init__(self):
'''
Constructor
'''
import os
self.dmidecodePath="" #final path to dmidecode binary

def parseDMI(self):
# First, find dmidecode.exe

for root,dirs,files in os.walk('C:\\'):
for file in files:
if file == "dmidecode.exe":

self.dmidecodePath=[os.path.normcase(os.path.join(root,file))]
return "Found DMIDecode.exe at %s" % self.dmidecodePath
break

Keep in mind that this is very early stage code and does nothing so
far other than telling me where to find dmidecode.exe...

But why does importing in the init not make os available to every
other function in the class?  Do I have to import OS into every
function like this:

class ClassA():

def func1(self):
import os

def func2(self):
import os

Also, when I DO have the import statement inside the function
definition, I'm screwing up the join and getting this:
"C:\\dmidecod\\sbin\\dmidecode.exe"

What am I doing that's creating the two \ characters??

Anyway, the first question I'm just asking because I'm still trying to
figure out the heirarchy and how classes work in Python, the other is
because I'm just annoyed...

Also, that brings up a third question:

So in my class, I import OS.  Now, in my calling script, do I also
import OS, or will I inheirit OS as part of the class object?

IOW:

script.py

thisclass=myclass():  #where myclass imports os

for r,d,f in os.walk(path)

or would I have to do something like:

for r,d,f in thisclass.os.wall(path)

Or do I just have to also import OS into script.py??

Thanks!  I know enough to be dangerous, so trying to become less so now...

Jeff
-- 

Ogden Nash  - "The trouble with a kitten is that when it grows up,
it's always a cat." -
http://www.brainyquote.com/quotes/authors/o/ogden_nash.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: question about imports in a class

[J]

On Mon, Dec 7, 2009 at 16:13, J  wrote:

> But why does importing in the init not make os available to every
> other function in the class?  Do I have to import OS into every
> function like this:
>
> class ClassA():
>
>    def func1(self):
>        import os
>
>    def func2(self):
>        import os

A little more education and playing around and I'm still not quite
sure how to do this...

for the class i'm writing, I want to import os, sys and wmi globally
for the class...
if I put the import at the beginning of the class, it just dawned on
me that perhaps I still have to explicitly call the function by class:

sysinfo.py

class myclass():
import os
def findDMIDecode(self):
for r,d,f in myclass.os.walk(args)

is that correct?

How about if I have two classes in sysinfo.py and want to import os
globally for BOTH classes to use? Same thing?

> Also, when I DO have the import statement inside the function
> definition, I'm screwing up the join and getting this:
> "C:\\dmidecod\\sbin\\dmidecode.exe"

Still not sure about this, but I DO know where I glaringly screwed up.

Originally, the function that finds the dmidecode executable was doing this:

return dmidecodePath # dmidecodePath is, I learned, a list that
contains the results of
 #
self.dmidecodePath=[os.path.normcase(os.path.join(root,file))]

which is wrong, because on the calling side, I was just using the
whole return, which was actually a list, not a string, so this was
fixed simply by:

return dmidecodePath[0]

which now returns simply:

c:\dmidecode\sbin\dmidecode.exe

So that seems correct.

Also, I apologize if my code examples are incomplete.  If they are not
complete "enough" please let me know. I'm doing this for work, and not
sure just how much I can really post without violating any obligations
on my end.. so I'm trying to provide generic examples based on the
code I'm actually trying to write.



-- 

Stephen Leacock  - "I detest life-insurance agents: they always argue
that I shall some day die, which is not so." -
http://www.brainyquote.com/quotes/authors/s/stephen_leacock.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: question about imports in a class

[J]

On Mon, Dec 7, 2009 at 16:57, Diez B. Roggisch  wrote:

>> if I put the import at the beginning of the class, it just dawned on
>> me that perhaps I still have to explicitly call the function by class:
>>
>> sysinfo.py
>>
>> class myclass():
>>    import os
>>    def findDMIDecode(self):
>>        for r,d,f in myclass.os.walk(args)
>>
>> is that correct?
>
> It is correct in the sense that it works, but it is by no means good
> practice.

So good practice (I'm doing this for real as well as for my own
education) would be:

sysinfo.py:

import os

class ClassA():
sysinfo.os.walk()

class ClassB():
sysinfo.os.walk()

Sorry for the incredibly entry-level questions...  I'm a more senior
member on a few Linux related lists/groups, and I can certainly
understand if questions as basic as these are annoying, so I'm really
trying to not ask them unless I'm just really confused about
something.

OTOH, if there's something more akin to a python newbies kind of list
where this kind of basic thing is more appropriate, please let me
know, because really, I don't want to clutter up python-list with
stuff that would be better off elsewhere.


-- 

Marie von Ebner-Eschenbach  - "Even a stopped clock is right twice a
day." - 
http://www.brainyquote.com/quotes/authors/m/marie_von_ebnereschenbac.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Problem using commands.getoutput()

[J]

Reading up on ways to run commands in a shell and capture output...

So I was looking at os.exec*() and that's not the correct thing here.
If I understand the docs correctly, the os.exec*() functions actually
end the calling program and replace it with the program called by
os.exec*() even as far as giving the new process the calling process's
PID:

"These functions all execute a new program, replacing the current
process; they do not return. On Unix, the new executable is loaded
into the current process, and will have the same process id as the
caller. Errors will be reported as OSError exceptions."

So from there, I found the commands module and getoutput() which
again, if I'm right, is supposed to run a command in a shell, and
return the output of that command a string.

commands.getoutput() is this:

def getstatusoutput(cmd):
"""Return (status, output) of executing cmd in a shell."""
import os
pipe = os.popen('{ ' + cmd + '; } 2>&1', 'r')
text = pipe.read()
sts = pipe.close()
if sts is None: sts = 0
if text[-1:] == '\n': text = text[:-1]
return sts, text

or at least it calls this function, and returns text, ignorint sts...

However, when I try using it, I get this:
>>> print commands.getoutput('dir')
'{' is not recognized as an internal or external command,
operable program or batch file.
>>>

Which looks like it's choking on the format of the "pipe = os.popen"
line, but that's just a guess... because this works:

p = os.popen('dir','r')
p.read()
p.close()

So what's the point of the commands module, or is that one that only
works in Linux, and not Windows?  I can do what I want, I think, by
using os.popen(), but I wanted to know if there was a better way of
doing it.

Cheers,
Jeff

-- 

Jonathan Swift  - "May you live every day of your life." -
http://www.brainyquote.com/quotes/authors/j/jonathan_swift.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem using commands.getoutput()

[J]

On Tue, Dec 8, 2009 at 14:44, Jerry Hill  wrote:

> At the very top of http://docs.python.org/library/commands.html it
> says "Platforms: Unix", so yes, it's Unix-only.

That sound you hear is me beating my head against the table now...
sigh... I was too wrapped up in reading the actual code to notice the
comments...

> Also at the top of that same bit of documentation is this: "The
> subprocess module provides more powerful facilities for spawning new
> processes and retrieving their results. Using the subprocess module is
> preferable to using the commands module."
>
> I would start with http://docs.python.org/library/subprocess.html and
> http://www.doughellmann.com/PyMOTW/subprocess/ for more information
> about the subprocess module and how it works.

And thanks for the pointer to subprocess.  I'll read over that (ALL of
it) and learn some more...  right now I'm doing it just using
os.popen() and dumping the read() results to a list, but I'm all for
learning different ways of accomplishing the task.  I do appreciate
the info!

Jeff



-- 

Samuel Goldwyn  - "I'm willing to admit that I may not always be
right, but I am never wrong." -
http://www.brainyquote.com/quotes/authors/s/samuel_goldwyn.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Help with parsing a list

[J]

Hi all,

I need some help in turning a list into a dictionary...

The list looks something like this:

['key1: data1','key2: data2','key3: data3',' key4: ',' \tdata4.1','
\tdata4.2',' \tdata4.3','key5: data5']

and it's derived from output (via subprocess.Popen) that in a terminal
would look like this:

key1: data1
key2: data2
key3: data3
key4:
data4.1
data4.2
data4.3
key5: data5

So what I want to do is turn this into a dictionary that looks like this:

{'key1':'data1','key2':'data2','key3':'data3','key4':['data4.1','data4.2','data4.3'],'key5':'data5']

So the problem I am having is just with the key 4 stuff... right now,
I'm looking at something like this (pseudocode because I'm still
trying to work this out)

loop through list
  if item is 'flags'
make flags the key
add every following item until item + 1 does not start with \t
  partition item where item[0] is key and item[2] is value

Sorry for not having actual code yet... I am still trying to work out
how to handle this on paper before actually writing code.

And no, it's not a homework assignment ;-) just a small part of a
project I'm working on.

Cheers,

Jeff

-- 

Pablo Picasso  - "Computers are useless. They can only give you
answers." - http://www.brainyquote.com/quotes/authors/p/pablo_picasso.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Thanks for the help not given :)

[J]

I just wanted to take a break from writing some code to thank the list
for all the help you've NOT given me :-)

And I do mean that but I should explain...

At least three times today, I've been working on various bits of code,
and run into a problem that I just could not figure out.

And each time, I started writing a post, complete with code samples
and my thoughts... and in EVERY case, so far, I've found the answer on
my own ONLY after writing that post.  Thankfully, I've not actually
posted them, saving myself from looking like TOO much of an idiot
(especially since one case was a REALLY glaring example of a Logic
Error).

I could say that it's all because I'm learning and figuring things out
on my own, but because I don't want to post poorly formed help
requests on any list I belong to, actually writing out a post helps me
organize my thoughts, the problem, and what I've done so far, and
usually THAT is when the solution actually hits me. Even when I don't
send the post, just having a target that forces me to actually
organize and spell out what the problem is helps immensely.

So though I've only posted a small bit here and on python-win, I did
want to thank y'all for helping me when you have, and even when you
actually haven't!

Cheers
Jeff


-- 

Ted Turner  - "Sports is like a war without the killing." -
http://www.brainyquote.com/quotes/authors/t/ted_turner.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Thanks for the help not given :)

[J]

On Tue, Dec 29, 2009 at 17:58, Phlip  wrote:
> To the OP - adding "... because Python sucks" to your subject lines will
> increase the quantity of answers - but possibly not the quality.

Unlike Stephan, I hope this was intended in humor/irony rather than
the way it looks, because my OP was meant humorously/ironically.

> You can also learn a little about good questions by answering others's here.

I don't answer questions without having $CLUE, fortunately.  I've been
on lists like this for well over 15 years, going back to when the only
place to find them was usenet, and I've been party to and target of my
fair share of flame wars due to improperly formatted questions or
uninformed answers ;-)

> And I hope you answered your questions here, if no one else did, to avoid
> dead search trails in the archives. You learn by teaching!

Indeed, which is why I teach Linux 101 at the community college level
(credited, not con-ed, an actual, honest to goodness Linux course!!).
I have not answered my own questions here, because the ones that I can
and eventually answer myself, such as the three I mentioned in my OP,
were really so basic as to be just noise on this list (though I guess
this discussion probably borders on it too).  My original POINT was
that the help/answers that I HAVE received here, in addition to the
various threads I read as they come up, has given me enough background
education to really start figuring things out on my own, as opposed to
constantly asking someone else for an answer or trying to grep my way
through various documents/tutorials/HOW-TOs/books/etc.

So it really was meant as a thanks for the people here who do have
$CLUE for helping me start getting one of my own.

> As for me, my questions are usually some combination of hopelessly retarded
> and beyond the blue sky envelope that I take pity on anyone even reading
> them, less answering them...

I sometimes think that of my own questions, though I really do try to
keep the basic questions to a minimum anywhere... then again, we all
have to start somewhere, which is why I try to not haze newbies, even
when that's my first reaction.

> Jon Clements wrote:
>
>> You have a bear that likes a Python? The one I have just keeps going
>> on about Piglet and eating my honey reserves...

As for Jon, and Aahz, I'd try the teddy bear approach, but the last
one I knew led me down the dark path to Perl and thus I try to avoid
them whenever possible ;-)

Cheers,

Jeff

-- 

Jonathan Swift  - "May you live every day of your life." -
http://www.brainyquote.com/quotes/authors/j/jonathan_swift.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multivariable assignment

[J]

On Thu, Dec 31, 2009 at 11:13, davidj411  wrote:
> I am not sure why this behavior is this way.
> at beginning of script, i want to create a bunch of empty lists and
> use each one for its own purpose.
> however, updating one list seems to update the others.
>
>>>> a = b = c = []
>>>> a.append('1')
>>>> a.append('1')
>>>> a.append('1')
>>>> c
> ['1', '1', '1']
>>>> a
> ['1', '1', '1']
>>>> b
> ['1', '1', '1']

That's because you're saying c is an empty list object, b is a pointer
to the same list object and a is a pointer to the same list object.
It looks better if you look at it like this:

a \
b  --  []
c /

You just created one actual object and created three pointers to it.

Though I wondered something similar the other day...

obviously this won't work:

a,b,c = []

but does this do it properly:

a,b,c = [],[],[]

For now, in my newbieness, I've been just creating each one separately:

a=[]
b=[]
c=[]

Cheers,

Jeff
-- 

Jonathan Swift  - "May you live every day of your life." -
http://www.brainyquote.com/quotes/authors/j/jonathan_swift.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python books, literature etc

[J]

On Wed, Jan 6, 2010 at 09:35, Shawn Milochik  wrote:
> Search Google. You'll find it all.
>
> Search this list's archives. This kind of thing has been discussed a thousand 
> times.
>
> It also wouldn't hurt to brush up on this:
> http://catb.org/~esr/faqs/smart-questions.html

Heh... I've seen that link mentioned many many times on some other
lists I belong to :)  But I agree.  HOWEVER, that is a valid question.

BUT, the answer is really up to the person asking it.

For example, I too am a relative beginner with Python.  Luckily I DO
have at least an educational background in OOP, and at least a
professional background in basic coding (some perl, BASH, etc).  So I
do tend to rely a lot on google when I run into a snag, THEN if that
doesn't work, I come here with specific questions.

FWIW, my Google searches always look something like "python " and 99% of the time, that gives me the answers I seek.

That being said, however, I also am somewhat old school and prefer to
have hard copy at hand too.  Sometimes, it's just that much more
satisfying to have a physical book handy to look things up in.  Maybe
it's a comfort thing, I don't know...

So in that vein, while I can't suggest any specific books, I can say
this... right now, I have a copy of O'Reilly's Learning Python that I
use as a reference, as well as a much older copy of the Python 2.1
Bible that I picked up almost a decade ago now the first time I messed
around with Python.

My personal preference, while searching Google and asking on lists
like this is quick and invaluable for the "real world" knowledge that
is shared, is to also have A: something along the lines of the
Learning Foo books from O'Reilly because they tend to have lengthy
explanations that I usually can understand on my own, and B: some sort
of Cookbook on the topic at hand.  I haven't found a Python cookbook
that I like enough to buy yet, but I'm still looking.  Those come in
handy for simple snippets and examples when doing specific tasks.

So yeah, Google and this list are certainly invaluable, but some of us
actually like having the heft of a lofty tome to peruse for the
answers to the mysteries of the universe.

Cheers
Jeff



-- 

Charles de Gaulle  - "The better I get to know men, the more I find
myself loving dogs." -
http://www.brainyquote.com/quotes/authors/c/charles_de_gaulle.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python books, literature etc

[J]

A good point was brought up to me privately, and I agree completely,
that the OP should re-state the request with a bit more specifics...

Since the OP says he is at least familiar with Python, does he need
info on beginner level books that are general purpose, or is he
interested in resources that are more specific (e.g. geared toward web
programming, mathematical analysis, data modeling, etc)

My suggestions were meant just as an example of what I use in the
course of learning something at the basic and intermediate level, once
it goes beyond that, it's useful to know WHAT you intend to do so you
can find the right resources to go in that direction.

Just want to make sure I am not misunderstood or anything :-)

For what it's worth, I also tend to collect technical books for some
reason... My wife is just barely tolerant of my bookshelf full of
things on various computer topics, astronomy, photography, radio and
antenna theory and so forth ;-)  I just let her keep her shoe
collection, and we have a quid pro quo.

Cheers
Jeff



-- 

Mike Ditka  - "If God had wanted man to play soccer, he wouldn't have
given us arms." -
http://www.brainyquote.com/quotes/authors/m/mike_ditka.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Scripting (was Re: Python books, literature etc)

[J]

On Fri, Jan 8, 2010 at 09:37, Jorgen Grahn  wrote:
> Regarding the book's title: is it just me, or are Python programmers
> in general put off when people call it "scripting"?
>
> I won't attempt a strict definition of the term "scripting language",
> but it seems like non-programmers use it to mean "less scary than what
> you might think of as programming", while programmers interpret it as
> "not useful as a general-purpose language".

I dunno...  I consider it programming when I'm writing bash scripts.
Same with running python scripts.

My personal take on it, so YMMV, is that scripting is just a synonym
for programming an interpreted language, as opposed to programming
(common parlance) which is usually meant writing code for a compiled
language (C, C++, VB, etc...)

Then again, I also tend to use scripting, coding and programming
interchangeably too.

And sometimes scripting = just writing a quick and dirty program to do
a small task, programming = writing something much larger for long
term use.

Either way, I'm not offended by any of those terms as they all involve
programming, regardless of whether or not someone actually calls it
programming.

For another analogy, what do they call Chinese food in China?  Food.

Cheers

Jeff



-- 

Ted Turner  - "Sports is like a war without the killing." -
http://www.brainyquote.com/quotes/authors/t/ted_turner.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: table from csv file

[J]

On Fri, Jan 8, 2010 at 13:55, Jon Clements  wrote:
> On Jan 8, 5:59 pm, marlowe  wrote:
>> I am trying to create a table in python from a csv file where I input
>> which columns I would like to see, and the table only shows those
>> columns. I have attached an example of the csv file i am using, and
>> some of the code I have written. I am having trouble converting
>> variables between lists, dictionaries and tuples. Is there a name for
>> what I am attempting to do? any help to get me on the right track with
>> this is appreciated.
>>
>> test.csv

I had to edit that and comma delimit it, because cut and paste gave me
random numbers/types of whitespace...

[code snipped]

> This might be a useful starting point (I'm guessing this is what
> you're after...)
>
> Let's assume your 'CSV' file is tab separated as it's certainly not
> comma separated :)
>
> import csv
> csvin = csv.reader(open('test.csv'), delimiter='\t')
> header = dict( (val.strip(),idx) for idx, val in enumerate(next
> (csvin)) )
>
> We can use header as a column name->column index lookup eg header
> ['Open'] == 1
>
> from operator import itemgetter
> wanted = ['Open', 'Close'] # Although you'll want to use raw_input and
> split on ','
> getcols = itemgetter(*[header[col] for col in wanted])
>
> getcols is a helper function that'll return a tuple of the columns in
> the requested order...
>
> for row in csvin:
>    print getcols(row)
>
> Loop over the rest of the file and output the required columns.

As someone who knows just enough to be dangerous... what about this:

import csv

reader = open('C:/test.txt','rb')
data = csv.DictReader(reader,restval='000',restkey='Misc')

print "Options are: Date, Open, Close, High, Low, Volume, Adj Close"
column = raw_input('What do you want? (comma delimited)?')
choice = column.split(',')
f = 12

print ''.join([s.center(f) for s in choice])

for row in data:
print ''.join([(row.get(s)).center(f) for s in choice])



-- 

Mike Ditka  - "If God had wanted man to play soccer, he wouldn't have
given us arms." -
http://www.brainyquote.com/quotes/authors/m/mike_ditka.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: table from csv file

[J]

On Sat, Jan 9, 2010 at 05:26, Jon Clements  wrote:

>> reader = open('C:/test.txt','rb')
>> data = csv.DictReader(reader,restval='000',restkey='Misc')
>
> [snip]
>
> DictReader works, but what use to bug me was the fact you couldn't
> then output the cols in the 'correct' order afterwards, so you had
> to store the header row anyway to re-order the rows...
> (although admittedly this doesn't affect the OP's question).
>
> However, I see that 2.6+ offers .fieldnames on DictReader objects.

At a guess, and I stipulate that because I am just learning about all
this, I'd say that's because DictReader returns a dictionary for each
row retrieved...  It DOES take the first row and make those the dict
keys, but after that, it just returns a Dict for each rows and
dictionaries seem to be randomly ordered.

It may not be that great if you're doing  something that needs order,
but all he's doing is pulling arbitrary columns from a file and
printing as a table, so that works great in this case...

It was quite interesting messing around with that yesterday.  I saw
the question, and it looked interesting, so I went and learned and
applied ;-)  I'm rather smugly satisfied with myself ...

Cheers,

Jeff

-- 

Samuel Goldwyn  - "I'm willing to admit that I may not always be
right, but I am never wrong." -
http://www.brainyquote.com/quotes/authors/s/samuel_goldwyn.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What is built-in method sub

[J]

On Mon, Jan 11, 2010 at 15:02, Jeremy  wrote:
> I am using the re.sub command to remove trailing whitespace from lines
> in a text file.  The commands I use are copied below.  If you have any
> suggestions on how they could be improved, I would love to know.

Just curious, but if each line is simply a string, wouldn't it be
easier to pass each one through .strip() instead?

Asking because I'm keen to know.  It's been a while since I bothered
with any real regluar expressions, and while I've touched on them in
bash programming and some perl, I've never touched them in Python.

But just from that little big of code, it seems to me that if each
item in lines is just a string object that is one line from the file,
then just passing it to .strip() to remove the whitespace is quick and
simple.  Unless there's a reason to using the overhead (assuming
python has the same or similar overhead to regular expressions that
sometimes occur in other languages).

Cheers,

Jeff



-- 

Pablo Picasso  - "Computers are useless. They can only give you
answers." - http://www.brainyquote.com/quotes/authors/p/pablo_picasso.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Inheriting methods but over-riding docstrings

[J]

On Sun, Jan 17, 2010 at 19:45, Alf P. Steinbach  wrote:
> Steven: on a personal note, earlier when I saw you (I think it was you)
> using the "Norwegian Parrot" example I thought it referred to me because
> that was the only sense I could make of it, it followed right after some
> discussion we had. Thus my impression of you or or responses in this group
> was colored by a false interpretation. But, checking, which is often a good
> idea!, and *which I should have done then*, as far as I can see the term was
> first used in this group in April 2001,  http://groups.google.com/group/comp.lang.python/browse_thread/thread/12a125ceddd401e2/c021547a1dc14a41>.
>
> It's still a mystery to me what it refers to, though... :-)

I can't really comment on decorators and subclassing yet (still a
newb) but I am finding this discussion very informative... and lo
there IS something I can comment on.

The Norwegian Blue is a remarkable bird.  Beautiful plumage!  Though
they do like to lie around on their backs a bit, and seem to spend
their time either resting, or pining for the fjords.  Yes lovely
plumage, indeed!

Google the remarkable Norwegian Blue Parrot and you'll fall in love as
I did so long ago :)

-- 

Marie von Ebner-Eschenbach  - "Even a stopped clock is right twice a
day." - 
http://www.brainyquote.com/quotes/authors/m/marie_von_ebnereschenbac.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: not and is not problem

[J]

On Mon, Jan 18, 2010 at 10:43, superpollo  wrote:
> hi:
>
> #!/usr/bin/env python
> data = "seq=123"
> name , value = data.split("=")
> print name
> print value
> if not name == "seq":
>    print "DOES NOT PRINT OF COURSE..."
> if name is not "seq":
>    print "WTF! WHY DOES IT PRINT?"

is will return True if two variables point to the same object, == if
the objects referred to by the variables are equal.

>> if not name == "seq":

says if the object that the variable name points to is NOT the same as
the string "seq" then do the following.
Since the object that "name" points to contains the string "seq" and
the string "seq" is identical in value to the "seq" in your
comparison, the result is TRUE (b = a) and your if statment only
proceeds if the comparison result is FALSE.

>> if name is not "seq"

is and is not relate to pointing to an object, not the object's contents.

Example:

>> name = "foo"
>> name1 = name
>> print name
foo
>> print name1
foo
>> name is name1
True
>> name1 is name
True
>> name = "bar"
>> print name
bar
>> print name1
foo
>> name is name1
False
>> name is not name1
True

Or even better:

>> name = "foo"
>> name1 = name
>> id(name)
11875840
>> id(name1)
11875840
>> name = "bar"
>> id(name)
11875744
>> id(name1)
11875840
>> id("foo")
11875840
>> id("bar")
11875744

shows the location for each object in relation to the name pointing to
that object...

-- 

Joan Crawford  - "I, Joan Crawford, I believe in the dollar.
Everything I earn, I spend." -
http://www.brainyquote.com/quotes/authors/j/joan_crawford.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Good Book

[J]

On Thu, Jan 28, 2010 at 11:09, talal awadh  wrote:
> Many people who want to learn Islam or are new converts find it hard

I just wanted to thank you for reminding me that I needed to write a
mail filter to delete this kind of drivel.  I appreciate the reminder!

Cheers,

Jeff
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: What is Islam?-1

[J]

On Tue, Oct 13, 2009 at 20:05, Aahz  wrote:
There's no point in trying to reason with a Muslim.
>>>
>>> That's not funny, and if you were being serious, that was incredibly
>>> rude.
>>
>>Not as much as posting in comp.lang.python.
>
> What exactly are you claiming is rude?

This entire thread is rude and really has no place on a list like this.

jeff
-- 

Marie von Ebner-Eschenbach  - "Even a stopped clock is right twice a
day." - 
http://www.brainyquote.com/quotes/authors/m/marie_von_ebnereschenbac.html
-- 
http://mail.python.org/mailman/listinfo/python-list

A stupid newbie question about output...

[J]

Can someone explain why this code results in two different outputs?

> for os in comp.CIM_OperatingSystem ():
>  print os.Name.split("|")[0] + " Service Pack", os.ServicePackMajorVersion
>  osVer = os.Name.split("|")[0] + " Service Pack", os.ServicePackMajorVersion
>  print osVer

the first print statement gives me this:
  Microsoft Windows XP Professional Service Pack 3

and the second print statement gives me this:
(u'Microsoft Windows XP Professional Service Pack', 3)

I know this is a grossly oversimplified example.

The ultimate goal is to get certain things from the system via WMI and
then use that data in other scripts to generate test case scripts for
a testing tool.

So given this example, what I ultimately want is to create a class
full of this stuff, that I can call from another program like this:

> import SystemInformation

> comp = SystemInformation()

> # Get OS info
> OSVersion = comp.osVer

OR just print the result

> print comp.osVer

Can this be cleared up by using some sort of formatting in print (like
sprintf) or am I going to have to manipulate the WMI results before
sending them out??

Thanks

Jeff

-- 

Ted Turner  - "Sports is like a war without the killing." -
http://www.brainyquote.com/quotes/authors/t/ted_turner.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A stupid newbie question about output...

[J]

On Tue, Oct 20, 2009 at 14:53, Ethan Furman  wrote:

> osVer = "%s Service Pack %d" % (os.Name.split("|")[0],
>        os.ServicePackMajorVersion)
>
> This way, osVer is a string, and not a tuple.

Thanks for the help...

The tuple thing is a new concept to me... at least the vocabulary is,
I'll go look that up now and learn info on tuples. It's been ages
since I did any python programming, and even back then it was fairly
simple stuff (this was about 9 years ago)... so I'm relearning by
working on this little project.

I also forgot that you can do printf style output too... that helps a
lot and is something I need to write on my forehead or something :-)


-- 

Stephen Leacock  - "I detest life-insurance agents: they always argue
that I shall some day die, which is not so." -
http://www.brainyquote.com/quotes/authors/s/stephen_leacock.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A stupid newbie question about output...

[J]

On Tue, Oct 20, 2009 at 16:25, Aahz  wrote:
> In article ,
> J   wrote:
>>
>>The tuple thing is a new concept to me... at least the vocabulary is,
>>I'll go look that up now and learn info on tuples. It's been ages
>>since I did any python programming, and even back then it was fairly
>>simple stuff (this was about 9 years ago)... so I'm relearning by
>>working on this little project.
>
> Tuples were certainly in Python ten years ago...

Oh I have no doubt... I just didn't really need to worry about it
then.  Like I said, I was just doing fairly simple stuff then.
Actually, at THAT time I was trying to teach myself Python by working
on a project... the project was using cuecat scanners to create a
home-built barcode inventory system... BUT, right after I got all the
front end stuff done, and was ready to actually build the back end db
to actually store real data, I got laid off and just stopped caring.

So now I'm older, wiser, and care again, and am teaching myself once
more by working on a project that interests me... well, it mostly
interests me.  I'm starting off with a simple hardware/os info
gathering program that will work on both Windows and Linux when
finished...



-- 

Mike Ditka  - "If God had wanted man to play soccer, he wouldn't have
given us arms." -
http://www.brainyquote.com/quotes/authors/m/mike_ditka.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Beginning Question about Python functions, parameters...

[j]

On Nov 23, 12:37 pm, Neo  wrote:
> astral orange schrieb:
>
>
>
> > Hi, I am trying to teach myself Python and have a good book to help me
> > but I am stuck on something and I would like for someone to explain
> > the following piece of code for me and what it's actually doing.
> > Certain parts are very clear but once it enters the "def store(data,
> > full_name): " function and the "def lookup()..." function things
> > get a little confusing for me. Specifically, lines 103-108 *and* Lines
> > 110-111.
>
> > Lastly, I am not sure how to print the results I've put into this
> > program either, the book I'm reading doesn't tell me. As you can tell,
> > I am a beginner and I don't truly understand everything that is going
> > on here...a lot, but not all
>
> > Here is the code:
>
> >  92 def init(data):
> >  93     data['first'] = {}
> >  94     data['middle'] = {}
> >  95     data['last'] = {}
> >  96
> >  97 def store(data, full_name):
> >  98     names = full_name.split()
> > 100     if len(names) == 2: names.insert(1, '')
> > 101     labels = 'first', 'middle', 'last'
> > 103     for label, name in zip(labels, names):
> > 104         people = lookup(data, label, name)
> > 105     if people:
> > 106         people.append(full_name)
> > 107     else:
> > 108         data[label][name] = [full_name]
> > 109
> > 110 def lookup(data, label, name):
> > 111     return data[label].get(name)
> > 112
> > 113
> > 114 MyNames = {}
> > 115 init(MyNames)
> > 116 store(MyNames, 'John Larry Smith')
> > 117 lookup(MyNames, 'middle', 'Smith')
>
> If it tells you so I'm not really sure its a good book - partially for
> teaching you into the unpythonic way to do things (above stuff, if its
> not a counter example, should really go into a class)
>
> Have you tried the tutorial first? Its online and very easy to follow
> from the very beginning but you can also skip parts if you are sure you
> already understand it:
>
> http://docs.python.org/tutorial/
>
> HTH
> Tino

The book is "Apress Beginning Python 2nd Edition". I think what you
are saying is if I don't understand all of the code I should go into a
class? Unfortunately I don't have the money or time to enroll into a
class and even if I did they wouldn't be teaching Python it would be
Java instead...and I've already taken Java before...so that's not
really an option...

Regardless, I understand up to the part "for label, name in zip
(labels, names):" which zips the two sequences 'names' and 'labels'
into a list of tuples...

What I am not totally sure about is when the store function callsthe
lookup function and does "return data[label].get(name)", that line
"trips" me up somethen the lookup function returns that back to
the store function, assigns the data to the variable 'people', THEN
does this, which also isn't that clear to me what it's all doing:

if people:
people.append(full_name)
else:
data[label][name] = [full_name]

My main problem is finding out what's it's actually *doing*? I know
and understand the terminologies (what 'append' does, what a
'sequence' is, a 'list', a 'tuple', 'if'...'else'...etc...et)

Thanks for the reply back
-- 
http://mail.python.org/mailman/listinfo/python-list

Ideas for creating processes

[J]

I'm working on a project and thought I'd ask for a suggestion on how
to proceed (I've got my own ideas, but I wanted to see if I was on the
right track)

For now, I've got this:

def main():
## get our list of directories to refresh
releases=sys.argv[1:]
if len(releases) < 1:
print "You need to provide at least one dir to update"
sys.exit()

## Lets figure out what there is to update
updateDirs = []
for rel in releases:
currentDir = os.path.join(homedir, rel)
for item in os.listdir(currentDir):
updateDirs += [os.path.join(homedir, rel, item)]

which returns a list of full pathnames to directories that need to be
updated (updates will be carried out by calling rsync or zsync
eventually)

The directory hierarchy looks like this:

/home/user/files
/home/user/files/version1
/home/user/files/version1/type1
/home/user/files/version1/type2
/home/user/files/version2
/home/user/files/version2/type1

and the list ends up looking like this:

['/home/user/files/version1/type1','/home/user/files/version1/type2','/home/user/files/version2/type1','/home/user/files/version2/type2']

the next thing I need to do is figure out how to update those.

the quick and dirty would be (as I'm imagining it at the moment):
for path in pathlist:
chdir into path
execute rsync or zsync

but that gets me moving into one dir, updating, then moving into another.

What I was wondering about though, is spawning off separate rsync
processes to run concurrently (each rsync will hit a different remote
dir for each local dir)

so I'm trying to figure out a good way to do this:

for path in pathlist:
kick off an individual rsync|zsync process to update path

wait for all child processes to end
exit program.

I've been looking at subprocess because at the moment, that's all I
really know...
But is there a better way of kicking off multiple simultaneous
processes so I can update all dirs at once instead of one at a time?

No, this isn't homework, it's something I'm working on to sync a
couple directories of ISO images to grab nightly builds
Yes, there are plenty of pre-made scripts out there, but I don't want
to even look at those because I figured this would be a good learning
experience, and I want to try to solve this as much on my own as I can
without just cut and pasting from someone elses program.

So, with that said, any ideas on the best way to proceed?  I'm going
to start looking at ways to use subprocess to do this, or would there
be a better way (multi-threading maybe?)

Or am I even in the right ballpark?

Cheers
Jeff
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Ideas for creating processes

[J]

On Wed, Mar 10, 2010 at 16:52, J  wrote:
> the quick and dirty would be (as I'm imagining it at the moment):
> for path in pathlist:
>    chdir into path
>    execute rsync or zsync
>
> but that gets me moving into one dir, updating, then moving into another.
>
> What I was wondering about though, is spawning off separate rsync
> processes to run concurrently (each rsync will hit a different remote
> dir for each local dir)
>
> so I'm trying to figure out a good way to do this:
>
> for path in pathlist:
>    kick off an individual rsync|zsync process to update path

And now I'm looking at subprocess and I can set shell=True and it will
intrepret special characters like &

So could I do something like this:

for item in pathlist:
subprocess.Popen('rsync command &', shell=True)

and simply wait unti they are all done?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Ideas for creating processes

[J]

On Wed, Mar 10, 2010 at 18:03, MRAB  wrote:
> Are you sure that you would gain from doing more than one at a time?
>
> The bottleneck will probably be the speed of your network connection,
> and if that's working at its maximum speed with one sync then doing
> several concurrently won't save any time. (The syncs will also be
> completing for disk I/O.)

Good point MRAB... thanks for making it.

I started to wonder the same thing, but it took me a couple hours away
from working on it and a good dinner before I saw that...

Yeah, looking at it freshly now, I agree, I think I probably would be
better off running them one at a time.

The goal is to update several directories of ISO images with nightly
builds.  I can use rsync and only download images that are changed, or
I can use zsync and only download diffs (I believe that's how it
works) which is faster anyway.

The only reason I was considering doing them all simultaneously, or in
batches at least, is that I have to update 10 - 12 ISOs daily... BUT,
now that I look at it from your perspective, yeah, it would probably
be faster overall to do them one at a time instead because of the
bottleneck.

Thanks!  That helped a lot (and probably saved me a lot of headache
later on) :-)

Cheers
Jeff
-- 
http://mail.python.org/mailman/listinfo/python-list

Get Eclipse/PyDev to run scripts that don't end in .py

[J]

Is there any way to tell PyDev in Eclipse to run a script that doesn't
end in .py?  Even if I have to go and manually set something for each
file...

I've inherited (in a manner of speaking) a dev project that is done in
python2.6... I pulled the latest dev branch and have it opened as a
project in Eclipse, however, none of the python files end in .py, so
PyDev only sees them as text files.  And because of that, I can't run
them in Eclipse to try my changes, debug, etc.

So, is there a way to make them show up as python code without the .py
extension?  Because of this, I also don't get any of the fancy
indenting and highlighting that I'd normally get...

Cheers,

Jeff
-- 
http://mail.python.org/mailman/listinfo/python-list

Regex driving me crazy...

[J]

Can someone make me un-crazy?

I have a bit of code that right now, looks like this:

status = getoutput('smartctl -l selftest /dev/sda').splitlines()[6]
status = re.sub(' (?= )(?=([^"]*"[^"]*")*[^"]*$)', ":",status)
print status

Basically, it pulls the first actual line of data from the return you
get when you use smartctl to look at a hard disk's selftest log.

The raw data looks like this:

# 1  Short offline   Completed without error   00%   679 -

Unfortunately, all that whitespace is arbitrary single space
characters.  And I am interested in the string that appears in the
third column, which changes as the test runs and then completes.  So
in the example, "Completed without error"

The regex I have up there doesn't quite work, as it seems to be
subbing EVERY space (or at least in instances of more than one space)
to a ':' like this:

# 1: Short offline:: Completed without error:: 00%:: 679 -

Ultimately, what I'm trying to do is either replace any space that is
> one space wiht a delimiter, then split the result into a list and
get the third item.

OR, if there's a smarter, shorter, or better way of doing it, I'd love to know.

The end result should pull the whole string in the middle of that
output line, and then I can use that to compare to a list of possible
output strings to determine if the test is still running, has
completed successfully, or failed.

Unfortunately, my google-fu fails right now, and my Regex powers were
always rather weak anyway...

So any ideas on what the best way to proceed with this would be?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Regex driving me crazy...

[J]

On Wed, Apr 7, 2010 at 22:45, Patrick Maupin  wrote:

> When I saw "And I am interested in the string that appears in the
> third column, which changes as the test runs and then completes" I
> assumed that, not only could that string change, but so could the one
> before it.
>
> I guess my base assumption that anything with words in it could
> change.  I was looking at the OP's attempt at a solution, and he
> obviously felt he needed to see two or more spaces as an item
> delimiter.

I apologize for the confusion, Pat...

I could have worded that better, but at that point I was A:
Frustrated, B: starving, and C: had my wife nagging me to stop working
to come get something to eat ;-)

What I meant was, in that output string, the phrase in the middle
could change in length...
After looking at the source code for smartctl (part of the
smartmontools package for you linux people) I found the switch that
creates those status messages they vary in character length, some
with non-text characters like ( and ) and /, and have either 3 or 4
words...

The spaces between each column, instead of being a fixed number of
spaces each, were seemingly arbitrarily created... there may be 4
spaces between two columns or there may be 9, or 7 or who knows what,
and since they were all being treated as individual spaces instead of
tabs or something, I was having trouble splitting the output into
something that was easy to parse (at least in my mind it seemed that
way).

Anyway, that's that... and I do apologize if my original post was
confusing at all...

Cheers
Jeff
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Regex driving me crazy...

[J]

On Thu, Apr 8, 2010 at 01:16, Kushal Kumaran
 wrote:
>
> Is there any particular reason you absolutely must extract the status
> message?  If you already have a list of possible status messages, you
> could just test which one of those is present in the line...

Yes and no...

Mostly, it's for the future.  Right now, this particular test script
(and I mean test script in the sense it's part of a testing framework,
not in the sense that I'm being tested on it ;-)  ) is fully
automated.

Once the self-test on the HDD is complete, the script will return
either a 0 or 1 for PASS or FAIL respectively.

However, in the future, it may need to be changed to or also handled
manually instead of automatically. And if we end up wanting it to be
automatic, then having that phrase would be important for logging or
problem determination.  We don't so much care about the rest of the
string I want to parse as the data it gives is mostly meaningless, but
having it pull things like:

Completed: Electrical error

or

Completed: Bad Sectors Found

could as useful as

Completed without error

or

Aborted by user

So that's why I was focusing on just extracting that phrase from the
output.  I could just pull the entire string and do a search for the
phrases in question, and that's probably the simplest thing to do:

re.search("Search Phrase",outputString)

but I do have a tendency to overthink things some times and besides
which, having just that phrase for the logs, or for use in a future
change would be cool, and this way, I've already got that much of it
done for later on.
-- 
http://mail.python.org/mailman/listinfo/python-list

question about list extension

[J]

Ok... I know pretty much how .extend works on a list... basically it
just tacks the second list to the first list... like so:

>>> lista=[1]
>>> listb=[2,3]
>>> lista.extend(listb)
>>> print lista;
[1, 2, 3]

what I'm confused on is why this returns None:

>>> lista=[1]
>>> listb=[2,3]
>>> print lista.extend(listb)
None
>>> print lista
[1, 2, 3]

So why the None? Is this because what's really happening is that
extend() and append() are directly manipulating the lista object and
thus not actuall returning a new object?

Even if that assumption of mine is correct, I would have expected
something like this to work:

>>> lista=[1]
>>> listb=[2,3]
>>> print (lista.extend(listb))
None

The reason this is bugging me right now is that I'm working on some
code.  My program has a debugger class that takes a list as it's only
(for now) argument.  That list, is comprised of messages I want to
spit out during debug, and full output from system commands being run
with Popen...

So the class looks something like this:

class debugger():
def printout(self,info):
for line in info:
print "DEBUG: %s" % line


and later on, it get's called like this

meminfo = Popen('cat
/proc/meminfo',shell=True,stdout=PIPE).communicate[0].splitlines()

if debug:
debugger.printout(meminfo)

easy enough

BUT, what if I have several lists I want to pass on multiple lists...

So changing debugger.printout() to:

   def printout(self,*info):

lets me pass in multiple lists... and I can do this:

   for tlist in info:
   for item in tlist:
print item

which works well...

So, what I'm curious about, is there a list comprehension or other
means to reduce that to a single line?

I tried this, which didn't work because I'm messing up the syntax, somehow:

def func(*info)
print [ i for tlist in info for i in tlist ]

so can someone help me understand a list comprehension that will turn
those three lines into on?

It's more of a curiosity thing at this point...  and not a huge
difference in code... I was just curious about how to make that work.

Cheers,

Jeff
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: question about list extension

[J]

On Fri, Apr 16, 2010 at 15:16, Terry Reedy  wrote:
> On 4/16/2010 9:41 AM, J wrote:
>>
>> Ok... I know pretty much how .extend works on a list... basically it
>> just tacks the second list to the first list... like so:
>>
>>>>> lista=[1]
>>>>> listb=[2,3]
>>>>> lista.extend(listb)
>>>>> print lista;
>>
>> [1, 2, 3]
>
> This shows right here that lista is extended in place. If you are not
> convinced, print(id(lista)) before and after.

Thanks for the explanations, everyone...

I was just a bit confused about how .extend was working... I
originally thought that it returned a new object and linked lista to
that, but I realize that it directly manipulates the list object
instead...

I'm not sure why I got that idea stuck in my head, but it was there,
so I asked :)
-- 
http://mail.python.org/mailman/listinfo/python-list

Is it better to extend a class, or do something repetitious in the main part of a program?

[J]

First, before I get farther,

Is there a way for the logging module to natively handle lists and
dict objects when logging?

e.g. take this {'key1':'val1','key2':'val2'} and have it logged like this:

INFO: key1: val1
INFO: key2: val2

If I pass the dict or list directly to the logger, it is logged the
same as if you simply did this:

mydict={1:1, 2:2}
mylist=[1,2,3]

print mydict
print mylist

>>> {1:1,2:2}
>>> [1,2,3]

It came up that I wanted to have logging present command line options
from optparse if the log level was set to debug...  so they'd look
something like this in the output:

debug: True
sleep_time: 30
log_file: /tmp/testlog

So the options I've managed to work out are to either parse the list
or dict object item by item and feed those items one at a time into
the logger:

for i in mylist:
logging.info(i)

Or to extend the StreamHandler class to handle this by creating a new
report.msg...

Then the discussion came up: which is better?  To parse a dictionary
or list in the main code and pass each item to the logger one at a
time, or to extend the logger to handle it natively?

Any thoughts on which is the more proper way to handle cases like this?
-- 
http://mail.python.org/mailman/listinfo/python-list

Difference between Popen and open() for reading a file

[J]

I was reading something from a code review a little while ago and saw
something that's got my curiosity up...

Say I had a file, foo.txt that I wanted to read from, only one time
and only read.

So what's the difference between this:

mylist = Popen(["cat","foo.txt"], stdout=PIPE).communicate()[0].splitlines()

and this:

f = open('foo.txt')
mylist = f.readlines()
f.close

Is there a reason why you would not use subprocess.Popen for this,
other than just not relying on external programs to perfrom the task?

what if that file only has one line in it, and that's all you're
interested in, and the file is guaranteed to only have that one line
it it?

For example:

say foo.txt contained only the number 123456789

what's wrong with doing this:

my_int = int(commands.getoutput('cat foo.txt')

or via the subprocess.Popen method mentioned above?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Difference between Popen and open() for reading a file

[J]

On Thu, Apr 22, 2010 at 15:18, Dave Angel  wrote:

> The same difference as between handing the paper boy three bucks, versus
> flying to London to open an account, making a deposit, going to a branch in
> Sydney and asking for a bank check, then flying back home and taking the
> paper boy with you to the bank to cash it.
>
> When you use subprocess, you're creating a whole new process in order the
> copy the file into another one, which you then read.  Takes much more
> resources, time, and is much more likely to break.  For example, what if
> this program runs on a system without "cat" ?
>
> DaveA

Thanks for this (and the other responses everyone)...  I'm still
really in the "learning" process and the reasons to NOT do it that way
make perfect sense to me (and I love DaveA's explanation)...

Learning something every day...

Though, if I could afford it, I wouldn't mind a trip to London and
then Sydney...
-- 
http://mail.python.org/mailman/listinfo/python-list

Is there a better way to set a system clock in Python (on a Linux system)

[J]

Is there a better way to do this?

def SkewTime():
'''
Optional function. We can skew time by 1 hour if we'd like to see real sync
changes being enforced
'''
TIME_SKEW=1
logging.info('Time Skewing has been selected. Setting clock ahead 1 hour')
# Let's get our current time
t = TimeCheck()
logging.info('Current time is: %s' % time.asctime(t))
# Now create new time string in the form MMDDhhmm for the date program
hr = t.tm_hour + TIME_SKEW
date_string = time.strftime('%m%d%H%M%Y',(t.tm_year,
t.tm_mon,
t.tm_mday,
hr,
t.tm_min,
t.tm_sec,
t.tm_wday,
t.tm_yday,
t.tm_isdst))
logging.debug('New date string is: %s' % date_string)
logging.debug('Setting new system time/date')
status = SilentCall('/bin/date %s' % date_string)
logging.info('Pre-sync time is: %s' % time.asctime())

TimeCheck() as referenced above is a simple function that just returns
the time.time_struct object from time.localtime().  I pull time a few
times and it was a little cleaner to put that into a function and just
call the function whenever I needed to.

SilentCall() is a modification of subprocess.call() (which in reality
just calls Popen(*popenargs,**kwargs).wait()) but it defaults to
redirecting stdin and stdout to /dev/null to suppress shell output
from the command being called.

Anyway, what I'm wondering, is, while this works, is there a better
way to do it than using part of the originally returned time_struct
and injecting my own new hour argument (hr).

The goal of this function is to just set the system clock one hour
ahead, so when I call the Linux command 'ntpdate' I can get a real
time change when it syncs the local clock to an NTP server.

This just looks... well, big to me.  I tried passing only the things I
really needed to time.strftime(), but apparently, that requires the
full 9-tuple from time_struct, not just individual parts of it.

Like I said, it works well, I just wonder if there is a cleaner way of
setting the local clock to a different time in python without having
to do all this.  The reason most of that exists, is because the linux
date command expects to see the new date/time like this:
MMDDhhmm.ss.

Or am I just looking at this too hard and really did work it out nicely?

Cheers
Jeff
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: write a 20GB file

[J]

On Fri, May 14, 2010 at 07:32, Jackie Lee  wrote:
> Thx, Dave,
>
> The code works fine. I just don't know how f.write works. It says that
> file.write won't write the file until file.close or file.flush. So I
> don't know if the following one is more efficient (sorry I forget to
> add condition to break the loop):

someone smarter than me can correct me, but file.write() will write
when it's buffer is filled, or close() or flush() are called.
I don't know what the default buffer size for file.write() is though.
close() flushes the buffer before closing the file, and flush()
flushes the buffer and leaves the file open for further writing.

> try:
>        f=open(sys.argv[1],'rb+')
> except (IOError,Exception):
>    print '''usage:
>        scriptname segyfilename
> '''

You can just add a f.flush() every time you write to the file, but, I
tend to open files with 0 buffer size like this:

f = open(filename,"rb+",0)

Then again, I don't deal with files of that size, so there could be a
problem with my way once you start scaling up to the 20GB or larger
that you're working with.

Again, I could be wrong about all of that, so if so, I hope someone
will correct me and fix my understanding...

Cheers,

Jeff
-- 
http://mail.python.org/mailman/listinfo/python-list