Re: my email

[MRAB]

On 18/07/2012 02:44, Maria Hanna Carmela Dionisio wrote:

mmdionisio1...@yahoo.com.ph

Just a newbhie here :>


...who has just revealed her password!

[remainder snipped]
--
http://mail.python.org/mailman/listinfo/python-list

Python Programming expert - adding user

[Maria Hanna Carmela Dionisio]

Im just a student :)

Our prof gave as a task that we need to make a program using python (for 
redhat) and c++(for windows)

Our objective is to make a program file and we will said it remotely to another 
computer via network ( its easy and i could do it lolz)..

the hard part is the programming..that when the user click the file a box will 
prompt and ask for username and a password(even without a password) then the 
username that he inputted will add to the user account of his/her computer :DD

HELP HELP EXPERTSS

THANKS :D
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: my email

[Karim]

Le 18/07/2012 04:34, Simon Cropper a écrit :

On 18/07/12 11:44, Maria Hanna Carmela Dionisio wrote:

mmdionisio1...@yahoo.com.ph

Just a newbhie here :>


[snip]


You must know your password to change your options (including changing
the password, itself) or to unsubscribe.  It is:

   sweet103093



I suggest you change you password now that you have told the world 
what it is.



??

--
http://mail.python.org/mailman/listinfo/python-list

Re: Foxpro goto command and deleted records

[Ian Kelly]

On Wed, Jul 18, 2012 at 8:40 AM, Grant Edwards  wrote:
> On 2012-07-17, Ethan Furman  wrote:
>
>> In Foxpro if you do a
>
> Foxpro?

http://en.wikipedia.org/wiki/Visual_FoxPro
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A thread import problem

[Dieter Maurer]

Bruce Sherwood  writes:

> I'm trying to do something rather tricky, in which a program imports a
> module that starts a thread that exec's a (possibly altered) copy of
> the source in the original program, and the module doesn't return.
> This has to do with an attempt to run VPython in the Mac Cocoa
> context, in which Cocoa is required to be the primary thread, making
> it necessary to turn the environment inside out, as currently VPython
> invokes the Carbon context as a secondary thread.
>
> I've created a simple test case, displayed below, that illustrates
> something I don't understand. The module reads the source of the
> program that imported it, comments out the import statement in that
> source, and performs an exec of the modified source. The module then
> enters an infinite loop, so that there is no return to the original
> program; only the exec-ed program runs, and it runs in a secondary
> thread.
>
> The puzzle is that if there is any later import statement in the exec
> source, the exec program halts on that import statement, with no error
> message. I saw a discussion that suggested a need for the statement
> "global math" to make the math import work, but that doesn't fix the
> problem. I've tried with no success various versions of the exec
> statement, with respect to its global and local environment.

In a recent discussion in this list someone mentioned that
on module import, you should not start a thread. The reason: apparently,
Python uses some kind of locking during import which can interfere
with "import"s in the started thread.

You can (in principle) easily avoid starting the thread on module import.
Instead of starting the thread as a side effect of the import,
put the start in a function, import the module and then call
the thread starting function.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Encapsulation, inheritance and polymorphism

[Lipska the Kat]

On 19/07/12 07:09, rusi wrote:

On Jul 19, 6:34 am, Steven D'Aprano  wrote:

On Wed, 18 Jul 2012 15:40:00 +0100, Lipska the Kat wrote:

Object Oriented programming is all about encapsulating human concepts in
a way that makes sense to human beings. Make no mistake, it is NEVER the
case that a software system is written for any other reason than to
serve human beings. OO is more than just the mechanics of writing code,
it's a state of mind.


Um, yes?



Its not so much a question of language as in programming as language
as in layman-speak.
One characteristic with our field is that we take ordinary words and
then distort them so much the original meaning is completely lost.

Take 'computer' for example.  For Turing a computer was a
mathematician doing a computation with a pen and paper.  He then
showed how to de-skill the mathematician so much that a machine could
do what he was doing.  In trying that he also hit upon the limits of
such 'de-skilling' -- human-computers routinely detect infinite loops,
whereas machine-computers can never do so (in full generality).

Ironically the important lesson is lost and today 'computer' is
synonymous with machine.

Here is Dijkstra on similar distortions with 'user' and
'intelligent':
http://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD618.html

'Object' (and OO) are similar messes.


Well this is all very interesting.

The terminology is quite concise if you allow it to be
Take "An Object is an instance of a Class"

A Class describes a human concept (be it concrete like a 'Parrot' or 
more abstract like a 'Session') an Object is an actual representation of 
that concept that exists in the memory of a computer (what else should 
we call it). Objects don't exist in the mind of a human, concepts do. A 
class is a way of representing that concept so that other humans can 
understand it. That's it, really, there is no more, anything else is 
implementation.



In layman-speak and object is well, a thing.


But we are not talking in 'layman-speak' we are discussing concepts that 
are familiar to us in the 'language of the domain' at least I though we 
were. Academic twiddling with the distorted meaning of words spun by 
vested interests is all very interesting I'm sure but doesn't really 
advance the discussion does it?



- subject to space-laws like can only exist at one place at a time,
there cannot be two objects at the same place and time etc.
- subject to time-laws like coming into existence at some time and
going out at some other
- connotes inanimateness unlike other 'creatures' or 'beings'.


Well here I have to agree with you. Anyone who invents a 'Person' Class 
should be dismissed forthwith. Parrots are OK though.



When this metaphor works (for example as in guis and simulation) then
we have success-stories like smalltalk and simula.

When it doesn't the success is poorer. eg a programmer writing math
software in/on a OO system may for example 'clone' a matrix.  This may
be good science-fiction; its bad math.

And one of the most pervasive (and stupidist) metaphors is the parent-
child relation of classes.
Just for the record, in the normal world 'creatures/beings' reproduce
and therefore inherit.


But we are not talking about the 'real world' are we, we are talking 
about representing complex interacting human concepts in a way that can 
be understood by humans and translated into a form that can be executed 
on a binary machine


> Objects dont.

Well they do, it's a fact, you can call a method on a sub class that 
only exists in the super class. What else would you call it.


Well it's been fun but I have bills to pay so I suppose I'd better do 
some work. TTFN



--
Lipska the Kat: Troll hunter, Sandbox destroyer
and Farscape dreamer of Aeryn Sun.
--
http://mail.python.org/mailman/listinfo/python-list

Incorrect detection of futimesns/futimes in 3.3.0b1

[RICHARD MOSELEY]

I have recently been compiling the source for 3.3.0 beta1 and have
discovered that even though the functions futimens, futimes and lutimes are
found by the configure script, since they exist as entry points in libc,
their actually only stub functions that simply return ENOSYS (Not
Implemented) when called.

This causes the installation stage to fail with a fatal error, 'Error:
Function Not Implemented', during the installation of the libraries into
the lib-dynload directory, leading me to initially think that there was a
problem with the distutils module (fairly unlikely I know).

The platform I am compiling on, is an ASUS Eee PC 1000 running the original
Xandros operating system.

I can make the installation complete, by modifying the pyconfig.h and
undefining the erroneous defines: HAVE_FUNC_FUTIMENS, HAVE_FUNC_FUTIMES and
HAVE_FUNC_LUTIMES.

I am now considering providing a general patch to the configure.ac file
which will more correctly detect all the various flavours of utimes
(futimens, futimes, lutimes, futimesat, utimensat and utimes) using a
different check other than AC_CHECK_FUNCS. Or would it be better simply to
keep it private and apply the fix each time I upgrade the source code to
the latest version?

Richard Moseley
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Incorrect detection of futimesns/futimes in 3.3.0b1

[Christian Heimes]

Am 19.07.2012 11:03, schrieb RICHARD MOSELEY:
> I am now considering providing a general patch to the configure.ac
>  file which will more correctly detect all the
> various flavours of utimes (futimens, futimes, lutimes, futimesat,
> utimensat and utimes) using a different check other than AC_CHECK_FUNCS.
> Or would it be better simply to keep it private and apply the fix each
> time I upgrade the source code to the latest version?

Please create a bug report http://bugs.python.org/ and attach your patch
when it's ready. Other people will find your fix useful, too. The issue
should be fixed in the official distribution.

Thanks
Christian

-- 
http://mail.python.org/mailman/listinfo/python-list

reloading code and multiprocessing

[andrea crotti]

We need to be able to reload code on a live system.  This live system
has a daemon process always running but it runs many subprocesses with
multiprocessing, and the subprocesses might have a short life...

Now I found a way to reload the code successfully, as you can see from
this testcase:


def func():
from . import a
print(a.ret())


class TestMultiProc(unittest.TestCase):
def setUp(self):
open(path.join(cur_dir, 'a.py'), 'w').write(old_a)

def tearDown(self):
remove(path.join(cur_dir, 'a.py'))

def test_reloading(self):
"""Starting a new process gives a different result
"""
p1 = Process(target=func)
p2 = Process(target=func)
p1.start()
res = p1.join()
open(path.join(cur_dir, 'a.py'), 'w').write(new_a)
remove(path.join(cur_dir, 'a.pyc'))

p2.start()
res = p2.join()


As long as I import the code in the function and make sure to remove the
"pyc" files everything seems to work..
Are there any possible problems which I'm not seeing in this approach or
it's safe?

Any other better ways otherwise?
-- 
http://mail.python.org/mailman/listinfo/python-list

Odd csv column-name truncation with only one column

[Tim Chase]

tim@laptop:~/tmp$ python
Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import csv
>>> from cStringIO import StringIO
>>> s = StringIO('Email\n...@example.com\n...@example.org\n')
>>> s.seek(0)
>>> d = csv.Sniffer().sniff(s.read())
>>> s.seek(0)
>>> r = csv.DictReader(s, dialect=d)
>>> r.fieldnames
['Emai', '']

I get the same results using Python 3.1.3 (also readily available on
Debian Stable), as well as working directly on a file rather than a
StringIO.

Any reason I'm getting ['Emai', ''] (note the missing ell) instead
of ['Email'] as my resulting fieldnames?  Did I miss something in
the docs?

-tkc






-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Odd csv column-name truncation with only one column

[Peter Otten]

Tim Chase wrote:

> tim@laptop:~/tmp$ python
> Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
> [GCC 4.4.5] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
 import csv
 from cStringIO import StringIO
 s = StringIO('Email\n...@example.com\n...@example.org\n')
 s.seek(0)
 d = csv.Sniffer().sniff(s.read())
 s.seek(0)
 r = csv.DictReader(s, dialect=d)
 r.fieldnames
> ['Emai', '']
> 
> I get the same results using Python 3.1.3 (also readily available on
> Debian Stable), as well as working directly on a file rather than a
> StringIO.
> 
> Any reason I'm getting ['Emai', ''] (note the missing ell) instead
> of ['Email'] as my resulting fieldnames?  Did I miss something in
> the docs?

Judging from 

>>> import csv
>>> sniffer = csv.Sniffer()
>>> sniffer.sniff("abc").delimiter
'c'
>>> sniffer.sniff("abc\naba").delimiter
'b'
>>> sniffer.sniff("abc\naba\nxyz").delimiter
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.7/csv.py", line 184, in sniff
raise Error, "Could not determine delimiter"
_csv.Error: Could not determine delimiter
>>> sniffer.sniff("abc\n"*10 + "xyz").delimiter
'c'
>>> sniffer.sniff("abc\n"*9 + "xyz").delimiter
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.7/csv.py", line 184, in sniff
raise Error, "Could not determine delimiter"
_csv.Error: Could not determine delimiter

the Sniffer heuristics determines a character that occurs on all of the 
first 10 lines to be the delimiter. There are of course examples where that 
doesn't make sense to a human observer...


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Odd csv column-name truncation with only one column

[Steven D'Aprano]

On Thu, 19 Jul 2012 06:21:58 -0500, Tim Chase wrote:

> tim@laptop:~/tmp$ python
> Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48) [GCC 4.4.5] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
 import csv
 from cStringIO import StringIO
 s = StringIO('Email\n...@example.com\n...@example.org\n') s.seek(0)
 d = csv.Sniffer().sniff(s.read())
 s.seek(0)
 r = csv.DictReader(s, dialect=d)
 r.fieldnames
> ['Emai', '']


I get the same results for Python 2.6 and 2.7. Curiously, 2.5 returns 
fieldnames as None.

I'm not entirely sure that a single column is legitimate for CSV -- if 
there's only one column, it is hardly comma-separated, or any other 
separated for that matter. But perhaps the csv module should raise an 
exception in that case.

I think you've found a weird corner case where the sniffer goes nuts. You 
should probably report it as a bug:

py> s = StringIO('Email\n...@example.com\n...@example.org\n')
py> s.seek(0)
py> d = csv.Sniffer().sniff(s.read())
py> d.delimiter
'l'

py> s = StringIO('Spam\n...@example.com\n...@example.org\n')
py> s.seek(0)
py> d = csv.Sniffer().sniff(s.read())
py> d.delimiter
'p'

py> s = StringIO('Spam\nham\ncheese\n')
py> s.seek(0)
py> d = csv.Sniffer().sniff(s.read())
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python2.7/csv.py", line 184, in sniff
raise Error, "Could not determine delimiter"
_csv.Error: Could not determine delimiter


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Encapsulation, inheritance and polymorphism

[Steven D'Aprano]

On Wed, 18 Jul 2012 23:09:13 -0700, rusi wrote:

> Its not so much a question of language as in programming as language as
> in layman-speak.
> One characteristic with our field is that we take ordinary words and
> then distort them so much the original meaning is completely lost.

All technical fields have jargon. Those jargon terms are often more 
precise than the ordinary terms they are derived from, or have a slightly 
different meaning, or both.

This is not unique to programming, nor is it anything to be disturbed by. 
Words change. What matters is whether the new meanings cause confusion or 
clarity.


> Take 'computer' for example.  For Turing a computer was a mathematician
> doing a computation with a pen and paper.  He then showed how to
> de-skill the mathematician so much that a machine could do what he was
> doing.  In trying that he also hit upon the limits of such 'de-skilling'
> -- human-computers routinely detect infinite loops, whereas
> machine-computers can never do so (in full generality).

Do they really? I doubt that. Human-computers *sometimes* detect infinite 
loops, but there's no evidence that they can detect ∞-loops in full 
generality, or even that they are better at it than electrical computers.

In fact, the sheer number of accidental ∞-loops programmed by human 
beings suggests that people *cannot* routinely detect them, at least not 
without special training, and even then not *routinely*.

Generally people detect ∞-loops with an extremely simple-minded 
heuristic: "if the loop hasn't finished by now, it will never finish".

The fact that we don't, as a general rule, program our computers to 
detect ∞-loops does not mean that they cannot do so at least as well as 
humans, and probably better, when we bother to program them to.

For example, both ML and Haskell can, under some circumstances, report a 
type-error for an infinite loop, *at compile time*.

http://static.usenix.org/publications/library/proceedings/vhll/full_papers/koenig.a

If you think that people can routinely detect infinite loops, then 
perhaps you would care to tell me whether this is an infinite loop or not:

i = 1
while not is_perfect(i):
i += 2
print "odd perfect number discovered"


where is_perfect() returns True if the integer argument is perfect, and 
False otherwise.

http://mathworld.wolfram.com/PerfectNumber.html
http://mathworld.wolfram.com/OddPerfectNumber.html


 
[...]
> 'Object' (and OO) are similar messes.
> 
> In layman-speak and object is well, a thing.

Define "thing".

Is a cloud a thing? What about a fog bank? An ocean?

A desert? The atmosphere?

Is the paint on your car a thing?

I have an axe that has been passed down for generations through my 
family, from my father, his father before him, and his father, and his 
father before him. Occasionally we replace the handle, or put on a new 
head, but that axe is almost as good as the day my great-great-
grandfather made it.

Is that axe a thing?

Just because laymen toss around a word, doesn't mean that it is a well-
defined, meaningful word.


> When it doesn't the success is poorer. eg a programmer writing math
> software in/on a OO system may for example 'clone' a matrix.  This may
> be good science-fiction; its bad math.

In what way is it bad maths?



> And one of the most pervasive (and stupidist) metaphors is the parent-
> child relation of classes.
> Just for the record, in the normal world 'creatures/beings' reproduce
> and therefore inherit.

Incorrect. In the normal world, "inherit" is used for at least four 
senses:

1) to inherit wealth, property, a title, debts etc. from an ancestor 
   upon their death;

2) to receive or take by birth, to have by nature, physical or mental
   qualities, e.g. "he inherited his tendency to melancholy from his
   father";

3) to come into possession of, e.g. "the meek shall inherit the earth";

4) to receive from a predecessor, e.g. "the Prime Minister has inherited
   an economy in dire straits".

It is clear that the sense of inheritance used in OO programming is sense 
#2, to have by nature.


> Objects dont.

Irrelevant.


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Odd csv column-name truncation with only one column

[Tim Chase]

On 07/19/12 06:21, Tim Chase wrote:
> tim@laptop:~/tmp$ python
> Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
> [GCC 4.4.5] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
 import csv
 from cStringIO import StringIO
 s = StringIO('Email\n...@example.com\n...@example.org\n')
 s.seek(0)
 d = csv.Sniffer().sniff(s.read())
 s.seek(0)
 r = csv.DictReader(s, dialect=d)
 r.fieldnames
> ['Emai', '']

I think I may have stumbled across the "what the heck is happening"
factor:

>>> import csv
>>> from cStringIO import StringIO
>>> s = StringIO('Email\n...@example.org\n...@test.test\n')
>>> d = csv.Sniffer().sniff(s.read())
>>> s.seek(0)
>>> r = csv.DictReader(s, dialect=d)
>>> r.fieldnames
['Em', 'il']

It appears that it's finding something repeated [ed: Peter's &
Steven's replies came in as I finished typing this].  In my first,
it was the "l" appearing on each line, and in the 2nd example here,
it's the "a" on each line, so the csv module thinks that's the
delimiter.  The source file comes from an Excel-dialect generation:

>>> s = StringIO()
>>> w = csv.writer(s)
>>> w.writerows([["email"], ["f...@example.com"], ["b...@example.org"]])
>>> s.seek(0)
>>> d = csv.Sniffer().sniff(s.read())
>>> d.delimiter
'l'
>>> s.seek(0)
>>> r = csv.DictReader(s, dialect=d)
>>> r.fieldnames
['emai', '']


I guess it then takes the Python community to make the call on
whether the csv module is doing the right thing in the degenerate
case.  I.e., you can't get back out what you put in when you try to
sniff.

-tkc




-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Encapsulation, inheritance and polymorphism

[Roy Smith]

In article <500804cc$0$29978$c3e8da3$54964...@news.astraweb.com>,
 Steven D'Aprano  wrote:

> On Wed, 18 Jul 2012 23:09:13 -0700, rusi wrote:
> 
> > Its not so much a question of language as in programming as language as
> > in layman-speak.
> > One characteristic with our field is that we take ordinary words and
> > then distort them so much the original meaning is completely lost.
> 
> All technical fields have jargon. Those jargon terms are often more 
> precise than the ordinary terms they are derived from, or have a slightly 
> different meaning, or both.

Heh.  This reminds me of one of my current pet peeves.  I've run across 
documentation for more than one Python project (django is the one that 
comes to mind, but I'm sure there's others) which misuse words like 
"set" and "list".  They're often used in the generic sense of "a 
collection of things", but they're also the names of specific Python 
data types.  It can sometimes get confusing trying to figure out which 
meaning they intended.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Programming expert - adding user

[William R. Wing (Bill Wing)]

On Jul 17, 2012, at 9:58 PM, Maria Hanna Carmela Dionisio wrote:

> Im just a student :)
> 
> Our prof gave as a task that we need to make a program using python (for 
> redhat) and c++(for windows)
> 
> Our objective is to make a program file and we will said it remotely to 
> another computer via network ( its easy and i could do it lolz)..
> 
> the hard part is the programming..that when the user click the file a box 
> will prompt and ask for username and a password(even without a password) then 
> the username that he inputted will add to the user account of his/her 
> computer :DD
> 
> HELP HELP EXPERTSS
> 
> THANKS :D
> -- 
> http://mail.python.org/mailman/listinfo/python-list

I'm sorry if I'm not understanding exactly what your professor wants, I think 
he is asking you to add a new user to the list of possible users of the 
computer - that is, add a new user account.  If so, then the answer is pretty 
simple, since Linux (Red Hat) has a "useradd" command that you just have to 
supply inputs to.  Your python program will start by importing "subprocess" and 
then using it to spawn a child process in which the useradd command will run.

Read the subprocess documentation here:


http://docs.python.org/library/subprocess.html?highlight=subprocess#subprocess

and see if this, plus the man page for useradd, doesn't get you going.

-Bill
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Odd csv column-name truncation with only one column

[Hans Mulder]

On 19/07/12 13:21:58, Tim Chase wrote:
> tim@laptop:~/tmp$ python
> Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
> [GCC 4.4.5] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
 import csv
 from cStringIO import StringIO
 s = StringIO('Email\n...@example.com\n...@example.org\n')
 s.seek(0)
 d = csv.Sniffer().sniff(s.read())
 s.seek(0)
 r = csv.DictReader(s, dialect=d)
 r.fieldnames
> ['Emai', '']
> 
> I get the same results using Python 3.1.3 (also readily available on
> Debian Stable), as well as working directly on a file rather than a
> StringIO.
> 
> Any reason I'm getting ['Emai', ''] (note the missing ell) instead
> of ['Email'] as my resulting fieldnames?  Did I miss something in
> the docs?

The sniffer tries to guess the column separator.  If none of the
usual suspects seems to work, it tries to find a character that
occurs with the same frequency in every row.  In your sample,
the letter 'l' occurs exactly once on each line, so it is the
most plausible separator, or so the Sniffer thinks.

Perhaps it should be documented that the Sniffer doesn't work
on single-column data.

If you really need to read a one-column csv file, you'll have
to find some other way to produce a Dialect object.  Perhaps the
predefined 'cvs.excel' dialect matches your data.  If not, the
easiest way might be to manually define a csv.Dialect subclass.

Hope this helps,

-- HansM

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Google the video "9/11 Missing Links". 9/11 was a Jew Job!

[Neal Becker]

Google the video "Go fuck yourself"

-- 
http://mail.python.org/mailman/listinfo/python-list

Use of compile flags in regular expressions.

[Steven W. Orr]

I have a problem that I'm solving using a regex. (Yeah, I know, now I have two 
problems...) ;-)>


Anyways, the regex is about a couple of pages long and it works just peachy. 
There's just one thing I'd like to do to make it more elegant.


I need to compile the regex with MULTILINE and DOTALL. But there are a few 
sections where I wish it was *not* compiled with DOTALL. For those cases, I 
use (something like)


[^\n]*

instead of

.*

I see that I can use the (?MS) construct but I also see that it applies 
globally and not to the subgroup that I'm using it in.


* Is there a way to make it apply locally to a subgroup?
* If not, is there another way?
* Also, is this an incredible stroke of genius that I came up with this idea 
of applying flags to a subgroup, or have others thought of this too and found 
out that it's not really a good idea?


TIA

--
Time flies like the wind. Fruit flies like a banana. Stranger things have  .0.
happened but none stranger than this. Does your driver's license say Organ ..0
Donor?Black holes are where God divided by zero. Listen to me! We are all- 000
individuals! What if this weren't a hypothetical question?
steveo at syslang.net
--
http://mail.python.org/mailman/listinfo/python-list

Re: Google the video "9/11 Missing Links". 9/11 was a Jew Job!

[Matty Sarro]

I must be a Jew or a traitor as I keep deleting this email. Seriously
guys, don't reply to it, it's not worth the time.

On Thu, Jul 19, 2012 at 10:22 AM, Neal Becker  wrote:
> Google the video "Go fuck yourself"
>
> --
> http://mail.python.org/mailman/listinfo/python-list
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: reloading code and multiprocessing

[88888 Dihedral]

andrea crotti於 2012年7月19日星期四UTC+8下午6時15分11秒寫道：
> We need to be able to reload code on a live system.  This live system
> has a daemon process always running but it runs many subprocesses with
> multiprocessing, and the subprocesses might have a short life...
> 
> Now I found a way to reload the code successfully, as you can see from
> this testcase:
> 
> 
> def func():
> from . import a
> print(a.ret())
> 
> 
> class TestMultiProc(unittest.TestCase):
> def setUp(self):
> open(path.join(cur_dir, 'a.py'), 'w').write(old_a)
> 
> def tearDown(self):
> remove(path.join(cur_dir, 'a.py'))
> 
> def test_reloading(self):
> """Starting a new process gives a different result
> """
> p1 = Process(target=func)
> p2 = Process(target=func)
> p1.start()
> res = p1.join()
> open(path.join(cur_dir, 'a.py'), 'w').write(new_a)
> remove(path.join(cur_dir, 'a.pyc'))
> 
> p2.start()
> res = p2.join()
> 
> 
> As long as I import the code in the function and make sure to remove the
> "pyc" files everything seems to work..
> Are there any possible problems which I'm not seeing in this approach or
> it's safe?
> 
> Any other better ways otherwise?

If a byte code interpreter is embedded in the executable, then the program 
can obtain or reload code objects in the run time.

In C/C++, unless one can swap some DLL safely or spawn in another process with 
new executables obtained from other places, or a script interpreter is 
embedded, otherwise it's not easy for an instance in C/C++ to get new methods 
dynamically.

or 

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Use of compile flags in regular expressions.

[MRAB]

On 19/07/2012 15:22, Steven W. Orr wrote:

I have a problem that I'm solving using a regex. (Yeah, I know, now I have two
problems...) ;-)>

Anyways, the regex is about a couple of pages long and it works just peachy.
There's just one thing I'd like to do to make it more elegant.

I need to compile the regex with MULTILINE and DOTALL. But there are a few
sections where I wish it was *not* compiled with DOTALL. For those cases, I
use (something like)

[^\n]*

instead of

.*

I see that I can use the (?MS) construct but I also see that it applies
globally and not to the subgroup that I'm using it in.

* Is there a way to make it apply locally to a subgroup?


It with the re module.


* If not, is there another way?


The DOTALL . could be replaced with, say, [\d\D].


* Also, is this an incredible stroke of genius that I came up with this idea
of applying flags to a subgroup, or have others thought of this too and found
out that it's not really a good idea?


Many other regex implementations do support scoped flags, including the
one here:

http://pypi.python.org/pypi/regex

which is designed to be backwards-compatible with the re module.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Encapsulation, inheritance and polymorphism

[rusi]

On Jul 19, 1:56 pm, Lipska the Kat  wrote:
> Academic twiddling with the distorted meaning of words spun by
> vested interests is all very interesting I'm sure but doesn't really
> advance the discussion does it?

Well lets back up the discussion a bit. You coming from a Java
background have one view of what OO means. Others coming from a python
background have a different view. In particular you said:

> Python looks like an interesting language and I will certainly spend
> time getting to know it but at the moment it seems to me that calling it
> an Object Oriented language is just plain misleading.


> On 19/07/12 07:09, rusi wrote:
> > In layman-speak and object is well, a thing.
>
> But we are not talking in 'layman-speak' we are discussing concepts that
> are familiar to us in the 'language of the domain' at least I though we
> were.


> > And one of the most pervasive (and stupidist) metaphors is the parent-
> > child relation of classes.
> > Just for the record, in the normal world 'creatures/beings' reproduce
> > and therefore inherit.
>
> But we are not talking about the 'real world' are we, we are talking
> about representing complex interacting human concepts in a way that can
> be understood by humans and translated into a form that can be executed
> on a binary machine

So when multiple technical understandings are in conflict it seems
reasonable to find a frame in which the different viewpoints could
resolve.  The ultimate such frame is the completely de-jargonized
frame ie laymanspeak.  [How far one can/should go toward that ultimate
is another quesion]


-- 
http://mail.python.org/mailman/listinfo/python-list

shutil ignore fails on passing a tuple?

[Alex van der Spek]

This beats me:


ipatterns

('*.txt', '*.hdf', '*.pdf', '*.png')

igf = shutil.ignore_patterns(ipatterns)
ignorethis = igf(ddftopdir,os.listdir(ddftopdir))


Traceback (most recent call last):
 File "", line 1, in 
   ignorethis = igf(ddftopdir,os.listdir(ddftopdir))
 File "C:\Python27\lib\shutil.py", line 138, in _ignore_patterns
   ignored_names.extend(fnmatch.filter(names, pattern))
 File "C:\Python27\lib\fnmatch.py", line 49, in filter
   pat=os.path.normcase(pat)
 File "C:\Python27\lib\ntpath.py", line 46, in normcase
   return s.replace("/", "\\").lower()
AttributeError: 'tuple' object has no attribute 'replace'


igg = shutil.ignore_patterns('*.txt', '*.hdf', '*.pdf', '*.png')
ignorethat = igg(ddftopdir, os.listdir(ddftopdir))
ignorethat
set(['Chi2.png', 'DTSdata.hdf', 'TST.hdf', 'BullNoseDiffs.png', 
'DTSall.hdf', 'Symmetry.pdf'])




Why does it fail on passing in a tuple of ignore strings? I thought the , 
(comma) is pretty much the tuple constructor (if that is the right word).


How can I solve this? Is there a way to convert a tuple of strings in a form 
that will be accepted?


Thank you in advance,
Alex van der Spek


--
http://mail.python.org/mailman/listinfo/python-list

RE: shutil ignore fails on passing a tuple?

[Prasad, Ramit]

> >>> ipatterns
> ('*.txt', '*.hdf', '*.pdf', '*.png')
> >>> igf = shutil.ignore_patterns(ipatterns)
> >>> ignorethis = igf(ddftopdir,os.listdir(ddftopdir))
> 
> Traceback (most recent call last):
>   File "", line 1, in 
> ignorethis = igf(ddftopdir,os.listdir(ddftopdir))
>   File "C:\Python27\lib\shutil.py", line 138, in _ignore_patterns
> ignored_names.extend(fnmatch.filter(names, pattern))
>   File "C:\Python27\lib\fnmatch.py", line 49, in filter
> pat=os.path.normcase(pat)
>   File "C:\Python27\lib\ntpath.py", line 46, in normcase
> return s.replace("/", "\\").lower()
> AttributeError: 'tuple' object has no attribute 'replace'
> 
> >>> igg = shutil.ignore_patterns('*.txt', '*.hdf', '*.pdf', '*.png')
> >>> ignorethat = igg(ddftopdir, os.listdir(ddftopdir))
> >>> ignorethat
> set(['Chi2.png', 'DTSdata.hdf', 'TST.hdf', 'BullNoseDiffs.png',
> 'DTSall.hdf', 'Symmetry.pdf'])
> >>>
> 
> Why does it fail on passing in a tuple of ignore strings? I thought the ,
> (comma) is pretty much the tuple constructor (if that is the right word).
> 
> How can I solve this? Is there a way to convert a tuple of strings in a form
> that will be accepted?
>

Untested, but I think it should work.
igf = shutil.ignore_patterns(*ipatterns)

Ramit

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: shutil ignore fails on passing a tuple?

[Joel Goldstick]

On Thu, Jul 19, 2012 at 12:43 PM, Alex van der Spek  wrote:
> This beats me:
> 

 ipatterns
>
> ('*.txt', '*.hdf', '*.pdf', '*.png')

 igf = shutil.ignore_patterns(ipatterns)
 ignorethis = igf(ddftopdir,os.listdir(ddftopdir))
>
>
> Traceback (most recent call last):
>  File "", line 1, in 
>ignorethis = igf(ddftopdir,os.listdir(ddftopdir))
>  File "C:\Python27\lib\shutil.py", line 138, in _ignore_patterns
>ignored_names.extend(fnmatch.filter(names, pattern))
>  File "C:\Python27\lib\fnmatch.py", line 49, in filter
>pat=os.path.normcase(pat)
>  File "C:\Python27\lib\ntpath.py", line 46, in normcase
>return s.replace("/", "\\").lower()
> AttributeError: 'tuple' object has no attribute 'replace'

What is s?  It needs to be a string.  If s is a tuple with string
values, you will have to iterate over each to replace.  Or I believe
you could use map.
>
 igg = shutil.ignore_patterns('*.txt', '*.hdf', '*.pdf', '*.png')
 ignorethat = igg(ddftopdir, os.listdir(ddftopdir))
 ignorethat
>
> set(['Chi2.png', 'DTSdata.hdf', 'TST.hdf', 'BullNoseDiffs.png',
> 'DTSall.hdf', 'Symmetry.pdf'])


> 
> Why does it fail on passing in a tuple of ignore strings? I thought the ,
> (comma) is pretty much the tuple constructor (if that is the right word).
>
> How can I solve this? Is there a way to convert a tuple of strings in a form
> that will be accepted?
>
> Thank you in advance,
> Alex van der Spek
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list



-- 
Joel Goldstick
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Odd csv column-name truncation with only one column

[Tim Chase]

On 07/19/12 08:52, Hans Mulder wrote:
> Perhaps it should be documented that the Sniffer doesn't work
> on single-column data.

I think this would  involve the least change in existing code, and
go a long way towards removing my surprise.  :-)

> If you really need to read a one-column csv file, you'll have
> to find some other way to produce a Dialect object.  Perhaps the
> predefined 'cvs.excel' dialect matches your data.  If not, the
> easiest way might be to manually define a csv.Dialect subclass.

The problem I'm trying to solve is "here's a filename that might be
comma/pipe/tab delimited, it has an 'email' column at minimum, and
perhaps a couple others of interest if they were included"  It's
improbable that it's ONLY an email column, but my tests happened to
snag this edge case.  I can likely do my own sniffing by reading the
first line, checking for tabs then pipes then commas (perhaps
biasing the order based on the file-extension of .csv vs. .txt), and
then building my own dialect information to pass to csv.DictReader
 It just seems unfortunate that the sniffer would ever consider
[a-zA-Z0-9] as a valid delimiter.

-tkc


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Encapsulation, inheritance and polymorphism

[Paul Rudin]

Steven D'Aprano  writes:

> For example, both ML and Haskell can, under some circumstances, report a 
> type-error for an infinite loop, *at compile time*.

... and in Charity all programs are guaranteed to terminate. Of course
it's not Turing complete.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Encapsulation, inheritance and polymorphism

[Tim Chase]

> If you think that people can routinely detect infinite loops, then 
> perhaps you would care to tell me whether this is an infinite loop or not:
> 
> i = 1
> while not is_perfect(i):
> i += 2
> print "odd perfect number discovered"
> 
> 
> where is_perfect() returns True if the integer argument is perfect, and 
> False otherwise.

Sure it terminates...If you don't run out of RAM to represent the
number "i" in question, there's also this "heat death of the
universe" limit I keep hearing about ;-)

-tkc


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Encapsulation, inheritance and polymorphism

[Chris Angelico]

On Fri, Jul 20, 2012 at 4:20 AM, Tim Chase
 wrote:
> Sure it terminates...If you don't run out of RAM to represent the
> number "i" in question, there's also this "heat death of the
> universe" limit I keep hearing about ;-)

I'd be more worried about the heat death of your computer, it's likely
to be sooner. How many people have access to a computer that'll still
be running in ten years, much less a thousand? And while a thousand
years is extremely large, it still falls pitifully short of
infinity[1], or even the heat death of the universe.

ChrisA
[1] http://tools.ietf.org/html/rfc2795
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Encapsulation, inheritance and polymorphism

[Tim Chase]

On 07/19/12 13:28, Chris Angelico wrote:
> On Fri, Jul 20, 2012 at 4:20 AM, Tim Chase
>  wrote:
>> Sure it terminates...If you don't run out of RAM to represent the
>> number "i" in question, there's also this "heat death of the
>> universe" limit I keep hearing about ;-)
> 
> I'd be more worried about the heat death of your computer, it's likely
> to be sooner. How many people have access to a computer that'll still
> be running in ten years, much less a thousand?

Just putting a maximum bound on the problem, providing a time-frame
in which I can be fairly certain that the program will have
terminated. :-)

-tkc


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Finding duplicate file names and modifying them based on elements of the path

[larry.mart...@gmail.com]

On Jul 18, 6:36 pm, Simon Cropper
 wrote:
> On 19/07/12 08:20, larry.mart...@gmail.com wrote:
>
>
>
>
>
>
>
>
>
> > I have an interesting problem I'm trying to solve. I have a solution
> > almost working, but it's super ugly, and know there has to be a
> > better, cleaner way to do it.
>
> > I have a list of path names that have this form:
>
> > /dir0/dir1/dir2/dir3/dir4/dir5/dir6/file
>
> > I need to find all the file names (basenames) in the list that are
> > duplicates, and for each one that is a dup, prepend dir4 to the
> > filename as long as the dir4/file pair is unique. If there are
> > multiple dir4/files in the list, then I also need to add a sequence
> > number based on the sorted value of dir5 (which is a date in ddMONyy
> > format).
>
> > For example, if my list contains:
>
> > /dir0/dir1/dir2/dir3/qwer/09Jan12/dir6/file3
> > /dir0/dir1/dir2/dir3/abcd/08Jan12/dir6/file1
> > /dir0/dir1/dir2/dir3/abcd/08Jan12/dir6/file2
> > /dir0/dir1/dir2/dir3/xyz/08Jan12/dir6/file1
> > /dir0/dir1/dir2/dir3/qwer/07Jan12/dir6/file3
>
> > Then I want to end up with:
>
> > /dir0/dir1/dir2/dir3/qwer/09Jan12/dir6/qwer_01_file3
> > /dir0/dir1/dir2/dir3/abcd/08Jan12/dir6/abcd_file1
> > /dir0/dir1/dir2/dir3/abcd/08Jan12/dir6/file2
> > /dir0/dir1/dir2/dir3/xyz/08Jan12/dir6/xyz_file1
> > /dir0/dir1/dir2/dir3/qwer/07Jan12/dir6/qwer_00_file3
>
> > My solution involves multiple maps and multiple iterations through the
> > data. How would you folks do this?
>
> Hi Larry,
>
> I am making the assumption that you intend to collapse the directory
> tree and store each file in the same directory, otherwise I can't think
> of why you need to do this.

Hi Simon, thanks for the reply. It's not quite this - what I am doing
is creating a zip file with relative path names, and if there are
duplicate files the parts of the path that are not be carried over
need to get prepended to the file names to make then unique,
>
> If this is the case, then I would...
>
> 1. import all the files into an array
> 2. parse path to extract forth level directory name and base name.
> 3. reiterate through the array
>     3.1 check if base filename exists in recipient directory
>     3.2 if not, copy to recipient directory
>     3.3 if present, append the directory path then save
>     3.4 create log of success or failure
>
> Personally, I would not have some files with abcd_file1 and others as
> file2 because if it is important enough to store a file in a separate
> directory you should also note where file2 came from as well. When
> looking at your results at a later date you are going to have to open
> file2 (which I presume must record where it relates to) to figure out
> where it came from. If it is in the name it is easier to review.
>
> In short, consistency is the name of the game; if you are going to do it
> for some then do it for all; and finally it will be easier for others
> later to work out what you have done.

Yeah, I know, but this is for a client, and this is what they want.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Finding duplicate file names and modifying them based on elements of the path

[larry.mart...@gmail.com]

On Jul 18, 4:49 pm, Paul Rubin  wrote:
> "larry.mart...@gmail.com"  writes:
> > I have an interesting problem I'm trying to solve. I have a solution
> > almost working, but it's super ugly, and know there has to be a
> > better, cleaner way to do it. ...
>
> > My solution involves multiple maps and multiple iterations through the
> > data. How would you folks do this?
>
> You could post your code and ask for suggestions how to improve it.
> There are a lot of not-so-natural constraints in that problem, so it
> stands to reason that the code will be a bit messy.  The whole
> specification seems like an antipattern though.  You should just give a
> sensible encoding for the filename regardless of whether other fields
> are duplicated or not.  You also don't seem to address the case where
> basename, dir4, and dir5 are all duplicated.
>
> The approach I'd take for the spec as you wrote it is:
>
> 1. Sort the list on the (basename, dir4, dir5) triple, saving original
>    location (numeric index) of each item
> 2. Use itertools.groupby to group together duplicate basenames.
> 3. Within the groups, use groupby again to gather duplicate dir4's,
> 4. Within -those- groups, group by dir5 and assign sequence numbers in
>    groups where there's more than one file
> 5. Unsort to get the rewritten items back into the original order.
>
> Actual code is left as an exercise.

I replied to this before, but I don't see, so if this is a duplicate,
sorry.

Thanks for the reply Paul. I had not heard of itertools. It sounds
like just what I need for this. But I am having 1 issue - how do you
know how many items are in each group? Without knowing that I have to
either make 2 passes through the data, or else work on the previous
item (when I'm in an iteration after the first then I know I have
dups). But that very quickly gets crazy with trying to keep the
previous values.
-- 
http://mail.python.org/mailman/listinfo/python-list

RE: Finding duplicate file names and modifying them based on elements of the path

[Prasad, Ramit]

> > I am making the assumption that you intend to collapse the directory
> > tree and store each file in the same directory, otherwise I can't think
> > of why you need to do this.
> 
> Hi Simon, thanks for the reply. It's not quite this - what I am doing
> is creating a zip file with relative path names, and if there are
> duplicate files the parts of the path that are not be carried over
> need to get prepended to the file names to make then unique,

Depending on the file system of the client, you can hit file name
length limits. I would think it would be better to just create
the full structure in the zip.

Just something to keep in mind, especially if you see funky behavior.

Ramit

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Finding duplicate file names and modifying them based on elements of the path

[larry.mart...@gmail.com]

On Jul 19, 1:02 pm, "Prasad, Ramit"  wrote:
> > > I am making the assumption that you intend to collapse the directory
> > > tree and store each file in the same directory, otherwise I can't think
> > > of why you need to do this.
>
> > Hi Simon, thanks for the reply. It's not quite this - what I am doing
> > is creating a zip file with relative path names, and if there are
> > duplicate files the parts of the path that are not be carried over
> > need to get prepended to the file names to make then unique,
>
> Depending on the file system of the client, you can hit file name
> length limits. I would think it would be better to just create
> the full structure in the zip.
>
> Just something to keep in mind, especially if you see funky behavior.

Thanks, but it's not what the client wants.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: shutil ignore fails on passing a tuple?

[Dave Angel]

On 07/19/2012 12:43 PM, Alex van der Spek wrote:
> This beats me:
> 
 ipatterns
> ('*.txt', '*.hdf', '*.pdf', '*.png')
 igf = shutil.ignore_patterns(ipatterns)
 ignorethis = igf(ddftopdir,os.listdir(ddftopdir))
>
> 
> Why does it fail on passing in a tuple of ignore strings? I thought
> the , (comma) is pretty much the tuple constructor (if that is the
> right word).
>

ignore_patterns() is not looking for a tuple, it's looking for one or
more strings, as separate arguments.  If you already have the tuple, you
can expand it by using the asterisk, as Prasad ha pointed out.

igf = shutil.ignore_patterns(*ipatterns)

-- 

DaveA

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Finding duplicate file names and modifying them based on elements of the path

[larry.mart...@gmail.com]

On Jul 18, 4:49 pm, Paul Rubin  wrote:
> "larry.mart...@gmail.com"  writes:
> > I have an interesting problem I'm trying to solve. I have a solution
> > almost working, but it's super ugly, and know there has to be a
> > better, cleaner way to do it. ...
>
> > My solution involves multiple maps and multiple iterations through the
> > data. How would you folks do this?
>
> You could post your code and ask for suggestions how to improve it.
> There are a lot of not-so-natural constraints in that problem, so it
> stands to reason that the code will be a bit messy.  The whole
> specification seems like an antipattern though.  You should just give a
> sensible encoding for the filename regardless of whether other fields
> are duplicated or not.  You also don't seem to address the case where
> basename, dir4, and dir5 are all duplicated.
>
> The approach I'd take for the spec as you wrote it is:
>
> 1. Sort the list on the (basename, dir4, dir5) triple, saving original
>    location (numeric index) of each item
> 2. Use itertools.groupby to group together duplicate basenames.
> 3. Within the groups, use groupby again to gather duplicate dir4's,
> 4. Within -those- groups, group by dir5 and assign sequence numbers in
>    groups where there's more than one file
> 5. Unsort to get the rewritten items back into the original order.
>
> Actual code is left as an exercise.

Thanks very much for the reply Paul. I did not know about itertools.
This seems like it will be perfect for me. But I'm having 1 issue, how
do I know how many of a given basename (and similarly how many
basename/dir4s) there are? I don't know that I have to modify a file
until I've passed it, so I have to do all kinds of contortions to save
the previous one, and deal with the last one after I fall out of the
loop, and it's getting very nasty.

reports_list is the list sorted on basename, dir4, dir5 (tool is dir4,
file_date is dir5):

for file, file_group in groupby(reports_list, lambda x: x[0]):
# if file is unique in file_group do nothing, but how can I tell
if file is unique?
for tool, tool_group in groupby(file_group, lambda x: x[1]):
# if tool is unique for file, change file to tool_file, but
how can I tell if tool is unique for file?
for file_date, file_date_group in groupby(tool_group, lambda
x: x[2]):


You can't do a len on the iterator that is returned from groupby, and
I've tried to do something with imap or  defaultdict, but I'm not
getting anywhere. I guess I can just make 2 passes through the data,
the first time getting counts. Or am I missing something about how
groupby works?

Thanks!
-larry
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Finding duplicate file names and modifying them based on elements of the path

[Paul Rubin]

"larry.mart...@gmail.com"  writes:
> Thanks for the reply Paul. I had not heard of itertools. It sounds
> like just what I need for this. But I am having 1 issue - how do you
> know how many items are in each group?

Simplest is:

  for key, group in groupby(xs, lambda x:(x[-1],x[4],x[5])):
 gs = list(group)  # convert iterator to a list
 n = len(gs)   # this is the number of elements

there is some theoretical inelegance in that it requires each group to
fit in memory, but you weren't really going to have billions of files
with the same basename.

If you're not used to iterators and itertools, note there are some
subtleties to using groupby to iterate over files, because an iterator
actually has state.  It bumps a pointer and maybe consumes some input
every time you advance it.  In a situation like the above, you've got
some nexted iterators (the groupby iterator generating groups, and the
individual group iterators that come out of the groupby) that wrap the
same file handle, so bad confusion can result if you advance both
iterators without being careful (one can consume file input that you
thought would go to another).

This isn't as bad as it sounds once you get used to it, but it can be
a source of frustration at first.  

BTW, if you just want to count the elements of an iterator (while
consuming it),

 n = sum(1 for x in xs)

counts the elements of xs without having to expand it into an in-memory
list.

Itertools really makes Python feel a lot more expressive and clean,
despite little kinks like the above.
-- 
http://mail.python.org/mailman/listinfo/python-list

properly catch SIGTERM

[Eric Frederich]

So I wrote a script which acts like a daemon.
And it starts with something like this

### Begin Code

import signal

STOPIT = False

def my_SIGTERM_handler(signum, frame):
global STOPIT
print '\n--- Caught SIGTERM; Attempting to quit gracefully ---'
STOPIT = True

signal.signal(signal.SIGTERM, my_SIGTERM_handler)
signal.signal(signal.SIGINT , my_SIGTERM_handler)

### End Code

My main loop looks something like this...

login()
while not STOPIT:
foo1()
foo2()
foo3()
if STOPIT:
break
bar1()
bar2()
bar3()

print 'bye'
logout()

This seems to work okay but just now I got this while hitting ctrl-c
It seems to have caught the signal at or in the middle of a call to
sys.stdout.flush()


--- Caught SIGTERM; Attempting to quit gracefully ---
Traceback (most recent call last):
  File "/home/user/test.py", line 125, in 
sys.stdout.flush()
IOError: [Errno 4] Interrupted system call


How should I fix this?
Am I doing this completely wrong?

Thanks,
~Eric
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Finding duplicate file names and modifying them based on elements of the path

[Paul Rubin]

"larry.mart...@gmail.com"  writes:
> You can't do a len on the iterator that is returned from groupby, and
> I've tried to do something with imap or  defaultdict, but I'm not
> getting anywhere. I guess I can just make 2 passes through the data,
> the first time getting counts. Or am I missing something about how
> groupby works?

I posted another reply to your other message, which reached me earlier.
If you're still stuck, post again, though I probably won't be able to
reply til tomorrow or the next day.
-- 
http://mail.python.org/mailman/listinfo/python-list

logging time format millisecond precision

[Alex van der Spek]

I use this formatter in logging:

formatter = logging.Formatter(fmt='%(asctime)s \t %(name)s \t %(levelname)s 
\t %(message)s')


Sample output:

2012-07-19 21:34:58,382   root   INFO   Removed - C:\Users\ZDoor\Documents

The time stamp has millisecond precision but the decimal separator is a 
comma.


Can I change the comma (,) into a period (.) and if so how?

Thanks in advance,
Alex van der Spek 


--
http://mail.python.org/mailman/listinfo/python-list

Let child process to run while parent is out (multiprocessing)

[John Wong]

def main(...):
build_id = create_build_id(...)
build_stuff
return build_id

Suppose build_stuff compiles a C program. It could take days to finish, and
notify users their builds are ready. I was thinking about using
mutliprocessing to handle the build_stuff.

So here is a sample:

#!/usr/bin/python

import multiprocessing as mp
import time

def build():
print 'I am building HUGE things'
time.sleep(10)

def main():
build_p = mp.Process(name='build process', target=build)
build_p.start()
return 'abcd12345'

if __name__ == '__main__':

v = main()
print v
print 'done'

Here is output:
yeukhon@fermat:~$ python c2.py
abcd12345
done  [now hangs for 10 seconds]
I build things

When I looked at `ps -elf|grep python`, I can see two processes running,
and one of the python c2.py process is `wait`.  But this is not ideal,
especially this is a web app. I can't implement any advanced queue / event
system right now (I use Pylon, I believe I have gevent installed). But just
with multiprocessing, is it possible to send the id first, while running
child in the backgroud?

Right now it hangs there as long as the child process is alive. Any
solutions?

Thanks.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Encapsulation, inheritance and polymorphism

[John Gordon]

In  Dennis Lee Bieber 
 writes:

> > Sure it terminates...If you don't run out of RAM to represent the
> > number "i" in question, there's also this "heat death of the
> > universe" limit I keep hearing about ;-)
> >
>   Since the current evidence indicates the universe will just keep
> expanding, it's more of a "deep freeze death..."

Heat death means *lack* of heat.

-- 
John Gordon   A is for Amy, who fell down the stairs
gor...@panix.com  B is for Basil, assaulted by bears
-- Edward Gorey, "The Gashlycrumb Tinies"

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Finding duplicate file names and modifying them based on elements of the path

[MRAB]

On 19/07/2012 20:06, larry.mart...@gmail.com wrote:

On Jul 19, 1:02 pm, "Prasad, Ramit"  wrote:

> > I am making the assumption that you intend to collapse the directory
> > tree and store each file in the same directory, otherwise I can't think
> > of why you need to do this.

> Hi Simon, thanks for the reply. It's not quite this - what I am doing
> is creating a zip file with relative path names, and if there are
> duplicate files the parts of the path that are not be carried over
> need to get prepended to the file names to make then unique,

Depending on the file system of the client, you can hit file name
length limits. I would think it would be better to just create
the full structure in the zip.

Just something to keep in mind, especially if you see funky behavior.


Thanks, but it's not what the client wants.


Here's another solution, not using itertools:

from collections import defaultdict
from os.path import basename, dirname
from time import strftime, strptime

# Starting with the original paths

paths = [
"/dir0/dir1/dir2/dir3/qwer/09Jan12/dir6/file3",
"/dir0/dir1/dir2/dir3/abcd/08Jan12/dir6/file1",
"/dir0/dir1/dir2/dir3/abcd/08Jan12/dir6/file2",
"/dir0/dir1/dir2/dir3/xyz/08Jan12/dir6/file1",
"/dir0/dir1/dir2/dir3/qwer/07Jan12/dir6/file3",
]

def make_dir5_key(path):
date = strptime(path.split("/")[6], "%d%b%y")
return strftime("%y%b%d", date)

# Collect the paths into a dict keyed by the basename

files = defaultdict(list)
for path in paths:
files[basename(path)].append(path)

# Process a list of paths if there's more than one entry

renaming = []

for name, entries in files.items():
if len(entries) > 1:
# Collect the paths in each subgroup into a dict keyed by dir4

subgroup = defaultdict(list)
for path in entries:
subgroup[path.split("/")[5]].append(path)

for dir4, subentries in subgroup.items():
# Sort the subentries by dir5 (date)
subentries.sort(key=make_dir5_key)

if len(subentries) > 1:
for index, path in enumerate(subentries):
renaming.append((path, 
"{}/{}_{:02}_{}".format(dirname(path), dir4, index, name)))

else:
path = subentries[0]
renaming.append((path, "{}/{}_{}".format(dirname(path), 
dir4, name)))

else:
path = entries[0]

for old_path, new_path in renaming:
print("Rename {!r} to {!r}".format(old_path, new_path))

--
http://mail.python.org/mailman/listinfo/python-list

Re: Encapsulation, inheritance and polymorphism

[OKB (not okblacke)]

MRAB wrote:

> """Thus spake the Lord: Thou shalt indent with four spaces. No
> more, no less.
> Four shall be the number of spaces thou shalt indent, and the
> number of thy indenting shall be four. Eight shalt thou not indent,
> nor either indent thou
> two, excepting that thou then proceed to four. Tabs are right
> out.""" 
>   -- Georg Brandl

Yes, and the Lord was Wrong.

-- 
--OKB (not okblacke)
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is
no path, and leave a trail."
--author unknown
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Encapsulation, inheritance and polymorphism

[Chris Angelico]

On Fri, Jul 20, 2012 at 7:01 AM, John Gordon  wrote:
> In  Dennis Lee Bieber 
>  writes:
>
>> > Sure it terminates...If you don't run out of RAM to represent the
>> > number "i" in question, there's also this "heat death of the
>> > universe" limit I keep hearing about ;-)
>> >
>>   Since the current evidence indicates the universe will just keep
>> expanding, it's more of a "deep freeze death..."
>
> Heat death means *lack* of heat.

The second law of thermodynamics states that energy tends to go from
higher states to lower, with heat being the very lowest. It's possible
to do work using (say) kinetic energy, and in the process, some of
that energy becomes heat. It's also possible to do work with any
difference in temperature (eg Stirling engines), so the state of the
universe in which it's no longer possible to do any work will be one
in which all energy is heat and everything's at the same temperature.
That doesn't mean a lack of heat; in fact, it implies that there'll be
rather more heat than there now is, because we currently have a whole
lot of chemical energy available to be used.

But in any case, that's a long way off...

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Encapsulation, inheritance and polymorphism

[John Gordon]

In  Chris Angelico 
 writes:

> The second law of thermodynamics states that energy tends to go from
> higher states to lower, with heat being the very lowest. It's possible
> to do work using (say) kinetic energy, and in the process, some of
> that energy becomes heat. It's also possible to do work with any
> difference in temperature (eg Stirling engines), so the state of the
> universe in which it's no longer possible to do any work will be one
> in which all energy is heat and everything's at the same temperature.
> That doesn't mean a lack of heat; in fact, it implies that there'll be
> rather more heat than there now is, because we currently have a whole
> lot of chemical energy available to be used.

*mind blown*

-- 
John Gordon   A is for Amy, who fell down the stairs
gor...@panix.com  B is for Basil, assaulted by bears
-- Edward Gorey, "The Gashlycrumb Tinies"

-- 
http://mail.python.org/mailman/listinfo/python-list

help

[Miriam Gomez Rios]

Hello, sorry for bothering you, but I have a doubt,

Is there a way to turn this string into a tuplelist??, I need it for gurobi

('per1','persona1.1','pro1'),('per1','persona1.1','pro2'),('per1','persona1.1','pro3'),('per1','persona1.1','pro4'),('per1','persona1.1','pro5'),('per2','persona2.1','pro1'),('per2','persona2.1','pro2'),('per2','persona2.1','pro3'),('per2','persona2.1','pro4'),('per2','persona2.1','pro5'),('per2','persona2.2','pro1'),('per2','persona2.2','pro2'),('per2','persona2.2','pro3'),('per2','persona2.2','pro4'),('per2','persona2.2','pro5'),('per2','persona2.3','pro1'),('per2','persona2.3','pro2'),('per2','persona2.3','pro3'),('per2','persona2.3','pro4'),('per2','persona2.3','pro5'),('per2','persona2.4','pro1'),('per2','persona2.4','pro2'),('per2','persona2.4','pro3'

the string is made with fors, using data from another file, and I need it as a 
tuplelist

Thank you

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Let child process to run while parent is out (multiprocessing)

[Chris Angelico]

On Fri, Jul 20, 2012 at 6:34 AM, John Wong  wrote:
> Here is output:
> yeukhon@fermat:~$ python c2.py
> abcd12345
> done  [now hangs for 10 seconds]
> I build things

Side point: I just tried your code on Python 2.6 on Windows, and it
produced all three lines of output before waiting the ten seconds.
Same in Python 3.3 on Linux (after putting parentheses around the
prints, the only 2->3 change needed). Your experience of it delaying
the "building" message probably has something to do with output
buffering.

As to the fundamental, though: I don't know that Python truly supports
what you're trying to do. A skim of the docs doesn't show up anything
obvious, so I'm hoping that someone else has already tried this and
can give an authoritative or empirical answer.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: help

[Benjamin Kaplan]

On Jul 19, 2012 4:04 PM, "Miriam Gomez Rios" 
wrote:
>
> Hello, sorry for bothering you, but I have a doubt,
>
> Is there a way to turn this string into a tuplelist??, I need it for
gurobi
>
>
('per1','persona1.1','pro1'),('per1','persona1.1','pro2'),('per1','persona1.1','pro3'),('per1','persona1.1','pro4'),('per1','persona1.1','pro5'),('per2','persona2.1','pro1'),('per2','persona2.1','pro2'),('per2','persona2.1','pro3'),('per2','persona2.1','pro4'),('per2','persona2.1','pro5'),('per2','persona2.2','pro1'),('per2','persona2.2','pro2'),('per2','persona2.2','pro3'),('per2','persona2.2','pro4'),('per2','persona2.2','pro5'),('per2','persona2.3','pro1'),('per2','persona2.3','pro2'),('per2','persona2.3','pro3'),('per2','persona2.3','pro4'),('per2','persona2.3','pro5'),('per2','persona2.4','pro1'),('per2','persona2.4','pro2'),('per2','persona2.4','pro3'
>
> the string is made with fors, using data from another file, and I need it
as a tuplelist
>
> Thank you
>
>
> --

It looks like you can just put brackets around it and call ast.literal_eval.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: help

[Tim Chase]

On 07/19/12 17:31, Miriam Gomez Rios wrote:
> Hello, sorry for bothering you, but I have a doubt,
> 
> Is there a way to turn this string into a tuplelist??, I
> need it for gurobi
> 
> ('per1','persona1.1','pro1'),('per1','persona1.1','pro2'),('per1','persona1.1','pro3'),...,('per2','persona2.4','pro3'

Assuming a closing paren, you can safely use ast.literal_eval on the
raw string:

  import ast
  s = "('per1','persona1.1','pro1'),('per1','persona1.1','pro2')"
  resulting_tuple_of_tuples = ast.literal_eval(s)

-tkc



-- 
http://mail.python.org/mailman/listinfo/python-list

RE: help

[Prasad, Ramit]

>('per1','persona1.1','pro1'),('per1','persona1.1','pro2'),('per1','persona1.1','pro3'),('per1','persona1.1','pro4'),('per1','persona1.1','pro5'),('per2','persona2.1','pro1'),('per2','persona2.1','pro2'),('per2','persona2.1','pro3'),('per2','persona2.1','pro4'),('per2','persona2.1','pro5'),('per2','persona2.2','pro1'),('per2','persona2.2','pro2'),('per2','persona2.2','pro3'),('per2','persona2.2','pro4'),('per2','persona2.2','pro5'),('per2','persona2.3','pro1'),('per2','persona2.3','pro2'),('per2','persona2.3','pro3'),('per2','persona2.3','pro4'),('per2','persona2.3','pro5'),('per2','persona2.4','pro1'),('per2','persona2.4','pro2'),('per2','persona2.4','pro3'

You can use ast.literal_eval if you add an enclosing () or [] in the string 
(and add a finishing paren for the last tuple).

>the string is made with fors, using data from another file, and I need it as a 
>tuplelist

If you are transferring this data (network/file) you may want to take a 
look at pickle/json. If you are doing this to convert to/from a CSV then look at
the csv module. If you say what are you trying to do and why, someone on the
list might be able to recommend a better way to do this.

Ramit


Ramit Prasad | JPMorgan Chase Investment Bank | Currencies Technology
712 Main Street | Houston, TX 77002
work phone: 713 - 216 - 5423

--


This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: help

[Tim Chase]

On 07/19/12 18:26, Prasad, Ramit wrote:
>> ('per1','persona1.1','pro1'),('per1','persona1.1','pro2'),('per1','persona1.1','pro3'),('per1','persona1.1','pro4'),('per1','persona1.1','pro5'),('per2','persona2.1','pro1'),('per2','persona2.1','pro2'),('per2','persona2.1','pro3'),('per2','persona2.1','pro4'),('per2','persona2.1','pro5'),('per2','persona2.2','pro1'),('per2','persona2.2','pro2'),('per2','persona2.2','pro3'),('per2','persona2.2','pro4'),('per2','persona2.2','pro5'),('per2','persona2.3','pro1'),('per2','persona2.3','pro2'),('per2','persona2.3','pro3'),('per2','persona2.3','pro4'),('per2','persona2.3','pro5'),('per2','persona2.4','pro1'),('per2','persona2.4','pro2'),('per2','persona2.4','pro3'
> 
> You can use ast.literal_eval if you add an enclosing () or [] in the string 
> (and add a finishing paren for the last tuple).

If you want a resulting tuple-of-tuple and you have at least one
comma after the first tuple, you don't even have to include the ()
around it.

-tkc


-- 
http://mail.python.org/mailman/listinfo/python-list

best way to handle this in Python

[Rita]

Hello,

I have data in many files (/data/year/month/day/) which are named like
YearMonthDayHourMinute.gz.

I would like to build a data structure which can easily handle querying the
data. So for example, if I want to query data from 3 weeks ago till today,
i can do it rather quickly.

each YearMonthDayHourMinute.gz file look like this and they are about 4to
6kb
red 34
green 44
blue 88
orange 4
black 3
while 153

I would like to query them so I can generate a plot rather quickly but not
sure what is the best way to do this.



-- 
--- Get your facts first, then you can distort them as you please.--
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Which lib can diff two dict tree?

[alex23]

On Jul 19, 7:51 pm, esbat...@gmail.com wrote:
> Q1：
> which lib can diff two dict tree？
> Q2
> How can I diff two json file?How can I get struct_diff and value_diff?
> Q3
> How can I diff two xml file? How can I get struct_diff and value_diff?

Rather than trying to come up with a method for finding the diffs of
three different data structures, it might be easier to transform all
the structures into the same form and then just use one mechanism for
diffs.

There's a 3rd-party python library for producing diffs of XML
documents:
http://www.logilab.org/projects/xmldiff

Python dicts can be converted to XML using one of the suggestions
here:
http://stackoverflow.com/questions/1019895/serialize-python-dictionary-to-xml

JSON can be transformed into dicts by using the standard library's
json module.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: help

[Tim Chase]

On 07/19/12 19:32, Miriam Gomez Rios wrote:
   s = "('per1','persona1.1','pro1'),('per1','persona1.1','pro2')"
>>   resulting_tuple_of_tuples = ast.literal_eval(s)
>
> Traceback (most recent call last):
>   File "C:\Users\Miriam\scripts\prueba_datos", line 124, in 
> print resulting_tuple.select('perf1','*','*')
> AttributeError: 'tuple' object has no attribute 'select'

[copying the list, trimming your reply and changing to inline
quoting for easier reading]

I'm not sure why you expect there to be a .select method on the
resulting object.  Neither a list-of-tuples or a tuple-of-tuples has
a .select method.  It *sounds* like what you want to do is then to
do something like

  d = ast.literal_eval(s) # data as a tuple-of-tuples
  filtered = [t for t in d where t[0] == "perf1"]

or you can do

  filtered = [(v1,v2,v3) for (v1,v2,v3) in d where v1=="perf1"]

which allows you to assign more meaningful names to the bits, as
well as rearrange/select the contents:

  filtered = [(v3,v2) for (v1,v2,v3) in d where v1=="perf1"]

or you can even do further work like instantiating objects:

  filtered = [MyCls(v3,v2) for (v1,v2,v3) in d where v1=="perf1"]

Espero q' eso le ayuda a usted.

-tkc



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: best way to handle this in Python

[Dave Angel]

On 07/19/2012 07:51 PM, Rita wrote:
> Hello,
>
> I have data in many files (/data/year/month/day/) which are named like
> YearMonthDayHourMinute.gz.
>
> I would like to build a data structure which can easily handle querying the
> data. So for example, if I want to query data from 3 weeks ago till today,
> i can do it rather quickly.
>
> each YearMonthDayHourMinute.gz file look like this and they are about 4to
> 6kb
> red 34
> green 44
> blue 88
> orange 4
> black 3
> while 153
>
> I would like to query them so I can generate a plot rather quickly but not
> sure what is the best way to do this.
>
>
>

What part of your code is giving you difficulty?  You didn't post any
code.  You don't specify the OS, nor version of your Python, nor what
other programs you expect to use along with Python.



-- 

DaveA

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Finding duplicate file names and modifying them based on elements of the path

[larry.mart...@gmail.com]

On Jul 19, 1:56 pm, Paul Rubin  wrote:
> "larry.mart...@gmail.com"  writes:
> > You can't do a len on the iterator that is returned from groupby, and
> > I've tried to do something with imap or      defaultdict, but I'm not
> > getting anywhere. I guess I can just make 2 passes through the data,
> > the first time getting counts. Or am I missing something about how
> > groupby works?
>
> I posted another reply to your other message, which reached me earlier.
> If you're still stuck, post again, though I probably won't be able to
> reply til tomorrow or the next day.

I really appreciate the offer, but I'm going to go with MRAB's
solution. It works, and I understand it ;-)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Finding duplicate file names and modifying them based on elements of the path

[larry.mart...@gmail.com]

On Jul 19, 3:32 pm, MRAB  wrote:
> On 19/07/2012 20:06, larry.mart...@gmail.com wrote:
>
>
>
>
>
>
>
> > On Jul 19, 1:02 pm, "Prasad, Ramit"  wrote:
> >> > > I am making the assumption that you intend to collapse the directory
> >> > > tree and store each file in the same directory, otherwise I can't think
> >> > > of why you need to do this.
>
> >> > Hi Simon, thanks for the reply. It's not quite this - what I am doing
> >> > is creating a zip file with relative path names, and if there are
> >> > duplicate files the parts of the path that are not be carried over
> >> > need to get prepended to the file names to make then unique,
>
> >> Depending on the file system of the client, you can hit file name
> >> length limits. I would think it would be better to just create
> >> the full structure in the zip.
>
> >> Just something to keep in mind, especially if you see funky behavior.
>
> > Thanks, but it's not what the client wants.
>
> Here's another solution, not using itertools:
>
> from collections import defaultdict
> from os.path import basename, dirname
> from time import strftime, strptime
>
> # Starting with the original paths
>
> paths = [
>      "/dir0/dir1/dir2/dir3/qwer/09Jan12/dir6/file3",
>      "/dir0/dir1/dir2/dir3/abcd/08Jan12/dir6/file1",
>      "/dir0/dir1/dir2/dir3/abcd/08Jan12/dir6/file2",
>      "/dir0/dir1/dir2/dir3/xyz/08Jan12/dir6/file1",
>      "/dir0/dir1/dir2/dir3/qwer/07Jan12/dir6/file3",
> ]
>
> def make_dir5_key(path):
>      date = strptime(path.split("/")[6], "%d%b%y")
>      return strftime("%y%b%d", date)
>
> # Collect the paths into a dict keyed by the basename
>
> files = defaultdict(list)
> for path in paths:
>      files[basename(path)].append(path)
>
> # Process a list of paths if there's more than one entry
>
> renaming = []
>
> for name, entries in files.items():
>      if len(entries) > 1:
>          # Collect the paths in each subgroup into a dict keyed by dir4
>
>          subgroup = defaultdict(list)
>          for path in entries:
>              subgroup[path.split("/")[5]].append(path)
>
>          for dir4, subentries in subgroup.items():
>              # Sort the subentries by dir5 (date)
>              subentries.sort(key=make_dir5_key)
>
>              if len(subentries) > 1:
>                  for index, path in enumerate(subentries):
>                      renaming.append((path,
> "{}/{}_{:02}_{}".format(dirname(path), dir4, index, name)))
>              else:
>                  path = subentries[0]
>                  renaming.append((path, "{}/{}_{}".format(dirname(path),
> dir4, name)))
>      else:
>          path = entries[0]
>
> for old_path, new_path in renaming:
>      print("Rename {!r} to {!r}".format(old_path, new_path))

Thanks a million MRAB. I really like this solution. It's very
understandable and it works! I had never seen .format before. I had to
add the index of the positional args to them to make it work.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Finding duplicate file names and modifying them based on elements of the path

[larry.mart...@gmail.com]

On Jul 19, 1:43 pm, Paul Rubin  wrote:
> "larry.mart...@gmail.com"  writes:
> > Thanks for the reply Paul. I had not heard of itertools. It sounds
> > like just what I need for this. But I am having 1 issue - how do you
> > know how many items are in each group?
>
> Simplest is:
>
>   for key, group in groupby(xs, lambda x:(x[-1],x[4],x[5])):
>      gs = list(group)  # convert iterator to a list
>      n = len(gs)       # this is the number of elements
>
> there is some theoretical inelegance in that it requires each group to
> fit in memory, but you weren't really going to have billions of files
> with the same basename.
>
> If you're not used to iterators and itertools, note there are some
> subtleties to using groupby to iterate over files, because an iterator
> actually has state.  It bumps a pointer and maybe consumes some input
> every time you advance it.  In a situation like the above, you've got
> some nexted iterators (the groupby iterator generating groups, and the
> individual group iterators that come out of the groupby) that wrap the
> same file handle, so bad confusion can result if you advance both
> iterators without being careful (one can consume file input that you
> thought would go to another).

It seems that if you do a list(group) you have consumed the list. This
screwed me up for a while, and seems very counter-intuitive.

> This isn't as bad as it sounds once you get used to it, but it can be
> a source of frustration at first.
>
> BTW, if you just want to count the elements of an iterator (while
> consuming it),
>
>      n = sum(1 for x in xs)
>
> counts the elements of xs without having to expand it into an in-memory
> list.
>
> Itertools really makes Python feel a lot more expressive and clean,
> despite little kinks like the above.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: best way to handle this in Python

[Rita]

Using linux 2.6.31; Python 2.7.3.
I am not necessary looking for code just a pythonic way of doing it.
Eventually, I would like to graph the data using matplotlib




On Thu, Jul 19, 2012 at 8:52 PM, Dave Angel  wrote:

> On 07/19/2012 07:51 PM, Rita wrote:
> > Hello,
> >
> > I have data in many files (/data/year/month/day/) which are named like
> > YearMonthDayHourMinute.gz.
> >
> > I would like to build a data structure which can easily handle querying
> the
> > data. So for example, if I want to query data from 3 weeks ago till
> today,
> > i can do it rather quickly.
> >
> > each YearMonthDayHourMinute.gz file look like this and they are about 4to
> > 6kb
> > red 34
> > green 44
> > blue 88
> > orange 4
> > black 3
> > while 153
> >
> > I would like to query them so I can generate a plot rather quickly but
> not
> > sure what is the best way to do this.
> >
> >
> >
>
> What part of your code is giving you difficulty?  You didn't post any
> code.  You don't specify the OS, nor version of your Python, nor what
> other programs you expect to use along with Python.
>
>
>
> --
>
> DaveA
>
>


-- 
--- Get your facts first, then you can distort them as you please.--
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Need help connecting via a remote VPN

[The Coca Cola Kid]

"Dave Angel"  wrote in message 
news:mailman.2284.1342663213.4697.python-l...@python.org...



Starting a VPN simply makes it possible for IP packets to get to the
specified machine.  You still need to match some protocol that the
particular remote machine can handle.  SSH is probably the most common,
but there are lots of other possibilities, some standard and some
custom.  So the question is what does the remote machine (server)
support for file access?

glob will work if the server supports mounting the remote partition as a
file system.  But there are no guarantees that that's possible.  You
haven't even said what the two operating systems involved are, though I
can guess that the local machine is some version of Windows.


The local machine is running Windows 7.  I'm not sure what the other machine 
is running on, but I assume it can handle ssh.  If it helps, I usually 
connect to it manually via an https web link, and then after entering my 
user name and password in a Firepass session manager it runs an active-x 
cache cleaner program and will then let me navigate to server B which is 
behind the firewall. 


--
http://mail.python.org/mailman/listinfo/python-list

Re: Need help connecting via a remote VPN

[Dave Angel]

On 07/19/2012 09:41 PM, The Coca Cola Kid wrote:
> "Dave Angel"  wrote in message
> news:mailman.2284.1342663213.4697.python-l...@python.org...
>
>> Starting a VPN simply makes it possible for IP packets to get to the
>> specified machine.  You still need to match some protocol that the
>> particular remote machine can handle.  SSH is probably the most common,
>> but there are lots of other possibilities, some standard and some
>> custom.  So the question is what does the remote machine (server)
>> support for file access?
>>
>> glob will work if the server supports mounting the remote partition as a
>> file system.  But there are no guarantees that that's possible.  You
>> haven't even said what the two operating systems involved are, though I
>> can guess that the local machine is some version of Windows.
>
> The local machine is running Windows 7.  I'm not sure what the other
> machine is running on, but I assume it can handle ssh.  If it helps, I
> usually connect to it manually via an https web link, and then after
> entering my user name and password in a Firepass session manager it
> runs an active-x cache cleaner program and will then let me navigate
> to server B which is behind the firewall.

If the remote machine is Linux, running sshd, and if you have an account
there, then use scp to copy the file.  And if you want to automate it,
run scp from within python, using the subprocess module.  But make sure
it works manually before bothering to automate it.

To get ssh on Windows, one approach is to install cygwin.

If you want more control than subprocess, there are a whole bunch of 3rd
party ssh libraries for Python.  Just google  python ssh library  to get
at least 6 on the first page.

-- 

DaveA

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Finding duplicate file names and modifying them based on elements of the path

[larry.mart...@gmail.com]

On Jul 19, 7:01 pm, "larry.mart...@gmail.com"
 wrote:
> On Jul 19, 3:32 pm, MRAB  wrote:
>
>
>
>
>
>
>
>
>
> > On 19/07/2012 20:06, larry.mart...@gmail.com wrote:
>
> > > On Jul 19, 1:02 pm, "Prasad, Ramit"  wrote:
> > >> > > I am making the assumption that you intend to collapse the directory
> > >> > > tree and store each file in the same directory, otherwise I can't 
> > >> > > think
> > >> > > of why you need to do this.
>
> > >> > Hi Simon, thanks for the reply. It's not quite this - what I am doing
> > >> > is creating a zip file with relative path names, and if there are
> > >> > duplicate files the parts of the path that are not be carried over
> > >> > need to get prepended to the file names to make then unique,
>
> > >> Depending on the file system of the client, you can hit file name
> > >> length limits. I would think it would be better to just create
> > >> the full structure in the zip.
>
> > >> Just something to keep in mind, especially if you see funky behavior.
>
> > > Thanks, but it's not what the client wants.
>
> > Here's another solution, not using itertools:
>
> > from collections import defaultdict
> > from os.path import basename, dirname
> > from time import strftime, strptime
>
> > # Starting with the original paths
>
> > paths = [
> >      "/dir0/dir1/dir2/dir3/qwer/09Jan12/dir6/file3",
> >      "/dir0/dir1/dir2/dir3/abcd/08Jan12/dir6/file1",
> >      "/dir0/dir1/dir2/dir3/abcd/08Jan12/dir6/file2",
> >      "/dir0/dir1/dir2/dir3/xyz/08Jan12/dir6/file1",
> >      "/dir0/dir1/dir2/dir3/qwer/07Jan12/dir6/file3",
> > ]
>
> > def make_dir5_key(path):
> >      date = strptime(path.split("/")[6], "%d%b%y")
> >      return strftime("%y%b%d", date)
>
> > # Collect the paths into a dict keyed by the basename
>
> > files = defaultdict(list)
> > for path in paths:
> >      files[basename(path)].append(path)
>
> > # Process a list of paths if there's more than one entry
>
> > renaming = []
>
> > for name, entries in files.items():
> >      if len(entries) > 1:
> >          # Collect the paths in each subgroup into a dict keyed by dir4
>
> >          subgroup = defaultdict(list)
> >          for path in entries:
> >              subgroup[path.split("/")[5]].append(path)
>
> >          for dir4, subentries in subgroup.items():
> >              # Sort the subentries by dir5 (date)
> >              subentries.sort(key=make_dir5_key)
>
> >              if len(subentries) > 1:
> >                  for index, path in enumerate(subentries):
> >                      renaming.append((path,
> > "{}/{}_{:02}_{}".format(dirname(path), dir4, index, name)))
> >              else:
> >                  path = subentries[0]
> >                  renaming.append((path, "{}/{}_{}".format(dirname(path),
> > dir4, name)))
> >      else:
> >          path = entries[0]
>
> > for old_path, new_path in renaming:
> >      print("Rename {!r} to {!r}".format(old_path, new_path))
>
> Thanks a million MRAB. I really like this solution. It's very
> understandable and it works! I had never seen .format before. I had to
> add the index of the positional args to them to make it work.

Also, in make_dir5_key the format specifier for strftime should be %y%m
%d so they sort properly.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: properly catch SIGTERM

[Dieter Maurer]

Eric Frederich  writes:
> ...
> This seems to work okay but just now I got this while hitting ctrl-c
> It seems to have caught the signal at or in the middle of a call to
> sys.stdout.flush()
> --- Caught SIGTERM; Attempting to quit gracefully ---
> Traceback (most recent call last):
>   File "/home/user/test.py", line 125, in 
>     sys.stdout.flush()
> IOError: [Errno 4] Interrupted system call
> How should I fix this?

This is normal *nix behavior. Any signal, even if caught by a signal
handler, can interrupt system calls.

Modern *nix versions might allow to control whether a signal interrupts
a system call or not. Check the signal documentation for your operating
system for the control you have over signal handling. Likely,
you cannot directly control the feature via Python, but the
"ctypes" module allows you to call C functions directly.

-- 
http://mail.python.org/mailman/listinfo/python-list