Archives and magic bytes

[andrea]

Hi everybody,
this is my first post but I've read already many of yours interesting 
posts... (sorry for my bad english)

Anyway for my little project I need a module that given an archive (zip, 
bz2, tar ...) gives me back the archive decompressed.

I looked at the modules in the library reference and used some of them, 
but the problem is that they all behave in a different way, and just one 
has a useful command for decompressing files easily, bz2.

/decompress(...)
   decompress(data) -> decompressed data
  
   Decompress data in one shot. If you want to decompress data 
sequentially,
   use an instance of BZ2Decompressor instead.
/
I can't get even that one working, what does it mean data? A file?

Maybe I could implement myself the compression algorithm (or copy that 
from the modules) and implement myself compress/decompress functions, 
what do you think?
Do you know if there is already something similar??

Another thing, I work on linux (gentoo) and I would like to use the 
"file" command to retrieve informations about type of file instead of 
using extensions, do you think this can be done?

Thanks, and sorry if I've done some stupid questions, I'm still a python 
novice (but it's a great language)

Andrea
--
http://mail.python.org/mailman/listinfo/python-list

Re: newbie help

[andrea]

[EMAIL PROTECTED] wrote:
I want to pass and index number  and a file name from the command line.
This index number corresponds to a buffer. I need to maintain 2 such
buffers for my test.
 

Passing options is very easy, just import sys and you can find your 
options in sys.argv.
To do something more complicated you should use getopt, or even more 
complicated optparse.

In C you would do someting like this
unsigned char buf1[512];
In python is this as simple as?
buf 1 = 512
buf 2 = 512
And how do i keep track of buf1 and buf2
Thanks.
Thanks.
 

I think I didn't understand the question, can't you just use a list??
--
http://mail.python.org/mailman/listinfo/python-list

Re: Archives and magic bytes

[andrea]

Chris Rebert (cybercobra) wrote:
Have you tried the tarfile or zipfile modules? You might need to ugrade
your python if you don't have them. They look pretty easy and should
make this a snap.
You can grab the output from the *nix "file" command using the new
subprocess module.
Good Luck
- Chris
===
PYTHON POWERs all!
All your code are belong to Python!
 

I've got them (I'm still using python 2.3 because I use gentoo) but they 
are not very easy to use as they seem...
I'll try again, thanks!
--
http://mail.python.org/mailman/listinfo/python-list

creating very small types

[andrea]

I was thinking to code the huffman algorithm and trying to compress
something with it, but I've got a problem.
How can I represent for example a char with only 3 bits??
I had a look to the compression modules but I can't understand them much...

Thank you very much
Any good link would be appreciated of course :)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: creating very small types

[andrea]

Jeremy Bowers wrote:

>On Wed, 27 Apr 2005 22:17:07 +0200, andrea wrote:
>
>  
>
>>I was thinking to code the huffman algorithm and trying to compress
>>something with it, but I've got a problem.
>>How can I represent for example a char with only 3 bits??
>>I had a look to the compression modules but I can't understand them much...
>>
>>Thank you very much
>>Any good link would be appreciated of course :)
>>
>>
>
>I think the answer to this question very much depends on why you want to
>do this. Easy and fast are pretty opposed in this particular domain in
>Python (IMHO anyways, it's an easy bit-bashing language but it's slow
>for that), and you're in one of the rare domains where it matters.
>
>The answers strongly vary if you're trying to create something performant,
>or just for your learning purposes. Or homework purposes... ;-)
>
>  
>
No it's not for homework but for learning purposes...
I understand I can't do it easily in python, but maybe I could define a
new type in C and use it to do those dirty works, what do you think?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Automatic import PEP

[Andrea]

> Opinions?

Great :)

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: 2.5 updates for syntax file

[Andrea]

> Hi everyone. I'm updating my UltraEdit syntax file for Python 2.5 and
> was wondering if anyone knew of other additions to make besides these. I
> found all of these in the What's New document, but wasn't sure if there
> were other important things to add for highlight.

Great :)

- Two new built-in functions, any() and all()
- A new warning, UnicodeWarning

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Designing a graph study program

[andrea]

On 10 Mag, 16:52, Alexander Schliep <[EMAIL PROTECTED]> wrote:
> andrea <[EMAIL PROTECTED]> writes:
> > On 9 Mag, 09:10, Alexander Schliep <[EMAIL PROTECTED]> wrote:
> >> Check outhttp://gato.sf.net(LGPLlicense). It does exactly what you
> >> want to do and there is a binary for MacOS X. Algorithms are implemented
> >> using Gato's graph class and rudimentary visualizations you get for free
> >> by replacing standard data structures (e.g., a vertex queue) by
> >> animated ones (AnimatedVertexQueue).
>
> > Very very nice well done!
>
> Thanks.
>
> > I'd like to do something similar, just to learn something new...
>
> Gato is open source and I'd be happy to collaborate. There are quite a
> few areas (e.g. SVG export, displays for data structure contents, more
> complete 3D support, polygon edges, running backwards?) which need
> work.
>
> > Could you explain me how you designed it?? How is the step mechanism
> > done??
>
> The algorithm is executed by a subclass of the Python debugger (see
> Gato.py).  A Tk event mechanism is used to step to the next line if
> you are in running mode. Otherwise the user steps manually. The
> visualizations are done with animated data structures, which animate
> changes to their internal state with respect to the graph: e.g., when
> you add or remove v to/from an AnimatedVertexQueue it changes v's
> color.
>
> Tk has a canvas which does object-oriented drawing. A line is not
> just a bunch of pixels but rather an object which you can move,
> scale, has callbacks. I dislike Tcl, but Tk is amazing, even it
> it looks 1980s.
>
> There is a 
> paperhttp://algorithmics.molgen.mpg.de/preprints/2000-CATBox-MTCM.pdf
> describing the ideas.
>
> Best,
> Alexander

Ok thanks a lot for the explanation Alexander...

Well I changed my implementation of edges and nodes to this:

class node:
"""nodi di un grafo"""
def __init__(self, label, color=None, parent=None, distance=None):
self.label = label
self.color = color
self.parent = parent
self.distance = distance

def __eq__(self,y):
"""uguaglianza fra nodi"""
return self.label == y.label

def __repr__(self):
"""docstring for __repr__"""
return str(self.label)


class edge: # CHANGED tutta la gestione di nodi e lati
"""lato di un grafo"""
def __init__(self, u, v, directed=False, w=1):
"due lati ed eventualmente il peso associato"
self.u = u
self.v = v
self.w = w
self.directed = directed


def __repr__(self):
"""docstring for __repr__"""
if self.directed:
sym = " --> "
else:
sym = " --- "
return str(self.u) + sym + str(self.v)

def __eq__(self,y):
if self.directed != y.directed:
return False
r = (self.u == y.u and self.v == y.v)
l = (self.v == y.u and self.u == y.v)
if self.directed: # gia controllato che non siano diversi
return r
else:
return r or l


Does it make sense??
But now I have some troubles with all the rest ;)
For example this code

def make_adj_list(self,nodes,edges):
"""crea la lista di adiacenza"""
adj_list = {}
for v in nodes:
adj_list[v] = []
   ...
builds an adjacency list of the graph, but now the object Node is not
hashable of course!
How do I manage this? I think I can use the label
adj_list[v.label] = [u.label, etc etc]
but then I need another dictionary to go from labels to objects, it's
not difficult but look ugly to me, other solutions??

Thanks a lot

-- 
http://mail.python.org/mailman/listinfo/python-list

textmate and execute line

[andrea]

Hi,
I have the latest bundle for python (upgraded from svn) but I don't
understand how execute line works..

It only works if I play with arithmetic operations, something like
this works:
2*2
4


but for example trying to execute this line gives me this error, any
help?
x = 1; x + 1

Traceback (most recent call last):
  File "/tmp/temp_textmate.31Kc2h", line 19, in ?
stdout.write(exc)
TypeError: argument 1 must be string or read-only character buffer,
not list


Thanks a lot

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: textmate and execute line

[andrea]

On 19 Giu, 14:20, andrea <[EMAIL PROTECTED]> wrote:
> Hi,
> I have the latest bundle for python (upgraded from svn) but I don't
> understand how execute line works..
>
> It only works if I play with arithmetic operations, something like
> this works:
> 2*2
> 4
>
> but for example trying to execute this line gives me this error, any
> help?
> x = 1; x + 1
>
> Traceback (most recent call last):
>   File "/tmp/temp_textmate.31Kc2h", line 19, in ?
> stdout.write(exc)
> TypeError: argument 1 must be string or read-only character buffer,
> not list
>
> Thanks a lot

Noone using textmate for python?

-- 
http://mail.python.org/mailman/listinfo/python-list

Rappresenting infinite

[andrea]

I would like to have a useful rappresentation of infinite, is there
already something??

I was thinking to something like this

class Inf(int):
"""numero infinito"""
def __init__(self,negative=False):
self.negative = negative
def __cmp__(self,y):
"""infinito maggiore di qualasiasi cosa"""
if self.negative:
return -1
else:
return 1
def __repr__(self):
"""docstring for __repr__"""
if self.negative:
return '-inf'
else:
return 'inf'
def __add__(self,y):
"""addizione con infinito"""
return self
def __mul__(self,y):
"""moltiplicazione"""
import types
if isinstance(y,types.IntType):
if y > 0:
return self
if y == 0:
return 'indefinito'
else:
return Inf(negative)


I'd like to be able to compare between any value and infinite always
getting the right result, and paying attention to the special cases
like
inf / inf => not determinate

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Rappresenting infinite

[andrea]

Mm very interesting thread, for my needs from numpy import inf is more
than enough :)


-- 
http://mail.python.org/mailman/listinfo/python-list

Designing a graph study program

[andrea]

I'm studying some graphs algorithm (minumum spanning tree, breath
search first topological sort etc etc...) and to understand better the
theory I'm implementing them with python...

I made my own graph class, the constructor is simply this:
class graph:
"in forma di matrice e' una matrice normale, in forma di lista uso un
dizionario"
def __init__(self,nodes,edges,dir=False,weight=[]): # inizializzatore
dell'oggetto, di default in forma di lista di adiacenza e undirected

# il grafo puo' essere anche pesato
"di default uso la lista di adiacenza per rappresentare il 
grafo,
usare set"
self.adj_list = {}
self.nodes = nodes
self.edges = edges # in modo da avere comodi questi dati, se
bidirezionale non ci sono tutti
self.weight = weight
self.dir = dir # anche questo deve essere raggiungibile
for n in nodes:
self.adj_list[n] = []
for n in nodes:
for e in edges:
if dir: # se ho la direzione guardo l'ordine 
dei vertici nel lato
if n == e[0]:
self.adj_list[n].append(e[1])
elif n in e:
other = e[((e.index(n))+1) % 2]
self.adj_list[n].append(other)
if weight:
self.w = {}
for idx in range(len(edges)):
self.w[edges[idx]] = weight[idx] # assegno in 
corrispondenza

Well then I wanted to draw graphs and I found that pydot is working
really nicely.
BUT I'd like to do this, an interactive program to see ho the
algorithms works...
For example in the breath search first, every time the algorithm
colors a node, the program should redraw the graphs. Which modules
should I use for graphics (I use macosX and I'd like something cross
platforms).

Now I'm doing something like this

def draw_graph(self):
"""disegna un grafo con pydot"""
import os
output = 'graph.png'
self.dot_graph.write_png(output)
com = 'open '+output
os.popen(com)

which I think is very ugly and not fitting very well for my purpose.

I also created a class to represent matrix (for the matrix view of the
graph) but I found that numpy has a very complete implementation, I
think I'll go with it.

Thank you very much, if you have any kind of suggestions/links please
write it :)

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Designing a graph study program

[andrea]

On 8 Mag, 13:02, "Diez B. Roggisch" <[EMAIL PROTECTED]> wrote:
> > Well then I wanted to draw graphs and I found that pydot is working
> > really nicely.
> > BUT I'd like to do this, an interactive program to see ho the
> > algorithms works...
> > For example in the breath search first, every time the algorithm
> > colors a node, the program should redraw the graphs. Which modules
> > should I use for graphics (I use macosX and I'd like something cross
> > platforms).
>
> Use the bundled Tkinter. I've implemented a similar thing back in my under
> graduation days using TCL/Tk, and Tk is perfectly suited for that task.
>
> Diez

Ok thank you very much I'll try with that.
But I have some design doubts, I'd like to keep the algorithm (for
example bfs) as clean as possible, being independent from the drawing
methods.
And how could I make it step through algorithms without having a more
complicated code? Maybe using threads?

Thanks

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Designing a graph study program

[andrea]

On 8 Mag, 13:55, "Diez B. Roggisch" <[EMAIL PROTECTED]> wrote:
> > Ok thank you very much I'll try with that.
> > But I have some design doubts, I'd like to keep the algorithm (for
> > example bfs) as clean as possible, being independent from the drawing
> > methods.
> > And how could I make it step through algorithms without having a more
> > complicated code? Maybe using threads?
>
> Along the lines of what Marc said:
>
> Use two graph-implementations that share the same interface regarding the
> algorithms, but one will emit events to some observer for each operation on
> the graph - edge/node adding/removal, attribute changing and so forth.
>
> Thus the algorithm is kept clean, and all that you can visualize anyway is
> available to you.
>
> diez
Interesting but what do you mean for two graph-implementatio that
share the same interface??
I don't have abstract classes or interfaces in python, am I wrong?
Thanks

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Designing a graph study program

[andrea]

On 9 Mag, 09:10, Alexander Schliep <[EMAIL PROTECTED]> wrote:
> andrea <[EMAIL PROTECTED]> writes:
> > Well then I wanted to draw graphs and I found that pydot is working
> > really nicely.
> > BUT I'd like to do this, an interactive program to see ho the
> > algorithms works...
> > For example in the breath search first, every time the algorithm
> > colors a node, the program should redraw the graphs. Which modules
> > should I use for graphics (I use macosX and I'd like something cross
> > platforms).
>
> Check outhttp://gato.sf.net(LGPL license). It does exactly what you
> want to do and there is a binary for MacOS X. Algorithms are implemented
> using Gato's graph class and rudimentary visualizations you get for free
> by replacing standard data structures (e.g., a vertex queue) by
> animated ones (AnimatedVertexQueue).
>
> There is a Springer textbook forthcoming. We are also starting to collect
> contributed algorithms, which we would like to make available from
> our website.
>
> Full disclosure: I am the author of Gato
>
> Best,
> Alexander

Very very nice well done!
I'd like to do something similar, just to learn something new...
Could you explain me how you designed it?? How is the step mechanism
done??
Any advices?

-- 
http://mail.python.org/mailman/listinfo/python-list

Path python versions and Macosx

[andrea]

Hi everyone,
I use python on macosx with textmate as editor (great program).

I also use macport to install unix programs from the command line and
I find it great too.
Well I would like to have all my modules in the path when I'm using
textmate AND when I use the commandline (ipython), but because
textmate and the command line use different python versions they also
search in different places..

I found somewhere to write .pythonrc.py like this

#!/usr/bin/env python
import sys
PATH='/opt/local/lib/python2.4/site-packages/'
sys.path.append(PATH)
del sys

But it doesn't work either, I also tried to append this
PATH="/Library/Frameworks/Python.framework/Versions/Current/bin:/opt/
local/lib/python2.4/site-packages:${PATH}
to .bash_profile but nothing.

Where should I set this variables??

Thanks a lot

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Path python versions and Macosx

[andrea]

On 12 Mag, 01:09, [EMAIL PROTECTED] wrote:
> On May 11, 1:36 pm, andrea <[EMAIL PROTECTED]> wrote:
>
>
>
> > Hi everyone,
> > I use python on macosx with textmate as editor (great program).
>
> > I also use macport to install unix programs from the command line and
> > I find it great too.
> > Well I would like to have all my modules in the path when I'm using
> > textmate AND when I use the commandline (ipython), but because
> > textmate and the command line use different python versions they also
> > search in different places..
>
> > I found somewhere to write .pythonrc.py like this
>
> > #!/usr/bin/env python
> > import sys
> > PATH='/opt/local/lib/python2.4/site-packages/'
> > sys.path.append(PATH)
> > del sys
>
> > But it doesn't work either, I also tried to append this
> > PATH="/Library/Frameworks/Python.framework/Versions/Current/bin:/opt/
> > local/lib/python2.4/site-packages:${PATH}
> > to .bash_profile but nothing.
>
> > Where should I set this variables??
>
> > Thanks a lot
>
> You can set environment variables for gui apps with this 
> freeware:http://www.versiontracker.com/dyn/moreinfo/macosx/15073
>
> You can edit some text files as well, but this thing just makes it
> much easier.
>
> ~Sean

Ok thanks, but why it doesn't work setting the .bash_profile?? What
should I set manually theorically?? The problem is also that I have
many modules for python 2.4 and trying to import them from the 2.5 of
course gives errors..
I installed them with macports (compiling), how can I make them switch
to 2.5???
thanks

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Path python versions and Macosx

[andrea]

On 14 Mag, 23:00, [EMAIL PROTECTED] wrote:
> On May 14, 4:46 am, andrea <[EMAIL PROTECTED]> wrote:
>
>
>
> > On 12 Mag, 01:09, [EMAIL PROTECTED] wrote:
>
> > > On May 11, 1:36 pm, andrea <[EMAIL PROTECTED]> wrote:
>
> > > > Hi everyone,
> > > > I use python on macosx with textmate as editor (great program).
>
> > > > I also use macport to install unix programs from the command line and
> > > > I find it great too.
> > > > Well I would like to have all my modules in the path when I'm using
> > > > textmate AND when I use the commandline (ipython), but because
> > > > textmate and the command line use different python versions they also
> > > > search in different places..
>
> > > > I found somewhere to write .pythonrc.py like this
>
> > > > #!/usr/bin/env python
> > > > import sys
> > > > PATH='/opt/local/lib/python2.4/site-packages/'
> > > > sys.path.append(PATH)
> > > > del sys
>
> > > > But it doesn't work either, I also tried to append this
> > > > PATH="/Library/Frameworks/Python.framework/Versions/Current/bin:/opt/
> > > > local/lib/python2.4/site-packages:${PATH}
> > > > to .bash_profile but nothing.
>
> > > > Where should I set this variables??
>
> > > > Thanks a lot
>
> > > You can set environment variables for gui apps with this 
> > > freeware:http://www.versiontracker.com/dyn/moreinfo/macosx/15073
>
> > > You can edit some text files as well, but this thing just makes it
> > > much easier.
>
> > > ~Sean
>
> > Ok thanks, but why it doesn't work setting the .bash_profile?? What
> > should I set manually theorically?? The problem is also that I have
> > many modules for python 2.4 and trying to import them from the 2.5 of
> > course gives errors..
> > I installed them with macports (compiling), how can I make them switch
> > to 2.5???
> > thanks
>
> Gui applications are not launched through the bash shell.
> The .bash_profile never gets read.
>
> You can accomplish the same thing as the freewware app I mentioned by
> editing the xml file at ~/.MacOSX/environment.plist.  Its a simple xml
> file with keys and their corresponding values.
>
> Not sure about the modules.  You can force a script to look for libs
> in a given directory like you mentioned, but that needs to go at the
> top of each python file instead of a .pythonrc file.  Maybe the
> modules will work with 2.5, maybe not.
>
> ~Sean

If I put it on the top of every file it works, it just need to be of
the same version...
The problem is that I would like it to run everywhere, putting a
specific path is not good idea imho (I could use try except but it's
annoying).

-- 
http://mail.python.org/mailman/listinfo/python-list

Problem with extremely small real number

[Andrea]

Hi,
I need to calculate this probability P!/{n \choose p}, varying both n
and p, n takes values in this range [512:1024] and p in [2:12].
So i write this code in python:

def factorial(n):
result=1
if n==0: return 1
for i in xrange(1, abs(n)+1):
result = i*result
if n >= 0:
return result
else:
return -result
def binomial(n, k):
assert n>0 and isinstance(n, (int, long)) and isinstance(k,
(int,long))
if k < 0 or k > n:
return 0
if k == 0 or k == n:
return 1
return factorial(n) // (factorial(k) * factorial(n-k))

I want to call factorial(2)//binomial(1024,2) for example, in this way
trivially I obtain 0 as probability, how can I obtain the probability
forcing this division to output extremely small real numbers

I want to produce a program that provide a set of data of this
probability on varying n and p, in order to plot a surface with this
data set with gnuplot, any comments or suggestion?
thanks in advance,
Andrea

-- 
http://mail.python.org/mailman/listinfo/python-list

Pattern_Recognition_School+Conf_2006:Deadline_Approaching

[andrea]

Approaching Deadlines!!

PATTERN RECOGNITION EVENTS THIS SUMMER, 2006
___
4TH INTERNATIONAL SUMMER SCHOOL ON PATTERN RECOGNITION (ISSPR, 2006), 23-28 
JULY, UK
WWW.PatternRecognitionSchool.com
The 4th International Summer School on Pattern Recognition will be organised at 
the University of Plymouth, UK (23-28 July, 2006). The  school programme is 
listed below. Please pass on this email to your interested colleagues and 
students. This is a  great summer school which I would recommend for everyone 
to attend. 
DEADLINE: Register BEFORE 01 MAY, 2006 through the website to get a discount on 
the fee. More than 100 participants in 2005!

Speakers at the Summer School (ISSPR'2006)
Dr. Sam Roberts Mathworks, UK (Introduction to Matlab)
Prof. Wojtek Krzanowski University of Exeter, UK (Multivariate Statistics: Data 
Description)
Dr. Mike Tipping, Microsoft Research, UK (Bayesian Pattern Recognition: 
Principles and Practice)
Prof. Chris Bishop, Microsoft Research, UK (Latent Variables, Mixture Models 
and EM)
Dr. Richard Everson, University of Exeter, UK (Dimensionality Reduction)
Dr. Peter Tino  University of Birmingham, UK (Probabilistic framework for 
model-based topographic map formation)
Prof. Chris Williams, University of Edinburgh, UK (Neural Networks and Kernel 
Machines) 
Dr. Colin Campbell, University of Bristol, UK (Support Vector Machines and 
Kernel Methods: An Introduction and  Review)
Prof. John Shawe- Taylor, University of Southampton, UK (Kernel Based Methods)
Dr. Steve Gunn, University of Southampton, UK (Matlab for Support Vector 
Machines)
Prof. Mahesan Niranjan, University of Sheffield, UK (Classifier Performance 
Particle Filters for Tracking and  Sequential Problems) 
Dr. Andrew Webb Qinetiq, UK (Decision TreesData Clustering)
Prof. Xin Yao, University of Birmingham, UK (A Gentle Introduction to 
Evolutionary Computation; ATutorial on  Evolutionary Multi-objective 
Optimisation) 
Dr. Richard Deardon, University of Birmingham, UK (Sequential Decision Making; 
Markov Chains Monte Carlo Methods)
Dr. Jeremy Wyatt, University of Birmingham, UK (An Introduction to 
Reinforcement Learning)
Dr. Ludmila Kuncheva, University of Wales, UK   (Classifier Combination)
Prof. Joseph Kittler, University of Surrey, UK  (Feature Selection and 
Extraction)
Prof. Sameer Singh, Loughborough University, UK (Multiresolution Pattern 
Recognition)
Prof. Susan Craw, Robert Gordon University, UK  (Case Based Reasoning)
SUPPORTED BY: Microsoft, Springer, British Computer Society, and Mathworks. 
___
IEEE CVPR CONFERENCE
www.CVPR.org/2006
17-22 June, 2006
___
18th INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION 
www.comp.hkbu.edu.hk/~icpr06
August 20-24, 2006
__
ICIAR CONFERENCE
www.iciar.uwaterloo.ca/iciar06
18-20 September, 2006
__
Please see the event websites to get FULL information.

Compiled by: Dr. Heather Mclleland 
-- 
http://mail.python.org/mailman/listinfo/python-list

ISSPR2006_School_etc.:NEW_EXTENDED_Deadline

[andrea]

REGISTER NOW FOR PATTERN RECOGNITION EVENTS THIS SUMMER, 2006

___
4TH INTERNATIONAL SUMMER SCHOOL ON PATTERN RECOGNITION (ISSPR, 2006)
23-28 JULY, UK

http://www.PatternRecognitionSchool.com

NEW...EXTENDED Early Bird Deadline for Registration. 
New Deadline 25th May 2006! and payments can be made Online

CALL FOR PARTICIPATION
The 4th International Summer School on Pattern Recognition will be organised at 
the University of Plymouth, UK (23-28 July, 2006). The  school programme is 
listed below. Please pass on this email to your interested colleagues and 
students. This is a  great summer school which I would recommend for everyone 
to attend. 
DEADLINE: Register BEFORE 01 MAY, 2006 through the website to get a discount on 
the fee. More than 100 participants in 2005!

Speakers at the Summer School (ISSPR'2006)
Dr. Sam Roberts Mathworks, UK (Introduction to Matlab)
Prof. Wojtek Krzanowski University of Exeter, UK (Multivariate Statistics: Data 
Description)
Dr. Mike Tipping, Microsoft Research, UK (Bayesian Pattern Recognition: 
Principles and Practice)
Prof. Chris Bishop, Microsoft Research, UK (Latent Variables, Mixture Models 
and EM)
Dr. Richard Everson, University of Exeter, UK (Dimensionality Reduction)
Dr. Peter Tino  University of Birmingham, UK (Probabilistic framework for 
model-based topographic map formation)
Prof. Chris Williams, University of Edinburgh, UK (Neural Networks and Kernel 
Machines) 
Dr. Colin Campbell, University of Bristol, UK (Support Vector Machines and 
Kernel Methods: An Introduction and  Review)
Prof. John Shawe- Taylor, University of Southampton, UK (Kernel Based Methods)
Dr. Steve Gunn, University of Southampton, UK (Matlab for Support Vector 
Machines)
Prof. Mahesan Niranjan, University of Sheffield, UK (Classifier Performance 
Particle Filters for Tracking and  Sequential Problems) 
Dr. Andrew Webb Qinetiq, UK (Decision TreesData Clustering)
Prof. Xin Yao, University of Birmingham, UK (A Gentle Introduction to 
Evolutionary Computation; ATutorial on  Evolutionary Multi-objective 
Optimisation) 
Dr. Richard Deardon, University of Birmingham, UK (Sequential Decision Making; 
Markov Chains Monte Carlo Methods)
Dr. Jeremy Wyatt, University of Birmingham, UK (An Introduction to 
Reinforcement Learning)
Dr. Ludmila Kuncheva, University of Wales, UK   (Classifier Combination)
Prof. Joseph Kittler, University of Surrey, UK  (Feature Selection and 
Extraction)
Prof. Sameer Singh, Loughborough University, UK (Multiresolution Pattern 
Recognition)
Prof. Susan Craw, Robert Gordon University, UK  (Case Based Reasoning)
SUPPORTED BY: Microsoft, Springer, British Computer Society, and Mathworks. 
___
IEEE CVPR CONFERENCE
www.CVPR.org/2006
17-22 June, 2006
Registration Deadline is now 2nd of June, 2006
___
18th INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION 
www.comp.hkbu.edu.hk/~icpr06
August 20-24, 2006
Early Bird Registration: 31 May, 2006
__
Please see the event websites to get FULL information.

Compiled by: Dr. Heather Mclleland 
-- 
http://mail.python.org/mailman/listinfo/python-list

Final_Call_SummerSchool2006

[andrea]

REGISTER NOW FOR PATTERN RECOGNITION EVENTS THIS SUMMER, 2006
Early Registration Deadline for ISSPR is 25 MAY, 2006
___
4TH INTERNATIONAL SUMMER SCHOOL ON PATTERN RECOGNITION (ISSPR, 2006)
23-28 JULY, UK

http://www.PatternRecognitionSchool.com

NEW...EXTENDED Early Bird Deadline for Registration. 
New Deadline 25th May 2006! and payments can be made Online

CALL FOR PARTICIPATION
The 4th International Summer School on Pattern Recognition will be organised at 
the University of Plymouth, UK (23-28 July, 2006). The  school programme is 
listed below. Please pass on this email to your interested colleagues and 
students. This is a  great summer school which I would recommend for everyone 
to attend. 
DEADLINE: Register BEFORE 25 MAY, 2006 through the website to get a discount on 
the fee. Limited seats left!

Speakers at the Summer School (ISSPR'2006)
Dr. Sam Roberts Mathworks, UK (Introduction to Matlab)
Prof. Wojtek Krzanowski University of Exeter, UK (Multivariate Statistics: Data 
Description)
Dr. Mike Tipping, Microsoft Research, UK (Bayesian Pattern Recognition: 
Principles and Practice)
Prof. Chris Bishop, Microsoft Research, UK (Latent Variables, Mixture Models 
and EM)
Dr. Richard Everson, University of Exeter, UK (Dimensionality Reduction)
Dr. Peter Tino  University of Birmingham, UK (Probabilistic framework for 
model-based topographic map formation)
Prof. Chris Williams, University of Edinburgh, UK (Neural Networks and Kernel 
Machines) 
Dr. Colin Campbell, University of Bristol, UK (Support Vector Machines and 
Kernel Methods: An Introduction and  Review)
Prof. John Shawe- Taylor, University of Southampton, UK (Kernel Based Methods)
Dr. Steve Gunn, University of Southampton, UK (Matlab for Support Vector 
Machines)
Prof. Mahesan Niranjan, University of Sheffield, UK (Classifier Performance 
Particle Filters for Tracking and  Sequential Problems) 
Dr. Andrew Webb Qinetiq, UK (Decision TreesData Clustering)
Prof. Xin Yao, University of Birmingham, UK (A Gentle Introduction to 
Evolutionary Computation; ATutorial on  Evolutionary Multi-objective 
Optimisation) 
Dr. Richard Deardon, University of Birmingham, UK (Sequential Decision Making; 
Markov Chains Monte Carlo Methods)
Dr. Jeremy Wyatt, University of Birmingham, UK (An Introduction to 
Reinforcement Learning)
Dr. Ludmila Kuncheva, University of Wales, UK   (Classifier Combination)
Prof. Joseph Kittler, University of Surrey, UK  (Feature Selection and 
Extraction)
Prof. Sameer Singh, Loughborough University, UK (Multiresolution Pattern 
Recognition)
Prof. Susan Craw, Robert Gordon University, UK  (Case Based Reasoning)
SUPPORTED BY: Microsoft, Springer, British Computer Society, and Mathworks. 
___
IEEE CVPR CONFERENCE
www.CVPR.org/2006
17-22 June, 2006
Registration Deadline is now 2nd of June, 2006
___
18th INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION 
www.comp.hkbu.edu.hk/~icpr06
August 20-24, 2006
Early Bird Registration: 31 May, 2006
__
Please see the event websites to get FULL information.

Compiled by: Dr. Heather Mclleland 
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Building truth tables

[andrea]

On 26 Set, 20:01, "Aaron \"Castironpi\" Brady" <[EMAIL PROTECTED]>
wrote:
>
> Good idea.  If you want prefixed operators: 'and( a, b )' instead of
> 'a and b', you'll have to write your own.  ('operator.and_' is bitwise
> only.)  It may be confusing to mix prefix with infix: 'impl( a and b,
> c )', so you may want to keep everything prefix, but you can still use
> table( f, n ) like Tim said.

After a while I'm back, thanks a lot, the truth table creator works,
now I just want to parse some strings to make it easier to use.

Like

(P \/ Q) -> S == S

Must return a truth table 2^3 lines...

I'm using pyparsing and this should be really simple, but it doesn't
allow me to recurse and that makes mu stuck.
The grammar BNF is:

Var :: = [A..Z]
Exp ::= Var | !Exp  | Exp \/ Exp | Exp -> Exp | Exp /\ Exp | Exp ==
Exp

I tried different ways but I don't find a smart way to get from the
recursive bnf grammar to the implementation in pyparsing...
Any hint?
--
http://mail.python.org/mailman/listinfo/python-list

Re: Creating huge data in very less time.

[andrea]

On 31 Mar, 12:14, "venutaurus...@gmail.com" 
wrote:
>
> That time is reasonable. The randomness should be in such a way that
> MD5 checksum of no two files should be the same.The main reason for
> having such a huge data is for doing stress testing of our product.


In randomness is not necessary (as I understood) you can just create
one single file and then modify one bit of it iteratively for 1000
times.
It's enough to make the checksum change.

Is there a way to create a file to big withouth actually writing
anything in python (just give me the garbage that is already on the
disk)?
--
http://mail.python.org/mailman/listinfo/python-list

Building truth tables

[andrea]

Well I would like to make a little program that given a certain
logical expression gives the complete truth table.

It's not too difficult in fact, I just have some doubts on how to
design it.

I thought something like that:

class Term:

class Table:

   def and(...
   def or(...


But I'm not convinced...
I would like something like that more or less:
a,b,c = Term()

table([a,b,c], impl(and(or(a,b)),c))

Any idea??
Thanks
--
http://mail.python.org/mailman/listinfo/python-list

pip and different branches?

[andrea crotti]

We use github and we work on many different branches at the same time.

The problem is that we have >5 repos now, and for each repo we might
have the same branches on all of them.

Now we use pip and install requirements such as:
git+ssh://g...@github.com/repo.git@dev

Now the problem is that the requirements file are also under revision
control, and constantly we end up in the situation that when we merge
branches the branch settings get messed up, because we forget to change
them.

I was looking for a solution for this that would allow me to:
- use the branch of the "main" repo for all the dependencies
- fallback on master if that branch doesn't exist

I thought about a few options:
1. create a wrapper for PIP that manipulates the requirement file, that now
   would become templates.
   In this way I would have to know however if a branch exist or not,
   and I didn't find a way to do that without cloning the repo.

2. modify PIP to not fail when checking out a non existing branch, so
   that if it's not found it falls back on master automatically.

3. use some git magic hooks but I'm not sure what exactly

4. stop using virtualenv + pip and use something smarter that handles
   this.

Any suggestions?
Thanks
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Getting a callable for any value?

[andrea crotti]

On 05/29/2013 06:46 PM, Croepha wrote:

Is there anything like this in the standard library?

class AnyFactory(object):
def __init__(self, anything):
self.product = anything
def __call__(self):
return self.product
def __repr__(self):
return "%s.%s(%r)" % (self.__class__.__module__, 
self.__class__.__name__, self.product)


my use case is: 
collections.defaultdict(AnyFactory(collections.defaultdict(AnyFactory(None




I think I would scratch my head for a good half an hour if I see a 
string like this, so I hope there isn't anything in the standard library 
to do that :D
-- 
http://mail.python.org/mailman/listinfo/python-list

decorator to fetch arguments from global objects

[andrea crotti]

Using a CouchDB server we have a different database object potentially for
every request.

We already set that db in the request object to make it easy to pass it
around form our django app, however it would be nice if I could set it once
in the API and automatically fetch it from there.

Basically I have something like

class Entity:
 def save_doc(db)
...

I would like basically to decorate this function in such a way that:
- if I pass a db object use it
- if I don't pass it in try to fetch it from a global object
- if both don't exist raise an exception

Now it kinda of works already with the decorator below.
The problem is that the argument is positional so I end up maybe passing it
twice.
So I have to enforce that 'db' if there is passed as first argument..

It would be a lot easier removing the db from the arguments but then it
would look too magic and I didn't want to change the signature.. any other
advice?

def with_optional_db(func):
"""Decorator that sets the database to the global current one if
not passed in or if passed in and None
"""
@wraps(func)
def _with_optional_db(*args, **kwargs):
func_args = func.func_code.co_varnames
db = None
# if it's defined in the first elements it needs to be
# assigned to *args, otherwise to kwargs
if 'db' in func_args:
assert 'db' == func_args[0], "Needs to be the first defined"
else:
db = kwargs.get('db', None)

if db is None:
kwargs['db'] = get_current_db()

assert kwargs['db'] is not None, "Can't have a not defined database"
ret = func(*args, **kwargs)
return ret

return _with_optional_db
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python-django for dynamic web survey?

[andrea crotti]

Django makes your life a lot easier in many ways, but you still need some
time to learn it.
The task you're trying it's not trivial though, depending on your
experience it might take a while with any library/framework..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: decorator to fetch arguments from global objects

[andrea crotti]

2013/6/18 Wolfgang Maier 

> andrea crotti  gmail.com> writes:
>
> >
> >
> > Using a CouchDB server we have a different database object potentially
> for
> every request.
> >
> > We already set that db in the request object to make it easy to pass it
> around form our django app, however it would be nice if I could set it once
> in the API and automatically fetch it from there.
> >
> > Basically I have something like
> >
> > class Entity:
> >  def save_doc(db)
> > ...
> >
> > I would like basically to decorate this function in such a way that:
> > - if I pass a db object use it
> > - if I don't pass it in try to fetch it from a global object
> > - if both don't exist raise an exception
> >
> > Now it kinda of works already with the decorator below.
> > The problem is that the argument is positional so I end up maybe passing
> it twice.
> > So I have to enforce that 'db' if there is passed as first argument..
> >
> > It would be a lot easier removing the db from the arguments but then it
> would look too magic and I didn't want to change the signature.. any other
> advice?
> >
> > def with_optional_db(func):
> > """Decorator that sets the database to the global current one if
> > not passed in or if passed in and None
> > """
> >   wraps(func)
> > def _with_optional_db(*args, **kwargs):
> > func_args = func.func_code.co_varnames
> > db = None
> > # if it's defined in the first elements it needs to be
> > # assigned to *args, otherwise to kwargs
> > if 'db' in func_args:
> > assert 'db' == func_args[0], "Needs to be the first defined"
> > else:
> > db = kwargs.get('db', None)
> >
> > if db is None:
> > kwargs['db'] = get_current_db()
> >
> > assert kwargs['db'] is not None, "Can't have a not defined
> database"
> > ret = func(*args, **kwargs)
> > return ret
> >
> > return _with_optional_db
> >
>
> I'm not sure, whether your code would work. I get the logic for the db in
> kwargs case, but why are you checking whether db is in func_args? Isn't the
> real question whether it's in args ?? In general, I don't understand why
> you
> want to use .func_code.co_varnames here. You know how you defined your
> function (or rather method):
> class Entity:
> def save_doc(db):
> ...
> Maybe I misunderstood the problem?
> Wolfgang
>
>
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>


Well the point is that I could allow someone to not use "db" as argument of
the function if he only wants to use the global db object..

Or at least I want to check that it's the first argument and not in another
position, just as a sanity check.

I might drop some magic and make it a bit simpler though, even the default
argument DEFAULT_DB could be actually good, and I would not even need the
decorator at that point..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: decorator to fetch arguments from global objects

[andrea crotti]

2013/6/18 Terry Reedy 

> On 6/18/2013 5:47 AM, andrea crotti wrote:
>
>> Using a CouchDB server we have a different database object potentially
>> for every request.
>>
>> We already set that db in the request object to make it easy to pass it
>> around form our django app, however it would be nice if I could set it
>> once in the API and automatically fetch it from there.
>>
>> Basically I have something like
>>
>> class Entity:
>>   def save_doc(db)
>>
>
> If save_doc does not use an instance of Entity (self) or Entity itself
> (cls), it need not be put in the class.


I missed a self it's a method actually..


>
>
>   ...
>>
>> I would like basically to decorate this function in such a way that:
>> - if I pass a db object use it
>> - if I don't pass it in try to fetch it from a global object
>> - if both don't exist raise an exception
>>
>
> Decorators are only worthwhile if used repeatedly. What you specified can
> easily be written, for instance, as
>
> def save_doc(db=None):
>   if db is None:
> db = fetch_from_global()
>   if isinstance(db, dbclass):
> save_it()
>   else:
> raise ValueError('need dbobject')
>
>
>
Yes that's exactly why I want a decorator, to avoid all this boilerplate
for every function method that uses a db object..
-- 
http://mail.python.org/mailman/listinfo/python-list

using SQLalchemy

[andrea crotti]

We have a very chaotic database (on MySql) at the moment, with for
example table names used as keys to query other tables (but that's just
an example).

We are going to redesign it but first I still have to replace the
perl+vbscript system with only one Python program, so I still have to
deal with the current state.

I'm trying to use SQLalchemy and it looks absolutely great, but in
general as a policy we don't use external dependencies..

To try to do an exception in this case:
- are there any problems with SQLalchemy on Windows?
- are there any possibly drawbacks of using SQLalchemy instead of the
  MySqlDB interface?

  For the second point I guess that we might have a bit less fine
  tuning, but the amount of data is not so much and speed is also not a
  bit issue (also because all the queries are extremely inefficient
  now).

  Any other possible issue?

Thanks,
Andrea
-- 
http://mail.python.org/mailman/listinfo/python-list

retry many times decorator

[andrea crotti]

Hi everyone, I'm replacing a perl system that has to work a lot with
databases and perforce (and a few other things).

This script have to run completely unsupervisioned, so it's important
that it doesn't just quit at the first attempt waiting for human
intervent..

They say that the network sometimes has problems so all over the code
there are things like:
until ($dbh = DBI->connect('...'))
{
sleep( 5 * 60 );
)


Since I really don't want to do that I tried to do a decorator:

class retry_n_times:
def __init__(self, ntimes=3, timeout=3, fatal=True):
self.ntimes = ntimes
self.timeout = timeout
self.fatal = fatal

def __call__(self, func):
def _retry_n_times(*args, **kwargs):
attempts = 0
while True:
logger.debug("Attempt number %s of %s" % (attempts,
func.__name__))
ret = func(*args, **kwargs)
if ret:
return ret
else:
sleep(self.timeout)

attempts += 1
if attempts == self.ntimes:
logger.error("Giving up the attempts while running
%s" % func.__name__)
if self.fatal:
exit(100)

return _retry_n_times

which can be used as

@retry_n_times(ntimes=10)
def connect():
try:
conn = mysql_connection()
except Exception:
return False
else:
return True


So the function to be decorated has to return a boolean..  The problem
is that I would like to keep the exception message to report a bit
better what could be the problem, in case the retry fails.

Any idea about how to improve it?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: retry many times decorator

[andrea crotti]

> Returning a boolean isn't very Pythonic. It would be better, IMHO, if
> it could swallow a specified exception (or specified exceptions?)
> raised when an attempt failed, up to the maximum permitted number of
> attempts. If the final attempt fails, propagate the exception.
> --
> http://mail.python.org/mailman/listinfo/python-list


Yes right I also didn't like it..  Now it become something as below,
so I capture every possible exception (since it must be generic) and
log what exception was raised.  I'm not re-raising because if it fails
and it's fatal I should just exit, and if it's not fatal it should
just continue..

class retry_n_times:
def __init__(self, ntimes=3, timeout=3, fatal=True):
self.ntimes = ntimes
self.timeout = timeout
self.fatal = fatal

def __call__(self, func):
def _retry_n_times(*args, **kwargs):
attempts = 0
while True:
logger.debug("Attempt number %s of %s" % (attempts,
func.__name__))
try:
ret = func(*args, **kwargs)
except Exception as e:
logger.error("Got exception %s with error %s" %
(type(e), str(e)))
sleep(self.timeout)
else:
return ret

attempts += 1
if attempts == self.ntimes:
logger.error("Giving up the attempts while running
%s" % func.__name__)
if self.fatal:
exit(100)

return _retry_n_times
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: retry many times decorator

[Andrea Crotti]

On 06/28/2012 06:43 PM, Steven D'Aprano wrote:

On Thu, 28 Jun 2012 17:26:36 +0100, andrea crotti wrote:


I disagree. If you make a coding error in your function, why do you think
it is useful to retry that buggy code over and over again? It's never
going to get less buggy unless you see the exception and fix the bug.

For any operation that you want to retry, identify the *temporary*
errors, catch them, and retry the request. *Permanent* errors should
immediately fail, without retrying. *Unexpected* errors should not be
caught, since they probably represent a bug in your code.


Ah well maybe I wasn't clear, but I'm not going to retry random things, 
I will only decorate
the functions that I know for sure that could go wrong for temporary 
network problems.


For example they told me that sometimes mysql just doesn't respond in 
time for some reasons,

but there's nothing permanently wrong, so retrying is the best option..

It would be good of course, however, to catch the exceptions that are 
known to be permanent problems

in the function at least, and leave the retry as last resource..

Thanks for the idea of the exponential backoff, which is also a better 
name than timeout for the variable..

--
http://mail.python.org/mailman/listinfo/python-list

Re: retry many times decorator

[andrea crotti]

On the other hand now that I think again even supposing there is a
permanent error like MySql completely down, retrying continuosly
won't do any harm anyway because the machine will not be able to do
anything else anyway, when someone will fix MySql it would
restart again without human intervention.

So I think I could even just let it retry and use maybe a SMTPHanlder
for the logging errors, to make the notification of problems very
quick..
-- 
http://mail.python.org/mailman/listinfo/python-list

adding a simulation mode

[andrea crotti]

I'm writing a program which has to interact with many external
resources, at least:
- mysql database
- perforce
- shared mounts
- files on disk

And the logic is quite complex, because there are many possible paths to
follow depending on some other parameters.
This program even needs to run on many virtual machines at the same time
so the interaction is another thing I need to check...

Now I successfully managed to mock the database with sqlalchemy and only
the fields I actually need, but I now would like to simulate also
everything else.

I would like for example that if I simulate I can pass a fake database,
a fake configuration and get the log of what exactly would happen.  But
I'm not sure how to implement it now..  One possibility would be to have
a global variable (PRETEND_ONLY = False) that if set should be checked
before every potentially system-dependent command.

For example

copytree(src, dest) becomes:
if not PRETEND_ONLY:
copytree(src, dest)

But I don't like it too much because I would have to add a lot of
garbage around..

Another way is maybe to set the sys.excepthook to something that catchs
all the exceptions that would be thrown by these comands, but I'm not
sure is a good idea either..

Any suggestion?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: adding a simulation mode

[andrea crotti]

2012/7/4 Steven D'Aprano :
>
> Then, have your code accept the thing as an argument.
>
> E.g. instead of having a hard-coded database connection, allow the
> database connection to be set (perhaps as an argument, perhaps as a
> config option, etc.).
>
> There are libraries to help with mocks, e.g.:
>
> http://garybernhardt.github.com/python-mock-comparison/

Ah yes this part is already done, I pass an object to the entry point
of the program which represents the database connection, which looks
like this:

class MockMysqlEngine(MySqlEngine):
# TODO: make the engine more generic would avoid this dirty hack
def __init__(self, *args, **kwargs):
# self.engine =
create_engine('sqlite:home/andrea/testdb.sqlite', echo=True)
self.engine = create_engine('sqlite://', echo=True)
self.meta = MetaData(bind=self.engine)
self.session_maker = sessionmaker(bind=self.engine)


Now I populate statically the schema and populate with some test data
too, but I'm also implementing a weay to just pass some CSV files so
that other people can easily write some test cases with some other
possible database configurations.

(And I use mock for a few other things)


>
>
>> For example
>>
>> copytree(src, dest) becomes:
>> if not PRETEND_ONLY:
>> copytree(src, dest)
>
> Ewww :(
>
> Mocking the file system is probably the hardest part, because you
> generally don't have a "FileSystem" object available to be replaced. In
> effect, your program has one giant global variable, the file system.
> Worse, it's not even a named variable, it's hard-coded everywhere you use
> it.
>
> I don't know of any good solution for that. I've often thought about it,
> but don't have an answer.
>
> I suppose you could monkey-patch a bunch of stuff:
>
> if ONLY_PRETEND:
>  open = my_mock_open
>  copytree = my_mock_copytree
>  # etc.
> main()  # run your application
>
>
>
> but that would also be painful.
>

Yes there is no easy solution apparently..  But I'm also playing
around with vagrant and virtual machine generations, suppose I'm able
to really control what will be on the machine at time X, creating it
on demand with what I need, it might be a good way to solve my
problems (a bit overkill and slow maybe though).

I'll try the sys.excepthook trick first, any error should give me an
exception, so if I catch them all I think it might work already..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: adding a simulation mode

[andrea crotti]

> Yes there is no easy solution apparently..  But I'm also playing
> around with vagrant and virtual machine generations, suppose I'm able
> to really control what will be on the machine at time X, creating it
> on demand with what I need, it might be a good way to solve my
> problems (a bit overkill and slow maybe though).
>
> I'll try the sys.excepthook trick first, any error should give me an
> exception, so if I catch them all I think it might work already..


I actually thought that the sys.excepthook would be easy but it's not
so trivial apparently:
This simple sample never reaches the print("here"), because even if
the exception is catched it still quits with return code=1.

I also tried to catch the signal but same result, how do I make it
continue and just don't complain?

The other option if of course to do a big try/except, but I would
prefer the excepthook solution..


import sys
from shutil import copy

def my_except_hook(etype, value, tb):
print("got an exception of type", etype)


if __name__ == '__main__':
sys.excepthook = my_except_hook
copy('sdflsdk')
print("here")
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: adding a simulation mode

[andrea crotti]

2012/7/5 Dieter Maurer :
>
> There is a paradigm called "inversion of control" which can be used
> to handle those requirements.
>
> With "inversion of control", the components interact on the bases
> of interfaces. The components themselves do not know each other, they
> know only the interfaces they want to interact with. For the interaction
> to really take place, a component asks a registry "give me a component
> satisfying this interface", gets it and uses the interface.
>
> If you follow this paradigm, it is easy to switch components: just
> register different alternatives for the interface at hand.
>
>
> "zope.interface" and "zope.component" are python packages that
> support this paradigm. Despite the "zope" in their name, they can be
> used outside of "Zope".
>
> "zope.interface" models interfaces, while "zope.component" provides
> so called "utilities" (e.g. "database utility", "filesystem utility", ...)
> and "adapters" and the corresponding registries.
>
>
> Of course, they contain only the infrastructure for the "inversion of control"
> paradigm. Up to you to provide the implementation for the various
> mocks.
>


Thanks that's a good point, in short I could do something like:


class FSActions:
  @classmethod
  def copy_directories(cls, src, dest)
  raise NotImplementedError

  @classmethod
  

And then have different implementations of this interface.  This would
work, but I don't really like the idea of constructing interfaces that
provide only the few things I need.

Instead of being good APIs they might become just random
functionalities put together to make my life easier, and at that point
maybe just some clear mocking might be even better..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: stuck in files!!

[andrea crotti]

2012/7/6 Chirag B :
> i want to kno how to link two applications using python for eg:notepad
> txt file and some docx file. like i wat to kno how to take path of
> those to files and run them simultaneously.like if i type something in
> notepad it has to come in wordpad whenever i run that code.
> --
> http://mail.python.org/mailman/listinfo/python-list

I don't think that "I want to know" ever lead to some useful answers,
it would be not polite even if
people were actually paid to answer ;)

Anyway it's quite an application-os specific question, probably not
very easy either..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: adding a simulation mode

[andrea crotti]

One thing that I don't quite understand is why some calls even if I
catch the exception still makes the whole program quit.
For example this

try:
copytree('sjkdf', 'dsflkj')
Popen(['notfouhd'], shell=True)
except Exception as e:
print("here")


behaves differently from:

try:
Popen(['notfouhd'], shell=True)
copytree('sjkdf', 'dsflkj')
except Exception as e:
print("here")

because if copytree fails it quits anyway.
I also looked at the code but can't quite get why.. any idea?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: adding a simulation mode

[andrea crotti]

One way instead that might actually work is this

def default_mock_action(func_name):
def _default_mock_action(*args, **kwargs):
print("running {} with args {} and {}".format(func_name, args, kwargs))

return _default_mock_action


def mock_fs_actions(to_run):
"""Take a function to run, and run it in an environment which
mocks all the possibly dangerous operations
"""
side_effect = [
'copytree',
'copy',
]

acts = dict((s, default_mock_action(s)) for s in side_effect)

with patch('pytest.runner.commands.ShellCommand.run',
default_mock_action('run')):
with patch.multiple('shutil', **acts):
to_run()


So I can just pass the main function inside the mock like
mock_fs_actions(main)

and it seems to do the job, but I have to list manually all the things
to mock and I'm not sure is the best idea anyway..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: adding a simulation mode

[andrea crotti]

Well that's what I thought, but I can't find any explicit exit
anywhere in shutil, so what's going on there?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: adding a simulation mode

[andrea crotti]

2012/7/12 John Gordon :
> In  andrea crotti 
>  writes:
>
>> Well that's what I thought, but I can't find any explicit exit
>> anywhere in shutil, so what's going on there?
>
> Try catching SystemExit specifically (it doesn't inherit from Exception,
> so "except Exception" won't catch it.)
>
> --

Ah yes that actually works, but I think is quite dodgy, why was it
done like this?
In shutil there is still no mention of SystemExit, and trying to raise
the single exceptions by and
doens't still make it exit, so I would still like to know how it is
possible just for curiosity..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: adding a simulation mode

[andrea crotti]

2012/7/13 Steven D'Aprano :

> Well of course it does. If copytree fails, the try block ends and
> execution skips straight to the except block, which runs, and then the
> program halts because there's nothing else to be done.
>
> That at least is my guess, based on the described symptoms.
>

Well I think that's what I was stupidly missing, I always had only one
possibly failing thing in a try/except block,
and I always gave for granted that it doesn't jump to the except block
on first error, but of course it makes
more sense if it does...

Thanks a lot
-- 
http://mail.python.org/mailman/listinfo/python-list

assertraises behaviour

[andrea crotti]

I found that the behaviour of assertRaises used as a context manager a
bit surprising.

This small example doesn't fail, but the OSError exception is cathed
even if not declared..
Is this the expected behaviour (from the doc I would say it's not).
(Running on arch-linux 64 bits and Python 2.7.3, but it doesn the same
with Python 3.2.3)

import unittest

class TestWithRaises(unittest.TestCase):
def test_ass(self):
with self.assertRaises(AssertionError):
assert False, "should happen"
raise OSError("should give error")


if __name__ == '__main__':
unittest.main()
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: assertraises behaviour

[andrea crotti]

2012/7/16 Christian Heimes :
>
> The OSError isn't catched as the code never reaches the line with "raise
> OSError". In other words "raise OSError" is never executed as the
> exception raised by "assert False" stops the context manager.
>
> You should avoid testing more than one line of code in a with
> self.assertRaises() block.
>
> Christian
>
> --
> http://mail.python.org/mailman/listinfo/python-list

Ok now it's more clear, and normally I only test one thing in the block.
But I find this quite counter-intuitive, because in the block I want
to be able to run something that raises the exception I catch (and
which I'm aware of), but if another exception is raised it *should*
show it and fail in my opinion..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: assertraises behaviour

[andrea crotti]

Good thanks, but there is something that implements this behaviour..
For example nose runs all the tests, and if there are failures it goes
on and shows the failed tests only in the end, so I think it is
possible to achieve somehow, is that correct?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Diagramming code

[Andrea Crotti]

On 07/16/2012 02:26 AM, hamilton wrote:

Is there any software to help understand python code ?

Thanks

hamilton


Sometimes to get some nice graphs I use gprof2dot 
(http://code.google.com/p/jrfonseca/wiki/Gprof2Dot)

or doxygen (http://www.stack.nl/~dimitri/doxygen/)

gprof2dot analyses the output of the profiling that you get running the 
code through the python profiler,

doing for example:

python -m cProfile -o $STATS $FNAME $@
$GPROF2DOT -f pstats $STATS | dot -T$TYPE -o $OUT

doxygen is more useful for C++ but it's also able to infer a few things 
(without running) from a python project..

--
http://mail.python.org/mailman/listinfo/python-list

Re: assertraises behaviour

[andrea crotti]

2012/7/16 Peter Otten <__pete...@web.de>:
> No, I don't see how the code you gave above can fail with an OSError.
>
> Can you give an example that produces the desired behaviour with nose? Maybe
> we can help you translate it to basic unittest.
>
> --
> http://mail.python.org/mailman/listinfo/python-list


Well this is what I meant:

import unittest

class TestWithRaises(unittest.TestCase):
def test_first(self):
assert False

def test_second(self):
print("also called")
assert True

if __name__ == '__main__':
unittest.main()

in this case also the second test is run even if the first fails..
But that's probably easy because we just need to catch exceptions for
every method call, so it's not exactly the same thing..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: assertraises behaviour

[Andrea Crotti]

To clarify my "problem", I just thought that assertRaises if used as 
context manager should behave as following:

- keep going if the exception declared is raised
- re-raise an error even if catched after the declared exception was catched

I was also confused by the doc a bit:
"Test that an exception is raised when /callable/ is called with any 
positional or keyword arguments that are also passed to assertRaises() 
. 
The test passes if /exception/ is raised, is an error if another 
exception is raised, or fails if no exception is raised"


which speaks about when it's not used as a context manager..

I understand why it's not possible and it's not a big issue though..
-- 
http://mail.python.org/mailman/listinfo/python-list

reloading code and multiprocessing

[andrea crotti]

We need to be able to reload code on a live system.  This live system
has a daemon process always running but it runs many subprocesses with
multiprocessing, and the subprocesses might have a short life...

Now I found a way to reload the code successfully, as you can see from
this testcase:


def func():
from . import a
print(a.ret())


class TestMultiProc(unittest.TestCase):
def setUp(self):
open(path.join(cur_dir, 'a.py'), 'w').write(old_a)

def tearDown(self):
remove(path.join(cur_dir, 'a.py'))

def test_reloading(self):
"""Starting a new process gives a different result
"""
p1 = Process(target=func)
p2 = Process(target=func)
p1.start()
res = p1.join()
open(path.join(cur_dir, 'a.py'), 'w').write(new_a)
remove(path.join(cur_dir, 'a.pyc'))

p2.start()
res = p2.join()


As long as I import the code in the function and make sure to remove the
"pyc" files everything seems to work..
Are there any possible problems which I'm not seeing in this approach or
it's safe?

Any other better ways otherwise?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: reloading code and multiprocessing

[andrea crotti]

2012/7/20 Chris Angelico :
> On Thu, Jul 19, 2012 at 8:15 PM, andrea crotti
>  wrote:
>> We need to be able to reload code on a live system.  This live system
>> has a daemon process always running but it runs many subprocesses with
>> multiprocessing, and the subprocesses might have a short life...
>> ...
>> As long as I import the code in the function and make sure to remove the
>> "pyc" files everything seems to work..
>> Are there any possible problems which I'm not seeing in this approach or
>> it's safe?
>
> Python never promises reloading reliability, but from my understanding
> of what you've done here, it's probably safe. However, you may find
> that you're using the wrong language for the job; it depends on how
> expensive it is to spin off all those processes and ship their work to
> them. But if that's not an issue, I'd say you have something safe
> there. (Caveat: I've given this only a fairly cursory examination, and
> I'm not an expert. Others may have more to say. I just didn't want the
> resident Markov chainer to be the only one to respond!!)
>
> ChrisA
> --
> http://mail.python.org/mailman/listinfo/python-list


Thanks Chris, always nice to get a "human" answer ;)

Anyway the only other problem which I found is that if I start the
subprocesses after many other things are initialised, it might happen
that the reloading doesn't work correctly, is that right?

Because sys.modules will get inherited from the subprocesses and it
will not reimport what has been already imported as far as I
understood..

So or I make sure I import everything only where it is needed or (and
maybe better and more explicit) I remove manually from sys.modules all
the modules that I want to reload, what do you think?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: reloading code and multiprocessing

[andrea crotti]

2012/7/23 Chris Angelico :
>
> That would probably be correct. However, I still think you may be
> fighting against the language instead of playing to its strengths.
>
> I've never fiddled with sys.modules like that, but I know some have,
> without problem.
>
> ChrisA
> --
> http://mail.python.org/mailman/listinfo/python-list


I would also like to avoid this in general, but we have many
subprocesses to launch and some of them might take weeks, so we need
to have a process which is always running, because there is never a
point in time where we can just say let's stop everything and start again..

Anyway if there are better solutions I'm still glad to hear them, but
I would also like to keep it simple..

Another thing which now we need to figure out is how to communicate
with the live process..  For example we might want to submit something
manually, which should pass from the main process.

The first idea is to have a separate process that opens a socket and
listens for data on a local port, with a defined protocol.

Then the main process can parse these commands and run them.
Are there easier ways otherwise?
-- 
http://mail.python.org/mailman/listinfo/python-list

Dumping all the sql statements as backup

[andrea crotti]

I have some long running processes that do very long simulations which
at the end need to write things on a database.

At the moment sometimes there are network problems and we end up with
half the data on the database.

The half-data problem is probably solved easily with sessions and
sqlalchemy (a db-transaction), but still we would like to be able to
keep a backup SQL file in case something goes badly wrong and we want to
re-run it manually..

This might also be useful if we have to rollback the db for some reasons
to a previous day and we don't want to re-run the simulations..

Anyone did something similar?
It would be nice to do something like:

with CachedDatabase('backup.sql'):
# do all your things
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Dumping all the sql statements as backup

[andrea crotti]

2012/7/25 Jack  Since you know the content of what the sql code is, why not just build
> the sql file(s) needed and store them so that in case of a burp you can
> just execute the code file. If you don't know the exact sql code, dump
> it to a file as the statements are constructed... The only problem you
> would run into in this scenario is duplicate data, which is also easily
> solvable by using transaction-level commits to the db.
> --
> http://mail.python.org/mailman/listinfo/python-list


Yes but how do I construct them with SqlAlchemy?
One possible option I found is to enable the logging of some parts of
SqlAlchemy, and use that log, (echo=True in create_engine does
something similar) but maybe there is a better option..

But I need to filter only the insert/update/delete probably..

And in general the processes have to run independently so in case of
database connection problems I would just let them retry until it
actually works.

When the transaction actually works then in the backed up log I can
add a marker(or archive the log), to avoid replaying it.
-- 
http://mail.python.org/mailman/listinfo/python-list

regexps to objects

[andrea crotti]

I have some complex input to parse (with regexps), and I would like to
create nice objects directy from them.
The re module doesn't of course try to conver to any type, so I was
playing around to see if it's worth do something as below, where I
assign a constructor to every regexp and build an object from the
result..

Do you think it makes sense in general or how do you cope with this problem?

import re
from time import strptime
TIME_FORMAT_INPUT = '%m/%d/%Y %H:%M:%S'

def time_string_to_obj(timestring):
return strptime(timestring, TIME_FORMAT_INPUT)


REGEXPS = {
'num': ('\d+', int),
'date': ('[0-9/]+ [0-9:]+', time_string_to_obj),
}


def reg_to_obj(reg, st):
reg, constr = reg
found = re.match(reg, st)
return constr(found.group())


if __name__ == '__main__':
print reg_to_obj(REGEXPS['num'], '100')
print reg_to_obj(REGEXPS['date'], '07/24/2012 06:23:13')
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: reloading code and multiprocessing

[andrea crotti]

2012/7/25 andrea crotti :
>
> I would also like to avoid this in general, but we have many
> subprocesses to launch and some of them might take weeks, so we need
> to have a process which is always running, because there is never a
> point in time where we can just say let's stop everything and start again..
>
> Anyway if there are better solutions I'm still glad to hear them, but
> I would also like to keep it simple..
>
> Another thing which now we need to figure out is how to communicate
> with the live process..  For example we might want to submit something
> manually, which should pass from the main process.
>
> The first idea is to have a separate process that opens a socket and
> listens for data on a local port, with a defined protocol.
>
> Then the main process can parse these commands and run them.
> Are there easier ways otherwise?


So I was trying to do this, removing the module from sys.modules and
starting a new process (after modifying the file), but it doesn't work
as I expected.
The last assertion fails, but how?

The pyc file is not generated, the module is actually not in
sys.modules, and the function doesn't in the subprocess doesn't fail
but still returns the old value.
Any idea?

old_a = "def ret(): return 0"
new_a = "def ret(): return 1"


def func_no_import(queue):
queue.put(a_glob.ret())


class TestMultiProc(unittest.TestCase):

def test_reloading_with_global_import(self):
"""In this case the import is done before the process are started,
so we need to clean sys.modules to make sure we reload everything
"""
queue = Queue()
open(path.join(CUR_DIR, 'old_a.py'), 'w').write(old_a)

p1 = Process(target=func_no_import, args=(queue, ))
p1.start()
p1.join()
self.assertEqual(queue.get(), 0)

open(path.join(CUR_DIR, 'old_a.py'), 'w').write(new_a)
del sys.modules['auto_tester.tests.a_glob']

p2 = Process(target=func_no_import, args=(queue, ))
p2.start()
p2.join()
self.assertEqual(queue.get(), 1)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: py2c - an open source Python to C/C++ is looking for developers

[andrea crotti]

2012/7/30  :
> I created py2c ( http://code.google.com/p/py2c )- an open source Python to 
> C/C++ translator!
> py2c is looking for developers!
> To join create a posting in the py2c-discuss Google Group or email me!
> Thanks
> PS:I hope this is the appropiate group for this message.
> --
> http://mail.python.org/mailman/listinfo/python-list

It looks like a very very hard task, and really useful or for exercise?

The first few lines I've seen there are the dangerous * imports and
LazyStrin looks like a typo..

from ast import *
import functools
from c_types import *
from lazystring import *
#constant data
empty = LazyStrin
ordertuple = ((Or,),(And
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pass data to a subprocess

[andrea crotti]

>
>
> def procs():
> mp = MyProcess()
> # with the join we are actually waiting for the end of the running time
> mp.add([1,2,3])
> mp.start()
> mp.add([2,3,4])
> mp.join()
> print(mp)
>

I think I got it now, if I already just mix the start before another
add, inside the Process.run it won't see the new data that has been
added after the start.

So this way is perfectly safe only until the process is launched, if
it's running I need to use some multiprocess-aware data structure, is
that correct?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pass data to a subprocess

[andrea crotti]

2012/7/31 Laszlo Nagy :
>> I think I got it now, if I already just mix the start before another add,
>> inside the Process.run it won't see the new data that has been added after
>> the start. So this way is perfectly safe only until the process is launched,
>> if it's running I need to use some multiprocess-aware data structure, is
>> that correct?
>
> Yes. Read this:
>
> http://docs.python.org/library/multiprocessing.html#exchanging-objects-between-processes
>
> You can use Queues and Pipes. Actually, these are basic elements of the
> multiprocessing module and they are well documented. I wonder if you read
> the documentation at all, before posting questions here.
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list


As I wrote "I found many nice things (Pipe, Manager and so on), but
actually even
this seems to work:" yes I did read the documentation.

I was just surprised that it worked better than I expected even
without Pipes and Queues, but now I understand why..

Anyway now I would like to be able to detach subprocesses to avoid the
nasty code reloading that I was talking about in another thread, but
things get more tricky, because I can't use queues and pipes to
communicate with a running process that it's noit my child, correct?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pass data to a subprocess

[andrea crotti]

2012/8/1 Laszlo Nagy :
>> I was just surprised that it worked better than I expected even
>> without Pipes and Queues, but now I understand why..
>>
>> Anyway now I would like to be able to detach subprocesses to avoid the
>> nasty code reloading that I was talking about in another thread, but
>> things get more tricky, because I can't use queues and pipes to
>> communicate with a running process that it's noit my child, correct?
>>
> Yes, I think that is correct. Instead of detaching a child process, you can
> create independent processes and use other frameworks for IPC. For example,
> Pyro.  It is not as effective as multiprocessing.Queue, but in return, you
> will have the option to run your service across multiple servers.
>
> The most effective IPC is usually through shared memory. But there is no OS
> independent standard Python module that can communicate over shared memory.
> Except multiprocessing of course, but AFAIK it can only be used to
> communicate between fork()-ed processes.


Thanks, there is another thing which is able to interact with running
processes in theory:
https://github.com/lmacken/pyrasite

I don't know though if it's a good idea to use a similar approach for
production code, as far as I understood it uses gdb..  In theory
though I could be able to set up every subprocess with all the data
they need, so I might not even need to share data between them.

Anyway now I had another idea to avoid to be able to stop the main
process without killing the subprocesses, using multiple forks.  Does
the following makes sense?  I don't really need these subprocesses to
be daemons since they should quit when done, but is there anything
that can go wrong with this approach?

from os import fork
from time import sleep
from itertools import count
from sys import exit

from multiprocessing import Process, Queue

class LongProcess(Process):
def __init__(self, idx, queue):
Process.__init__(self)
# self.daemon = True
self.queue = queue
self.idx = idx

def run(self):
for i in count():
self.queue.put("%d: %d"  % (self.idx, i))
print("adding %d: %d"  % (self.idx, i))
sleep(2)


if __name__ == '__main__':
qu = Queue()

# how do I do a multiple fork?
for i in range(5):
pid = fork()
# if I create here all the data structures I should still be
able to do things
if pid == 0:
lp = LongProcess(1, qu)
lp.start()
lp.join()
exit(0)
else:
print("started subprocess with pid ", pid)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pass data to a subprocess

[andrea crotti]

2012/8/1 Laszlo Nagy :
> On thing is sure: os.fork() doesn't work under Microsoft Windows. Under
> Unix, I'm not sure if os.fork() can be mixed with
> multiprocessing.Process.start(). I could not find official documentation on
> that.  This must be tested on your actual platform. And don't forget to use
> Queue.get() in your test. :-)
>

Yes I know we don't care about Windows for this particular project..
I think mixing multiprocessing and fork should not harm, but probably
is unnecessary since I'm already in another process after the fork so
I can just make it run what I want.

Otherwise is there a way to do same thing only using multiprocessing?
(running a process that is detachable from the process that created it)
-- 
http://mail.python.org/mailman/listinfo/python-list

CRC-checksum failed in gzip

[andrea crotti]

We're having some really obscure problems with gzip.
There is a program running with python2.7 on a 2.6.18-128.el5xen (red
hat I think) kernel.

Now this program does the following:
if filename == 'out2.txt':
 out2 = open('out2.txt')
elif filename == 'out2.txt.gz'
 out2 = open('out2.txt.gz')

text = out2.read()

out2.close()

very simple right? But sometimes we get a checksum error.
Reading the code I got the following:

 - CRC is at the end of the file and is computed against the whole
file (last 8 bytes)
 - after the CRC there is the \ marker for the EOF
 - readline() doesn't trigger the checksum generation in the
beginning, but only when the EOF is reached
 - until a file is flushed or closed you can't read the new content in it

but the problem is that we can't reproduce it, because doing it
manually on the same files it works perfectly,
and the same files some time work some time don't work.

The files are on a shared NFS drive, I'm starting to think that it's a
network/fs problem, which might truncate the file
adding an EOF before the end and thus making the checksum fail..
But is it possible?
Or what else could it be?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: CRC-checksum failed in gzip

[andrea crotti]

2012/8/1 Laszlo Nagy :
> On 2012-08-01 12:39, andrea crotti wrote:
>>
>> We're having some really obscure problems with gzip.
>> There is a program running with python2.7 on a 2.6.18-128.el5xen (red
>> hat I think) kernel.
>>
>> Now this program does the following:
>> if filename == 'out2.txt':
>>   out2 = open('out2.txt')
>> elif filename == 'out2.txt.gz'
>>   out2 = open('out2.txt.gz')
>
> Gzip file is binary. You should open it in binary mode.
>
> out2 = open('out2.txt.gz',"b")
>
> Otherwise carriage return and newline characters will be converted
> (depending on the platform).
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list


Ah no sorry I just wrote wrong that part of the code, it was
otu2 = gzip.open('out2.txt.gz') because otherwise nothing would possibly work..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: CRC-checksum failed in gzip

[andrea crotti]

Full traceback:

Exception in thread Thread-8:
Traceback (most recent call last):
  File "/user/sim/python/lib/python2.7/threading.py", line 530, in
__bootstrap_inner
self.run()
  File "/user/sim/tests/llif/AutoTester/src/AutoTester2.py", line 67, in run
self.processJobData(jobData, logger)
  File "/user/sim/tests/llif/AutoTester/src/AutoTester2.py", line 204,
in processJobData
self.run_simulator(area, jobData[1] ,log)
  File "/user/sim/tests/llif/AutoTester/src/AutoTester2.py", line 142,
in run_simulator
report_file, percentage, body_text = SimResults.copy_test_batch(log, area)
  File "/user/sim/tests/llif/AutoTester/src/SimResults.py", line 274,
in copy_test_batch
out2_lines = out2.read()
  File "/user/sim/python/lib/python2.7/gzip.py", line 245, in read
self._read(readsize)
  File "/user/sim/python/lib/python2.7/gzip.py", line 316, in _read
self._read_eof()
  File "/user/sim/python/lib/python2.7/gzip.py", line 338, in _read_eof
hex(self.crc)))
IOError: CRC check failed 0x4f675fba != 0xa9e45aL


- The file is written with the linux gzip program.
- no I can't reproduce the error with the same exact file that did
failed, that's what is really puzzling,
  there seems to be no clear pattern and just randmoly fails. The file
is also just open for read from this program,
  so in theory no way that it can be corrupted.

  I also checked with lsof if there are processes that opened it but
nothing appears..

- can't really try on the local disk, might take ages unfortunately
(we are rewriting this system from scratch anyway)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pass data to a subprocess

[andrea crotti]

2012/8/1 Roy Smith :
> In article ,
>  Laszlo Nagy  wrote:
>
>> Yes, I think that is correct. Instead of detaching a child process, you
>> can create independent processes and use other frameworks for IPC. For
>> example, Pyro.  It is not as effective as multiprocessing.Queue, but in
>> return, you will have the option to run your service across multiple
>> servers.
>
> You might want to look at beanstalk (http://kr.github.com/beanstalkd/).
> We've been using it in production for the better part of two years.  At
> a 30,000 foot level, it's an implementation of queues over named pipes
> over TCP, but it takes care of a zillion little details for you.
>
> Setup is trivial, and there's clients for all sorts of languages.  For a
> Python client, go with beanstalkc (pybeanstalk appears to be
> abandonware).
>>
>> The most effective IPC is usually through shared memory. But there is no
>> OS independent standard Python module that can communicate over shared
>> memory.
>
> It's true that shared memory is faster than serializing objects over a
> TCP connection.  On the other hand, it's hard to imagine anything
> written in Python where you would notice the difference.
> --
> http://mail.python.org/mailman/listinfo/python-list


That does look nice and I would like to have something like that..
But since I have to convince my boss of another external dependency I
think it might be worth
to try out zeromq instead, which can also do similar things and looks
more powerful, what do you think?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: CRC-checksum failed in gzip

[andrea crotti]

2012/8/1 Laszlo Nagy :
>>there seems to be no clear pattern and just randmoly fails. The file
>> is also just open for read from this program,
>>so in theory no way that it can be corrupted.
>
> Yes, there is. Gzip stores CRC for compressed *blocks*. So if the file is
> not flushed to the disk, then you can only read a fragment of the block, and
> that changes the CRC.
>
>>
>>I also checked with lsof if there are processes that opened it but
>> nothing appears..
>
> lsof doesn't work very well over nfs. You can have other processes on
> different computers (!) writting the file. lsof only lists the processes on
> the system it is executed on.
>
>>
>> - can't really try on the local disk, might take ages unfortunately
>> (we are rewriting this system from scratch anyway)
>>
>


Thanks a lotl, someone that writes on the file while reading might be
an explanation, the problem is that everyone claims that they are only
reading the file.

Apparently this file is generated once and a long time after only read
by two different tools (in sequence), so this could not be possible
either in theory.. I'll try to investigate more in this sense since
it's the only reasonable explation I've found so far.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pass data to a subprocess

[andrea crotti]

2012/8/1 Laszlo Nagy :
>
> So detaching the child process will not make IPC stop working. But exiting
> from the original parent process will. (And why else would you detach the
> child?)
>
> --
> http://mail.python.org/mailman/listinfo/python-list


Well it makes perfect sense if it stops working to me, so or
- I use zeromq or something similar to communicate
- I make every process independent without the need to further
communicate with the parent..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: CRC-checksum failed in gzip

[andrea crotti]

2012/8/1 Steven D'Aprano :
> On Wed, 01 Aug 2012 14:01:45 +0100, andrea crotti wrote:
>
>> Full traceback:
>>
>> Exception in thread Thread-8:
>
> "DANGER DANGER DANGER WILL ROBINSON!!!"
>
> Why didn't you say that there were threads involved? That puts a
> completely different perspective on the problem.
>
> I *was* going to write back and say that you probably had either file
> system corruption, or network errors. But now that I can see that you
> have threads, I will revise that and say that you probably have a bug in
> your thread handling code.
>
> I must say, Andrea, your initial post asking for help was EXTREMELY
> misleading. You over-simplified the problem to the point that it no
> longer has any connection to the reality of the code you are running.
> Please don't send us on wild goose chases after bugs in code that you
> aren't actually running.
>
>
>>   there seems to be no clear pattern and just randmoly fails.
>
> When you start using threads, you have to expect these sorts of
> intermittent bugs unless you are very careful.
>
> My guess is that you have a bug where two threads read from the same file
> at the same time. Since each read shares state (the position of the file
> pointer), you're going to get corruption. Because it depends on timing
> details of which threads do what at exactly which microsecond, the effect
> might as well be random.
>
> Example: suppose the file contains three blocks A B and C, and a
> checksum. Thread 8 starts reading the file, and gets block A and B. Then
> thread 2 starts reading it as well, and gets half of block C. Thread 8
> gets the rest of block C, calculates the checksum, and it doesn't match.
>
> I recommend that you run a file system check on the remote disk. If it
> passes, you can eliminate file system corruption. Also, run some network
> diagnostics, to eliminate corruption introduced in the network layer. But
> I expect that you won't find anything there, and the problem is a simple
> thread bug. Simple, but really, really hard to find.
>
> Good luck.
>

Thanks a lot, that makes a lot of sense..  I haven't given this detail
before because I didn't write this code, and I forgot that there were
threads involved completely, I'm just trying to help to fix this bug.

Your explanation makes a lot of sense, but it's still surprising that
even just reading files without ever writing them can cause troubles
using threads :/
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: CRC-checksum failed in gzip

[andrea crotti]

2012/8/1 Laszlo Nagy :
>
>> Thanks a lot, that makes a lot of sense..  I haven't given this detail
>> before because I didn't write this code, and I forgot that there were
>> threads involved completely, I'm just trying to help to fix this bug.
>>
>> Your explanation makes a lot of sense, but it's still surprising that
>> even just reading files without ever writing them can cause troubles
>> using threads :/
>
> Make sure that file objects are not shared between threads. If that is
> possible. It will probably solve the problem (if that is related to
> threads).


Well I just have to create a lock I guess right?
with lock:
# open file
# read content
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: CRC-checksum failed in gzip

[andrea crotti]

2012/8/1 Steven D'Aprano :
>
> When you start using threads, you have to expect these sorts of
> intermittent bugs unless you are very careful.
>
> My guess is that you have a bug where two threads read from the same file
> at the same time. Since each read shares state (the position of the file
> pointer), you're going to get corruption. Because it depends on timing
> details of which threads do what at exactly which microsecond, the effect
> might as well be random.
>
> Example: suppose the file contains three blocks A B and C, and a
> checksum. Thread 8 starts reading the file, and gets block A and B. Then
> thread 2 starts reading it as well, and gets half of block C. Thread 8
> gets the rest of block C, calculates the checksum, and it doesn't match.
>
> I recommend that you run a file system check on the remote disk. If it
> passes, you can eliminate file system corruption. Also, run some network
> diagnostics, to eliminate corruption introduced in the network layer. But
> I expect that you won't find anything there, and the problem is a simple
> thread bug. Simple, but really, really hard to find.
>
> Good luck.

One last thing I would like to do before I add this fix is to actually
be able to reproduce this behaviour, and I thought I could just do the
following:

import gzip
import threading


class OpenAndRead(threading.Thread):
def run(self):
fz = gzip.open('out2.txt.gz')
fz.read()
fz.close()


if __name__ == '__main__':
for i in range(100):
OpenAndRead().start()


But no matter how many threads I start, I can't reproduce the CRC
error, any idea how I can try to help it happening?

The code in run should be shared by all the threads since there are no
locks, right?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: CRC-checksum failed in gzip

[andrea crotti]

2012/8/2 Laszlo Nagy :
>
> Your example did not share the file object between threads. Here an example
> that does that:
>
> class OpenAndRead(threading.Thread):
> def run(self):
> global fz
> fz.read(100)
>
> if __name__ == '__main__':
>
>fz = gzip.open('out2.txt.gz')
>for i in range(10):
> OpenAndRead().start()
>
> Try this with a huge file. And here is the one that should never throw CRC
> error, because the file object is protected by a lock:
>
> class OpenAndRead(threading.Thread):
> def run(self):
> global fz
> global fl
> with fl:
> fz.read(100)
>
> if __name__ == '__main__':
>
>fz = gzip.open('out2.txt.gz')
>fl = threading.Lock()
>for i in range(2):
> OpenAndRead().start()
>
>
>>
>> The code in run should be shared by all the threads since there are no
>> locks, right?
>
> The code is shared but the file object is not. In your example, a new file
> object is created, every time a thread is started.
>


Ok sure that makes sense, but then this explanation is maybe not right
anymore, because I'm quite sure that the file object is *not* shared
between threads, everything happens inside a thread..

I managed to get some errors doing this with a big file
class OpenAndRead(threading.Thread):
 def run(self):
 global fz
 fz.read(100)

if __name__ == '__main__':

fz = gzip.open('bigfile.avi.gz')
for i in range(20):
 OpenAndRead().start()

and it doesn't fail without the *global*, but this is definitively not
what the code does, because every thread gets a new file object, it's
not shared..

Anyway we'll read once for all the threads or add the lock, and
hopefully it should solve the problem, even if I'm not convinced yet
that it was this.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: CRC-checksum failed in gzip

[andrea crotti]

2012/8/2 andrea crotti :
>
> Ok sure that makes sense, but then this explanation is maybe not right
> anymore, because I'm quite sure that the file object is *not* shared
> between threads, everything happens inside a thread..
>
> I managed to get some errors doing this with a big file
> class OpenAndRead(threading.Thread):
>  def run(self):
>  global fz
>  fz.read(100)
>
> if __name__ == '__main__':
>
> fz = gzip.open('bigfile.avi.gz')
> for i in range(20):
>  OpenAndRead().start()
>
> and it doesn't fail without the *global*, but this is definitively not
> what the code does, because every thread gets a new file object, it's
> not shared..
>
> Anyway we'll read once for all the threads or add the lock, and
> hopefully it should solve the problem, even if I'm not convinced yet
> that it was this.


Just for completeness as suggested this also does not fail:

class OpenAndRead(threading.Thread):
def __init__(self, lock):
threading.Thread.__init__(self)
self.lock = lock

def run(self):
 global fz
 with self.lock:
 fz.read(100)

if __name__ == '__main__':
lock = threading.Lock()
fz = gzip.open('bigfile.avi.gz')
for i in range(20):
 OpenAndRead(lock).start()
-- 
http://mail.python.org/mailman/listinfo/python-list

Sharing code between different projects?

[andrea crotti]

I am in the situation where I am working on different projects that
might potentially share a lot of code.

I started to work on project A, then switched completely to project B
and in the transiction I copied over a lot of code with the
corresponding tests, and I started to modify it.

Now it's time to work again on project A, but I don't want to copy
things over again.

I would like to design a simple and nice way to share between projects,
where the things I want to share are simple but useful things as for
example:

class TempDirectory:
"""Create a temporary directory and cd to it on enter, cd back to
the original position and remove it on exit
"""
def __init__(self):
self.oldcwd = getcwd()
self.temp_dir = mkdtemp()

def __enter__(self):
logger.debug("create and move to temp directory %s" % self.temp_dir)
return self.temp_dir

def __exit__(self, type, value, traceback):
# I first have to move out
chdir(self.oldcwd)
logger.debug("removing the temporary directory and go back to
the original position %s" % self.temp_dir)
rmtree(self.temp_dir)


The problem is that there are functions/classes from many domains, so it
would not make much sense to create a real project, and the only name I
could give might be "utils or utilities"..

In plus the moment the code is shared I must take care of versioning and
how to link different pieces together (we use perforce by the way).

If then someone else except me will want to use these functions then of
course I'll have to be extra careful, designing really good API's and so
on, so I'm wondering where I should set the trade-off between ability to
share and burden to maintain..

Anyone has suggestions/real world experiences about this?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Sharing code between different projects?

[andrea crotti]

2012/8/13 Rob Day :
> I'd just create a module - called shared_utils.py or similar - and import
> that in both projects. It might be a bit messy if there's no 'unifying
> theme' to the module - but surely it'd be a lot less messy than your
> TempDirectory class, and anyone else who knows Python will understand
> 'import shared_utils' much more easily.
>
> I realise you might not want to say, but if you could give some idea what
> sort of projects these are, and what sorts of code you're trying to share,
> it might make things a bit clearer.
>
> I'm not really sure what your concerns about 'versioning and how to link
> different pieces together' are - what d you think could go wrong here?
>

It's actually not so simple..

Because the two projects live in different parts of the repository
with different people allowed to work on them, and they have to run on
different machines..

In plus I'm using perforce which doesn't have any svn:externals-like
thing as far as I know..  The thing I should do probably is to set up
workspace (which contains *absolute* paths of the machines) with the
right setting to make module available in the right position.

Second problem is that one of the two projects has a quite insane
requirement, which is to be able to re-run itself on a specific
version depending on a value fetched from the database.

This becomes harder if divide code around, but in theory I can use the
changeset number which is like a SVN revision so this should be fine.

The third problem is that from the moment is not just me using these
things, how can I be sure that changing something will not break
someone else code?

I have unit tests on both projects plus the tests for the utils, but
as soon as I separate them it becomes harder to test everything..

So well everything can have a solution probably, I just hope it's
worth the effort..

Another thing which would be quite cool might be a import hook which
fetches things from the repository when needed, with a simple
bootstrap script for every project to be able to use this feature, but
it only makes sense if I need this kind of feature in many projects.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Sharing code between different projects?

[andrea crotti]

2012/8/14 Jean-Michel Pichavant :
>
> I can think of logilab-common (http://www.logilab.org/848/)
>
> Having a company-wide python module properly distributed is one to achieve
> your goal. Without distributing your module to the public, there's a way to
> have a pypi-like server runnning on your private network :
>
> http://pypi.python.org/pypi/pypiserver/
>
> JM
>
> Note : looks like pypi.python.org is having some trouble, the above link is
> broken. Search for recent announcement about pypiserver.
>


Thanks, yes we need something like this..
I'll copy the name probably, I prefer "common" to "utils/utilities"..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Sharing code between different projects?

[andrea crotti]

2012/8/14 Cameron Simpson :
>
> Having just skimmed this thread, one thing I haven't quite seen suggested is
> this:
>
> Really do make a third "utilities" project, and treat "the project" and
> "deploy" as separate notions. So to actually run/deploy project A's code
> you'd have a short script that copied project A and the utilities project
> code into a tree and ran off that. Or even a simple process/script to
> update the copy of "utilities" in "project A"'s area.
>
> So you don't "share" code on an even handed basis but import the
> "utilities" library into each project as needed.
>
> I do this (one my own very small scale) in one of two ways:
>
>   - as needed, copy the desired revision of utilities into the project's
> library space and do perforce's equivalent of Mercurial's addremove
> on that library tree (comment "update utilities to revision X").
>
>   - keep a perforce work area for the utilities in your project A area,
> where your working project A can hook into it with a symlink or some
> deploy/copy procedure as suggested above.
> With this latter one you can push back into the utilities library
> from your "live" project, because you have a real checkout. So:
>
>   projectAdir
> projectA-perforce-checkout
> utilities-perforce-checkout
>   projectBdir
> projectB-perforce-checkout
> utilities-perforce-checkout
>

Thanks, is more or less what I was going to do..  But I would not use
symlinks and similar things, because then every user should set it up
accordingly.

Potentially we could instead use the perforce API to change the
workspace mappings at run-time, and thus "force" perforce to checkout
the files in the right place..

There is still the problem that people should checkout things from two
places all the time instead of one..

> Personally I become more and more resistent to cut/paste even for small
> things as soon as multiple people use it; you will never get to backport
> updates to even trivial code to all the copies.
>
> Cheers,


Well sure, but on the other end as soon as multiple people use it you
can't change any of the public functions signatures without being
afraid that you'll break something..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Sharing code between different projects?

[andrea crotti]

Also looking at logilab-common I thought that it would be great if we
could actually make this "common" library even open source, and use it
as one of the other many external libraries.

Since Python code is definitively not the the core business of this
company I might even convince them, but the problem is that then all
the internal people working on it would not be able to use the
standard tools that they use with everything else..

Did anyone manage to convince his company to do something similar?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Sharing code between different projects?

[andrea crotti]

2012/8/16 Jean-Michel Pichavant :
>
> SVN allows to define external dependencies, where one repository will
> actually checkout another one at a specific version. If SVN does it, I guess
> any decent SCM also provide such feature.
>
> Assuming our project is named 'common', and you have 2 projects A and B :
>
> A
>- common@rev1
>
> B
>- common@rev2
>
> Project A references the lib as "A.common", B as "B.common". You need to be
> extra carefull to never reference common as 'common' in any place.
>
> JM
>


Unfortunately I think you guess wrong
http://forums.perforce.com/index.php?/topic/553-perforce-svnexternals-equivalent/
Anyway with views and similar things is not that hard to implement the
same thing..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how to call perl script from html using python

[andrea crotti]

2012/8/16 Pervez Mulla :
>
> Hey Steven ,
>
> Thank you for your response,
>
> I will in detail now about my project,
>
> Actually the project entire backend in PERL language , Am using Django 
> framework for my front end .

> I have written code for signup page in python , which is working perfectly .
>
> In HTml when user submit POST method, it calling Python code Instead of 
> this I wanna call perl script for sign up ..
>
> below in form for sign up page in python 

Good that's finally an explanation, so the question you can ask google
was "how do I call an external process from Python",
which has absolutely nothing to with HTML, and is very easy to find
out (hint: subprocess).
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Sharing code between different projects?

[andrea crotti]

2012/8/16 andrea crotti :
>
>
> Unfortunately I think you guess wrong
> http://forums.perforce.com/index.php?/topic/553-perforce-svnexternals-equivalent/
> Anyway with views and similar things is not that hard to implement the
> same thing..


I'm very happy to say that I finally made it!

It took 3 hours to move / merge a few thousand lines around but
everything seems to work perfectly now..

At the moment I'm just using symlinks, I'll see later if something
smarter is necessary, thanks to everyone for the ideas.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Why doesn't Python remember the initial directory?

[andrea crotti]

2012/8/20 kj :
> In  Roy Smith  
> writes> This means that no library code can ever count on, for example,
> being able to reliably find the path to the file that contains the
> definition of __main__.  That's a weakness, IMO.  One manifestation
> of this weakness is that os.chdir breaks inspect.getmodule, at
> least on Unix.  If you have some Unix system handy, you can try
> the following.  First change the argument to os.chdir below to some
> valid directory other than your working directory.  Then, run the
> script, making sure that you refer to it using a relative path.
> When I do this on my system (OS X + Python 2.7.3), the script bombs
> at the last print statement, because the second call to inspect.getmodule
> (though not the first one) returns None.
>
> import inspect
> import os
>
> frame = inspect.currentframe()
>
> print inspect.getmodule(frame).__name__
>
> os.chdir('/some/other/directory') # where '/some/other/directory' is
>   # different from the initial directory
>
> print inspect.getmodule(frame).__name__
>
> ...
>
> % python demo.py
> python demo.py
> __main__
> Traceback (most recent call last):
>   File "demo.py", line 11, in 
> print inspect.getmodule(frame).__name__
> AttributeError: 'NoneType' object has no attribute '__name__'
>
>..

As in many other cases the programming language can't possibly act
safely on all the possible stupid things that the programmer wants to
do, and not understanding how an operating system works doesn't help
either..

In the specific case there is absolutely no use of os.chdir, since you
can:
- use absolute paths
- things like subprocess.Popen accept a cwd argument
- at worst you can chdir back to the previous position right after the
broken thing that require a certain path that you are calling is run
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Why doesn't Python remember the initial directory?

[andrea crotti]

2012/8/20 Roy Smith :
> In article ,
>  Walter Hurry  wrote:
>
>> It is difficult to think of a sensible use for os.chdir, IMHO.
>
> It is true that you can mostly avoid chdir() by building absolute
> pathnames, but it's often more convenient to just cd somewhere and use
> names relative to that.  Fabric (a very cool tool for writing remote
> sysadmin scripts), gives you a cd() command which is a context manager,
> making it extra convenient.
>
> Also, core files get created in the current directory.  Sometimes
> daemons will cd to some fixed location to make sure that if they dump
> core, it goes in the right place.
>
> On occasion, you run into (poorly designed, IMHO) utilities which insist
> of reading or writing a file in the current directory.  If you're
> invoking one of those, you may have no choice but to chdir() to the
> right place before running them.
> --
> http://mail.python.org/mailman/listinfo/python-list


I've done quite a lot of system programming as well, and changing
directory is only a source of possible troubles in general.

If I really have to for some reasons I do this


class TempCd:
"""Change temporarily the current directory
"""
def __init__(self, newcwd):
self.newcwd = newcwd
self.oldcwd = getcwd()

def __enter__(self):
chdir(self.newcwd)
return self

def __exit__(self, type, value, traceback):
chdir(self.oldcwd)


with TempCd('/tmp'):
# now working in /tmp

# now in the original

So it's not that hard to avoid problems..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: pyQT performance?

[Andrea Crotti]

On 09/10/2012 07:29 PM, jayden.s...@gmail.com wrote

Have you ever used py2exe? After converting the python codes to executable, 
does it save the time of interpreting the script language? Thank a lot!


Py2exe normally never speeds up anything, simply because it doesn't 
convert to executable, but simply
package everything together, so I haven't tried in this particular case 
but it shouldn't make a difference..

--
http://mail.python.org/mailman/listinfo/python-list

main and dependent objects

[andrea crotti]

I am in a situation where I have a class Obj which contains many
attributes, and also contains logically another object of class
Dependent.

This dependent_object, however, also needs to access many fields of the
original class, so at the moment we did something like this:


class Dependent:
def __init__(self, orig):
self.orig = orig

def using_other_attributes(self):
print("Using attr1", self.orig.attr1)


class Obj:
def __init__(self):
self.attr1 = "attr1"
self.attr2 = "attr2"
self.attr3 = "attr3"

self.dependent_object = Dependent(self)


But I'm not so sure it's a good idea, it's a bit smelly..
Any other suggestion about how to get a similar result?

I could of course passing all the arguments needed to the constructor of
Dependent, but it's a bit tedious..


Thanks,
Andrea
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: main and dependent objects

[andrea crotti]

2012/9/13 Jean-Michel Pichavant :
>
> Nothing shocking right here imo. It looks like a classic parent-child 
> implementation.
> However it seems the relation between Obj and Dependent are 1-to-1. Since 
> Dependent need to access all Obj attributes, are you sure that Dependent and 
> Obj are not actually the same class ?
>
>
> JM

Yes well the main class is already big enough, and the relation is 1-1
but the dependent class can be also considered separate to split
things more nicely..

So I think it will stay like this for now and see how it goes.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python presentations

[andrea crotti]

2012/9/13 William R. Wing (Bill Wing) :
>
> [byte]
>
> Speaking from experience as both a presenter and an audience member, please 
> be sure that anything you demo interactively you include in your slide deck 
> (even if only as an addendum).  I assume your audience will have access to 
> the deck after your talk (on-line or via hand-outs), and you want them to be 
> able to go home and try it out for themselves.
>
> Nothing is more frustrating than trying to duplicate something you saw a 
> speaker do, and fail because of some detail you didn't notice at the time of 
> the talk.  A good example is one that was discussed on the matplotlib-users 
> list several weeks ago:
>
> http://www.loria.fr/~rougier/teaching/matplotlib/
>
> -Bill


Yes that's a good point thanks, in general everything is already in a
git repository, now only in my dropbox but later I will make it
public.

Even the code that I should write there should already written anyway,
and to make sure everything is available I could use the save function
of IPython and add it to the repository...

In general I think that explaining code on a slide (if it involves
some new concepts in particular) it's better, but then showing what it
does it's always a plus.

It's not the same if you say this will go 10x faster than the previous
one, and showing that it actually does on your machine..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python presentations

[Andrea Crotti]

On 09/13/2012 11:58 PM, Miki Tebeka wrote:

What do you think work best in general?

I find typing during class (other than small REPL examples) time consuming and 
error prone.

What works well for me is to create a slidy HTML presentation with asciidoc, 
then I can include code snippets that can be also run from the command line.
(Something like:

 [source,python,numbered]
 ---
 include::src/sin.py[]
 ---

Output example: http://i.imgur.com/Aw9oQ.png
)

Let me know if you're interested and I'll send you a example project.

HTH,
--
Miki


Yes please send me something and I'll have a look.
For my slides I'm using hieroglyph:
http://heiroglyph.readthedocs.org/en/latest/index.html

which works with sphinx, so in theory I might be able to run the code as 
well..


But in general probably the best way is to copy and paste in a ipython 
session, to show

that what I just explained actually works as expected..
--
http://mail.python.org/mailman/listinfo/python-list

Re: Decorators not worth the effort

[andrea crotti]

I think one very nice and simple example of how decorators can be used is this:

def memoize(f, cache={}, *args, **kwargs):
def _memoize(*args, **kwargs):
key = (args, str(kwargs))
if not key in cache:
cache[key] = f(*args, **kwargs)

return cache[key]

return _memoize

def fib(n):
if n <= 1:
return 1
return fib(n-1) + fib(n-2)

@memoize
def fib_memoized(n):
if n <= 1:
return 1
return fib_memoized(n-1) + fib_memoized(n-2)


The second fibonacci looks exactly the same but while the first is
very slow and would generate a stack overflow the second doesn't..

I might use this example for the presentation, before explaining what it is..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Decorators not worth the effort

[andrea crotti]

2012/9/14 Chris Angelico :
>
> Trouble is, you're starting with a pretty poor algorithm. It's easy to
> improve on what's poor. Memoization can still help, but I would start
> with a better algorithm, such as:
>
> def fib(n):
> if n<=1: return 1
> a,b=1,1
> for i in range(1,n,2):
> a+=b
> b+=a
> return b if n%2 else a
>
> def fib(n,cache=[1,1]):
> if n<=1: return 1
> while len(cache)<=n:
> cache.append(cache[-1] + cache[-2])
> return cache[n]
>
> Personally, I don't mind (ab)using default arguments for caching, but
> you could do the same sort of thing with a decorator if you prefer. I
> think the non-decorated non-recursive version is clear and efficient
> though.
>
> ChrisA
> --
> http://mail.python.org/mailman/listinfo/python-list


The poor algorithm is much more close to the mathematical definition
than the smarter iterative one..  And in your second version you
include some ugly caching logic inside it, so why not using a
decorator then?

I'm not saying that with the memoization is the "good" solution, just
that I think it's a very nice example of how to use a decorator, and
maybe a good example to start with a talk on decorators..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: subprocess call is not waiting.

[andrea crotti]

I have a similar problem, something which I've never quite understood
about subprocess...
Suppose I do this:

proc = subprocess.Popen(['ls', '-lR'], stdout=subprocess.PIPE,
stderr=subprocess.PIPE)

now I created a process, which has a PID, but it's not running apparently...
It only seems to run when I actually do the wait.

I don't want to make it waiting, so an easy solution is just to use a
thread, but is there a way with subprocess?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: subprocess call is not waiting.

[andrea crotti]

2012/9/18 Dennis Lee Bieber :
>
> Unless you have a really massive result set from that "ls", that
> command probably ran so fast that it is blocked waiting for someone to
> read the PIPE.

I tried also with "ls -lR /" and that definitively takes a while to run,
when I do this:

proc = subprocess.Popen(['ls', '-lR', '/'], stdout=subprocess.PIPE,
stderr=subprocess.PIPE)

nothing is running, only when I actually do
proc.communicate()

I see the process running in top..
Is it still an observation problem?

Anyway I also need to know when the process is over while waiting, so
probably a thread is the only way..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: subprocess call is not waiting.

[andrea crotti]

2012/9/19 Hans Mulder :
> Yes: using "top" is an observation problem.
>
> "Top", as the name suggests, shows only the most active processes.

Sure but "ls -lR /" is a very active process if you try to run it..
Anyway as written below I don't need this anymore.

>
> It's quite possible that your 'ls' process is not active, because
> it's waiting for your Python process to read some data from the pipe.
>
> Try using "ps" instead.  Look in thte man page for the correct
> options (they differ between platforms).  The default options do
> not show all processes, so they may not show the process you're
> looking for.
>
>> Anyway I also need to know when the process is over while waiting, so
>> probably a thread is the only way..
>
> This sounds confused.
>
> You don't need threads.  When 'ls' finishes, you'll read end-of-file
> on the proc.stdout pipe.  You should then call proc.wait() to reap
> its exit status (if you don't, you'll leave a zombie process).
> Since the process has already finished, the proc.wait() call will
> not actually do any waiting.
>
>
> Hope this helps,
>


Well there is a process which has to do two things, monitor
periodically some external conditions (filesystem / db), and launch a
process that can take very long time.

So I can't put a wait anywhere, or I'll stop everything else.  But at
the same time I need to know when the process is finished, which I
could do but without a wait might get hacky.

So I'm quite sure I just need to run the subprocess in a subthread
unless I'm missing something obvious..
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python presentations

[andrea crotti]

2012/9/19 Trent Nelson :
>
> FWIW, I gave a presentation on decorators to the New York Python
> User Group back in 2008.  Relevant blog post:
>
> http://blogs.onresolve.com/?p=48
>
> There's a link to the PowerPoint presentation I used in the first
> paragraph.  It's in .pptx format; let me know if you'd like it in
> some other form.
>
> Regards,
>
> Trent.


Ok thanks a lot, how long did it take for you to present that material?

Interesting the part about the learning process, I had a similar
experience, but probably skip this since I only have 30 minutes.

Another thing which I would skip or only explain how it works are
parametrized decorators, in the triple-def form they just look to ugly
to be worth the effort (but at least should be understood).
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: 'str' object does not support item assignment

[Andrea Crotti]

On 09/23/2012 07:31 PM, jimbo1qaz wrote:

spots[y][x]=mark fails with a "'str' object does not support item assignment" 
error,even though:

a=[["a"]]
a[0][0]="b"

and:

a=[["a"]]
a[0][0]=100

both work.
Spots is a nested list created as a copy of another list.


But
a = "a"
a[0] = 'c'
fails for the same reason, which is that strings in Python are immutable..
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python presentations

[andrea crotti]

For anyone interested, I already moved the slides on github
(https://github.com/AndreaCrotti/pyconuk2012_slides)
and for example the decorator slides will be generated from this:

https://raw.github.com/AndreaCrotti/pyconuk2012_slides/master/deco_context/deco.rst

Notice the literalinclude with :pyobject: which allows to include any
function or class automatically very nicely from external files ;)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PHP vs. Python

[andrea crotti]

2012/9/25  :
> On Thursday, 23 December 2004 03:33:36 UTC+5:30, (unknown)  wrote:
>> Anyone know which is faster?  I'm a PHP programmer but considering
>> getting into Python ... did searches on Google but didn't turn much up
>> on this.
>>
>> Thanks!
>> Stephen
>
>
> Here some helpful gudance.
>
> http://hentenaar.com/serendipity/index.php?/archives/27-Benchmark-PHP-vs.-Python-vs.-Perl-vs.-Ruby.html
> --
> http://mail.python.org/mailman/listinfo/python-list


Quite ancient versions of everything, would be interesting to see if
things are different now..

Anyway you can switch to Python happily, it might not be faster but
99% of the times that's not an issue..
-- 
http://mail.python.org/mailman/listinfo/python-list