Reddit broke - should have remained on Lisp?
While posting a comment on http://www.reddit.com I got an error page with the following curious statement on it. "reddit broke (sorry)" "looks like we shouldn't have stopped using lisp..." See screenshot at http://photos1.blogger.com/blogger/1773/1980/1600/reddit-broke.jpg Whether they truly repent not using lisp or otherwise, their site appears to be 3 times slower ... Alok -- http://mail.python.org/mailman/listinfo/python-list
Re: Reddit broke - should have remained on Lisp?
Luis M. González wrote: > Alok wrote: > > While posting a comment on http://www.reddit.com I got an error page > > with the following curious statement on it. > > > > "reddit broke (sorry)" > > "looks like we shouldn't have stopped using lisp..." > > > > See screenshot at > > http://photos1.blogger.com/blogger/1773/1980/1600/reddit-broke.jpg > > > > Whether they truly repent not using lisp or otherwise, their site > > appears to be 3 times slower ... > > > > Alok > > > I don't know if this is true or not, but blaming a language for a poor > development is a little bit ridiculous... I was merely describing my experience and inviting others' response about theirs. Please don't misconstrue that as a blame on any language. And of course, I disagree with your comments about ridicule etc. -- http://mail.python.org/mailman/listinfo/python-list
Re: Reddit broke - should have remained on Lisp?
Luis M. González wrote: > Alok wrote: > > I was merely describing my experience and inviting others' response > > about theirs. > > That's exactly what I'm doing. You misinterpret, I was talking about my experience with the site and inviting response from other people about their experience with the site. Now, I don't know much about python or lisp, but I have read about this web-site's history with the two programming languages. And hence I chose to post in comp.lang.lisp and comp.lang.python. Now if there were a reddit.technology.lisp or reddit.technology.python, and if they had the same traction as these usenet groups, then I could have posted there. But that was not an option was it. > > > Please don't misconstrue that as a blame on any language. > > I think it can be interpreted in many ways. Can you please explain what your interpretation is? > Now if you're not ready to read other people's oppinions, don't ask. > Now, I think you are unfairly prejudiced about my not wanting to read other people's opinion. I would rather have a 1000 acrid responses to something I write and learn from it, than post into a responseless vacuum. > > And of course, I disagree with your comments about ridicule etc. > > Ok. -- http://mail.python.org/mailman/listinfo/python-list
Re: Reddit broke - should have remained on Lisp?
Luis M. González wrote: Luis, Thank you very much for you detailed and careful response. I very much appreciate your comments. > Alok wrote: > > Luis M. González wrote: > > > Alok wrote: > > > > I was merely describing my experience and inviting others' response > > > > about theirs. > > > > > > That's exactly what I'm doing. > > > > You misinterpret, I was talking about my experience with the site and > > inviting response from other people about their experience with the > > site. Now, I don't know much about python or lisp, but I have read > > about this web-site's history with the two programming languages. And > > hence I chose to post in comp.lang.lisp and comp.lang.python. > > > > Now if there were a reddit.technology.lisp or > > reddit.technology.python, and if they had the same traction as these > > usenet groups, then I could have posted there. But that was not an > > option was it. > > > > > > > > > Please don't misconstrue that as a blame on any language. > > > > > > I think it can be interpreted in many ways. > > > > Can you please explain what your interpretation is? > > > > > Now if you're not ready to read other people's oppinions, don't ask. > > > > > > > Now, I think you are unfairly prejudiced about my not wanting to read > > other people's opinion. I would rather have a 1000 acrid responses to > > something I write and learn from it, than post into a responseless > > vacuum. > > > > > > And of course, I disagree with your comments about ridicule etc. > > > > > > Ok. > > Dear Alok, > > I think is you who are misinterpreting me (or I didn't make myself > clear). > You posted a link to a joke from reddit, which is perfectly fine, is > very funny. > But below, you added a comment wondering whether the change from Lisp > to Python may be a reason for a huge performance loss. I did not intend to comment on Python or Lisp but yes, I have aired my subjective observations about the site and in the context of the technologies used. And I hoped that other users may respond with comments if they have experienced any similar issues using the site. If I have implied an indication that the change from Lisp to Python was the reason that the site became 3 times slower, then I have slipped in unintended innuendos. And I must accept that I have to be more careful about it. > > I'm not a python bigot, but I just said that blamming a language choice > for such a deficieny was ridiculous. There are hundreds of very complex > web sites developed in python that doesn't suffer these problems, and, > even if they are developed in python, you must take into account which > other technologies they are using for deployment (web server, > framework, mod_python, cgi, fastcgi, hardware, etc, etc...). > Also, the fact that someone could be more versed or knowledgeable of > one language compared to another one, doesn't make it better or worse. > This is s subjective fact, and has nothing to do with the language. It > has to do with who's using the language. > > Also, as someone pointed out above, don't you think that the huge > incresement of traffic since they switched to python may have had > something to do with this problem? > Perhaps, their lack of experise in a new language didn't prepare them > for facing such a challenge. Who knows? Yeah, perhaps if they sticked > to Lisp, they could have handled the problem better. They are lispers, > aren't they? > > Anyway, you posted a comment, you asked for opinons, you got mine. > Sorry if I ofended you. It wasn't my intention. There is no offense taken or hurt feelings. But yes, detailed responses (and smileys) do make conversations more rewarding. > I'll make sure I place more smileys in my posts to avoid hurting > feelings from now on :-) > > Luis -- http://mail.python.org/mailman/listinfo/python-list
Re: Reddit broke - should have remained on Lisp?
Fredrik Lundh wrote: > "Alok" wrote: > > > While posting a comment on http://www.reddit.com I got an error page > > with the following curious statement on it. > > > > "reddit broke (sorry)" > > "looks like we shouldn't have stopped using lisp..." > > > > See screenshot at > > http://photos1.blogger.com/blogger/1773/1980/1600/reddit-broke.jpg > > > > Whether they truly repent not using lisp or otherwise, their site > > appears to be 3 times slower ... > > since when? Their site was more responsive last year. Especially the form submission page to create new posts. And also being able to refresh the page with your recent comments in them. > don't you think that the massive growth in usage that they've been > seeing lately might have something to do with that ? > You are right it definitely has a major impact, I wish they scale up their hardware in line with their possible growth. -A -- http://mail.python.org/mailman/listinfo/python-list
Re: Reddit broke - should have remained on Lisp?
Fredrik Lundh wrote: > Alok wrote: > > > Their site was more responsive last year. > > http://tinyurl.com/zhary > > Thats an interesting URL my friend. For those of you wondering what the spike in April was due to, check out this one http://www.joelonsoftware.com/items/2006/03/20.html -- http://mail.python.org/mailman/listinfo/python-list
weave and 64 bit issues
nverters=weave.c onve rters.blitz,headers=[""],compiler="gcc",verbose=2) File "c:\python27\lib\site-packages\scipy\weave\inline_tools.py", line 355, in inline **kw) File "c:\python27\lib\site-packages\scipy\weave\inline_tools.py", line 488, in compile_function exec 'import ' + module_name File "", line 1, in ImportError: DLL load failed: Invalid access to memory location. Y:\STMM\alpha\klse\PROD> For everytime there is an error I have one observation, the message has notification "repairing catalog by removing key" which leads to DLL load failed error. Any ideas? - Alok === Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html === -- http://mail.python.org/mailman/listinfo/python-list
weave in 64 bit strange behavior
Hi everyone,
I realize my previous post was quite unreadable, thanks to my email
client. I am going to report my question here, with slight enhancements.
Apologies for inconvenience caused and spamming your mailboxes.
I am facing a strange problem using weave on 64 bit machine.
Specifically with weave's inline function. It has something to do with
weave's catalog.
Similar issues I found in the past (very old)
http://mail.scipy.org/pipermail/scipy-dev/2006-June/005908.html
http://mail.scipy.org/pipermail/scipy-dev/2005-June/003042.html
Common things I have in my observation are:
- Already working setup in 32 bit doesn't work in same manner
in 64 bit env
- Weave recompiles inline code which does not require any
recompilation. This is random behavior. Whenever weave recompiles I see
a notification "repairing catalog by removing key" in the output which
ends up in the error message "ImportError: DLL load failed: Invalid
access to memory location"
- Sometimes gcc gets into an infinite loop printing error
message "Looking for python27.dll". Even though the dll is on the path.
This process doesn't end. Had to kill it forcefully. G++ process became
ghost even after killing python process.
Could someone advise what am I missing here. Is there any specific setup
that I need to do? Is there an issue with python 27 64 bit weave
implementation?
Regards,
Alok
I have a simple script to calculate moving average using weave's inline
function.
source mvg.py
import numpy as np
import scipy.weave as weave
import distutils.sysconfig
import distutils.dir_util
import os
distutils.sysconfig._config_vars["LDSHARED"]="-LC:\strawberry\c\x86_64-w
64-mingw32\lib"
def ExpMovAvg(data,time,lag):
if (data.size!=time.size):
print "error in EMA, data and time have different size"
return None
result=np.repeat(0.0,data.size)
code="""
#line 66 "basics.py"
result(0)=data(0);
for (int i=0;i1)
{
alpha=10;
}
result(i+1)=(1-alpha)*data(i)+alpha*result(i);
}
"""
weave.inline(code,["data","time","lag","result"],type_converters=weave.c
onverters.blitz,headers=[""],compiler="gcc",verbose=2)
return result
source test.py
import string
import numpy as np
import mvg
print(mvg.ExpMovAvg(np.array(range(10)),np.array(range(10)),2))
Output:
Working output:
Y:\STMM\alpha\klse\PROD>c:\python27\python.exe
s:\common\tools\python\python-2.7-64bit\test.py
[ 0. 0. 0.63212774 1.49679774 2.44701359 3.42869938
4.42196209 5.41948363 6.41857187 7.41823646]
Now if I keep running the script multiple times, sometimes I see correct
output... but suddenly sometimes I get below error.
Y:\STMM\alpha\klse\PROD>c:\python27\python.exe
s:\common\tools\python\python-2.7-64bit\test.py
repairing catalog by removing key
Looking for python27.dll
running build_ext
running build_src
build_src
building extension "sc_44f3fe3c65d5c3feecb45d9269ac207f5" sources
build_src: building npy-pkg config files
Looking for python27.dll
customize Mingw32CCompiler
customize Mingw32CCompiler using build_ext
Looking for python27.dll
customize Mingw32CCompiler
customize Mingw32CCompiler using build_ext
building 'sc_44f3fe3c65d5c3feecb45d9269ac207f5' extension
compiling C++ sources
C compiler: g++ -g -DDEBUG -DMS_WIN64 -O0 -Wall
compile options: '-Ic:\python27\lib\site-packages\scipy\weave
-Ic:\python27\lib\site-packages\scipy\weave\scxx
-Ic:\python27\lib\site-packages\scipy\weave\blitz
-Ic:\python27\lib\site-packages\numpy\core\include -Ic:\python27\include
-Ic:\python27\PC -c'
g++ -g -DDEBUG -DMS_WIN64 -O0 -Wall
-Ic:\python27\lib\site-packages\scipy\weave
-Ic:\python27\lib\site-packages\scipy\weave\scxx
-Ic:\python27\lib\site-packages\scipy\weave\blitz
-Ic:\python27\lib\site-packages\numpy\core\include -Ic:\python27\include
-Ic:\python27\PC -c
c:\users\ajadhav2\appdata\local\temp\ajadhav2\python27_compiled\sc_44f3f
e3c65d5c3feecb45d9269ac207f5.cpp -o
c:\users\ajadhav2\appdata\local\temp\ajadhav2\python27_intermediate\comp
iler_2d3e1e2e4de6a91419d2376b162e5342\Release\users\ajadhav2\appdata\loc
al\temp\ajadhav2\python27_compiled\sc_44f3fe3c65d5c3feecb45d9269ac207f5.
o
Found executable C:\strawberry\c\bin\g++.exe
g++ -g -DDEBUG -DMS_WIN64 -O0
-Wall-Ic:\python27\lib\site-packages\scipy\weave-Ic:\python27\lib\site-p
ackages\scipy\weave\scxx
-Ic:\python27\lib\site-packages\scipy\weave\blitz
-Ic:\python27\lib\site-packages\numpy\core\include-Ic:\python27\include
-Ic:\python27\PC -c
c:\python27\lib\site-packages\scipy\weave\scxx\weave_imp.cpp -o
c:\users\ajadhav2\appdata\local\temp\ajadha
Python garbage collector/memory manager behaving strangely
Hi Everyone,
I have a simple program which reads a large file containing few million
rows, parses each row (`numpy array`) and converts into an array of
doubles (`python array`) and later writes into an `hdf5 file`. I repeat
this loop for multiple days. After reading each file, i delete all the
objects and call garbage collector. When I run the program, First day
is parsed without any error but on the second day i get `MemoryError`. I
monitored the memory usage of my program, during first day of parsing,
memory usage is around **1.5 GB**. When the first day parsing is
finished, memory usage goes down to **50 MB**. Now when 2nd day starts
and i try to read the lines from the file I get `MemoryError`. Following
is the output of the program.
source file extracted at C:\rfadump\au\2012.08.07.txt
parsing started
current time: 2012-09-16 22:40:16.829000
50 lines parsed
100 lines parsed
150 lines parsed
200 lines parsed
250 lines parsed
300 lines parsed
350 lines parsed
400 lines parsed
450 lines parsed
500 lines parsed
parsing done.
end time is 2012-09-16 23:34:19.931000
total time elapsed 0:54:03.102000
repacking file
done
> s:\users\aaj\projects\pythonhf\rfadumptohdf.py(132)generateFiles()
-> while single_date <= self.end_date:
(Pdb) c
*** 2012-08-08 ***
source file extracted at C:\rfadump\au\2012.08.08.txt
cought an exception while generating file for day 2012-08-08.
Traceback (most recent call last):
File "rfaDumpToHDF.py", line 175, in generateFile
lines = self.rawfile.read().split('|\n')
MemoryError
I am very sure that windows system task manager shows the memory usage
as **50 MB** for this process. It looks like the garbage collector or
memory manager for Python is not calculating the free memory correctly.
There should be lot of free memory but it thinks there is not enough.
Any idea?
Thanks.
Alok Jadhav
CREDIT SUISSE AG
GAT IT Hong Kong, KVAG 67
International Commerce Centre | Hong Kong | Hong Kong
Phone +852 2101 6274 | Mobile +852 9169 7172
alok.jad...@credit-suisse.com | www.credit-suisse.com
<http://www.credit-suisse.com/>
===
Please access the attached hyperlink for an important electronic communications
disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===
--
http://mail.python.org/mailman/listinfo/python-list
RE: Python garbage collector/memory manager behaving strangely
Thanks Dave for clean explanation. I clearly understand what is going on
now. I still need some suggestions from you on this.
There are 2 reasons why I was using self.rawfile.read().split('|\n')
instead of self.rawfile.readlines()
- As you have seen, the line separator is not '\n' but its '|\n'.
Sometimes the data itself has '\n' characters in the middle of the line
and only way to find true end of the line is that previous character
should be a bar '|'. I was not able specify end of line using
readlines() function, but I could do it using split() function.
(One hack would be to readlines and combine them until I find '|\n'. is
there a cleaner way to do this?)
- Reading whole file at once and processing line by line was must
faster. Though speed is not of very important issue here but I think the
tie it took to parse complete file was reduced to one third of original
time.
Regards,
Alok
-Original Message-
From: Dave Angel [mailto:d...@davea.name]
Sent: Monday, September 17, 2012 10:13 AM
To: Jadhav, Alok
Cc: python-list@python.org
Subject: Re: Python garbage collector/memory manager behaving strangely
On 09/16/2012 09:07 PM, Jadhav, Alok wrote:
> Hi Everyone,
>
>
>
> I have a simple program which reads a large file containing few
million
> rows, parses each row (`numpy array`) and converts into an array of
> doubles (`python array`) and later writes into an `hdf5 file`. I
repeat
> this loop for multiple days. After reading each file, i delete all the
> objects and call garbage collector. When I run the program, First day
> is parsed without any error but on the second day i get `MemoryError`.
I
> monitored the memory usage of my program, during first day of parsing,
> memory usage is around **1.5 GB**. When the first day parsing is
> finished, memory usage goes down to **50 MB**. Now when 2nd day starts
> and i try to read the lines from the file I get `MemoryError`.
Following
> is the output of the program.
>
>
>
>
>
> source file extracted at C:\rfadump\au\2012.08.07.txt
>
> parsing started
>
> current time: 2012-09-16 22:40:16.829000
>
> 50 lines parsed
>
> 100 lines parsed
>
> 150 lines parsed
>
> 200 lines parsed
>
> 250 lines parsed
>
> 300 lines parsed
>
> 350 lines parsed
>
> 400 lines parsed
>
> 450 lines parsed
>
> 500 lines parsed
>
> parsing done.
>
> end time is 2012-09-16 23:34:19.931000
>
> total time elapsed 0:54:03.102000
>
> repacking file
>
> done
>
> >
s:\users\aaj\projects\pythonhf\rfadumptohdf.py(132)generateFiles()
>
> -> while single_date <= self.end_date:
>
> (Pdb) c
>
> *** 2012-08-08 ***
>
> source file extracted at C:\rfadump\au\2012.08.08.txt
>
> cought an exception while generating file for day 2012-08-08.
>
> Traceback (most recent call last):
>
> File "rfaDumpToHDF.py", line 175, in generateFile
>
> lines = self.rawfile.read().split('|\n')
>
> MemoryError
>
>
>
> I am very sure that windows system task manager shows the memory usage
> as **50 MB** for this process. It looks like the garbage collector or
> memory manager for Python is not calculating the free memory
correctly.
> There should be lot of free memory but it thinks there is not enough.
>
>
>
> Any idea?
>
>
>
> Thanks.
>
>
>
>
>
> Alok Jadhav
>
> CREDIT SUISSE AG
>
> GAT IT Hong Kong, KVAG 67
>
> International Commerce Centre | Hong Kong | Hong Kong
>
> Phone +852 2101 6274 | Mobile +852 9169 7172
>
> alok.jad...@credit-suisse.com | www.credit-suisse.com
> <http://www.credit-suisse.com/>
>
>
>
Don't blame CPython. You're trying to do a read() of a large file,
which will result in a single large string. Then you split it into
lines. Why not just read it in as lines, in which case the large string
isn't necessary. Take a look at the readlines() function. Chances are
that even that is unnecessary, but i can't tell without seeing more of
the code.
lines = self.rawfile.read().split('|\n')
lines = self.rawfile.readlines()
When a single large item is being allocated, it's not enough to have
sufficient free space, the space also has to be contiguous. After a
program runs for a while, its space naturally gets fragmented more and
more. it's the nature of the C runtime, and CPython is stuck with it.
--
DaveA
===
Please access the attached hyperlink for an important electronic communications
disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===
--
http://mail.python.org/mailman/listinfo/python-list
RE: Python garbage collector/memory manager behaving strangely
I am thinking of calling a new subprocess which will do the memory
hungry job and then release the memory as specified in the link below
http://stackoverflow.com/questions/1316767/how-can-i-explicitly-free-mem
ory-in-python/1316799#1316799
Regards,
Alok
-Original Message-
From: Dave Angel [mailto:d...@davea.name]
Sent: Monday, September 17, 2012 10:13 AM
To: Jadhav, Alok
Cc: python-list@python.org
Subject: Re: Python garbage collector/memory manager behaving strangely
On 09/16/2012 09:07 PM, Jadhav, Alok wrote:
> Hi Everyone,
>
>
>
> I have a simple program which reads a large file containing few
million
> rows, parses each row (`numpy array`) and converts into an array of
> doubles (`python array`) and later writes into an `hdf5 file`. I
repeat
> this loop for multiple days. After reading each file, i delete all the
> objects and call garbage collector. When I run the program, First day
> is parsed without any error but on the second day i get `MemoryError`.
I
> monitored the memory usage of my program, during first day of parsing,
> memory usage is around **1.5 GB**. When the first day parsing is
> finished, memory usage goes down to **50 MB**. Now when 2nd day starts
> and i try to read the lines from the file I get `MemoryError`.
Following
> is the output of the program.
>
>
>
>
>
> source file extracted at C:\rfadump\au\2012.08.07.txt
>
> parsing started
>
> current time: 2012-09-16 22:40:16.829000
>
> 50 lines parsed
>
> 100 lines parsed
>
> 150 lines parsed
>
> 200 lines parsed
>
> 250 lines parsed
>
> 300 lines parsed
>
> 350 lines parsed
>
> 400 lines parsed
>
> 450 lines parsed
>
> 500 lines parsed
>
> parsing done.
>
> end time is 2012-09-16 23:34:19.931000
>
> total time elapsed 0:54:03.102000
>
> repacking file
>
> done
>
> >
s:\users\aaj\projects\pythonhf\rfadumptohdf.py(132)generateFiles()
>
> -> while single_date <= self.end_date:
>
> (Pdb) c
>
> *** 2012-08-08 ***
>
> source file extracted at C:\rfadump\au\2012.08.08.txt
>
> cought an exception while generating file for day 2012-08-08.
>
> Traceback (most recent call last):
>
> File "rfaDumpToHDF.py", line 175, in generateFile
>
> lines = self.rawfile.read().split('|\n')
>
> MemoryError
>
>
>
> I am very sure that windows system task manager shows the memory usage
> as **50 MB** for this process. It looks like the garbage collector or
> memory manager for Python is not calculating the free memory
correctly.
> There should be lot of free memory but it thinks there is not enough.
>
>
>
> Any idea?
>
>
>
> Thanks.
>
>
>
>
>
> Alok Jadhav
>
> CREDIT SUISSE AG
>
> GAT IT Hong Kong, KVAG 67
>
> International Commerce Centre | Hong Kong | Hong Kong
>
> Phone +852 2101 6274 | Mobile +852 9169 7172
>
> alok.jad...@credit-suisse.com | www.credit-suisse.com
> <http://www.credit-suisse.com/>
>
>
>
Don't blame CPython. You're trying to do a read() of a large file,
which will result in a single large string. Then you split it into
lines. Why not just read it in as lines, in which case the large string
isn't necessary. Take a look at the readlines() function. Chances are
that even that is unnecessary, but i can't tell without seeing more of
the code.
lines = self.rawfile.read().split('|\n')
lines = self.rawfile.readlines()
When a single large item is being allocated, it's not enough to have
sufficient free space, the space also has to be contiguous. After a
program runs for a while, its space naturally gets fragmented more and
more. it's the nature of the C runtime, and CPython is stuck with it.
--
DaveA
===
Please access the attached hyperlink for an important electronic communications
disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===
--
http://mail.python.org/mailman/listinfo/python-list
RE: Python garbage collector/memory manager behaving strangely
Thanks for your valuable inputs. This is very helpful.
-Original Message-
From: Python-list
[mailto:python-list-bounces+alok.jadhav=credit-suisse@python.org] On
Behalf Of Dave Angel
Sent: Monday, September 17, 2012 6:47 PM
To: alex23
Cc: python-list@python.org
Subject: Re: Python garbage collector/memory manager behaving strangely
On 09/16/2012 11:25 PM, alex23 wrote:
> On Sep 17, 12:32 pm, "Jadhav, Alok"
> wrote:
>> - As you have seen, the line separator is not '\n' but its '|\n'.
>> Sometimes the data itself has '\n' characters in the middle of the
line
>> and only way to find true end of the line is that previous character
>> should be a bar '|'. I was not able specify end of line using
>> readlines() function, but I could do it using split() function.
>> (One hack would be to readlines and combine them until I find '|\n'.
is
>> there a cleaner way to do this?)
> You can use a generator to take care of your readlines requirements:
>
> def readlines(f):
> lines = []
> while "f is not empty":
> line = f.readline()
> if not line: break
> if len(line) > 2 and line[-2:] == '|\n':
> lines.append(line)
> yield ''.join(lines)
> lines = []
> else:
> lines.append(line)
There's a few changes I'd make:
I'd change the name to something else, so as not to shadow the built-in,
and to make it clear in caller's code that it's not the built-in one.
I'd replace that compound if statement with
if line.endswith("|\n":
I'd add a comment saying that partial lines at the end of file are
ignored.
>> - Reading whole file at once and processing line by line was must
>> faster. Though speed is not of very important issue here but I think
the
>> tie it took to parse complete file was reduced to one third of
original
>> time.
You don't say what it was faster than. Chances are you went to the
other extreme, of doing a read() of 1 byte at a time. Using Alex's
approach of a generator which in turn uses the readline() generator.
> With the readlines generator above, it'll read lines from the file
> until it has a complete "line" by your requirement, at which point
> it'll yield it. If you don't need the entire file in memory for the
> end result, you'll be able to process each "line" one at a time and
> perform whatever you need against it before asking for the next.
>
> with open(u'infile.txt','r') as infile:
> for line in readlines(infile):
> ...
>
> Generators are a very efficient way of processing large amounts of
> data. You can chain them together very easily:
>
> real_lines = readlines(infile)
> marker_lines = (l for l in real_lines if l.startswith('#'))
> every_second_marker = (l for i,l in enumerate(marker_lines) if (i
> +1) % 2 == 0)
> map(some_function, every_second_marker)
>
> The real_lines generator returns your definition of a line. The
> marker_lines generator filters out everything that doesn't start with
> #, while every_second_marker returns only half of those. (Yes, these
> could all be written as a single generator, but this is very useful
> for more complex pipelines).
>
> The big advantage of this approach is that nothing is read from the
> file into memory until map is called, and given the way they're
> chained together, only one of your lines should be in memory at any
> given time.
--
DaveA
--
http://mail.python.org/mailman/listinfo/python-list
===
Please access the attached hyperlink for an important electronic communications
disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===
--
http://mail.python.org/mailman/listinfo/python-list
Re: numpy array operation
On Tue, 29 Jan 2013 00:41:54 -0800, C. Ng wrote: > Is there a numpy operation that does the following to the array? > > 1 2 ==> 4 3 > 3 4 2 1 > > Thanks in advance. How about: >>> import numpy as np >>> a = np.array([[1,2],[3,4]]) >>> a array([[1, 2], [3, 4]]) >>> a[::-1, ::-1] array([[4, 3], [2, 1]]) -- http://mail.python.org/mailman/listinfo/python-list
Regarding problem in python 3.8.0 installation
Sir, My system is windows 7 SP1 32-bit . after installing python in my system,when i try to launch it using command prompt then a message is shown. I am attaching a screenshot of the following. kindly look seriously into my problem and tell me the solution.. thank you -- https://mail.python.org/mailman/listinfo/python-list
Hiring Lead Python Programmer for Mozilla-Based Project
We're a US-based start-up building new software with the Mozilla Platform and are at present seeking a lead programmer/architect for our project. At present, we're planning to build a team in India, so we'd need the lead to be on-site for 6 to 12 months. We may, however, end up developing in the US or elsewhere. We're also building websites (ideally in Python) which will work in conjunction with the client application. Our projects are quite challenging and exciting. They're motivated by the general transition to a net-based lifestyle--we need better tools to help organize and improve the functionality of critical web services. They thus have the potential to reach a lot of people, and involve a breadth of fun but difficult technical challenges. We're building a small team of extremely capable developers. We're seeking independently motivated, dedicated and generally brilliant programmers--the sort of programmers who coded a C compiler for fun in college. We offer a competitive salary, intense, challenging and rewarding work, and the potential for spectacular bonuses. If you're seeking a new challenge and would like to build exciting new products, please send us your CV or resume and a letter of interest in technology/programming. Thanks, Alok, MetaStart metastart at gmail.com -- http://mail.python.org/mailman/listinfo/python-list
libxml2 Installation for Python 2.4
Hi, Can someone point me how to install libxml2 for python. It works fine on my Redhat 4.0 linux Box, but when I am trying to run my script on ARM target , it throws import error for libxml2. I am using xpath in my script and in header trying to import libxml2. Thanks in advance. Alok -- http://mail.python.org/mailman/listinfo/python-list
Struct usages in Python
Dear All, This might be very basic question for python, but I have been struggling with it. Your help will be highly appreciated. I have a class like below: class Event(): def __init__(self, cameraEventType="", zone=99, setDay="",setTime ="", clrTime=""): self.cameraEventType = cameraEventType self.zone = zone self.setDay = setDay self.setTime = setTime self.clrTime = clrTime I want to use array of Event class object in another class like below: class EventTimeFilter: def __init__(self): self.event[] = Event() *# Seems this is not allowed ?? * Can someone help me how can I use array of Event object in EventTimeFilter class. I planned to fill array of Event object in EventTimeFilter class method. Thanks in advance. Regards Alok -- http://mail.python.org/mailman/listinfo/python-list
multi dimensional dictionary
Dear All, I am using dictionary for filling my xpath parsed data. I wanted to use in the following manner. mydict[index] ["key1"] ["key2"]#Can someone help me with right declaration. So that I can fill my XML xpath parsed data mydict[0] ["person"] ["setTime"] = "12:09:30" mydict[0] ["person"] ["clrTime"] = "22:09:30" Can someone help me with right declaration usages. Your help will be highly appreciated. Regards Alok -- http://mail.python.org/mailman/listinfo/python-list
Re: Struct usages in Python
while traversing I get out of index error as mentioned below.
class EventTimeFilter:
def __init__(self):
* self.event = [Event()]*
def populateScheduleData(self):
self.doc = libxml2.parseFile(self.FILENAME)
for eachcamera in
self.doc.xpathEval('SetDeviceConfiguration/Camera/.'):
cameraIndex = eachcamera.get_properties()
#print cameraIndex
index = int(cameraIndex.content,10)
print index
xpathEventType = 'SetDeviceConfiguration/[EMAIL PROTECTED]' +
cameraIndex.content +']/Filter/Event'
for searchResults in self.doc.xpathEval(xpathEventType):
eventType = searchResults.get_properties()
* self.event[index-1].cameraEventType = eventType.content #
Error*
*#Error as below*
self.event[index-1].cameraEventType = eventType.content
IndexError: list index out of range
Any guidance why I am getting *list index out of range error*?
index value runs from 1 to 4.
Thanks for all your help.
Alok
On Wed, May 28, 2008 at 1:09 AM, Casey McGinty <[EMAIL PROTECTED]>
wrote:
>
>self.event[] = Event() *# Seems this is not allowed ?? *
>>
>
> self.event = [Event()]
>
> - Casey
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
Regards
Alok Kumar
--
http://mail.python.org/mailman/listinfo/python-list
Re: Struct usages in Python
I am getting following error when tried as you suggested. self.event = [] #Create an empty list, bind to the name "event" under the "self" namespace self.event.append(Event()) #Create an event object and append to the end of the list *class Event(): ^ SyntaxError: invalid syntax* On Wed, May 28, 2008 at 1:07 AM, Patrick Mullen <[EMAIL PROTECTED]> wrote: > I don't know if this will go through (my posts seem to have become blocked > lately), but I'll give it a shot anyhow. > > You seem to be under a misconception that a python list is similar to a > list in say, Java or other languages that have a rigid idea of variables and > types. In python, a list is a list of objects - any type of object can be > stored in a list. Just as you don't declare types for variables, you also > don't declare types for lists. Here is your modified code: > > class Event(): >def __init__(self, cameraEventType="", zone=99, setDay="",setTime ="", > clrTime=""): > self.cameraEventType = cameraEventType > self.zone = zone > self.setDay = setDay > self.setTime = setTime > self.clrTime = clrTime > > class EventTimeFilter: >def __init__(self): >self.event = [] #Create an empty list, bind to the name "event" > under the "self" namespace >self.event.append(Event()) #Create an event object and append > to the end of the list > > > Python won't stop you from putting other objects into self.event besides > Event objects, but in practice this isn't often an issue. The real benefit, > is if you subclass event or make some other type of object that is similar > to events, with maybe some of the same fields, you can still store them in > the list and it will play along with the rest of your code. > > -- > http://mail.python.org/mailman/listinfo/python-list > -- Regards Alok Kumar -- http://mail.python.org/mailman/listinfo/python-list
Re: multi dimensional dictionary
Thanks to all for their kind help and time.
Alok
On Wed, May 28, 2008 at 4:11 AM, Peter Otten <[EMAIL PROTECTED]> wrote:
> Gary Herron wrote:
>
> > Alok Kumar wrote:
> >> Dear All,
> >>
> >> I am using dictionary for filling my xpath parsed data.
> >>
> >> I wanted to use in the following manner.
> >>
> >> mydict[index] ["key1"] ["key2"]#Can someone help me with right
> >> declaration.
> >>
> >> So that I can fill my XML xpath parsed data
> >>
> >> mydict[0] ["person"] ["setTime"] = "12:09:30"
> >> mydict[0] ["person"] ["clrTime"] = "22:09:30"
>
> [I didn't see the original post]
>
> >>> from collections import defaultdict
> >>> def make_inner():
> ... return defaultdict(lambda: defaultdict(make_inner))
> ...
> >>> mydict = make_inner()
> >>> mydict[0]["person"]["setTime"] = "12:09:30"
> >>> mydict[0]["person"]["shoes"]["color"] = "bright yellow"
> >>> mydict
> defaultdict( at 0x2b7afd0025f0>, {0:
> defaultdict( make_inner at 0x2b7afd002578>, {'person': defaultdict( at
> 0x2b7afd002668>, {'setTime': '12:09:30', 'shoes': defaultdict( make_inner at 0x2b7afd002578>, {'color': 'bright yellow'})})})})
>
> If that looks too messy, try a subclass:
>
> >>> class Dict(defaultdict):
> ... def __init__(self):
> ... defaultdict.__init__(self, Dict)
> ... def __repr__(self):
> ... return dict.__repr__(self)
> ...
> >>> mydict = Dict()
> >>> mydict[0]["person"]["setTime"] = "12:09:30"
> >>> mydict
> {0: {'person': {'setTime': '12:09:30'}}}
> >>> mydict[0]["person"]["shoes"]["color"] = "bright yellow"
> >>> mydict
> {0: {'person': {'setTime': '12:09:30', 'shoes': {'color': 'bright
> yellow'
>
> Peter
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
Regards
Alok Kumar
--
http://mail.python.org/mailman/listinfo/python-list
Re: Struct usages in Python
Thanks to everyone for your help. I am able to use array of structure (here Event is a class) in the following manner. But here I am fixing the array index as 4. Is there any easy way to keep it appending dynamically. self.event = [Event() for x in range(4)] # Event is a class as posted in original message. I have one set of structures and want to process them, subsequently couple of them will not be able to pass the criteria and all passed ones I wanted to put in another list where I can dynamically append the structure. So that I can post this list to server. Thanks in advance. Regards Alok On Wed, May 28, 2008 at 1:53 PM, Arnaud Delobelle <[EMAIL PROTECTED]> wrote: > Arnaud Delobelle <[EMAIL PROTECTED]> writes: > > > "Alex Gusarov" <[EMAIL PROTECTED]> writes: > > > >>> class Event(object): > >>> > >>> Always subclass object, unless you have a very compelling reason not > to, > >>> or you are subclassing something else. > >>> > >> > >> I've thought that if I write > >> > >> class Event: > >> pass > >> > >> , it'll be subclass of object too, I was wrong? > > > > You are wrong for Python 2.X, but right for Python 3 where old-style > > classes are gone for good. > > > > What you define with the statement > > > > class Event: pass > > > > is an 'old-style' class. Witness: > > > > >>> class Event: pass > > ... > > >>> class NewEvent(object): pass > > ... > > >>> type(Event) > > > > >>> type(NewEvent) > > > > >>> type(Event()) > > > > del>>> type(NewEvent()) > > > > > > All old-style classes are actually objects of type 'classobj' (they > > all have the same type!), all their instances are all of type 'instance'. > > Oops somthing disappeared in the copy/paste process: > >>>> class FooBar: pass >... > > > >>> type(FooBar) == type(Event) > > True > > >>> type(FooBar()) == type(Event()) > > True > > > > Whereas instances of new-style classes are of type their class: > > > > >>> class NewFooBar(object): pass > > ... > > >>> type(NewFooBar) == type(NewEvent) > > True > > >>> type(NewFooBar()) == type(NewEvent()) > > False > > > > However, in python 2.X (X > 2?), you can force all classes to of a > > certain type by setting the global variable '__metaclass__'. So: > > > > >>> type(Event) # Event is an old-style class > > > > >>> __metaclass__ = type > > >>> class Event: pass > > ... > > >>> type(Event) # Now Event is new-style! > > > > > > HTH > > > > -- > > Arnaud > -- > http://mail.python.org/mailman/listinfo/python-list > -- Regards Alok Kumar -- http://mail.python.org/mailman/listinfo/python-list
UTC datetime.fromtimestamp
Dear All,
I have UTC datetime as datetime.fromtimestamp(ParseDateTimeUTC("2007-12-06
20:37:05"))
How can I add a day into this. Your help will be highly appreciated.
Regards
Alok
--
http://mail.python.org/mailman/listinfo/python-list
ImportError: No module named _md5
I am trying to move from python 2.4 to python 2.5.2. After compiling the python 2.5.2, I found it is throwing "ImportError: No module named _md5". I found this topic on the mailing list, but no solution to fix it. Can someone help me out. It will be highly appreciated. Regards Alok -- http://mail.python.org/mailman/listinfo/python-list
Python HTTPS Bus Error
Hi, Can someone help me what to look for fixing Bus error. I must mention that I am working in embedded environment and it does have selective installation of python I am getting following error from client when it is trying to talk with server over *HTTPS*. HTTP works fine. *Alignment trap: python2.5 (1030) PC=0x4058a738 Instr=0xe5902004 Address=0x01* *35 FSR 0x001* *Bus error* Server throws SSL handshake failure error. It will be great if someone can throw any pointers. Regards Alok -- http://mail.python.org/mailman/listinfo/python-list
Basic Question about Python WebServer File handling
Hi, I need to have a python webserver which can handle Get request from the clients and upload the *files* from 4 different directories. Can someone please point me what to exactly look for. Thanks you very much for this great help. Regards Alok Kumar -- http://mail.python.org/mailman/listinfo/python-list
XML Parsing
Hello,
I am new to XML parsing.Could you kindly tell me whats the
problem with the following code:
import xml.dom.minidom
import xml.parsers.expat
document = """LettermanisbetterthanJayLeno"""
# 3 handler functions
def start_element(name, attrs):
print 'Start element:', name, attrs
def end_element(name):
print 'End element:', name
def char_data(data):
print 'Character data:', repr(data)
p = xml.parsers.expat.ParserCreate()
p.StartElementHandler = start_element
p.EndElementHandler = end_element
p.CharacterDataHandler = char_data
p.Parse(document, 1)
OUTPUT:
Start element: token {u'pos': u'nn'}
Character data: u'Letterman'
End element: token
Traceback (most recent call last):
File "C:/Python25/Programs/eg.py", line 20, in
p.Parse(document, 1)
ExpatError: junk after document element: line 1, column 33
--
http://mail.python.org/mailman/listinfo/python-list
Re: XML Parsing
Thanks ! it worked ! On Wed, Apr 2, 2008 at 1:31 AM, Konstantin Veretennicov < [EMAIL PROTECTED]> wrote: > On Tue, Apr 1, 2008 at 10:42 PM, Alok Kothari <[EMAIL PROTECTED]> > wrote: > > > Hello, > > I am new to XML parsing.Could you kindly tell me whats the > > problem with the following code: > > > > import xml.dom.minidom > > import xml.parsers.expat > > document = """Lettermanis > token>betterthan > token>JayLeno""" > > > > This document is not well-formed. It doesn't have root element. > > ... > > > > > > Traceback (most recent call last): > > File "C:/Python25/Programs/eg.py", line 20, in > >p.Parse(document, 1) > > ExpatError: junk after document element: line 1, column 33 > > > > Told ya :) > > > Try wrapping your document in root element, like > ".." > > -- > kv > -- http://mail.python.org/mailman/listinfo/python-list
HTTP Error code Info
Hi I could get the HTTP error code using try: .. except: .. else: block. Is there any better way to get the HTTP Error code using urllib2 lib. Something like know the exact response number (200, 404 etc) without the above block. Regards Alok Kumar -- http://mail.python.org/mailman/listinfo/python-list
State machine Example in Python
Can someone please redirect me for a state machine example or design pattern used in Python. Regards Alok -- http://mail.python.org/mailman/listinfo/python-list
Python Minidom Help
Hi, I am reading a XML data from HTTP server and giving it to parse method. dom = parse(dataread) It throws following error File "/usr/lib/python2.3/site-packages/_xmlplus/dom/minidom.py", line 1908, in parse return expatbuilder.parse(file) File "/usr/lib/python2.3/site-packages/_xmlplus/dom/expatbuilder.py", line 922, in parse fp = open(file, 'rb') Can anyone please help? Regards Alok Kumar -- http://mail.python.org/mailman/listinfo/python-list
editing video using python
Hello everyone, I want to join 2 or 3 video and want to add effect of fading in and out. I also want to add text strip on the video. I am not able to decide what to adopt for doing this task. I have option of MLT ( http://www.mltframework.org/bin/view/MLT/WebHome ) but I guess its too complex and I didnt find any documentation of MLT I have another option of FFmpeg, but I am not sure whether I can do all three requirement with ffmpeg. if I adopt ffmpeg then should I use ffmpeg's wrapper for python like http://code.google.com/p/pyffmpeg/ or I should use pure ffmpeg in python as subprocess? how is pymedia? can I do my task with pymedia? please tell me easy way to do my task in python. Thank you -- https://mail.python.org/mailman/listinfo/python-list
web development in python without using any webframework
Hello everyone, few months back I started learning python and now I got nice familiarity. now i want to use python for creating dynamic database driven websites. and I dont want to use existing web frameworks for my work. I am learning things so I wont feel lazy to write all the code myself because I want to learn. could anyone suggest me any books/site from where I can start. I want to follow MVC architecture. so please suggest me some links/book or anything thank you in advance -- http://mail.python.org/mailman/listinfo/python-list
django vs zope vs web2py
Hi everyone, few months back I decided to adopt python for my all sort of work including web progra -- http://mail.python.org/mailman/listinfo/python-list
Re: django vs zope vs web2py
I am sorry by mistake I sent incomplete mail here is my mail. Hi everyone, few months back I decided to adopt python for my all sort of work including web programming. and I have wasted long time deciding which to adopt out of django, zope and web2py. I am from php and drupal background. which framework would be better for me. I am open to learn anything, anything new. but I want to adopt best thing full of features and lot of plugins/extensions and easy to use and have better documentation and books etc. please suggest me so without wasting more time I can start learning thanks in advance :) -- http://mail.python.org/mailman/listinfo/python-list
Re: django vs zope vs web2py
On Sun, Apr 21, 2013 at 5:49 PM, Modulok wrote: > > Hi everyone, > > few months back I decided to adopt python for my all sort of work > including > > web programming... > > -- > > http://mail.python.org/mailman/listinfo/python-list > > > > Pick Django or web2py. You'll be happy with either. (I have no experience > with > zope.) > > They're both full featured do-everything-you-ever-wanted frameworks with > great > communities and lots of documentation. You can buy books on either. I'd say > web2py is a little more elegant and easier to get started with. (An > admittedly > subjective claim.) Django has a little larger community and has more third > party stuff. > > If you just need to "get it done" and don't care about how it happens, > they're > both excellent. You'll meet deadlines with either of them. The communities > are > smart the docs are great. You can't really go wrong any way you slice it. > There's more third party documentation and books for Django right now but > that's just because Django came out first. Give it another couple years and > there won't be much difference. > > Basically, flip a coin and just go with it. > > > > And now for the gritty details approach... > > The problem with web frameworks is they are "magic", i.e. things just > happen. > It's the price we pay for using a high level abstraction. The higher the > abstraction the more magic there is. Often times magic is good. It saves us > time and money. However depending on your needs, other options are worth > considering. > > If you are willing to invest a lot of time not being initially productive > but > learn a *ton* in exchange, you can use something like cherrypy. (Don't get > me > wrong, I love and often use cherrypy.) It's dirt simple and works. However, > because it's so simple it doesn't do half of what you need for a > non-trivial > production site. Result? You'll have to fill in the tool chain gaps with > other > modules. This is what web frameworks do for you. > > If you go the cherrypy route you'll need to learn other things like like > markup > languages and some kind of way to talk to a database. Security is also > entirely > in your hands. You'll learn a ton about HTTP, SQL, markup languages, web > security, encryption, etc. You'll be basically re-creating a web framework > of > your own brand. Again it's a time spent vs. knowledge gained trade off. > > For a template language I really liked wheezy.template but it's a little > too > new for me to feel comfortable using it on any big project. It has a very > simple inheritance model, which is refreshing. I hope to use it more in the > future. > > I usually use Mako for my templates. By 'template' I mean any template, not > just HTML. I use mako for HTML, documentation, I even use mako to write SQL > templates. The inheritance model of Mako takes a little more mental > gymnastics > to wrap your head around than the simpler (read nicer) wheezy.template > model, > but it's a more mature code base. (Not as mature as cheetah.) I had only > minor > experience with cheetah but found I preferred Mako. It was a matter of > taste. > There's nothing wrong with cheetah. > > As for database access: sqlalchemy is truly excellent and very flexible. > For > most things sqlalchemy is great. However for some projects it may contain > too > much magic. (Again we're going deeper.) Sometimes a backend-specific > module is > called for, in which case psycopg2 on postgresql is nice. The ability to > use > python context managers as transaction blocks is very clean. > > In short, how much do you want to learn? Do you prefer a top-down or > bottom-up > approach? Gritty details or elegant abstractions? > > -Modulok- > thanks a lot Rusi, Roy Smith, Surya and Modulok I am sticking to django. In future I will touch web2py also -- http://mail.python.org/mailman/listinfo/python-list
