Re: What is heating the memory here? hashlib?
tracemalloc module may help you to investigate leaks. 2016/02/14 午後4:05 "Paulo da Silva" : > I was unable to reproduce the situation using a simple program just > walking through all files>4K, with or without the seek, and computing > their shasums. > Only some fluctuations of about 500MB in memory consumption. > > I'll look at this when I get more time, taking in consideration the > suggestions here posted. > > For the time being, my work is done. With a small buffer size (16k) the > results produced were correct and no memory was leaked! > > If I can find any explanation (if not embarrassing :-) ), I'll post it > here. > > Thank you all. > Paulo > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
ftplib throws: IndexError: tuple index out of range
I have written a small backup program, that uses ftplib to make
remote backups. However recentely the program starts to regularly
raise IndexErrors, as far as I can see the problem is in socket.py
Can anyone shed some light?
This is the traceback:
Traceback (most recent call last):
File "/usr/local/lib/python-apps/rmtdump/pymain.py", line 93, in main
Exit_Nr = process(sys.argv)
File "/usr/local/lib/python-apps/rmtdump/rmtdump.py", line 228, in program
process(path)
File "/usr/local/lib/python-apps/rmtdump/rmtdump.py", line 170, in process
of = rmt.open(rmtsep.join([rmt5lcl(rt), archive]), "wb")
File "/usr/local/lib/python-apps/rmtdump/ftputil.py", line 198, in open
return ftpfile(ftp, fn, mode, True)
File "/usr/local/lib/python-apps/rmtdump/ftputil.py", line 69, in __init__
self.cnct = self.ftp.transfercmd("%s %s" % (cmd, rfn))
File "/usr/lib/python2.7/ftplib.py", line 376, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "/usr/lib/python2.7/ftplib.py", line 693, in ntransfercmd
conn, size = FTP.ntransfercmd(self, cmd, rest)
File "/usr/lib/python2.7/ftplib.py", line 339, in ntransfercmd
resp = self.sendcmd(cmd)
File "/usr/lib/python2.7/ftplib.py", line 249, in sendcmd
return self.getresp()
File "/usr/lib/python2.7/ftplib.py", line 215, in getresp
resp = self.getmultiline()
File "/usr/lib/python2.7/ftplib.py", line 201, in getmultiline
line = self.getline()
File "/usr/lib/python2.7/ftplib.py", line 186, in getline
line = self.file.readline(self.maxline + 1)
File "/usr/lib/python2.7/socket.py", line 478, in readline
if e.args[0] == EINTR:
IndexError: tuple index out of range
Locals by frame, innermost last
Frame main in /usr/local/lib/python-apps/rmtdump/pymain.py at line 111
backtrace =
pn = 'rmtdump'
process =
Frame program in /usr/local/lib/python-apps/rmtdump/rmtdump.py at line 261
aborted = True
argv = ['/usr/local/sbin/rmtdump']
path = '/home'
Frame process in /usr/local/lib/python-apps/rmtdump/rmtdump.py at line 190
b_rt = '/home'
backupflag = 'r--'
backupstamp = 1453021366
checkstamp = 1450374304
dirs = ['antoon', 'jenny', 'lost+found', 'photo']
entries = ['0-home', 'antoon', 'homelady', 'jenny', 'lost+found',
'photo', 'recover']
entry = 'recover'
fileinfo = posix.stat_result(st_mode=16877, st_ino=2490369,
st_dev=2058L, st_nlink=5, st_uid=0, st_gid=0, st_size=4 ...
files = []
fqpn = '/home/recover'
islink =
ismount =
last = 1455445804.979914
newfn = 'home/!2016-01-17@10_02_46-tbz'
now = 1455445804
nr_of_tries = 0
of = None
oldfn = 'home/!ENTRY-tbz'
path = '/home'
rt = '/home'
stamp = '1450374304'
start = 1455445804
timefn = '!2016-01-17@10_02_46-tbz'
tryal = 1
update = True
Frame open in /usr/local/lib/python-apps/rmtdump/ftputil.py at line 198
fn = 'home/!ENTRY-tbz'
ftp =
mode = 'wb'
self =
Frame __init__ in /usr/local/lib/python-apps/rmtdump/ftputil.py at line 69
bound = True
ch = 'b'
cmd = 'STOR'
cn =
count = 1
mode = 'wb'
rfn = 'home/!ENTRY-tbz'
self = None
Frame transfercmd in /usr/lib/python2.7/ftplib.py at line 376
cmd = 'STOR home/!ENTRY-tbz'
rest = None
self =
Frame ntransfercmd in /usr/lib/python2.7/ftplib.py at line 693
cmd = 'STOR home/!ENTRY-tbz'
rest = None
self =
Frame ntransfercmd in /usr/lib/python2.7/ftplib.py at line 352
cmd = 'STOR home/!ENTRY-tbz'
conn =
host = '192.110.0.54'
port = 64649
rest = None
self =
size = None
Frame sendcmd in /usr/lib/python2.7/ftplib.py at line 249
cmd = 'STOR home/!ENTRY-tbz'
self =
Frame getresp in /usr/lib/python2.7/ftplib.py at line 215
self =
Frame getmultiline in /usr/lib/python2.7/ftplib.py at line 201
self =
Frame getline in /usr/lib/python2.7/ftplib.py at line 186
self =
Frame readline in /usr/lib/python2.7/socket.py at line 478
buf =
buf_len = 0
e = timeout()
self =
size = 8193
--
https://mail.python.org/mailman/listinfo/python-list
Re: ftplib throws: IndexError: tuple index out of range
Hi Antoon,
EINTR, is an error when there is an emited signal to your process.
http://man7.org/linux/man-pages/man2/read.2.html
Look for EINTR in this page
On 02/14, Antoon Pardon wrote:
I have written a small backup program, that uses ftplib to make
remote backups. However recentely the program starts to regularly
raise IndexErrors, as far as I can see the problem is in socket.py
Can anyone shed some light?
This is the traceback:
Traceback (most recent call last):
File "/usr/local/lib/python-apps/rmtdump/pymain.py", line 93, in main
Exit_Nr = process(sys.argv)
File "/usr/local/lib/python-apps/rmtdump/rmtdump.py", line 228, in program
process(path)
File "/usr/local/lib/python-apps/rmtdump/rmtdump.py", line 170, in process
of = rmt.open(rmtsep.join([rmt5lcl(rt), archive]), "wb")
File "/usr/local/lib/python-apps/rmtdump/ftputil.py", line 198, in open
return ftpfile(ftp, fn, mode, True)
File "/usr/local/lib/python-apps/rmtdump/ftputil.py", line 69, in __init__
self.cnct = self.ftp.transfercmd("%s %s" % (cmd, rfn))
File "/usr/lib/python2.7/ftplib.py", line 376, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "/usr/lib/python2.7/ftplib.py", line 693, in ntransfercmd
conn, size = FTP.ntransfercmd(self, cmd, rest)
File "/usr/lib/python2.7/ftplib.py", line 339, in ntransfercmd
resp = self.sendcmd(cmd)
File "/usr/lib/python2.7/ftplib.py", line 249, in sendcmd
return self.getresp()
File "/usr/lib/python2.7/ftplib.py", line 215, in getresp
resp = self.getmultiline()
File "/usr/lib/python2.7/ftplib.py", line 201, in getmultiline
line = self.getline()
File "/usr/lib/python2.7/ftplib.py", line 186, in getline
line = self.file.readline(self.maxline + 1)
File "/usr/lib/python2.7/socket.py", line 478, in readline
if e.args[0] == EINTR:
IndexError: tuple index out of range
Locals by frame, innermost last
Frame main in /usr/local/lib/python-apps/rmtdump/pymain.py at line 111
backtrace =
pn = 'rmtdump'
process =
Frame program in /usr/local/lib/python-apps/rmtdump/rmtdump.py at line 261
aborted = True
argv = ['/usr/local/sbin/rmtdump']
path = '/home'
Frame process in /usr/local/lib/python-apps/rmtdump/rmtdump.py at line 190
b_rt = '/home'
backupflag = 'r--'
backupstamp = 1453021366
checkstamp = 1450374304
dirs = ['antoon', 'jenny', 'lost+found', 'photo']
entries = ['0-home', 'antoon', 'homelady', 'jenny', 'lost+found',
'photo', 'recover']
entry = 'recover'
fileinfo = posix.stat_result(st_mode=16877, st_ino=2490369,
st_dev=2058L, st_nlink=5, st_uid=0, st_gid=0, st_size=4 ...
files = []
fqpn = '/home/recover'
islink =
ismount =
last = 1455445804.979914
newfn = 'home/!2016-01-17@10_02_46-tbz'
now = 1455445804
nr_of_tries = 0
of = None
oldfn = 'home/!ENTRY-tbz'
path = '/home'
rt = '/home'
stamp = '1450374304'
start = 1455445804
timefn = '!2016-01-17@10_02_46-tbz'
tryal = 1
update = True
Frame open in /usr/local/lib/python-apps/rmtdump/ftputil.py at line 198
fn = 'home/!ENTRY-tbz'
ftp =
mode = 'wb'
self =
Frame __init__ in /usr/local/lib/python-apps/rmtdump/ftputil.py at line 69
bound = True
ch = 'b'
cmd = 'STOR'
cn =
count = 1
mode = 'wb'
rfn = 'home/!ENTRY-tbz'
self = None
Frame transfercmd in /usr/lib/python2.7/ftplib.py at line 376
cmd = 'STOR home/!ENTRY-tbz'
rest = None
self =
Frame ntransfercmd in /usr/lib/python2.7/ftplib.py at line 693
cmd = 'STOR home/!ENTRY-tbz'
rest = None
self =
Frame ntransfercmd in /usr/lib/python2.7/ftplib.py at line 352
cmd = 'STOR home/!ENTRY-tbz'
conn =
host = '192.110.0.54'
port = 64649
rest = None
self =
size = None
Frame sendcmd in /usr/lib/python2.7/ftplib.py at line 249
cmd = 'STOR home/!ENTRY-tbz'
self =
Frame getresp in /usr/lib/python2.7/ftplib.py at line 215
self =
Frame getmultiline in /usr/lib/python2.7/ftplib.py at line 201
self =
Frame getline in /usr/lib/python2.7/ftplib.py at line 186
self =
Frame readline in /usr/lib/python2.7/socket.py at line 478
buf =
buf_len = 0
e = timeout()
self =
size = 8193
--
https://mail.python.org/mailman/listinfo/python
Re: problem with dateutil
On 02/13/2016 09:45 PM, Gary Herron wrote: On 02/13/2016 12:27 PM, Tom P wrote: On 02/13/2016 07:13 PM, Gary Herron wrote: On 02/13/2016 09:58 AM, Tom P wrote: I am writing a program that has to deal with various date/time formats and convert these into timestamps. It looks as if dateutil.parser.parse should be able to handle about any format, but what I get is: datetimestr = '2012-10-22 11:22:33' print(dateutil.parser.parse(datetimestr)) result: datetime.datetime(2012, 10, 22, 11, 22, 33) However: datetimestr = '2012:10:22 11:22:33' print(dateutil.parser.parse(datetimestr)) result: datetime.datetime(2016, 2, 13, 11, 22, 33) In other words, it's getting the date wrong when colons are used to separate :MM:DD. Is there a way to include this as a valid format? Yes, there is a way to specify your own format. Search the datetime documentation for datetime.strptime(date_string, format) Gary Herron Thanks. I started out with datetime.strptime but AFAICS that means I have to go through try/except for every conceivable format. Are you saying that I can't use dateutil.parser? Well now... If by "every conceivable format" you are including formats that the author of dateutil.parser did not conceive of, then of course you cannot use dateutil.parser. But you have the code for dateutil.parser -- perhaps you could modify it to accept whatever odd formats you care about. Gary Herron I had a look at the code for dateutil.parser. Have you looked at it? Meanwhile I'm living with try: dt = datetime.datetime.strptime(datetimestr, '%Y:%m:%d %H:%M:%S') except ValueError: dt = dateutil.parser.parse(datetimestr) unixtime = time.mktime(dt.timetuple()) -- https://mail.python.org/mailman/listinfo/python-list
Re: problem with dateutil
On 02/13/2016 10:01 PM, Mark Lawrence wrote: On 13/02/2016 17:58, Tom P wrote: I am writing a program that has to deal with various date/time formats and convert these into timestamps. It looks as if dateutil.parser.parse should be able to handle about any format, but what I get is: datetimestr = '2012-10-22 11:22:33' print(dateutil.parser.parse(datetimestr)) result: datetime.datetime(2012, 10, 22, 11, 22, 33) However: datetimestr = '2012:10:22 11:22:33' print(dateutil.parser.parse(datetimestr)) result: datetime.datetime(2016, 2, 13, 11, 22, 33) In other words, it's getting the date wrong when colons are used to separate :MM:DD. Is there a way to include this as a valid format? From http://labix.org/python-dateutil#head-a23e8ae0a661d77b89dfb3476f85b26f0b30349c parserinfo This parameter allows one to change how the string is parsed, by using a different parserinfo class instance. Using it you may, for example, intenationalize the parser strings, or make it ignore additional words. HTH. Thanks, let me look at that. -- https://mail.python.org/mailman/listinfo/python-list
Syntax error (The Python Book) Linux User and Developer Bookazine
Hi,
Noob at the Python thing so here goes,
I have copied a program to demonstrate control structures in Python but get a
syntax error at line 31, isint = False. I'm using Python 2.7.6 and Linux Mint
based around ubuntu14.04.1. I have pasted all the code below,
#!/usr/bin/env python2
'''
We are going to write a program that will ask for the user to input an arbitary
number of integers, store them in a collection, and then demonstrate how
the collection would be used with various control structures
'''
import sys # Used for the sys.exit function
target_int=raw_input("How many integers?")
'''
By now the variable target_int contains a string representation of
whatever the user typed. We nee to try and convert that to an integer but
be ready to deal with the error if it's not. Otherwise the program will crash
'''
try:
target_int=int(target_int)
except ValueError:
sys.exit("You must enter an integer")
ints=list() # list to store the integers
count = 0 # Track how many integers have been inputted
# Keep asking for a number until we have reached the required number
while count < target_int:
new_int=raw_input("Please enter integer {0}:".format(count +1)
isint = False
try:
new_int=int(new_int) # If the above succeeds then isint will
#be set to true: isint = True
except:
print("You must enter an integer")
'''
Only carry on if we have an integer. If not we will loop again.
The == below is a comparision operator, a single = is an asignment operator
'''
if isnit==True:
ints.append(new_int) # Adds the integer to the collection
count += 1 # Count is incremented by 1
# The for loop
print ("Using a for loop")
for values in ints:
print (str(value))
# The while loop
print ("Using a while loop")
total=len(ints) # We already have the total from above but using len we
can determine from the ints list.
count = 0
while count < total:
print (str(ints[count]))
count += 1
--
https://mail.python.org/mailman/listinfo/python-list
Re: ftplib throws: IndexError: tuple index out of range
Antoon Pardon wrote: > I have written a small backup program, that uses ftplib to make > remote backups. However recentely the program starts to regularly > raise IndexErrors, as far as I can see the problem is in socket.py > Can anyone shed some light? > > This is the traceback: [...] > File "/usr/lib/python2.7/socket.py", line 478, in readline > if e.args[0] == EINTR: > IndexError: tuple index out of range The offending line seems to be part of try: data = self._sock.recv(self._rbufsize) except error, e: if e.args[0] == EINTR: continue raise > Locals by frame, innermost last [...] > Frame readline in /usr/lib/python2.7/socket.py at line 478 > buf = > buf_len = 0 >e = timeout() > self = > size = 8193 It looks like the actual error is socket.timeout which is probably raised from somewhere inside the stdlib without args. I think this is a bug in socket.py; the error handler should special-case with either try: ... except timeout, e: raise except error, e: ... or try: ... except error, e: if e.args and e.args[0] == EINTR: continue raise You will still have to find the cause and handle the effects of the timeout. PS: How did you produce the overview over the local variables? That looks nice. -- https://mail.python.org/mailman/listinfo/python-list
Re: How do i instantiate a class_name passed via cmd line
Gregory Ewing wrote: > Something like this should do it: > >instance = getattr(module, class_name)(module_name, product) > > If the class name is always the same as the module name with the > first letter capitalized, you could use > >instance = getattr(module, module_name.capitalize())(module_name, >product) If the convention is that strict you could also expose the class under an alias: # in ebay.py Site = Ebay # in the client module module = importlib.import_module(module_name) instance = module.Site(product) -- https://mail.python.org/mailman/listinfo/python-list
Re: Syntax error (The Python Book) Linux User and Developer Bookazine
Geoff Munn wrote:
> Noob at the Python thing so here goes,
>
> I have copied a program to demonstrate control structures in Python but
> get a syntax error at line 31, isint = False.
Often the actual syntax error in your code is in one of the lines preceding
the one that Python is complaining about.
> new_int=raw_input("Please enter integer {0}:".format(count +1)
> isint = False
Hint: Do the parentheses match?
--
https://mail.python.org/mailman/listinfo/python-list
Re: Syntax error (The Python Book) Linux User and Developer Bookazine
On Mon, Feb 15, 2016 at 12:39 AM, Geoff Munn wrote:
> Hi,
>
> Noob at the Python thing so here goes,
>
> I have copied a program to demonstrate control structures in Python but get a
> syntax error at line 31, isint = False. I'm using Python 2.7.6 and Linux Mint
> based around ubuntu14.04.1. I have pasted all the code below,
>
Peter's already explained your actual problem, so I'll make a few
other comments about the code - starting with one about... comments.
> '''
> By now the variable target_int contains a string representation of
> whatever the user typed. We nee to try and convert that to an integer but
> be ready to deal with the error if it's not. Otherwise the program will crash
> '''
This isn't a comment. It's a triple-quoted string literal. Since a
string literal as an expression on its own is legal and insignificant,
this doesn't hurt much, but just be aware that these blocks aren't
comments. (BTW, is the typo "nee to try" (s/be "need") from the
original?)
> try:
> target_int=int(target_int)
> except ValueError:
> sys.exit("You must enter an integer")
What does "crash" mean? In the case of Python, failing to check for
this exception will result in a message printed to stderr and program
termination. Instead, the exception is caught... and a message is
printed to stderr and the program terminated. Is it really worth the
effort?
> while count < target_int:
> new_int=raw_input("Please enter integer {0}:".format(count +1)
> isint = False
> try:
> new_int=int(new_int) # If the above succeeds then isint will
> #be set to true: isint = True
>
> except:
> print("You must enter an integer")
A bare except clause! Bad idea. Never do this. In the *extremely* rare
cases when you actually do want to catch absolutely everything, you
can spell it "except BaseException:", but most of the time, you want
to catch one specific exception.
> '''
> Only carry on if we have an integer. If not we will loop again.
> The == below is a comparision operator, a single = is an asignment
> operator
> '''
> if isnit==True:
> ints.append(new_int) # Adds the integer to the collection
> count += 1 # Count is incremented by 1
Despite your comments, isint is never set to true - and isnit is
always going to be a NameError. Based on the number of typos here, I'm
wondering if we can actually depend on the code that IS having
trouble, which Peter mentioned as being a parenthesis count; maybe
these are all transcription errors?
> # The for loop
> print ("Using a for loop")
> for values in ints:
> print (str(value))
> # The while loop
> print ("Using a while loop")
> total=len(ints) # We already have the total from above but using len we
> can determine from the ints list.
> count = 0
> while count < total:
> print (str(ints[count]))
> count += 1
These two loops are inside your outer while loop - is that intentional?
I strongly recommend not using the name 'count' for two completely
different jobs in the same loop. Although the first one isn't actually
used, which kinda makes it a bit pointless.
This is not an example of Python best practice. It may be teaching you
some things, but don't imitate its style.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: ftplib throws: IndexError: tuple index out of range
Op 14-02-16 om 14:40 schreef Peter Otten: > Antoon Pardon wrote: > >> I have written a small backup program, that uses ftplib to make >> remote backups. However recentely the program starts to regularly >> raise IndexErrors, as far as I can see the problem is in socket.py >> Can anyone shed some light? >> >> This is the traceback: > > [...] > >> File "/usr/lib/python2.7/socket.py", line 478, in readline >> if e.args[0] == EINTR: >> IndexError: tuple index out of range > > The offending line seems to be part of > > try: > data = self._sock.recv(self._rbufsize) > except error, e: > if e.args[0] == EINTR: > continue > raise > > >> Locals by frame, innermost last > > [...] > >> Frame readline in /usr/lib/python2.7/socket.py at line 478 >> buf = >> buf_len = 0 >>e = timeout() >> self = >> size = 8193 > > It looks like the actual error is socket.timeout which is probably raised > from somewhere inside the stdlib without args. I think I know what is going on. I have my own timeout mechanism at work here, that works with signals and alarm. When the SIGALRM fires I just raise the socket.timeout without arguments, which then causes the problem. I defined my own timeout exception class and now raise that and the problem disappeared. Thanks for the extra pair of eyes. -- Antoon Pardon. -- https://mail.python.org/mailman/listinfo/python-list
Re: ftplib throws: IndexError: tuple index out of range
Op 14-02-16 om 14:40 schreef Peter Otten: > > PS: How did you produce the overview over the local variables? That looks > nice. > I started from this recipe: http://code.activestate.com/recipes/52215-get-more-information-from-tracebacks/ made it more to my liking and turned it into a module. So that instead of a program being run like this: def program(argv): ... if __name__ == '__main__': program(sys.argv) I start it now like this from pymain import main def program(argv): ... if __name__ == '__main__': main(program) pymain.main will then trap almost all exceptions and produces the overview over the local variables. -- Antoon. -- https://mail.python.org/mailman/listinfo/python-list
Make a unique filesystem path, without creating the file
Howdy all, How should a program generate a unique filesystem path and *not* create the filesystem entry? The ‘tempfile.mktemp’ function is strongly deprecated, and rightly so https://docs.python.org/3/library/tempfile.html#tempfile.mktemp> because it leaves the program vulnerable to insecure file creation. In some code (e.g. unit tests) I am calling ‘tempfile.mktemp’ to generate a unique path for a filesystem entry that I *do not want* to exist on the real filesystem. In this case the filesystem security concerns are irrelevant because there is no file. The deprecation of that function is a concern still, because I don't want code that makes every conscientious reader need to decide whether the code is a problem. Instead the code should avoid rightly-deprecated APIs. It is also prone to that API function disappearing at some point in the future, because it is explicitly and strongly deprecated. So I agree with the deprecation, but the library doesn't appear to provide a replacement. What standard library function should I be using to generate ‘tempfile.mktemp’-like unique paths, and *not* ever create a real file by that path? -- \“If you have the facts on your side, pound the facts. If you | `\ have the law on your side, pound the law. If you have neither | _o__) on your side, pound the table.” —anonymous | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Ben Finney wrote: > How should a program generate a unique filesystem path and *not* create > the filesystem entry? The Python documentation suggests that it should not. > The ‘tempfile.mktemp’ function is strongly deprecated, and rightly so > https://docs.python.org/3/library/tempfile.html#tempfile.mktemp> > because it leaves the program vulnerable to insecure file creation. > > In some code (e.g. unit tests) I am calling ‘tempfile.mktemp’ to > generate a unique path for a filesystem entry that I *do not want* to > exist on the real filesystem. In this case the filesystem security > concerns are irrelevant because there is no file. I do not think that you have properly understood the problems with tmpfile.mktemp(). > […] > It is also prone to that API function disappearing at some point in the > future, because it is explicitly and strongly deprecated. > > So I agree with the deprecation, but the library doesn't appear to > provide a replacement. | mktemp() usage can be replaced easily with NamedTemporaryFile(), passing | it the delete=False parameter: [example] > What standard library function should I be using to generate > ‘tempfile.mktemp’-like unique paths, and *not* ever create a real file > by that path? I do not think it is possible to avoid the creation of a real file using the PSL; in fact, that a file is created appears to be precisely what fixes the problems with tempfile.mktemp() because then it cannot happen that someone else creates a file with the same name at the same time: | tempfile.NamedTemporaryFile(mode='w+b', buffering=None, encoding=None, | newline=None, suffix=None, prefix=None, dir=None, delete=True) | | This function operates exactly as TemporaryFile() does, except that the | file is guaranteed to have a visible name in the file system (on Unix, the | directory entry is not unlinked). […] If delete is true (the default), the | file is deleted as soon as it is closed. […] It is of course possible to generate a filename that is not currently used, but I am not aware of a PSL feature that does this, and if there were such a feature there would be the same problems with it as with mktemp(). -- PointedEars Twitter: @PointedEars2 Please do not cc me. / Bitte keine Kopien per E-Mail. -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On 14 Feb 2016 21:46, "Ben Finney" wrote: > What standard library function should I be using to generate > ‘tempfile.mktemp’-like unique paths, and *not* ever create a real file > by that path? Could you use tempfile.TemporaryDirectory and then just use a consistent name within that directory. It's guaranteed not to exist because the directory was only just created and only you can write to it? Has the added bonus of still being reasonably secure, to appease people like Mr PointedEars. (If you need multiple nonexistent paths in the same dir then perhaps use tempfile.NamedTemporaryFile with your newly created temp dir and an arbitrary suffix, and strip the suffix off to get the name you actually use.) -- Matt Wheeler http://funkyh.at -- https://mail.python.org/mailman/listinfo/python-list
LIGO and Python
On the PSF-Community list, a Python user points out that LIGO uses Python, with numpy and IPython, extensively. He linked to https://losc.ligo.org/s/events/GW150914/GW150914_tutorial.html which explains, with code, how the gravity wave signal was pulled out of the (publicly available) raw data. I found it an interesting read. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Matt Wheeler writes: > On 14 Feb 2016 21:46, "Ben Finney" wrote: > > What standard library function should I be using to generate > > ‘tempfile.mktemp’-like unique paths, and *not* ever create a real > > file by that path? > > Could you use tempfile.TemporaryDirectory and then just use a > consistent name within that directory. That fails because it touches the filesystem. I want to avoid using a real file or a real directory. > It's guaranteed not to exist I am unconcerned with whether there is a real filesystem entry of that name; the goal entails having no filesystem activity for this. I want a valid unique filesystem path, without touching the filesystem. -- \ “I believe our future depends powerfully on how well we | `\ understand this cosmos, in which we float like a mote of dust | _o__) in the morning sky.” —Carl Sagan, _Cosmos_, 1980 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
How to properly override the default factory of defaultdict?
I want to pass in the key to the default_factory of defaultdict and I found that defaultdict somehow can intercept my call to dict.__getitem__(self, key), so my class's __getitem__ have to catch a TypeError instead instead of KeyError. The following class is my code: class DefaultDictWithEnhancedFactory(defaultdict): """Just like the standard python collections.dict, but the default_factory takes the missing key as argument. Args: default_factory: A function that takes the missing key as the argument and return a value for the missing key. *a: arguments passing to the defaultdict constructor **kw: keyword arguments passing to the defaultdict constructor """ def __init__(self, default_factory, *a, **kw): defaultdict.__init__(self, default_factory, *a, **kw) def __getitem__(self, key): try: return dict.__getitem__(self, key) except KeyError: # Normally, you would expect this line to be # called for missing keys... return self.default_factory(key) except TypeError as ex: # However, this is actually getting called # because for some reason, defaultdict still # intercepts the __getitem__ call and raises: # TypeError: () takes exactly 1 argument (0 given) # So we have to catch that instead... if "lambda" in str(ex): return self.default_factory(key) -- https://mail.python.org/mailman/listinfo/python-list
Creating a List from a Loop
I'm in an Intro to Python class and have an assignment where I need to implement a function calculating distance between two points, then create a loop to find points within a certain distance and create a list from those points. I was able to create the function, but I cant get my loop to create the list. Any help would be appreciated. I can't figure out the solution from my textbooks or other web sources. my current work: https://ghostbin.com/paste/qypvb -- https://mail.python.org/mailman/listinfo/python-list
Re: How to properly override the default factory of defaultdict?
Herman writes: > I want to pass in the key to the default_factory of defaultdict and I > found that defaultdict somehow can intercept my call to > dict.__getitem__(self, key), so my class's __getitem__ have to catch a > TypeError instead instead of KeyError. The following class is my code: I don't see an example of using that code, so that we can use it the same way you are. So I'll comment on some things that may not be relevant but nevertheless should be addressed: > class DefaultDictWithEnhancedFactory(defaultdict): > """Just like the standard python collections.dict, > but the default_factory takes the missing key as argument. Your doc string should have only a brief (about 50–60 characters) single line synopsis. If it's longer, you may be writing something too complex. (See PEP 257.) > Args: The class itself doesn't take arguments; its numerous methods do. I think you mean these descriptions to be in the ‘__init__’ method's doc string. > def __init__(self, default_factory, *a, **kw): > defaultdict.__init__(self, default_factory, *a, **kw) > > def __getitem__(self, key): > try: > return dict.__getitem__(self, key) you are using the inheritance hierarchy but thwarting it by not using ‘super’. Instead:: super().__init__(self, default_factory, *a, **kw) and:: super().__getitem__(self, key) If you're not using Python 3 (and you should, for new code), ‘super’ is a little more complex. Migrating to Python 3 has this advantage among many others. -- \ "Those who will not reason, are bigots, those who cannot, are | `\fools, and those who dare not, are slaves." —“Lord” George | _o__)Gordon Noel Byron | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Creating a List from a Loop
On Mon, Feb 15, 2016 at 11:23 AM, Kaley Tucker wrote: > I'm in an Intro to Python class and have an assignment where I need to > implement a function calculating distance between two points, then create a > loop to find points within a certain distance and create a list from those > points. I was able to create the function, but I cant get my loop to create > the list. Any help would be appreciated. I can't figure out the solution from > my textbooks or other web sources. > > my current work: > > https://ghostbin.com/paste/qypvb Hi Kaley! Welcome to python-list; you're learning one of the best languages in the world. When you have questions like this, it's usually best to include your source code right in the body of the email, rather than including a link or an attachment. Attachments often don't survive, and links can break (plus, some people will simply refuse to click on them). Here's the code: import math blue_points = [[ 30 , 536254.99137 , 3659453.06343 ], [ 33 , 536721.584912 , 3659162.97207 ], [ 50 , 535807.099324 , 3659576.92825 ], [ 112 , 536827.131371 , 3657913.01245 ], [ 117 , 536473.254082 , 3659433.57702 ], [ 120 , 536196.9844 , 3658713.72722 ], [ 127 , 536387.547701 , 3658527.70015 ], [ 133 , 537397.838429 , 3659554.48657 ], [ 144 , 537715.931243 , 3658625.59997 ], [ 166 , 538367.648437 , 3658867.34288 ], [ 172 , 537112.662366 , 3657921.28957 ], [ 173 , 536418.315024 , 3658715.47946 ], [ 209 , 538096.28422 , 3658514.93514 ], [ 211 , 538077.87716 , 3658138.39337 ], [ 223 , 536220.396985 , 3659243.54161 ], [ 242 , 536102.087002 , 3658703.61054 ], [ 244 , 536968.755886 , 3659409.42857 ], [ 246 , 535996.903591 , 3658705.08691 ], [ 275 , 538078.165429 , 3659022.35547 ], [ 303 , 535999.885405 , 3658521.91524 ]] red_point = [1, 1073706.744,3658967.925] #points [ID, x coordinate, y coordinate] def identifyNeighbor(point, c_point): cid = c_point[0] cx = c_point[1] cy = c_point[2] bid = point[0] x = point[1] y = point [2] c_point = (cid, cx, cy) point = (bid, x, y) dx = cx - x dy = cy - y dsquared = dx**2 + dy**2 edistance = math.sqrt(dsquared) if edistance <= 536000: print "True" else: print "False" neighbor = [] def neighborloop(p): while p <= 18: p = p + 1 identifyNeighbor(blue_points[p], red_point) if identifyNeighbor == "True": neighbor.append(blue_points[p][0]) What you've done here is make a function which *prints* the word "True" or "False". Instead, you want to *return* something. Check your class textbook or other course materials; you should have been introduced to this concept. Once you figure that part out, you'll probably want to do the same thing in neighborloop too; don't use a global list, but instead, create a new list inside the function, and return that list. I'll let you explore the details on your own, rather than giving it all away (where's the fun in that!). Incidentally, your print statements indicate that you're using Python 2 here. I strongly recommend using Python 3 instead; the differences aren't huge, but it's much easier if you start on the modern side of the fence, instead of the backward-compatibility mode. There are a few small syntactic changes, but the most significant change involves Unicode handling, and trust me, it's a LOT easier if you learn that correctly from the beginning, rather than going through ASCII -> Extended ASCII -> DBCS -> Extended ASCII -> Unicode, the way I did! Nothing in your code actually calls any of these functions. Is that how you intend to do this? Any time you have a 'while' loop that counts upward, have a look to see if you can do this as a 'for' loop instead. Again, I'm assuming you've been taught about them already; if you haven't, ignore this recommendation, and just know that the way you're looping is wordy and unnecessary :) Have fun learning Python. It's a great language, and you seem to have most of it sorted out. If you have more trouble, don't hesitate to come back to this list, or to the python-tutor list if you think your problems are more at the tutorial level than general. All the best! ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: How to properly override the default factory of defaultdict?
On Mon, Feb 15, 2016 at 11:17 AM, Herman wrote: > I want to pass in the key to the default_factory of defaultdict and I found > that defaultdict somehow can intercept my call to dict.__getitem__(self, > key), so my class's __getitem__ have to catch a TypeError instead instead > of KeyError. The following class is my code: Save yourself a lot of trouble, and just override __missing__: class DefaultDictWithEnhancedFactory(collections.defaultdict): def __missing__(self, key): return self.default_factory(key) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Mon, 15 Feb 2016 11:08:52 +1100, Ben Finney wrote: > I am unconcerned with whether there is a real filesystem entry of that > name; the goal entails having no filesystem activity for this. I want > a valid unique filesystem path, without touching the filesystem. That's an odd use case. If it's really just one valid filesystem path (your original post said *paths*, plural), then how about __file__? or os.__file__? -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Dan Sommers writes: > On Mon, 15 Feb 2016 11:08:52 +1100, Ben Finney wrote: > > > I am unconcerned with whether there is a real filesystem entry of > > that name; the goal entails having no filesystem activity for this. > > I want a valid unique filesystem path, without touching the > > filesystem. > > That's an odd use case. It's very common to want filesystem paths divorced from accessing a filesystem entry. For example: test paths in a unit test. Filesystem access is orders of magnitude slower than accessing fake files in memory only, it is more complex and prone to irrelevant failures. So in such a test case filesystem access should be avoided as unnecessary. > If it's really just one valid filesystem path (your original post said > *paths*, plural), then how about __file__? or os.__file__? One valid filesystem path each time it's accessed. That is, behaviour equivalent to ‘tempfile.mktemp’. My question is because the standard library clearly has this useful functionality implemented, but simultaneously warns strongly against its use. I'm looking for how to get at that functionality in a non-deprecated way, without re-implementing it myself. -- \ “The most common way people give up their power is by thinking | `\ they don't have any.” —Alice Walker | _o__) | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: What is heating the memory here? hashlib?
Às 07:04 de 14-02-2016, Paulo da Silva escreveu: > I was unable to reproduce the situation using a simple program just > walking through all files>4K, with or without the seek, and computing > their shasums. > Only some fluctuations of about 500MB in memory consumption. Today I gave another try to the program using 40MB bfsz on the same circumstances except for a previous reboot and, surprisingly, it worked pretty fine. The fluctuations in memory were of the same magnitude of those of the simple program. No swaps at all! Some history ... The 1st. time the problem occurred, I found an issue that I thought could cause that behavior. An equivalent statement for h=hashlib.sha256() was out of the files loop. I put it in the arguments parser because the user could choose the algorithm to use. And instead of testing the option for each file I put it there. Apart from the memory leakage hashlib seemed to work fine. After the "digest" I started feeding it with the contents of another file. 1. Is it possible that the memory exhaustion caused some sort of problem that left the system in a way to cause gc malfunction on the next runs? 2. The filesystem is btrfs. So, is it possible some "fight" among btrfs, gc and my program cause inability to gc free memory in time? This seems unlikely because I was only reading and the filesystem is mounted with noatime. However I don't know if btrfs takes some organization work during the readings. Anyway, I tried at least 3 times the failed tests one of which updating hashlib with 8KB chunks and another with 1MB bfsz. This last one ran until the end but used ~5GB swap. 3. There is another small change I made since then. Some (few) times hashlib was fed with empty data (zero length). That was fixed. So far I tried the program twice and it ran perfectly. When I need to run it in future, out of this confusion, and if the same problem occurs, I'll try to see things more carefully. Once more thank you all. Paulo -- https://mail.python.org/mailman/listinfo/python-list
Re: How to properly override the default factory of defaultdict?
On Monday 15 February 2016 11:17, Herman wrote: > I want to pass in the key to the default_factory of defaultdict Just use a regular dict and define __missing__: class MyDefaultDict(dict): def __missing__(self, key): return "We gotcha key %r right here!" % key If you want a per-instance __missing__, do something like this: class MyDefaultDict(dict): def __init__(self, factory): self._factory = factory def __missing__(self, key): return self._factory(self, key) d = MyDefaultDict(lambda self, key: ...) -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Monday 15 February 2016 11:08, Ben Finney wrote: > I am unconcerned with whether there is a real filesystem entry of that > name; the goal entails having no filesystem activity for this. I want a > valid unique filesystem path, without touching the filesystem. Your phrasing is ambiguous. If you are unconcerned whether or not a file of that name exists, then just pick a name and use that: unique_path = /tmp/foo is guaranteed to be valid on POSIX systems and unique, and it may or may not exist. If you actually do care that /tmp/foo *doesn't* exist, then you have a problem: whatever name you pick *now* may no longer "not exist" a millisecond later. In general there's no way to create a valid pathname which doesn't exist *now* and is guaranteed to continue to not exist unless you touch the file system. But if you explain in more detail why you want this filename, perhaps we can come up with some ideas that will help. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Steven D'Aprano writes: > On Monday 15 February 2016 11:08, Ben Finney wrote: > > > I am unconcerned with whether there is a real filesystem entry of > > that name; the goal entails having no filesystem activity for this. > > I want a valid unique filesystem path, without touching the > > filesystem. > > Your phrasing is ambiguous. The existing behaviour of ‘tempfile.mktemp’ – actually of its internal class ‘tempfile._RandomNameSequence’ – is to generate unpredictable, unique, valid filesystem paths that are different each time. That's the behaviour I want, in a public API that exposes what ‘tempfile’ already has implemented, documented in a way that doesn't create a scare about security. > But if you explain in more detail why you want this filename, perhaps > we can come up with some ideas that will help. The behaviour is already implemented in the standard library. What I'm looking for is a way to use it (not re-implement it) that is public API and isn't scolded by the library documentation. -- \ “Try adding “as long as you don't breach the terms of service – | `\ according to our sole judgement” to the end of any cloud | _o__) computing pitch.” —Simon Phipps, 2010-12-11 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Good evening/morning Ben, >> > I am unconcerned with whether there is a real filesystem entry of >> > that name; the goal entails having no filesystem activity for this. >> > I want a valid unique filesystem path, without touching the >> > filesystem. >> >> Your phrasing is ambiguous. > >The existing behaviour of ‘tempfile.mktemp’ – actually of its >internal class ‘tempfile._RandomNameSequence’ – is to generate >unpredictable, unique, valid filesystem paths that are different >each time. > >That's the behaviour I want, in a public API that exposes what >‘tempfile’ already has implemented, documented in a way that >doesn't create a scare about security. If your code is not actually touching the filesystem, then it will not be affected by the race condition identified in the tempfile.mktemp() warning anyway. So, I'm unsure of your worry. >> But if you explain in more detail why you want this filename, perhaps >> we can come up with some ideas that will help. > >The behaviour is already implemented in the standard library. What >I'm looking for is a way to use it (not re-implement it) that is >public API and isn't scolded by the library documentation. I might also suggest the (bound) method _create_tmp() on class mailbox.Maildir, which achieves roughly the same goals, but for a permanent file. Of course, that particular method also touches the filesystem. The Maildir naming approach is based on the assumptions* that time is monotonically increasing, that system nodes never share the same name and that you don't need more than 1 uniquely named file per directory per millisecond. If so, then you can use the 9 or 10 lines of that method. Good luck, -Martin * I was tempted to joke about these two guarantees, but I think that undermines my basic message. To wit, you can probably rely on this naming technique about as much as you can rely on your system clock. I'll assume that you aren't naming all of your nodes 'franklin.p.gundersnip'. -- Martin A. Brown http://linux-ip.net/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Monday 15 February 2016 12:19, Ben Finney wrote: > One valid filesystem path each time it's accessed. That is, behaviour > equivalent to ‘tempfile.mktemp’. > > My question is because the standard library clearly has this useful > functionality implemented, but simultaneously warns strongly against its > use. If you can absolutely guarantee that this string will never actually be used on a real filesystem, then go right ahead and use it. There's nothing wrong with (for instance) calling mktemp to generate *strings* that merely *look* like pathnames. If you want to guarantee that these faux pathnames can't leak out of your test suite and touch the file system, prepend an ASCII NUL to them. That will make it an illegal path on all file systems that I'm aware of. > I'm looking for how to get at that functionality in a non-deprecated > way, without re-implementing it myself. You probably can't, not if you want to future-proof your code against the day when tempfile.mktemp is removed. But you can simply fork that module, delete all the irrelevant bits, and make the mktemp function a private utility in your test suite. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Steven D'Aprano writes: > If you can absolutely guarantee that this string will never actually > be used on a real filesystem, then go right ahead and use it. I'm giving advice in examples in documentation. It's not enough to have some private usage that I know is good, I am looking for a standard API that when the reader looks it up will not be laden with big scary warnings. Currently I can write about the public API ‘tempfile.mktemp’ in documentation, but the conscientious reader will be correct to have concerns when the examples I give are sternly deprecated in the standard library documentation. Or I can write about the private API ‘tempfile._RandomNameSequence’ in the documentation, and the conscientious reader will be correct to have concerns about use of an undocumented private-use API. I'm looking for a way to give examples that use that standard library functionality, with an API that is both public and not discouraged. > > I'm looking for how to get at that functionality in a non-deprecated > > way, without re-implementing it myself. > > You probably can't, not if you want to future-proof your code against > the day when tempfile.mktemp is removed. That's disappointing. It is already implemented and well-tested, it is useful as is. Forking and duplicating it is poor practice if it can simply be used in a standard place. I have reported https://bugs.python.org/issue26362> for this request. -- \ “Nothing worth saying is inoffensive to everyone. Nothing worth | `\saying will fail to make you enemies. And nothing worth saying | _o__)will not produce a confrontation.” —Johann Hari, 2011 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
fetchall is taking much longer time while getting data from Sybase module in Python
Hi All, I am retrieving data from Sybase database using Sybase module of Python. My query is not taking time however fecthall is taking longer time. Below is the test script : def run_query(db,query): ## Run query and resturn record result t1 = datetime.now() cursorObj = db.cursor() t2 = datetime.now() cursorObj.execute(query) t3 = datetime.now() import pdb pdb.set_trace() rowset = cursorObj.fetchall() t4 = datetime.now() cursorObj.close() print "Time taken to make cursor --%s"%(t2-t1) print "Time taken to execute query --%s"%(t3-t2) print "Time taken for fetchall--%s"%(t4-t3) return rowset Output: Time taken to make cursor --0:00:00.37 Time taken to execute query --0:00:00.379443 Time taken for fetchall--0:00:14.739064 -- https://mail.python.org/mailman/listinfo/python-list
asyncio - run coroutine in the background
Hi all Using asyncio, there are times when I want to execute a coroutine which is time-consuming. I do not need the result immediately, and I do not want to block the current task, so I want to run it in the background. run_in_executor() can run an arbitrary function in the background, but a coroutine needs an event loop. After some experimenting I came up with this - class BackgroundTask: async def run(self, coro, args, callback=None): loop = asyncio.get_event_loop() loop.run_in_executor(None, self.task_runner, coro, args, callback) def task_runner(self, coro, args, callback): loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) fut = asyncio.ensure_future(coro(*args)) if callback is not None: fut.add_done_callback(callback) loop.run_until_complete(fut) loop.close() Usage - bg_task = BackgroundTask() args = (arg1, arg2 ...) callback = my_callback_function await bg_task.run(coro, args, callback) Although it 'awaits' bk_task.run(), it returns immediately, as it is simply waiting for run_in_executor() to be launched. Hope this is of some interest. Frank Millman -- https://mail.python.org/mailman/listinfo/python-list
Re: asyncio - run coroutine in the background
"Frank Millman" : > Using asyncio, there are times when I want to execute a coroutine which > is time-consuming. I do not need the result immediately, and I do not > want to block the current task, so I want to run it in the background. You can't run code "in the background" using asyncio. Coroutines perform cooperative multitasking in a single thread on a single CPU. Parallel processing requires the use of threads or, often preferably, processes. To put it in another way, never run time-consuming code in asyncio. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: asyncio - run coroutine in the background
"Marko Rauhamaa" wrote in message news:8737sumpjl@elektro.pacujo.net... "Frank Millman" : > Using asyncio, there are times when I want to execute a coroutine which > is time-consuming. I do not need the result immediately, and I do not > want to block the current task, so I want to run it in the background. You can't run code "in the background" using asyncio. Coroutines perform cooperative multitasking in a single thread on a single CPU. Parallel processing requires the use of threads or, often preferably, processes. To put it in another way, never run time-consuming code in asyncio. No arguments there. I started with a task that ran quickly, but as I added stuff it started to slow down. The execution of the task involves calling some existing functions, which are themselves coroutines. As you have noted elsewhere, once you turn one function into a coroutine, all calls higher up the chain have to be coroutines as well. The benefit of my class is that it enables me to take the coroutine and run it in another thread, without having to re-engineer the whole thing. Hope this makes sense. Frank -- https://mail.python.org/mailman/listinfo/python-list
Re: asyncio - run coroutine in the background
"Frank Millman" : > The benefit of my class is that it enables me to take the coroutine > and run it in another thread, without having to re-engineer the whole > thing. > > Hope this makes sense. Sure. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: What is heating the memory here? hashlib?
Às 09:49 de 14-02-2016, INADA Naoki escreveu: > tracemalloc module may help you to investigate leaks. > 2016/02/14 午後4:05 "Paulo da Silva" : > Thanks. I didn't know it! Paulo -- https://mail.python.org/mailman/listinfo/python-list
Re: asyncio - run coroutine in the background
"Frank Millman" writes: > The benefit of my class is that it enables me to take the coroutine > and run it in another thread, without having to re-engineer the whole > thing. Threads in Python don't get you parallelism either, of course. I haven't used async/await yet and it's looking painful. I've been wanting to read this: http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5 but I start to think it isn't all that great an approach to concurrency. -- https://mail.python.org/mailman/listinfo/python-list
