ANN: psutil 4.2.0 with Windows service support is out
Full blog post: http://grodola.blogspot.com/2016/05/psutil-420-windows-services-and-python.html -- Giampaolo - http://grodola.blogspot.com -- https://mail.python.org/mailman/listinfo/python-list
ANN: psutil 4.2.0 with Windows service support is out
Full blog post: http://grodola.blogspot.com/2016/05/psutil-420-windows-services-and-python.html -- Giampaolo - http://grodola.blogspot.com -- https://mail.python.org/mailman/listinfo/python-list
How to get a directory list sorted by date?
I have a little Python program I wrote myself which copies images from a camera (well, any mounted directory) to my picture archive. The picture archive is simply a directory hierarchy of dates with years at the top, then months, then days. My Python program simply extracts the date from the image (put there by the camera) and copies the image to the appropriate place in the picture archive. There is one small bit of speeding up done by my program, it checks if the file is already in the archive and doesn't copy if it's already there. I don't generally clear pictures off the camera memory cards so it's often the case that most of the pcitures are already in the archive and all I actually want to do is copy the last couple of weeks of new pictures. As camera memory card sizes get bigger (and images get more detailed) this is beginning to take rather a long time. So, to the question, how can I get a list of the files on the memory card (i.e. files in a directory) in date/time order, latest first. I can then copy them until the first one I find which is already in the archive. The file date may not *exactly* match the date/time in the image file but it will produce the right order which is what I need. What I want is a list in the order produced by:- ls --sort=time I suppose I could use a system call but that seems a little inelegant. -- Chris Green · -- https://mail.python.org/mailman/listinfo/python-list
Re: How to get a directory list sorted by date?
c...@isbd.net wrote: > I have a little Python program I wrote myself which copies images from > a camera (well, any mounted directory) to my picture archive. The > picture archive is simply a directory hierarchy of dates with years at > the top, then months, then days. > > My Python program simply extracts the date from the image (put there > by the camera) and copies the image to the appropriate place in the > picture archive. > > There is one small bit of speeding up done by my program, it checks if > the file is already in the archive and doesn't copy if it's already > there. I don't generally clear pictures off the camera memory cards > so it's often the case that most of the pcitures are already in the > archive and all I actually want to do is copy the last couple of weeks > of new pictures. > > As camera memory card sizes get bigger (and images get more detailed) > this is beginning to take rather a long time. > > So, to the question, how can I get a list of the files on the memory > card (i.e. files in a directory) in date/time order, latest first. I > can then copy them until the first one I find which is already in the > archive. The file date may not *exactly* match the date/time in the > image file but it will produce the right order which is what I need. > > What I want is a list in the order produced by:- > ls --sort=time > > I suppose I could use a system call but that seems a little inelegant. Personally I'd be worried that something goes wrong with this approach (one changed file time leading you to discard many pictures), but to answer the question in the subject: you can build the path from name and folder, then find the time with os.path.getmtime(), then sort: def sorted_dir(folder): def getmtime(name): path = os.path.join(folder, name) return os.path.getmtime(path) return sorted(os.listdir(folder), key=getmtime, reverse=True) The same idea will work with pathlib and os.scandir(): def _getmtime(entry): return entry.stat().st_mtime def sd_sorted_dir(folder): return sorted(os.scandir(folder), key=_getmtime, reverse=True) def pl_sorted_dir(folder): """folder: a pathlib.Path() object""" return sorted(folder.iterdir(), key=_getmtime, reverse=True) -- https://mail.python.org/mailman/listinfo/python-list
Re: How to get a directory list sorted by date?
On Sunday 15 May 2016 18:47, c...@isbd.net wrote: > What I want is a list in the order produced by:- > ls --sort=time I'm not sure if this is the best way to do it, but what I would do is sort the directory list by the file's time metadata, which you can access using os.stat. To sort the file names, I use a key function which takes the filename, joins it to the parent directory, grabs its stat data, and extracts the time field. This is done only once per file name. import os import functools def by_date(where, fname): return os.stat(os.path.join(where, fname)).st_mtime location = '/tmp/' files = sorted(os.listdir(location), key=functools.partial(by_date, location)) The mysterious call to functools.partial creates a new function which is exactly the same as by_date except the "where" argument is already filled in. If functools.partial scares you, you can replace it with: key=lambda fname: by_date(location, fname) and if lambda scares you, you can create a top-level function: def keyfunc(fname): return by_date(location, fname) Also, remember that most operating systems provide (at least) three different times. I'm not sure which one you want, but if I had to guess, I would probably guess mtime. If I remember correctly: atime is usually the last access time; mtime is usually the last modification time; ctime is the creation time, but only on OS-X; ctime on Linux and Windows is, um, something else... To use one of the other two, change "st_mtime" to "st_ctime" or "st_atime". -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Design: Idiom for classes and methods that are customizable by the user?
Hi Marko, Am 13.05.2016 um 07:53 schrieb Marko Rauhamaa: Dirk Bächle : For example, why do you need a key? Couldn't you simply pass the task master class as an argument? The idea behind this is, to be able to select classes by giving a parameter on the command-line. So at some point a translation from a given "key" to its actual class has to happen, I guess. I see. So are the task masters interchangeable? I would have imagined they would modify the default operational semantics. If so, the task manager should be fixed programmatically in the SConstruct file. yes, I'd like to be able to replace Taskmasters as well. For example, switching from the "default" file-oriented taskmaster, to one that also supports pseudo targets (always run) like in "make" or "aqualid". After all, we're mainly a "build framework"...so supporting different approaches to a given "build process" makes sense to me. Apart from that, it's basically just an example of what I'm trying to achieve. To which classes in our architecture the idiom is actually applied, is left as a decision that is made sometime later. Dirk -- https://mail.python.org/mailman/listinfo/python-list
Re: Design: Idiom for classes and methods that are customizable by the user?
Hi Gregory, thanks a lot for your answer and comments. Am 13.05.2016 um 08:35 schrieb Gregory Ewing: Dirk Bächle wrote: [...] I'd even suggest that *all* of the build logic should be in the Nodes, and the Taskmaster class shouldn't exist at all. The top level logic should just tell the final Nodes to bring themselves up to date, and they recursively do likewise for their dependent nodes. This is pretty much what we have now, and the starting point for the redesign. Our current Node class already has too much knowledge about the build logic, and users frequently complain that it's too hard to derive from our base "Node" to build their own upon. Our aim is to make the Node class in particular "slimmer", instead of "fatter". ;) I'm currently following the "Factory" pattern (more or less) as I know it from C++ and similar languages. This statement sets off alarm bells for me. If you're using some design pattern in Python just because you learned to do it that way in C++/Java/whatever, you're probably making it more complicated than it needs to be. Yes, I know what you mean. I used the term "factory" for the idiom of addressing a function or class by a key (=string), and the "more or less" was supposed to clarify that I'm not planning to implement a dedicated "TaskmasterFactory" as class...as you'd find in a design book when looking up the "Factory" pattern. It may well be that some of my wordings and expressions are a bit misleading...sorry, I'm not a native speaker. Best regards, Dirk -- https://mail.python.org/mailman/listinfo/python-list
Re: Design: Idiom for classes and methods that are customizable by the user?
Hello Dieter, Am 13.05.2016 um 09:21 schrieb dieter: Dirk Bächle writes: ... My questions - Is this a good approach, that I could use for other parts of the architecture as well, e.g. the Node class mentioned above? You might look at the "adpater" pattern. It is heavily used in Zope - and there looks something like: thanks a lot for bringing up Zope, I'll definitely have a look at it. Best regards, Dirk -- https://mail.python.org/mailman/listinfo/python-list
Re: How to get a directory list sorted by date?
On 2016-05-15 20:48, Steven D'Aprano wrote: > Also, remember that most operating systems provide (at least) three > different times. I'm not sure which one you want, but if I had to > guess, I would probably guess mtime. If I remember correctly: > > atime is usually the last access time; > mtime is usually the last modification time; > ctime is the creation time, but only on OS-X; > ctime on Linux and Windows is, um, something else... > > To use one of the other two, change "st_mtime" to "st_ctime" or > "st_atime". As an added wrinkle, some of us turn off "atime" in our /etc/ftab because it tends to slow things down with little benefit in return. So Steven was right to recommend "mtime" as it's usually what people want. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: How to get a directory list sorted by date?
On 2016-05-15 11:46, Peter Otten wrote: > def sorted_dir(folder): > def getmtime(name): > path = os.path.join(folder, name) > return os.path.getmtime(path) > > return sorted(os.listdir(folder), key=getmtime, reverse=True) > > The same idea will work with pathlib and os.scandir(): > > def _getmtime(entry): > return entry.stat().st_mtime > > def sd_sorted_dir(folder): > return sorted(os.scandir(folder), key=_getmtime, reverse=True) unless sorted() returns a lazy sorter, you lose most of the advantages of scandir() being lazy since you have to read the entire directory list into memory to sort it. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: How to get a directory list sorted by date?
On Sun, May 15, 2016 at 9:15 PM, Tim Chase wrote: > On 2016-05-15 11:46, Peter Otten wrote: >> def sorted_dir(folder): >> def getmtime(name): >> path = os.path.join(folder, name) >> return os.path.getmtime(path) >> >> return sorted(os.listdir(folder), key=getmtime, reverse=True) >> >> The same idea will work with pathlib and os.scandir(): >> >> def _getmtime(entry): >> return entry.stat().st_mtime >> >> def sd_sorted_dir(folder): >> return sorted(os.scandir(folder), key=_getmtime, reverse=True) > > unless sorted() returns a lazy sorter, you lose most of the advantages > of scandir() being lazy since you have to read the entire directory > list into memory to sort it. I'm not sure a lazy sorter exists. You can't know which is the oldest/newest file in a directory without knowing all their mtimes. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: How to get a directory list sorted by date?
Tim Chase wrote: > On 2016-05-15 11:46, Peter Otten wrote: >> def sorted_dir(folder): >> def getmtime(name): >> path = os.path.join(folder, name) >> return os.path.getmtime(path) >> >> return sorted(os.listdir(folder), key=getmtime, reverse=True) >> >> The same idea will work with pathlib and os.scandir(): >> >> def _getmtime(entry): >> return entry.stat().st_mtime >> >> def sd_sorted_dir(folder): >> return sorted(os.scandir(folder), key=_getmtime, reverse=True) > > unless sorted() returns a lazy sorter, you lose most of the advantages > of scandir() being lazy since you have to read the entire directory > list into memory to sort it. I mentioned it because it avoids the duplicate stat call of the older approach which is rather inelegant. Taking another look at the OP's problem I think sorting could be entirely avoided. Instead have the memory card remember when its data was last archived. Pseudo-code: try: cut_off = load_cut_off(memory_card) pics = (pic for pic in memory_card if gettime(pic) > cut_off) except: pics = memory_card for pic in pics: copy_to_archive(pic) save_cut_off(now(), memory_card) As long as computer and camera time are in sync... -- https://mail.python.org/mailman/listinfo/python-list
Developers using Python QT framework ??
Hi folks, Are there people using QT cross-platform framework on the list? I appreciate for your advices and references regarding this. I am a novice python programmer who started using this framework. Thanks a lot in advance! Tomo -- https://mail.python.org/mailman/listinfo/python-list
Re: How to get a directory list sorted by date?
Le 15/05/2016 10:47, c...@isbd.net a écrit : I have a little Python program I wrote myself which copies images from a camera (well, any mounted directory) to my picture archive. The picture archive is simply a directory hierarchy of dates with years at the top, then months, then days. My Python program simply extracts the date from the image (put there by the camera) and copies the image to the appropriate place in the picture archive. There is one small bit of speeding up done by my program, it checks if the file is already in the archive and doesn't copy if it's already there. I don't generally clear pictures off the camera memory cards so it's often the case that most of the pcitures are already in the archive and all I actually want to do is copy the last couple of weeks of new pictures. As camera memory card sizes get bigger (and images get more detailed) this is beginning to take rather a long time. So, to the question, how can I get a list of the files on the memory card (i.e. files in a directory) in date/time order, latest first. I can then copy them until the first one I find which is already in the archive. The file date may not *exactly* match the date/time in the image file but it will produce the right order which is what I need. What I want is a list in the order produced by:- ls --sort=time I suppose I could use a system call but that seems a little inelegant. Hi, Seeing all the answers, I think there's a misunderstood here. The date used to sort the picture is not the file date provided by os.stat but the shooting date written into the metadata of each picture. Can the OP confirm that ? So, if the name of the picture are unchanged why not just compare the file name ? Vincent -- https://mail.python.org/mailman/listinfo/python-list
Re: Developers using Python QT framework ??
Le 15/05/2016 13:19, tommy yama a écrit : Hi folks, Are there people using QT cross-platform framework on the list? I appreciate for your advices and references regarding this. I am a novice python programmer who started using this framework. Thanks a lot in advance! Tomo Yes. Note, it is also a mailing list PyQt: https://www.riverbankcomputing.com/mailman/listinfo/pyqt Vincent -- https://mail.python.org/mailman/listinfo/python-list
Re: How to get a directory list sorted by date?
On 2016-05-15, Tim Chase wrote: > On 2016-05-15 11:46, Peter Otten wrote: >> def sorted_dir(folder): >> def getmtime(name): >> path = os.path.join(folder, name) >> return os.path.getmtime(path) >> >> return sorted(os.listdir(folder), key=getmtime, reverse=True) >> >> The same idea will work with pathlib and os.scandir(): >> >> def _getmtime(entry): >> return entry.stat().st_mtime >> >> def sd_sorted_dir(folder): >> return sorted(os.scandir(folder), key=_getmtime, reverse=True) > > unless sorted() returns a lazy sorter, What's a lazy sorter? -- Grant -- https://mail.python.org/mailman/listinfo/python-list
Re: Developers using Python QT framework ??
Hey there, Many thanks, Vincent, On Sun, May 15, 2016 at 11:15 PM, Vincent Vande Vyvre < vincent.vande.vy...@telenet.be> wrote: > Le 15/05/2016 13:19, tommy yama a écrit : > >> Hi folks, >> >> Are there people using QT cross-platform framework on the list? >> I appreciate for your advices and references regarding this. >> >> I am a novice python programmer who started using this framework. >> >> >> Thanks a lot in advance! >> >> Tomo >> > Yes. > > Note, it is also a mailing list PyQt: > https://www.riverbankcomputing.com/mailman/listinfo/pyqt > > Vincent > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: How to get a directory list sorted by date?
On Sun, May 15, 2016 at 10:36 AM, Grant Edwards wrote: > On 2016-05-15, Tim Chase wrote: >> On 2016-05-15 11:46, Peter Otten wrote: >>> def sorted_dir(folder): >>> def getmtime(name): >>> path = os.path.join(folder, name) >>> return os.path.getmtime(path) >>> >>> return sorted(os.listdir(folder), key=getmtime, reverse=True) >>> >>> The same idea will work with pathlib and os.scandir(): >>> >>> def _getmtime(entry): >>> return entry.stat().st_mtime >>> >>> def sd_sorted_dir(folder): >>> return sorted(os.scandir(folder), key=_getmtime, reverse=True) >> >> unless sorted() returns a lazy sorter, > > What's a lazy sorter? Some postal workers? just kidding > > -- > Grant > > -- > https://mail.python.org/mailman/listinfo/python-list -- Joel Goldstick http://joelgoldstick.com/blog http://cc-baseballstats.info/stats/birthdays -- https://mail.python.org/mailman/listinfo/python-list
intermittent ValueErrors from subprocess
Hi,
I get intermittent ValueErrors[1] from subprocess when I check if an IP
is assigned to loopback interface by running:
/sbin/ip address show dev lo to 10.52.12.2/32
I use subprocess.check_output like this:
cmd = [
'/sbin/ip',
'address',
'show',
'dev',
"{}".format(self.config['interface']),
'to',
"{}".format(self.config['ip_prefix']),
]
try:
out = subprocess.check_output(
cmd,
universal_newlines=True,
timeout=1)
except subprocess.CalledProcessError as error:
return True
except subprocess.TimeoutExpired:
return True
else:
if self.config['ip_prefix'] in out:
return True
else:
return False
and I get the following exception:
ValueError: Invalid file object: <_io.TextIOWrapper name=11
encoding='UTF-8'>
As a consequence of the raised exception thread dies and I now catch
ValueError in the same try block in order to avoid the crash.
It has happened ~5 times and this code is executed from multiple
threads and every ~10secs on several(~40) systems for more than
18months. So, it is quite impossible for me to reproduce it.
It could be that the system returns corrupted data or python fails for
some unknown reason to parse the output.
The program uses Python 3.4.4 and runs on Debian stable and CentOS 6/7.
The full code path in question can be found here:
https://github.com/unixsurfer/anycast_healthchecker/blob/master/anycast_healthchecker/servicecheck.py#L90
I did a search on bugs.python.org about it but I didn't find anything.
Has anyone seen this behavior before?
Cheers,
Pavlos
[1] https://gist.github.com/unixsurfer/67db620d87f667423f6f6e3a04e0bff5
signature.asc
Description: OpenPGP digital signature
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to get a directory list sorted by date?
On Sun, May 15, 2016, 10:37 AM Grant Edwards wrote: > On 2016-05-15, Tim Chase wrote: > > On 2016-05-15 11:46, Peter Otten wrote: > >> def sorted_dir(folder): > >> def getmtime(name): > >> path = os.path.join(folder, name) > >> return os.path.getmtime(path) > >> > >> return sorted(os.listdir(folder), key=getmtime, reverse=True) > >> > >> The same idea will work with pathlib and os.scandir(): > >> > >> def _getmtime(entry): > >> return entry.stat().st_mtime > >> > >> def sd_sorted_dir(folder): > >> return sorted(os.scandir(folder), key=_getmtime, reverse=True) > > > > unless sorted() returns a lazy sorter, > > What's a lazy sorter? > One that doesn't calculate the next item in the sequence until you ask for it. It's impossible unless you don't mind an approximation rather than correct sort. > -- https://mail.python.org/mailman/listinfo/python-list
Re: How to get a directory list sorted by date?
On 2016-05-15, Michael Selik wrote: > On Sun, May 15, 2016, 10:37 AM Grant Edwards > wrote: >> On 2016-05-15, Tim Chase wrote: >>> >>> unless sorted() returns a lazy sorter, >> >> What's a lazy sorter? > > One that doesn't calculate the next item in the sequence until you > ask for it. It's impossible Why? As long as the function has access to the entire sequence, it should be trivial. Just find the min (or max) item in the sequence, remove it, then return it. It's horribly inefficient, but... > unless you don't mind an approximation rather than correct sort. I have a feeling that I've missed the joke somewhere. -- Grant -- https://mail.python.org/mailman/listinfo/python-list
Pandas GroupBy does not behave consistently
Hello, Michael,
Pandas GroupBy does not behave consistently.
Last time, when we had conversation, I used grouby. It works well.
Now, I thought to re-write the program, so that I can end up with a clean
script.
But, the problem is that a lot of columns are missing after groupby application.
Any idea?
Regards.
David
On Saturday, 14 May 2016, 17:00, Michael Selik
wrote:
This StackOverflow question was the first search result when I Googled for
"Python why is there a little
u"http://stackoverflow.com/questions/11279331/what-does-the-u-symbol-mean-in-front-of-string-values
On Sat, May 14, 2016, 11:40 AM David Shi wrote:
Hello, Michael,
Why there is a little u ? u'ID',?
Why can be done to it? How to handle such objects?
Can it be turn into list easily?
Regards.
David
On Saturday, 14 May 2016, 15:34, Michael Selik
wrote:
You might also be interested in "Python for Data Analysis" for a thorough
discussion of Pandas.http://shop.oreilly.com/product/0636920023784.do
On Sat, May 14, 2016 at 10:29 AM Michael Selik wrote:
David, it sounds like you'll need a thorough introduction to the basics of
Python.Check out the tutorial: https://docs.python.org/3/tutorial/
On Sat, May 14, 2016 at 6:19 AM David Shi wrote:
Hello, Michael,
I discovered that the problem is "two columns of data are put together" and
"are recognised as one column".
This is very strange. I would like to understand the subject well.
And, how many ways are there to investigate into the nature of objects
dynamically?
Some object types only get shown as an object. Are there anything to be typed
in Python, to reveal objects.
Regards.
David
On Saturday, 14 May 2016, 4:30, Michael Selik
wrote:
What were you hoping to get from ``df[0]``?When you say it "yields nothing" do
you mean it raised an error? What was the error message?
Have you tried a Google search for "pandas set
index"?http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.set_index.html
On Fri, May 13, 2016 at 11:18 PM David Shi wrote:
Hello, Michael,
I tried to discover the problem.
df[0] yields nothingdf[1] yields nothingdf[2] yields nothing
However, df[3] gives the following:sid
-9223372036854775808 NaN
1 133738.70
4 295256.11
5 137733.09
6 409413.58
8 269600.97
9 12852.94
Can we split this back to normal? or turn it into a dictionary, so that I can
put values back properly.
I like to use sid as index, some way.
Regards.
David
On Friday, 13 May 2016, 22:58, Michael Selik
wrote:
What have code you tried? What error message are you receiving?
On Fri, May 13, 2016, 5:54 PM David Shi wrote:
Hello, Michael,
How to convert a float type column into an integer or label or string type?
On Friday, 13 May 2016, 22:02, Michael Selik
wrote:
To clarify that you're specifying the index as a label, use df.iloc
>>> df = pd.DataFrame({'X': range(4)}, index=list('abcd')) >>> df
X a 0 b 1 c 2 d 3 >>> df.loc['a'] X 0 Name: a,
dtype: int64 >>> df.iloc[0] X 0 Name: a, dtype: int64
On Fri, May 13, 2016 at 4:54 PM David Shi wrote:
Dear Michael,
To avoid complication, I only groupby using one column.
It is OK now. But, how to refer to new row index? How do I use floating index?
Float64Index([ 1.0, 4.0, 5.0, 6.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 16.0,
17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0,
28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0,
39.0, 40.0, 41.0, 42.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0,
51.0, 53.0, 54.0, 55.0, 56.0],
dtype='float64', name=u'StateFIPS')
Regards.
David
On Friday, 13 May 2016, 21:43, Michael Selik
wrote:
Here's an example.
>>> import pandas as pd >>> df = pd.DataFrame({'group': list('AB') * 2,
'data': range(4)}, index=list('wxyz')) >>> df data group w 0
A x 1 B y 2 A z 3 B >>> df =
df.reset_index() >>> df index data group 0 w 0 A 1
x 1 B 2 y 2 A 3 z 3 B >>>
df.groupby('group').max() index data group A y 2
B z 3
If that doesn't help, you'll need to explain what you're trying to accomplish
in detail -- what variables you started with, what transformations you want to
do, and what variables you hope to have when finished.
On Fri, May 13, 2016 at 4:36 PM David Shi wrote:
Hello, Michael,
I changed groupby with one column.
The index is different.
Index([ u'AL',u'AR',u'AZ',u'CA',u'CO',u'CT',u'DC',
u'DE',u'FL',u'GA',u'IA',u'ID',u'IL',u'IN',
u'KS',u'KY',u'LA',u'MA',u'MD',u'ME',u'MI',
u
Re: How to get a directory list sorted by date?
On 2016-05-15 14:36, Grant Edwards wrote: > On 2016-05-15, Tim Chase wrote: > > unless sorted() returns a lazy sorter, > > What's a lazy sorter? A hypothetical algorithm that can spool out a sorted sequence without holding the entire sequence in memory at the same time. Though I typed that initial reply a little quickly, as I hadn't known at the time that scandir()'s results also provide stat() results with reduced overhead (whereas listdir() just returns the filenames, so you have to stat() each one). Thanks, Peter, for highlighting this feature. So in light of my newfound knowledge, I humbly retract my snark and agree that it's better to exploit scandir()'s inclusion of stat() details without additional calls. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: Pandas GroupBy does not behave consistently
On Sun, May 15, 2016 at 7:07 AM David Shi wrote: > Hello, Michael, > > Pandas GroupBy does not behave consistently. > > Last time, when we had conversation, I used grouby. It works well. > > Now, I thought to re-write the program, so that I can end up with a clean > script. > > But, the problem is that a lot of columns are missing after groupby > application. > > Any idea? > I'd guess that the columns lost after a groupby had an inappropriate datatype for the operation you were doing. If you'd like a more thorough response, you'll need to post the code you tried and the results you got. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to get a directory list sorted by date?
Tim Chase wrote: > On 2016-05-15 14:36, Grant Edwards wrote: > > On 2016-05-15, Tim Chase wrote: > > > unless sorted() returns a lazy sorter, > > > > What's a lazy sorter? > > A hypothetical algorithm that can spool out a sorted sequence without > holding the entire sequence in memory at the same time. > > Though I typed that initial reply a little quickly, as I hadn't known > at the time that scandir()'s results also provide stat() results with > reduced overhead (whereas listdir() just returns the filenames, so > you have to stat() each one). Thanks, Peter, for highlighting this > feature. > > So in light of my newfound knowledge, I humbly retract my snark and > agree that it's better to exploit scandir()'s inclusion of stat() > details without additional calls. > Thanks everyone for the ideas etc. -- Chris Green · -- https://mail.python.org/mailman/listinfo/python-list
Re: How to get a directory list sorted by date?
On Mon, May 16, 2016 at 2:00 AM, Grant Edwards wrote: > On 2016-05-15, Michael Selik wrote: >> On Sun, May 15, 2016, 10:37 AM Grant Edwards >> wrote: >>> On 2016-05-15, Tim Chase wrote: unless sorted() returns a lazy sorter, >>> >>> What's a lazy sorter? >> >> One that doesn't calculate the next item in the sequence until you >> ask for it. It's impossible > > Why? As long as the function has access to the entire sequence, it > should be trivial. Just find the min (or max) item in the sequence, > remove it, then return it. It's horribly inefficient, but... > >> unless you don't mind an approximation rather than correct sort. > > I have a feeling that I've missed the joke somewhere. Sure, it's not impossible to implement, but generally the point of lazy generation is that it's more efficient. Going from O(N log N) with O(N) memory to O(N*N) with O(1) memory is not usually what you want! ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: intermittent ValueErrors from subprocess
On Mon, May 16, 2016 at 1:49 AM, Pavlos Parissis
wrote:
> I use subprocess.check_output like this:
>
> cmd = [
> '/sbin/ip',
> 'address',
> 'show',
> 'dev',
> "{}".format(self.config['interface']),
> 'to',
> "{}".format(self.config['ip_prefix']),
> ]
Unrelated to your problem: "{}".format(x) is the same as str(x), and
if x is already a string, it does exactly nothing. Most likely you can
just put your configs straight in there.
> try:
> out = subprocess.check_output(
> cmd,
> universal_newlines=True,
> timeout=1)
> except subprocess.CalledProcessError as error:
> return True
> except subprocess.TimeoutExpired:
> return True
> else:
> if self.config['ip_prefix'] in out:
> return True
> else:
> return False
>
>
> and I get the following exception:
>
> ValueError: Invalid file object: <_io.TextIOWrapper name=11
> encoding='UTF-8'>
Searching the CPython sources for that exception shows one hit:
selectors.py, where it converts a file object to an integer file
descriptor. (You could have helped out by showing us the full
traceback.) Is it possible you were running out of file descriptors,
or in some other way unable to create the pipe?
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: intermittent ValueErrors from subprocess
On 16/05/2016 12:08 πμ, Chris Angelico wrote:
> On Mon, May 16, 2016 at 1:49 AM, Pavlos Parissis
> wrote:
>> I use subprocess.check_output like this:
>>
>> cmd = [
>> '/sbin/ip',
>> 'address',
>> 'show',
>> 'dev',
>> "{}".format(self.config['interface']),
>> 'to',
>> "{}".format(self.config['ip_prefix']),
>> ]
>
> Unrelated to your problem: "{}".format(x) is the same as str(x), and
> if x is already a string, it does exactly nothing. Most likely you can
> just put your configs straight in there.
>
Thanks for the tip, I will change it as you suggested.
>> try:
>> out = subprocess.check_output(
>> cmd,
>> universal_newlines=True,
>> timeout=1)
>> except subprocess.CalledProcessError as error:
>> return True
>> except subprocess.TimeoutExpired:
>> return True
>> else:
>> if self.config['ip_prefix'] in out:
>> return True
>> else:
>> return False
>>
>>
>> and I get the following exception:
>>
>> ValueError: Invalid file object: <_io.TextIOWrapper name=11
>> encoding='UTF-8'>
>
> Searching the CPython sources for that exception shows one hit:
> selectors.py, where it converts a file object to an integer file
> descriptor. (You could have helped out by showing us the full
> traceback.)
I did, https://gist.github.com/unixsurfer/67db620d87f667423f6f6e3a04e0bff5
> Is it possible you were running out of file descriptors,
> or in some other way unable to create the pipe?
I don't think as I see right now only 8 FDs:
sudo ls -1 /proc/22706/fd|wc
8 8 16
Thanks,
Pavlos
signature.asc
Description: OpenPGP digital signature
--
https://mail.python.org/mailman/listinfo/python-list
Re: intermittent ValueErrors from subprocess
On Mon, May 16, 2016 at 8:32 AM, Pavlos Parissis wrote: >> Searching the CPython sources for that exception shows one hit: >> selectors.py, where it converts a file object to an integer file >> descriptor. (You could have helped out by showing us the full >> traceback.) > > I did, https://gist.github.com/unixsurfer/67db620d87f667423f6f6e3a04e0bff5 Ah. I didn't click that link in your original post - didn't know it was the traceback. Better would have been to at least say so; best would have been to include it inline. >> Is it possible you were running out of file descriptors, >> or in some other way unable to create the pipe? > > I don't think as I see right now only 8 FDs: > > sudo ls -1 /proc/22706/fd|wc > 8 8 16 > If you can recreate the problem consistently, it would be worth messing around with slightly lower level APIs - using subprocess.Popen rather than check_output, for instance - and see what you can do without the pipes. Somewhere, something's failing, and it's not easy to see what. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: intermittent ValueErrors from subprocess
On 16/05/2016 12:59 πμ, Chris Angelico wrote: > On Mon, May 16, 2016 at 8:32 AM, Pavlos Parissis > wrote: >>> Searching the CPython sources for that exception shows one hit: >>> selectors.py, where it converts a file object to an integer file >>> descriptor. (You could have helped out by showing us the full >>> traceback.) >> >> I did, https://gist.github.com/unixsurfer/67db620d87f667423f6f6e3a04e0bff5 > > Ah. I didn't click that link in your original post - didn't know it > was the traceback. Better would have been to at least say so; best > would have been to include it inline. > I don't usually include traces as they have long lines and several e-mail clients mess with them in a way that makes the trace unreadable. >>> Is it possible you were running out of file descriptors, >>> or in some other way unable to create the pipe? >> >> I don't think as I see right now only 8 FDs: >> >> sudo ls -1 /proc/22706/fd|wc >> 8 8 16 >> > > If you can recreate the problem consistently, I can't. This is my main problem. This code has been executed ~100K and that exception has occurred only ~5 times. > it would be worth > messing around with slightly lower level APIs - using subprocess.Popen > rather than check_output, for instance - and see what you can do > without the pipes. What do you mean by that? > Somewhere, something's failing, and it's not easy > to see what. > > ChrisA > Thanks once again, Pavlos signature.asc Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: intermittent ValueErrors from subprocess
On Mon, May 16, 2016 at 9:17 AM, Pavlos Parissis wrote: > On 16/05/2016 12:59 πμ, Chris Angelico wrote: >> Ah. I didn't click that link in your original post - didn't know it >> was the traceback. Better would have been to at least say so; best >> would have been to include it inline. >> > > I don't usually include traces as they have long lines and several > e-mail clients mess with them in a way that makes the trace unreadable. If your client can't send them, fix your client. If someone else's client can't read them, that's not your problem. >> If you can recreate the problem consistently, > > I can't. This is my main problem. This code has been executed ~100K and > that exception has occurred only ~5 times. Yeah, I figured. Makes it tough. >> it would be worth >> messing around with slightly lower level APIs - using subprocess.Popen >> rather than check_output, for instance - and see what you can do >> without the pipes. > > What do you mean by that? You're currently using a very high level API that says "run this program, make sure it exits zero, and give me its output". Somewhere inside there, something's going wrong. So the first thing I'd do would be to tease apart the job into smaller and simpler parts; that means, in this case, going to something like subprocess.Popen, with explicit use of pipes and such. You could look at the source code for check_output for a start. But being unable to recreate the problem on demand makes it hard. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
sqlite 3 attribute error __exit__
Hi
I have a file and want to create the sqlite3 db. Using with however i receive
an attribute error and it causes an exit.
The relevant section of the file is:
import sqlite3
conn = sqlite3.connect("trial.db")
with conn, conn.cursor() as cur:
# First, create tables.
cur.execute("drop table if exists meetings, races, horses")
cur.execute("create table meetings (" +
", ".join("%s varchar" % fld for fld in meetattrs)
+ ")")
cur.execute("create table races (" +
", ".join("%s varchar" % fld for fld in raceattrs)
+ ")")
cur.execute("create table horses (" +
", ".join("%s varchar" % fld for fld in horseattrs)
+ ")")
However it runs producing this error for line 30 which is
with conn, conn.cursor() as cur:
(pyXML) [sayth@manjaro pyXML]$ python racemeeting.py data/*xml
Traceback (most recent call last):
File "racemeeting.py", line 31, in
with conn, conn.cursor() as cur:
AttributeError: __exit__
Why would i get this error?
Sayth
--
https://mail.python.org/mailman/listinfo/python-list
Re: sqlite 3 attribute error __exit__
Sayth Renshaw writes: > with conn, conn.cursor() as cur: What are you expecting this ‘with’ statement to do? As you've written it, the statement declares your intent to enter both ‘conn’ and ‘conn.cursor()’ as context managers. https://docs.python.org/3/reference/compound_stmts.html#with> To “enter a context manager” entails calling the ‘__enter__’ method on the context manager object. So that will happen to both ‘conn’ and ‘conn.cursor()’. Are both of those objects context managers? If not, you will get the error that you reported. -- \ “Pinky, are you pondering what I'm pondering?” “I think so, | `\Brain, but if we get Sam Spade, we'll never have any puppies.” | _o__) —_Pinky and The Brain_ | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: sqlite 3 attribute error __exit__
On Monday, 16 May 2016 12:45:26 UTC+10, DFS wrote:
> On 5/15/2016 10:21 PM, Sayth Renshaw wrote:
> > Hi
> >
> > I have a file and want to create the sqlite3 db. Using with however i
> > receive an attribute error and it causes an exit.
> >
> > The relevant section of the file is:
> >
> > import sqlite3
> >
> > conn = sqlite3.connect("trial.db")
> > with conn, conn.cursor() as cur:
> > # First, create tables.
> > cur.execute("drop table if exists meetings, races, horses")
> > cur.execute("create table meetings (" +
> > ", ".join("%s varchar" % fld for fld in meetattrs)
> > + ")")
> > cur.execute("create table races (" +
> > ", ".join("%s varchar" % fld for fld in raceattrs)
> > + ")")
> > cur.execute("create table horses (" +
> > ", ".join("%s varchar" % fld for fld in horseattrs)
> > + ")")
> >
> >
> > However it runs producing this error for line 30 which is
> > with conn, conn.cursor() as cur:
> >
> > (pyXML) [sayth@manjaro pyXML]$ python racemeeting.py data/*xml
> > Traceback (most recent call last):
> > File "racemeeting.py", line 31, in
> > with conn, conn.cursor() as cur:
> > AttributeError: __exit__
> >
> > Why would i get this error?
> >
> > Sayth
>
>
> Answer #66 looks like the ticket:
>
> http://stackoverflow.com/questions/7447284/how-to-troubleshoot-an-attributeerror-exit-in-multiproccesing-in-python
>
>
> Have you tried:
>
> cur = conn.cursor()
cur = conn.cursor() does work thought I should have been able to use with to
handle it for me.
Thanks
Sayth
--
https://mail.python.org/mailman/listinfo/python-list
Re: Trouble porting glob bash behavior with argparse to windows shell
On Wednesday, 4 May 2016 17:57:32 UTC+10, Sayth Renshaw wrote:
> Oops sorry noticed you did in the glob. Sorry squinting at phone.
>
> Sayth
Hi
this seems to be causing me an error in my thinking as well as the program. I
am creating a function GetArgs to take a path and file extension from the
command line.
However I cannot call it effectively. I will clrify this is my function
import argparse
import glob
import os
import sqlite3
def GetArgs(parser):
'''parse XML from command line'''
parser.add_argument("path", nargs="+")
parser.add_argument('-e', '--extension', default='',
help='File extension to filter by.')
args = parser.parse_args()
files = set()
name_pattern = "*" + args.extension
for path in args.path:
files.update(glob.glob(os.path.join(path, name_pattern)))
return files
Then later in program I am attempting to call it an a for statement.
filesToProcess = GetArgs()
for meeting in filesToProcess:
meetdata = [meeting.get(attr) for attr in meetattrs]
cur.execute("insert into meetings values (" +
",".join(["%s"] * len(meetattrs)) + ")", meetdata)
this fails as i would expect, however if I declare a list as the GetArgs()
argument it fails as well.
Where my confusion is that I created the function to take arguments from the
command line, so I don't have that variable to supply until executed.
Have i overbaked the cake?
Sayth
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to use pip to install dtrx?
On Saturday, 14 May 2016 07:39:23 UTC+10, Ehsan Hajiramezanali wrote: > Hi, > > I want to use pip to install dtrx. However, I got the following error. > > ~~~ > $ pip install --allow-external dtrx dtrx > DEPRECATION: --allow-external has been deprecated and will be removed > in the future. Due to changes in the repository protocol, it no longer > has any effect. > Collecting dtrx > Could not find a version that satisfies the requirement dtrx (from > versions: ) > No matching distribution found for dtrx > ~~~ > > Is there any way to solve this problem? > > Thanks in advance. > > Best regards, > Ehsan You can sownload it and run this on it. python setup.py install --prefix=/usr/local Of course it appears to not be maintained in anyway for 9 years. I found pyunpack https://github.com/ponty/pyunpack that page has a link to similar projects to which you may find useful if pyunpack isn't. https://github.com/ponty/pyunpack#similar-projects Sayth -- https://mail.python.org/mailman/listinfo/python-list
Re: sqlite 3 attribute error __exit__
>
> As you've written it, the statement declares your intent to enter both
> ‘conn’ and ‘conn.cursor()’ as context managers.
>
> https://docs.python.org/3/reference/compound_stmts.html#with>
>
> To “enter a context manager” entails calling the ‘__enter__’ method on
> the context manager object. So that will happen to both ‘conn’ and
> ‘conn.cursor()’.
>
> Are both of those objects context managers? If not, you will get the
> error that you reported.
>
I actually am unsure about context managers. I was talking to ChrisA the other
day and using psycog2 this code worked.
conn = psycopg2.connect("")
with conn, conn.cursor() as cur:
# First, create tables.
cur.execute("drop table if exists meetings, races, horses")
cur.execute("create table meetings (" +
", ".join("%s varchar" % fld for fld in meetattrs)
+ ")")
cur.execute("create table races (" +
", ".join("%s varchar" % fld for fld in raceattrs)
+ ")")
cur.execute("create table horses (" +
", ".join("%s varchar" % fld for fld in horseattrs)
+ ")")
I tried to change it to sqlite3 and it fails. I thought it may have been sqlite
specific.
Found this http://initd.org/psycopg/articles/2013/04/07/psycopg-25-released/ I
found using a context managers search and psycopg2 provides this as a feature.
from the site.
Connections and cursors as context managers
A recent DBAPI extension has standardized the use of connections and cursors as
context managers: it is now possible to use an idiom such as:
with psycopg2.connect(DSN) as conn:
with conn.cursor() as curs:
curs.execute(SQL)
with the intuitive behaviour: when the cursor block exits the cursor is closed;
when the connection block exits normally the current transaction is committed,
if it exits with an exception instead the transaction is rolled back, in either
case the connection is ready to be used again (FIXED: the connection is NOT
closed as originally stated).
thanks
Sayth
--
https://mail.python.org/mailman/listinfo/python-list
