[issue13968] Support recursive globs

Marc added the comment: Please review one word documentation change at https://github.com/python/cpython/pull/12918 to clarify that recursive glob ** follows symbolic links to directories. -- nosy: +marc-hb pull_requests: +12844 ___ Python tracker

[issue13968] Support recursive globs

Changes by Serhiy Storchaka : -- resolution: -> fixed status: open -> closed ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue13968] Support recursive globs

STINNER Victor added the comment: > However perhaps we should consider as a bug if a test ran by regrtest doesn't > clean created files or directories => yes, I opened the issue #22390. -- ___ Python tracker

[issue13968] Support recursive globs

Serhiy Storchaka added the comment: Thank you Victor. The test was failed also when run it directly, omitting the test.regrtest module (which run a test inside temporary directory): ./python Lib/test/test_glob.py Now it is fixed. However perhaps we should consider as a bug if a test ran b

[issue13968] Support recursive globs

Roundup Robot added the comment: New changeset 180f5bf7d1b9 by Serhiy Storchaka in branch 'default': Issue #13968: Fixed newly added recursive glob test. http://hg.python.org/cpython/rev/180f5bf7d1b9 -- ___ Python tracker

[issue13968] Support recursive globs

STINNER Victor added the comment: The test failed on a buildbot, I reopen the issue. http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/10607/steps/test/logs/stdio == FAIL: test_selflink (test.test_glob.

[issue13968] Support recursive globs

Serhiy Storchaka added the comment: Thank you for your review Nick. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___

[issue13968] Support recursive globs

Roundup Robot added the comment: New changeset ff4b9d654691 by Serhiy Storchaka in branch 'default': Issue #13968: The glob module now supports recursive search in http://hg.python.org/cpython/rev/ff4b9d654691 -- nosy: +python-dev ___ Python tracker

[issue13968] Support recursive globs

Nick Coghlan added the comment: Looks good to me! -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://ma

[issue13968] Support recursive globs

Changes by Serhiy Storchaka : Added file: http://bugs.python.org/file36555/glob_recursive_6.patch ___ Python tracker ___ ___ Python-bugs-list

[issue13968] Support recursive globs

Serhiy Storchaka added the comment: Thank you Nick. > 1. Is there a reason the helper function is "glob2" rather than either > "_glob2" or else something more self-documenting? Only consistency with other helper functions (glob0, glob1). > 2. "match a files" in the docs and docstrings doesn't

[issue13968] Support recursive globs

Nick Coghlan added the comment: Mostly looks good to me, just two comments: 1. Is there a reason the helper function is "glob2" rather than either "_glob2" or else something more self-documenting? 2. "match a files" in the docs and docstrings doesn't read correctly. "match any files", perhaps

[issue13968] Support recursive globs

Serhiy Storchaka added the comment: Could you make a review Nick? -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscr

[issue13968] Support recursive globs

Changes by Serhiy Storchaka : -- keywords: +needs review ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:

[issue13968] Support recursive globs

Nick Coghlan added the comment: Oops, Python 3.4 has ** support in pathlib, but we missed Serhiy's patch for the glob module itself. We should resolve that discrepancy for 3.5 :) -- versions: +Python 3.5 -Python 3.4 ___ Python tracker

[issue13968] Support recursive globs

Serhiy Storchaka added the comment: In updated patch fixed warning/errors when ran with -b or -bb options. -- Added file: http://bugs.python.org/file32669/glob_recursive_4.patch ___ Python tracker _

[issue13968] Support recursive globs

Serhiy Storchaka added the comment: In fact glob() is already protected against an endless recursion (in the same way as Bash). The level of recursion is simply limited by the maximum length of the path. So I did not change the implementation, I have just added a test for symlink loop. I also

[issue13968] Support recursive globs

Serhiy Storchaka added the comment: I should add a symlink loop detecting to _rlistdir() as Antoine advised me on IRC. -- stage: patch review -> needs patch ___ Python tracker _

[issue13968] Support recursive globs

Changes by Serhiy Storchaka : Removed file: http://bugs.python.org/file28213/glob_recursive.patch ___ Python tracker ___ ___ Python-bugs-list

[issue13968] Support recursive globs

Changes by Serhiy Storchaka : -- stage: needs patch -> patch review ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue13968] Support recursive globs

Serhiy Storchaka added the comment: Patch updated for current tip. -- Added file: http://bugs.python.org/file28527/glob_recursive_2.patch ___ Python tracker ___ _

[issue13968] Support recursive globs

Changes by Serhiy Storchaka : -- assignee: -> serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue13968] Support recursive globs

Serhiy Storchaka added the comment: Here is a patch which implements recursive globbing which conforms to Bash globbing with "globstar" option. For backward compatibility recursive globbing off by default and works only if new argument "recursive" is true (default is False). I am not sure this

[issue13968] Support recursive globs

Changes by Serhiy Storchaka : -- stage: patch review -> needs patch ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue13968] Support recursive globs

Changes by Antoine Pitrou : -- versions: +Python 3.4 -Python 3.3 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue13968] Support recursive globs

Antoine Pitrou added the comment: I'm looking at the tests and I don't understand why '**/bcd/*' should match 'a/bcd/efg/ha'. Am I missing something? -- ___ Python tracker ___

[issue13968] Support recursive globs

Changes by Chris Rebert : -- nosy: +cvrebert ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python

[issue13968] Support recursive globs

Antoine Pitrou added the comment: > So, anybody for or against this patch? I'd really like to see this > feature make its way in... I think the feature is useful, but someone needs to review the patch. Sorry if it takes some time. -- ___ Python trac

[issue13968] Support recursive globs

Changes by Antoine Pitrou : -- stage: -> patch review ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://m

[issue13968] Support recursive globs

Yuval Greenfield added the comment: So, anybody for or against this patch? I'd really like to see this feature make its way in... -- ___ Python tracker ___

[issue13968] Support recursive globs

Yuval Greenfield added the comment: I added the doublestar functionality to iglob and updated the docs and tests. Also, a few readability renames in that module were a long time coming. I'd love to hear your feedback. -- Added file: http://bugs.python.org/file25360/glob.doublestars.pa

[issue13968] Support recursive globs

Yuval Greenfield added the comment: On Sun, Apr 1, 2012 at 4:42 PM, Serhiy Storchaka wrote: > For "**" globbing see http://ant.apache.org/manual/dirtasks.html#patterns They mention that "mypackage/test/ is interpreted as if it were mypackage/test/**" so that's not really an option. I'm pretty

[issue13968] Support recursive globs

Serhiy Storchaka added the comment: For "**" globbing see http://ant.apache.org/manual/dirtasks.html#patterns . If we extend pattern syntax of templates, why not implement Perl, Tcl or Bash extensions? -- nosy: +storchaka ___ Python tracker

[issue13968] Support recursive globs

Yuval Greenfield added the comment: I don't have a strong opinion on "rglob vs glob" so whichever way the majority here thinks is fine by me. -- ___ Python tracker ___

[issue13968] Support recursive globs

Antoine Pitrou added the comment: > I found this comprehensive description of the '**' convention at > http://www.codeproject.com/Articles/2809/Recursive-patterned-File-Globbing > that can translate directly to unittests. > > I'd like to fix the patch for these specs but should it be in a new

[issue13968] Support recursive globs

Yuval Greenfield added the comment: I found this comprehensive description of the '**' convention at http://www.codeproject.com/Articles/2809/Recursive-patterned-File-Globbing that can translate directly to unittests. I'd like to fix the patch for these specs but should it be in a new rglob

[issue13968] Support recursive globs

Changes by Eli Bendersky : -- nosy: -eli.bendersky ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail

[issue13968] Support recursive globs

Nick Coghlan added the comment: FWIW, I've also come around to the point of view that it's worthwhile to provide stdlib support for the "**" convention (specifically due to the UI aspect that Michael mentions). -- ___ Python tracker

[issue13968] Support recursive globs

Michael.Elsdörfer added the comment: This should absolutely be implemented as **: - It is pretty much standard. Recursive globbing is not supported everywhere, but when it is, ** is used. - A separate function will not allow me to let the *user* to decide when recursion should be used. I fin

[issue13968] Support recursive globs

Antoine Pitrou added the comment: > But I think I like this API better as it really is easier to explain. I think you got it wrong. Looking at the implementation, it is no different from the API proposed above, except that it also lets you choose the start dir. --

[issue13968] Support recursive globs

Yuval Greenfield added the comment: I noticed this implementation on PyPI http://packages.python.org/rglob/ which sort of has rglob defined as def rglob(pattern, base='.'): Which seems like the most comprehensible way of doing this, though not the most compact. The code itself isn't in

[issue13968] Support recursive globs

R. David Murray added the comment: The last forumulation of what rglob does "apply the glob to the current directory and all subdirectories recursively, returning the joined list with filenames anchored in the current directory" is simple and intuitive enough for me. (I'm not sure it can be

[issue13968] Support recursive globs

Yuval Greenfield added the comment: Raymond Hettinger, by simple do you mean a single argument rglob function? Or do you mean you prefer glob doesn't get a new function? -- ___ Python tracker

[issue13968] Support recursive globs

Raymond Hettinger added the comment: I would like to see this one API remain simple and leave more complex tasks to os.walk et al. -- nosy: +rhettinger ___ Python tracker ___ _

[issue13968] Support recursive globs

Éric Araujo added the comment: FTR, packaging has an extended glob function (not my code) which supports sets (with { , }) and recursivity (**). -- ___ Python tracker ___ _

[issue13968] Support recursive globs

Antoine Pitrou added the comment: > Ah, OK, so what you are saying is that rglob returns the concatenation > of the results of running ls with the argument glob in each > subdirectory of a walk starting with the current directory, except > that the returned names have paths anchored in the curre

[issue13968] Support recursive globs

R. David Murray added the comment: Ah, OK, so what you are saying is that rglob returns the concatenation of the results of running ls with the argument glob in each subdirectory of a walk starting with the current directory, except that the returned names have paths anchored in the current d

[issue13968] Support recursive globs

Antoine Pitrou added the comment: > The alternative interpretation of the pattern, that it is fully > matched against the CWD, takes you back to having to explain what '..' > means when matched against a given location in the recursion, starting > with the CWD. I clearly don't have a good intui

[issue13968] Support recursive globs

R. David Murray added the comment: Oh, yeah, and there's still the question of whether or not directories are matched by the terminal glob pattern, which I would naively expect they would be, in either interpretation, but I wouldn't be sure. -- ___

[issue13968] Support recursive globs

R. David Murray added the comment: @antoine: no, my description involves recursion. It assumes that the path portion of the glob specifies the directories to *start* from, but that the filename glob portion then applies recursively to any of those start directories. The alternative interpre

[issue13968] Support recursive globs

Yuval Greenfield added the comment: As R. David Murray has suggested I think there may be a middle ground. def rglob(fn_filter, root='.'): That would mean the default use case is still easy to remember as rglob('*.py') and also there aren't any explanations needed for how this function wo

[issue13968] Support recursive globs

Antoine Pitrou added the comment: > I don't know, is it? From what has been said so far I'd expect > */c/d/*.py to look for *.py files in all c/d subdirectories of direct > subdirectories of the current directory, and subdirectories of those > c/d directories. But I wouldn't expect the c/d mat

[issue13968] Support recursive globs

R. David Murray added the comment: I don't know, is it? From what has been said so far I'd expect */c/d/*.py to look for *.py files in all c/d subdirectories of direct subdirectories of the current directory, and subdirectories of those c/d directories. But I wouldn't expect the c/d matchin

[issue13968] Support recursive globs

Antoine Pitrou added the comment: > This discussion does mean I plan to add path filtering though, that > works on the "dirpath" value directly (and *will* allow multi-part > matches). So the "Python source files in subdirectories named 'd' of > directories named 'c'" will be written something l

[issue13968] Support recursive globs

Nick Coghlan added the comment: Given the complexities proposed for the dir matching, I'm shifting back to a solid -1 on this. Trying to match multi-part directories with globs is a nightmare and I currently don't allow it at all in walkdir. Instead, dir filtering and file filtering are expre

[issue13968] Support recursive globs

R. David Murray added the comment: Well, in that case I would expect that the argument 'c/*.py' would start walking in the c directory, but I definitely did not get that impression from Antoine's explanation of how the function works. I again advocate separating the starting path specificatio

[issue13968] Support recursive globs

Yuval Greenfield added the comment: Given /home/a /home/a/k.py /home/a/c/j.py /home/b/z.py /home/b/c/f.py and a current directory of /home/a, we'd have: pattern matches --- --- *.pyk.py, c/j.py c/*.py c/j.py c* c ../*.py ..

[issue13968] Support recursive globs

R. David Murray added the comment: So given /home/a /home/a/k.py /home/a/c/j.py /home/b/z.py /home/b/c/f.py and a current directory of /home/a, we'd have: pattern matches --- --- *.py k.py, c/j.py c/*.pyc/j.py c*c [?] ../*

[issue13968] Support recursive globs

Antoine Pitrou added the comment: > My vote could change if the final semantics are intuitive and unambiguous. How about "match the given glob from this directory and any sub-directory"? That sounds quite intuitive and unambiguous to me. -- ___ Pyth

[issue13968] Support recursive globs

R. David Murray added the comment: How about having separate starting path and glob arguments, where the glob cannot contain any directory? I'm -1 on this function as it stands. My vote could change if the final semantics are intuitive and unambiguous. (It's OK if getting the correct intui

[issue13968] Support recursive globs

Antoine Pitrou added the comment: > > you should use the same algorithm for all globs (e.g. "a/*.py"), shouldn't > > you? > > That specific string would start the walk from the current directory IIUC. Yes but would it match b/a/setup.py? -- ___ Py

[issue13968] Support recursive globs

Yuval Greenfield added the comment: > you should use the same algorithm for all globs (e.g. "a/*.py"), shouldn't > you? That specific string would start the walk from the current directory IIUC. -- ___ Python tracker

[issue13968] Support recursive globs

Antoine Pitrou added the comment: > Yes, that's what I meant by a "single exemption for double dots". The > solution should start the walk from wherever the double dots lead it > to. I don't think that's a single exemption; you should use the same algorithm for all globs (e.g. "a/*.py"), should

[issue13968] Support recursive globs

Yuval Greenfield added the comment: > That depends how you implement it. If you detect that ".." exists and glob for "pn*" inside it, you will probably find "../png". Yes, that's what I meant by a "single exemption for double dots". The solution should start the walk from wherever the double d

[issue13968] Support recursive globs

Antoine Pitrou added the comment: > Again with 'a/b/c/d' and let's add a file 'a/b/png'. > > If the curdir is 'c' and you use rglob('../pn*') you won won't find > '../png' as you would be walking only in the curdir. That depends how you implement it. If you detect that ".." exists and glob for

[issue13968] Support recursive globs

Yuval Greenfield added the comment: > This would be quirky. I don't think '..' should be treated specially. > (there's also the symlinks problem) Again with 'a/b/c/d' and let's add a file 'a/b/png'. If the curdir is 'c' and you use rglob('../pn*') you won won't find '../png' as you would be w

[issue13968] Support recursive globs

Antoine Pitrou added the comment: > >> * Behave like a glob for every subdirectory. Meaning that every > >> relative path gets a '*/' prepended to it. Eg rglob('c/d') started > >> from the directory 'a' will yield 'a/b/c/d'. > > > That's what I would expect. That way, rglob('__init__.py') would

[issue13968] Support recursive globs

Yuval Greenfield added the comment: >> * Behave like a glob for every subdirectory. Meaning that every >> relative path gets a '*/' prepended to it. Eg rglob('c/d') started >> from the directory 'a' will yield 'a/b/c/d'. > That's what I would expect. That way, rglob('__init__.py') would > find

[issue13968] Support recursive globs

Antoine Pitrou added the comment: > * Behave like a glob for every subdirectory. Meaning that every > relative path gets a '*/' prepended to it. Eg rglob('c/d') started > from the directory 'a' will yield 'a/b/c/d'. That's what I would expect. That way, rglob('__init__.py') would find all files

[issue13968] Support recursive globs

R. David Murray added the comment: Hmm. Just to make it clear where I'm coming from, though, I should also point out that I use rdiff-backup, which uses the **/yadayada syntax, and I hate it any time I have to try to figure out what such a glob is going to actually match. -- __

[issue13968] Support recursive globs

Yuval Greenfield added the comment: > I have to say that the non-obvious subtleties you point out in your rglob > make me think I personally would probably opt to use Nick's module directly > instead, so that I was sure what I was getting. I didn't notice these corner cases initially because

[issue13968] Support recursive globs

Eli Bendersky added the comment: > I have to say that the non-obvious subtleties you point out in your rglob > make me think I personally would probably opt to use Nick's module directly > instead, so that I was sure what I was getting. > > -- +1 -- _

[issue13968] Support recursive globs

R. David Murray added the comment: I have to say that the non-obvious subtleties you point out in your rglob make me think I personally would probably opt to use Nick's module directly instead, so that I was sure what I was getting. -- nosy: +r.david.murray __

[issue13968] Support recursive globs

Yuval Greenfield added the comment: Thanks for the bug find Antoine, I worked surprisingly hard trying to make this right in more edge cases and while fixing it I noticed rglob/globtree has 3 options: * Behave like a glob for every subdirectory. Meaning that every relative path gets a '*/' p

[issue13968] Support recursive globs

Nick Coghlan added the comment: This discussion (particularly my final globtree recipe) made me realise that the exact same approach would greatly improve the usability of the all_paths, file_paths and dir_paths iterators in walkdir [1]. Accordingly, walkdir 0.4 will let you write a recursive

[issue13968] Support recursive globs

Eli Bendersky added the comment: >> Well, if glob() already lived in shutil, this decision would be a no-brainer :) Having glob() in the glob module and globtree() in the shutil module, though, looks a bit weird. (I agree having a separate module for glob isn't ideal) Would it be feasible to d

[issue13968] Support recursive globs

Nick Coghlan added the comment: We do have the option of aliasing glob.iglob as shutil.glob... -- ___ Python tracker ___ ___ Python-b

[issue13968] Support recursive globs

Antoine Pitrou added the comment: > However, along with that itertools inspired iterator pipeline based > design, I've also inherited Raymond's preference that particular *use > cases* start life as recipes in the documentation. I think it's important to remember where we are coming from. Many

[issue13968] Support recursive globs

Nick Coghlan added the comment: I think it's important to be clear on what the walkdir API aims to be: a composable toolkit of utilities for directory tree processing. It's overall design is inspired directly by the itertools module. Yes, it started life as a simple proposal to add shutil.fil

[issue13968] Support recursive globs

Eli Bendersky added the comment: Oops, Éric, sorry about the title. I didn't even notice :) -- ___ Python tracker ___ ___ Python-bugs

[issue13968] Support recursive globs

Éric Araujo added the comment: >>> list(glob.rglob('*/setu*.py')) It looks quite strange to me that '/' should be allowed in a function that recurses down directories (see my messages above). OTOH fnmatch is not really appropriate, contrary to my earlier feeling. (Restoring my title change:

[issue13968] Support recursive globs

Changes by Éric Araujo : -- title: Add a recursive function to the glob package -> Support recursive globs ___ Python tracker ___ ___