[issue1521950] shlex.split() does not tokenize like the shell

Roundup Robot added the comment: New changeset ea99e2f0b829 by Vinay Sajip in branch 'default': Closes #1521950: Made shlex parsing more shell-like. https://hg.python.org/cpython/rev/ea99e2f0b829 -- nosy: +python-dev resolution: -> fixed stage: patch review -> resolved status: open -> c

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip : -- assignee: -> vinay.sajip ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:/

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip added the comment: Okay, I've updated with a new patch addressing SilentGhost's comments, and addressed the comments on that patch. If I don't hear any objections by Friday, I plan to commit this change. -- ___ Python tracker

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip : Added file: http://bugs.python.org/file43831/refresh-2016.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue1521950] shlex.split() does not tokenize like the shell

R. David Murray added the comment: No objection from me. I'm not likely to have the time to give it the kind of thorough review I'd *like* to, but I don't think it is really needed. -- ___ Python tracker __

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip added the comment: This has been knocking around since 3.3, but never got enough attention to make it in. Barring objections from anyone, I'd like to commit this patch once I check that it applies cleanly against 3.6, before we get into 3.6 beta. -- versions: +Python 3.6 -P

[issue1521950] shlex.split() does not tokenize like the shell

Andrey Kislyuk added the comment: Is there any chance of getting this into 3.6? We are still in a situation where the shlex module misleads developers into believing that it has functionality to parse things the way the shell does. I've had to vendor the copy of shlex with patches from this bu

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip : Added file: http://bugs.python.org/file36772/80eea6bd898c.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Chris Rebert : -- nosy: +cvrebert ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyt

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip added the comment: Let's hope we can get this into 3.5. I updated my patch a while ago to address RDM's comments. -- versions: +Python 3.5 -Python 3.4 ___ Python tracker _

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip : -- versions: +Python 3.4 -Python 3.3 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip : Added file: http://bugs.python.org/file25809/388411be9b61.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue1521950] shlex.split() does not tokenize like the shell

R. David Murray added the comment: Review, including a code-but-not-algorithm review :), posted. -- ___ Python tracker ___ ___ Pyth

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip added the comment: I've updated the patch following comments by RDM - it probably could do with a code review (now that I've addressed RDM's comments on the docs). -- ___ Python tracker

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Ezio Melotti : -- nosy: +ezio.melotti ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip : Added file: http://bugs.python.org/file25365/9252961a03e7.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue1521950] shlex.split() does not tokenize like the shell

R. David Murray added the comment: I am interested in shell stuff in general. The unicode bug is issue 1170. -- ___ Python tracker ___ ___

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo added the comment: > I'd like to take a look at this (I wasn't aware of it before). Are you interested in shlex in general or only this bug? If the former, then I’ll try to remember to make you nosy on future issues. BTW, what is the shlex unicode bug you mentioned a few times on

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip added the comment: I believe Dan meant that the behaviour of shlex.split() now is different from what it was when he first raised the issue (in July 2006). Looking at the default branch of CPython, this is what I see: Python 3.3.0a2+ (default:ff6593aa8376, Apr 22 2012, 12:39:08)

[issue1521950] shlex.split() does not tokenize like the shell

R. David Murray added the comment: I'd like to take a look at this (I wasn't aware of it before). I'll try to do that some time in the next 24 hours, and if I don't you shouldn't wait for me :) Did you address Dan's concern about 'old' possibly not matching the old behavior completely?

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip added the comment: I've received no comments on the latest revision of my patch (incorporating comments on the previous version); is it OK to commit this? -- ___ Python tracker _

[issue1521950] shlex.split() does not tokenize like the shell

Dan Christian added the comment: I haven't been following this much. Sorry. My day job isn't in this area any more (and I'm stuck using 2.4 :-(). Looking at the docs, I notice the "old" is different from what it used to be. Notably: 'e;' gets split into two tokens; and ">'abc';" gets split

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip added the comment: >Éric Araujo added the comment: >I did not fully get what you meant here, but the example you added to the doc >made it clear.  Is this covered by tests? Yes, I believe that testSyntaxSplitCustom covers this. >Overall great patch!  Dan, do you have time to tes

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Gustavo Niemeyer : -- nosy: -niemeyer ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo added the comment: > Previously when punctuation chars were set, wordchars was being augmented by > '-'. This was > incomplete, so the augmentation is now with '~-./*?=' which allows for > wildcards, filename > chars and argument flags. I did not fully get what you meant here, but

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip added the comment: Plus I also changed a few instances of the anachronism a = a + b to a += b -- ___ Python tracker ___

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip added the comment: I updated the patch to reflect Éric's comments on Rietveld, but there are also some other changes: Previously when punctuation chars were set, wordchars was being augmented by '-'. This was incomplete, so the augmentation is now with '~-./*?=' which allows for

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip : Added file: http://bugs.python.org/file24590/079ab75d29a4.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo added the comment: This time you should have received an email from Rietveld, I made sure that your ID was expanded to an email address. I like all the suggestions you made in reply to my comments. -- ___ Python tracker

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip : Added file: http://bugs.python.org/file24158/9e12275eec25.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip added the comment: I've made a patch which implements this functionality, together with docs and tests. Please review. -- hgrepos: +99 nosy: +vinay.sajip stage: test needed -> patch review ___ Python tracker

[issue1521950] shlex.split() does not tokenize like the shell

Dan Christian added the comment: > Sure :)  That’s why I suggest using dash for quick tests and rely on the work > of other people who did read the POSIX spec.  I’ll have to check it too > before committing a patch. The point of ref_shlex.py is that all shells act the same for common cases an

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo added the comment: > I was just looking for a reference where I didn't have to sift through tons > of documentation. Sure :) That’s why I suggest using dash for quick tests and rely on the work of other people who did read the POSIX spec. I’ll have to check it too before committ

[issue1521950] shlex.split() does not tokenize like the shell

Dan Christian added the comment: On Sat, Nov 26, 2011 at 7:12 AM, Éric Araujo wrote: > Your script passes with dash, which is probably the most POSIX-compliant > shell we can find.  (bash has extensions, zsh/csh don’t use the POSIX shell > language, so I think the behavior of dash should be o

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo added the comment: Thanks for the diff and test. (I removed the older versions; there are “edit” links in the list of files leading to pages where it’s possible to remove them, if one has the required permissions.) Your script passes with dash, which is probably the most POSIX-co

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Éric Araujo : Removed file: http://bugs.python.org/file23779/test_shlex.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Éric Araujo : Removed file: http://bugs.python.org/file23778/ref_shlex.py ___ Python tracker ___ ___ Python-bugs-list mailing lis

[issue1521950] shlex.split() does not tokenize like the shell

Dan Christian added the comment: I just realized that I left out a major case. The shell will also split (). I think this is now complete. If you do "man bash" and skip down to DEFINITONS it lists all the control characters. I've attached updated versions of ref_shlex.py and test_shlex.diff.

[issue1521950] shlex.split() does not tokenize like the shell

Dan Christian added the comment: I've attached a diff to test_shlex.py and a script that I used to verify what the shells actually do. Both are relative to Python-3.2.2/Lib/test I'm completely ignoring the quotes issue for now. That should probably be an enhancement. I don't think it really m

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo added the comment: > Of course, that's how it's used. That's all it can do right now. :) What I meant is that it is *meant* to be used in this way. > I was was splitting and combining commands (using ;, &&, and ||) and then > running the resulting > (mega) one liners over ssh. It

[issue1521950] shlex.split() does not tokenize like the shell

Dan Christian added the comment: Of course, that's how it's used. That's all it can do right now. I was was splitting and combining commands (using ;, &&, and ||) and then running the resulting (mega) one liners over ssh. It still gets run by a shell, but I was specifying the control flow. 0

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo added the comment: Thanks for the comments. > There are really two cases in one bug. > The first part is that the shell will split tokens at characters that shlex > doesn't. The handling > of &, |, ;, >, and < could be done by adjusting the definition of > shlex.wordchars. The s

[issue1521950] shlex.split() does not tokenize like the shell

Dan Christian added the comment: It's been a while since I looked at this.  I'm not really in a position to contribute code/tests right now; but I can comment. I don't think POSIX mode existed when I first reported this, but that's where it makes sense.  I think all POSIX shells (borne, C, korn

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo added the comment: Thanks for the report. Would you like to work on a patch, or translate your examples into unit tests? The docs do not mention “&” at all, and platform discrepancies have to be taken into account too, so I really don’t know if this is a bug fix for the normal mo

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Mark Lawrence : -- versions: +Python 3.2 -Python 2.7 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Daniel Diniz : -- stage: -> test needed type: -> feature request versions: +Python 2.7 -Python 2.3 ___ Python tracker ___ _