Re: PEP suggestion: Uniform way to indicate Python language version
On Monday, August 22, 2016 at 2:04:39 AM UTC-4, Random832 wrote: > On Mon, Aug 22, 2016, at 01:35, Steven D'Aprano wrote: > > Could somebody (the OP?) please explain what is the purpose of this > > proposal, what it does, how it works, and when would people use it? > > I think what he wants is a way for a module which uses features > (syntactic or otherwise, but I suppose especially syntactic features > since this can't as easily be done with a runtime check using existing > mechanisms) from a particular python version and which makes no > provision to run under earlier versions to fail with a message like > "This script requires Python 3.4 or later" rather than a mysterious > syntax error or worse a runtime error after the program has been running > for some time. Right. People are focusing on specific code instead of the problem: a simple and uniform way to indicate a specific Python dialect in force for that program. The language continues to evolve over time: there are many Python 2.7 programs that won't work on Python 2.5 or earlier and vice versa. When you expand the range from Python 1.5 to Python 3.6 the likelihood of the program running becomes even smaller. Furthermore, I am not aware of any program that when given a Python source code will tell you which or versions or dialects of Python it will run on. The fact that there has been all this much discussion over specific code to me enforces the need for a simple an uniform mechanism. -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On 2016-08-22, Steve D'Aprano wrote: > On Mon, 22 Aug 2016 10:38 am, eryk sun wrote: >> To me it's scary that this check misses cases because it's trying to >> be cross-platform instead of simply relying on GetFullPathName to do >> the work. For example, it misses at least the following cases: > > Instead of shaking in your boots over a simple bug in a non-critical > library, how about reporting these cases on the bug tracker with an > explanation of the problem? That seems a rather unnecessarily harsh response. Also, it's not "non-critical", this is a security bug. -- https://mail.python.org/mailman/listinfo/python-list
Re: PEP suggestion: Uniform way to indicate Python language version
On Monday, August 22, 2016 at 1:36:07 AM UTC-4, Steven D'Aprano wrote:
> On Monday 22 August 2016 14:33, Chris Angelico wrote:
>
> > On Mon, Aug 22, 2016 at 1:37 PM, rocky wrote:
> >> Sorry should have been:
> >>
> >> assert sys.version_info >= (3,0)
> >
> > The next question is: How common is code like this? I don't put
> > version checks in any of my modules. Adding magic comments would be of
> > value only if this sort of thing is common enough to need its own
> > syntax.
>
> Most of my code is supposed to work on Python 2.4 through the latest 3.x
> version. That means I have lots of version-specific code.
>
> But most of it uses feature detection rather than version checks. So instead
> of
> writing:
>
> if sys.version < '3':
> def next(iterator):
> return iterator.next()
>
>
> I write this:
>
> try:
> next
> except NameError:
> def next(iterator):
> return iterator.next()
>
> or possibly this:
>
>
> if not hasattr(builtins, 'next'):
> def next(iterator):
> return iterator.next()
>
>
> But some features depend on new syntax, and you can't check for the
> availability of new syntax at runtime. (Well, you can, by clever use of
> exec(),
> but its too much trouble than its worth.) So very, very occasionally I have
> to
> do a version check:
>
>
> if sys.version < '3':
> import mymodule2 as mymodule
> else:
> import mymodule3 as mymodule
>
>
> But... I don't understand what this proposal actually is. We already have a
> uniform way to indicate the Python language version: check sys.version, or
> sys.version_info, or sys.hexversion, whichever is more appropriate for your
> needs.
The fact that you mention 3 different ways says to me this isn't uniform.
> Because this is just an ordinary comparison, you can then do whatever
> you like, no matter how outlandish or unusual:
>
>
> if sys.version < '3':
> try:
> import mymodule3
> except SyntaxError:
> print("Yay, a syntax error!")
> else:
> print("Now that's weird, that shouldn't happen...")
> del mymodule3
> import mymodule2 as mymodule
> elif config[RUN_OLD_VERSION]:
> import mymodule2 as mymodule
> else:
> import mymodule3 as mymodule
>
>
> Could somebody (the OP?) please explain what is the purpose of this proposal,
> what it does, how it works, and when would people use it?
All of the ways you mention involve running the program. Suppose you have a
tool that analyzes source code. It reads the code, but doesn't run it. For
example possibly code climate. How it it figure out which dialect of Python is
being used?
>
>
>
> --
> Steve
--
https://mail.python.org/mailman/listinfo/python-list
Re: PEP suggestion: Uniform way to indicate Python language version
On Monday, August 22, 2016 at 3:53:27 PM UTC+5:30, rocky wrote: > On Monday, August 22, 2016 at 2:04:39 AM UTC-4, Random832 wrote: > > On Mon, Aug 22, 2016, at 01:35, Steven D'Aprano wrote: > > > Could somebody (the OP?) please explain what is the purpose of this > > > proposal, what it does, how it works, and when would people use it? > > > > I think what he wants is a way for a module which uses features > > (syntactic or otherwise, but I suppose especially syntactic features > > since this can't as easily be done with a runtime check using existing > > mechanisms) from a particular python version and which makes no > > provision to run under earlier versions to fail with a message like > > "This script requires Python 3.4 or later" rather than a mysterious > > syntax error or worse a runtime error after the program has been running > > for some time. > > Right. People are focusing on specific code instead of the problem: a simple > and uniform way to indicate a specific Python dialect in force for that > program. The language continues to evolve over time: there are many Python > 2.7 programs that won't work on Python 2.5 or earlier and vice versa. When > you expand the range from Python 1.5 to Python 3.6 the likelihood of the > program running becomes even smaller. > > Furthermore, I am not aware of any program that when given a Python source > code will tell you which or versions or dialects of Python it will run on. > > The fact that there has been all this much discussion over specific code to > me enforces the need for a simple an uniform mechanism. Prior Art: I may mention that the web-language curl has this feature builtin See very first example here: https://en.wikipedia.org/wiki/Curl_(programming_language) Note this is version(s) specification Auto version detection is hard and likely impossible -- https://mail.python.org/mailman/listinfo/python-list
Re: PEP suggestion: Uniform way to indicate Python language version
On Monday, August 22, 2016 at 6:44:43 AM UTC-4, Rustom Mody wrote: > On Monday, August 22, 2016 at 3:53:27 PM UTC+5:30, rocky wrote: > > On Monday, August 22, 2016 at 2:04:39 AM UTC-4, Random832 wrote: > > > On Mon, Aug 22, 2016, at 01:35, Steven D'Aprano wrote: > > > > Could somebody (the OP?) please explain what is the purpose of this > > > > proposal, what it does, how it works, and when would people use it? > > > > > > I think what he wants is a way for a module which uses features > > > (syntactic or otherwise, but I suppose especially syntactic features > > > since this can't as easily be done with a runtime check using existing > > > mechanisms) from a particular python version and which makes no > > > provision to run under earlier versions to fail with a message like > > > "This script requires Python 3.4 or later" rather than a mysterious > > > syntax error or worse a runtime error after the program has been running > > > for some time. > > > > Right. People are focusing on specific code instead of the problem: a > > simple and uniform way to indicate a specific Python dialect in force for > > that program. The language continues to evolve over time: there are many > > Python 2.7 programs that won't work on Python 2.5 or earlier and vice > > versa. When you expand the range from Python 1.5 to Python 3.6 the > > likelihood of the program running becomes even smaller. > > > > Furthermore, I am not aware of any program that when given a Python source > > code will tell you which or versions or dialects of Python it will run on. > > > > The fact that there has been all this much discussion over specific code to > > me enforces the need for a simple an uniform mechanism. > > Prior Art: > I may mention that the web-language curl has this feature builtin > See very first example here: > https://en.wikipedia.org/wiki/Curl_(programming_language) > > Note this is version(s) specification > Auto version detection is hard and likely impossible As mentioned in the original post Perl has it too. http://perldoc.perl.org/functions/use.html And I gotta say it is pretty clever how they worked in the duplicate match up between common English usage and valid programming language syntax. -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On Mon, Aug 22, 2016 at 8:33 PM, Jon Ribbens wrote: > On 2016-08-22, Steve D'Aprano wrote: >> On Mon, 22 Aug 2016 10:38 am, eryk sun wrote: >>> To me it's scary that this check misses cases because it's trying to >>> be cross-platform instead of simply relying on GetFullPathName to do >>> the work. For example, it misses at least the following cases: >> >> Instead of shaking in your boots over a simple bug in a non-critical >> library, how about reporting these cases on the bug tracker with an >> explanation of the problem? > > That seems a rather unnecessarily harsh response. > Also, it's not "non-critical", this is a security bug. Explain how? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: PEP suggestion: Uniform way to indicate Python language version
On Monday, August 22, 2016 at 12:33:53 AM UTC-4, Chris Angelico wrote:
> On Mon, Aug 22, 2016 at 1:37 PM, rocky wrote:
> > Sorry should have been:
> >
> > assert sys.version_info >= (3,0)
>
> The next question is: How common is code like this? I don't put
> version checks in any of my modules. Adding magic comments would be of
> value only if this sort of thing is common enough to need its own
> syntax.
>
> ChrisA
I'll answer your question with a question: How common are "software as service"
code-analysis tools for Python used?
A slightly different but related problem is noting the Python dialect at the
package-level. If a package is in Pypi then PKGINFO can contain the a list of
Python dialects supported. If the program is not in Pypi, then what? As with
"use perl" inside source code, the same mechanism ("use perl") is used inside
its build system(s) to indicate which dialects a Perl package supports.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On 2016-08-22, Chris Angelico wrote: > On Mon, Aug 22, 2016 at 8:33 PM, Jon Ribbens > wrote: >> On 2016-08-22, Steve D'Aprano wrote: >>> On Mon, 22 Aug 2016 10:38 am, eryk sun wrote: To me it's scary that this check misses cases because it's trying to be cross-platform instead of simply relying on GetFullPathName to do the work. For example, it misses at least the following cases: >>> >>> Instead of shaking in your boots over a simple bug in a non-critical >>> library, how about reporting these cases on the bug tracker with an >>> explanation of the problem? >> >> That seems a rather unnecessarily harsh response. >> Also, it's not "non-critical", this is a security bug. > > Explain how? I don't know what purpose you are envisaging this function being used for, but the only one I can think of is input sanitisation. e.g. a web form where you receive a file from the Internet and store it somewhere, and you want to use the filename given to you rather than choose your own randomly-generated one. Under Unix all you need to do is check for the filename starting with "." or containing "/." (or "/", depending on your requirements). Under Windows you would use this function, which apparently doesn't work, hence: security hole. -- https://mail.python.org/mailman/listinfo/python-list
Re: PEP suggestion: Uniform way to indicate Python language version
rocky : > A slightly different but related problem is noting the Python dialect > at the package-level. I don't know what if anything is needed support this idea, but one option would be to just use "import": import python3_5_17 That would require Python and modules to install such empty modules to indicate that they support the given API. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On Mon, 22 Aug 2016 08:33 pm, Jon Ribbens wrote: > On 2016-08-22, Steve D'Aprano wrote: >> On Mon, 22 Aug 2016 10:38 am, eryk sun wrote: >>> To me it's scary that this check misses cases because it's trying to >>> be cross-platform instead of simply relying on GetFullPathName to do >>> the work. For example, it misses at least the following cases: >> >> Instead of shaking in your boots over a simple bug in a non-critical >> library, how about reporting these cases on the bug tracker with an >> explanation of the problem? > > That seems a rather unnecessarily harsh response. Eryksun bought into Lawrence's over-the-top rhetorical question "does this scare you?" by answering "Yes", and repeating the ridiculous term "scary". He specifically said that it scares him *because* it is cross-platform code, as if cross-platform code is a bad thing. Now I'm sure that Eryksun isn't *actually* scared of cross-platform code. I'm sure he is quite capable of using (say) os.listdir() without widdling himself in terror *wink*. And I don't know if he was intentionally using the word "scary" or whether it was just an ill-thought out choice of words. Either way, yes, I'm making a gentle dig at Eryksun for exaggerating the magnitude of the supposed problem and for taking something which is clearly a mere bug and treating it as a feature that is broken by design. There's nothing wrong with writing cross-platform code, and there's no reason why non-Windows users shouldn't be permitted to explicitly query whether a file name could be valid on a Windows system. > Also, it's not "non-critical", this is a security bug. How is this a security bug? What's the nature of the vulnerability? -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On Mon, Aug 22, 2016 at 9:50 PM, Jon Ribbens wrote: > On 2016-08-22, Chris Angelico wrote: >> On Mon, Aug 22, 2016 at 8:33 PM, Jon Ribbens >> wrote: >>> On 2016-08-22, Steve D'Aprano wrote: On Mon, 22 Aug 2016 10:38 am, eryk sun wrote: > To me it's scary that this check misses cases because it's trying to > be cross-platform instead of simply relying on GetFullPathName to do > the work. For example, it misses at least the following cases: Instead of shaking in your boots over a simple bug in a non-critical library, how about reporting these cases on the bug tracker with an explanation of the problem? >>> >>> That seems a rather unnecessarily harsh response. >>> Also, it's not "non-critical", this is a security bug. >> >> Explain how? > > I don't know what purpose you are envisaging this function being used > for, but the only one I can think of is input sanitisation. e.g. a web > form where you receive a file from the Internet and store it somewhere, > and you want to use the filename given to you rather than choose your > own randomly-generated one. > > Under Unix all you need to do is check for the filename starting with > "." or containing "/." (or "/", depending on your requirements). > Under Windows you would use this function, which apparently doesn't > work, hence: security hole. Nope. On Windows, you would try/except it. There are myriad other ways something could fail, and the only correct action is to attempt it. Most of the reserved names will simply give an error; the only way you'd actually get incorrect behaviour is if the file name, including extension, is exactly a device name. (Caveat: My knowledge of Windows is rusty and my testing just now was cursory. I could be wrong.) So you can check for a few exact strings... or just slap some extra text onto the beginning or end of the file name (beginning meaning "after the last slash", not the beginning of the file *path*) and you're safe. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: PEP suggestion: Uniform way to indicate Python language version
On Mon, Aug 22, 2016 at 10:05 PM, Marko Rauhamaa wrote: > rocky : > >> A slightly different but related problem is noting the Python dialect >> at the package-level. > > I don't know what if anything is needed support this idea, but one > option would be to just use "import": > > import python3_5_17 > > That would require Python and modules to install such empty modules to > indicate that they support the given API. Preferable: from python import v35 Then a single 'python.py' could have lines for every supported version, rather than forcing dozens of separate modules. However, I don't think it's particularly necessary. Explicit version number checks should be very rare, and shouldn't be encouraged. Instead, encourage feature checks, as Steve gave some examples of. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: PEP suggestion: Uniform way to indicate Python language version
On Monday, August 22, 2016 at 8:05:15 AM UTC-4, Marko Rauhamaa wrote: > rocky : > > > A slightly different but related problem is noting the Python dialect > > at the package-level. > > I don't know what if anything is needed support this idea, but one > option would be to just use "import": > > import python3_5_17 > > That would require Python and modules to install such empty modules to > indicate that they support the given API. > > > Marko Yes, that is the intent of File/module python30.py in the first post. The only difference is that because there is code too, when run, it can enforce the version declared. As mentioned originally, a problem with this is that you have a proliferation of names and files. Better, I htink would be a way to pass a parameter string. Perl's "use" mechanism happens to allow passing in such a parameter. In Python without syntactic sugar this would be a two-step process of first importing and then calling something with the version range. -- https://mail.python.org/mailman/listinfo/python-list
Re: PEP suggestion: Uniform way to indicate Python language version
On Mon, Aug 22, 2016, at 08:44, Chris Angelico wrote: > However, I don't think it's particularly necessary. Explicit version > number checks should be very rare, and shouldn't be encouraged. > Instead, encourage feature checks, as Steve gave some examples of. The problem is when you want to write a large body of code that just *uses* lots of features (including syntactic features), *without* checking for them. Steve's examples were of how to support earlier versions, not how *not* to. At some point you want to just write Python 3 code and say to hell with Python 2, or the same for any particular version of Python 3. -- https://mail.python.org/mailman/listinfo/python-list
Re: PEP suggestion: Uniform way to indicate Python language version
On Monday, August 22, 2016 at 8:45:05 AM UTC-4, Chris Angelico wrote: > On Mon, Aug 22, 2016 at 10:05 PM, Marko Rauhamaa wrote: > > rocky : > > > >> A slightly different but related problem is noting the Python dialect > >> at the package-level. > > > > I don't know what if anything is needed support this idea, but one > > option would be to just use "import": > > > > import python3_5_17 > > > > That would require Python and modules to install such empty modules to > > indicate that they support the given API. > > Preferable: > > from python import v35 > > Then a single 'python.py' could have lines for every supported > version, rather than forcing dozens of separate modules. > > However, I don't think it's particularly necessary. Explicit version > number checks should be very rare, and shouldn't be encouraged. > Instead, encourage feature checks, as Steve gave some examples of. > > ChrisA Yes, "from python import v35" is better. How do feature checks in the code address the problem static analysis of source code? -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On Mon, Aug 22, 2016, at 08:39, Chris Angelico wrote: > Nope. On Windows, you would try/except it. No, you can't, because the failure mode often isn't "file refuses to open" but "data is written to a serial port". There are myriad other ways > something could fail, and the only correct action is to attempt it. > Most of the reserved names will simply give an error; the only way > you'd actually get incorrect behaviour is if the file name, including > extension, is exactly a device name. I think the reason you believe this can be traced back to the "C:\con\con" trick, which crashed the system by trying to use the name as a directory. > (Caveat: My knowledge of Windows > is rusty and my testing just now was cursory. I could be wrong.) Eryk Sun already posted an example using "NUL .txt". -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On Mon, Aug 22, 2016 at 12:39 PM, Chris Angelico wrote: > > Nope. On Windows, you would try/except it. There are myriad other ways No, I would not rely on exceptions in this case. Both \\.\CON and \\.\NUL can be opened for both reading and writing, so you may not detect the problem. I think it's best to check for this via os.path.abspath, which calls GetFullPathName, which is implemented by RtlGetFullPathName_Ustr, which is exactly what RtlDosPathNameToNtPathName_U uses to normalize a path before it converts it to the NT namespace. > or just slap some extra text onto the beginning or end of the file name > (beginning meaning "after the last slash", not the beginning of the file > *path*) and you're safe. Adding to the beginning is the safe bet without having to worry about the rules for trailing spaces, dots, and colons at the end of the name. Adding a single underscore prefix will suffice (or however many to make a unique name) if for some reason you can't ask the user for a new name. -- https://mail.python.org/mailman/listinfo/python-list
Re: PEP suggestion: Uniform way to indicate Python language version
Random832 : > The problem is when you want to write a large body of code that just > *uses* lots of features (including syntactic features), *without* > checking for them. Ordinarily, that's the job of package management. The installer will perform the necessary checks for you. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On Mon, Aug 22, 2016 at 10:56 PM, Random832 wrote: >> Most of the reserved names will simply give an error; the only way >> you'd actually get incorrect behaviour is if the file name, including >> extension, is exactly a device name. > > I think the reason you believe this can be traced back to the > "C:\con\con" trick, which crashed the system by trying to use the name > as a directory. I tried things like "con.txt" and it simply failed (no such file or directory), without printing anything to the console. But as Eryk says, adding an underscore is safe; and to be honest, I wouldn't accept file names from untrusted sources on *any* system - at very least, I'd prefix/suffix them with something to ensure uniqueness, which would deal with this issue as a convenient side effect. (Or alternatively, I'd use arbitrary numbers or hashes as the file names, and store the originally-submitted file name in some sort of metadata repository, like a Postgres table.) So I still don't see this as a security problem, just a practicality one. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: PEP suggestion: Uniform way to indicate Python language version
On Mon, 22 Aug 2016 08:32 pm, rocky wrote:
> On Monday, August 22, 2016 at 1:36:07 AM UTC-4, Steven D'Aprano wrote:
[...]
>> But... I don't understand what this proposal actually is. We already have
>> a uniform way to indicate the Python language version: check sys.version,
>> or sys.version_info, or sys.hexversion, whichever is more appropriate for
>> your needs.
>
> The fact that you mention 3 different ways says to me this isn't uniform.
What do you mean? Of course it is uniform -- every compliant Python
interpreter must support sys.version, sys.hexversion and sys.version_info.
sys.version and sys.hexversion go back to at least Python 1.5 and possibly
older. I'm not sure when version_info was introduced, but anything offering
Python 2.4 or better must support it. So as far as current and future
versions of Python go, you can trust that *any* version of *any*
implementation will support those three attributes, be it CPython, PyPy,
Nuitika, Jython, IronPython, Stackless, or something else.
Perhaps you mean "unique" rather than uniform -- but of course you cannot
have a unique way of doing this, since people can simply test for the
existence of features, or inspect the version number, whichever they
prefer. Adding something like Perl's "use" won't change that -- it still
won't be unique.
>> Could somebody (the OP?) please explain what is the purpose of this
>> proposal, what it does, how it works, and when would people use it?
>
> All of the ways you mention involve running the program. Suppose you have
> a tool that analyzes source code. It reads the code, but doesn't run it.
> For example possibly code climate.
What is "code climate"?
> How it it figure out which dialect of Python is being used?
In general, you can't, because Python code will run under many different
versions and implementations.
I'm not strongly opposed to this suggestion. I just don't see the point.
I've never found myself missing this feature. And I don't like version
testing except as an absolute last resort: I much prefer feature detection.
But the biggest problem with this suggestion is the practical issue that the
absolute earliest it could be introduced will be Python 3.6, and more
likely 3.7. Python 3.6 has only a couple of weeks left before feature
freeze, and all older versions are likewise frozen or out of support.
Realistically, by the time you convince people this is a useful feature,
write a patch and have the patch reviewed, you'll be looking at Python 3.7.
So 3.7 will have this new syntax "use version", and *no other version*.
That means you can't use it until you are ready to abandon Python 2.4
through 3.6.
I'm assuming you want a new syntactic feature like Perl's "use version". If
its just a function, then you can use a version check to test for it first:
if sys.version_info > (3, 7):
sys.use("v3.7")
but that makes "use" completely redundant. (If you have to test the version
before you can test the version, what's the point of the second test?)
If this feature had been built into Python from the start, it would merely
be an attractive nuisance, encouraging people to test the version instead
of doing feature-detection. But since it wasn't, I don't see any point to
add it now: if we added it to Python 3.7, it wouldn't be until (probably)
2028 or significantly later that folks could rely on it being available.
(Currently, I think that Red Hat offer the longest paid commercial support
for Python versions: ten years. Suppose they start using Python 3.6 as
their standard Python next year, it won't fall out of support until 2028.
That's the absolute earliest pre-3.7 versions could be considered out of
support.)
Earlier, in a previous email, Rocky also wrote:
> Right. People are focusing on specific code instead of the problem: a
> simple and uniform way to indicate a specific Python dialect in force for
> that program.
That's not describing the problem. That's describing one specific solution
to some unknown problem. The *solution* is "a simple and uniform way to
indicate..." but what is the problem? Under what circumstances would people
use this? Why would they use it?
Normally, there's no question about what version of Python a module is meant
to use:
- if it is a script, it has a hash-bang line at the top which specifies the
location of the Python interpreter;
- if its a library, it will be installed in the site-packages directory of a
specific version of the interpreter;
- and more importantly, many (possibly even most) Python code will run under
multiple versions.
There are exceptions to this of course. Sometimes libraries will be just
dumped any old place, and scripts may not have hash-bang lines. Neither of
these are compulsory. But neither will your proposed "use version".
> The language continues to evolve over time: there are many
> Python 2.7 programs that won't work on Python 2.5 or earlier and vice
> versa. When you expand the range from Python 1.5 to Python 3.6 the
> like
Re: PEP suggestion: Uniform way to indicate Python language version
On Monday, August 22, 2016 at 7:00:36 PM UTC+5:30, Steve D'Aprano wrote: > Realistically, by the time you convince people this is a useful feature, > write a patch and have the patch reviewed, you'll be looking at Python 3.7. > So 3.7 will have this new syntax "use version", and *no other version*. > That means you can't use it until you are ready to abandon Python 2.4 > through 3.6. Yes First-class version detection and management needs to be put into the language right at the outset if it is to work If you do it late in the history (like now for python), you need to 1 First detect if first-class version management is supported 2 If yes use it 3 If no use the old ad-hoc ways So might as well use 3 alone!! -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On Mon, 22 Aug 2016 09:50 pm, Jon Ribbens wrote: > On 2016-08-22, Chris Angelico wrote: >> On Mon, Aug 22, 2016 at 8:33 PM, Jon Ribbens >> wrote: >>> On 2016-08-22, Steve D'Aprano wrote: On Mon, 22 Aug 2016 10:38 am, eryk sun wrote: > To me it's scary that this check misses cases because it's trying to > be cross-platform instead of simply relying on GetFullPathName to do > the work. For example, it misses at least the following cases: Instead of shaking in your boots over a simple bug in a non-critical library, how about reporting these cases on the bug tracker with an explanation of the problem? >>> >>> That seems a rather unnecessarily harsh response. >>> Also, it's not "non-critical", this is a security bug. >> >> Explain how? > > I don't know what purpose you are envisaging this function being used > for, but the only one I can think of is input sanitisation. e.g. a web > form where you receive a file from the Internet and store it somewhere, > and you want to use the filename given to you rather than choose your > own randomly-generated one. > > Under Unix all you need to do is check for the filename starting with > "." or containing "/." (or "/", depending on your requirements). > Under Windows you would use this function, which apparently doesn't > work, hence: security hole. That's backwards: it works under Windows, but is buggy under non-Windows. In any case, I don't think that what you suggest is even close to sufficient. For starters, even on Unix I probably wouldn't want to use "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\.txt" as a file name, even if it is technically legal. Nor would I allow "foo.jpg .exe" either, especially not on Windows. But let's suppose my application can handle newlines and other control characters in filenames (I never `ls` the storage directory, all my scripts are newline safe, etc.) And I don't care about users uploading malware with disguised file names. Am I done? No, not at all. I can't assume that just because a file name is legal that I can write to it. Even after sanitising the name (or deciding I don't need to bother) I still need to be prepared to catch I/O errors when writing the file -- including attempts to write to reserved file names. None of this makes the issue a security issue. How would you exploit it? The attacker would have to ensure that the file name is sanitised under Linux but then written to a file under Windows, in which case the worst that happens is that I get an unexpected I/O error. Technically that could be called a DoS attack, but the same could be said for ANY bug that raises an exception. I think that it takes more than that to be a security bug. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: PEP suggestion: Uniform way to indicate Python language version
On Mon, 22 Aug 2016 10:52 pm, Random832 wrote:
> On Mon, Aug 22, 2016, at 08:44, Chris Angelico wrote:
>> However, I don't think it's particularly necessary. Explicit version
>> number checks should be very rare, and shouldn't be encouraged.
>> Instead, encourage feature checks, as Steve gave some examples of.
>
> The problem is when you want to write a large body of code that just
> *uses* lots of features (including syntactic features), *without*
> checking for them. Steve's examples were of how to support earlier
> versions, not how *not* to. At some point you want to just write Python
> 3 code and say to hell with Python 2, or the same for any particular
> version of Python 3.
Surely that's not a problem -- it's what people have been doing with Python
code for decades. When was the last time you saw somebody do an explicit
version check?
if sys.version_info() < (2, 2):
raise RuntimeError('Python too old')
Rather, you just use the features you rely on, document the minimum
supported version, and if somebody is silly enough to try running your code
under Python 1.4, they'll get a SyntaxError or an exception when you try to
do something that is not supported.
Maybe if Python supported this from the beginning it would be worth the
bother. But it seems of very marginal utility to me.
--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On Mon, 22 Aug 2016 10:56 pm, Random832 wrote: > On Mon, Aug 22, 2016, at 08:39, Chris Angelico wrote: >> Nope. On Windows, you would try/except it. > > No, you can't, because the failure mode often isn't "file refuses to > open" but "data is written to a serial port". Ah, that's a good point. I hadn't thought of that. But... what are the consequences if you write to the serial port? Unless you actually have an external device plugged into it, isn't that equivalent to writing to /dev/null? (Bytes go into the serial port, and just disappear.) The user uploads their file, and cleverly fools you into discarding their file? I'm not seeing how this is an attack. I suppose they could write to CON and display a lot of garbage on the screen. But if you're running this on Windows, surely you've already dealt with these issues, in which case it's a non-issue. Or you haven't dealt with them, in which case it's an existing bug and the code Lawrence demonstrated doesn't change anything. >> There are myriad other ways >> something could fail, and the only correct action is to attempt it. >> Most of the reserved names will simply give an error; the only way >> you'd actually get incorrect behaviour is if the file name, including >> extension, is exactly a device name. > > I think the reason you believe this can be traced back to the > "C:\con\con" trick, which crashed the system by trying to use the name > as a directory. \con\con hasn't been an issue since Windows 98. If you're running your web application under Win 98, you deserve to be blue-screened :-) -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)
Chris Angelico writes: > […] to be honest, I wouldn't accept file names from untrusted sources > on *any* system […] That's one of the wiser things said in this whole thread. > I'd use arbitrary numbers or hashes as the file names, and store the > originally-submitted file name in some sort of metadata repository, > like a Postgres table.) The failure modes of using filenames from untrusted input are shockingly diverse, as Tom Eastman describes: The scope for abuse is eye-widening: The contents of the file, the type of the file, the size and encoding of the file, even the *name* of the file can be a potent vector for attacking your system. The scariest part? Even the best and most secure web-frameworks (yes, I'm talking about Django) can't protect you from all of it. In this talk, I'll show you every scary thing I know about that can be done with a file upload, and how to protect yourself from -- hopefully -- most of them. https://2016.pycon-au.org/schedule/148/view_talk> Tom presented to us at this year's PyCon AU https://www.youtube.com/watch?v=HS8KQbswZkU>. So yes, filenames from arbitrary sources should be *completely* untrusted, and never used to access any file on the system. Throw the entire filename away and make a filename locally, without using any part of the original name. -- \“I saw a sign: ‘Rest Area 25 Miles’. That's pretty big. Some | `\ people must be really tired.” —Steven Wright | _o__) | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: PEP suggestion: Uniform way to indicate Python language version
On Mon, Aug 22, 2016, at 09:21, Steve D'Aprano wrote: > Rather, you just use the features you rely on, document the minimum > supported version, and if somebody is silly enough to try running your > code > under Python 1.4, they'll get a SyntaxError or an exception when you try > to > do something that is not supported. Receiving a SyntaxError or whatever other exception, which provides no suggestion about how to actually fix the issue (install a later version of python / run with "python3" instead of "python"), is a bad user experience. It will continue to be a bad user experience when people are using features that only work on python 5.0 and later and other people are trying to run their scripts under python 4.0, so it not having existed all along is not a sufficient justification to not consider adding it,. -- https://mail.python.org/mailman/listinfo/python-list
Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)
On Mon, Aug 22, 2016, at 10:21, Ben Finney wrote:
> So yes, filenames from arbitrary sources should be *completely*
> untrusted, and never used to access any file on the system. Throw the
> entire filename away and make a filename locally, without using any part
> of the original name.
To be fair, this particular case is unique in presenting a possibility
to cause problems even for a filename that consists only of whitelisted
characters (for a reasonable-sounding whitelist such as "ASCII letters
and numbers and underscore only; all other characters to be scrubbed and
replaced with {underscore, hex escape, nothing}"). I don't think there's
any other precedent.
--
https://mail.python.org/mailman/listinfo/python-list
Re: saving octet-stream png file
On Sun, Aug 21, 2016 at 5:24 PM, Jon Ribbens wrote:
> On 2016-08-19, Larry Martell wrote:
>> fd.write(request.POST[key])
>
> You could try:
>
> request.encoding = "iso-8859-1"
> fd.write(request.POST[key].encode("iso-8859-1"))
>
> It's hacky and nasty and there might be a better "official" method
> but I think it should work.
For some reason that messes up the request structure:
(Pdb) type(request.POST[key])
(Pdb) request.encoding = "iso-8859-1"
(Pdb) type(request.POST[key])
*** MultiValueDictKeyError:
"u'right-carotidartery:63B2E474-D690-445F-B92A-31EBADDC9D93.png'"
--
https://mail.python.org/mailman/listinfo/python-list
Re: saving octet-stream png file
On Fri, Aug 19, 2016 at 4:24 PM, Chris Kaynor wrote:
> On Fri, Aug 19, 2016 at 12:00 PM, Larry Martell
> wrote:
>
>> On Fri, Aug 19, 2016 at 1:24 PM, Chris Angelico wrote:
>> > On Sat, Aug 20, 2016 at 3:10 AM, Larry Martell
>> wrote:
>> >> I have some python code (part of a django app) that processes a
>> >> request that contains a png file. The request is send with
>> >> content_type = 'application/octet-stream'
>> >>
>> >> In the python code I want to write this data to a file and still have
>> >> it still be a valid png file.
>> >>
>> >> The data I get looks like this:
>> >>
>> >> u'\ufffdPNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01\ufffd\
>> x00\x00\x01\ufffd
>> >> ..'
>> >>
>> >> If I try and write that to a file it fails with a UnicodeEncodeError.
>> >> If I write it with encode('utf8') it writes the file, but then it's no
>> >> longer a valid png file.
>> >>
>> >> Anyone know how I can do this?
>> >
>> > At that point, you've already lost information. Each U+FFFD (shown as
>> > "\ufffd" above) is a marker saying "a byte here was not valid UTF-8"
>> > (or whatever was being used). Something somewhere took the .png file's
>> > bytes and tried to interpret them as text, which they're not.
>> >
>> > What sent you that data? How did you receive it?
>>
>> The request is sent by a client app written in C++ with Qt. It's
>> received by a django based server. I am trying to port a falcon server
>> to django. The falcon server code did this:
>>
>> form = cgi.FieldStorage(fp=req.stream, environ=req.env)
>>
>> and then wrote the png like this:
>>
>> fd.write(form[key].file.read())
>>
>> Whereas in the django server I am doing:
>>
>> fd.write(request.POST[key])
>>
>> I've never used the cgi module. I guess I can try that. I've written a
>> lot with django but never had to receive a PNG file.
>>
>>
> I don't know Django, however a quick search makes it seem like you might
> need to use request.FILES[key] (1) rather than request.POST[key]. You may
> also be able to use request.POST if you set request.encoding first (2). If
> both of those fail, you may need to use request.body and parse the HTTP
> form data manually, though I'd imagine there is an easier way.
>
> [1]
> https://docs.djangoproject.com/en/1.10/ref/request-response/#django.http.HttpRequest.FILES
>
> [2]
> https://docs.djangoproject.com/en/1.10/ref/request-response/#django.http.HttpRequest.encoding
Thanks for the reply. When I get the request, request.FILES is empty.
Yet the content type is multipart/form-data and the method is POST:
(Pdb) print request.META['CONTENT_TYPE']
multipart/form-data;
boundary="boundary_.oOo._NzEwNjIzMTM4MTI4NjUxOTM5OQ==MTY2NjE4MDk5Nw=="
(Pdb) print request.META['REQUEST_METHOD']
POST
(Pdb) print request.FILES
Tried setting request.encoding, but that messes up the request structure:
(Pdb) type(request.POST[key])
(Pdb) request.encoding = "iso-8859-1"
(Pdb) type(request.POST[key])
*** MultiValueDictKeyError:
"u'right-carotidartery:63B2E474-D690-445F-B92A-31EBADDC9D93.png'"
--
https://mail.python.org/mailman/listinfo/python-list
Re: PEP suggestion: Uniform way to indicate Python language version
On Tue, Aug 23, 2016 at 12:29 AM, Random832 wrote: > On Mon, Aug 22, 2016, at 09:21, Steve D'Aprano wrote: >> Rather, you just use the features you rely on, document the minimum >> supported version, and if somebody is silly enough to try running your >> code >> under Python 1.4, they'll get a SyntaxError or an exception when you try >> to >> do something that is not supported. > > Receiving a SyntaxError or whatever other exception, which provides no > suggestion about how to actually fix the issue (install a later version > of python / run with "python3" instead of "python"), is a bad user > experience. It will continue to be a bad user experience when people are > using features that only work on python 5.0 and later and other people > are trying to run their scripts under python 4.0, so it not having > existed all along is not a sufficient justification to not consider > adding it,. How bad, exactly? If you installed the program using a package manager (apt, yum, brew, etc etc etc), there should be versioning directives in that, just the same as if there's some particular feature of libc or openssl or libjpeg that you need. And if you go to someone's web site to download it, you should get told "This requires Python X.Y". Sometimes, when there's a really common failure, you can toss in a try/except, or even just a comment: import tkinter # ImportError? You may need Python 3 which will then show up if something goes wrong. There are much, MUCH better ways of solving this than trying to magically make older Pythons, uhh uhh do what, exactly? Nobody's yet said what ought to happen instead of SyntaxError/ImportError. Should the process be terminated with an error message? Should a different exception be raised (InsufficientVersionError) and a check added to the default error handler so it displays differently (the way SystemExit does, or rather doesn't display at all)? And above all, how is all this an improvement over the status quo, which is either a SyntaxError/ImportError straight away, or the ability to put a straight-forward run-time check that does anything you like? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)
On 2016-08-23 00:21, Ben Finney wrote: > So yes, filenames from arbitrary sources should be *completely* > untrusted, and never used to access any file on the system. Throw > the entire filename away and make a filename locally, without using > any part of the original name. Sadly, this ideal advice too often conflicts with the shoddy Code Other People Wrote In Our "Enterprise" System™. :-/ -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)
On Tue, Aug 23, 2016 at 12:34 AM, Random832 wrote:
> On Mon, Aug 22, 2016, at 10:21, Ben Finney wrote:
>> So yes, filenames from arbitrary sources should be *completely*
>> untrusted, and never used to access any file on the system. Throw the
>> entire filename away and make a filename locally, without using any part
>> of the original name.
>
> To be fair, this particular case is unique in presenting a possibility
> to cause problems even for a filename that consists only of whitelisted
> characters (for a reasonable-sounding whitelist such as "ASCII letters
> and numbers and underscore only; all other characters to be scrubbed and
> replaced with {underscore, hex escape, nothing}"). I don't think there's
> any other precedent.
Windows has some other issues, including that arbitrary files can
become executable very easily (eg if %PATHEXT% includes its file
extension), and since the current directory is always at the beginning
of your path, this can easily turn into a remote code execution
exploit. And any GUI that automatically calculates thumbnails from
image files (this includes Windows, Mac OS, and more than one Linux
window manager) could potentially be attacked via a malformed file,
simply by having it appear on the file system. So the idea that some
file names are dangerous is far FAR broader than "a file called
prn.txt will get saved to the printer".
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)
On Tue, Aug 23, 2016 at 12:21 AM, Ben Finney wrote: > > So yes, filenames from arbitrary sources should be *completely* > untrusted, and never used to access any file on the system. Throw the > entire filename away and make a filename locally, without using any part > of the original name. Oh, and I wish you could convince some other parts of the world about this. When you mix file uploads with Apache+PHP web applications, you basically get remote code execution right there. As sysadmin, I have to constantly play whack-a-mole with stupid exploits that just wouldn't happen if Joomla's web uploads followed this model. Seriously, how hard is it for something that *already has a database* to simply insert a row into jos_uploaded_files and then use that row's ID eg "uploads/file_"+id as the file name? In one stroke, you eliminate code execution (it can't be ".php" or any variant thereof), collisions (two uploads with the same file name will get different IDs), and even detritus from incomplete transactions (if you find "uploads/file_12345" but there's no row with ID 12345, you can assume the transaction got rolled back, and safely delete the file). S'not that hard, folks. Be safe. Be smart. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On Mon, Aug 22, 2016 at 1:17 PM, Chris Angelico wrote:
> I tried things like "con.txt" and it simply failed (no such file or
> directory), without printing anything to the console.
Are you using IDLE or some other IDE that uses pythonw.exe instead of
python.exe? If so, first use ctypes to allocate a console:
import ctypes
ctypes.WinDLL('kernel32').AllocConsole()
The CON device should work if the process is attached to a console
(i.e. a conhost.exe instance).
You can detect any of the classic DOS devices via isatty(), since
they're all character devices. Below I'm using Windows 10, so the
console uses a kernel device (condrv.sys was added in Windows 8), for
which the name can be queried:
>>> f = open('con.txt')
>>> h = msvcrt.get_osfhandle(f.fileno())
>>> devname = UNICODE_STRING()
>>> ctypes.resize(devname, sizeof(devname) + 32767*2)
>>> ntdll.NtQueryObject(h, 1, byref(devname), sizeof(devname))
0
>>> print(devname.Buffer)
\Device\ConDrv
> I wouldn't accept file names from untrusted sources on *any* system
There are still desktop applications that ask users to name their files.
--
https://mail.python.org/mailman/listinfo/python-list
Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)
On Mon, Aug 22, 2016, at 11:40, Chris Angelico wrote: > Windows has some other issues, including that arbitrary files can > become executable very easily (eg if %PATHEXT% includes its file > extension), and since the current directory is always at the beginning > of your path, this can easily turn into a remote code execution > exploit. I didn't include dot in my example whitelist, and there's no mechanism for an attacker to add random extensions to your PATHEXT. > And any GUI that automatically calculates thumbnails from > image files (this includes Windows, Mac OS, and more than one Linux > window manager) could potentially be attacked via a malformed file, > simply by having it appear on the file system. This has nothing to do with the filename, unless you additionally assume that this will only happen if the file is called .jpg -- https://mail.python.org/mailman/listinfo/python-list
Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman
Random832 : > On Mon, Aug 22, 2016, at 11:40, Chris Angelico wrote: >> Windows has some other issues, including that arbitrary files can >> become executable very easily (eg if %PATHEXT% includes its file >> extension), and since the current directory is always at the >> beginning of your path, this can easily turn into a remote code >> execution exploit. > > I didn't include dot in my example whitelist, and there's no mechanism > for an attacker to add random extensions to your PATHEXT. Years back, my FTP server was hacked by exploiting a buffer overflow. The anonymous input directory contained a very long filename that apparently contained some valid x86 code. Did you vet your whitelist so it couldn't possibly be interpreted by the CPU as meaningful instructions? Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On Tue, Aug 23, 2016 at 1:54 AM, eryk sun wrote:
> On Mon, Aug 22, 2016 at 1:17 PM, Chris Angelico wrote:
>> I tried things like "con.txt" and it simply failed (no such file or
>> directory), without printing anything to the console.
>
> Are you using IDLE or some other IDE that uses pythonw.exe instead of
> python.exe? If so, first use ctypes to allocate a console:
>
> import ctypes
> ctypes.WinDLL('kernel32').AllocConsole()
>
> The CON device should work if the process is attached to a console
> (i.e. a conhost.exe instance).
No, I used Pike (to avoid any specifically-Python issues or
protections) running in a console. Attempting to write to "Logs/con"
wrote to the console, so I know the console device is active.
Attempting to write to "Logs/con.txt" failed as described.
> > I wouldn't accept file names from untrusted sources on *any* system
>
> There are still desktop applications that ask users to name their files.
A person running a desktop application is generally considered a
trusted source. In a kiosk environment, you have a lot more to worry
about than special device names (eg someone could overwrite a key
file), so again, allowing an untrusted user to name a file in that
situation would be inappropriate. If the user owns the computer, s/he
should be allowed to attempt any name, and there'd simply be some that
fail - same as any other invalid characters (Windows won't let you put
a colon or question mark in a file name, for instance, which annoys my
brother no end when I give him files like "What's Up, Doc?.mkv" or
"Operation: Rabbit.mkv") or over-long names or anything like that.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)
On Tue, Aug 23, 2016 at 1:56 AM, Random832 wrote: >> And any GUI that automatically calculates thumbnails from >> image files (this includes Windows, Mac OS, and more than one Linux >> window manager) could potentially be attacked via a malformed file, >> simply by having it appear on the file system. > > This has nothing to do with the filename, unless you additionally assume > that this will only happen if the file is called .jpg It generally will (or rather, only if the file has one of a particular set of extensions). Automatic thumbnailing is usually done only for certain file names. I don't know of anything that opens every single file to see if it has a JFIF signature (etc for PNG and whatever other types). ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)
On Mon, Aug 22, 2016 at 3:40 PM, Chris Angelico wrote: > Windows has some other issues, including that arbitrary files can > become executable very easily (eg if %PATHEXT% includes its file > extension), cmd uses PATHEXT to augment its search by appending each extension in the list, in addition to searching for the exact filename. cmd will always attempt to run any match, regardless of the extension. You must be thinking of PowerShell, which for some reason reinterprets how this environment variable has worked since probably OS/2 in the late 80s. PowerShell only executes files found in PATH that have an extension that's listed in PATHEXT. CreateProcess checks the user's execute access in the file security, which can prevent the execution of .BAT/.CMD files and PE executables, regardless of extension. But ShellExecute(Ex) has an MS-DOS brain (so much of the entire Explorer/shell32 implementation has an MS-DOS brain; it's like they think they're still supporting Windows 9x), so scripts and data files are always 'executable'. You get some help here from cmd, which always tries CreateProcess, regardless of extension, and won't continue to ShellExecuteEx if CreateProcess failed because access was denied. PowerShell... not so much. > and since the current directory is always at the beginning > of your path, this can easily turn into a remote code execution > exploit. Since Vista, both CreateProcess and cmd.exe support the environment variable NoDefaultCurrentDirectoryInExePath. If this is set, you have to explicitly reference the current directory. PowerShell always required this. -- https://mail.python.org/mailman/listinfo/python-list
Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman
On Tue, Aug 23, 2016 at 2:08 AM, Marko Rauhamaa wrote: > Random832 : > >> On Mon, Aug 22, 2016, at 11:40, Chris Angelico wrote: >>> Windows has some other issues, including that arbitrary files can >>> become executable very easily (eg if %PATHEXT% includes its file >>> extension), and since the current directory is always at the >>> beginning of your path, this can easily turn into a remote code >>> execution exploit. >> >> I didn't include dot in my example whitelist, and there's no mechanism >> for an attacker to add random extensions to your PATHEXT. > > Years back, my FTP server was hacked by exploiting a buffer overflow. > The anonymous input directory contained a very long filename that > apparently contained some valid x86 code. > > Did you vet your whitelist so it couldn't possibly be interpreted by the > CPU as meaningful instructions? Step 1: Don't have buffers. Step 2: Profit! Anyone who's using fixed-sized buffers for application-level code deserves to be exploited. A program designed to be accessed via the internet is never (well, hardly ever) going to need so much performance that it can't afford to be written in a high level language - it's going to spend most of its time waiting for the network. The rare exceptions (*maybe* DNS, but even there, I'd be quite happy to replace my DNS server with one written in Pike, if BIND9 ever becomes a major threat vector) should be monitored closely - preferably statically checked with something like Coverity - because they're remotely-accessible and thus a major risk. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: saving octet-stream png file
On Mon, Aug 22, 2016 at 10:36 AM, Larry Martell wrote:
> On Fri, Aug 19, 2016 at 4:24 PM, Chris Kaynor
> wrote:
>> On Fri, Aug 19, 2016 at 12:00 PM, Larry Martell
>> wrote:
>>
>>> On Fri, Aug 19, 2016 at 1:24 PM, Chris Angelico wrote:
>>> > On Sat, Aug 20, 2016 at 3:10 AM, Larry Martell
>>> wrote:
>>> >> I have some python code (part of a django app) that processes a
>>> >> request that contains a png file. The request is send with
>>> >> content_type = 'application/octet-stream'
>>> >>
>>> >> In the python code I want to write this data to a file and still have
>>> >> it still be a valid png file.
>>> >>
>>> >> The data I get looks like this:
>>> >>
>>> >> u'\ufffdPNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01\ufffd\
>>> x00\x00\x01\ufffd
>>> >> ..'
>>> >>
>>> >> If I try and write that to a file it fails with a UnicodeEncodeError.
>>> >> If I write it with encode('utf8') it writes the file, but then it's no
>>> >> longer a valid png file.
>>> >>
>>> >> Anyone know how I can do this?
>>> >
>>> > At that point, you've already lost information. Each U+FFFD (shown as
>>> > "\ufffd" above) is a marker saying "a byte here was not valid UTF-8"
>>> > (or whatever was being used). Something somewhere took the .png file's
>>> > bytes and tried to interpret them as text, which they're not.
>>> >
>>> > What sent you that data? How did you receive it?
>>>
>>> The request is sent by a client app written in C++ with Qt. It's
>>> received by a django based server. I am trying to port a falcon server
>>> to django. The falcon server code did this:
>>>
>>> form = cgi.FieldStorage(fp=req.stream, environ=req.env)
>>>
>>> and then wrote the png like this:
>>>
>>> fd.write(form[key].file.read())
>>>
>>> Whereas in the django server I am doing:
>>>
>>> fd.write(request.POST[key])
>>>
>>> I've never used the cgi module. I guess I can try that. I've written a
>>> lot with django but never had to receive a PNG file.
>>>
>>>
>> I don't know Django, however a quick search makes it seem like you might
>> need to use request.FILES[key] (1) rather than request.POST[key]. You may
>> also be able to use request.POST if you set request.encoding first (2). If
>> both of those fail, you may need to use request.body and parse the HTTP
>> form data manually, though I'd imagine there is an easier way.
>>
>> [1]
>> https://docs.djangoproject.com/en/1.10/ref/request-response/#django.http.HttpRequest.FILES
>>
>> [2]
>> https://docs.djangoproject.com/en/1.10/ref/request-response/#django.http.HttpRequest.encoding
>
> Thanks for the reply. When I get the request, request.FILES is empty.
> Yet the content type is multipart/form-data and the method is POST:
>
> (Pdb) print request.META['CONTENT_TYPE']
> multipart/form-data;
> boundary="boundary_.oOo._NzEwNjIzMTM4MTI4NjUxOTM5OQ==MTY2NjE4MDk5Nw=="
>
> (Pdb) print request.META['REQUEST_METHOD']
> POST
>
> (Pdb) print request.FILES
>
>
> Tried setting request.encoding, but that messes up the request structure:
>
> (Pdb) type(request.POST[key])
>
> (Pdb) request.encoding = "iso-8859-1"
> (Pdb) type(request.POST[key])
> *** MultiValueDictKeyError:
> "u'right-carotidartery:63B2E474-D690-445F-B92A-31EBADDC9D93.png'"
Thanks to everyone for they replied. I solved this by changing the
client to set the file and filename fields each part of the multipart
and then I was able it iterate through request.FILES and successfully
write the files as PNG.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On Mon, Aug 22, 2016 at 4:18 PM, Chris Angelico wrote:
>
>> The CON device should work if the process is attached to a console
>> (i.e. a conhost.exe instance).
>
> No, I used Pike (to avoid any specifically-Python issues or
> protections) running in a console. Attempting to write to "Logs/con"
> wrote to the console, so I know the console device is active.
> Attempting to write to "Logs/con.txt" failed as described.
What version of Windows is this? If it's Windows 7 I'll have to check
that later. If "Logs" is an existing directory, then both "Logs/con"
and "Logs/con.txt" should refer to the console. If "Logs" doesn't
exist, then both should fail. Virtual DOS devices only exist in
existing directories.
>> > I wouldn't accept file names from untrusted sources on *any* system
>>
>> There are still desktop applications that ask users to name their files.
> If the user owns the computer, s/he should be allowed to attempt any
> name, and there'd simply be some that fail - same as any other invalid
> characters
Silently writing a file to a (possibly hidden) console or NUL device
is not the same thing as failing with an error. If users explicitly
open "NUL" or "\\.\NUL", I can detect that, and I have no problem with
it (with some reservations about the former). But if they open files
like "C:\Users\JoeUser\Documents\Nul.20160822.doc", I want to make
sure they know that they just asked to save to "\\.\NUL". It's not a
common problem. I just find the system's behavior abhorrent. I'd like
to have a manifest setting that opts the process out of this stupid
DOS legacy behavior.
> Windows won't let you put a colon or question mark in a file name, for
> instance,
> which annoys my brother no end when I give him files like
> "What's Up, Doc?.mkv" or "Operation: Rabbit.mkv")
A colon is reserved for NTFS streams, so it can silently bite you. If
you get an error for "Operation: Rabbit.mkv", you're probably using
FAT32. On NTFS that creates a file named "Operation" with a $DATA
stream named " Rabbit.mkv".
Every NTFS file has at least the anonymous stream, e.g.
"filename::$DATA". Every NTFS directory has at least the $I30 stream,
e.g. "dirname:$I30:$INDEX_ALLOCATION". Directories can also have named
$DATA streams.
As to question mark and other wildcard characters, those are all
unambiguously handled as exceptions. No Microsoft filesystem allows
them because wildcard matching is baked into the system, such as the
FileName parameter of NtQueryDirectoryFile when listing the contents
of a directory. I guess no one on the NT I/O team wanted to bother
with the possibility of escaping these wildcards since they weren't
allowed in DOS filenames.
FsRtlIsNameInExpression [1] is the kernel-mode filesystem function
that implements wildcard matching. Note that DOS wildcard semantics
are implemented by DOS_STAR (<), DOS_QM (>), and DOS_DOT (").
FindFirstFile has to convert DOS wildcard patterns to NT semantics
before calling NtQueryDirectoryFile. It's not as simple as
substituting the corresponding DOS_ wildcard.
Pipe and ASCII control characters are also reserved, except not in
NTFS stream names. Reserving pipe is just a DOS legacy. I don't think
it's used for anything internally.
> or over-long names or anything like that.
Good news. In Windows 10, systems and applications can opt in to using
long paths with up to about 32760 characters. Python 3.6 will be
opting in.
[1]: https://msdn.microsoft.com/en-us/library/ff546850
--
https://mail.python.org/mailman/listinfo/python-list
Re: saving octet-stream png file
On Fri, Aug 19, 2016 at 4:51 PM, Lawrence D’Oliveiro wrote: > On Saturday, August 20, 2016 at 6:03:53 AM UTC+12, Terry Reedy wrote: >> >> An 'octet' is a byte of 8 bits. > > Is there any other size of byte? Many, many years ago, probably c. 1982 my Dad came into my house and saw a Byte Magazine laying on the coffee table. He asked "What is a byte?" I replied "Half a word." He then asked "What is the other half of the word?" I said "That is also a byte." He thought for a moment, then said "So the full word is 'byte byte'?" -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On Tue, Aug 23, 2016 at 3:13 AM, eryk sun wrote: > On Mon, Aug 22, 2016 at 4:18 PM, Chris Angelico wrote: >> >>> The CON device should work if the process is attached to a console >>> (i.e. a conhost.exe instance). >> >> No, I used Pike (to avoid any specifically-Python issues or >> protections) running in a console. Attempting to write to "Logs/con" >> wrote to the console, so I know the console device is active. >> Attempting to write to "Logs/con.txt" failed as described. > > What version of Windows is this? If it's Windows 7 I'll have to check > that later. If "Logs" is an existing directory, then both "Logs/con" > and "Logs/con.txt" should refer to the console. If "Logs" doesn't > exist, then both should fail. Virtual DOS devices only exist in > existing directories. Yes, it was Windows 7 (running in a VM under Debian Jessie, though I doubt that makes any difference). The Logs directory did exist (that's why I used that otherwise-odd choice of name). >> Windows won't let you put a colon or question mark in a file name, for >> instance, >> which annoys my brother no end when I give him files like >> "What's Up, Doc?.mkv" or "Operation: Rabbit.mkv") > > A colon is reserved for NTFS streams, so it can silently bite you. If > you get an error for "Operation: Rabbit.mkv", you're probably using > FAT32. On NTFS that creates a file named "Operation" with a $DATA > stream named " Rabbit.mkv". I think it was FAT32 that we were using - it was a USB stick for sharing files between my Linux systems and his Windows. Linux was quite happy to put files into that FS that Windows refused to touch. (Also, due to some renamings, I ended up giving him two copies of a file that differed only in case - something like "Name The File Something.mkv" and "Name the File Something.mkv" - and Windows took issue with that too.) > As to question mark and other wildcard characters, those are all > unambiguously handled as exceptions. No Microsoft filesystem allows > them because wildcard matching is baked into the system, such as the > FileName parameter of NtQueryDirectoryFile when listing the contents > of a directory. I guess no one on the NT I/O team wanted to bother > with the possibility of escaping these wildcards since they weren't > allowed in DOS filenames. Yeah. They're nice straight-forward errors, as long as you're running on Windows. Detecting that on POSIX platforms and avoiding those names is why we have PureWindowsPath. If it doesn't currently have a quick little "is_valid", it could possibly benefit from one - PurePosixPath would simply "return '\0' not in name", while PureWindowsPath can do the more extensive checks with wildcard characters and stuff. >> or over-long names or anything like that. > > Good news. In Windows 10, systems and applications can opt in to using > long paths with up to about 32760 characters. Python 3.6 will be > opting in. > > [1]: https://msdn.microsoft.com/en-us/library/ff546850 Great. So just as soon as all previous Windowses die... Remind me how many people are still using Python 3.4 because they're still on Windows XP? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On 2016-08-22, Chris Angelico wrote: > I tried things like "con.txt" and it simply failed (no such file or > directory), without printing anything to the console. I'm not sure how you got that to fail, but writing to "con.txt" certainly does write to the console in Windows 10 - I just tried it: C:\>echo hello >con.txt hello -- https://mail.python.org/mailman/listinfo/python-list
Re: saving octet-stream png file
On 2016-08-22, Larry Martell wrote:
> On Sun, Aug 21, 2016 at 5:24 PM, Jon Ribbens
> wrote:
>> On 2016-08-19, Larry Martell wrote:
>>> fd.write(request.POST[key])
>>
>> You could try:
>>
>> request.encoding = "iso-8859-1"
>> fd.write(request.POST[key].encode("iso-8859-1"))
>>
>> It's hacky and nasty and there might be a better "official" method
>> but I think it should work.
>
> For some reason that messes up the request structure:
>
> (Pdb) type(request.POST[key])
>
> (Pdb) request.encoding = "iso-8859-1"
> (Pdb) type(request.POST[key])
> *** MultiValueDictKeyError:
> "u'right-carotidartery:63B2E474-D690-445F-B92A-31EBADDC9D93.png'"
Sounds like you should be filing a bug report with Django.
--
https://mail.python.org/mailman/listinfo/python-list
Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)
On 2016-08-23 02:20, Chris Angelico wrote: > It generally will (or rather, only if the file has one of a > particular set of extensions). Automatic thumbnailing is usually > done only for certain file names. I don't know of anything that > opens every single file to see if it has a JFIF signature (etc for > PNG and whatever other types). How about a web server that opens arbitrary files. Compare any of https://technet.microsoft.com/en-us/library/nonexistent.aspx https://technet.microsoft.com/en-us/library/doesnotexist.aspx https://technet.microsoft.com/en-us/library/asdf.aspx vs https://technet.microsoft.com/en-us/library/con.aspx https://technet.microsoft.com/en-us/library/lpt1.aspx https://technet.microsoft.com/en-us/library/com1.aspx https://technet.microsoft.com/en-us/library/nul.aspx This is FREAKING MICROSOFT and it breaks things. It's not like anybody would open arbitrarily-named files... -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On 2016-08-22 22:39, Chris Angelico wrote: > Nope. On Windows, you would try/except it. There are myriad other > ways something could fail, and the only correct action is to > attempt it. Most of the reserved names will simply give an error; The problem is that when opening such a pseudo-file, you can get unexpected behavior. In the Unix world, we're used to files-that-aren-t-files (such as things in /dev ). But a lot of Windows developers don't handle these cases, and so opening something like COM1 can end up hanging a program indefinitely instead of actually returning either an error or a file-handle. If you have a web-server running on Windows and can manage to coerce a file to have such a name, you might be able to hang the web-server process while it tries to read from (or write to) the serial port. And poof, near-instant DoS. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On 2016-08-22, Steve D'Aprano wrote:
> On Mon, 22 Aug 2016 09:50 pm, Jon Ribbens wrote:
>> I don't know what purpose you are envisaging this function being used
>> for, but the only one I can think of is input sanitisation. e.g. a web
>> form where you receive a file from the Internet and store it somewhere,
>> and you want to use the filename given to you rather than choose your
>> own randomly-generated one.
>>
>> Under Unix all you need to do is check for the filename starting with
>> "." or containing "/." (or "/", depending on your requirements).
>> Under Windows you would use this function, which apparently doesn't
>> work, hence: security hole.
>
> That's backwards: it works under Windows, but is buggy under non-Windows.
I'm not sure what you're getting at there, but you appear to be wrong:
C:\>python
Python 3.5.1 (v3.5.1:37a07cee5969, Dec 6 2015, 01:38:48) [MSC v.1900
32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pathlib
>>> pathlib.WindowsPath("con .txt").is_reserved()
False
> For starters, even on Unix I probably wouldn't want to
> use "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\.txt" as a
> file name, even if it is technically legal.
That might be your personal preference, but it's not a security hole
per se.
> No, not at all. I can't assume that just because a file name is legal that I
> can write to it. Even after sanitising the name (or deciding I don't need
> to bother) I still need to be prepared to catch I/O errors when writing the
> file -- including attempts to write to reserved file names.
As mentioned by others, this is nothing to do with lack of error-checking.
> Technically that could be called a DoS attack, but the same could be said
> for ANY bug that raises an exception. I think that it takes more than that
> to be a security bug.
It doesn't raise an exception. It interacts with the hardware of the
server.
--
https://mail.python.org/mailman/listinfo/python-list
Re: The dangerous, exquisite art of safely handing user-uploaded files: Tom Eastman (was: Does This Scare You?)
On Tue, Aug 23, 2016 at 3:32 AM, Tim Chase wrote: > On 2016-08-23 02:20, Chris Angelico wrote: >> It generally will (or rather, only if the file has one of a >> particular set of extensions). Automatic thumbnailing is usually >> done only for certain file names. I don't know of anything that >> opens every single file to see if it has a JFIF signature (etc for >> PNG and whatever other types). > > How about a web server that opens arbitrary files. Compare any of > > https://technet.microsoft.com/en-us/library/nonexistent.aspx > https://technet.microsoft.com/en-us/library/doesnotexist.aspx > https://technet.microsoft.com/en-us/library/asdf.aspx > > vs > > https://technet.microsoft.com/en-us/library/con.aspx > https://technet.microsoft.com/en-us/library/lpt1.aspx > https://technet.microsoft.com/en-us/library/com1.aspx > https://technet.microsoft.com/en-us/library/nul.aspx > > This is FREAKING MICROSOFT and it breaks things. It's not like > anybody would open arbitrarily-named files... Oh, brilliant. Brilliant brilliant brilliant. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
DED processing
I have to go out for a while, so for DED processing two options from my end: 1. Process as you all have been in the past for now. If you all do this, the records that have not been mailed prior to the latest list are contained in a MailManage Job name DED_master. If you chose to process as in the past, these records should be exported to a spreadsheet and added to whatever new file you have for today. 2. I can come in tomorrow morning at about 9:00 and walk Greg thru the way I process the DED job. Let me know. -- https://mail.python.org/mailman/listinfo/python-list
Re: DED processing
On Mon, Aug 22, 2016 at 1:52 PM, Gary Sublett wrote: > I have to go out for a while, so for DED processing two options from > my end: > > 1. Process as you all have been in the past for now. If you all do > this, the records that have not been mailed prior to the latest list > are contained in a MailManage Job name DED_master. If you chose to > process as in the past, these records should be exported to a > spreadsheet and added to whatever new file you have for today. > > 2. I can come in tomorrow morning at about 9:00 and walk Greg thru > the way I process the DED job. > > Let me know. I think you should let Greg know. What if you get hit by a bus, or win the lottery? -- https://mail.python.org/mailman/listinfo/python-list
Re: Something wrong with the PIP, lots of friend have the same problem
even though I try to upgrade PIP[cid:beb9e7c5-7d98-4506-ba67-9e67890edec1] From: Wentao Liang Sent: Sunday, 21 August 2016 4:35:31 PM To: python-list@python.org Subject: Something wrong with the PIP, lots of friend have the same problem [cid:1c20248f-a678-40c0-9563-da4f8a1ef3bd] Is it the problem from PIP itself, or I used it in a wrong way? [😊] Looking forwards to your reply! Thank you Wentao Liang -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On Tue, 23 Aug 2016 03:13 am, eryk sun wrote: > But if they open files > like "C:\Users\JoeUser\Documents\Nul.20160822.doc", I want to make > sure they know that they just asked to save to "\\.\NUL". It's not a > common problem. I just find the system's behavior abhorrent. I'd like > to have a manifest setting that opts the process out of this stupid > DOS legacy behavior. Aren't we getting further and further away from the original topic? I don't think this is about Window's support for legacy behaviour from ancient DOS days. Yes, it would be nice if Windows dropped support for these horrid magic file names, but that's nothing to do with Python. I'm not really sure what the question is -- we've established that there's a bug in the non-Windows implementation that tries to emulate Window's behaviour. What else is there to argue about? - Does anyone wish to argue that Python shouldn't provide PureWindowsPath.is_reserved on non-Windows systems? For what reason? - Is anyone still arguing that there's a new security vulnerability here because of the pathlib functions? If so, how do you see this attack working? (Existing filename-based attacks are not new.) I don't see what the issue is. Eryksun found a bug in pathlib, well done. (I mean that, I'm not being sarcastic.) I still don't understand why Lawrence posed his question in the first place. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On 2016-08-22, Steve D'Aprano wrote: > I'm not really sure what the question is -- we've established that there's a > bug in the non-Windows implementation that tries to emulate Window's > behaviour. What else is there to argue about? It doesn't seem to be "the non-Windows implementation", it seems to be "the implementation". > - Does anyone wish to argue that Python shouldn't provide > PureWindowsPath.is_reserved on non-Windows systems? For what reason? It shouldn't provide it at all unless it works. The reason I'm putting it that way is that making it work may be a very great deal of effort (What is the actual full list of special filenames? Does it vary between supported Windows versions? How can anyone tell test that the function is correct?) > - Is anyone still arguing that there's a new security vulnerability > here because of the pathlib functions? If so, how do you see this > attack working? (Existing filename-based attacks are not new.) Already answered. > I don't see what the issue is. Eryksun found a bug in pathlib, well done. (I > mean that, I'm not being sarcastic.) I still don't understand why Lawrence > posed his question in the first place. Presumably because of the security implications as described. -- https://mail.python.org/mailman/listinfo/python-list
Re: DED processing
On Tue, Aug 23, 2016 at 3:58 AM, Larry Martell wrote: > On Mon, Aug 22, 2016 at 1:52 PM, Gary Sublett wrote: >> I have to go out for a while, so for DED processing two options from >> my end: >> >> 1. Process as you all have been in the past for now. If you all do >> this, the records that have not been mailed prior to the latest list >> are contained in a MailManage Job name DED_master. If you chose to >> process as in the past, these records should be exported to a >> spreadsheet and added to whatever new file you have for today. >> >> 2. I can come in tomorrow morning at about 9:00 and walk Greg thru >> the way I process the DED job. >> >> Let me know. > > I think you should let Greg know. What if you get hit by a bus, or win > the lottery? DED Processing Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited; please print this email out on minimum 100 GSM paper, cross-cut shred it, burn the remains, and send the ashes back to the sender. Failure to do so will result in the internet police being informed of your IP address, street address, ICBM address, and t-shirt size. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: ANN: pygene - genetic algorithms package
On Tuesday, December 6, 2005 at 11:10:56 AM UTC+3:30, aum wrote: > Hi all, > > I looked at a few genetic algorithms/genetic programming packages for > Python, and found them somewhat convoluted, complicated and > counter-intuitive to use. > > So I've written a genetic algorithms package which I hope will be more > approachable to beginners. > > The first release of pygene is up at: > http://www.freenet.org.nz/python/pygene > > The package includes full api documentation, and an implementation of > the travelling salesman problem, plus a couple of simpler cases. > > -- > > Cheers > aum -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On Mon, 22 Aug 2016 17:27:13 +, Jon Ribbens wrote: > On 2016-08-22, Chris Angelico wrote: >> I tried things like "con.txt" and it simply failed (no such file or >> directory), without printing anything to the console. > > I'm not sure how you got that to fail, but writing to "con.txt" > certainly does write to the console in Windows 10 - I just tried it: > > C:\>echo hello >con.txt > hello I got the same result with Windows 7, Vista, XP, 2000, NT4, 98SE and 3.11. Yes, I have a lot VM's although I don't have 8.x. I would, however, expect the same result. On Linux a file is created named con.txt that contains hello/n as expected. -- GNU/Linux user #557453 "The Constitution only gives people the right to pursue happiness. You have to catch it yourself." -Benjamin Franklin -- https://mail.python.org/mailman/listinfo/python-list
Re: saving octet-stream png file
On Mon, 22 Aug 2016 13:21:43 -0400, Larry Martell wrote: > On Fri, Aug 19, 2016 at 4:51 PM, Lawrence D’Oliveiro > wrote: >> On Saturday, August 20, 2016 at 6:03:53 AM UTC+12, Terry Reedy wrote: >>> >>> An 'octet' is a byte of 8 bits. >> >> Is there any other size of byte? > > Many, many years ago, probably c. 1982 my Dad came into my house and > saw a Byte Magazine laying on the coffee table. He asked "What is a > byte?" I replied "Half a word." He then asked "What is the other half > of the word?" I said "That is also a byte." He thought for a moment, > then said "So the full word is 'byte byte'?" LOL! Did you explain to him that a full word could also be 'nibble nibble nibble nibble' or 'bit bit bit bit bit bit bit bit bit bit bit bit bit bit bit bit'? -- GNU/Linux user #557453 The cow died so I don't need your bull! -- https://mail.python.org/mailman/listinfo/python-list
Dynamically import specific names from a module vs importing full module
Python 3.5: Is there a way to dynamically import specific names from a module vs importing the full module? By dynamic I mean via some form of importlib machinery, eg. I'm looking for the dynamic "from import " equivalent of "import "'s importlib.import_module. Thank you, Malcolm -- https://mail.python.org/mailman/listinfo/python-list
Re: Dynamically import specific names from a module vs importing full module
On Monday, August 22, 2016 at 4:22:09 PM UTC-4, Malcolm Greene wrote:
> Python 3.5: Is there a way to dynamically import specific names from a
> module vs importing the full module?
>
> By dynamic I mean via some form of importlib machinery, eg. I'm looking
> for the dynamic "from import " equivalent of "import
> "'s importlib.import_module.
>
> Thank you,
> Malcolm
You can use:
the_mod = importlib.import_module("module")
a = the_mod.a
b = the_mod.b
# or
a = getattr(the_mod, 'a')
b = getattr(the_mod, 'b')
--Ned.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Dynamically import specific names from a module vs importing full module
On Mon, Aug 22, 2016, at 16:21, Malcolm Greene wrote: > Python 3.5: Is there a way to dynamically import specific names from a > module vs importing the full module? > > By dynamic I mean via some form of importlib machinery, eg. I'm looking > for the dynamic "from import " equivalent of "import > "'s importlib.import_module. You do know that "from import " still loads the whole module into sys.modules, right? And you're free to assign the result from importlib.import_module to a local variable and delete it once you've gotten the names you want out of it. -- https://mail.python.org/mailman/listinfo/python-list
Re: saving octet-stream png file
On Mon, Aug 22, 2016 at 1:25 PM, Jon Ribbens wrote:
> On 2016-08-22, Larry Martell wrote:
>> On Sun, Aug 21, 2016 at 5:24 PM, Jon Ribbens
>> wrote:
>>> On 2016-08-19, Larry Martell wrote:
fd.write(request.POST[key])
>>>
>>> You could try:
>>>
>>> request.encoding = "iso-8859-1"
>>> fd.write(request.POST[key].encode("iso-8859-1"))
>>>
>>> It's hacky and nasty and there might be a better "official" method
>>> but I think it should work.
>>
>> For some reason that messes up the request structure:
>>
>> (Pdb) type(request.POST[key])
>>
>> (Pdb) request.encoding = "iso-8859-1"
>> (Pdb) type(request.POST[key])
>> *** MultiValueDictKeyError:
>> "u'right-carotidartery:63B2E474-D690-445F-B92A-31EBADDC9D93.png'"
>
> Sounds like you should be filing a bug report with Django.
Turns out that is expected behavior -- request.encoding clears
existing GET/POST data. Not sure how it's useful to set that then.
--
https://mail.python.org/mailman/listinfo/python-list
Re: saving octet-stream png file
On 2016-08-22, Larry Martell wrote: > On Mon, Aug 22, 2016 at 1:25 PM, Jon Ribbens > wrote: >> On 2016-08-22, Larry Martell wrote: >>> (Pdb) type(request.POST[key]) >>> >>> (Pdb) request.encoding = "iso-8859-1" >>> (Pdb) type(request.POST[key]) >>> *** MultiValueDictKeyError: >>> "u'right-carotidartery:63B2E474-D690-445F-B92A-31EBADDC9D93.png'" >> >> Sounds like you should be filing a bug report with Django. > > Turns out that is expected behavior -- request.encoding clears > existing GET/POST data. Not sure how it's useful to set that then. Yeah it clears the cached parsed data, but when you fetch a value afterwards it should automatically recalculate the data, not fail. -- https://mail.python.org/mailman/listinfo/python-list
Re: Does This Scare You?
On 08/19/2016 06:11 PM, Wildman via Python-list wrote: Since I am fairly new to Python, I realize there is much that I still don't know but I don't understand how Windows can have reserved names on a Linux system. What am I missing? A PureWindowsPath (and PurePosixPath and PurePath) is a theoretical -- it's still, basically, a string and hasn't been checked against the file system to see if it actually exists (or to create it, etc.). A WindowsPath (PosixPath) does actually have a tie-in to the actual file system, and can only be instantiated on Windows (Posix) systems. -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Raw Data from Website
Hi, I am hoping someone is able to help me. Is there a way to pull as much raw data from a website as possible. The webpage that I am looking for is as follows: http://www.homepriceguide.com.au/Research/ResearchSeeFullList.aspx?LocationType=LGA&State=QLD&LgaID=632 The main variable that is important is the "632" at the end, by adjusting this it changes the postcodes. Each postcode contains a large amount of data. Is there a way this all able to be exported into an excel document? Any help with this would be amazing. Thank you. Adam -- https://mail.python.org/mailman/listinfo/python-list
