Re: Sometimes bottle takes a lot of time
Cecil Westerhof wrote: > I created a simple application with bottle: > https://github.com/CecilWesterhof/PublishedPhotos > > But sometimes it needs a lot of time. For example: > 127.0.0.1 - - [21/Aug/2015 23:16:40] "GET / HTTP/1.1" 304 0 > 127.0.0.1 - - [21/Aug/2015 23:16:40] "GET /static/css/default.css > HTTP/1.1" 304 0 127.0.0.1 - - [21/Aug/2015 23:16:46] "GET > /static/JavaScript/angular.js HTTP/1.1" 304 0 127.0.0.1 - - [21/Aug/2015 > 23:16:46] "GET /static/appPublishedPhotos.js HTTP/1.1" 304 0 127.0.0.1 - - > [21/Aug/2015 23:16:46] "GET /links/data HTTP/1.1" 200 2884 127.0.0.1 - - > [21/Aug/2015 23:16:52] "GET /versionPython HTTP/1.1" 200 5 127.0.0.1 - - > [21/Aug/2015 23:16:52] "GET /versionSQLite HTTP/1.1" 200 5 > > Between default.css and angular.js there are six seconds. And between > /links/data and /versionPytjon is again six seconds. What is happening > here? I don't know. Is it bottle, or the browser, or something completely different that eats the extra time? > Just before everything was done in a second: > 127.0.0.1 - - [21/Aug/2015 23:16:22] "GET / HTTP/1.1" 200 956 > 127.0.0.1 - - [21/Aug/2015 23:16:22] "GET /static/css/default.css > HTTP/1.1" 304 0 127.0.0.1 - - [21/Aug/2015 23:16:22] "GET > /static/appPublishedPhotos.js HTTP/1.1" 304 0 127.0.0.1 - - [21/Aug/2015 > 23:16:22] "GET /static/JavaScript/angular.js HTTP/1.1" 304 0 127.0.0.1 - - > [21/Aug/2015 23:16:23] "GET /versionPython HTTP/1.1" 200 5 127.0.0.1 - - > [21/Aug/2015 23:16:23] "GET /links/data HTTP/1.1" 200 2884 127.0.0.1 - - > [21/Aug/2015 23:16:23] "GET /versionSQLite HTTP/1.1" 200 5 > -- https://mail.python.org/mailman/listinfo/python-list
Re: SQLite3 and web server
Cecil Westerhof wrote: > Thanks. I made a first try: > https://github.com/CecilWesterhof/PublishedPhotos > > The thing I do not like is that all my static files have to be put in > /static/, but I think I can life with it. Is that really required by bottle? Where would you like to put the files? -- https://mail.python.org/mailman/listinfo/python-list
Re: Sometimes bottle takes a lot of time
On 21.08.2015 23:22, Cecil Westerhof wrote: > Just before everything was done in a second: Since you're on GitHub, why don't you git bisect and find out where you screwed up instead of trying to get people to remotely debug and profile your broken code? Cheers, Johannes -- >> Wo hattest Du das Beben nochmal GENAU vorhergesagt? > Zumindest nicht öffentlich! Ah, der neueste und bis heute genialste Streich unsere großen Kosmologen: Die Geheim-Vorhersage. - Karl Kaos über Rüdiger Thomas in dsa -- https://mail.python.org/mailman/listinfo/python-list
Re: Every character of a string becomes a binding
On 21.08.2015 19:04, Cecil Westerhof wrote:
>> Because the execute method expects the bindings to be passed as a
>> sequence,
>
> Yeah, I found that. I solved it a little differently:
> urls = c.execute('SELECT URL FROM links WHERE URL = ?', [url]).fetchall()
You continuously ask more than amateurish questions here and frequently,
after having received dozens of helpful and quick replies you respond
with "yeah, I found that" or similar.
Wouldn't it be advisable to tinker with your code roughly 10 more
minutes the next time you run into an amateur's problem instead of
firing off a Usenet post, oh very wise Senior Software Engineer?
Cheers,
Johannes
--
>> Wo hattest Du das Beben nochmal GENAU vorhergesagt?
> Zumindest nicht öffentlich!
Ah, der neueste und bis heute genialste Streich unsere großen
Kosmologen: Die Geheim-Vorhersage.
- Karl Kaos über Rüdiger Thomas in dsa
--
https://mail.python.org/mailman/listinfo/python-list
Re: Sometimes bottle takes a lot of time
On Saturday 22 Aug 2015 11:41 CEST, Johannes Bauer wrote: > On 21.08.2015 23:22, Cecil Westerhof wrote: > >> Just before everything was done in a second: > > Since you're on GitHub, why don't you git bisect and find out where > you screwed up instead of trying to get people to remotely debug and > profile your broken code? Are you beaten up by your wife that you need to be this harsh? If you would have taken a little more time you would have seen that there where 20 seconds between both logs. I am fast, but not that fast. It is exactly the same code. I suppose it has to do something with bottle. Something I use since yesterday. Is it that strange to look if someone else has had the same problem and maybe a solution? -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: Sometimes bottle takes a lot of time
On Friday 21 Aug 2015 23:55 CEST, gst wrote: > What if you try with all the SQLite code commented ? I do not think that is the problem. First of all I do not think receiving 25 records takes 6 seconds. Secondly the first part is: 127.0.0.1 - - [21/Aug/2015 23:16:40] "GET / HTTP/1.1" 304 0 127.0.0.1 - - [21/Aug/2015 23:16:40] "GET /static/css/default.css HTTP/1.1" 304 0 127.0.0.1 - - [21/Aug/2015 23:16:46] "GET /static/JavaScript/angular.js HTTP/1.1" 304 0 127.0.0.1 - - [21/Aug/2015 23:16:46] "GET /static/appPublishedPhotos.js HTTP/1.1" 304 0 There is no SQLite code at all there. Only Not Modified replies. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: SQLite3 and web server
On Saturday 22 Aug 2015 09:51 CEST, Peter Otten wrote: > Cecil Westerhof wrote: > >> Thanks. I made a first try: >> https://github.com/CecilWesterhof/PublishedPhotos >> >> The thing I do not like is that all my static files have to be put >> in /static/, but I think I can life with it. > > Is that really required by bottle? Where would you like to put the > files? That is how I interpret this: http://bottlepy.org/docs/dev/tutorial.html#routing-static-files -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: Sometimes bottle takes a lot of time
On Saturday 22 Aug 2015 09:49 CEST, Peter Otten wrote: > Cecil Westerhof wrote: > >> I created a simple application with bottle: >> https://github.com/CecilWesterhof/PublishedPhotos >> >> But sometimes it needs a lot of time. For example: 127.0.0.1 - - >> [21/Aug/2015 23:16:40] "GET / HTTP/1.1" 304 0 127.0.0.1 - - >> [21/Aug/2015 23:16:40] "GET /static/css/default.css HTTP/1.1" 304 0 >> 127.0.0.1 - - [21/Aug/2015 23:16:46] "GET >> /static/JavaScript/angular.js HTTP/1.1" 304 0 127.0.0.1 - - >> [21/Aug/2015 23:16:46] "GET /static/appPublishedPhotos.js HTTP/1.1" >> 304 0 127.0.0.1 - - [21/Aug/2015 23:16:46] "GET /links/data >> HTTP/1.1" 200 2884 127.0.0.1 - - [21/Aug/2015 23:16:52] "GET >> /versionPython HTTP/1.1" 200 5 127.0.0.1 - - [21/Aug/2015 23:16:52] >> "GET /versionSQLite HTTP/1.1" 200 5 >> >> Between default.css and angular.js there are six seconds. And >> between /links/data and /versionPytjon is again six seconds. What >> is happening here? > > I don't know. Is it bottle, or the browser, or something completely > different that eats the extra time? I really do not know. I suspect bottle, but I am new to this, so I value the suspicion of someone who has more experience more. :-D >> Just before everything was done in a second: 127.0.0.1 - - >> [21/Aug/2015 23:16:22] "GET / HTTP/1.1" 200 956 127.0.0.1 - - >> [21/Aug/2015 23:16:22] "GET /static/css/default.css HTTP/1.1" 304 0 >> 127.0.0.1 - - [21/Aug/2015 23:16:22] "GET >> /static/appPublishedPhotos.js HTTP/1.1" 304 0 127.0.0.1 - - >> [21/Aug/2015 23:16:22] "GET /static/JavaScript/angular.js HTTP/1.1" >> 304 0 127.0.0.1 - - [21/Aug/2015 23:16:23] "GET /versionPython >> HTTP/1.1" 200 5 127.0.0.1 - - [21/Aug/2015 23:16:23] "GET >> /links/data HTTP/1.1" 200 2884 127.0.0.1 - - [21/Aug/2015 23:16:23] >> "GET /versionSQLite HTTP/1.1" 200 5 >> -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: SQLite3 and web server
On Friday 21 Aug 2015 16:57 CEST, Michael Torrie wrote: > On 08/21/2015 08:19 AM, Michael Torrie wrote: >> But of course you can use straight CGI with Apache too. Probably >> this is the easiest way for you to get started. Don't try to run >> your own http server. That's just a waste of time and effort. Run >> the CGIs under Apache. Google for Apache CGI to get info on how to >> set up apache to execute CGI scripts (in any language). > > On second thought, maybe your original course of action is better, > though I don't know the cause of the errors you are seeing. I switched to bottle. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: Sometimes bottle takes a lot of time
On 22.08.2015 13:28, Cecil Westerhof wrote: > If you would have taken a little more time you would have seen that > there where 20 seconds between both logs. I am fast, but not that > fast. It is exactly the same code. I suppose it has to do something > with bottle. Something I use since yesterday. Is it that strange to > look if someone else has had the same problem and maybe a solution? So let me get your story straight: You're new to bottle and, apparently, web-programming in Python as well. You're new to sqlite. You've built some web application using both. Yesterday it worked perfectly. Today it doesn't. Obviously, you suspect bottle is at fault. Being the very wise Senior Software Engineer that you are, do you really think that mature software, programmed by people who actually know what they're doing is at fault? Or do you rather think it maybe could be the piece-of-junk demo application written by someone who has proven his utter incompetence comprehensively time and time again? Since you're the very wise Senior Software Engineer here, why don't you use an approach that every schoolkid learns in a programming class? Namely, circle the error, reproduce it reliably. Change your machine, network setup and change between your software versions. Create a minimal example that demonstrate the issue. Then, should you find one, blame bottle. Not sooner, very wise Senior Software Engineer, not sooner. Cheers, Johannes -- >> Wo hattest Du das Beben nochmal GENAU vorhergesagt? > Zumindest nicht öffentlich! Ah, der neueste und bis heute genialste Streich unsere großen Kosmologen: Die Geheim-Vorhersage. - Karl Kaos über Rüdiger Thomas in dsa -- https://mail.python.org/mailman/listinfo/python-list
Re: SQLite3 and web server
Cecil Westerhof wrote:
> On Saturday 22 Aug 2015 09:51 CEST, Peter Otten wrote:
>
>> Cecil Westerhof wrote:
>>
>>> Thanks. I made a first try:
>>> https://github.com/CecilWesterhof/PublishedPhotos
>>>
>>> The thing I do not like is that all my static files have to be put
>>> in /static/, but I think I can life with it.
>>
>> Is that really required by bottle? Where would you like to put the
>> files?
>
> That is how I interpret this:
> http://bottlepy.org/docs/dev/tutorial.html#routing-static-files
No, /static/ is in no way special. To drive the point home here's a bogus
example that gives you a random file from the ~/.thumbnails/large folder:
@bottle.route("/")
def random_thumbnail(path):
root = os.path.expanduser("~/.thumbnails/large")
name = random.choice(os.listdir(root))
return bottle.static_file(name, root=root)
The path is not even taken into account.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Sometimes bottle takes a lot of time
On Saturday 22 Aug 2015 14:09 CEST, Johannes Bauer wrote: > On 22.08.2015 13:28, Cecil Westerhof wrote: > >> If you would have taken a little more time you would have seen that >> there where 20 seconds between both logs. I am fast, but not that >> fast. It is exactly the same code. I suppose it has to do something >> with bottle. Something I use since yesterday. Is it that strange to >> look if someone else has had the same problem and maybe a solution? > > So let me get your story straight: I wish you really meant that. You just like to bash people. I am afraid I need to ignore you. Have a happy life. Do not forget to look in the mirror. Will be painful in the short term, but very beneficial in the long term. Also: take a course in reading. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: Sometimes bottle takes a lot of time
On 22.08.2015 15:09, Cecil Westerhof wrote: >> So let me get your story straight: > > I wish you really meant that. I really do, did I get it wrong at all? I really don't think that I did. > Also: take a course in reading. Maybe you, oh very wise Senior Software Engineer, should take a course in Software Engineering. You wouldn't otherwise ask embarassingly stupid questions over and over and over again. Really eats away at your seniority if you ask me. Cheers, Johannes -- >> Wo hattest Du das Beben nochmal GENAU vorhergesagt? > Zumindest nicht öffentlich! Ah, der neueste und bis heute genialste Streich unsere großen Kosmologen: Die Geheim-Vorhersage. - Karl Kaos über Rüdiger Thomas in dsa -- https://mail.python.org/mailman/listinfo/python-list
Re: Sometimes bottle takes a lot of time
Am 22.08.15 um 15:51 schrieb Johannes Bauer: On 22.08.2015 15:09, Cecil Westerhof wrote: So let me get your story straight: I wish you really meant that. I really do, did I get it wrong at all? I really don't think that I did. Probably yes. You should take a look at the OP again and compare the time stamps. It says that in between two consecutive calls of the same program, the request was served once in a second, and once with serious delays. Despite that the server is localhost. In between both trials there are 20 seconds. I do not see, how git bisect would help here. Note that this says nothing about the location of the bug, in can still be either in the OPs code or in the framework. Christian -- https://mail.python.org/mailman/listinfo/python-list
Re: Sometimes bottle takes a lot of time
On 08/22/2015 05:37 AM, Cecil Westerhof wrote: >> I don't know. Is it bottle, or the browser, or something completely >> different that eats the extra time? > > I really do not know. I suspect bottle, but I am new to this, so I > value the suspicion of someone who has more experience more. :-D These are requests performed from browser Javascript (ajax), right? Could you write a shell script that fetches these urls in sequence using curl or wget, simulating the web browser? This would let you check times in a controlled way, without the variable of the browser itself. While it's true this particular problem is possibly beyond the scope of this python list (and may not be python-related at all), it's too bad a couple of people have taken the time to reply to your queries to simply berate you. -- https://mail.python.org/mailman/listinfo/python-list
Re: Sometimes bottle takes a lot of time
On Saturday, August 22, 2015 at 9:03:52 PM UTC+5:30, Michael Torrie wrote: > While it's true this particular problem is possibly beyond the scope of > this python list (and may not be python-related at all), it's too bad a > couple of people have taken the time to reply to your queries to simply > berate you. Yeah -- quite uncalled for. As for beyond scope, I believe Peter Otten recommended bottle just a few days ago. So I dont see whats improper about the question. At some point of course someone may say: "Bottle written in python doesn't mean this is a python question." If after that the questions continue and they are persistent and asinine and... then berating may be ok. Dont see that here -- https://mail.python.org/mailman/listinfo/python-list
Re: Bug!
On 22/08/2015 02:02, Chris Angelico wrote: The security concerns of XP aren't Python's problem, and Python isn't in the business of twisting people's arms to make them upgrade just for the sake of upgrading. However, every new version of Windows introduces new APIs and features, so maintaining support for an older version means ignoring all features added since then; conversely, dropping support for XP means taking advantage of anything that was added in Vista. That's why the change in support. Thanks for saying this, Chris. Just to add, from a Python developer perspective: any system -- Microsoft or not, open or not, old or new -- which core Python supports, brings a measure of complexity to the codebase. #ifdefs, conditional LoadLibrary calls &c. From the point of view simply of the maintenance burden, less code is better. Obviously there is more to deciding on platform support than code maintenance ... This isn't some kind of political move by the Python dev team to undercut Windows users: it's entirely pragmatic. And using the Windows support calendar is a common-sense way of giving ourselves a set of cut-off dates. TJG -- https://mail.python.org/mailman/listinfo/python-list
Re: Sometimes bottle takes a lot of time
On Saturday 22 Aug 2015 17:33 CEST, Michael Torrie wrote: > On 08/22/2015 05:37 AM, Cecil Westerhof wrote: >>> I don't know. Is it bottle, or the browser, or something >>> completely different that eats the extra time? >> >> I really do not know. I suspect bottle, but I am new to this, so I >> value the suspicion of someone who has more experience more. :-D > > These are requests performed from browser Javascript (ajax), right? > Could you write a shell script that fetches these urls in sequence > using curl or wget, simulating the web browser? This would let you > check times in a controlled way, without the variable of the browser > itself. I should have thought about that myself. :-( I was already thinking about writing debug statements in the routes. By the way does anybody know what the time-stamp is: the moment the requests is received, or the moment the request is finished? I just tried it again. Two almost immediately and then a long one again. What I find very peculiarly is that every-time there is a delay, there are two delays of six seconds. > While it's true this particular problem is possibly beyond the scope > of this python list (and may not be python-related at all), it's too > bad a couple of people have taken the time to reply to your queries > to simply berate you. It could be an AngularJS problem. ;-) -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Best strategy for testing class and subclasses in pytest?
Greetings, I'm writing a chess engine to learn about Python classes and inheritance, and using pytest for the unit test. I've created a Piece class, which has 99% of the functionality for a chess piece, and subclass the other pieces -- Bishop, King, Knight, Pawn, Queen, Rook -- that will implement the move-specific functionality. I'm not sure what's the best strategy is for testing the class and subclasses. I initially wrote unit tests for the class and subclasses (including tests for class-only functionality). That worked well until I started refactoring code and breaking the tests. Too much copy, paste and renaming for my taste. I tried to create a separate class and/or module to import the common tests for each class and subclass. My attempts often ended in failure with the "RuntimeError: super(): no arguments" message. I couldn't find a working example on the Internet on how to do that. The pytest documentation is all over the place. Is there a way to reuse tests in pytest? Or should I test everything in the class and test only the implemented functionality in the subclasses? Thank you, Chris R. -- https://mail.python.org/mailman/listinfo/python-list
Re: Sometimes bottle takes a lot of time
On Saturday 22 Aug 2015 20:03 CEST, Cecil Westerhof wrote:
> On Saturday 22 Aug 2015 17:33 CEST, Michael Torrie wrote:
>
>> On 08/22/2015 05:37 AM, Cecil Westerhof wrote:
I don't know. Is it bottle, or the browser, or something
completely different that eats the extra time?
>>>
>>> I really do not know. I suspect bottle, but I am new to this, so I
>>> value the suspicion of someone who has more experience more. :-D
>>
>> These are requests performed from browser Javascript (ajax), right?
>> Could you write a shell script that fetches these urls in sequence
>> using curl or wget, simulating the web browser? This would let you
>> check times in a controlled way, without the variable of the
>> browser itself.
>
> I should have thought about that myself. :-(
I used Python instead of a shell script of-course. :-P
#!/usr/bin/env python3
import time
from urllib.request import urlopen
server = 'http://localhost:8080'
urls = [
'/',
'/static/css/default.css',
'/static/JavaScript/angular.js',
'/static/appPublishedPhotos.js',
'/links/data',
'/versionPython',
'/versionSQLite',
]
for x in range(0, 10):
start_time = time.time()
for url in urls:
print(url)
urlopen(server + url).read()
end_time = time.time()
print('It took {0} seconds\n'.format(end_time - start_time), flush = True)
time.sleep(20)
It is not perfect code, no error checking and the last sleep is
superfluous, but for the current job good enough.
I have included the output as attachment. It is clear that bottle is
not the problem: fetching all the data takes at most 0.017 seconds.
/
/static/css/default.css
/static/JavaScript/angular.js
/static/appPublishedPhotos.js
/links/data
/versionPython
/versionSQLite
It took 0.012779951095581055 seconds
/
/static/css/default.css
/static/JavaScript/angular.js
/static/appPublishedPhotos.js
/links/data
/versionPython
/versionSQLite
It took 0.010144948959350586 seconds
/
/static/css/default.css
/static/JavaScript/angular.js
/static/appPublishedPhotos.js
/links/data
/versionPython
/versionSQLite
It took 0.010040283203125 seconds
/
/static/css/default.css
/static/JavaScript/angular.js
/static/appPublishedPhotos.js
/links/data
/versionPython
/versionSQLite
It took 0.013696908950805664 seconds
/
/static/css/default.css
/static/JavaScript/angular.js
/static/appPublishedPhotos.js
/links/data
/versionPython
/versionSQLite
It took 0.01705002784729004 seconds
/
/static/css/default.css
/static/JavaScript/angular.js
/static/appPublishedPhotos.js
/links/data
/versionPython
/versionSQLite
It took 0.009366750717163086 seconds
/
/static/css/default.css
/static/JavaScript/angular.js
/static/appPublishedPhotos.js
/links/data
/versionPython
/versionSQLite
It took 0.011032581329345703 seconds
/
/static/css/default.css
/static/JavaScript/angular.js
/static/appPublishedPhotos.js
/links/data
/versionPython
/versionSQLite
It took 0.015336990356445312 seconds
/
/static/css/default.css
/static/JavaScript/angular.js
/static/appPublishedPhotos.js
/links/data
/versionPython
/versionSQLite
It took 0.016999244689941406 seconds
/
/static/css/default.css
/static/JavaScript/angular.js
/static/appPublishedPhotos.js
/links/data
/versionPython
/versionSQLite
It took 0.010639429092407227 seconds
So my next bet is AngularJS.
--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
--
https://mail.python.org/mailman/listinfo/python-list
Re: Bug!
On Sun, Aug 23, 2015 at 3:08 AM, Tim Golden wrote: > On 22/08/2015 02:02, Chris Angelico wrote: >> >> The security concerns of XP aren't Python's problem, and Python isn't >> in the business of twisting people's arms to make them upgrade just >> for the sake of upgrading. However, every new version of Windows >> introduces new APIs and features, so maintaining support for an older >> version means ignoring all features added since then; conversely, >> dropping support for XP means taking advantage of anything that was >> added in Vista. That's why the change in support. > > > Thanks for saying this, Chris. Just to add, from a Python developer > perspective: any system -- Microsoft or not, open or not, old or new -- > which core Python supports, brings a measure of complexity to the codebase. > #ifdefs, conditional LoadLibrary calls &c. From the point of view simply of > the maintenance burden, less code is better. Obviously there is more to > deciding on platform support than code maintenance ... > > This isn't some kind of political move by the Python dev team to undercut > Windows users: it's entirely pragmatic. And using the Windows support > calendar is a common-sense way of giving ourselves a set of cut-off dates. > Precisely. Every time you support multiple versions of some dependency, you have to test your code on all of them, and in the common case (new features added in newer versions), you have to target the oldest and weakest version. When you're writing a Python program that has to run on CPython back as far as 2.4, there's a lot you can't do... dropping support for everything pre-2.7 lets you improve your code significantly. Does dropping support for Python 2.4 consist of "undercutting RHEL users"? Nope. It's that same pragmatism - I want a cleaner codebase. CPython 3.4 will continue to run on Windows XP. If you're still using an old Windows, you just have to keep using an old Python too. Eventually Python 3.4 will be out of support, but at that point, it's no different from the OS anyway. There's nothing stopping you from using an ancient OS, an ancient CPython, and an ancient Python application, if that's what it requires... I'm not sure what OS people are running Python 1.5 on, but if anyone complains that it doesn't install properly on Windows 10, I rather doubt that python.org will release a patch :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Sometimes bottle takes a lot of time
On Sun, Aug 23, 2015 at 7:06 AM, Cecil Westerhof wrote: > I have included the output as attachment. It is clear that bottle is > not the problem: fetching all the data takes at most 0.017 seconds. > Something to consider: You could be running into some weird interaction of caches. Try blowing your OS and browser caches, and see what the timings are like then. Also, if you can recreate the six-second delay, I'd want to know what's happening during that time - is there an open socket between the browser and the server? Is anything pegging the CPU? Is the disk heavily active? Finding any sort of saturation would help to pin down the cause of the delay. Do you have any network mounts in your file system, and could they be delaying some stat() call somewhere? Six seconds is a lot, but I do recall running into problems occasionally when I had a NETBIOS/NETBEUI mount on one of my boxes (Linux couldn't safely cache stuff, and the remote system was misconfigured as regards caching, I think - the upshot was terrible performance in certain situations, all of it spent waiting on the network). ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Sandboxing Python
I was always led to believe that the subject was a difficult thing to do, but here https://www.reddit.com/r/learnpython/comments/3huz4x/how_to_do_math_inside_raw_input/ is a safe solution in only 23 characters, or are there any discernable flaws in it? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Sandboxing Python
On Sun, Aug 23, 2015 at 9:25 AM, Mark Lawrence wrote: > I was always led to believe that the subject was a difficult thing to do, > but here > https://www.reddit.com/r/learnpython/comments/3huz4x/how_to_do_math_inside_raw_input/ > is a safe solution in only 23 characters, or are there any discernable flaws > in it? I'm sorry, I can't see which solution you're talking about there - maybe I just don't know how to read reddit properly. Can you paste the proposed code please? The best I can see there is "use eval but with no builtins". That's fundamentally flawed because you don't need builtins to break stuff. All you need is a literal, from which you can snag everything else via its attributes. However, for this situation, I would be recommending ast.literal_eval, which *is* safe. It's a lot more powerful than "split it into number, operator, number" as mentioned at the end, but still can't majorly break anything. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Sandboxing Python
On 23/08/2015 00:44, Chris Angelico wrote:
On Sun, Aug 23, 2015 at 9:25 AM, Mark Lawrence wrote:
I was always led to believe that the subject was a difficult thing to do,
but here
https://www.reddit.com/r/learnpython/comments/3huz4x/how_to_do_math_inside_raw_input/
is a safe solution in only 23 characters, or are there any discernable flaws
in it?
I'm sorry, I can't see which solution you're talking about there -
maybe I just don't know how to read reddit properly. Can you paste the
proposed code please?
The best I can see there is "use eval but with no builtins". That's
fundamentally flawed because you don't need builtins to break stuff.
All you need is a literal, from which you can snag everything else via
its attributes.
However, for this situation, I would be recommending ast.literal_eval,
which *is* safe. It's a lot more powerful than "split it into number,
operator, number" as mentioned at the end, but still can't majorly
break anything.
ChrisA
>>> import os
>>> eval("os.system('rm -rf /')", {"__builtins__":None})
Traceback (most recent call last):
File "", line 1, in
eval("os.system('rm -rf /')", {"__builtins__":None})
File "", line 1, in
TypeError: 'NoneType' object is not subscriptable
Surely I must I have missed your meaning because I needed just 23
characters and zero extra lines to create a safe sandbox for this, but
you've said that the core developers have tried and failed to do this.
It appears that I didn't just wipe out my entire filesystem and you've
stated quite matter-of-factly that there is no safe solution... so what
happened here? Why didn't my filesystem just get wiped out?
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.
Mark Lawrence
--
https://mail.python.org/mailman/listinfo/python-list
Re: Sandboxing Python
On Sun, Aug 23, 2015 at 9:52 AM, Mark Lawrence wrote:
> On 23/08/2015 00:44, Chris Angelico wrote:
>>
>> On Sun, Aug 23, 2015 at 9:25 AM, Mark Lawrence
>> wrote:
>>>
>>> I was always led to believe that the subject was a difficult thing to do,
>>> but here
>>>
>>> https://www.reddit.com/r/learnpython/comments/3huz4x/how_to_do_math_inside_raw_input/
>>> is a safe solution in only 23 characters, or are there any discernable
>>> flaws
>>> in it?
>>
>>
>>
>> I'm sorry, I can't see which solution you're talking about there -
>> maybe I just don't know how to read reddit properly. Can you paste the
>> proposed code please?
>>
>> The best I can see there is "use eval but with no builtins". That's
>> fundamentally flawed because you don't need builtins to break stuff.
>> All you need is a literal, from which you can snag everything else via
>> its attributes.
>>
>> However, for this situation, I would be recommending ast.literal_eval,
>> which *is* safe. It's a lot more powerful than "split it into number,
>> operator, number" as mentioned at the end, but still can't majorly
>> break anything.
>>
>> ChrisA
>>
>
>
import os
eval("os.system('rm -rf /')", {"__builtins__":None})
> Traceback (most recent call last):
> File "", line 1, in
> eval("os.system('rm -rf /')", {"__builtins__":None})
> File "", line 1, in
> TypeError: 'NoneType' object is not subscriptable
>
>
>
> Surely I must I have missed your meaning because I needed just 23 characters
> and zero extra lines to create a safe sandbox for this, but you've said that
> the core developers have tried and failed to do this. It appears that I
> didn't just wipe out my entire filesystem and you've stated quite
> matter-of-factly that there is no safe solution... so what happened here?
> Why didn't my filesystem just get wiped out?
>
Got it, thanks. The answer is: It's easy to make something you can't
yourself break out of. It just means you don't know all the tricks.
http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html
>>> cmd="""[c for c in ().__class__.__base__.__subclasses__() if c.__name__ ==
>>> 'catch_warnings'][0]()._module.__builtins__["__import__"]("os").system("echo
>>> Hello")"""
>>> eval(cmd,{"__builtins__":None})
Hello
0
Et voila. Arbitrary module loading, arbitrary code execution, have fun.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Sometimes bottle takes a lot of time
On Sunday 23 Aug 2015 01:13 CEST, Chris Angelico wrote: > On Sun, Aug 23, 2015 at 7:06 AM, Cecil Westerhof wrote: >> I have included the output as attachment. It is clear that bottle >> is not the problem: fetching all the data takes at most 0.017 >> seconds. >> > > Something to consider: You could be running into some weird > interaction of caches. Try blowing your OS and browser caches, and > see what the timings are like then. Also, if you can recreate the > six-second delay, I'd want to know what's happening during that time > - is there an open socket between the browser and the server? Is > anything pegging the CPU? Is the disk heavily active? Finding any > sort of saturation would help to pin down the cause of the delay. Do > you have any network mounts in your file system, and could they be > delaying some stat() call somewhere? Six seconds is a lot, but I do > recall running into problems occasionally when I had a > NETBIOS/NETBEUI mount on one of my boxes (Linux couldn't safely > cache stuff, and the remote system was misconfigured as regards > caching, I think - the upshot was terrible performance in certain > situations, all of it spent waiting on the network). How do I see if there is an open socket? But in principal I have found the problem. (Not the reason.) The problem is Firefox. (So it is not bottle and also not AngularJS.) When using Chrome there is no problem. Not even when I do 15 times a refresh. With Firefox there is this problem. Even when I restart it. So I have found the problem and it is certainly not a Python problem. I should post it on the Firefox, but if there will be an use-full reply … I have been reading it for a long time. And when someone mentioned that Firefox used to much CPU or memory the replies where in my opinion not very helpful. It even happens with Firefox in safe-mode. Less often and not two times, but max one time. Well, first some sleep and then trying to get something useful from the Firefox mailing-list. ;-) -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: Sometimes bottle takes a lot of time
On Sun, Aug 23, 2015 at 10:51 AM, Cecil Westerhof wrote: > How do I see if there is an open socket? Depends on your OS. On Linux, I can poke around in /proc or with commands like netstat and/or lsof. It may be easier to separate client and server across two computers, which would force the socket to be a "real" network connection, rather than being optimized away. > But in principal I have found the problem. (Not the reason.) The > problem is Firefox. (So it is not bottle and also not AngularJS.) When > using Chrome there is no problem. Not even when I do 15 times a > refresh. With Firefox there is this problem. Even when I restart it. Huh, interesting. I can't help there, but I wish you the very best of luck in finding it. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Bug!
On Sun, Aug 23, 2015 at 11:22 AM, Simon Ward wrote: > > > On 23 August 2015 00:06:44 BST, Chris Angelico wrote: >>Precisely. Every time you support multiple versions of some >>dependency, you have to test your code on all of them, and in the >>common case (new features added in newer versions), you have to target >>the oldest and weakest version. > > Just don't add features to older versions. They're in maintenance or bugfix > mode. That's not what I'm talking about... I'm talking about multiple versions of a dependency. If I write a Python script, and tell people "this requires CPython 3.6 running on Linux" because that's what I run... it's not going to be easy to use. Telling people that it requires Python 3.4 or newer cuts out a lot of people, requiring 3.3 or better is going to include a lot more. It's a tradeoff between usability and cleanliness of code. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Sandboxing Python
Ned Batchelder has researched this one quite a bit, see: see: http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html http://nedbatchelder.com/blog/201302/looking_for_python_3_builtins.html http://nedbatchelder.com/blog/201302/finding_python_3_builtins.html Laura -- https://mail.python.org/mailman/listinfo/python-list
Re: Sandboxing Python
Am 23.08.15 um 02:04 schrieb Chris Angelico:
import os
eval("os.system('rm -rf /')", {"__builtins__":None})
Traceback (most recent call last):
File "", line 1, in
eval("os.system('rm -rf /')", {"__builtins__":None})
File "", line 1, in
TypeError: 'NoneType' object is not subscriptable
Got it, thanks. The answer is: It's easy to make something you can't
yourself break out of. It just means you don't know all the tricks.
http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html
cmd="""[c for c in ().__class__.__base__.__subclasses__() if c.__name__ ==
'catch_warnings'][0]()._module.__builtins__["__import__"]("os").system("echo Hello")"""
eval(cmd,{"__builtins__":None})
Hello
0
Et voila. Arbitrary module loading, arbitrary code execution, have fun.
In one of my other favourite languages, you can create sandboxes very
easily. You create them as a new slave interpreter with restrictions:
interp create -safe myInterp
myInterp eval $userinput
In addition to removing "dangerous" functions, you can limit the mount
of time spent by the eval, or alias new functions to callbacks from the
main interpreter (though this may break security)
This was once built into it for a browser plugin (now extinct).
Would it be that difficult to get the same for Python? On the C side,
the interpreter is a structure and does not use global variables (as
opposed to CPython), therefore it is easy to create more than one
interpreter in a single program, and also to reflect that to the
scripting level.
Christian
--
https://mail.python.org/mailman/listinfo/python-list
Re: Sandboxing Python
On Sun, Aug 23, 2015 at 3:17 PM, Christian Gollwitzer wrote: > Would it be that difficult to get the same for Python? On the C side, the > interpreter is a structure and does not use global variables (as opposed to > CPython), therefore it is easy to create more than one interpreter in a > single program, and also to reflect that to the scripting level. There have been some explorations in that direction. However, it would be very difficult to pass objects from one interpreter to the other, so you'd be restricted to some form of serialization... at which point you may as well just use a subprocess, which you can isolate using OS facilities. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Bug!
On 23 August 2015 00:06:44 BST, Chris Angelico wrote: >Precisely. Every time you support multiple versions of some >dependency, you have to test your code on all of them, and in the >common case (new features added in newer versions), you have to target >the oldest and weakest version. Just don't add features to older versions. They're in maintenance or bugfix mode. > When you're writing a Python program >that has to run on CPython back as far as 2.4, there's a lot you can't >do... Just deprecate then drop the stuff you don't want to support any more. This is part of the standard software lifecyle to me. If you care about your users that might not be ready to upgrade provide security fixes for the older versions. If you don't care, well I wouldn't want to use your software in production. > dropping support for everything pre-2.7 lets you improve your >code significantly. Does dropping support for Python 2.4 consist of >"undercutting RHEL users"? Nope. It's that same pragmatism - I want a >cleaner codebase. RHEL 4 might still be supported (is it? haven't paid attention) but there should be no obligation to provide new features. Have a development branch or mainline, that doesn't stop you from having "stable" rele > >CPython 3.4 will continue to run on Windows XP. If you're still using >an old Windows, you just have to keep using an old Python too. >Eventually Python 3.4 will be out of support, but at that point, it's >no different from the OS anyway. There's nothing stopping you from >using an ancient OS, an ancient CPython, and an ancient Python >application, if that's what it requires... I'm not sure what OS people >are running Python 1.5 on, but if anyone complains that it doesn't >install properly on Windows 10, I rather doubt that python.org will >release a patch :) > >ChrisA -- Sent from Kaiten Mail. Please excuse my brevity. -- https://mail.python.org/mailman/listinfo/python-list
Qestion
hi dear friend I have python version 2.4.3 Which version of PIL is appropriate for me and how can I add it to my systems? Regards -- https://mail.python.org/mailman/listinfo/python-list
how to handle cpu cache in python ( or fastest way to call a function once)
I find out that python's VM seems to be very unfriendly with CPU-Cache. see: http://stackoverflow.com/questions/32163585/how-to-handle-cpu-cache-in-python-or-fastest-way-to-call-a-function-once http://stackoverflow.com/questions/32153178/python-functionor-a-code-block-runs-much-slower-with-a-time-interval-in-a-loop for example: *** import time a = range(500) sum(a) for i in range(100): #just to create a time interval, seems this disturb cpu cache? pass st = time.time() sum(a) print (time.time() - st)*1e6 * time:> 100us another case: * import time a = range(500) for i in range(10): st = time.time() sum(a) print (time.time() - st)*1e6 * time:~ 20us we can see when running frequently, the code becomes much faster. is there a solution? I feel this question is very difficult. one must has indepth unstanding about the mechanism of python virtual machine, c and cpu-cache. Do you have any suggestion about where to post this question for a possible answer? -- https://mail.python.org/mailman/listinfo/python-list
