Re: Question about asyncio and blocking operations
"Ian Kelly" wrote in message news:CALwzidkr-fT6S6wH2caNaxyQvUdAw=x7xdqkqofnrrwzwnj...@mail.gmail.com... On Wed, Jan 27, 2016 at 10:14 AM, Ian Kelly wrote: > Unfortunately this doesn't actually work at present. > EventLoop.run_in_executor swallows the StopIteration exception and > just returns None, which I assume is a bug. http://bugs.python.org/issue26221 Thanks for that. Fascinating discussion between you and GvR. Reading it gave me an idea. Run the database handler in a separate thread. Use a queue.Queue to send requests to the handler. Use an asyncio.Queue to send results back to the caller, which can call 'await q.get()'. I ran a quick test and it seems to work. What do you think? Frank -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about asyncio and blocking operations
On Thu, Jan 28, 2016 at 10:11 PM, Frank Millman wrote: >> >> The other risk is that the wrong result will be queried (two async >> tasks put something onto the queue - which one gets the first >> result?), which could either be coped with by simple sequencing (maybe >> this happens automatically, although I'd prefer a >> mathematically-provable result to "it seems to work"), or by wrapping >> the whole thing up in a function/class. >> > > I *think* I have this one covered. When the caller makes a request, it > creates an instance of an asyncio.Queue, and includes it with the request. > The db handler uses this queue to send the result back. > > Do you see any problem with this? Oh, I get it. In that case, you should be safe (at the cost of efficiency, presumably, but probably immeasurably so). The easiest way to thrash-test this is to simulate an ever-increasing stream of requests that eventually get bottlenecked by the database. For instance, have the database "process" a maximum of 10 requests a second, and then start feeding it 11 requests a second. Monitoring the queue length should tell you what's happening. Eventually, the queue will hit its limit (and you can make that happen sooner by creating the queue with a small maxsize), at which point the queue.put() will block. My suspicion is that that's going to lock your entire backend, unlocking only when the database takes something off its queue; you'll end up throttling at the database's rate (which is fine), but with something perpetually blocked waiting for the database (which is bad - other jobs, like socket read/write, won't be happening). As an alternative, you could use put_nowait or put(item, block=False) to put things on the queue. If there's no room, it'll fail immediately, which you can spit back to the user as "Database overloaded, please try again later". Tuning the size of the queue would then be an important consideration for real-world work; you'd want it long enough to cope with bursty traffic, but short enough that people's requests don't time out while they're blocked on the database (it's much tidier to fail them instantly if that's going to happen). ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about asyncio and blocking operations
On Thu, Jan 28, 2016 at 8:13 PM, Frank Millman wrote: > Run the database handler in a separate thread. Use a queue.Queue to send > requests to the handler. Use an asyncio.Queue to send results back to the > caller, which can call 'await q.get()'. > > I ran a quick test and it seems to work. What do you think? My gut feeling is that any queue can block at either get or put. Half of your operations are "correct", and the other half are "wrong". The caller can "await q.get()", which is the "correct" way to handle the blocking operation; the database thread can "jobqueue.get()" as a blocking operation, which is also fine. But your queue-putting operations are going to have to assume that they never block. Maybe you can have the database put its results onto the asyncio.Queue safely, but the requests going onto the queue.Queue could block waiting for the database. Specifically, this will happen if database queries come in faster than the database can handle them - quite literally, your jobs will be "blocked on the database". What should happen then? The other risk is that the wrong result will be queried (two async tasks put something onto the queue - which one gets the first result?), which could either be coped with by simple sequencing (maybe this happens automatically, although I'd prefer a mathematically-provable result to "it seems to work"), or by wrapping the whole thing up in a function/class. Both of these are risks seen purely by looking at the idea description, not at any sort of code. It's entirely possible I'm mistaken about them. But there's only one way to find out :) By the way, just out of interest... there's no way you can actually switch out the database communication for something purely socket-based, is there? PostgreSQL's protocol, for instance, is fairly straight-forward, and you don't *have* to use libpq; Pike's inbuilt pgsql module just opens a socket and says hello, and it looks like py-postgresql [1] is the same thing for Python. Taking something like that and making it asynchronous would be as straight-forward as converting any other socket-based code. Could be an alternative to all this weirdness. ChrisA [1] https://pypi.python.org/pypi/py-postgresql -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about asyncio and blocking operations
"Chris Angelico" wrote in message news:captjjmr162+k4lzefpxrur6wxrhxbr-_wkrclldyr7kst+k...@mail.gmail.com... On Thu, Jan 28, 2016 at 8:13 PM, Frank Millman wrote: > Run the database handler in a separate thread. Use a queue.Queue to send > requests to the handler. Use an asyncio.Queue to send results back to > the > caller, which can call 'await q.get()'. > > I ran a quick test and it seems to work. What do you think? My gut feeling is that any queue can block at either get or put H'mm, I will have to think about that one, and figure out how to create a worst-case scenario. I will report back on that. The other risk is that the wrong result will be queried (two async tasks put something onto the queue - which one gets the first result?), which could either be coped with by simple sequencing (maybe this happens automatically, although I'd prefer a mathematically-provable result to "it seems to work"), or by wrapping the whole thing up in a function/class. I *think* I have this one covered. When the caller makes a request, it creates an instance of an asyncio.Queue, and includes it with the request. The db handler uses this queue to send the result back. Do you see any problem with this? Frank -- https://mail.python.org/mailman/listinfo/python-list
class attribute
hello Here is a class from django framework from django.db import models class Article(models.Model): titre = models.CharField(max_length=100) auteur = models.CharField(max_length=42) contenu = models.TextField(null=True) date = models.DateTimeField(auto_now_add=True, auto_now=False, verbose_name="Date de parution") def __str__(self): return self.titre From a Python point of view, what are titre, auteur, contenu and date ? Are they class attributes, so common to all instance of Article ? It seems so to me. But if i do in a django shell (run with py manage.py shell) Article.titre it doesnt work, AttributeError: type object 'Article' has no attribute 'titre' why ? if I test on a small class class MyClass: i=0 MyClass.i 0 works When we create an object of class Article article = Article(titre="Bonjour", auteur="Maxime") article.contenu = "Les crêpes bretonnes sont trop bonnes !" we use the same names titre, auteur, contenu, which should be instance attribute this time. This is confusing to me thx -- https://mail.python.org/mailman/listinfo/python-list
Re: class attribute
On 01/28/2016 02:15 PM, ast wrote: hello Here is a class from django framework from django.db import models class Article(models.Model): titre = models.CharField(max_length=100) auteur = models.CharField(max_length=42) contenu = models.TextField(null=True) date = models.DateTimeField(auto_now_add=True, auto_now=False, verbose_name="Date de parution") def __str__(self): return self.titre From a Python point of view, what are titre, auteur, contenu and date ? Are they class attributes, so common to all instance of Article ? It seems so to me. But if i do in a django shell (run with py manage.py shell) Article.titre it doesnt work, AttributeError: type object 'Article' has no attribute 'titre' why ? if I test on a small class class MyClass: i=0 MyClass.i 0 works When we create an object of class Article article = Article(titre="Bonjour", auteur="Maxime") article.contenu = "Les crêpes bretonnes sont trop bonnes !" we use the same names titre, auteur, contenu, which should be instance attribute this time. This is confusing to me thx My guess is that models.Model has a metclass. Without going too much int details, the metaclass may change the class structure when it's created. django is very specific and very database oriented. " article = Article(titre="Bonjour", auteur="Maxime") article.contenu = "Les crêpes bretonnes sont trop bonnes !" " this is probably the wrong way to assign a value to 'contenu'. You should have a look at django help files, from what I remember it's very well documented with a lot of examples. jm -- https://mail.python.org/mailman/listinfo/python-list
Re: class attribute
"ast" a écrit dans le message de news:56aa1474$0$27833$426a7...@news.free.fr... OK, thank you for answer I didn't studied metaclass yet that's why it is not clear for me. I have to take a look at that concept first -- https://mail.python.org/mailman/listinfo/python-list
Re: class attribute
On Fri, Jan 29, 2016 at 12:56 AM, jmp wrote: > My guess is that models.Model has a metclass. Without going too much int > details, the metaclass may change the class structure when it's created. > > django is very specific and very database oriented. > > " > article = Article(titre="Bonjour", auteur="Maxime") > article.contenu = "Les crêpes bretonnes sont trop bonnes !" > " > > this is probably the wrong way to assign a value to 'contenu'. You should > have a look at django help files, from what I remember it's very well > documented with a lot of examples. > I suspect Django's using the descriptor protocol or other magic here. SQLAlchemy works in a similar way; when you create the class, you put a bunch of attributes on it to specify columns, and then you can simply assign to those attributes on an instance to set the values for a row to be saved to the database. As Todd explains in further detail, the metaclass gets to do whatever it likes. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
installation error
Hello, I am having an issue installing Python 3.51, I get an error message stating that there is a dll that is missing: API-MS-WIN-CRT-RUNTIME-l1-1-0.DLL I have tried reinstalling the program but keep getting the same error message. Please advise of fix. Please reply to jrsc...@msn.com Hello, -- https://mail.python.org/mailman/listinfo/python-list
[no subject]
Instalar python -- https://mail.python.org/mailman/listinfo/python-list
Re:
On Mon, Jan 25, 2016 at 8:58 PM, Karla Larrea < karlamishellkmlp1...@hotmail.com> wrote: > Instalar python > > -- > https://mail.python.org/mailman/listinfo/python-list > So, you are trying to install python? A wild guess. Can you tell us what OS, what version of python, what have you tried so far? -- Joel Goldstick http://joelgoldstick.com/stats/birthdays -- https://mail.python.org/mailman/listinfo/python-list
Re: how to completely uninstall python 2.7 on Windows
On Thu, Jan 28, 2016 at 10:40 AM, Ryan Gonzalez wrote: > Wrong mailing list! You probably want > https://mail.python.org/mailman/listinfo/python-list > > Anyway, does uninstalling from the Control Panel not work? > > On January 28, 2016 7:59:00 AM CST, "Shi, Jiajun" > wrote: > >Hi there, > > > >Please help me to uninstall Python 2.7 on Windows. I want to install > >Python 3, and new python packages are all installed for Python 2.7, > >even I tried uninstalling approaches i searched from web to > >uninstalling previous versions. > > > >Thanks, > >Jiajun Shi > >-- > >https://mail.python.org/mailman/listinfo/python-announce-list > > > >Support the Python Software Foundation: > >http://www.python.org/psf/donations/ > > -- > Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. > -- > https://mail.python.org/mailman/listinfo/python-list > You can have both installed at the same time -- Joel Goldstick http://joelgoldstick.com/stats/birthdays -- https://mail.python.org/mailman/listinfo/python-list
Re: class attribute
On 28/01/16 13:15, ast wrote: hello Here is a class from django framework from django.db import models class Article(models.Model): titre = models.CharField(max_length=100) auteur = models.CharField(max_length=42) contenu = models.TextField(null=True) date = models.DateTimeField(auto_now_add=True, auto_now=False, verbose_name="Date de parution") def __str__(self): return self.titre From a Python point of view, what are titre, auteur, contenu and date ? Are they class attributes, so common to all instance of Article ? It seems so to me. But if i do in a django shell (run with py manage.py shell) Article.titre it doesnt work, AttributeError: type object 'Article' has no attribute 'titre' why ? if I test on a small class class MyClass: i=0 MyClass.i 0 works When we create an object of class Article article = Article(titre="Bonjour", auteur="Maxime") article.contenu = "Les crêpes bretonnes sont trop bonnes !" we use the same names titre, auteur, contenu, which should be instance attribute this time. This is confusing to me thx Django Model classes are very different from a basic Python classes. It is worth having a look inside django/db/models/base.py for more information. But in summary, see below. The Model class has a metaclass of ModelBase which on __new__() returns a class constructed from the attributes which you have defined in your Model class (Article in this example). All the information is retained, it is just filed away neatly for other purposes. Have a look in the Article._meta attribute. You can find the fields you provided through Article._meta.fields. When you create Article (as above) you run the __init__() which reaches into the ._meta attribute of the class in order to create an object, which is then stored within the 'database'. This is done because you don't want to access the attributes on the Model, so they are removed when the Model is created. What you actually want to do is access them on the objects within the Model. Writing Article.objects.first().titre in Django Models is mostly equivalent to Article.titre from Python classes. Todd -- https://mail.python.org/mailman/listinfo/python-list
Re: installation error
On 27 January 2016 at 15:56, JAMES SCALF wrote: > > I am having an issue installing Python 3.51, > > I get an error message stating that there is a dll that is > missing: > > API-MS-WIN-CRT-RUNTIME-l1-1-0.DLL You need to get this file in a Windows update from Microsoft. I think you can download it here: https://support.microsoft.com/en-us/kb/2999226 Then you can install Python 3.5. Alternatively you can install 3.4 which does not need this dll file. -- Oscar -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about asyncio and blocking operations
On Thu, Jan 28, 2016 at 9:40 AM, Frank Millman wrote: > I have hit a snag. It feels like a bug in 'await q.get()', though I am sure > it is just me misunderstanding how it works. > > I can post some working code if necessary, but here is a short description. > > Here is the database handler - 'request_queue' is a queue.Queue - > >while not request_queue.empty(): >return_queue, sql = request_queue.get() >cur.execute(sql) >for row in cur: >return_queue.put_nowait(row) > return_queue.put_nowait(None) >request_queue.task_done() As I commented in my previous message, asyncio.Queue is not thread-safe, so it's very important that the put calls here be done on the event loop thread using event_loop.call_soon_threadsafe. This could be the cause of the strange behavior you're seeing in getting the results. > The caller requests some data from the database like this. > >return_queue = asyncio.Queue() >sql = 'SELECT ...' >request_queue.put((return_queue, sql)) Note that since this is a queue.Queue, the put call has the potential to block your entire event loop. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Unit test
Thanks for the contributions. Took a while but I got the function done. My problem was in the first two lines, I had to import the python file in my test code which I did not do earlier -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about asyncio and blocking operations
On Jan 28, 2016 4:13 AM, "Frank Millman" wrote: > > "Chris Angelico" wrote in message news:captjjmr162+k4lzefpxrur6wxrhxbr-_wkrclldyr7kst+k...@mail.gmail.com... >> >> >> On Thu, Jan 28, 2016 at 8:13 PM, Frank Millman wrote: >> > Run the database handler in a separate thread. Use a queue.Queue to send >> > requests to the handler. Use an asyncio.Queue to send results back to > the >> > caller, which can call 'await q.get()'. >> > >> > I ran a quick test and it seems to work. What do you think? >> >> My gut feeling is that any queue can block at either get or put >> > > H'mm, I will have to think about that one, and figure out how to create a worst-case scenario. I will report back on that. The get and put methods of asyncio queues are coroutines, so I don't think this would be a real issue. The coroutine might block, but it won't block the event loop. If the queue fills up, then effectively the waiting coroutines just become a (possibly unordered) extension of the queue. >> >> The other risk is that the wrong result will be queried (two async >> tasks put something onto the queue - which one gets the first >> result?), which could either be coped with by simple sequencing (maybe >> this happens automatically, although I'd prefer a >> mathematically-provable result to "it seems to work"), or by wrapping >> the whole thing up in a function/class. >> > > I *think* I have this one covered. When the caller makes a request, it creates an instance of an asyncio.Queue, and includes it with the request. The db handler uses this queue to send the result back. > > Do you see any problem with this? That seems reasonable to me. I assume that when you send the result back you would be queuing up individual rows and not just sending a single object across, which could be more easily with just a single future. The main risk of adding limited threads to an asyncio program is that threads make it harder to reason about concurrency. Just make sure the threads don't share any state and you should be good. Note that I can only see queues being used to move data in this direction, not in the opposite. It's unclear to me how queue.get would work from the blocking thread. Asyncio queues aren't threadsafe, but you couldn't just use call_soon_threadsafe since the result is important. You might want to use a queue.Queue instead in that case, but then you run back into the problem of queue.put being a blocking operation. -- https://mail.python.org/mailman/listinfo/python-list
Re: how to completely uninstall python 2.7 on Windows
Wrong mailing list! You probably want https://mail.python.org/mailman/listinfo/python-list Anyway, does uninstalling from the Control Panel not work? On January 28, 2016 7:59:00 AM CST, "Shi, Jiajun" wrote: >Hi there, > >Please help me to uninstall Python 2.7 on Windows. I want to install >Python 3, and new python packages are all installed for Python 2.7, >even I tried uninstalling approaches i searched from web to >uninstalling previous versions. > >Thanks, >Jiajun Shi >-- >https://mail.python.org/mailman/listinfo/python-announce-list > >Support the Python Software Foundation: >http://www.python.org/psf/donations/ -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. -- https://mail.python.org/mailman/listinfo/python-list
Re: class attribute
On Thu, Jan 28, 2016 at 9:56 AM, Chris Angelico wrote: > On Fri, Jan 29, 2016 at 12:56 AM, jmp wrote: > > My guess is that models.Model has a metclass. Without going too much int > > details, the metaclass may change the class structure when it's created. > > > > django is very specific and very database oriented. > > > > " > > article = Article(titre="Bonjour", auteur="Maxime") > > article.contenu = "Les crêpes bretonnes sont trop bonnes !" > > " > > > > this is probably the wrong way to assign a value to 'contenu'. You should > > have a look at django help files, from what I remember it's very well > > documented with a lot of examples. > > > > I suspect Django's using the descriptor protocol or other magic here. > SQLAlchemy works in a similar way; when you create the class, you put > a bunch of attributes on it to specify columns, and then you can > simply assign to those attributes on an instance to set the values for > a row to be saved to the database. > > As Todd explains in further detail, the metaclass gets to do whatever it > likes. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > There is an excellent django book: http://prodjango.com/ There is a section that explains how models are created. Its a great and clear read. -- Joel Goldstick http://joelgoldstick.com/stats/birthdays -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about asyncio and blocking operations
"Ian Kelly" wrote in message
news:CALwzidnGbz7kM=d7mkua2ta9-csfn9u0ohl0w-x5bbixpcw...@mail.gmail.com...
On Jan 28, 2016 4:13 AM, "Frank Millman" wrote:
>
> I *think* I have this one covered. When the caller makes a request, it
creates an instance of an asyncio.Queue, and includes it with the request.
The db handler uses this queue to send the result back.
>
> Do you see any problem with this?
That seems reasonable to me. I assume that when you send the result back
you would be queuing up individual rows and not just sending a single
object across, which could be more easily with just a single future.
I have hit a snag. It feels like a bug in 'await q.get()', though I am sure
it is just me misunderstanding how it works.
I can post some working code if necessary, but here is a short description.
Here is the database handler - 'request_queue' is a queue.Queue -
while not request_queue.empty():
return_queue, sql = request_queue.get()
cur.execute(sql)
for row in cur:
return_queue.put_nowait(row)
return_queue.put_nowait(None)
request_queue.task_done()
The caller requests some data from the database like this.
return_queue = asyncio.Queue()
sql = 'SELECT ...'
request_queue.put((return_queue, sql))
while True:
row = await return_queue.get()
if row is None:
break
print('got', row)
return_queue.task_done()
The first time 'await return_queue.get()' is called, return_queue is empty,
as the db handler has not had time to do anything yet.
It is supposed to pause there, wait for something to appear in the queue,
and then process it. I have confirmed that the db handler populates the
queue virtually instantly.
What seems to happen is that it pauses there, but then waits for some other
event in the event loop to occur before it continues. Then it processes all
rows very quickly.
I am running a 'counter' task in the background that prints a sequential
number, using await asyncio.sleep(1). I noticed a short but variable delay
before the rows were printed, and thought it might be waiting for the
counter. I increased the sleep to 10, and sure enough it now waits up to 10
seconds before printing any rows.
Any ideas?
Frank
--
https://mail.python.org/mailman/listinfo/python-list
Warehouse, the new PyPi codebase
Hi folks, A couple of links for those who may be interested and haven't heard. http://pyfound.blogspot.co.uk/2016/01/welcome-to-warehouse.html http://pyfound.blogspot.co.uk/2016/01/postscript-to-warehouse-post.html -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: how to get python version ?
namenobodywa...@gmail.com wrote: > hi > > is there something analogous to sys.platform that lets you get the version of > python you're using? sorry if the question is too see-spot-run. thanks if you > can help > > peace > stm Since everyone else has answered the question you asked, let me answer the question I think you might have wanted to ask instead. If what you're trying to determine is simply whether you're running under Python 2 or 3, then the six package has booleans for six.PY2 and six.PY3, so you can say things like: if six.PY2: def my_fn(): ... else: def my_fn(): ... Note that checking compatibility this way (calling Python 2 "the weird one") rather than "if six.PY3" will keep your code compatible in the future as well. Bringing in an entire extra package just for one boolean seems excessive, but if you're actually making that check it's because you've got a bunch of 2/3 compatibility crap you're having to work out; six will help will all of that too. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about asyncio and blocking operations
2016-01-28 17:53 GMT+01:00 Ian Kelly : > On Thu, Jan 28, 2016 at 9:40 AM, Frank Millman wrote: > > > The caller requests some data from the database like this. > > > >return_queue = asyncio.Queue() > >sql = 'SELECT ...' > >request_queue.put((return_queue, sql)) > > Note that since this is a queue.Queue, the put call has the potential > to block your entire event loop. > > Actually, I don't think you actually need an asyncio.Queue. You could use a simple deque as a buffer, and call fetchmany() when it is empty, like that (untested): class AsyncCursor: """Wraps a DB cursor and provide async method for blocking operations""" def __init__(self, cur, loop=None): if loop is None: loop = asyncio.get_event_loop() self._loop = loop self._cur = cur self._queue = deque() def __getattr__(self, attr): return getattr(self._cur, attr) def __setattr__(self, attr, value): return setattr(self._cur, attr, value) async def execute(self, operation, params): return await self._loop.run_in_executor(self._cur.execute, operation, params) async def fetchall(self): return await self._loop.run_in_executor(self._cur.fetchall) async def fetchone(self): return await self._loop.run_in_executor(self._cur.fetchone) async def fetchmany(self, size=None): return await self._loop.run_in_executor(self._cur.fetchmany, size) async def __aiter__(self): return self async def __anext__(self): if self._queue.empty(): rows = await self.fetchmany() if not rows: raise StopAsyncIteration() self._queue.extend(rows) return self._queue.popleft() -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about asyncio and blocking operations
On Thu, Jan 28, 2016 at 2:23 PM, Maxime S wrote: > > 2016-01-28 17:53 GMT+01:00 Ian Kelly : >> >> On Thu, Jan 28, 2016 at 9:40 AM, Frank Millman wrote: >> >> > The caller requests some data from the database like this. >> > >> >return_queue = asyncio.Queue() >> >sql = 'SELECT ...' >> >request_queue.put((return_queue, sql)) >> >> Note that since this is a queue.Queue, the put call has the potential >> to block your entire event loop. >> > > Actually, I don't think you actually need an asyncio.Queue. > > You could use a simple deque as a buffer, and call fetchmany() when it is > empty, like that (untested): True. The asyncio Queue is really just a wrapper around a deque with an interface designed for use with the producer-consumer pattern. If the producer isn't a coroutine then it may not be appropriate. This seems like a nice suggestion. Caution is advised if multiple cursor methods are executed concurrently since they would be in different threads and the underlying cursor may not be thread-safe. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about asyncio and blocking operations
On Jan 28, 2016 3:07 PM, "Maxime Steisel" wrote: > > But it is a pretty strange idea to call two fetch*() method concurrently anyways. If you want to process rows concurrently and aren't concerned with processing them in order, it may be attractive to create multiple threads / coroutines, pass the cursor to each, and let them each call fetchmany independently. I agree this is a bad idea unless you use a lock to isolate the calls or are certain that you'll never use a dbapi implementation with threadsafety < 3. I pointed it out because the wrapper makes it less obvious that multiple threads are involved; one could naively assume that the separate calls are isolated by the event loop. -- https://mail.python.org/mailman/listinfo/python-list
Re: psss...I want to move from Perl to Python
On 01/28/2016 07:01 PM, Fillmore wrote: I learned myself Perl as a scripting language over two decades ago. All through this time, I would revert to it from time to time whenever I needed some text manipulation and data analysis script. My problem? maybe I am stupid, but each time I have to go back and re-learn the syntax, the gotchas, the references and the derefercing, the different syntax between Perl 4 and Perl 5, that messy CPAN in which every author seems to have a different ideas of how things should be done I get this feeling I am wasting a lot of time restudying the wheel each tim... I look and Python and it looks so much more clean add to that that it is the language of choice of data miners... add to that that iNotebook looks powerful Does Python have Regexps? How was the Python 2.7 vs Python 3.X solved? which version should I go for? Do you think that switching to Python from Perl is a good idea at 45? Where do I get started moving from Perl to Python? which gotchas need I be aware of? Thank you Python does "have" regex.. it is not part of the language, but is in a separate module. If you are used to the way that regex feels "natural" in perl, you will have some "adjusting" to do. Python 2 vs python 3 is anything but "solved". You will hear arguments in favor of both. I will not make a suggestion, because I do not have on my flame-resistant clothing I am adding Python 3 to my "bag of tricks" after many, many years of perl programming, and I am significantly older than you. I suggest that you just plunge in, and see what you think. It just is NOT that difficult. There are lots of tutorials and books.. some good, some less so. The tutorials on python.org are (IMHO) a very good place to start, though. Gotchas? If you are learning Python after years of perl use, the "gotchas" would be another book just by themselves. Be prepared to do a LOT more typing. Long and short of it... just dig in. I would offer one piece of advice, though... you will probably be better off if you simply view Python totally separately from perl.. sort of like learning a new spoken/written language. Don't try to view concepts as a translation from perl to Python , but rather just view Python as if you were learning to program from scratch. Comparisons will make it much more difficult, in several ways. Best of luck to you, and I hope that you find Python to be too your liking. Nathan -- https://mail.python.org/mailman/listinfo/python-list
Re: psss...I want to move from Perl to Python
On 28Jan2016 19:01, Fillmore wrote: I learned myself Perl as a scripting language over two decades ago. All through this time, I would revert to it from time to time whenever I needed some text manipulation and data analysis script. My problem? maybe I am stupid, but each time I have to go back and re-learn the syntax, the gotchas, the references and the derefercing, the different syntax between Perl 4 and Perl 5, that messy CPAN in which every author seems to have a different ideas of how things should be done Yes. CPAN always seemed a royal PITA to me too. Pip and Python virtualenvs are much cleaner IMO. I look and Python and it looks so much more clean Oh yes. I wasted quite a while with Perl before shifting, largely because I had a large base of personal Perl modules I used a lot and felt like the bootstrap process would be painful. I do wish I'd jumped in a few years earlier. Does Python have Regexps? It has a regexp module named "re" and they're the same syntax as perl. It doesn't have regexp operations as part of the language itself (no "=~ /regexp/" stuff) - you compile a regexp and then use it. This does mean you use regexps more deliberately and less frequently than in Perl, but that is a Good Thing. Regexps are far far overused in Perl and elsewhere. They are cumbersome, cryptic and error prone. Python's strings have a bunch of fixed-string methods available which cover a surprisingly large number of cases before you need the complexity of a regexp. How was the Python 2.7 vs Python 3.X solved? which version should I go for? Go for 3. While not everything has been ported, many many things have and you will mostly be fine. Python 3 has a much cleaner bytes vs text separation (which is also a Good Thing), some library cleanups, and all the new work goes to Python 3. Python 2 exists, but is essentially frozen. I try to write code which will work in both (easy most of the time), but some of my code will only work in Python 3 because it works with bytes and text, and going back to the ambiguity of Python 2 would be a disaster (in fact, the module I'm thinking of had many potential bugs which got cleaned up as a direct result of the move to 3). Do you think that switching to Python from Perl is a good idea at 45? Yes. Yes yes yes. Where do I get started moving from Perl to Python? There are tutorials, if that kind of thing works for you. My approach was to just start writing small things in Python. And when those things got reused, start making them modules (== "packages" in Perl speak). You will end up with a little suite of tools. Also, join the "tu...@python.org" list. That is where to go for new Pythoners seeking help. which gotchas need I be aware of? Hard to say, I found it depended on what I was doing. I think the biggest for all new users is probably the "default argument value" semantics. When you write a Python function: def foo(a, b): ... you can specify default values. For example: def (a, b=3): ... means you can call foo as foo(1) and it will be like calling it as foo(1, 3). The important point here is that the default values are computed at function definition time, _not_ at function call time. For constants like the above this doesn't matter, but consider this: def foo(a, b=[]): so that "b" defaults to being an empty list. Because "[]" is computed when the function is defined, you will be reusing the _same_ list whenever you call the function without specifying "b"; it won't stay an empty list, becuase it is not a _new_ empty list of each call. So if you had "b.append(4)" inside the function, the list would progressively accrue 4s, and so forth. For this reason the standard idiom for default arguments goes: def foo(a, b=None): if b is None: b = [] Since that test runs on each call (being inside the function), it does make a new empty list for "b" if it was not specified. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
psss...I want to move from Perl to Python
I learned myself Perl as a scripting language over two decades ago. All through this time, I would revert to it from time to time whenever I needed some text manipulation and data analysis script. My problem? maybe I am stupid, but each time I have to go back and re-learn the syntax, the gotchas, the references and the derefercing, the different syntax between Perl 4 and Perl 5, that messy CPAN in which every author seems to have a different ideas of how things should be done I get this feeling I am wasting a lot of time restudying the wheel each tim... I look and Python and it looks so much more clean add to that that it is the language of choice of data miners... add to that that iNotebook looks powerful Does Python have Regexps? How was the Python 2.7 vs Python 3.X solved? which version should I go for? Do you think that switching to Python from Perl is a good idea at 45? Where do I get started moving from Perl to Python? which gotchas need I be aware of? Thank you -- https://mail.python.org/mailman/listinfo/python-list
Re: psss...I want to move from Perl to Python
Fillmore writes:
> I look and Python and it looks so much more clean
Yes it is, I forgot everything I knew about Perl shortly after starting
to use Python.
> add to that that it is the language of choice of data miners...
There are a bunch of statistical libraries and that sort of thing but
you have to know what you're doing to use them. It's not a matter of
the language itself, which is pretty simple.
> add to that that iNotebook looks powerful
People seem to like it. I don't use it myself (haven't felt the need).
> Does Python have Regexps?
Not built into the language the way Perl regexps are. There's a regexp
library that's pretty capable, but you have to say things like
re.search('foo.*bar', x)
instead of x ~ /foo.*bar/ or whatever. Generally I'd say Perl has far
more built-in creature comforts where Python makes you call a library.
The capabilities end up being about the same but one could say Python
doesn't require you to remember as much stuff.
Perl 6 is another story, interesting but apparently monstrously
complicated. I haven't looked at it closely but followed some of the
development history, which was amazing.
> How was the Python 2.7 vs Python 3.X solved? which version should I go for?
It hasn't really been solved and probably never will be. It's probably
preferable to start with 3.x, but in practice the differences aren't
that large and you should be able to work with both. Despite the
screams of this newsgroup, most professional projects that I've been
involved with are still using 2.x.
> Do you think that switching to Python from Perl is a good idea at 45?
Sure.
> Where do I get started moving from Perl to Python?
Read the tutorial ( https://docs.python.org/3/tutorial/ )
and hack some code.
> which gotchas need I be aware of?
There's lots of minor ones, just like with any other complicated system.
You'll run into them as you go along, and deal with them. There's
nothing really horrible that I can think of.
--
https://mail.python.org/mailman/listinfo/python-list
Re: psss...I want to move from Perl to Python
On Thu, Jan 28, 2016 at 5:01 PM, Fillmore wrote: > > I learned myself Perl as a scripting language over two decades ago. All > through this time, I would revert to it from time to time whenever I needed > some text manipulation and data analysis script. > > My problem? maybe I am stupid, but each time I have to go back and re-learn > the syntax, the gotchas, the references and the derefercing, the different > syntax between Perl 4 and Perl 5, that messy CPAN in which every author > seems to have a different ideas of how things should be done I know what you mean about Perl. I have probably re-learned Perl 12 times. If you will not need to maintain any existing code, go directly to Python 3.x. Not every third-party support Python 3 yet, but it is coming for most. If you can eschew 2.x, then do. Maintaining a source base targeting both 2.x and 3.x can be done, but it is not always a simple picture. Learning a new language at 45 is a great idea. Do it again at 65. -- https://mail.python.org/mailman/listinfo/python-list
Re: psss...I want to move from Perl to Python
On Fri, Jan 29, 2016 at 1:06 PM, Paul Rubin wrote: > Fillmore writes: >> I look and Python and it looks so much more clean > > Yes it is, I forgot everything I knew about Perl shortly after starting > to use Python. https://xkcd.com/353/ Particularly the hover text. :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: psss...I want to move from Perl to Python
On Fri, Jan 29, 2016 at 11:01 AM, Fillmore wrote: > Does Python have Regexps? Yes! Although there are often other ways of doing things, so you don't *have* to use a regex for everything. But yes, Python has a powerful regular expression library. > How was the Python 2.7 vs Python 3.X solved? which version should I go for? Go for 3.x, unless you have a particular reason for using 2.7. There are heaps of cool new features in 3.x, and most of the modules you'll need support it. There's now very little reason for going 2.7 for new projects. > Do you think that switching to Python from Perl is a good idea at 45? Definitely. Never too old to learn new tricks. > Where do I get started moving from Perl to Python? Start with the Python tutorial in the docs; flip through that and get an idea of broadly how things work. Then port a simple script, so you know you can make Python do stuff for you. Get your code reviewed by experts (whether it works or not - you'll learn a lot from people saying "Hey, here's a different way you could do that"), and don't take anything personally. > which gotchas need I be aware of? Probably the biggest thing to take note of is the Python object model - how names and objects and assignment work. It's pretty straight-forward, but if it's not what you're used to, you'll need to get your head around it. Here's a good talk on the subject: http://nedbatchelder.com/text/names1.html > Thank you You're most welcome! And welcome to the list. This is a pretty friendly place; don't be afraid to post your code and ask for advice. All the best! ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: psss...I want to move from Perl to Python
On Fri, Jan 29, 2016 at 2:23 PM, Rick Johnson wrote: > On Thursday, January 28, 2016 at 6:28:25 PM UTC-6, Nathan Hilterbrand wrote: >> Python 2 vs python 3 is anything but "solved". You will hear arguments >> in favor of both. I will not make a suggestion, because I do not have >> on my flame-resistant clothing > > Indeed. About the only subject more volatile than the Py2<=>Py3 issue is > abortion. I'm fairly sure propane is more volatile... ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about asyncio and blocking operations
"Ian Kelly" wrote in message news:calwzidn6nft_o0cfhw1itwja81+mw3schuecadvcen3ix6z...@mail.gmail.com... As I commented in my previous message, asyncio.Queue is not thread-safe, so it's very important that the put calls here be done on the event loop thread using event_loop.call_soon_threadsafe. This could be the cause of the strange behavior you're seeing in getting the results. Using call_soon_threadsafe makes all the difference. The rows are now retrieved instantly. I have read the other messages, and I can see that there are some clever ideas there. However, having found something that seems to work and that I feel comfortable with, I plan to run with this for the time being. Thanks to all for the very stimulating discussion. Frank -- https://mail.python.org/mailman/listinfo/python-list
Re: psss...I want to move from Perl to Python
On Thursday, January 28, 2016 at 6:28:25 PM UTC-6, Nathan Hilterbrand wrote: > Python 2 vs python 3 is anything but "solved". You will hear arguments > in favor of both. I will not make a suggestion, because I do not have > on my flame-resistant clothing Indeed. About the only subject more volatile than the Py2<=>Py3 issue is abortion. -- https://mail.python.org/mailman/listinfo/python-list
Python Scraps, Napkins & Nuggets
Hi This may seem an odd request, however i thought i would ask do you have any diagrams, scribbles, presentations you have done when coaching someone at work that just seems to work for others consistently? In coaching non programming topics at work i have noticed the variance in learning styles and many times my off hand scribbles and comments are what gets their attention and understanding. I would love to see what you have created. If its python awesome and in keeping with list topic if not i wont tell. Thanks Sayth -- https://mail.python.org/mailman/listinfo/python-list
Re: psss...I want to move from Perl to Python
On 01/28/2016 07:34 PM, Chris Angelico wrote: > On Fri, Jan 29, 2016 at 1:06 PM, Paul Rubin wrote: >> Fillmore writes: >>> I look and Python and it looks so much more clean >> >> Yes it is, I forgot everything I knew about Perl shortly after starting >> to use Python. > > https://xkcd.com/353/ > > Particularly the hover text. :) Another particularly appropriate xkcd comic, given the OP's experiencein Perl is this one (and the hover text again!): https://xkcd.com/224/ -- https://mail.python.org/mailman/listinfo/python-list
Re: psss...I want to move from Perl to Python
On Fri, 29 Jan 2016 11:25 am, Chris Angelico wrote: > Probably the biggest thing to take note of is the Python object model > - how names and objects and assignment work. It's pretty > straight-forward, but if it's not what you're used to, you'll need to > get your head around it. Here's a good talk on the subject: > > http://nedbatchelder.com/text/names1.html Every time I make a half-hearted attempt to learn enough Perl syntax to get started, I keep running into the differences between $foo, %foo and @foo and dire warnings about what happens if you use the wrong sigil, and then I get confused. Is there a good discussion of how names and references work in Perl, equivalent to Ned's discussion? -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: >>> %matplotlib inline results in SyntaxError: invalid syntax
On Fri, Jan 29, 2016 at 6:04 PM, Mike S via Python-list wrote: > > I get an error on the last line. I am running this code in Idle Python 3.4.4 > Shell... > > Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit > (Intel)] on win32 > Type "copyright", "credits" or "license()" for more information. import numpy as np import pandas as pd import random import matplotlib.pyplot as plt %matplotlib inline > SyntaxError: invalid syntax > > What am I doing wrong? Suggested reading? That's a magic command for ipython - it's not actual Python syntax. You can either install ipython and do the exercises there, or continue with Idle and accept that things will be slightly different. Skim through a few more of the examples; do you see more lines starting with percent signs? If not, you can probably get away with using any interactive Python interpreter (including Idle as you're using above). Alternatively, take the certain route and install ipython - then everything should work. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
