Make synchronous generator from an asynchronous generator
Hello, I have recently got the need to convert a synchronous function to asynchronous function because I needed to run three of them concurrently. Since I am using Python 3.6, I used async def and asyncio and that has worked out great. Thank you guys for making this possible. Because I wanted to keep the synchronous function for scripts which used it, without unnecessarily duplicating the code, I built also a synchronous function from this new asynchronous one, like that: def acquire_to_files(self, *args, **kwargs): loop = asyncio.get_event_loop() loop.run_until_complete(self.acquire_to_files_async(*args, **kwargs)) loop.close() So far, it works fine. I am quite happy. I can await acquire_to_files_async in situations where I need concurrency, and call acquire_to_files in situations where I don't, and no code is duplicated. Now I wish to do the same game with a generator (which happens to be called by the acquire_to_files_async function). So I made it into an asynchronous generator, and it works as expected. My problem: how do I proceed if I wish to build a synchronous generator from this asynchronous generator ? (for situations where concurrency is not needed) i.e. what is the equivalent of the above snippet for generators ? Thanks, Julien PS: I was surprised however that enumerate does not work when applied to an asynchronous generator, i.e.: async for ii, img in enumerate(async_gen()): pass raises TypeError: 'async_gen' object is not iterable. -- https://mail.python.org/mailman/listinfo/python-list
Re: Make synchronous generator from an asynchronous generator
On Thu, Mar 15, 2018 at 1:35 PM, Julien Salort wrote: > Because I wanted to keep the synchronous function for scripts which used it, > without unnecessarily duplicating the code, I built also a synchronous > function from this new asynchronous one, like that: > > def acquire_to_files(self, *args, **kwargs): > loop = asyncio.get_event_loop() > loop.run_until_complete(self.acquire_to_files_async(*args, **kwargs)) > loop.close() Note that this function can't be called more than once, because it closes the event loop at the end. Next time you call it it will get the closed event loop and try to schedule on it and then raise an exception because it's closed. > So far, it works fine. I am quite happy. > > I can await acquire_to_files_async in situations where I need concurrency, > and call acquire_to_files in situations where I don't, and no code is > duplicated. > > > Now I wish to do the same game with a generator (which happens to be called > by the acquire_to_files_async function). So I made it into an asynchronous > generator, and it works as expected. > > My problem: how do I proceed if I wish to build a synchronous generator from > this asynchronous generator ? > > (for situations where concurrency is not needed) > > i.e. what is the equivalent of the above snippet for generators ? Well, it's not pretty, but this seems to work: async def agen(): yield 1 yield 2 yield 3 def gen(): loop = asyncio.get_event_loop() ait = agen().__aiter__() try: while True: yield loop.run_until_complete(ait.__anext__()) except StopAsyncIteration: return py> list(gen()) [1, 2, 3] It would take some additional code if you wanted to wire up gen().send() and gen().throw() to work as well, but it should be doable. -- https://mail.python.org/mailman/listinfo/python-list
Re: Make synchronous generator from an asynchronous generator
Le 16/03/2018 à 16:55, Ian Kelly a écrit : Note that this function can't be called more than once, because it closes the event loop at the end. Next time you call it it will get the closed event loop and try to schedule on it and then raise an exception because it's closed. Ah ok. So, if I replace get_event_loop() with new_event_loop(), then it's fine ? Well, it's not pretty, but this seems to work: [...] Thanks ! Julien -- https://mail.python.org/mailman/listinfo/python-list
Re: urllib.request.urlopen fails with https - SOLVED
Thank you, thank you, thank you. That fixed it (at least on my computer, I'll see if I can do that at my school). Irv > On Mar 15, 2018, at 7:39 PM, Ned Deily wrote: > > On 2018-03-14 18:04, Irv Kalb wrote: >> File >> "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", >> line 1320, in do_open >> raise URLError(err) >> urllib.error.URLError: > certificate verify failed (_ssl.c:749)> > > If you are using Python 3.6 for macOS from a python.org installer, did > you follow the instructions displayed in the installer ReadMe and also > saved at: > > /Applications/Python 3.6/ReadMe.rtf > > to run the "Install Certificates.command" ? > > Either double-click on it in the Finder or, from a shell command line, type: > > open "/Applications/Python 3.6/Install Certificates.command" > > > Certificate verification and OpenSSL > > **NEW** This variant of Python 3.6 now includes its own private copy of > OpenSSL 1.0.2. Unlike previous releases, the deprecated Apple-supplied > OpenSSL libraries are no longer used. This also means that the trust > certificates in system and user keychains managed by the Keychain Access > application and the security command line utility are no longer used as > defaults by the Python ssl module. For 3.6.0, a sample command script > is included in /Applications/Python 3.6 to install a curated bundle of > default root certificates from the third-party certifi package > (https://pypi.python.org/pypi/certifi). If you choose to use certifi, > you should consider subscribing to the project's email update service to > be notified when the certificate bundle is updated. > > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Treatment of NANs in the statistics module
The bug tracker currently has a discussion of a bug in the median(), median_low() and median_high() functions that they wrongly compute the medians in the face of NANs in the data: https://bugs.python.org/issue33084 I would like to ask people how they would prefer to handle this issue: (1) Put the responsibility on the caller to strip NANs from their data. If there is a NAN in your data, the result of calling median() is implementation-defined. This is the current behaviour, and is likely to be the fastest. (2) Return a NAN. (3) Raise an exception. (4) median() should strip out NANs. (5) All of the above, selected by the caller. (In which case, which would you prefer as the default?) Thank you. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Treatment of NANs in the statistics module
On 3/16/2018 7:16 PM, Steven D'Aprano wrote: The bug tracker currently has a discussion of a bug in the median(), median_low() and median_high() functions that they wrongly compute the medians in the face of NANs in the data: https://bugs.python.org/issue33084 I would like to ask people how they would prefer to handle this issue: (1) Put the responsibility on the caller to strip NANs from their data. 1 to 3 all put responsibility on the caller to strip NANs to get a sane answer. The question is what to do if the caller does not (1) If there is a NAN in your data, the result of calling median() is implementation-defined. This is the current behaviour, and is likely to be the fastest. I hate implementation-defined behavior. (2) Return a NAN. I don't like NANs as implemented and used, or unused. (3) Raise an exception. That leave this. (4) median() should strip out NANs. and then proceed in a deterministic fashion to give an answer. (5) All of the above, selected by the caller. (In which case, which would you prefer as the default?) I would frame this as an alternative: 'ignore_nan=False (3) or =True (4). Or nan='ignore' versus 'raise' (or 'strict') These are like the choices encoding. What do statistics.mean() and other functions do? The proposed quantile() will have the same issue. BMDP and other packages had and have general options for dealing with missing values, and that is what NAN is. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Treatment of NANs in the statistics module
On Fri, 16 Mar 2018 22:08:42 -0400, Terry Reedy wrote:
> On 3/16/2018 7:16 PM, Steven D'Aprano wrote:
>> The bug tracker currently has a discussion of a bug in the median(),
>> median_low() and median_high() functions that they wrongly compute the
>> medians in the face of NANs in the data:
[...]
>> (4) median() should strip out NANs.
>
> and then proceed in a deterministic fashion to give an answer.
Indeed.
>> (5) All of the above, selected by the caller. (In which case, which
>> would you prefer as the default?)
>
> I would frame this as an alternative: 'ignore_nan=False (3) or =True
> (4). Or nan='ignore' versus 'raise' (or 'strict') These are like the
> choices encoding.
That's what I'm thinking. But which would you have as default? I'm
guessing "raise".
> What do statistics.mean() and other functions do?
Because they do actual arithmetic on the data points, the presence of a
float NAN will propagate through to the end of the calculation.
A Decimal NAN will behave in three different ways:
- a signalling NAN will raise when any operation is performed on it;
- a quiet NAN will raise if the current Decimal context is set to raise
on invalid operations, and propagate otherwise.
It seems reasonable for median() to handle NANs better than it currently
does, in which case I'd expect the rest of the statistics module to do
the same.
> The proposed quantile() will have the same issue.
>
> BMDP and other packages had and have general options for dealing with
> missing values, and that is what NAN is.
I don't wish to get into an argument about whether NANs are missing
values or could be missing values, but R supports both NANs and a
dedicated NA ("not available") missing value. By default, either will
cause median to return NA, but there is an option to ignore NANs:
> median(c(1, 2, 3, 4, NaN))
[1] NA
> median(c(1, 2, 3, 4, NaN), na.rm=TRUE)
[1] 2.5
--
Steve
--
https://mail.python.org/mailman/listinfo/python-list
A better solution
Hello,
im not entirely happy with my solution and would love to hear your
suggestions on how to improve the solution.
I simplified the task while keeping the code working.
Task:
financial accounts are described by XML documents. I want to architecture
the code to be easily extendible ( easy to add more accounts) and with
minimum repetitive code in each class.
All XML documents have a few common (SSN, first/last, sex, etc.) and unique
fields. Both common and unique fields may or may not be present in the
passed parameters.
I came up with the following solution:
one class as a common "account" with children that have specific
functionality for their specific fields.
What i _really_ don't like is the a repetitive code in each _init_ and
"update" method.
Code:
-start
from flask import Flask, jsonify, request
import xml_oper
app = Flask(__name__)
@app.route('/')
def index():
return "Hello, World!"
@app.route('/INT/API/1.0/Accounts/IRA_Simple', methods =['POST'])
def post_IRA_Simple():
json = request.get_json()
account = xml_oper.account_type['IRA']
account.update_xml(output_file_name='IRA.xml',json=json)
return jsonify( 'all good ), 201
if __name__ == '__main__':
app.run(debug=True)
xml_oper,py
from lxml import etree
from bs4 import BeautifulSoup as Soup
class Account():
'''
Parent for all account types
'''
_file_path_source = 'XXX/XML_Docs/Sample_ApplicationXML/'
_file_path_dest = 'X/src/XML/'
def __init__(self):
pass
def _open_xml(self, file_name):
self._soup = Soup(open(Account._file_path_source + file_name,
'r'), 'lxml')
def _write_xml(self, file_name):
f = open(Account._file_path_dest + file_name, 'w')
f.write(self._soup.prettify())
f.close()
def _update(self, json):
# update only common fields
self._soup.application.customer['email'] = 'b...@email.com'
class IRA(Account):
def __init__(self, source_file_name):
super().__init__()
super()._open_xml(file_name=source_file_name)
def update_xml(self, output_file_name, json):
super()._update(json= json)
# update specific to IRA account fields
# values may or may not exit
super()._write_xml(file_name=output_file_name)
class Checking(Account):
def __init__(self, source_file_name):
super().__init__()
super()._open_xml(file_name=source_file_name)
def update_xml(self, output_file_name, json):
super()._update(json= json)
# update specific to CHK account fields
# values may or may not exit
super()._write_xml(file_name=output_file_name)
account_type = {
'IRA' : IRA('IRA_Sample.xml'),
'Checking' : Checking('Checking_Sample.xml')
}
--- end
Thank you.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Treatment of NANs in the statistics module
Steven D'Aprano writes: > I would like to ask people how they would prefer to handle [the > computation of median when the data set contains NaN]: > > (1) Put the responsibility on the caller to strip NANs from their > data. If there is a NAN in your data, the result of calling median() > is implementation-defined. This is the least Pythonic; there is no good reason IMO for specifying a behaviour in the implementation. > (2) Return a NAN. I think this makes the most sense. A result of ‘NaN’ communicates effectively that there is no meaningful number which is the median of the data set. It is then up to the caller to decide, based on that unambiguous result, what to do about it. > (3) Raise an exception. To raise an exception might be justifiable if there were no better option; but it creates the problem that a program which has not been tested for data containing NaN will catastrophically fail, instead of continuing. To raise an exception also forecloses the decision of whether “NaN in the data set” is a problem. That is, IMO, a decision for the caller to make; it could well be that the caller is happy to use ‘NaN’ as the result. That should IMO be up to the caller to decide. > (4) median() should strip out NANs. Too much magic. This needlessly conflates different inputs as though they are the same. > (5) All of the above, selected by the caller. (In which case, which > would you prefer as the default?) Please, no. The function should have one well-defined behaviour, and all the options (raise an exception, strip the NaN from the input, etc.) can be done in a wrapper function by those who want it done. My vote, among those options, goes for “when the input contains ‘NaN’, return ‘NaN’”. -- \ “But Marge, what if we chose the wrong religion? Each week we | `\ just make God madder and madder.” —Homer, _The Simpsons_ | _o__) | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Treatment of NANs in the statistics module
Ben Finney writes: > Steven D'Aprano writes: > > > (1) […] If there is a NAN in your data, the result of calling > > median() is implementation-defined. > > This is the least Pythonic; there is no good reason IMO for specifying a > behaviour in the implementation. That's a confused statement. I mean to say: there's no good reason IMO to leave the behaviour undefined in the function specification. -- \“I don't accept the currently fashionable assertion that any | `\ view is automatically as worthy of respect as any equal and | _o__) opposite view.” —Douglas Adams | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
