Could probably jury rig it together with a Queue. (Will need to run Parquet in
a separate thread.)
class FileLikeIter:
def __init__(self):
self.q = Queue(1)
def write(self, data):
self.q.put(data)
def __iter__(self):
try:
while True:
yield self.q.get()
except Empty:
...
— Theron
> On Aug 9, 2022, at 11:27 AM, Michael Merickel <[email protected]> wrote:
>
> This harkens back to the discouraged write() callable in WSGI PEP 3333
> returned by the start_response() invocation. The PEP as well as Pyramid as a
> framework would encourage you to map the logic into an app_iter as Bert
> suggested.
>
> I think you'll want to define a file-like object that you can write to and
> set as the app_iter. The question will be whether you try to do this by
> writing from a separate thread or in some other way because once you return
> control to the WSGI server to iterate on your app_iter then you are no longer
> in control - you'll need some buffer between where you're generating your
> parquet file and what you're returning from the app_iter. I don't think a
> simple tempfile is good enough because you want the app_iter to wait instead
> of stopping when it hits an EOF unless you know that you've reached the end
> of the buffer.
>
> On Tue, Aug 9, 2022 at 12:59 PM Bert JW Regeer <[email protected]
> <mailto:[email protected]>> wrote:
> Some WSGI servers pass you the raw file descriptor as wsgi.input, but this is
> not guaranteed (wsgiref does for instance). Instead you should return an
> iterator that can be read incrementally so that your WSGI server can chunk
> responses.
>
>
>
>> On Aug 9, 2022, at 10:38, Mikko Ohtamaa <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Hi all,
>>
>> I'd like to stream dynamically generated Parquet-files from Pyramid server.
>> Parquet library itself offers writing to any file-like object. I am aware of
>> app_iter and FileResponse interfaces in Pyramid. However, does Pyramid (or
>> any example or utility class) offer a Python file-like interface, where I
>> could just dynamically .write() stuff?
>>
>> Br,
>> Mikko
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected]
>> <mailto:[email protected]>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/pylons-discuss/CAK8RCUudomfEGYnqAUk57XgOvatDLtFGGk%2Be5tFzu0w81Ez4Lg%40mail.gmail.com
>>
>> <https://groups.google.com/d/msgid/pylons-discuss/CAK8RCUudomfEGYnqAUk57XgOvatDLtFGGk%2Be5tFzu0w81Ez4Lg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected]
> <mailto:[email protected]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/2DB1D478-20A5-4F02-96BE-8F0EA792AFBF%400x58.com
>
> <https://groups.google.com/d/msgid/pylons-discuss/2DB1D478-20A5-4F02-96BE-8F0EA792AFBF%400x58.com?utm_medium=email&utm_source=footer>.
>
>
> --
>
> Michael
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected]
> <mailto:[email protected]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwHF%2BJjGydJ28bHneP1_GXvN_P%3DyN3W2FnSAQ_Cea-xbjw%40mail.gmail.com
>
> <https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwHF%2BJjGydJ28bHneP1_GXvN_P%3DyN3W2FnSAQ_Cea-xbjw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/E70BB1DE-7B0B-4A0B-AE57-DB5F9F7DE15A%40luhn.com.
