I'm not able to use asyncIO because the file will not fit in memory. I
thought that flatmap will allow me to enrich/process records while
downloading instead of waiting for the whole file to get downloaded. The
solution works but its not scalable because i'm not able to use
AsynFunction in Flatmap.
You can use flatMap to flatten and have an asyncIO after it.
On Wed, Mar 9, 2022 at 8:08 AM Diwakar Jha wrote:
> Thanks Gen, I will look into customized Source and SpiltEnumerator.
>
> On Mon, Mar 7, 2022 at 10:20 PM Gen Luo wrote:
>
>> Hi Diwakar,
>>
>> An asynchronous flatmap function without
Thanks Gen, I will look into customized Source and SpiltEnumerator.
On Mon, Mar 7, 2022 at 10:20 PM Gen Luo wrote:
> Hi Diwakar,
>
> An asynchronous flatmap function without the support of the framework can
> be problematic. You should not call collector.collect outside the main
> thread of the
Hi Diwakar,
An asynchronous flatmap function without the support of the framework can
be problematic. You should not call collector.collect outside the main
thread of the task, i.e. outside the flatMap method.
I'd suggest using a customized Source instead to process the files, which
uses a SplitE
