Re: pipeline with parquet and sql

2018-08-01 Thread Chamikara Jayalath
exact requirement. Thanks, Cham [1] https://lists.apache.org/thread.html/fe327866c6c81b7e55af28f81cedd9b2e588279def330940e8b8ebd7@%3Cdev.beam.apache.org%3E > > > > Regards, > > Akanksha > > > -- > *From:* Łukasz Gajowy > *Sent:* Tuesday, July 31, 2018 12:43:32 PM > *To:* user@beam.apa

Re: pipeline with parquet and sql

2018-08-01 Thread Akanksha Sharma B
Regards, Akanksha From: Łukasz Gajowy Sent: Tuesday, July 31, 2018 12:43:32 PM To: user@beam.apache.org Cc: d...@beam.apache.org Subject: Re: pipeline with parquet and sql In terms of schema and ParquetIO source/sink, there was an answer in some previous thread: Currently

Re: pipeline with parquet and sql

2018-07-31 Thread Łukasz Gajowy
ibed below was understandable. I hope it is a >> valid use-case. Please let me know if I need to explain the scenario >> better. >> >> >> Regards, >> >> Akanksha >> >> ---------- >> *From:* Akanksha Sharma B >>

Re: pipeline with parquet and sql

2018-07-31 Thread Łukasz Gajowy
andable. I hope it is a > valid use-case. Please let me know if I need to explain the scenario > better. > > > Regards, > > Akanksha > > -- > *From:* Akanksha Sharma B > *Sent:* Friday, July 27, 2018 9:44 AM > *To:* d...@beam.apache.

Re: pipeline with parquet and sql

2018-07-31 Thread Akanksha Sharma B
Sent: Friday, July 27, 2018 9:44 AM To: d...@beam.apache.org Subject: Re: pipeline with parquet and sql Hi, Please consider following pipeline:- Source is Parquet file, having hundreds of columns. Sink is Parquet. Multiple output parquet files are generated after applying some sql joins. Sql

pipeline with parquet and sql

2018-07-24 Thread Akanksha Sharma B
Hi, Please consider following pipeline:- Source is Parquet file, having hundreds of columns. Sink is Parquet. Multiple output parquet files are generated after applying some sql joins. Sql joins to be applied differ for each output parquet file. Lets assume we have a sql queries generator or