Re: CSV File Sink in Streaming Use Case

2023-03-12 Thread Shammon FY
Hi Chirag CSVBulkWriter implements BulkWriter and no special methods are added in CSVBulkWriter. I think you can use BulkWriter instead of CSVBulkWriter in your application directly. You can have a try, thanks Best, Shammon On Fri, Mar 10, 2023 at 4:05 PM Shammon FY wrote: > >

Re: CSV File Sink in Streaming Use Case

2023-03-10 Thread Chirag Dewan via user
Thanks for the reply Shammom. I looked at the DataStreamCsvITCase - it gives a very good example. I can implement something similar. However, the CSVBulkWriter that is uses to create a factory, has a default package access which can be accessed from this test case, but not from my application. 

Re: CSV File Sink in Streaming Use Case

2023-03-07 Thread ramkrishna vasudevan
Hi all, One thing to note is that, the CSVBulkReader does not support the splittable property. Previously with TextInputFormat we were able to use the block size to split them, but in Streaming world this is not there. Regards Ram On Wed, Mar 8, 2023 at 7:22 AM yuxia wrote: > Hi, as the doc sa

Re: CSV File Sink in Streaming Use Case

2023-03-07 Thread yuxia
Hi, as the doc said: 'The BulkFormat reads and decodes batches of records at a time.' So, the bulk is not binded to column format, the bulk writer for csv is indeed implemented in the Flink code. Actaully, you can use either Row or Bulk depending on what style you would like to write data. As

Re: CSV File Sink in Streaming Use Case

2023-03-07 Thread Shammon FY
Hi You can create a `BulkWriter.Factory` which will create `CsvBulkWriter` and create `FileSink` by `FileSink.forBulkFormat`. You can see the detail in `DataStreamCsvITCase.testCustomBulkWriter` Best, Shammon On Tue, Mar 7, 2023 at 7:41 PM Chirag Dewan via user wrote: > Hi, > > I am working o