Re: Parquet S3 Sink Part files are not rolling over with checkpoint

2020-04-09 Thread Roshan Punnoose
something odd going on? On Thu, Apr 9, 2020 at 7:49 AM Roshan Punnoose wrote: > Nope just the s3a. I'll keep looking around to see if there is anything > else I can see. If you think of anything else to try, let me know. > > On Thu, Apr 9, 2020, 7:41 AM Kostas Kloudas wrote: > &

Re: Parquet S3 Sink Part files are not rolling over with checkpoint

2020-04-09 Thread Roshan Punnoose
as the scheme for s3. > Are you using "s3p" for Presto? This should also be done in order for > Flink to understand where to use the one or the other. > > On Thu, Apr 9, 2020 at 1:30 PM Roshan Punnoose wrote: > > > > Lastly, could it be the way I built the flink image

Re: Parquet S3 Sink Part files are not rolling over with checkpoint

2020-04-09 Thread Roshan Punnoose
Lastly, could it be the way I built the flink image for kube? I added both the presto and Hadoop plugins On Thu, Apr 9, 2020, 7:29 AM Roshan Punnoose wrote: > Sorry realized this came off the user list by mistake. Adding the thread > back in. > > On Thu, Apr 9, 2020, 7:26 AM Ros

Re: Parquet S3 Sink Part files are not rolling over with checkpoint

2020-04-09 Thread Roshan Punnoose
Sorry realized this came off the user list by mistake. Adding the thread back in. On Thu, Apr 9, 2020, 7:26 AM Roshan Punnoose wrote: > Yes sorry, no errors on the task manager. However, I am new to flink so > don't know all the places to look for the logs. Been looking at the tas

Parquet S3 Sink Part files are not rolling over with checkpoint

2020-04-08 Thread Roshan Punnoose
Hi, I am trying to get the parquet writer to write to s3; however, the files do not seem to be rolling over. The same file "part-0-0.parquet" is being created each time. Like the 'partCounter" is not being updated? Maybe the Bucket is being recreated each time? I don't really know... Here are some