Re: Flink to S3 streaming

2017-04-18 Thread Aljoscha Krettek
Hi, You would have to write your own SinkFunction that uses the AWS S3 sdk to write to S3. You might be interested in the work proposed in this Jira: https://issues.apache.org/jira/browse/FLINK-6306 As to buffering elements, I’m afraid you woul

Flink to S3 streaming

2017-04-07 Thread pradeep s
Hi, I have a use case to stream messages from Kafka to Amazon S3. I am not using the s3 file system way since i need to have Object tags to be added for each object written in S3. So i am planning to use the AWS S3 sdk . But i have a query on how to hold the data till the message size is in few MBs