Why don’t you get an S3 notification on SQS and do the actions from there?
You will probably need to write the content of the files to a no sql database . Alternatively send the s3 notification to Kafka and read flink from there. https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html > Am 01.09.2020 um 16:46 schrieb orionemail <orionem...@protonmail.com>: > > > Hi, > > I have a S3 bucket that is continuously written to by millions of devices. > These upload small compressed archives. > > What I want to do is treat the tar gzipped (.tgz) files as a streaming source > and process each archive. The archive contains three files that each might > need to be processed. > > I see that > env.readFile(f, bucket, FileProcessingMode.PROCESS_CONTINUOUSLY, > 10000L).print(); > might do what I need, but I am unsure how best to implement 'f' - the > InputFileFormat. Is there a similar example for me to reference? > > Or is this idea not workable with this method? I need to ensure exactly once, > and also trigger removal of the files after processing. > > Thanks, > > > Sent with ProtonMail Secure Email. >
