Hi Farshid, Take a look at this example on github - https://github.com/hienluu/structured-streaming-sources.
Cheers, Hien On Thu, Jul 12, 2018 at 12:52 AM Farshid Zavareh <fhzava...@gmail.com> wrote: > Hello. > > I need to create a custom streaming source by extending *FileStreamSource*. > The idea is to override *commit*, so that processed files (S3 objects in > my case) are renamed to have a certain prefix. However, I don't know how to > use this custom source. Obviously I don't want to compile Spark -- the > application will be running on Amazon EMR clusters. > > Thanks, > Farshid > -- Regards,
