Re: SQLContext load. Filtering files

If you have enabled checkpointing the spark will handle that for you. Thanks Best Regards On Thu, Aug 27, 2015 at 4:21 PM, Masf wrote: > Thanks Akhil, I will have a look. > > I have a dude regarding to spark streaming and filestream. If spark > streaming crashs and while spark was down new file

Re: SQLContext load. Filtering files

Thanks Akhil, I will have a look. I have a dude regarding to spark streaming and filestream. If spark streaming crashs and while spark was down new files are created in input folder, when spark streaming is launched again, how can I process these files? Thanks. Regards. Miguel. On Thu, Aug 27,

Re: SQLContext load. Filtering files

Have a look at the spark streaming. You can make use of the ssc.fileStream. Eg: val avroStream = ssc.fileStream[AvroKey[GenericRecord], NullWritable, AvroKeyInputFormat[GenericRecord]](input) You can also specify a filter function