Re: read multiple files

Hi Divya, There are a number of ways you can do this Get today's date in epoch format. These are my package imports import java.util.Calendar import org.joda.time._ import java.math.BigDecimal import java.sql.{Timestamp, Date} import org.joda.time.format.DateTimeFormat // Get epoch time now sc

Re: read multiple files

If you're up for a fancy but excellent solution: - Store your data in Cassandra. - Use the expiring data feature (TTL) so data will automatically be removed a month later. - Now in your Spark process, just read f

Re: Read multiple files from S3

textFile does reads all files in a directory. We have modified the sparkstreaming code base to read nested files from S3, you can check this function