Re: Using Spark SQL with multiple (avro) files

2015-01-15 Thread David Jones
4, 2015 at 3:53 PM, David Jones wrote: > Should I be able to pass multiple paths separated by commas? I haven't > tried but didn't think it'd work. I'd expected a function that accepted a > list of strings. > > On Wed, Jan 14, 2015 at 3:20 PM, Yana Kadiyska

Re: Using Spark SQL with multiple (avro) files

2015-01-14 Thread David Jones
; Here was my question for reference: > > http://mail-archives.apache.org/mod_mbox/spark-user/201412.mbox/%3ccaaswr-5rfmu-y-7htluj2eqqaecwjs8jh+irrzhm7g1ex7v...@mail.gmail.com%3E > > On Wed, Jan 14, 2015 at 4:34 AM, David Jones > wrote: > >> Hi, >> >> I have a p

Using Spark SQL with multiple (avro) files

2015-01-14 Thread David Jones
Hi, I have a program that loads a single avro file using spark SQL, queries it, transforms it and then outputs the data. The file is loaded with: val records = sqlContext.avroFile(filePath) val data = records.registerTempTable("data") ... Now I want to run it over tens of thousands of Avro file