Re: Spark Streaming - Inserting into Tables

2015-07-14 Thread Tathagata Das
Why is .remember not ideal? On Sun, Jul 12, 2015 at 7:22 PM, Brandon White wrote: > Hi Yin, > > Yes there were no new rows. I fixed it by doing a .remember on the > context. Obviously, this is not ideal. > > On Sun, Jul 12, 2015 at 6:31 PM, Yin Huai wrote: > >> Hi Brandon, >> >> Can you explai

Re: Spark Streaming - Inserting into Tables

2015-07-12 Thread Brandon White
Hi Yin, Yes there were no new rows. I fixed it by doing a .remember on the context. Obviously, this is not ideal. On Sun, Jul 12, 2015 at 6:31 PM, Yin Huai wrote: > Hi Brandon, > > Can you explain what did you mean by "It simply does not work"? You did > not see new data files? > > Thanks, > >

Re: Spark Streaming - Inserting into Tables

2015-07-12 Thread Yin Huai
Hi Brandon, Can you explain what did you mean by "It simply does not work"? You did not see new data files? Thanks, Yin On Fri, Jul 10, 2015 at 11:55 AM, Brandon White wrote: > Why does this not work? Is insert into broken in 1.3.1? It does not throw > any errors, fail, or throw exceptions. I

Spark Streaming - Inserting into Tables

2015-07-10 Thread Brandon White
Why does this not work? Is insert into broken in 1.3.1? It does not throw any errors, fail, or throw exceptions. It simply does not work. val ssc = new StreamingContext(sc, Minutes(10)) val currentStream = ssc.textFileStream(s"s3://textFileDirectory/") val dayBefore = sqlContext.jsonFile(s"s3://