Spark not working with Hadoop 4mc compression

found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:132) ... 70 more Thank you, Abhijeet Kumar

Re: Why does join use rows that were sent after watermark of 20 seconds?

> If you want the intended behaviour you should add time stamp conditions or > window operator in join condition > > > > On Mon, 10 Dec 2018 at 5:23 PM, Abhijeet Kumar <mailto:abhijeet.ku...@sentienz.com>> wrote: > Hello, > > I’m using watermark to

Why does join use rows that were sent after watermark of 20 seconds?

even after watermark got finished Spark is holding the data in memory. I even tried after 45 seconds and that was getting joined too. I’m sending streams from two Kafka queues and tstamp_trans I’m creating with current timestamp values. This is creating confusion in my mind regarding watermark. Thank you, Abhijeet Kumar

Join happening after watermark time

ems like even after watermark got finished it’s holding the data in memory. I tried even after 45 seconds and that was getting joined too. This is creating confusion in my mind regarding watermark. Thank you, Abhijeet Kumar

Re: Spark Streaming join taking long to process

are cores on the machine." > > Just check if this is reducing the time taken for processing, since by this > local[*] we are going to use all cores available, not just one core? > > Regards, > Sriram Srikanth > > On Tue, Nov 27, 2018 at 1:46 PM Abhijeet Kumar

Not able to overwrite cassandra table using Spark

but when I’m checking the Cassandra table back, there is no content inside it. Everything is deleted. I’m really worried about this behaviour because this may delete some useful content (I’m sure about overwriting the content and fully understand the consequences). Thanks, Abhijeet Kumar

Re: Spark / Scala code not recognising the path?

med. > > On 9. Jun 2018, at 09:44, Abhijeet Kumar > wrote: > > I need to rename the file. I can write a separate program for this, I > think. > > Thanks, > Abhijeet Kumar > > On 09-Jun-2018, at 1:10 PM, Jörn Franke wrote: > > That would be an anti pattern an

Re: Spark / Scala code not recognising the path?

I need to rename the file. I can write a separate program for this, I think. Thanks, Abhijeet Kumar > On 09-Jun-2018, at 1:10 PM, Jörn Franke wrote: > > That would be an anti pattern and would lead to bad software. > Please don’t do it for the sake of the people that use your soft

Re: Spark / Scala code not recognising the path?

Can you please tell the estimated time. So, that my program will wait for that time period. Thanks, Abhijeet Kumar > On 09-Jun-2018, at 12:01 PM, Jörn Franke wrote: > > You need some time until the information of the file creation is propagated. > > On 9. Jun 2018, at 08:07,

Spark / Scala code not recognising the path?

ss verify whether it is taking the path of hdfs or not I have added one more println statement in my code, providing the path which is already there in HDFS. It's showing true in that case. So, what could be the reason? Thanks, Abhijeet Kumar