Hi,
I have scenario like below
https://stackoverflow.com/questions/58134379/how-to-handle-backup-scenario-in-spark-structured-streaming-using-joins
How to handle this use-case ( back-up scenario) in
spark-structured-streaming?
Any clues would be highly appreciated.
Thanks,
Shyam
1) this is not a use case, but a technical solution. Hence nobody can tell you
if it make sense or not
2) do an upsert in Cassandra. However keep in mind that the application
submitting to the Kafka topic and the one consuming from the Kafka topic need
to ensure that they process messages in the
What exactly is your requirement?
Is the read before write mandatory?
Are you maintaining states in Cassandra?
Regards
Prathmesh Ranaut
https://linkedin.com/in/prathmeshranaut
> On Aug 29, 2019, at 3:35 PM, Shyam P wrote:
>
>
> thanks Aayush. For every record I need to get the data from
thanks Aayush.
For every record I need to get the data from cassandra table and
update it ? Else it may not update the existing record.
What is this datastax-spark-connector ? is that not a "Cassandra
connector library written for spark"?
If not , how to write ourselves.
Where and how to
Cassandra is upsert, you should be able to do what you need with a single
statement unless you’re looking to maintain counters.
I’m not sure if there is a Cassandra connector library written for spark
streaming because we wrote one ourselves when we wanted to do the same.
Regards
Prathmesh Ran
Hi,
I need to do a PoC for a business use-case.
*Use case :* Need to update a record in Cassandra table if exists.
Will spark streaming support compare each record and update existing
Cassandra record ?
For each record received from kakfa topic , If I want to check and compare
each record wheth
Jul 15, 2014 at 6:53 PM, Tathagata Das <
> tathagata.das1...@gmail.com> wrote:
>
>> Oh yes, we have run sql, streaming and mllib all together.
>>
>> You can take a look at the demo <https://databricks.com/cloud> that
>> DataBricks gave at the spark summit.
Thanks Tathagata, we actually found the problem. I created SQLContext and
StreamContext from different SparkContext. But thanks for your help
Best,
Siyuan
On Tue, Jul 15, 2014 at 6:53 PM, Tathagata Das
wrote:
> Oh yes, we have run sql, streaming and mllib all together.
>
> You c
Oh yes, we have run sql, streaming and mllib all together.
You can take a look at the demo <https://databricks.com/cloud> that
DataBricks gave at the spark summit.
I think I get the problem is. Sql("") returns a RDD, and println(rdd)
prints only the RDD's name. An
By the way, have you ever run SQL and stream together? Do you know any
example that works? Thanks!
On Tue, Jul 15, 2014 at 4:28 PM, hsy...@gmail.com wrote:
> Hi Tathagata,
>
> I could see the output of count, but no sql results. Run in standalone is
> meaningless for me and I just run in my loc
Hi Tathagata,
I could see the output of count, but no sql results. Run in standalone is
meaningless for me and I just run in my local single node yarn cluster.
Thanks
On Tue, Jul 15, 2014 at 12:48 PM, Tathagata Das wrote:
> Could you run it locally first to make sure it works, and you see outp
Could you run it locally first to make sure it works, and you see output?
Also, I recommend going through the previous step-by-step approach to
narrow down where the problem is.
TD
On Mon, Jul 14, 2014 at 9:15 PM, hsy...@gmail.com wrote:
> Actually, I deployed this on yarn cluster(spark-submit
Actually, I deployed this on yarn cluster(spark-submit) and I couldn't find
any output from the yarn stdout logs
On Mon, Jul 14, 2014 at 6:25 PM, Tathagata Das
wrote:
> Can you make sure you are running locally on more than 1 local cores? You
> could set the master in the SparkConf as conf.setM
Can you make sure you are running locally on more than 1 local cores? You
could set the master in the SparkConf as conf.setMaster("local[4]"). Then
see if there are jobs running on every batch of data in the Spark web ui
(running on localhost:4040). If you still dont get any output, try first
simpl
No errors but no output either... Thanks!
On Mon, Jul 14, 2014 at 4:59 PM, Tathagata Das
wrote:
> Could you elaborate on what is the problem you are facing? Compiler error?
> Runtime error? Class-not-found error? Not receiving any data from Kafka?
> Receiving data but SQL command throwing error
Could you elaborate on what is the problem you are facing? Compiler error?
Runtime error? Class-not-found error? Not receiving any data from Kafka?
Receiving data but SQL command throwing error? No errors but no output
either?
TD
On Mon, Jul 14, 2014 at 4:06 PM, hsy...@gmail.com wrote:
> Hi Al
Hi All,
Couple days ago, I tried to integrate SQL and streaming together. My
understanding is I can transform RDD from Dstream to schemaRDD and execute
SQL on each RDD. But I got no luck
Would you guys help me take a look at my code? Thank you very much!
object KafkaSpark {
def main(args: Arr
17 matches
Mail list logo
