lliams
To: Reynold Xin , Mario Ds
Briggs/India/IBM@IBMIN
Cc: "dev@spark.apache.org"
Date: 15/09/2016 11:19 pm
Subject:Re: Compatibility of 1.6 spark.eventLog with a 2.0 History
Server
What is meant by:
"""
(This is because clicking the ref
Hi,
I would like to use a Spark 2.0 History Server instance on spark1.6
generated eventlogs. (This is because clicking the refresh button in
browser, updates the UI with latest events, where-as in the 1.6 code base,
this does not happen)
My question is whether this is safe to do and are they an
That was my initial thought as well. But then i was wondering if this
approach could help remove
a - the little extra latency overhead we have with the DirectApproach
(compared to Receiver) and
b - the data duplication in-efficiency (replication to WAL) and single
version of the truth of the offs
Look at
org.apache.spark.streaming.scheduler.JobGenerator
it has a RecurringTimer (timer) that will simply post 'JobGenerate' events
to a EventLoop at the batchInterval time.
This EventLoop's thread then picks up these events, uses the
streamingContext.graph' to generate a Job (InputDstream's
sounds sane for a first cut.
Since all creation methods take a KafkaParams, i was thinking along lines
of maybe a temp property in there which trigger usage of new consumer.
thanks
Mario
From: Cody Koeninger
To: Mario Ds Briggs/India/IBM@IBMIN
Cc: "dev@spark.apache.org&q
>>
forcing people on kafka 8.x to upgrade their brokers is questionable.
<<
I agree and i was more thinking maybe there is a way to support both for a
period of time (of course means some more code to maintain :-)).
thanks
Mario
From: Cody Koeninger
To: Mario Ds Brig
Hi,
Wanted to pick Cody's mind on what he thinks about
DirectKafkaInputDStream/KafkaRDD internally using the new Kafka consumer
API. I know the latter is documented as beta-quality, but yet wanted to
know if he sees any blockers as to why shouldn't go there shortly. On my
side the consideration
