Gourav, Yes, sorry. Apparently I failed to mention I'm having these problems
with Spark consuming from a kinesis stream. Been putting in late nights to
figure this out and it's affecting my brain. :^)
-jeremy
--
Jeremy Kelley | Technical Director, Data
jkel...@carbonblack.com | Carbon Bl
Hi Jeremy,
just out of curiosity - you do know that this is a SPARK user group?
Regards,
Gourav
On Thu, Dec 14, 2017 at 7:03 PM, Jeremy Kelley
wrote:
> We have a largeish kinesis stream with about 25k events per second and
> each record is around 142k. I have tried multiple cluster sizes, mu
We have a largeish kinesis stream with about 25k events per second and each
record is around 142k. I have tried multiple cluster sizes, multiple batch
sizes, multiple parameters... I am doing minimal transformations on the data.
Whatever happens I can sustain consuming 25k with minimal effort
