Hi Max,
is the distinct() operation reducing the size of the DataSet? If so, I
assume you have an idempotent update and the job is faster because fewer
updates are done?
if the distinct() operator is not changing anything, then, the job might be
faster because the INSERT is done while Flink is still executing the
distinct() operation. So the insert is over when the updates are starting.
This would mean that concurrent inserts and updates on the database are
much slower than doing this sequentially.
I'm wondering if there is a way in Flink to explicitly ask for spilling an
intermediate operator to "pause" execution:
Source ----- > (spill for pausing) ---> (update sink)
\
------- > (insert)
I don't have a lot of practical experience with RDBMS, but I guess updates
are slower because an index lookup + update is necessary. Maybe optimizing
the database configuration / schema / indexes is more promising. I think
its indeed much nicer to avoid any unnecessary steps in Flink.
Did you do any "microbenchmarks" for the update and insert part? I guess
that would help a lot to understand the impact of certain index structures,
batching sizes, or database drivers.
Regards,
Robert
On Thu, Jan 21, 2016 at 3:35 PM, Maximilian Bode <
maximilian.b...@tngtech.com> wrote:
> Hi everyone,
>
> in a Flink (0.10.1) job with two JDBCOutputFormat sinks, one of them
> (doing a database update) is performing slower than the other one (an
> insert). The job as a whole is also slow as upstream operators are slowed
> down due to backpressure. I am able to speed up the whole job by
> introducing an a priori unnecessary .distinct(), which of course blocks
> downstream execution of the slow sink, which in turn seems to be able to
> execute faster when given all data at once.
>
> Any ideas what is going on here? Is there something I can do without
> introducing unnecessary computation steps?
>
> Cheers,
> Max
> —
> Maximilian Bode * Junior Consultant * maximilian.b...@tngtech.com * 0176
> 1000 75 50
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>
>
