Hello,
I'm an OpenLineage committer - and previously, a minor Flink contributor.
OpenLineage community is very interested in conversation about Flink
metadata, and we'll be happy to cooperate with the Flink community.
Best,
Maciej Obuchowski
czw., 13 sty 2022 o 18:12 Martijn Visser
Hey.
As far as I see, you're not overriding functions like open,
setRuntimeContext, snapshotState, initializeState - the calls needs to
be passed to the inner sink function.
pon., 2 sie 2021 o 19:31 Rion Williams napisał(a):
>
> Hi again Maciek (and all),
>
> I just recently returned to start in
Hey,
We had similar problem, but with 1000s of tables. I've created issue [1]
and PR with internally used solution [2], but unfortunately, there seems to
be no interest in upstreaming this feature.
Thanks,
Maciej
[1] https://issues.apache.org/jira/browse/FLINK-21643
[2] https://github.com/apache/
Hey Rion,
We solved this issue by using usual, unbounded streams, and using
awaitility library to express conditions that would end the test - for
example, having particular data in a table.
IMO this type of testing has the advantage that you won't have divergent
behavior from production as you h
Hey Rion,
I had exactly the same problem and implemented this functionality in
my Flink fork with XA sink taken from the development branch.
As I see that it's not only my problem, I've created a Jira task for
it - FLINK-21643 - and will provide draft PR for it.
@David - for traditional relationa
Hey Lasse,
I've had a similar case, albeit with Avro. I was reading from multiple
Kafka topics, which all had different objects and did some metadata
driven operations on them.
I could not go with any concrete predefined types for them, because
there were hundreds of different object types.
My sol
ight regarding Flink compression settings will not have any impact
> with incremental checkpoints.
>
> Regards,
> Roman
>
>
> On Wed, Feb 24, 2021 at 11:01 AM Maciej Obuchowski
> wrote:
>>
>> Hey. Let me send simplified example, because I don't think this
&g
MemTable or in an SST file. So it doesn't matter as long as the same
> serializer is used.
>
> You probably should try enabling compression if you didn't already:
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#compression
>
>
Hey.
We have deduplication job that has a large amount of keyed ValueState. We
want to decrease state size as much as possible, so we're using
ValueState as it's smallest possible Java non-primitive. However,
as per https://www.baeldung.com/java-size-of-object (and my measurements)
Java Integer ha
