/DynamicKafkaRecordSerializationSchema.java#L128
It converts UPDATE_BEFORE into tombstones, does it means for workload with lots
of updates, we will see intermediate tombstones.
How can we avoid/mitigate this? I expect we only emit tombstones for actual
deletion, not just update_before.
Thanks
Qing Lim | Marshall Wace
Thank you for the tips, I will try these out
From: Josh Mahonin
Sent: 18 January 2024 21:07
To: Qing Lim
Cc: Jun Qin ; User
Subject: Re: Use different S3 access key for different S3 bucket
Oops my syntax was a bit off there, as shown in the Hadoop docs, it looks like:
fs.s3a.bucket..
Josh
should do when getting this error?
I don't see anything in config that can
I am on Flink 1.16.
Thanks
Qing Lim | Marshall Wace LLP, George House, 131 Sloane Street, London, SW1X 9AT
| E-mail: q@mwam.com<mailto:q@mwam.com> | Tel: +44 207 925 4865
This e-mail and any
://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/#configure-access-credentials
Thanks
From: Jun Qin
Sent: 18 January 2024 10:51
To: User ; Qing Lim
Subject: Re: Use different S3 access key for different S3 bucket
Hi Qing
The S3 credentials are associated with Flink SQL tables.
I
Hi Sun,
As Dulce said, running in a cluster is typically recommended. However if for
some reason you need to run in standalone mode, why do you recreate Cluster on
each job, can you try to reuse the MiniCluster?
I happen to have a similar setup when we are running in standalone mode, and
run m
Hi, I am using Flink SQL to create table backed by S3 buckets.
We are not using AWS S3, so we have to use access key and secret for Auth.
My pipeline depends on 2 different buckets, each requires different
credentials, can flink support this?
Qing Lim | Marshall Wace LLP, George House, 131
onder if there's workaround? I also don't fully understand why it is not
supported, is it just a missing feature or is there some underlying fundamental
reason making this hard/impossible?
Kind regards
Qing Lim | Marshall Wace LLP, George House, 131 Sloane Street, London | E-mail:
That’s my understanding as well, thanks for your confirmation.
From: Yanfei Lei
Sent: 04 November 2022 16:03
To: Qing Lim
Cc: User
Subject: Re: Does reduce function on keyed window gives any guarantee on the
order of elements?
Hi Qing,
> am I right to think that there will be 1 red
Sent: 03 November 2022 03:06
To: Qing Lim
Cc: User
Subject: Re: Does reduce function on keyed window gives any guarantee on the
order of elements?
Hi Qing,
> Does it guarantee that it will be called in the same order of elements in the
> stream, where value2 is always 1 element after
element after value1?
Kind regards
Qing Lim | Marshall Wace LLP, George House, 131 Sloane Street, London | E-mail:
q@mwam.com<mailto:q@mwam.com> | Tel: +44 207 925 4865
This e-mail and any attachments are confidential to the addressee(s) and may
contain information that is l
27;ve found is from this SO:
https://stackoverflow.com/questions/50570605/why-broadcast-state-can-store-the-dynamic-rules-however-broadcast-operator-c
Which seems to suggest Broadcast State broadcast() then maintain state in each
parallel operator under the hood?
Kind regards.
Qing Lim | Mar
Thanks both for your advice, I will give them a try!
From: Schwalbe Matthias
Sent: 10 October 2022 08:35
To: 仙路尽头谁为峰 ; Qing Lim
Cc: User
Subject: RE: Re:Question about Flink Broadcast State event ordering
Hi Qing again,
Another point to consider: broadcast streams are subject to watermarking
Oh, thank you for your explanation!
From: 仙路尽头谁为峰
Sent: 05 October 2022 09:13
To: Qing Lim
Cc: User
Subject: 回复: Re:Question about Flink Broadcast State event ordering
Hi Qing:
The key point is that the broadcast side may have different partitions that
interleaves. If you can make sure
: 05 October 2022 03:02
To: Qing Lim
Cc: User
Subject: Re:Question about Flink Broadcast State event ordering
Hi Qing:
> I think this is refering to the order between broadcasted element and non
> broadcasted element, right?
No, as broadcast and nonbroadcast stream are different streams
Hi Flink user group,
I have a question around broadcast.
Reading the docs
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/#important-considerations,
it says the following:
> Order of events in Broadcast State may differ across tasks: Alt
June 2022 16:16
To: Qing Lim
Cc: User
Subject: Re:How can I convert a SQL String to a ResolvedExpression?
Hi, what about use `FieldReferenceExpression`[1] in test just like here[2] ?
[1]
https://github.com/apache/flink/blob/5bcef81356f965ee9e6a8ab54b5faca1e3979873/flink-table/flink-table-common
Hi Flink User Group,
Is there a way to convert SQL String to a
org.apache.flink.table.expressions.ResolvedExpression easily? I wish use this
in test. I manage to reverse engineer part of it but I don't think it matches
what Flink is really doing, below is my ugly code.
private List resolveSQLF
Thanks both, that’s perfect!
From: Jing Ge
Sent: 01 June 2022 14:29
To: yuxia
Cc: Qing Lim ; User
Subject: Re: Can we resume a job from a savepoint from Java api?
Hi,
yuxia has already pointed out the correct direction. The exact line for using
the savepoint path to resume the job from a
Hi, is it possible to resume a job from a savepoint in Java code?
I wish to test failure recovery in my test code, I am thinking to simulate
failure recovery by saving state to a save point and the recover from it, is
this possible with local MiniCluster setup?
Kind regards
This e-mail and any
2 03:57
To: Qing Lim
Cc: user@flink.apache.org
Subject: Re: Can we use CheckpointedFunction with the new Source api?
Hi Qing,
I’m afraid CheckpointedFunction cannot be applied to the new source API, but
could you share the abstractions of your source implementation, like which
component a split m
Hi, is it possible to use CheckpointedFunction with the new Source api? (The
one in package org.apache.flink.api.connector.source)
My use case:
I have a custom source that emit individual nodes update from a tree, and I
wish to create a stream of the whole Tree snapshots, so I will have to
acc
21 matches
Mail list logo
