Re: Rebuilding flink

[Ferenc Csaky]

The cluster will only be refreshed if you rebuild the module with
changes and "flink-dist". So in this particular case:

$ mvn clean install -DskipTests -Dfast -pl flink-runtime,flink-dist -am

If you have a fairly fresh SNAPSHOT build already in your local maven
repo, then "-am" can be omitted and it will be a lot more faster, cause
it won't compile half of the world. :)

Hope this helps!

Cheers,
Ferenc




On Saturday, August 10th, 2024 at 08:33, Gabor Somogyi 
 wrote:

> 
> 
> Flink does, but maven not always detect the changes for god knows why.
> Gradle is slightly better but also not flawless. So for testing it may
> work, but for releases I suggest not to use.
> 
> G
> 
> On Sat, Aug 10, 2024, 00:17 Dan Plyukhin dplyuk...@gmail.com wrote:
> 
> > Does Flink support incremental builds? I just started hacking around on
> > flink-runtime, but I can’t see my changes unless I do a full “mvn clean
> > package”. I’ve tried “mvn clean package -pl flink-runtime -am”, but when I
> > run the cluster from “./build-target/bin/start-cluster.sh” I don’t see any
> > change in behavior.
> > 
> > Thanks in advance!
> > Dan

Re: [DISCUSS] FLIP-474: Store operator name and UID in state metadata

[Zakelly Lan]

Hi Gabor,

I apologize for any confusion. Let me clarify my position.

The concept of state observability is important for users, and the current
FLIP seems to be a step in the right direction. However, before we proceed,
I suggest we discuss the final presentation of the state observability to
the user and consider the high-level vision for achieving this. It's
essential to ensure that the current FLIP aligns with the overall
objective. I'm not suggesting a comprehensive FLIP to address all the
missing pieces, and one FLIP for each piece is fine for me. I just want to
ensure that we are on the same page in terms of vision. The last thing I
want is a fragmented approach resulting in refactoring or deprecation of
code when we need a complete feature.

Actually, I would hesitate about the current proposal of adding uid *in*
the state metadata. It may cause state incompatibility issues across
versions. In theory we can do this but it is better not if we are adding
data not for fault tolerance but only for human readability. And it could
be worse if we add one or two columns sporadically in future.

In fact, I expect the state metadata store to exist next to the checkpoint
metadata, rather than within it. This gives us enough flexibility to polish
this function as users need it, and without breaking checkpoint
compatibility too often. Or moreover we don't have to stick to the form of
checkpoint and we could choose a more human readable format like json for
the metadata store. This is where I think this FLIP is inconsistent with my
expectation of the state observability approach. These considerations
deserve a discussion before proceeding with other details. WDTY?


Best,
Zakelly

On Fri, Aug 9, 2024 at 8:22 PM Gabor Somogyi 
wrote:

> Hi David,
>
> Thanks for sharing your thoughts!
>
> > It sounds like you might already have an end-to-end solution in mind. It
> would be really helpful if you could put that into writing so we can all
> align our thinking.
>
> It makes sense to create a high level vision.
>
> > I’m not a fan of the mindset of “this is how it was done in Spark, so
> we’ll
> just replicate it” without proper discussion. We’ve had similar
> conversations before.
>
> I think we've had this conversation already in case of delegation token
> framework
> and I can say the same. No intention to take over things blindly but it's
> not a shame
> to be inspired by solutions which are welcome by users.
> The intention is similar just like in scalable authentication area where
> Flink is now ahead of Spark.
>
> > Would it be too much to ask for a FLIP that outlines the overall vision
> (without delving too deeply into the details) to ensure everyone is aligned
> and moving in the same direction?
>
> That's a fair point and a constructive way how we can proceed.
> I'm going to come back with the details...
>
> BR,
> G
>
>
> On Fri, Aug 9, 2024 at 1:36 PM David Morávek  wrote:
>
> > Hi Gabor,
> >
> > Thanks for taking the initiative on this. It’s clear that significant
> > improvements are needed in this area, and parsing state files can be
> > incredibly challenging, even for those who are well-versed in it.
> >
> > > Just to make it crystal clear, I’m not shooting for an ad-hoc tiny fix
> > but started a path where we fill each and every gap which will end up in
> a
> > functionality and UX bar just like the Spark solution.
> >
> > It sounds like you might already have an end-to-end solution in mind. It
> > would be really helpful if you could put that into writing so we can all
> > align our thinking.
> >
> > I’m not a fan of the mindset of “this is how it was done in Spark, so
> we’ll
> > just replicate it” without proper discussion. We’ve had similar
> > conversations before.
> >
> > > But this doesn’t mean we create a single giga big FLIP after several
> > months of discussion.
> >
> > I don’t think anyone is asking for a massive FLIP after lengthy
> > discussions, but having a document that outlines the overall vision could
> > be incredibly valuable, especially in a distributed setting. It also
> opens
> > the door for others to contribute to and shape this shared vision, which
> is
> > a core principle of community-driven open-source development.
> >
> > Would it be too much to ask for a FLIP that outlines the overall vision
> > (without delving too deeply into the details) to ensure everyone is
> aligned
> > and moving in the same direction?
> >
> > Best,
> > D.
> >
> > On Fri, Aug 9, 2024 at 11:44 AM Gabor Somogyi  >
> > wrote:
> >
> > > Hi Zakelly,
> > >
> > > > I'd suggest we could think of this as a whole
> > >
> > > In general I think we have the same idea in our mind about considering
> > the
> > > state observability as a whole, just we need to agree about the
> physical
> > > task scheduling.
> > >
> > > > But such a solution requires more design and discussion
> > >
> > > I can't even agree more. But this doesn't mean we create a single giga
> > big
> > > FLIP after several months of discuss

[jira] [Created] (FLINK-36027) [cdc-connector][MySQL] added metrics numRecordsOut for each OperationType(update, delete, insert)

[Lee SeungMin (Jira)]

Lee SeungMin created FLINK-36027:


 Summary: [cdc-connector][MySQL] added metrics numRecordsOut for 
each OperationType(update, delete, insert)
 Key: FLINK-36027
 URL: https://issues.apache.org/jira/browse/FLINK-36027
 Project: Flink
  Issue Type: Improvement
  Components: Flink CDC
Reporter: Lee SeungMin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] re-open FLINK-28842 Add client.id.prefix for the KafkaSink

[Rich C.]

Hi all,

I'm reaching out to you about FLINK-28842
. We've encountered the
`javax.management.InstanceAlreadyExistsException` in production, and I
believe resolving this issue could mitigate the problem.

I've prepared a fix in a private fork that has been working well in our
environment. I’d be happy to contribute this fix to the project.

Please let me know the best way to proceed with this contribution. Thank
you for your time and support.

Regards,
Rich

[jira] [Created] (FLINK-36028) Doc: kafka source additional properties

[Rich (Jira)]

Rich created FLINK-36028:


 Summary: Doc: kafka source additional properties
 Key: FLINK-36028
 URL: https://issues.apache.org/jira/browse/FLINK-36028
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Rich


In DataStream Connectors -> Kafka -> Additional Properties, it should add 
`client.id` is always overwritten by `client.id.prefix + subTaskId`. If 
`client.id.prefix` is not set, it is consumer group id or "KafkaSource-\{random 
integer}"

 

In Table API Connectors -> Kafka -> properties.*, it should mention there are 
additional properties from Flink and link to DataStream Connectors -> Kafka -> 
Additional Properties



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-36029) Add @Nullable Annotation to ValueState's Value Method Return Type

[LEE SOMANG (Jira)]

LEE SOMANG created FLINK-36029:
--

 Summary: Add @Nullable Annotation to ValueState's Value Method 
Return Type
 Key: FLINK-36029
 URL: https://issues.apache.org/jira/browse/FLINK-36029
 Project: Flink
  Issue Type: Improvement
Reporter: LEE SOMANG






--
This message was sent by Atlassian Jira
(v8.20.10#820010)