tartarus created FLINK-36946:
Summary: optimize sink operator name truncate
Key: FLINK-36946
URL: https://issues.apache.org/jira/browse/FLINK-36946
Project: Flink
Issue Type: Improvement
y Lan (binding)
- Rui Fan (binding)
- Yun Tang (binding)
There were no votes against it.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-474%3A+Store+operator+name+and+UID+in+state+metadata
BR,
G
+1(binding)
Hopes the big picture could involve the SQL operators.
Best
Yun Tang
From: Rui Fan <1996fan...@gmail.com>
Sent: Tuesday, August 27, 2024 11:04
To: dev@flink.apache.org
Subject: Re: [VOTE] FLIP-474: Store operator name and UID in state metada
>
> > From: Peter Huang
> > Sent: Monday, August 26, 2024 7:29:12 AM
> > To: dev@flink.apache.org
> > Subject: Re: [VOTE] FLIP-474: Store operator name and UID in state
> metadata
> >
> > +1 (Non binding)
&g
e.org
> Subject: Re: [VOTE] FLIP-474: Store operator name and UID in state metadata
>
> +1 (Non binding)
>
>
> Peter Huang
>
> On Mon, Aug 26, 2024 at 6:49 AM Gyula Fóra wrote:
>
> > +1
> >
> > Gyula
> >
> > On Mon, Aug 26, 2024 at 3:47 PM Fer
r we provide a tool to list those
> >> > > metadata.
> >> > > > My
> >> > > > > thoughts is to provide a complete solution instead of adding one
> >> or
> >> > two
> >> > > > > specific data alongside the che
+1( non binding)
Get Outlook for iOS<https://aka.ms/o0ukef>
From: Peter Huang
Sent: Monday, August 26, 2024 7:29:12 AM
To: dev@flink.apache.org
Subject: Re: [VOTE] FLIP-474: Store operator name and UID in state metadata
+1 (Non binding)
Peter Huang
; >
> > > Thanks for clarifying the big picture [1], that helps a lot.
> > >
> > > [1]
> > >
> >
> https://docs.google.com/document/d/1Du1-TShoOjaNDCahs3sgLWIpYkXzJPdSkgHcWLpELyw/edit?usp=sharing
> > >
> > > On Mon, Aug 26, 2024 at 10:02
; >
> > > Thanks for clarifying the big picture [1], that helps a lot.
> > >
> > > [1]
> > >
> >
> https://docs.google.com/document/d/1Du1-TShoOjaNDCahs3sgLWIpYkXzJPdSkgHcWLpELyw/edit?usp=sharing
> > >
> > > On Mon, Aug 26, 2024 at 10:02 AM Gabor Somogyi
> gab
g
> >
> > On Mon, Aug 26, 2024 at 10:02 AM Gabor Somogyi gabor.g.somo...@gmail.com
> >
> > wrote:
> >
> > > Hi All,
> > >
> > > I would like to start a vote on FLIP-474: Store operator name and UID
> in
> > > state metadata [1].
> >
gt;
> [1]
> https://docs.google.com/document/d/1Du1-TShoOjaNDCahs3sgLWIpYkXzJPdSkgHcWLpELyw/edit?usp=sharing
>
> On Mon, Aug 26, 2024 at 10:02 AM Gabor Somogyi gabor.g.somo...@gmail.com
>
> wrote:
>
> > Hi All,
> >
> > I would like to start a vote on FLIP-474: Store
74: Store operator name and UID in
> state metadata [1].
> The discussion thread can be found here [2].
>
> During the discussion a clear need formed to have an umbrella document
> about the global view
> which manifested in Flink state observability umbrella [3].
>
> The vote w
Hi All,
I would like to start a vote on FLIP-474: Store operator name and UID in
state metadata [1].
The discussion thread can be found here [2].
During the discussion a clear need formed to have an umbrella document
about the global view
which manifested in Flink state observability umbrella [3
gt; the
>> > > > > current issue you are facing, here's my idea: If you could get
>> access
>> > > to
>> > > > > the web UI, you can get the hash (vertex id) in the url by
>> clicking
>> > and
>> > > > > zooming in on th
> >
> > > > > Best,
> > > > > Zakelly
> > > > >
> > > > > On Thu, Aug 8, 2024 at 11:03 PM Gabor Somogyi <
> > > gabor.g.somo...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > &
ow.
> > > > >
> > > > > Up until now our state processor API was good enough but now we're
> > > > lagging
> > > > > behind. We see users (just like Spark) where the first class
> citizen
> > is
> > > > the
> >
that there are cases where users are not
> able
> > to
> > > > identify operators because hash is a one way conversion.
> > > > I'm open to any suggestion but somehow the initial operator human
> > > readable
> > > > identifier must b
doesn't
> know
> > > what they set when debugging?
> > >
> > > There are cases where the user is setting the UID in the job, such case
> > > it's not user friendly to parse git repos but doable.
> > > But there are cases where the user h
user friendly to parse git repos but doable.
> > But there are cases where the user has limited or no control related
> UIDs:
> > * SQL jobs are generating operators with meaningful names, but I think
> it's
> > not realistic to enforce users to understand all the internals o
rce users to understand all the internals of Flink SQL
> implementation (which operator named where and how).
> * Iceberg is using the given UID as prefix and generating more operators
> with it
> * Weak justification but exists: Since operator name and UID are both
> optional som
ators
with it
* Weak justification but exists: Since operator name and UID are both
optional some of the users are setting name only. Such case Flink generates
a random hash, where only name can give some pointers.
Hope I've given better context.
[1]
https://www.databricks.com/blog/anno
> Regarding the checkpoint, it is not designed to be self-contained or
> human-readable. I suggest not introducing such columns for debugging
> purposes.
>
>
> Best,
> Zakelly
>
> On Wed, Aug 7, 2024 at 10:07 PM Gabor Somogyi
> wrote:
>
> > Hi Devs,
> >
ng such columns for debugging
purposes.
Best,
Zakelly
On Wed, Aug 7, 2024 at 10:07 PM Gabor Somogyi
wrote:
> Hi Devs,
>
> I would like to start a discussion on FLIP-474: Store operator name and UID
> in state metadata[1].
>
> In short users are interested in what kind of operato
Hi Devs,
I would like to start a discussion on FLIP-474: Store operator name and UID
in state metadata[1].
In short users are interested in what kind of operators are inside a
checkpoint data which can be enhanced from user experience perspective. The
details can be found in FLIP-474[1].
Please
Gabor Somogyi created FLINK-36001:
-
Summary: Store operator name and UID in state metadata
Key: FLINK-36001
URL: https://issues.apache.org/jira/browse/FLINK-36001
Project: Flink
Issue Type
Weihua Hu created FLINK-31808:
-
Summary: wrong examples of how to set operator name in documents
Key: FLINK-31808
URL: https://issues.apache.org/jira/browse/FLINK-31808
Project: Flink
Issue
Wenlong Lyu created FLINK-25070:
---
Summary: FLIP-195: Improve the name and structure of vertex and
operator name for job
Key: FLINK-25070
URL: https://issues.apache.org/jira/browse/FLINK-25070
Project
!
>
> Best,
> Yun--
> Sender:Xianxun Ye
> Date:2021/11/24 16:49:44
> Recipient:dev@flink.apache.org
> Theme:[VOTE][FLIP-195] Improve the name and structure of vertex and
> operator name for job
>
> +1 (no
+1 (binding).
Very thanks Wenlong for the proposal!
Best,
Yun--
Sender:Xianxun Ye
Date:2021/11/24 16:49:44
Recipient:dev@flink.apache.org
Theme:[VOTE][FLIP-195] Improve the name and structure of vertex and operator
name for job
]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+job
Best,
Wenlong Lyu
--
Best regards,
Sergey
; > > > > > > > would
> > > > > like
> > > > > > to
> > > > > > > > start a vote on FLIP-195 [2].
> > > > > > > > Thanks for all of your feedback.
> > > > > > > >
> > > > > > > > The vote will last for at least 72 hours (Nov 26th 16:00 GMT)
> > > unless
> > > > > > > > there is an objection or insufficient votes.
> > > > > > > >
> > > > > > > > [1]
> > > https://lists.apache.org/thread/kvdxr8db0l5s6wk7hwlt0go5fms99b8t
> > > > > > > > [2]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+job
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Wenlong Lyu
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> >
> > --
> > Best regards,
> > Sergey
> >
> start a vote on FLIP-195 [2].
> > > > > > > Thanks for all of your feedback.
> > > > > > >
> > > > > > > The vote will last for at least 72 hours (Nov 26th 16:00 GMT)
> > unless
> > > > > > > there is an objection or insufficient votes.
> > > > > > >
> > > > > > > [1]
> > https://lists.apache.org/thread/kvdxr8db0l5s6wk7hwlt0go5fms99b8t
> > > > > > > [2]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+job
> > > > > > >
> > > > > > > Best,
> > > > > > > Wenlong Lyu
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
>
> --
> Best regards,
> Sergey
>
t; > > > to
> > > > > > start a vote on FLIP-195 [2].
> > > > > > Thanks for all of your feedback.
> > > > > >
> > > > > > The vote will last for at least 72 hours (Nov 26th 16:00 GMT)
> unless
> > > > > > there is an objection or insufficient votes.
> > > > > >
> > > > > > [1]
> https://lists.apache.org/thread/kvdxr8db0l5s6wk7hwlt0go5fms99b8t
> > > > > > [2]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+job
> > > > > >
> > > > > > Best,
> > > > > > Wenlong Lyu
> > > > > >
> > > > >
> > > >
> > >
>
--
Best regards,
Sergey
gt; like
> > > to
> > > > > start a vote on FLIP-195 [2].
> > > > > Thanks for all of your feedback.
> > > > >
> > > > > The vote will last for at least 72 hours (Nov 26th 16:00 GMT) unless
> > > > > there is an
ike
> > > to
> > > > > start a vote on FLIP-195 [2].
> > > > > Thanks for all of your feedback.
> > > > >
> > > > > The vote will last for at least 72 hours (Nov 26th 16:00 GMT)
> unless
> > > > > there is an obje
16:00 GMT) unless
> > > > there is an objection or insufficient votes.
> > > >
> > > > [1] https://lists.apache.org/thread/kvdxr8db0l5s6wk7hwlt0go5fms99b8t
> > > > [2]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+job
> > > >
> > > > Best,
> > > > Wenlong Lyu
> > > >
> > >
> >
>
(Nov 26th 16:00 GMT) unless
> > > there is an objection or insufficient votes.
> > >
> > > [1] https://lists.apache.org/thread/kvdxr8db0l5s6wk7hwlt0go5fms99b8t
> > > [2]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+job
> > >
> > > Best,
> > > Wenlong Lyu
> > >
> >
>
t
> > [2]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+job
> >
> > Best,
> > Wenlong Lyu
> >
>
s (Nov 26th 16:00 GMT) unless
> there is an objection or insufficient votes.
>
> [1] https://lists.apache.org/thread/kvdxr8db0l5s6wk7hwlt0go5fms99b8t
> [2]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+job
>
> Best,
> Wenlong Lyu
>
; The vote will last for at least 72 hours (Nov 26th 16:00 GMT) unless
> there is an objection or insufficient votes.
>
> [1] https://lists.apache.org/thread/kvdxr8db0l5s6wk7hwlt0go5fms99b8t
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+str
Hi,
Very thanks Wenlong for the update! Also +1 for the proposal~
Best,
Yun Gao
--
From:Yun Gao
Send Time:2021 Nov. 23 (Tue.) 16:09
To:dev
Subject:Re: [DISCUSS] Improve the name and structure of job vertex and operator
name
/thread/kvdxr8db0l5s6wk7hwlt0go5fms99b8t
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+job
Best,
Wenlong Lyu
org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+job
> to
> add a prefix [vertex-idx] for vertex name. please checkout to see whether
> you have more comments, thanks.
>
> Add a prefix [vertex-idx] for vertex name, so that we can
Hi, Yun Gao, Yun Tang, and Aitozi,
thanks for the suggestion again, I have added following section in proposed
changes at
https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+job
to
add a prefix [vertex-idx] for vertex name
Hi,
Very thanks Wenlong for bringing up this issue and very thanks for the warm
discussion! Big +1 to improve the name and structure of job vertex for it make
the life of the developers much easier and shorten the operator name would
saves a lot of money for reducing the amount of logs to
ncise
> > enough, we can get the vertex id easily.
> > Does anyone have some comments on this?
> >
> >
> > Best,
> > Wenlong Lyu
> >
> > On Thu, 18 Nov 2021 at 19:03, Yun Tang wrote:
> >
> > > Hi Wenlong,
> > >
>
eve many guys have ever
> > suffered from the long and unreadable operator name for long time.
> >
> > I have another suggestion which inspired by Aitozi, that we could add
> some
> > hint to tell the vertex index. Such as make the pipeline from "source -->
wrote:
> Hi Wenlong,
>
> Thanks for bringing up this discussion and I believe many guys have ever
> suffered from the long and unreadable operator name for long time.
>
> I have another suggestion which inspired by Aitozi, that we could add some
> hint to tell the vertex in
Hi Wenlong,
Thanks for bringing up this discussion and I believe many guys have ever
suffered from the long and unreadable operator name for long time.
I have another suggestion which inspired by Aitozi, that we could add some hint
to tell the vertex index. Such as make the pipeline from
> On Tue, 16 Nov 2021 at 19:14, Chesnay Schepler
> > wrote:
> > >
> > > > Why should this be specific to the table API? The datastream API has
> > > > similar issues with long operator names (like windowing).
> > > >
> > > > On 16/11/2021 1
hesnay Schepler
> wrote:
> >
> > > Why should this be specific to the table API? The datastream API has
> > > similar issues with long operator names (like windowing).
> > >
> > > On 16/11/2021 11:22, wenlong.lwl wrote:
> > > > Thank
API has
> > similar issues with long operator names (like windowing).
> >
> > On 16/11/2021 11:22, wenlong.lwl wrote:
> > > Thanks Godfrey for the suggestion.
> > > Regarding 1, how about table.optimizer.simplify-operator-name-enabled,
>
wenlong.lwl wrote:
> > Thanks Godfrey for the suggestion.
> > Regarding 1, how about table.optimizer.simplify-operator-name-enabled,
> > which means that we would simplify the name of operator and keep the
> > details in description only.
> > "table.optimizer.oper
Why should this be specific to the table API? The datastream API has
similar issues with long operator names (like windowing).
On 16/11/2021 11:22, wenlong.lwl wrote:
Thanks Godfrey for the suggestion.
Regarding 1, how about table.optimizer.simplify-operator-name-enabled,
which means that we
Thanks Godfrey for the suggestion.
Regarding 1, how about table.optimizer.simplify-operator-name-enabled,
which means that we would simplify the name of operator and keep the
details in description only.
"table.optimizer.operator-name.description-enabled" can not describe what
it mea
Thanks for creating this FLIP Wenlong.
The FLIP already looks pretty solid, I think the config options can be
improved a little:
1) about table.optimizer.separate-name-and-description, I think
"operator-name" should be considered in the option,
how about table.optimizer.operator-name.d
Hi, all, FYI the FLIP doc has been created ：
https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+sql+job
Best,
Wenlong
On Mon, 15 Nov 2021 at 11:41, wenlong.lwl wrote:
> Hi all,
> Thanks for the feedback, It seem
Hi all,
Thanks for the feedback, It seems that the proposal is accepted by all of
you guys. I will prepare a formal FLIP document and then go ahead to the
vote stage.
If any one has any other comments or suggestions, please let me know,
thanks.
Best,
Wenlong
On Fri, 12 Nov 2021 at 05:54, Neng Lu
+1 (non-binding)
This change will really help to ease developer life.
On Thu, Nov 11, 2021 at 6:33 AM Guowei Ma wrote:
> +1
> This would be very helpful for our debugging online job.
>
> Best,
> Guowei
>
>
> On Thu, Nov 11, 2021 at 8:03 PM Yuepeng Pan wrote:
>
> > +1. It's useful to understand
+1
This would be very helpful for our debugging online job.
Best,
Guowei
On Thu, Nov 11, 2021 at 8:03 PM Yuepeng Pan wrote:
> +1. It's useful to understand the job topology.
> Looking forward to this feature.
> Best,
> Yuepeng Pan.
>
>
>
>
>
>
> At 2021-11-11 19:44:44, "Yangze Guo" wrote:
> >
+1. It's useful to understand the job topology.
Looking forward to this feature.
Best,
Yuepeng Pan.
At 2021-11-11 19:44:44, "Yangze Guo" wrote:
>+1. That's gonna help a lot for debugging.
>
>Best,
>Yangze Guo
>
>On Thu, Nov 11, 2021 at 7:37 PM Till Rohrmann wrote:
>>
>> This improvement loo
+1. That's gonna help a lot for debugging.
Best,
Yangze Guo
On Thu, Nov 11, 2021 at 7:37 PM Till Rohrmann wrote:
>
> This improvement looks like it makes the life of our users a lot easier
> when it comes to understanding logs and reading the UI. Hence +1.
>
> Cheers,
> Till
>
> On Thu, Nov 11,
This improvement looks like it makes the life of our users a lot easier
when it comes to understanding logs and reading the UI. Hence +1.
Cheers,
Till
On Thu, Nov 11, 2021 at 11:59 AM JING ZHANG wrote:
> Big +1.
>
> This is a problem frequently encountered in our production platform, look
> for
Big +1.
This is a problem frequently encountered in our production platform, look
forward to this improvement.
Best,
Jing Zhang
Martijn Visser 于2021年11月11日周四 下午6:26写道：
> +1. Looks much better now
>
> On Thu, 11 Nov 2021 at 11:07, godfrey he wrote:
>
> > Thanks for driving this, this improveme
+1. Looks much better now
On Thu, 11 Nov 2021 at 11:07, godfrey he wrote:
> Thanks for driving this, this improvement solves a long-complained
> problem, +1
>
> Best,
> Godfrey
>
> Jark Wu 于2021年11月11日周四 下午5:40写道：
> >
> > +1 for this. It looks much more clear and structured.
> >
> > Best,
> > J
Thanks for driving this, this improvement solves a long-complained problem, +1
Best,
Godfrey
Jark Wu 于2021年11月11日周四 下午5:40写道：
>
> +1 for this. It looks much more clear and structured.
>
> Best,
> Jark
>
> On Thu, 11 Nov 2021 at 17:23, Chesnay Schepler wrote:
>
> > I'm generally in favor of it,
+1 for this. It looks much more clear and structured.
Best,
Jark
On Thu, 11 Nov 2021 at 17:23, Chesnay Schepler wrote:
> I'm generally in favor of it, and there are already tickets that
> proposed a dedicated operator/vertex description:
>
> https://issues.apache.org/jira/browse/FLINK-20388
> h
I'm generally in favor of it, and there are already tickets that
proposed a dedicated operator/vertex description:
https://issues.apache.org/jira/browse/FLINK-20388
https://issues.apache.org/jira/browse/FLINK-21858
On 11/11/2021 10:02, wenlong.lwl wrote:
Hi, all, I would like to start a discus
Hi, all, I would like to start a discussion about an improvement on name
and structure of job vertex name, mainly to improve experience of debugging
and analyzing sql job at runtime.
the main proposed changes including:
1. separate description and name for operator, so that we can have detailed
in
Qingsheng Ren created FLINK-24376:
-
Summary: Operator name in OperatorCoordinator should not use
chained name
Key: FLINK-24376
URL: https://issues.apache.org/jira/browse/FLINK-24376
Project: Flink
Lu Niu created FLINK-22162:
--
Summary: Make Max Operator name Length Configurable
Key: FLINK-22162
URL: https://issues.apache.org/jira/browse/FLINK-22162
Project: Flink
Issue Type: Improvement
Danny Chen created FLINK-20368:
--
Summary: Supports custom operator name for Flink SQL
Key: FLINK-20368
URL: https://issues.apache.org/jira/browse/FLINK-20368
Project: Flink
Issue Type
Boyang Jerry Peng created FLINK-19468:
-
Summary: Metrics not returned when data stream / operator name
contains "+"
Key: FLINK-19468
URL: https://issues.apache.org/jira/browse/FLINK-19468
Jark Wu created FLINK-18846:
---
Summary: Set a meaningful operator name for the filesystem sink
Key: FLINK-18846
URL: https://issues.apache.org/jira/browse/FLINK-18846
Project: Flink
Issue Type
mzz created FLINK-18443:
---
Summary: The operator name select: (ip, ts, count,
environment.access AS access, environment.brand AS brand, sid, params.adid AS
adid, eventid) exceeded the 80 characters length limit and was truncated
Key: FLINK
Zou created FLINK-16372:
---
Summary: Generate unique operator name
Key: FLINK-16372
URL: https://issues.apache.org/jira/browse/FLINK-16372
Project: Flink
Issue Type: Improvement
Reporter
lining created FLINK-11212:
--
Summary: operator metric key can repalce operator name by operator
id
Key: FLINK-11212
URL: https://issues.apache.org/jira/browse/FLINK-11212
Project: Flink
Issue Type
Chesnay Schepler created FLINK-10021:
Summary: All-round E2E tests query metric with wrong operator name
Key: FLINK-10021
URL: https://issues.apache.org/jira/browse/FLINK-10021
Project: Flink
Julian Stephen created FLINK-9653:
-
Summary: Add operator name to latency metrics
Key: FLINK-9653
URL: https://issues.apache.org/jira/browse/FLINK-9653
Project: Flink
Issue Type: New Feature
Hi to all,
in many of my Flink job it is helpful to give a name to operators in order
to make JobManager UI simpler to read.
Currently, to give a name to operators it is necessary to specify it of the
operator everytime it is used, for example:
- env.readAsCsv().map().name("My map function")
W
Chesnay Schepler created FLINK-4875:
---
Summary: operator name not correctly inferred
Key: FLINK-4875
URL: https://issues.apache.org/jira/browse/FLINK-4875
Project: Flink
Issue Type: Bug
81 matches
Mail list logo
