Re: [DISCUSS][Release 1.12] Stale blockers and build instabilities

[Dian Fu]

Thanks Rui for the info. This issue(hive related) 
https://issues.apache.org/jira/browse/FLINK-19025 
 is marked as a blocker.

Regards,
Dian

> 在 2020年8月25日，下午2:58，Rui Li  写道：
> 
> Hi Dian,
> 
> FLINK-18682 has been fixed. Is there any other blocker in the hive
> connector?
> 
> On Tue, Aug 25, 2020 at 2:41 PM Dian Fu  > wrote:
> 
>> Hi all,
>> 
>> Two weeks have passed and it seems that none of the test stabilities
>> issues have been addressed since then.
>> 
>> Here is an updated status report of Blockers and Test instabilities:
>> 
>> Blockers <
>> https://issues.apache.org/jira/browse/FLINK-18682?filter=12349334 
>>  <
>> https://issues.apache.org/jira/browse/FLINK-18682?filter=12349334 
>> >>:
>> Currently 2 blockers (1x Hive, 1x CI Infra)
>> 
>> Test-Instabilities <
>> https://issues.apache.org/jira/browse/FLINK-18869?filter=12348580 
>>  <
>> https://issues.apache.org/jira/browse/FLINK-18869?filter=12348580 
>> >>:
>> (total 80)
>> 
>> Besides the issues already posted in previous mail, here are the new
>> instability issues which should be taken care of:
>> 
>> - FLINK-19012 (https://issues.apache.org/jira/browse/FLINK-19012 
>>  <
>> https://issues.apache.org/jira/browse/FLINK-19012 
>> >)
>> E2E test fails with "Cannot register Closeable, this
>> subtaskCheckpointCoordinator is already closed. Closing argument."
>> 
>> -> This is a new issue occurred recently. It has occurred several times
>> and may indicate a bug somewhere and should be taken care of.
>> 
>> - FLINK-9992 (https://issues.apache.org/jira/browse/FLINK-9992 
>>  <
>> https://issues.apache.org/jira/browse/FLINK-9992 
>> >)
>> FsStorageLocationReferenceTest#testEncodeAndDecode failed in CI
>> 
>> -> There is already a PR for it and needs review.
>> 
>> - FLINK-18842 (https://issues.apache.org/jira/browse/FLINK-18842 
>>  <
>> https://issues.apache.org/jira/browse/FLINK-18842 
>> >)
>> e2e test failed to download "localhost:/flink.tgz" in "Wordcount on
>> Docker test"
>> 
>> 
>>> 在 2020年8月11日，下午2:08，Robert Metzger  写道：
>>> 
>>> Hi team,
>>> 
>>> 2 weeks have passed since the last update. None of the test stabilities
>>> I've mentioned have been addressed since then.
>>> 
>>> Here's an updated status report of Blockers and Test instabilities:
>>> 
>>> Blockers <
>> https://issues.apache.org/jira/browse/FLINK-18682?filter=12349334>:
>>> Currently 3 blockers (2x Hive, 1x CI Infra)
>>> 
>>> Test-Instabilities
>>> 
>> (total
>>> 79) which failed recently or frequently:
>>> 
>>> 
>>> - FLINK-18807 
>>> FlinkKafkaProducerITCase.testScaleUpAfterScalingDown
>>> failed with "Timeout expired after 6milliseconds while awaiting
>>> EndTxn(COMMIT)"
>>> 
>>> - FLINK-18634 
>>> FlinkKafkaProducerITCase.testRecoverCommittedTransaction
>>> failed with "Timeout expired after 6milliseconds while awaiting
>>> InitProducerId"
>>> 
>>> - FLINK-16908 
>>> FlinkKafkaProducerITCase
>>> testScaleUpAfterScalingDown Timeout expired while initializing
>>> transactional state in 6ms.
>>> 
>>> - FLINK-13733 
>>> FlinkKafkaInternalProducerITCase.testHappyPath fails on Travis
>>> 
>>> --> The first three tickets seem related.
>>> 
>>> 
>>> - FLINK-17260 
>>> StreamingKafkaITCase failure on Azure
>>> 
>>> --> This one seems really hard to reproduce
>>> 
>>> 
>>> - FLINK-16768 
>>> HadoopS3RecoverableWriterITCase.testRecoverWithStateWithMultiPart
>>> hangs
>>> 
>>> - FLINK-18374 
>>> 
>> HadoopS3RecoverableWriterITCase.testRecoverAfterMultiplePersistsStateWithMultiPart
>>> produced no output for 900 seconds
>>> 
>>> --> nobody seems to feel responsible for these tickets. My guess is that
>>> the S3 connector should have shorter timeouts / faster retries to finish
>>> within the 15 minutes test timeout. OR there is really something wrong
>> with
>>> the code.
>>> 
>>> 
>>> - FLINK-18333 UnsignedTypeConversionITCase failed caused by MariaDB4j
>>> "Asked to waitFor Program"
>>> 

Re: [DISCUSS][Release 1.12] Stale blockers and build instabilities

[Rui Li]

Thanks Dian for the pointer. I'll take a look.

On Tue, Aug 25, 2020 at 3:02 PM Dian Fu  wrote:

> Thanks Rui for the info. This issue(hive related)
> https://issues.apache.org/jira/browse/FLINK-19025 <
> https://issues.apache.org/jira/browse/FLINK-19025> is marked as a blocker.
>
> Regards,
> Dian
>
> > 在 2020年8月25日，下午2:58，Rui Li  写道：
> >
> > Hi Dian,
> >
> > FLINK-18682 has been fixed. Is there any other blocker in the hive
> > connector?
> >
> > On Tue, Aug 25, 2020 at 2:41 PM Dian Fu  dian0511...@gmail.com>> wrote:
> >
> >> Hi all,
> >>
> >> Two weeks have passed and it seems that none of the test stabilities
> >> issues have been addressed since then.
> >>
> >> Here is an updated status report of Blockers and Test instabilities:
> >>
> >> Blockers <
> >> https://issues.apache.org/jira/browse/FLINK-18682?filter=12349334 <
> https://issues.apache.org/jira/browse/FLINK-18682?filter=12349334> <
> >> https://issues.apache.org/jira/browse/FLINK-18682?filter=12349334 <
> https://issues.apache.org/jira/browse/FLINK-18682?filter=12349334>>>:
> >> Currently 2 blockers (1x Hive, 1x CI Infra)
> >>
> >> Test-Instabilities <
> >> https://issues.apache.org/jira/browse/FLINK-18869?filter=12348580 <
> https://issues.apache.org/jira/browse/FLINK-18869?filter=12348580> <
> >> https://issues.apache.org/jira/browse/FLINK-18869?filter=12348580 <
> https://issues.apache.org/jira/browse/FLINK-18869?filter=12348580>>>:
> >> (total 80)
> >>
> >> Besides the issues already posted in previous mail, here are the new
> >> instability issues which should be taken care of:
> >>
> >> - FLINK-19012 (https://issues.apache.org/jira/browse/FLINK-19012 <
> https://issues.apache.org/jira/browse/FLINK-19012> <
> >> https://issues.apache.org/jira/browse/FLINK-19012 <
> https://issues.apache.org/jira/browse/FLINK-19012>>)
> >> E2E test fails with "Cannot register Closeable, this
> >> subtaskCheckpointCoordinator is already closed. Closing argument."
> >>
> >> -> This is a new issue occurred recently. It has occurred several times
> >> and may indicate a bug somewhere and should be taken care of.
> >>
> >> - FLINK-9992 (https://issues.apache.org/jira/browse/FLINK-9992 <
> https://issues.apache.org/jira/browse/FLINK-9992> <
> >> https://issues.apache.org/jira/browse/FLINK-9992 <
> https://issues.apache.org/jira/browse/FLINK-9992>>)
> >> FsStorageLocationReferenceTest#testEncodeAndDecode failed in CI
> >>
> >> -> There is already a PR for it and needs review.
> >>
> >> - FLINK-18842 (https://issues.apache.org/jira/browse/FLINK-18842 <
> https://issues.apache.org/jira/browse/FLINK-18842> <
> >> https://issues.apache.org/jira/browse/FLINK-18842 <
> https://issues.apache.org/jira/browse/FLINK-18842>>)
> >> e2e test failed to download "localhost:/flink.tgz" in "Wordcount on
> >> Docker test"
> >>
> >>
> >>> 在 2020年8月11日，下午2:08，Robert Metzger  写道：
> >>>
> >>> Hi team,
> >>>
> >>> 2 weeks have passed since the last update. None of the test stabilities
> >>> I've mentioned have been addressed since then.
> >>>
> >>> Here's an updated status report of Blockers and Test instabilities:
> >>>
> >>> Blockers <
> >> https://issues.apache.org/jira/browse/FLINK-18682?filter=12349334>:
> >>> Currently 3 blockers (2x Hive, 1x CI Infra)
> >>>
> >>> Test-Instabilities
> >>> 
> >> (total
> >>> 79) which failed recently or frequently:
> >>>
> >>>
> >>> - FLINK-18807 
> >>> FlinkKafkaProducerITCase.testScaleUpAfterScalingDown
> >>> failed with "Timeout expired after 6milliseconds while awaiting
> >>> EndTxn(COMMIT)"
> >>>
> >>> - FLINK-18634 
> >>> FlinkKafkaProducerITCase.testRecoverCommittedTransaction
> >>> failed with "Timeout expired after 6milliseconds while awaiting
> >>> InitProducerId"
> >>>
> >>> - FLINK-16908 
> >>> FlinkKafkaProducerITCase
> >>> testScaleUpAfterScalingDown Timeout expired while initializing
> >>> transactional state in 6ms.
> >>>
> >>> - FLINK-13733 
> >>> FlinkKafkaInternalProducerITCase.testHappyPath fails on Travis
> >>>
> >>> --> The first three tickets seem related.
> >>>
> >>>
> >>> - FLINK-17260 
> >>> StreamingKafkaITCase failure on Azure
> >>>
> >>> --> This one seems really hard to reproduce
> >>>
> >>>
> >>> - FLINK-16768 
> >>> HadoopS3RecoverableWriterITCase.testRecoverWithStateWithMultiPart
> >>> hangs
> >>>
> >>> - FLINK-18374 
> >>>
> >>
> HadoopS3RecoverableWriterITCase.testRecoverAfterMultiplePersistsStateWithMultiPart
> >>> produced no output for 900 seconds
> >>>
> >>> --> nobody seems to feel responsible for these tickets. My guess is
> that
> >>> the S3 connector should have shorter ti

[jira] [Created] (FLINK-19042) HiveTableSourceITCase fails if object reuse is enabled

[Rui Li (Jira)]

Rui Li created FLINK-19042:
--

 Summary: HiveTableSourceITCase fails if object reuse is enabled
 Key: FLINK-19042
 URL: https://issues.apache.org/jira/browse/FLINK-19042
 Project: Flink
  Issue Type: Test
  Components: Connectors / Hive, Tests
Reporter: Rui Li


{{testNonPartitionStreamingSourceWithVectorizedReader}} fails because print 
table sink cannot process {{ColumnarRowData}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] Whether catalog table factory should be used for temporary tables

[Rui Li]

Hi Dev,

Currently temporary generic tables cannot work with hive catalog [1]. When
hive catalog is chosen as the current catalog, planner will use
HiveTableFactory to create source/sink for the temporary
table. HiveTableFactory cannot tell whether a table is temporary or not,
and considers it as a Hive table, which leads to job failure.
I've discussed with Jingsong offline and we believe one solution is to make
planner avoid using catalog table factory for temporary tables. But I'd
also like to hear more opinions from others whether this is the right way
to go. I think a possible alternative is to add an *isTemporary* field
to TableSourceFactory.Context & TableSinkFactory.Context, so that
HiveTableFactory knows how to handle such tables. What do you think?

[1] https://issues.apache.org/jira/browse/FLINK-18999

-- 
Best regards!
Rui Li

Re: [DISCUSS] Whether catalog table factory should be used for temporary tables

[Dawid Wysakowicz]

Hi Rui,

My take is that temporary tables should use the factory of the catalog
they were registered with.

What you are describing sounds very much like a limitation/bug in Hive
catalog only. I'd be in favor of passing the *isTemporary* flag.

Best,

Dawid

On 25/08/2020 09:37, Rui Li wrote:
> Hi Dev,
>
> Currently temporary generic tables cannot work with hive catalog [1]. When
> hive catalog is chosen as the current catalog, planner will use
> HiveTableFactory to create source/sink for the temporary
> table. HiveTableFactory cannot tell whether a table is temporary or not,
> and considers it as a Hive table, which leads to job failure.
> I've discussed with Jingsong offline and we believe one solution is to make
> planner avoid using catalog table factory for temporary tables. But I'd
> also like to hear more opinions from others whether this is the right way
> to go. I think a possible alternative is to add an *isTemporary* field
> to TableSourceFactory.Context & TableSinkFactory.Context, so that
> HiveTableFactory knows how to handle such tables. What do you think?
>
> [1] https://issues.apache.org/jira/browse/FLINK-18999
>



signature.asc
Description: OpenPGP digital signature

[jira] [Created] (FLINK-19043) Translate the page 'Logging' of 'Debugging & Monitoring' into Chinese

[Roc Marshal (Jira)]

Roc Marshal created FLINK-19043:
---

 Summary: Translate the page 'Logging' of 'Debugging & Monitoring' 
into Chinese
 Key: FLINK-19043
 URL: https://issues.apache.org/jira/browse/FLINK-19043
 Project: Flink
  Issue Type: Improvement
  Components: chinese-translation, Documentation
Affects Versions: 1.11.1, 1.10.2, 1.11.0, 1.10.1, 1.10.0
Reporter: Roc Marshal


The page url is : 
[Logging|http://https//ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/logging.html]
 The markdown file location is : flink/docs/monitoring/logging.zh.md



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Whether catalog table factory should be used for temporary tables

[Jingsong Li]

Hi Dawid,

But the temporary table does not belong to Catalog, actually
Catalog doesn't know the existence of the temporary table. Let the table
factory of catalog to create source/sink sounds a little sudden.

If we want to make temporary tables belong to Catalog, I think we need to
involve catalog when creating temporary tables.

Best,
Jingsong

On Tue, Aug 25, 2020 at 3:55 PM Dawid Wysakowicz 
wrote:

> Hi Rui,
>
> My take is that temporary tables should use the factory of the catalog
> they were registered with.
>
> What you are describing sounds very much like a limitation/bug in Hive
> catalog only. I'd be in favor of passing the *isTemporary* flag.
>
> Best,
>
> Dawid
>
> On 25/08/2020 09:37, Rui Li wrote:
> > Hi Dev,
> >
> > Currently temporary generic tables cannot work with hive catalog [1].
> When
> > hive catalog is chosen as the current catalog, planner will use
> > HiveTableFactory to create source/sink for the temporary
> > table. HiveTableFactory cannot tell whether a table is temporary or not,
> > and considers it as a Hive table, which leads to job failure.
> > I've discussed with Jingsong offline and we believe one solution is to
> make
> > planner avoid using catalog table factory for temporary tables. But I'd
> > also like to hear more opinions from others whether this is the right way
> > to go. I think a possible alternative is to add an *isTemporary* field
> > to TableSourceFactory.Context & TableSinkFactory.Context, so that
> > HiveTableFactory knows how to handle such tables. What do you think?
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-18999
> >
>
>

-- 
Best, Jingsong Lee

[jira] [Created] (FLINK-19044) Web UI reports incorrect timeline for the FINISHED state of a subtask

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-19044:
---

 Summary: Web UI reports incorrect timeline for the FINISHED state 
of a subtask
 Key: FLINK-19044
 URL: https://issues.apache.org/jira/browse/FLINK-19044
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Web Frontend
Affects Versions: 1.12.0
Reporter: Caizhi Weng
 Attachments: timeline.jpg

The timeline for the FINISHED state of a subtask is incorrect. The starting 
time of the FINISHED state is larger than its ending time. See the image below:

 !timeline.jpg! 

I discover this bug when running the TPCDS benchmark, but I think it can be 
reproduced by arbitrary tasks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Whether catalog table factory should be used for temporary tables

[Jark Wu]

Hi,

I'm wondering if we always fallback to using SPI for temporary tables, then
how does the create Hive temporary table using Hive dialect work?

IMO, adding an "isTemporary" to the factory context sounds reasonable to
me, because the factory context should describe the full content of create
table DDL.

Best,
Jark


On Tue, 25 Aug 2020 at 16:01, Jingsong Li  wrote:

> Hi Dawid,
>
> But the temporary table does not belong to Catalog, actually
> Catalog doesn't know the existence of the temporary table. Let the table
> factory of catalog to create source/sink sounds a little sudden.
>
> If we want to make temporary tables belong to Catalog, I think we need to
> involve catalog when creating temporary tables.
>
> Best,
> Jingsong
>
> On Tue, Aug 25, 2020 at 3:55 PM Dawid Wysakowicz 
> wrote:
>
> > Hi Rui,
> >
> > My take is that temporary tables should use the factory of the catalog
> > they were registered with.
> >
> > What you are describing sounds very much like a limitation/bug in Hive
> > catalog only. I'd be in favor of passing the *isTemporary* flag.
> >
> > Best,
> >
> > Dawid
> >
> > On 25/08/2020 09:37, Rui Li wrote:
> > > Hi Dev,
> > >
> > > Currently temporary generic tables cannot work with hive catalog [1].
> > When
> > > hive catalog is chosen as the current catalog, planner will use
> > > HiveTableFactory to create source/sink for the temporary
> > > table. HiveTableFactory cannot tell whether a table is temporary or
> not,
> > > and considers it as a Hive table, which leads to job failure.
> > > I've discussed with Jingsong offline and we believe one solution is to
> > make
> > > planner avoid using catalog table factory for temporary tables. But I'd
> > > also like to hear more opinions from others whether this is the right
> way
> > > to go. I think a possible alternative is to add an *isTemporary* field
> > > to TableSourceFactory.Context & TableSinkFactory.Context, so that
> > > HiveTableFactory knows how to handle such tables. What do you think?
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-18999
> > >
> >
> >
>
> --
> Best, Jingsong Lee
>

[DISCUSS] FLIP-138: Declarative Resource management

[Chesnay Schepler]

Hello,

in FLIP-138 we want to rework the way the JobMaster acquires slots, such 
that required resources are declared before a job is scheduled and th 
job execution is adjusted according to the provided resources (e.g., 
reducing parallelism), instead of asking for a fixed number of resources 
during scheduling and failing midway through if not enough resources are 
available.


This is a stepping stone towards reactive mode, where Flink will 
automatically make use of new TaskExecutors being started.


More details can be found here 
.

Re: [DISCUSS] Whether catalog table factory should be used for temporary tables

[Jingsong Li]

Hi Jark,

You raised a good point: Creating the Hive temporary table.
AFAIK, Hive temporary tables should be stored in metastore, Hive metastore
will maintain their life cycle. Correct me if I am wrong.

So actually, if we want to support Hive temporary tables, we should finish
one thing:
- A temporary table should belong to Catalog.
- Instead of current, a temporary table belongs to CatalogManager.

It means, `createTemporaryTable` and `dropTemporaryTable` should be proxied
into the Catalog.
In this situation, actually, we don't need the "isTemporary" flag. (But we
can have it too)

Best,
Jingsong

On Tue, Aug 25, 2020 at 4:32 PM Jark Wu  wrote:

> Hi,
>
> I'm wondering if we always fallback to using SPI for temporary tables, then
> how does the create Hive temporary table using Hive dialect work?
>
> IMO, adding an "isTemporary" to the factory context sounds reasonable to
> me, because the factory context should describe the full content of create
> table DDL.
>
> Best,
> Jark
>
>
> On Tue, 25 Aug 2020 at 16:01, Jingsong Li  wrote:
>
> > Hi Dawid,
> >
> > But the temporary table does not belong to Catalog, actually
> > Catalog doesn't know the existence of the temporary table. Let the table
> > factory of catalog to create source/sink sounds a little sudden.
> >
> > If we want to make temporary tables belong to Catalog, I think we need to
> > involve catalog when creating temporary tables.
> >
> > Best,
> > Jingsong
> >
> > On Tue, Aug 25, 2020 at 3:55 PM Dawid Wysakowicz  >
> > wrote:
> >
> > > Hi Rui,
> > >
> > > My take is that temporary tables should use the factory of the catalog
> > > they were registered with.
> > >
> > > What you are describing sounds very much like a limitation/bug in Hive
> > > catalog only. I'd be in favor of passing the *isTemporary* flag.
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 25/08/2020 09:37, Rui Li wrote:
> > > > Hi Dev,
> > > >
> > > > Currently temporary generic tables cannot work with hive catalog [1].
> > > When
> > > > hive catalog is chosen as the current catalog, planner will use
> > > > HiveTableFactory to create source/sink for the temporary
> > > > table. HiveTableFactory cannot tell whether a table is temporary or
> > not,
> > > > and considers it as a Hive table, which leads to job failure.
> > > > I've discussed with Jingsong offline and we believe one solution is
> to
> > > make
> > > > planner avoid using catalog table factory for temporary tables. But
> I'd
> > > > also like to hear more opinions from others whether this is the right
> > way
> > > > to go. I think a possible alternative is to add an *isTemporary*
> field
> > > > to TableSourceFactory.Context & TableSinkFactory.Context, so that
> > > > HiveTableFactory knows how to handle such tables. What do you think?
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-18999
> > > >
> > >
> > >
> >
> > --
> > Best, Jingsong Lee
> >
>


-- 
Best, Jingsong Lee

Re: [DISCUSS] Whether catalog table factory should be used for temporary tables

[Rui Li]

Hi,

Thanks everyone for your inputs.

Temporary hive table is not supported at the moment. If we want to support
it, I agree with Jingsong that the life cycle of the temporary table should
somehow be bound to the hive catalog. For example, hive catalog should be
responsible to delete the table folder when the temporary table is dropped,
or when hive catalog itself gets unregistered. Whether such a table should
be stored in metastore is another topic and open to discussion. Hive
doesn't store temporary tables in metastore, so we probably won't want to
do it either.

I'm fine with either of the solutions proposed. I think we can add the
`isTemporary` flag to factory context, even though temporary tables
currently don't belong to a catalog. Because conceptually, whether a table
is temporary should be part of the context that a factory may be interested
in.

On Tue, Aug 25, 2020 at 4:48 PM Jingsong Li  wrote:

> Hi Jark,
>
> You raised a good point: Creating the Hive temporary table.
> AFAIK, Hive temporary tables should be stored in metastore, Hive metastore
> will maintain their life cycle. Correct me if I am wrong.
>
> So actually, if we want to support Hive temporary tables, we should finish
> one thing:
> - A temporary table should belong to Catalog.
> - Instead of current, a temporary table belongs to CatalogManager.
>
> It means, `createTemporaryTable` and `dropTemporaryTable` should be
> proxied into the Catalog.
> In this situation, actually, we don't need the "isTemporary" flag. (But we
> can have it too)
>
> Best,
> Jingsong
>
> On Tue, Aug 25, 2020 at 4:32 PM Jark Wu  wrote:
>
>> Hi,
>>
>> I'm wondering if we always fallback to using SPI for temporary tables,
>> then
>> how does the create Hive temporary table using Hive dialect work?
>>
>> IMO, adding an "isTemporary" to the factory context sounds reasonable to
>> me, because the factory context should describe the full content of create
>> table DDL.
>>
>> Best,
>> Jark
>>
>>
>> On Tue, 25 Aug 2020 at 16:01, Jingsong Li  wrote:
>>
>> > Hi Dawid,
>> >
>> > But the temporary table does not belong to Catalog, actually
>> > Catalog doesn't know the existence of the temporary table. Let the table
>> > factory of catalog to create source/sink sounds a little sudden.
>> >
>> > If we want to make temporary tables belong to Catalog, I think we need
>> to
>> > involve catalog when creating temporary tables.
>> >
>> > Best,
>> > Jingsong
>> >
>> > On Tue, Aug 25, 2020 at 3:55 PM Dawid Wysakowicz <
>> dwysakow...@apache.org>
>> > wrote:
>> >
>> > > Hi Rui,
>> > >
>> > > My take is that temporary tables should use the factory of the catalog
>> > > they were registered with.
>> > >
>> > > What you are describing sounds very much like a limitation/bug in Hive
>> > > catalog only. I'd be in favor of passing the *isTemporary* flag.
>> > >
>> > > Best,
>> > >
>> > > Dawid
>> > >
>> > > On 25/08/2020 09:37, Rui Li wrote:
>> > > > Hi Dev,
>> > > >
>> > > > Currently temporary generic tables cannot work with hive catalog
>> [1].
>> > > When
>> > > > hive catalog is chosen as the current catalog, planner will use
>> > > > HiveTableFactory to create source/sink for the temporary
>> > > > table. HiveTableFactory cannot tell whether a table is temporary or
>> > not,
>> > > > and considers it as a Hive table, which leads to job failure.
>> > > > I've discussed with Jingsong offline and we believe one solution is
>> to
>> > > make
>> > > > planner avoid using catalog table factory for temporary tables. But
>> I'd
>> > > > also like to hear more opinions from others whether this is the
>> right
>> > way
>> > > > to go. I think a possible alternative is to add an *isTemporary*
>> field
>> > > > to TableSourceFactory.Context & TableSinkFactory.Context, so that
>> > > > HiveTableFactory knows how to handle such tables. What do you think?
>> > > >
>> > > > [1] https://issues.apache.org/jira/browse/FLINK-18999
>> > > >
>> > >
>> > >
>> >
>> > --
>> > Best, Jingsong Lee
>> >
>>
>
>
> --
> Best, Jingsong Lee
>


-- 
Best regards!
Rui Li

[flink-sql-connector-elasticsearch]相关问题

[Li,Qian(DXM,PB)]

Hi,all:

我在使用Flink SQL CLI向ES6写数据的时候，任务一直执行失败，
Log日志显示没有ElasticsearchSink类，请问是什么原因造成的呢？
我是用的jar包是这个flink-sql-connector-elasticsearch6_2.11-1.11.0.jar，ES版本是6.5。
谢谢～

2020-08-25 17:19:38,245 WARN  org.apache.flink.runtime.taskmanager.Task  [] - 
Source: TableSourceScan(table=[[default_catalog, default_database, 
order_info]], fields=[id, user_id, create_time, operate_time, province_id, or
der_status, total_amount]) -> Calc(select=[user_id, province_id]) ->
Sink: Sink(table=[default_catalog.default_database.user_log_sink_6], 
fields=[user_id, province_id]) (1/1)
(c2d1cb4035d826036f93f6e7749d7119) switched from RUNNING to FAILED.
org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user 
class: org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink
ClassLoader info: URL ClassLoader:
file: 
'/tmp/blobStore-b9a083f0-b8a8-46c0-a1fa-5eeff9ab3399/job_5f7e950500c23b8909dbda1ad41ef6c9/blob_p-949ecf57aaab045c3136da62fe2e2ab39f502c30-a4a8c21072eb5ee42e299ed5f2bba98e'
 (valid JAR)
Class not resolvable through given classloader.
at 
org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:288)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:471)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:393)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:459)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:393)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:155)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:453)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522) 
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) 
[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) 
[flink-dist_2.11-1.11.1.jar:1.11.1]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_51]
Caused by: java.lang.ClassNotFoundException: 
org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
~[?:1.8.0_51]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_51]
at 
org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:61)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
at 
org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
at 
org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_51]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_51]
at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_51]
at 
org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
at 
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613) 
~[?:1.8.0_51]
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518) 
~[?:1.8.0_51]
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774) 
~[?:1.8.0_51]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) 
~[?:1.8.0_51]
at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) 
~[?:1.8.0_51]
at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) 
~[?:1.8.0_51]
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) 
~[?:1.8.0_51]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) 
~[?:1.8.0_51]
at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) 
~[?:1.8.0_51]
at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) 
~[?:1.8.0_51]

[jira] [Created] (FLINK-19045) Remove obsolete option 'taskmanager.network.partition.force-release-on-consumption'

[Stephan Ewen (Jira)]

Stephan Ewen created FLINK-19045:


 Summary: Remove obsolete option 
'taskmanager.network.partition.force-release-on-consumption'
 Key: FLINK-19045
 URL: https://issues.apache.org/jira/browse/FLINK-19045
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Reporter: Stephan Ewen
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19046) Introduce separate classes for PipelinedResultPartition and BoundedBlockingResultPartition

[Stephan Ewen (Jira)]

Stephan Ewen created FLINK-19046:


 Summary: Introduce separate classes for PipelinedResultPartition 
and BoundedBlockingResultPartition
 Key: FLINK-19046
 URL: https://issues.apache.org/jira/browse/FLINK-19046
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.12.0


Currently, the SubPartition classes are specific to the partition type 
(pipelined, batched/blocking) but the parent Partition class is shared.

Given that the partitions behave differently regarding checkpoints, releasing, 
etc. the code is cleaner separated by introducing dedicated classes for the 
{{ResultPartitions}} based on the type.

This is also an important preparation to later have more different 
implementations, like sort-based shuffles.

Important: These new classes will not override any performance critical methods 
(like adding a buffer to the result). They merely specialize certain behaviors 
around checkpointing and cleanup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [flink-sql-connector-elasticsearch]相关问题

[taochanglian]

pom里面把这个依赖设置为provided,然后将这个jar包copy到服务器上flink的lib中，然后打包项目，在运行，试一下。

在 2020/8/25 17:38, Li,Qian(DXM,PB) 写道:

Hi,all:

我在使用Flink SQL CLI向ES6写数据的时候，任务一直执行失败，
Log日志显示没有ElasticsearchSink类，请问是什么原因造成的呢？
我是用的jar包是这个flink-sql-connector-elasticsearch6_2.11-1.11.0.jar，ES版本是6.5。
谢谢～

2020-08-25 17:19:38,245 WARN  org.apache.flink.runtime.taskmanager.Task  [] - 
Source: TableSourceScan(table=[[default_catalog, default_database, 
order_info]], fields=[id, user_id, create_time, operate_time, province_id, or
der_status, total_amount]) -> Calc(select=[user_id, province_id]) ->
Sink: Sink(table=[default_catalog.default_database.user_log_sink_6], 
fields=[user_id, province_id]) (1/1)
(c2d1cb4035d826036f93f6e7749d7119) switched from RUNNING to FAILED.
org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user 
class: org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink
ClassLoader info: URL ClassLoader:
 file: 
'/tmp/blobStore-b9a083f0-b8a8-46c0-a1fa-5eeff9ab3399/job_5f7e950500c23b8909dbda1ad41ef6c9/blob_p-949ecf57aaab045c3136da62fe2e2ab39f502c30-a4a8c21072eb5ee42e299ed5f2bba98e'
 (valid JAR)
Class not resolvable through given classloader.
 at 
org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:288)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
 at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:471)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
 at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:393)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
 at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:459)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
 at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:393)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
 at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:155)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
 at 
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:453)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
 at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522) 
~[flink-dist_2.11-1.11.1.jar:1.11.1]
 at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) 
[flink-dist_2.11-1.11.1.jar:1.11.1]
 at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) 
[flink-dist_2.11-1.11.1.jar:1.11.1]
 at java.lang.Thread.run(Thread.java:745) [?:1.8.0_51]
Caused by: java.lang.ClassNotFoundException: 
org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink
 at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
~[?:1.8.0_51]
 at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_51]
 at 
org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:61)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
 at 
org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
 at 
org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
 at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_51]
 at java.lang.Class.forName0(Native Method) ~[?:1.8.0_51]
 at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_51]
 at 
org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)
 ~[flink-dist_2.11-1.11.1.jar:1.11.1]
 at 
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613) 
~[?:1.8.0_51]
 at 
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518) 
~[?:1.8.0_51]
 at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774) 
~[?:1.8.0_51]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) 
~[?:1.8.0_51]
 at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) 
~[?:1.8.0_51]
 at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) 
~[?:1.8.0_51]
 at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) 
~[?:1.8.0_51]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) 
~[?:1.8.0_51]
 at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) 
~[?:1.8.0_51]
 at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) 
~[?:1.8.0_51]

[jira] [Created] (FLINK-19047) Move unaligned checkpoint methods from ResultPartition to separate interface.

[Stephan Ewen (Jira)]

Stephan Ewen created FLINK-19047:


 Summary: Move unaligned checkpoint methods from ResultPartition to 
separate interface.
 Key: FLINK-19047
 URL: https://issues.apache.org/jira/browse/FLINK-19047
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.12.0


All ResultPartitions have the unaligned checkpointing methods, because some do 
not support checkpoints and throw an {{UnsupportedOperationException}}.

I suggest to follow the idea of interface segregation to put the methods 
relating to unaligned checkpoints in a dedicated interface.





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19048) suuport line format for table connector

[badqiu (Jira)]

badqiu created FLINK-19048:
--

 Summary: suuport line format for table connector
 Key: FLINK-19048
 URL: https://issues.apache.org/jira/browse/FLINK-19048
 Project: Flink
  Issue Type: New Feature
  Components: API / Type Serialization System
Reporter: badqiu
 Attachments: LineFormatFactory.java, 
LineRowDataDeserializationSchema.java, LineRowDeserializationSchema.java, 
LineRowFormatFactory.java, LineRowSchemaConverter.java, 
LineRowSerializationSchema.java

Native string data format. No data conversion is done.
This format is particularly friendly to data without time attributes. With UDF, 
writing your own data analysis will be much more convenient.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [flink-sql-connector-elasticsearch]相关问题

[Xintong Song]

Hi all,

I'd like to kindly remind you that dev@flink.apache.org is an international
mailing list. It is important to have the discussions in English, so that
every subscriber can understand. Discussions in Chinese can be posted in
the user...@flink.apache.org mailing list. Please refer to the community
information page[1] for the available mailing lists and how to subscribe.

请注意，dev@flink.apache.org 是面向全球的邮件列表。为了便于所有订阅者理解邮件的内容，请使用英文进行讨论。中文讨论可以到
user...@apache.flink.org 进行。有关邮件列表的详细信息及订阅方式，请参考社区信息页面[1]。

Thank you~

Xintong Song


[1] https://flink.apache.org/community.html#mailing-lists

On Tue, Aug 25, 2020 at 5:57 PM taochanglian  wrote:

> pom里面把这个依赖设置为provided,然后将这个jar包copy到服务器上flink的lib中，然后打包项目，在运行，试一下。
>
> 在 2020/8/25 17:38, Li,Qian(DXM,PB) 写道:
> > Hi,all:
> >
> > 我在使用Flink SQL CLI向ES6写数据的时候，任务一直执行失败，
> > Log日志显示没有ElasticsearchSink类，请问是什么原因造成的呢？
> > 我是用的jar包是这个flink-sql-connector-elasticsearch6_2.11-1.11.0.jar，ES版本是6.5。
> > 谢谢～
> >
> > 2020-08-25 17:19:38,245 WARN  org.apache.flink.runtime.taskmanager.Task
> [] - Source: TableSourceScan(table=[[default_catalog, default_database,
> order_info]], fields=[id, user_id, create_time, operate_time, province_id,
> or
> > der_status, total_amount]) -> Calc(select=[user_id, province_id]) ->
> > Sink: Sink(table=[default_catalog.default_database.user_log_sink_6],
> fields=[user_id, province_id]) (1/1)
> > (c2d1cb4035d826036f93f6e7749d7119) switched from RUNNING to FAILED.
> > org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot
> load user class:
> org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink
> > ClassLoader info: URL ClassLoader:
> >  file:
> '/tmp/blobStore-b9a083f0-b8a8-46c0-a1fa-5eeff9ab3399/job_5f7e950500c23b8909dbda1ad41ef6c9/blob_p-949ecf57aaab045c3136da62fe2e2ab39f502c30-a4a8c21072eb5ee42e299ed5f2bba98e'
> (valid JAR)
> > Class not resolvable through given classloader.
> >  at
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:288)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]
> >  at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:471)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]
> >  at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:393)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]
> >  at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:459)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]
> >  at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:393)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]
> >  at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:155)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]
> >  at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:453)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]
> >  at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]
> >  at
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> >  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> >  at java.lang.Thread.run(Thread.java:745) [?:1.8.0_51]
> > Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink
> >  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> ~[?:1.8.0_51]
> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> ~[?:1.8.0_51]
> >  at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:61)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]
> >  at
> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]
> >  at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]
> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ~[?:1.8.0_51]
> >  at java.lang.Class.forName0(Native Method) ~[?:1.8.0_51]
> >  at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_51]
> >  at
> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]
> >  at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
> ~[?:1.8.0_51]
> >  at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
> ~[?:1.8.0_51]
> >  at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
> ~[?:1.8.0_51]
> >  at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> ~[?:1.8.0_5

Re: [DISCUSS] FLIP-36 - Support Interactive Programming in Flink Table API

[Xuannan Su]

Hi Timo,

Thanks for your comments. After the offline discussion, I have updated the FLIP 
with the following change.

1. Update the end to end process
a. The Table.cache method should only wrap the origin query operation with 
CacheOperation.
b. The planner will add the CacheSink or replace subtree with CacheSource 
while translating the QueryOperation to rel node. The CacheSink and CacheSource 
are treated as regular sinks and sources while the planner optimizes and 
translates the rel nodes.
c. The StreamGraphGenerator will recognize the CacheSink and CacheSource 
and generate the StreamGraph accordingly. When it sees the CacheSink, it will 
remove the CacheSink and set the cache flag in the StreamNode. When it sees the 
CacheSource, it will include the ClusterPartitionDescriptor in the StreamNode.
d. The JobGraph generator will not modify the graph. It only set the result 
partition type of the intermediate result to BLOCKING_PERSISTENT if it sees a 
StreamNode with the cache flag set or passes the ClusterPartitionDescriptor 
from the StreamNode to the JobVertex if the StreamNode has the 
ClusterPartitionDescriptor.
e. The ClusterPartitionDescriptor will be included in the JobResult and 
sent back to the client.
2. The metadata of the cached table will be stored in the CatalogManager 
instead of the TableEnvironment
3. The Table.cache method returns a CachedTable, which is a subclass of Table.
4. Add a paragraph to explain that the cache table will not work in Per-Job 
Mode cluster.
5. Add SQL integration and Cache Eviction in the future work section.

As of the @Public and @PublicEvolving interface, I think it should be covered 
in the public interface section, i.e., Table and TableEnvironment. Other than 
those, all the changes should only affect the internal class.

Please let me know if you have any further comments.

Best,
Xuannan
On Aug 10, 2020, 9:32 PM +0800, Timo Walther , wrote:
> Hi Xuannan,
>
> sorry for joining the discussion so late. I agree that this is a very
> nice and useful feature. However, the impact it has to many components
> in the stack requires more discussion in my opinion.
>
> 1) Separation of concerns:
> The current design seems to mix different layers. We should make sure
> that all layers do what they are supposed to do:
>
> 1a) The FLIP states: "The cache() method returns a new Table object with
> a flag set."
>
> The `Table` object is just a thin API class that wraps a
> `QueryOperation`. Other than that the `Table` object should not contain
> futher state. The tree of `QueryOperation` should be an immutable,
> independent data structure that can be passed around and will eventually
> be passed to the `Planner`.
>
> The mentioned `CacheSink` should be added by the optimizer. It is not
> the responsibility of the API do perform optimizer-like tasks. A call to
> `t1.cache()` should simply wrap the original operation into something
> like `CacheOperation(t1.getQueryOperation)`. A `CacheOperation` could
> still extend from `QueryOperation` and assign a unique string identifier
> already. A specialized `StreamTransformation` would be necessary during
> translation.
>
> 1b) The FLIP states: "The default table service stores the metadata in
> the client (e.g. TableEnvironment)"
>
> `TableEnvironment` is not a client. Similar to `Table`, it is an API
> class that just delegates to other session components. Currently, the
> table environment has (again) to many responsibilities that should
> better be split into other components. The table's `Executor` is the
> client that performs the cluster communication. But in general it also
> just delegates to `org.apache.flink.core.execution.PipelineExecutor`.
> IMO the `PipelineExecutor` is a better fit for a back-and-forth
> communication to determine existing cluster partitions and modify the
> job graph. Or even further down the stack, because as far as I know,
> `PipelineExecutor` works with `StreamGraph`.
>
> `flink-table-api-java` has no dependency to `flink-runtime`. This has
> been done on purpose.
>
> 2) API
> I still see the rejected option 2 a good fit to expose this feature.
>
> A `Table.cache(): CachedTable` with `CachedTable.invalidate(): void` and
> maybe `CachedTable.getId(): String` makes the feature and its operations
> very explicit. It also avoids following up questions such as:
>
> Will `invalidateCache()` be transitively propagated in
> `t1.cache().join(t2.cache()).invalidateCache()`?
>
> Or as the FLIP states:
>
> `Table t3 = t1.select(...) // cache will NOT be used.`
> but
> `t1.invalidateCache() // cache will be released`
>
> This sounds a bit contradicting to me. Because sometimes the
> `t1.cache()` has implications on t1 and sometimes not.
>
> 3) Big picture
>
> After reading the FLIP, I still don't understand how a user can
> configure or control the table service. Will we offer options through
> `TableConfig` or `TableEnvironment` or is this configuration done via
> ConfigOptions for lowe

[jira] [Created] (FLINK-19049) TableEnvironmentImpl.executeInternal() does not wait for the final job status

[Robert Metzger (Jira)]

Robert Metzger created FLINK-19049:
--

 Summary: TableEnvironmentImpl.executeInternal() does not wait for 
the final job status
 Key: FLINK-19049
 URL: https://issues.apache.org/jira/browse/FLINK-19049
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Runtime
Affects Versions: 1.12.0
Reporter: Robert Metzger


While working on another change, I realized that the 
{{FunctionITCase.testInvalidUseOfTableFunction()}} tests throws a 
NullPointerException during execution.

This error is not visible, because TableEnvironmentImpl.executeInternal() does 
not wait for the final job status.
It submits the job using the job client ({{JobClient jobClient = 
execEnv.executeAsync(pipeline);}}), and it doesn't wait for the job to complete 
before returning a result. 

This is the null pointer that is hidden:
{code}

Caused by: org.apache.flink.util.FlinkException: Failed to execute job 
'insert-into_default_catalog.default_database.SinkTable'.
at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1823)
at 
org.apache.flink.table.planner.delegation.ExecutorBase.executeAsync(ExecutorBase.java:57)
at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:681)
... 34 more
Caused by: java.util.concurrent.CompletionException: 
org.apache.flink.runtime.JobException: Recovery is suppressed by 
NoRestartBackoffTimeStrategy
at 
org.apache.flink.client.ClientUtils.waitUntilJobInitializationFinished(ClientUtils.java:148)
at 
org.apache.flink.client.program.PerJobMiniClusterFactory.lambda$submitJob$2(PerJobMiniClusterFactory.java:92)
at 
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
at 
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
at 
java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:457)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at 
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by 
NoRestartBackoffTimeStrategy
at 
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116)
at 
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78)
at 
org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:195)
at 
org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:188)
at 
org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:182)
at 
org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:523)
at 
org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:422)
at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:284)
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:199)
at 
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailb

Re: Next Stateful Functions Release

[David Anderson]

Igal,

The feature set you propose sounds great to me -- as a user I see
plenty there to get excited about. As for the feature freeze date, I
don't really have an informed opinion.

David

On Mon, Aug 24, 2020 at 10:15 AM Igal Shilman  wrote:
>
> Hi Flink devs,
>
> We have a few upcoming / implemented features for Stateful Functions on the
> radar, and would like to give a heads up on what to expect for the next
> release:
>
> 1. Upgrade support for Flink 1.11.x. [FLINK-18812]
> 2. Fine grained control on remote state configuration, such as state TTL.
> [FLINK-17954]
> 3. New state construct for dynamic state registration [FLINK-18316]
> 4. Add a DataStream API to StateFun [FLINK-19001]
> 5. Support async handlers for the Python SDK [FLINK-18518]
> 6. Add more metrics around async operations and backpressure [FLINK-19020]
> 7. Out-of-box support for common storage systems in flink-statefun Docker
> image [FLINK-19019]
>
> With these we think the project will be in a good spot for the next release.
> What do you think about aiming at 10.9.2020 for a feature freeze for
> StateFun 2.2?
>
> Kind regards,
> Igal.

[jira] [Created] (FLINK-19050) Doc of MAX_DECIMAL_PRECISION should be DECIMAL

[Pua (Jira)]

Pua created FLINK-19050:
---

 Summary: Doc of MAX_DECIMAL_PRECISION should be DECIMAL
 Key: FLINK-19050
 URL: https://issues.apache.org/jira/browse/FLINK-19050
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.11.1
Reporter: Pua


{code:java}
// Define MAX/MIN precision of TIMESTAMP type according to PostgreSQL docs:
// 
https://www.postgresql.org/docs/12/datatype-numeric.html#DATATYPE-NUMERIC-DECIMAL
private static final int MAX_DECIMAL_PRECISION = 1000;
private static final int MIN_DECIMAL_PRECISION = 1;
{code}
[https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/PostgresDialect.java#L43]

the doc of decimal precision constants should be DECIMAL not  TIMESTAMP



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Flink Committer: David Anderson

[Konstantin Knauf]

Congrats, David. Well deserved.

On Thu, Aug 20, 2020 at 5:06 AM Matt Wang  wrote:

> Congrats David!
>
>
> --
>
> Best,
> Matt Wang
>
>
> On 08/20/2020 10:44，Zhijiang wrote：
> Congratulations David!
>
>
> --
> From:Jeff Zhang 
> Send Time:2020年8月19日(星期三) 23:34
> To:dev 
> Subject:Re: [ANNOUNCE] New Flink Committer: David Anderson
>
> Congratulations David!
>
> Kostas Kloudas  于2020年8月19日周三 下午11:32写道：
>
> Congratulations David!
>
> Kostas
>
> On Wed, Aug 19, 2020 at 2:33 PM Arvid Heise  wrote:
>
> Congrats David!
>
> On Wed, Aug 19, 2020 at 11:17 AM Fabian Hueske 
> wrote:
>
> Congrats David, well deserved!
>
> Cheers,
> Fabian
>
> Am Mi., 19. Aug. 2020 um 11:05 Uhr schrieb Marta Paes Moreira <
> ma...@ververica.com>:
>
> Congrats, David! Thanks for being the Flink Stack Overflow hawk (on
> top
> of
> everything else, of course)!
>
> Marta
>
> On Thu, Aug 13, 2020 at 5:26 AM Roc Marshal  wrote:
>
>
>
>
>
>
>
> Congratulations David!
>
>
> Best,
> Roc Marshal.
>
>
>
>
>
>
>
>
>
>
>
> At 2020-08-12 15:50:47, "Robert Metzger" 
> wrote:
> Hi everyone,
>
> On behalf of the PMC, I'm very happy to announce David Anderson
> as a
> new
> Apache
> Flink committer.
>
> David has been a Flink community member for a long time. His first
> commit
> dates back to 2016, code changes mostly involve the
> documentation, in
> particular with the recent contribution of Flink training
> materials.
> Besides that, David has been giving numerous talks and trainings
> on
> Flink.
> On StackOverflow, he's among the most active helping Flink users
> to
> solve
> their problems (2nd in the all-time ranking, 1st in the last 30
> days). A
> similar level of activity can be found on the user@ mailing list.
>
> Please join me in congratulating David for becoming a Flink
> committer!
>
> Best,
> Robert
>
>
>
>
>
> --
>
> Arvid Heise | Senior Java Developer
>
> 
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward  - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Toni) Cheng
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>

-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: How jobmanager and task manager communicates with each other ?

[sidhant gupta]

Hi Till,

Thanks for the reply.

(1) If we are not using Flink's HA services then how we can dynamically
configure task manager nodes to connect to job manager? Any suggestions or
best practices?

(2) Which and how flink's HA service can be used for the service discovery
of job manager ?

Regards
Sidhant Gupta


On Tue, Aug 25, 2020, 11:51 AM Till Rohrmann  wrote:

> Hi Sidhant,
>
> the cluster components use tcp to communicate with each other. If you are
> not using Flink's HA services, then the TaskManager nodes need to be
> configured with the JobManager's address to connect to them. If you are
> using HA services, then the service discovery happens through the HA
> services. One requirement for Flink to work is that the different cluster
> nodes on which a Flink process is started can communicate with each other.
>
> Cheers,
> Till
>
> On Mon, Aug 24, 2020 at 6:26 PM sidhant gupta  wrote:
>
>> ++dev@flink.apache.org
>>
>> On Mon, Aug 24, 2020, 7:31 PM sidhant gupta  wrote:
>>
>> > Hi User
>> >
>> > How jobmanager and task manager communicates with each other ? How to
>> set
>> > connection between jobmanager and task manager running in different/same
>> > ec2 instance ? Is it http or tcp ? How the service discovery works ?
>> >
>> > Thanks
>> > Sidhant Gupta
>> >
>>
>

Re: How jobmanager and task manager communicates with each other ?

[Andrey Zagrebin]

Hi Sidhant,

(1) If we are not using Flink's HA services then how we can dynamically
> configure task manager nodes to connect to job manager? Any suggestions or
> best practices?

Not sure what you mean by 'dynamically'.
I think you have to restart the task manager with the new configuration
to connect to another job manager.

(2) Which and how flink's HA service can be used for the service discovery
> of job manager ?

You can check the docs for the zookeeper implementation of the HA in Flink
[1]

Best,
Andrey

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/jobmanager_high_availability.html

On Tue, Aug 25, 2020 at 5:45 PM sidhant gupta  wrote:

> Hi Till,
>
> Thanks for the reply.
>
> (1) If we are not using Flink's HA services then how we can dynamically
> configure task manager nodes to connect to job manager? Any suggestions or
> best practices?
>
> (2) Which and how flink's HA service can be used for the service discovery
> of job manager ?
>
> Regards
> Sidhant Gupta
>
>
> On Tue, Aug 25, 2020, 11:51 AM Till Rohrmann  wrote:
>
>> Hi Sidhant,
>>
>> the cluster components use tcp to communicate with each other. If you are
>> not using Flink's HA services, then the TaskManager nodes need to be
>> configured with the JobManager's address to connect to them. If you are
>> using HA services, then the service discovery happens through the HA
>> services. One requirement for Flink to work is that the different cluster
>> nodes on which a Flink process is started can communicate with each other.
>>
>> Cheers,
>> Till
>>
>> On Mon, Aug 24, 2020 at 6:26 PM sidhant gupta 
>> wrote:
>>
>>> ++dev@flink.apache.org
>>>
>>> On Mon, Aug 24, 2020, 7:31 PM sidhant gupta  wrote:
>>>
>>> > Hi User
>>> >
>>> > How jobmanager and task manager communicates with each other ? How to
>>> set
>>> > connection between jobmanager and task manager running in
>>> different/same
>>> > ec2 instance ? Is it http or tcp ? How the service discovery works ?
>>> >
>>> > Thanks
>>> > Sidhant Gupta
>>> >
>>>
>>

Re: [DISCUSS] Remove Kafka 0.10.x connector (and possibly 0.11.x)

[Konstantin Knauf]

Hi Aljoscha,

I am assuming you're asking about dropping the
flink-connector-kafka-0.10/0.11 modules, right? Or are you talking about
removing support for Kafka 0.10/0.11 from the universal connector?

I am in favor of removing flink-connector-kafka-0.10/0.11 in the next
release. These modules would still be available in Flink 1.11- as a
reference, and could be used with Flink 1.12+ with small or no
modifications. To my knowledge, you also use the universal Kafka connector
with 0.10 brokers, but there might be a performance penalty if I remember
correctly. In general, I find it important to continuously reduce baggage
that accumulates over time and this seems like a good opportunity.

Best,

Konstantin

On Tue, Aug 25, 2020 at 4:59 AM Paul Lam  wrote:

> Hi Aljoscha,
>
> I'm lightly leaning towards keeping the 0.10 connector, for Kafka 0.10
> still has a steady user base in my observation.
>
> But if we drop 0.10 connector, can we ensure the users would be able to
> smoothly migrate to 0.11 connector/universal connector?
>
> If I remember correctly, the universal connector is compatible with 0.10
> brokers, but I want to double check that.
>
> Best,
> Paul Lam
>
> 2020年8月24日 22:46，Aljoscha Krettek  写道：
>
> Hi all,
>
> this thought came up on FLINK-17260 [1] but I think it would be a good
> idea in general. The issue reminded us that Kafka didn't have an
> idempotent/fault-tolerant Producer before Kafka 0.11.0. By now we have had
> the "modern" Kafka connector that roughly follows new Kafka releases for a
> while and this one supports Kafka cluster versions as far back as 0.10.2.0
> (I believe).
>
> What are your thoughts on removing support for older Kafka versions? And
> yes, I know that we had multiple discussions like this in the past but I'm
> trying to gauge the current sentiment.
>
> I'm cross-posting to the user-ml since this is important for both users
> and developers.
>
> Best,
> Aljoscha
>
> [1] https://issues.apache.org/jira/browse/FLINK-17260
>
>
>

-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: [DISCUSS] Removing deprecated methods from DataStream API

[Konstantin Knauf]

I would argue that the guarantees of @Public methods that became
ineffective were broken when they became ineffective (and were deprecated).

   - ExecutionConfig#disable/enableSysoutLogging (deprecated in 1.10)
   - ExecutionConfig#set/isFailTaskOnCheckpointError (deprecated in 1.9)

Removing these methods seems like the better of two evils to me as it shows
users that they have been using no-ops for some time.

On Thu, Aug 20, 2020 at 10:50 AM Stephan Ewen  wrote:

> We have removed some public methods in the past, after a careful
> deprecation period, if they were not well working any more.
>
> The sentiment I got from users is that careful cleanup is in fact
> appreciated, otherwise things get confusing over time (the deprecated
> methods cause noise in the API).
> Still, we need to be very careful here.
>
> I would suggest to
>   - start with the non-public breaking methods
>   - remove fold() (very long deprecated)
>   - remove split() buggy
>
> Removing the env.socketStream() and env.fileStream() methods would
> probably be good, too. They are very long deprecated and they don't work
> well (with checkpoints) and the sources are the first thing a user needs to
> understand when starting with Flink, so removing noise there is super
> helpful.
>
>
> On Thu, Aug 20, 2020 at 8:53 AM Dawid Wysakowicz 
> wrote:
>
>> Hey Till,
>>
>> You've got a good point here. Removing some of the methods would cause
>> breaking the stability guarantees. I do understand if we decide not to
>> remove them for that reason, let me explain though why I am thinking it
>> might make sense to remove them already. First of all I am a bit afraid it
>> might take a long time before we arrive at the 2.0 version. We have not
>> ever discussed that in the community. At the same time a lot of the methods
>> already don't work or are buggy, and we do not fix them any more.
>>
>> Methods which removing would not break the Public guarantees:
>>
>>- ExecutionConfig#set/getCodeAnalysisMode (deprecated in 1.9)
>>- RuntimeContext#getAllAccumulators (deprecated in 0.10)
>>- ExecutionConfig#isLatencyTrackingEnabled (deprecated in 1.7)
>>- 
>> StreamExecutionEnvironment#setNumberOfExecutionRetries/getNumberOfExecutionRetries
>>(not the equivalent in the ExecutionConfig)
>>- StreamExecutionEnvironment#setStateBackend(AbstractStateBackend)
>>(deprecated in 1.5)
>>
>> Methods which removing would break the Public guarantees:
>>
>> which have no effect:
>>
>>- ExecutionConfig#disable/enableSysoutLogging (deprecated in 1.10)
>>- ExecutionConfig#set/isFailTaskOnCheckpointError (deprecated in 1.9)
>>
>> which are buggy or discouraged and thus we do not support fixing them:
>>
>>- DataStream#split (deprecated in 1.8)
>>- DataStream#fold and all related classes and methods such as
>>FoldFunction, FoldingState, FoldingStateDescriptor ... (deprecated in
>>1.3/1.4)
>>
>> The methods like:
>>
>>- 
>> StreamExecutionEnvironment#readFile,readFileStream(...),socketTextStream(...),socketTextStream(...),
>>
>>- methods in (Connected)DataStream that specify keys as either
>>indices or field names
>>-
>>ExecutionConfig#setNumberOfExecutionRetries/getNumberOfExecutionRetries
>>
>> should be working just fine and I feel the least eager to remove those.
>>
>> I'd suggest I will open PRs for removing the methods that will not cause
>> breakage of the Public guarantees as the general feedback was rather
>> positive. For the rest I do understand the resentment to do so and will not
>> do it in the 1.x branch. Still I think it is valuable to have the
>> discussion.
>>
>> Best,
>>
>> Dawid
>>
>>
>> On 18/08/2020 09:26, Till Rohrmann wrote:
>>
>> Having looked at the proposed set of methods to remove I've noticed that
>> some of them are actually annotated with @Public. According to our
>> stability guarantees, only major releases (1.0, 2.0, etc.) can break APIs
>> with this annotation. Hence, I believe that we cannot simply remove them
>> unless the community decides to change the stability guarantees we give or
>> by making the next release a major release (Flink 2.0).
>>
>> Cheers,
>> Till
>>
>> On Tue, Aug 18, 2020 at 5:57 AM Yun Gao  wrote:
>>
>>> +1 for removing the methods that are deprecated for a while & have
>>> alternative methods.
>>>
>>> One specific thing is that if we remove the DataStream#split, do we
>>> consider enabling side-output in more operators in the future ? Currently
>>> it should be only available in ProcessFunctions, but not available to other
>>> commonly used UDF like Source or AsyncFunction[1].
>>>
>>> One temporary solution occurs to me is to add a ProcessFunction after
>>> the operators want to use side-output. But I think the solution is not very
>>> direct to come up with and if it really works we might add it to the
>>> document of side-output.
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-7954
>>>
>>> Best,
>>>  Yun
>>>
>>> --Original Mail

[ANNOUNCE] Weekly Community Update 2020/31-34

[Konstantin Knauf]

Dear community,

The "weekly" community update is back after a short summer break! This time
I've tried to cover most of what happened during the last four weeks, but I
might pick up some older topics in the next weeks' updates, too.

Activity on the dev@ mailing list has picked up quite a bit as feature
development & design for the next releases of Apache Flink and Apache Flink
Stateful Functions is going at full steam. In detail:

Flink Development
==

* [releases] [Flink 1.12] The work on Flink 1.12 is well underway with
feature freeze planned for end of October [1]. Our release managers Robert
& Dian are periodically reminding the developer community of current
blockers to reduce time during release testing for this release [2].

* [releases] [Stateful Functions 2.2] Igal has started a discussion
releasing Stateful Functions 2.2. soon (proposed feature freeze:
September 10). The most notable feature is maybe the option to embed a
stateful functions module in a DataStream program via DataStream
Ingress/Egress. Checkout [3] for a full list of the planned features.

* [releases] [Flink 1.10] Flink 1.10.2 was released. [4]

* [apis] Besides the Stateful Functions API, Flink currently has three
top-level APIs: DataStream (streaming), DataSet (batch) and TableAPI/SQL
(unified). A major step towards the goal of a truly unified batch and
stream processing engine is the unification of the DataStream/DataSet APIs.
This is one of the main topics of the upcoming release(s), specifically:
* Aljoscha has published FLIP-131 [5] proposing to deprecate and
eventually drop the DataSet API. In order to still support the same breadth
of use cases, we need to make sure that all its use cases are covered by
the two remaining APIs: a unified DataStream API and the Table API. These
changes are not part of FLIP-131 itself, but are covered in other FLIPs,
which already exist (like FLIP-27 [6] or FLIP-129 [7]) or will be published
over the next few weeks like FLIP-134 (see below). [8]
* Most importantly, FLIP-134 [9] discusses how the DataStream API could
be used to efficiently execute batch workloads in the future. In essence
the FLIP proposes to introduce a BATCH and a STREAMING execution mode for
DataStream programs. The STREAMING mode corresponds to the current
behavior, while the BATCH mode adjusts the behavior in various areas to fit
the requirements of batch processing, e.g. pipelined scheduling with region
failover, blocking shuffles, no checkpointing, no watermarks, ... [10]

* [apis] Time proposes FLIP-136 to improve the interoperability between the
Data Stream and Table API. The FLIP covers the conversion between
DataStream <-> Table (incl. cnangelong streams, watermarks, etc.) as well
as more additional support for working with the Row type in the DataStream
API. [11]

* [datastream api] Dawid proposes to remove a set of deprecated methods
from the DataStream API. [12]

* [runtime] Yuan Mei has started a discussion on FLIP-135 to introduce
task-local recovery. The FLIP is about the introduction of a new
failover/recovery strategy for Flink Jobs, that trades consistency for
availability. Specifically, in the case of approximate task-local recovery
the failure of some tasks would not trigger a restart of the rest of the
job, but in turn you can expect data loss or duplication. [13]

* [python] Xingbo Huang proposes to extend the support of Pandas/vectorized
functions from scalar functions to aggregate functions. For more details on
Pandas support on PyFlink see the blog post linked below. [14]

* [connectors] Aljoscha has started a discussion on dropping support for
Kafka 0.10/0.11 in Flink 1.12+. [15]

* [connectors] Robert has revived the discussion on adding support for
Hbase 2.3.x. There is a consensus to add the HBase 2.x connector Apache
Flink, but no consensus yet on whether to move the existing HBase 1.x from
the Flink project to Apache Bahir, too. [16]

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Planning-Flink-1-12-tp43348.html
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Release-1-12-Stale-blockers-and-build-instabilities-tp43477.html
[3]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Next-Stateful-Functions-Release-tp44063.html
[4] https://flink.apache.org/news/2020/08/25/release-1.10.2.html
[5]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741
[6]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface?src=contextnavpagetreemode

[7]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-129%3A+Refactor+Descriptor+API+to+register+connectors+in+Table+API
[8]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-131-Consolidate-the-user-facing-Dataflow-SDKs-APIs-and-deprecate-the-DataSet-API-tp43521.html

[9]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=15

Re: [DISCUSS] Remove Kafka 0.10.x connector (and possibly 0.11.x)

[Chesnay Schepler]

+1 to remove both the 1.10 and 1.11 connectors.

The connectors have not been actively developed for some time. They are 
basically just sitting around causing noise by causing test 
instabilities and eating CI time.
It would  also allow us to really simplify the module structure of the 
Kafka connectors.


Users may continue to use the 1.11 version of the connectors with future 
Flink versions, and we may even provide critical bug fixes in a 1.11 
bugfix release (albeit unlikely).


While ultimately this is a separate topic I would also be in favor of 
removing any migration paths we have from 0.11 to the universal connector;
as these are already present in 1.11 users may migrate to the universal 
connector before jumping to Flink 1.12+.


On 25/08/2020 18:49, Konstantin Knauf wrote:

Hi Aljoscha,

I am assuming you're asking about dropping the 
flink-connector-kafka-0.10/0.11 modules, right? Or are you talking 
about removing support for Kafka 0.10/0.11 from the universal connector?


I am in favor of removing flink-connector-kafka-0.10/0.11 in the next 
release. These modules would still be available in Flink 1.11- as a 
reference, and could be used with Flink 1.12+ with small or no 
modifications. To my knowledge, you also use the universal Kafka 
connector with 0.10 brokers, but there might be a performance 
penalty if I remember correctly. In general, I find it important 
to continuously reduce baggage that accumulates over time and this 
seems like a good opportunity.


Best,

Konstantin

On Tue, Aug 25, 2020 at 4:59 AM Paul Lam > wrote:


Hi Aljoscha,

I'm lightly leaning towards keeping the 0.10 connector, for Kafka
0.10 still has a steady user base in my observation.

But if we drop 0.10 connector, can we ensure the users would be
able to smoothly migrate to 0.11 connector/universal connector?

If I remember correctly, the universal connector is compatible
with 0.10 brokers, but I want to double check that.

Best,
Paul Lam


2020年8月24日 22:46，Aljoscha Krettek mailto:aljos...@apache.org>> 写道：

Hi all,

this thought came up on FLINK-17260 [1] but I think it would be a
good idea in general. The issue reminded us that Kafka didn't
have an idempotent/fault-tolerant Producer before Kafka 0.11.0.
By now we have had the "modern" Kafka connector that roughly
follows new Kafka releases for a while and this one supports
Kafka cluster versions as far back as 0.10.2.0 (I believe).

What are your thoughts on removing support for older Kafka
versions? And yes, I know that we had multiple discussions like
this in the past but I'm trying to gauge the current sentiment.

I'm cross-posting to the user-ml since this is important for both
users and developers.

Best,
Aljoscha

[1] https://issues.apache.org/jira/browse/FLINK-17260




--

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

HadoopOutputFormat has issues with LocalExecutionEnvironment?

[Ken Krugler]

Hi devs,

In HadoopOutputFormat.close(), I see code that is trying to rename 
/tmp-r-1 to be /1

But when I run my Flink 1.9.2 code using a local MiniCluster, the actual 
location of the tmp-r-1 file is:

/_temporary/0/task___r_01/tmp-r-1

I think this is because the default behavior of Hadoop’s FileOutputCommitter 
(with algorithm == 1) is to put files in task-specific sub-dirs.

It’s depending on a post-completion “merge paths” action to be taken by what is 
(for Hadoop) the Application Master.

I assume that when running on a real cluster, the 
HadoopOutputFormat.finalizeGlobal() method’s call to commitJob() would do this, 
but it doesn’t seem to be happening when I run locally.

If I set the algorithm version to 2, then “merge paths” is handled by 
FileOutputCommitter immediately, and the HadoopOutputFormat code finds files in 
the expected location.

Wondering if Flink should always be using version 2 of the algorithm, as that’s 
more performant when there are a lot of results (which is why it was added).

Thanks,

— Ken

--
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr

Re: [ANNOUNCE] Apache Flink 1.10.2 released

[Yun Tang]

Thanks for Zhu's work to manage this release and everyone who contributed to 
this!

Best,
Yun Tang

From: Yangze Guo 
Sent: Tuesday, August 25, 2020 14:47
To: Dian Fu 
Cc: Zhu Zhu ; dev ; user 
; user-zh 
Subject: Re: [ANNOUNCE] Apache Flink 1.10.2 released

Thanks a lot for being the release manager Zhu Zhu!
Congrats to all others who have contributed to the release!

Best,
Yangze Guo

On Tue, Aug 25, 2020 at 2:42 PM Dian Fu  wrote:
>
> Thanks ZhuZhu for managing this release and everyone else who contributed to 
> this release!
>
> Regards,
> Dian
>
> 在 2020年8月25日，下午2:22，Till Rohrmann  写道：
>
> Great news. Thanks a lot for being our release manager Zhu Zhu and to all 
> others who have contributed to the release!
>
> Cheers,
> Till
>
> On Tue, Aug 25, 2020 at 5:37 AM Zhu Zhu  wrote:
>>
>> The Apache Flink community is very happy to announce the release of Apache 
>> Flink 1.10.2, which is the first bugfix release for the Apache Flink 1.10 
>> series.
>>
>> Apache Flink® is an open-source stream processing framework for distributed, 
>> high-performing, always-available, and accurate data streaming applications.
>>
>> The release is available for download at:
>> https://flink.apache.org/downloads.html
>>
>> Please check out the release blog post for an overview of the improvements 
>> for this bugfix release:
>> https://flink.apache.org/news/2020/08/25/release-1.10.2.html
>>
>> The full release notes are available in Jira:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12347791
>>
>> We would like to thank all contributors of the Apache Flink community who 
>> made this release possible!
>>
>> Thanks,
>> Zhu
>
>

[DISCUSS] FLIP-139: General Python User-Defined Aggregate Function on Table API

[Wei Zhong]

Hi everyone,

I would like to start discussion about how to support General Python 
User-Defined Aggregate Function on Table API.

FLIP-58[1] has already introduced the stateless Python UDF and has already been 
supported in the previous releases. However the stateful Python UDF, i.e. 
User-Defined Aggregate Function is not supported in PyFlink yet. We want to 
introduce the general Python user-defined aggregate function for PyFlink Table 
API.

Here is the design doc:

https://cwiki.apache.org/confluence/display/FLINK/FLIP-139%3A+General+Python+User-Defined+Aggregate+Function+Support+on+Table+API

Looking forward to your feedback!

Best,
Wei

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table

Re: [ANNOUNCE] Apache Flink 1.10.2 released

[Guowei Ma]

Hi,

Thanks a lot for being the release manager Zhu Zhu!
Thanks everyone contributed to this!

Best,
Guowei


On Wed, Aug 26, 2020 at 11:18 AM Yun Tang  wrote:

> Thanks for Zhu's work to manage this release and everyone who contributed
> to this!
>
> Best,
> Yun Tang
> 
> From: Yangze Guo 
> Sent: Tuesday, August 25, 2020 14:47
> To: Dian Fu 
> Cc: Zhu Zhu ; dev ; user <
> u...@flink.apache.org>; user-zh 
> Subject: Re: [ANNOUNCE] Apache Flink 1.10.2 released
>
> Thanks a lot for being the release manager Zhu Zhu!
> Congrats to all others who have contributed to the release!
>
> Best,
> Yangze Guo
>
> On Tue, Aug 25, 2020 at 2:42 PM Dian Fu  wrote:
> >
> > Thanks ZhuZhu for managing this release and everyone else who
> contributed to this release!
> >
> > Regards,
> > Dian
> >
> > 在 2020年8月25日，下午2:22，Till Rohrmann  写道：
> >
> > Great news. Thanks a lot for being our release manager Zhu Zhu and to
> all others who have contributed to the release!
> >
> > Cheers,
> > Till
> >
> > On Tue, Aug 25, 2020 at 5:37 AM Zhu Zhu  wrote:
> >>
> >> The Apache Flink community is very happy to announce the release of
> Apache Flink 1.10.2, which is the first bugfix release for the Apache Flink
> 1.10 series.
> >>
> >> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
> >>
> >> The release is available for download at:
> >> https://flink.apache.org/downloads.html
> >>
> >> Please check out the release blog post for an overview of the
> improvements for this bugfix release:
> >> https://flink.apache.org/news/2020/08/25/release-1.10.2.html
> >>
> >> The full release notes are available in Jira:
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12347791
> >>
> >> We would like to thank all contributors of the Apache Flink community
> who made this release possible!
> >>
> >> Thanks,
> >> Zhu
> >
> >
>

Re: [ANNOUNCE] Apache Flink 1.10.2 released

[Xingbo Huang]

Thanks Zhu for the great work and everyone who contributed to this release!

Best,
Xingbo

Guowei Ma  于2020年8月26日周三 下午12:43写道：

> Hi,
>
> Thanks a lot for being the release manager Zhu Zhu!
> Thanks everyone contributed to this!
>
> Best,
> Guowei
>
>
> On Wed, Aug 26, 2020 at 11:18 AM Yun Tang  wrote:
>
>> Thanks for Zhu's work to manage this release and everyone who contributed
>> to this!
>>
>> Best,
>> Yun Tang
>> 
>> From: Yangze Guo 
>> Sent: Tuesday, August 25, 2020 14:47
>> To: Dian Fu 
>> Cc: Zhu Zhu ; dev ; user <
>> u...@flink.apache.org>; user-zh 
>> Subject: Re: [ANNOUNCE] Apache Flink 1.10.2 released
>>
>> Thanks a lot for being the release manager Zhu Zhu!
>> Congrats to all others who have contributed to the release!
>>
>> Best,
>> Yangze Guo
>>
>> On Tue, Aug 25, 2020 at 2:42 PM Dian Fu  wrote:
>> >
>> > Thanks ZhuZhu for managing this release and everyone else who
>> contributed to this release!
>> >
>> > Regards,
>> > Dian
>> >
>> > 在 2020年8月25日，下午2:22，Till Rohrmann  写道：
>> >
>> > Great news. Thanks a lot for being our release manager Zhu Zhu and to
>> all others who have contributed to the release!
>> >
>> > Cheers,
>> > Till
>> >
>> > On Tue, Aug 25, 2020 at 5:37 AM Zhu Zhu  wrote:
>> >>
>> >> The Apache Flink community is very happy to announce the release of
>> Apache Flink 1.10.2, which is the first bugfix release for the Apache Flink
>> 1.10 series.
>> >>
>> >> Apache Flink® is an open-source stream processing framework for
>> distributed, high-performing, always-available, and accurate data streaming
>> applications.
>> >>
>> >> The release is available for download at:
>> >> https://flink.apache.org/downloads.html
>> >>
>> >> Please check out the release blog post for an overview of the
>> improvements for this bugfix release:
>> >> https://flink.apache.org/news/2020/08/25/release-1.10.2.html
>> >>
>> >> The full release notes are available in Jira:
>> >>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12347791
>> >>
>> >> We would like to thank all contributors of the Apache Flink community
>> who made this release possible!
>> >>
>> >> Thanks,
>> >> Zhu
>> >
>> >
>>
>