Re: SerializableHadoopConfiguration

[Till Rohrmann]

Hi Sivaprasanna,

we don't upload the source jars for the flink-shaded modules. However you
can build them yourself and install by cloning the flink-shaded repository
[1] and then call `mvn package -Dshade-sources`.

[1] https://github.com/apache/flink-shaded

Cheers,
Till

On Tue, Mar 3, 2020 at 6:29 PM Sivaprasanna 
wrote:

> BTW, can we leverage flink-shaded-hadoop-2? Reason why I ask, if any Flink
> module is going to use Hadoop in any way, it will most probably include
> flink-shaded-hadoop-2 as a dependency.
> However, flink-shaded modules don't have any source files. Is that a strict
> convention that the community follows?
>
> -
> Sivaprasanna
>
> On Tue, Mar 3, 2020 at 10:48 PM Sivaprasanna 
> wrote:
>
> > Hi Arvid,
> >
> > Thanks for the quick reply. Yes, it actually makes sense to avoid Hadoop
> > dependencies from getting into Flink's core modules but I also wonder if
> it
> > will be an overkill to add flink-hadoop-fs as a dependency just because
> we
> > want to use a utility class from that module.
> >
> > -
> > Sivaprasanna
> >
> > On Tue, Mar 3, 2020 at 4:17 PM Arvid Heise  wrote:
> >
> >> Hi Sivaprasanna,
> >>
> >> we actually want to remove Hadoop from all core modules, so we could not
> >> place it in some very common place like flink-core.
> >>
> >> But I think the module flink-hadoop-fs could be a fitting place.
> >>
> >> On Tue, Mar 3, 2020 at 11:25 AM Sivaprasanna  >
> >> wrote:
> >>
> >> > Hi
> >> >
> >> > The flink-sequence-file module has a class named
> >> > SerializableHadoopConfiguration[1] which is nothing but a wrapper
> class
> >> for
> >> > Hadoop Configuration. I believe this class can be moved to a common
> >> module
> >> > since this is not necessarily tightly coupled with sequence-file
> module,
> >> > and also because it can be used by many other modules, for ex.
> >> > flink-compress. Thoughts?
> >> >
> >> > -
> >> > Sivaprasanna
> >> >
> >>
> >
>

Re: [DISCUSS] FLIP-106: Support Python UDF in SQL Function DDL

[Dawid Wysakowicz]

Hi all,
I had a really quick look and from my perspective the proposal looks fine.
I share Jarks opinion that the instantiation could be done at a later
stage. I agree with Wei it requires some changes in the internal
implementation of the FunctionCatalog, to store temporary functions as
catalog functions instead of FunctionDefinitions, but we have that on our
agenda anyway. I would suggest investigating if we could do that as part of
this flip already. Nevertheless this in theory can be also done later.

Best,
Dawid

On Mon, 2 Mar 2020, 14:58 Jark Wu,  wrote:

> Thanks for the explanation, Wei!
>
> On Mon, 2 Mar 2020 at 20:59, Wei Zhong  wrote:
>
> > Hi Jark,
> >
> > Thanks for your suggestion.
> >
> > Actually, the timing of starting a Python process depends on the UDF
> type,
> > because the Python process is used to provide the necessary information
> to
> > instantiate the FunctionDefinition object of the Python UDF. For catalog
> > function, the FunctionDefinition will be instantiated when compiling the
> > job, which means the Python process is required during the compilation
> > instead of the registeration. For temporary system function and temporary
> > catalog function, the FunctionDefinition will be instantiated during the
> > UDF registeration, so the Python process need to be started at that time.
> >
> > But this FLIP will only support registering the temporary system function
> > and temporary catalog function in SQL DDL because registering Python UDF
> to
> > catalog is not supported yet. We plan to support the registeration of
> > Python catalog function (via Table API and SQL DDL) in a separate FLIP.
> > I'll add a non-goal section to the FLIP page to illustrate this.
> >
> > Best,
> > Wei
> >
> >
> > > 在 2020年3月2日，15:11，Jark Wu  写道：
> > >
> > > Hi Weizhong,
> > >
> > > Thanks for proposing this feature. In geneal, I'm +1 from the table's
> > view.
> > >
> > > I have one suggestion: I think the register python function into
> catalog
> > > doesn't need to startup python process (the "High Level Sequence
> Diagram"
> > > in your FLIP).
> > > Because only meta-information is persisted into catalog, we don't need
> to
> > > store "return type", "input types" into catalog.
> > > I guess the python process is required when compiling a SQL job.
> > >
> > > Best,
> > > Jark
> > >
> > >
> > >
> > > On Fri, 28 Feb 2020 at 19:04, Benchao Li  wrote:
> > >
> > >> Big +1 for this feature.
> > >>
> > >> We built our SQL platform on Java Table API, and most common UDF are
> > >> implemented in Java. However some python developers are not familiar
> > with
> > >> Java/Scala, and it's very inconvenient for these users to use UDF in
> > SQL.
> > >>
> > >> Wei Zhong  于2020年2月28日周五 下午6:58写道：
> > >>
> > >>> Thank for your reply Dan!
> > >>>
> > >>> By the way, this FLIP is closely related to the SQL API.  @Jark Wu <
> > >>> imj...@gmail.com> @Timo  could you please take a
> > >>> look?
> > >>>
> > >>> Thanks,
> > >>> Wei
> > >>>
> >  在 2020年2月25日，16:25，zoudan  写道：
> > 
> >  +1 for supporting Python UDF in Java/Scala Table API.
> >  This is a great feature and would be helpful for python users!
> > 
> >  Best,
> >  Dan Zou
> > 
> > 
> > >>>
> > >>>
> > >>
> > >> --
> > >>
> > >> Benchao Li
> > >> School of Electronics Engineering and Computer Science, Peking
> > University
> > >> Tel:+86-15650713730
> > >> Email: libenc...@gmail.com; libenc...@pku.edu.cn
> > >>
> > >>
> >
> >
>

[jira] [Created] (FLINK-16417) ConnectedComponents iterations with high parallelism end-to-end test fails with OutOfMemoryError: Direct buffer memory

[Robert Metzger (Jira)]

Robert Metzger created FLINK-16417:
--

 Summary: ConnectedComponents iterations with high parallelism 
end-to-end test fails with OutOfMemoryError: Direct buffer memory
 Key: FLINK-16417
 URL: https://issues.apache.org/jira/browse/FLINK-16417
 Project: Flink
  Issue Type: Bug
  Components: API / DataSet, Tests
Reporter: Robert Metzger


Logs: 
https://dev.azure.com/georgeryan1322/Flink/_build/results?buildId=74&view=logs&j=1f3ed471-1849-5d3c-a34c-19792af4ad16&t=ce095137-3e3b-5f73-4b79-c42d3d5f8283

{code}
2020-03-04T08:03:46.0786078Z 2020-03-04 08:03:42,628 INFO  
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] - starting 
iteration [1]:  Reduce (MIN(1), at 
main(HighParallelismIterationsTestProgram.java:61) (12/25)
2020-03-04T08:03:46.0787503Z 2020-03-04 08:03:42,875 ERROR 
org.apache.flink.runtime.io.network.netty.PartitionRequestQueue [] - 
Encountered error while consuming partitions
2020-03-04T08:03:46.0788060Z java.lang.OutOfMemoryError: Direct buffer memory
2020-03-04T08:03:46.0788460Zat java.nio.Bits.reserveMemory(Bits.java:175) 
~[?:?]
2020-03-04T08:03:46.0788904Zat 
java.nio.DirectByteBuffer.(DirectByteBuffer.java:118) ~[?:?]
2020-03-04T08:03:46.0789537Zat 
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
2020-03-04T08:03:46.0790381Zat 
org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:772)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0791491Zat 
org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:748)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0792483Zat 
org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0793416Zat 
org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena.allocate(PoolArena.java:215)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0794359Zat 
org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena.allocate(PoolArena.java:147)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0795385Zat 
org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:342)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0796471Zat 
org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0797575Zat 
org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0798718Zat 
org.apache.flink.shaded.netty4.io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0799951Zat 
org.apache.flink.shaded.netty4.io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0801172Zat 
org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
 ~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0802572Zat 
org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:779)
 [flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0803719Zat 
org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424)
 [flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0804763Zat 
org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
 [flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0806007Zat 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
 [flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0807050Zat 
org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
 [flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0807612Zat java.lang.Thread.run(Thread.java:834) [?:?]
2020-03-04T08:03:46.0808499Z 2020-03-04 08:03:43,572 ERROR 
org.apache.flink.runtime.operators.BatchTask [] - Error in task 
code:  Reduce (MIN(1), at main(HighParallelismIterationsTestProgram.java:61) 
(5/25)
2020-03-04T08:03:46.0810179Z java.lang.Exception: The

Re: [DISCUSS] FLIP-107: Reading table columns from different parts of source records

[Kurt Young]

Hi Dawid,

I have a couple of questions around key fields, actually I also have some
other questions but want to be focused on key fields first.

1. I don't fully understand the usage of "key.fields". Is this option only
valid during write operation? Because for
reading, I can't imagine how such options can be applied. I would expect
that there might be a SYSTEM_METADATA("key")
to read and assign the key to a normal field?

2. If "key.fields" is only valid in write operation, I want to propose we
can simplify the options to not introducing key.format.type and
other related options. I think a single "key.field" (not fields) would be
enough, users can use UDF to calculate whatever key they
want before sink.

3. Also I don't want to introduce "value.format.type" and
"value.format.xxx" with the "value" prefix. Not every connector has a
concept
of key and values. The old parameter "format.type" already good enough to
use.

Best,
Kurt


On Tue, Mar 3, 2020 at 10:40 PM Jark Wu  wrote:

> Thanks Dawid,
>
> I have two more questions.
>
> > SupportsMetadata
> Introducing SupportsMetadata sounds good to me. But I have some questions
> regarding to this interface.
> 1) How do the source know what the expected return type of each metadata?
> 2) Where to put the metadata fields? Append to the existing physical
> fields?
> If yes, I would suggest to change the signature to `TableSource
> appendMetadataFields(String[] metadataNames, DataType[] metadataTypes)`
>
> > SYSTEM_METADATA("partition")
> Can SYSTEM_METADATA() function be used nested in a computed column
> expression? If yes, how to specify the return type of SYSTEM_METADATA?
>
> Best,
> Jark
>
> On Tue, 3 Mar 2020 at 17:06, Dawid Wysakowicz 
> wrote:
>
> > Hi,
> >
> > 1. I thought a bit more on how the source would emit the columns and I
> > now see its not exactly the same as regular columns. I see a need to
> > elaborate a bit more on that in the FLIP as you asked, Jark.
> >
> > I do agree mostly with Danny on how we should do that. One additional
> > things I would introduce is an
> >
> > interface SupportsMetadata {
> >
> >boolean supportsMetadata(Set metadataFields);
> >
> >TableSource generateMetadataFields(Set metadataFields);
> >
> > }
> >
> > This way the source would have to declare/emit only the requested
> > metadata fields. In order not to clash with user defined fields. When
> > emitting the metadata field I would prepend the column name with
> > __system_{property_name}. Therefore when requested
> > SYSTEM_METADATA("partition") the source would append a field
> > __system_partition to the schema. This would be never visible to the
> > user as it would be used only for the subsequent computed columns. If
> > that makes sense to you, I will update the FLIP with this description.
> >
> > 2. CAST vs explicit type in computed columns
> >
> > Here I agree with Danny. It is also the current state of the proposal.
> >
> > 3. Partitioning on computed column vs function
> >
> > Here I also agree with Danny. I also think those are orthogonal. I would
> > leave out the STORED computed columns out of the discussion. I don't see
> > how do they relate to the partitioning. I already put both of those
> > cases in the document. We can either partition on a computed column or
> > use a udf in a partioned by clause. I am fine with leaving out the
> > partitioning by udf in the first version if you still have some concerns.
> >
> > As for your question Danny. It depends which partitioning strategy you
> use.
> >
> > For the HASH partitioning strategy I thought it would work as you
> > explained. It would be N = MOD(expr, num). I am not sure though if we
> > should introduce the PARTITIONS clause. Usually Flink does not own the
> > data and the partitions are already an intrinsic property of the
> > underlying source e.g. for kafka we do not create topics, but we just
> > describe pre-existing pre-partitioned topic.
> >
> > 4. timestamp vs timestamp.field vs connector.field vs ...
> >
> > I am fine with changing it to timestamp.field to be consistent with
> > other value.fields and key.fields. Actually that was also my initial
> > proposal in a first draft I prepared. I changed it afterwards to shorten
> > the key.
> >
> > Best,
> >
> > Dawid
> >
> > On 03/03/2020 09:00, Danny Chan wrote:
> > > Thanks Dawid for bringing up this discussion, I think it is a useful
> > feature ~
> > >
> > > About how the metadata outputs from source
> > >
> > > I think it is completely orthogonal, computed column push down is
> > another topic, this should not be a blocker but a promotion, if we do not
> > have any filters on the computed column, there is no need to do any
> > pushings; the source node just emit the complete record with full
> metadata
> > with the declared physical schema, then when generating the virtual
> > columns, we would extract the metadata info and output as full
> columns(with
> > full schema).
> > >
> > > About the type of metadata column
> > >
> > > Pe

Re: [DISCUSS] FLIP-107: Reading table columns from different parts of source records

[Kurt Young]

Sorry, forgot one question.

4. Can we make the value.fields-include more orthogonal? Like one can
specify it as "EXCEPT_KEY, EXCEPT_TIMESTAMP".
With current  EXCEPT_KEY and EXCEPT_KEY_TIMESTAMP, users can not config to
just ignore timestamp but keep key.

Best,
Kurt


On Wed, Mar 4, 2020 at 4:42 PM Kurt Young  wrote:

> Hi Dawid,
>
> I have a couple of questions around key fields, actually I also have some
> other questions but want to be focused on key fields first.
>
> 1. I don't fully understand the usage of "key.fields". Is this option only
> valid during write operation? Because for
> reading, I can't imagine how such options can be applied. I would expect
> that there might be a SYSTEM_METADATA("key")
> to read and assign the key to a normal field?
>
> 2. If "key.fields" is only valid in write operation, I want to propose we
> can simplify the options to not introducing key.format.type and
> other related options. I think a single "key.field" (not fields) would be
> enough, users can use UDF to calculate whatever key they
> want before sink.
>
> 3. Also I don't want to introduce "value.format.type" and
> "value.format.xxx" with the "value" prefix. Not every connector has a
> concept
> of key and values. The old parameter "format.type" already good enough to
> use.
>
> Best,
> Kurt
>
>
> On Tue, Mar 3, 2020 at 10:40 PM Jark Wu  wrote:
>
>> Thanks Dawid,
>>
>> I have two more questions.
>>
>> > SupportsMetadata
>> Introducing SupportsMetadata sounds good to me. But I have some questions
>> regarding to this interface.
>> 1) How do the source know what the expected return type of each metadata?
>> 2) Where to put the metadata fields? Append to the existing physical
>> fields?
>> If yes, I would suggest to change the signature to `TableSource
>> appendMetadataFields(String[] metadataNames, DataType[] metadataTypes)`
>>
>> > SYSTEM_METADATA("partition")
>> Can SYSTEM_METADATA() function be used nested in a computed column
>> expression? If yes, how to specify the return type of SYSTEM_METADATA?
>>
>> Best,
>> Jark
>>
>> On Tue, 3 Mar 2020 at 17:06, Dawid Wysakowicz 
>> wrote:
>>
>> > Hi,
>> >
>> > 1. I thought a bit more on how the source would emit the columns and I
>> > now see its not exactly the same as regular columns. I see a need to
>> > elaborate a bit more on that in the FLIP as you asked, Jark.
>> >
>> > I do agree mostly with Danny on how we should do that. One additional
>> > things I would introduce is an
>> >
>> > interface SupportsMetadata {
>> >
>> >boolean supportsMetadata(Set metadataFields);
>> >
>> >TableSource generateMetadataFields(Set metadataFields);
>> >
>> > }
>> >
>> > This way the source would have to declare/emit only the requested
>> > metadata fields. In order not to clash with user defined fields. When
>> > emitting the metadata field I would prepend the column name with
>> > __system_{property_name}. Therefore when requested
>> > SYSTEM_METADATA("partition") the source would append a field
>> > __system_partition to the schema. This would be never visible to the
>> > user as it would be used only for the subsequent computed columns. If
>> > that makes sense to you, I will update the FLIP with this description.
>> >
>> > 2. CAST vs explicit type in computed columns
>> >
>> > Here I agree with Danny. It is also the current state of the proposal.
>> >
>> > 3. Partitioning on computed column vs function
>> >
>> > Here I also agree with Danny. I also think those are orthogonal. I would
>> > leave out the STORED computed columns out of the discussion. I don't see
>> > how do they relate to the partitioning. I already put both of those
>> > cases in the document. We can either partition on a computed column or
>> > use a udf in a partioned by clause. I am fine with leaving out the
>> > partitioning by udf in the first version if you still have some
>> concerns.
>> >
>> > As for your question Danny. It depends which partitioning strategy you
>> use.
>> >
>> > For the HASH partitioning strategy I thought it would work as you
>> > explained. It would be N = MOD(expr, num). I am not sure though if we
>> > should introduce the PARTITIONS clause. Usually Flink does not own the
>> > data and the partitions are already an intrinsic property of the
>> > underlying source e.g. for kafka we do not create topics, but we just
>> > describe pre-existing pre-partitioned topic.
>> >
>> > 4. timestamp vs timestamp.field vs connector.field vs ...
>> >
>> > I am fine with changing it to timestamp.field to be consistent with
>> > other value.fields and key.fields. Actually that was also my initial
>> > proposal in a first draft I prepared. I changed it afterwards to shorten
>> > the key.
>> >
>> > Best,
>> >
>> > Dawid
>> >
>> > On 03/03/2020 09:00, Danny Chan wrote:
>> > > Thanks Dawid for bringing up this discussion, I think it is a useful
>> > feature ~
>> > >
>> > > About how the metadata outputs from source
>> > >
>> > > I think it is completely orthogonal, computed column push down is

Introduce flink-connector-hive-xx modules

[Jingsong Lee]

Hi all,

I'd like to propose introduce flink-connector-hive-xx modules.

We have documented the dependencies detailed information[2]. But still has
some inconvenient:
- Too many versions, users need to pick one version from 8 versions.
- Too many versions, It's not friendly to our developers either, because
there's a problem/exception, we need to look at eight different versions of
hive client code, which are often various.
- Too many jars, for example, users need to download 4+ jars for Hive 1.x
from various places.

We have discussed in [1] and [2], but unfortunately, we can not achieve an
agreement.

For improving this, I'd like to introduce few flink-connector-hive-xx
modules in flink-connectors, module contains all the dependencies related
to hive. And only support lower hive metastore versions:
- "flink-connector-hive-1.2" to support hive 1.0.0 - 1.2.2
- "flink-connector-hive-2.0" to support hive 2.0.0 - 2.0.1
- "flink-connector-hive-2.2" to support hive 2.1.0 - 2.2.0
- "flink-connector-hive-2.3" to support hive 2.3.0 - 2.3.6
- "flink-connector-hive-3.1" to support hive 3.0.0 - 3.1.2

Users can choose one and download to flink/lib. It includes all hive things.

I try to use a single module to deploy multiple versions, but I can not
find a suitable way, because different modules require different versions
and different dependencies.

What do you think?

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-have-separate-Flink-distributions-with-built-in-Hive-dependencies-td35918.html
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-109-Improve-Hive-dependencies-out-of-box-experience-td38290.html

Best,
Jingsong Lee

[DISCUSS] Introduce flink-connector-hive-xx modules

[Jingsong Lee]

Hi all,

I'd like to propose introduce flink-connector-hive-xx modules.

We have documented the dependencies detailed information[2]. But still has
some inconvenient:
- Too many versions, users need to pick one version from 8 versions.
- Too many versions, It's not friendly to our developers either, because
there's a problem/exception, we need to look at eight different versions of
hive client code, which are often various.
- Too many jars, for example, users need to download 4+ jars for Hive 1.x
from various places.

We have discussed in [1] and [2], but unfortunately, we can not achieve an
agreement.

For improving this, I'd like to introduce few flink-connector-hive-xx
modules in flink-connectors, module contains all the dependencies related
to hive. And only support lower hive metastore versions:
- "flink-connector-hive-1.2" to support hive 1.0.0 - 1.2.2
- "flink-connector-hive-2.0" to support hive 2.0.0 - 2.0.1
- "flink-connector-hive-2.2" to support hive 2.1.0 - 2.2.0
- "flink-connector-hive-2.3" to support hive 2.3.0 - 2.3.6
- "flink-connector-hive-3.1" to support hive 3.0.0 - 3.1.2

Users can choose one and download to flink/lib. It includes all hive things.

I try to use a single module to deploy multiple versions, but I can not
find a suitable way, because different modules require different versions
and different dependencies.

What do you think?

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-have-separate-Flink-distributions-with-built-in-Hive-dependencies-td35918.html
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-109-Improve-Hive-dependencies-out-of-box-experience-td38290.html

Best,
Jingsong Lee

Re: Introduce flink-connector-hive-xx modules

[Jingsong Lee]

Sorry to forget the title [DISCUSS].
Close this thread.

Best,
Jingsong Lee


On Wed, Mar 4, 2020 at 4:57 PM Jingsong Lee  wrote:

> Hi all,
>
> I'd like to propose introduce flink-connector-hive-xx modules.
>
> We have documented the dependencies detailed information[2]. But still has
> some inconvenient:
> - Too many versions, users need to pick one version from 8 versions.
> - Too many versions, It's not friendly to our developers either, because
> there's a problem/exception, we need to look at eight different versions of
> hive client code, which are often various.
> - Too many jars, for example, users need to download 4+ jars for Hive 1.x
> from various places.
>
> We have discussed in [1] and [2], but unfortunately, we can not achieve an
> agreement.
>
> For improving this, I'd like to introduce few flink-connector-hive-xx
> modules in flink-connectors, module contains all the dependencies related
> to hive. And only support lower hive metastore versions:
> - "flink-connector-hive-1.2" to support hive 1.0.0 - 1.2.2
> - "flink-connector-hive-2.0" to support hive 2.0.0 - 2.0.1
> - "flink-connector-hive-2.2" to support hive 2.1.0 - 2.2.0
> - "flink-connector-hive-2.3" to support hive 2.3.0 - 2.3.6
> - "flink-connector-hive-3.1" to support hive 3.0.0 - 3.1.2
>
> Users can choose one and download to flink/lib. It includes all hive
> things.
>
> I try to use a single module to deploy multiple versions, but I can not
> find a suitable way, because different modules require different versions
> and different dependencies.
>
> What do you think?
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-have-separate-Flink-distributions-with-built-in-Hive-dependencies-td35918.html
> [2]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-109-Improve-Hive-dependencies-out-of-box-experience-td38290.html
>
> Best,
> Jingsong Lee
>


-- 
Best, Jingsong Lee

[jira] [Created] (FLINK-16418) Hide hive version to avoid user confuse

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-16418:


 Summary: Hide hive version to avoid user confuse
 Key: FLINK-16418
 URL: https://issues.apache.org/jira/browse/FLINK-16418
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Reporter: Jingsong Lee
Assignee: Jingsong Lee
 Fix For: 1.11.0


Version in Yaml/HiveCatalog needs to be consistent with the dependencies 
version. There are three places: version in metastore, version in dependencies, 
version in Yaml/HiveCatalog, users are easy to make mistakes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16419) Avoid to recommit transactions which are known committed successfully to Kafka upon recovery

[Jun Qin (Jira)]

Jun Qin created FLINK-16419:
---

 Summary: Avoid to recommit transactions which are known committed 
successfully to Kafka upon recovery
 Key: FLINK-16419
 URL: https://issues.apache.org/jira/browse/FLINK-16419
 Project: Flink
  Issue Type: Improvement
Reporter: Jun Qin


When recovering from a snapshot (checkpoint/savepoint), FlinkKafkaProducer 
tries to recommit all pre-committed transactions which are in the snapshot, 
even if those transactions were successfully committed before (i.e., the call 
to {{kafkaProducer.commitTransaction()}} via {{notifyCheckpointComplete()}} 
returns OK). This may lead to recovery failures when recovering from a very old 
snapshot because the transactional IDs in that snapshot have been expired and 
removed from Kafka.  For example the following scenario:
 # Start a Flink job with FlinkKafkaProducer sink with exactly-once
 # Suspend the Flink job with a savepoint A
 # Wait for time longer than {{transactional.id.expiration.ms}} + 
{{transaction.remove.expired.transaction.cleanup.interval.ms}}
 # Recover the job with savepoint A.
 # The recovery will fail with the following error:

{noformat}
2020-02-26 14:33:25,817 INFO  
org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer 
 - Attempting to resume transaction Source: Custom Source -> Sink: 
Unnamed-7df19f87deec5680128845fd9a6ca18d-1 with producerId 2001 and epoch 
1202020-02-26 14:33:25,914 INFO  org.apache.kafka.clients.Metadata              
              - Cluster ID: RN0aqiOwTUmF5CnHv_IPxA
2020-02-26 14:33:26,017 INFO  org.apache.kafka.clients.producer.KafkaProducer   
           - [Producer clientId=producer-1, transactionalId=Source: Custom 
Source -> Sink: Unnamed-7df19f87deec5680128845fd9a6ca18d-1] Closing the Kafka 
producer with timeoutMillis = 92233720
36854775807 ms.
2020-02-26 14:33:26,019 INFO  org.apache.flink.runtime.taskmanager.Task         
           - Source: Custom Source -> Sink: Unnamed (1/1) 
(a77e457941f09cd0ebbd7b982edc0f02) switched from RUNNING to FAILED.
org.apache.kafka.common.KafkaException: Unhandled error in EndTxnResponse: The 
producer attempted to use a producer id which is not currently assigned to its 
transactional id.
        at 
org.apache.kafka.clients.producer.internals.TransactionManager$EndTxnHandler.handleResponse(TransactionManager.java:1191)
        at 
org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:909)
        at 
org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
        at 
org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:557)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549)
        at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:288)
        at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
        at java.lang.Thread.run(Thread.java:748)
{noformat}
After discussed with [~becket_qin], [~pnowojski] and [~aljoscha], a possible 
way is to let JobManager, after successfully notifies all operators the 
completion of a snapshot (via {{notifyCheckpoingComplete}}), record the 
success, e.g., write the successful transactional IDs somewhere in the 
snapshot. Then those transactions need not recommit upon recovery.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] FLIP-111: Docker image unification

[Andrey Zagrebin]

Hi All,

If you have ever touched the docker topic in Flink, you
probably noticed that we have multiple places in docs and repos which
address its various concerns.

We have prepared a FLIP [1] to simplify the perception of docker topic in
Flink by users. It mostly advocates for an approach of extending official
Flink image from the docker hub. For convenience, it can come with a set of
bash utilities and documented examples of their usage. The utilities allow
to:

   - run the docker image in various modes (single job, session master,
   task manager etc)
   - customise the extending Dockerfile
   - and its entry point

Eventually, the FLIP suggests to remove all other user facing Dockerfiles
and building scripts from Flink repo, move all docker docs to
apache/flink-docker and adjust existing docker use cases to refer to this
new approach (mostly Kubernetes now).

The first contributed version of Flink docker integration also contained
example and docs for the integration with Bluemix in IBM cloud. We also
suggest to maintain it outside of Flink repository (cc Markus Müller).

Thanks,
Andrey

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification

[Maven] Many artifactId contain ${scala.binary.version}

[Niels Basjes]

Hi,

When building Flink I see a LOT of messages from maven that are similar to
this:

[WARNING]
[WARNING] Some problems were encountered while building the effective model
for org.apache.flink:flink-runtime_2.11:jar:1.11-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @
org.apache.flink:flink-runtime_${scala.binary.version}:1.11-SNAPSHOT,
/home/nbasjes/workspace/Apache/flink/flink-runtime/pom.xml, line 32, column
14
[WARNING]
[WARNING] It is highly recommended to fix these problems because they
threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support
building such malformed projects.
[WARNING]

Apparently using the property ${scala.binary.version} in the artifactId is
not likes by Maven.
Right now it just produces a lot of noise during the build.
How big of a problem is this really?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [Maven] Many artifactId contain ${scala.binary.version}

[Chesnay Schepler]

It's not an actual problem for us since we make sure that only leaf 
modules have the suffix.


It can result in problems if parent modules have the suffix, since 
children _may_ no longer be able to resolve the path to the parent since 
the property value is usually defined in the parent.
On a related note, children overriding the scala.version.property would 
probably also result in a few weird issues.


In other words, it makes sense for maven to log these warnings, but they 
are not relevant for us.


On 04/03/2020 10:38, Niels Basjes wrote:

Hi,

When building Flink I see a LOT of messages from maven that are similar to
this:

[WARNING]
[WARNING] Some problems were encountered while building the effective model
for org.apache.flink:flink-runtime_2.11:jar:1.11-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @
org.apache.flink:flink-runtime_${scala.binary.version}:1.11-SNAPSHOT,
/home/nbasjes/workspace/Apache/flink/flink-runtime/pom.xml, line 32, column
14
[WARNING]
[WARNING] It is highly recommended to fix these problems because they
threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support
building such malformed projects.
[WARNING]

Apparently using the property ${scala.binary.version} in the artifactId is
not likes by Maven.
Right now it just produces a lot of noise during the build.
How big of a problem is this really?

Re: Creating TemporalTable based on Catalog table in SQL Client

[Gyula Fóra]

You are right but still if the default catalog is something else and that's
the one containing the table then it still wont work currently.

Gyula

On Wed, Mar 4, 2020 at 5:08 AM Bowen Li  wrote:

> Hi Gyula,
>
> What line 622 (the link you shared) does is not registering catalogs, but
> setting an already registered catalog as the current one. As you can see
> from the method and its comment, catalogs are loaded first before any
> tables in yaml are registered, so you should be able to achieve what you
> described.
>
> Bowen
>
> On Tue, Mar 3, 2020 at 5:16 AM Gyula Fóra  wrote:
>
> > Hi all!
> >
> > I was testing the TemporalTable functionality in the SQL client while
> using
> > the Hive Catalog and I ran into the following problem.
> >
> > I have a table created in the Hive catalog and I want to create a
> temporal
> > table over it.
> >
> > As we cannot create temporal tables in SQL directly I have to define it
> in
> > the environment yaml file. Unfortunately it seems to be impossible to
> > reference a table only present in the catalog (not in the yaml) as
> catalogs
> > are loaded only after creating the temporal table (see
> >
> >
> https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/ExecutionContext.java#L622
> > )
> >
> > I am wondering if it would make sense to set the catalogs before all else
> > or if that would cause some other problems.
> >
> > What do you think?
> > Gyula
> >
>

Re: Creating TemporalTable based on Catalog table in SQL Client

[Gyula Fóra]

I guess it will only work now if you specify the catalog name too when
referencing the table.


On Wed, Mar 4, 2020 at 11:15 AM Gyula Fóra  wrote:

> You are right but still if the default catalog is something else and
> that's the one containing the table then it still wont work currently.
>
> Gyula
>
> On Wed, Mar 4, 2020 at 5:08 AM Bowen Li  wrote:
>
>> Hi Gyula,
>>
>> What line 622 (the link you shared) does is not registering catalogs, but
>> setting an already registered catalog as the current one. As you can see
>> from the method and its comment, catalogs are loaded first before any
>> tables in yaml are registered, so you should be able to achieve what you
>> described.
>>
>> Bowen
>>
>> On Tue, Mar 3, 2020 at 5:16 AM Gyula Fóra  wrote:
>>
>> > Hi all!
>> >
>> > I was testing the TemporalTable functionality in the SQL client while
>> using
>> > the Hive Catalog and I ran into the following problem.
>> >
>> > I have a table created in the Hive catalog and I want to create a
>> temporal
>> > table over it.
>> >
>> > As we cannot create temporal tables in SQL directly I have to define it
>> in
>> > the environment yaml file. Unfortunately it seems to be impossible to
>> > reference a table only present in the catalog (not in the yaml) as
>> catalogs
>> > are loaded only after creating the temporal table (see
>> >
>> >
>> https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/ExecutionContext.java#L622
>> > )
>> >
>> > I am wondering if it would make sense to set the catalogs before all
>> else
>> > or if that would cause some other problems.
>> >
>> > What do you think?
>> > Gyula
>> >
>>
>

[jira] [Created] (FLINK-16420) Support CREATE TABLE with schema inference

[Jark Wu (Jira)]

Jark Wu created FLINK-16420:
---

 Summary: Support CREATE TABLE with schema inference
 Key: FLINK-16420
 URL: https://issues.apache.org/jira/browse/FLINK-16420
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / API
Reporter: Jark Wu


Support CREATE TABLE statement without specifying any columns, the schema can 
be inferenced from format or a schema registry. 

This an umbrella issue for the feature. It can be used to collect initial ideas 
and use cases until a FLIP is proposed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16421) Changing default catalog to hive without changing default database fails

[Gyula Fora (Jira)]

Gyula Fora created FLINK-16421:
--

 Summary: Changing default catalog to hive without changing default 
database fails
 Key: FLINK-16421
 URL: https://issues.apache.org/jira/browse/FLINK-16421
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive, Table SQL / Client
Reporter: Gyula Fora


The default database in Hive is caled "default" not "default_database". This 
causes an error when starting the SQL CLI with hive set as default catalog:


{code:java}
Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: A 
database with name [default_database] does not exist in the catalog: 
[hive].Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: A 
database with name [default_database] does not exist in the catalog: [hive]. at 
org.apache.flink.table.catalog.CatalogManager.setCurrentDatabase(CatalogManager.java:174)
 at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.useDatabase(TableEnvironmentImpl.java:631)
 at java.util.Optional.ifPresent(Optional.java:159) at 
org.apache.flink.table.client.gateway.local.ExecutionContext.initializeCatalogs(ExecutionContext.java:561)
 at 
org.apache.flink.table.client.gateway.local.ExecutionContext.initializeTableEnvironment(ExecutionContext.java:494)
 at 
org.apache.flink.table.client.gateway.local.ExecutionContext.(ExecutionContext.java:159)
 at 
org.apache.flink.table.client.gateway.local.ExecutionContext.(ExecutionContext.java:118)
 at 
org.apache.flink.table.client.gateway.local.ExecutionContext$Builder.build(ExecutionContext.java:744)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16422) Cannot use [catalog].[db].table with Hive catalog

[Gyula Fora (Jira)]

Gyula Fora created FLINK-16422:
--

 Summary: Cannot use [catalog].[db].table with Hive catalog
 Key: FLINK-16422
 URL: https://issues.apache.org/jira/browse/FLINK-16422
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive, Table SQL / Client
Affects Versions: 1.10.0
Reporter: Gyula Fora


When trying to select a table from the Hive , the SQL CLI automcompletes to the 
full table name:

select * from hive.default.itemtransactions ;

but then we get the following error:

[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.api.SqlParserException: SQL parse failed. Encountered ". 
default" at line 1, column 19.
Was expecting one of:...



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-100: Add Attempt Information

[Yadong Xie]

Hi Gary Kurt, and Jark

I am canceling the vote and restart it since the POC has some changes from
the initial one.

All the changes are following the proposal in this mail thread.

please vote again in the new thread, thanks

Jark Wu  于2020年3月4日周三 下午12:13写道：

> +1 from my side.
>
> Best,
> Jark
>
> On Wed, 4 Mar 2020 at 11:39, Kurt Young  wrote:
>
> > LGTM now, +1 from my side.
> >
> > Best,
> > Kurt
> >
> >
> > On Wed, Mar 4, 2020 at 12:27 AM Gary Yao  wrote:
> >
> >> Hi Yadong,
> >>
> >> Thank you for updating the wiki page.
> >>
> >> Only one minor suggestion – I would change:
> >>
> >> > If show-history is true return the information of attempt.
> >>
> >> to
> >>
> >> > If show-history is true, information for all attempts including
> >> previous ones will be returned
> >>
> >> That being said, FLIP-100 looks good to me. From my side there is not
> >> anything
> >> else to discuss.
> >>
> >> @Kurt and @Jark: Can you look into the improvements that have been made
> >> since
> >> the last time you looked at the PoC? If you are happy, we can restart
> the
> >> voting.
> >>
> >> Best,
> >> Gary
> >>
> >> On Tue, Mar 3, 2020 at 2:34 PM Yadong Xie  wrote:
> >>
> >>> Hi all
> >>>
> >>> The rest API part has been updated with Gary and Till's suggestions
> >>> here is the link:
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
> >>>
> >>> Yadong Xie  于2020年3月3日周二 下午9:14写道：
> >>>
> >>> > Hi Chesnay
> >>> >
> >>> > most discussions in this vote are about the more feature/demo request
> >>> in
> >>> > POC or discussion about response format, the main proposal the web UI
> >>> part
> >>> > which is not changed
> >>> >
> >>> > and the discussion about the response is converging, the response
> >>> format
> >>> > discussion could happen either here or at the code review stage,
> which
> >>> > would be a minor change from my point of view.
> >>> >
> >>> > Chesnay Schepler  于2020年3月3日周二 下午8:20写道：
> >>> >
> >>> >> I suggest to cancel this vote.
> >>> >> Several discussion items have been brought up during the vote, some
> of
> >>> >> which are still unresolved, others which resulted in changes to the
> >>> >> proposal.
> >>> >>
> >>> >> My conclusion is that this proposal needs more discussions.
> >>> >>
> >>> >>
> >>> >> On 20/02/2020 10:46, Yadong Xie wrote:
> >>> >> > Hi all
> >>> >> >
> >>> >> > I want to start the vote for FLIP-100, which proposes to add
> attempt
> >>> >> > information inside subtask and timeline in web UI.
> >>> >> >
> >>> >> > To help everyone better understand the proposal, we spent some
> >>> efforts
> >>> >> on
> >>> >> > making an online POC
> >>> >> >
> >>> >> > Timeline Attempt (click the vertex timeline to see the
> differences):
> >>> >> > previous web:
> >>> >> >
> >>> >>
> >>>
> http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/timeline
> >>> >> > POC web:
> >>> >> >
> >>> >>
> >>>
> http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/timeline
> >>> >> >
> >>> >> > Subtask Attempt (click the vertex and switch to subtask tab to see
> >>> the
> >>> >> > differences):
> >>> >> > previous web:
> >>> >> >
> >>> >>
> >>>
> http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/overview
> >>> >> > POC web:
> >>> >> >
> >>> >>
> >>>
> http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/overview
> >>> >> >
> >>> >> >
> >>> >> > The vote will last for at least 72 hours, following the consensus
> >>> voting
> >>> >> > process.
> >>> >> >
> >>> >> > FLIP wiki:
> >>> >> >
> >>> >>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
> >>> >> >
> >>> >> > Discussion thread:
> >>> >> >
> >>> >>
> >>>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> >>> >> >
> >>> >> > Thanks,
> >>> >> >
> >>> >> > Yadong
> >>> >> >
> >>> >>
> >>> >>
> >>>
> >>
>

[VOTE] FLIP-100[NEW]: Add Attempt Information

[Yadong Xie]

Hi all

I want to start the vote for FLIP-100, which proposes to add attempt
information inside subtask and timeline in web UI.

To help everyone better understand the proposal, we spent some efforts on
making an online POC

Timeline Attempt (click the vertex timeline to see the differences):
previous web:
http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/timeline
POC web:
http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/timeline

Subtask Attempt (click the vertex and switch to subtask tab to see the
differences):
previous web:
http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/overview
POC web:
http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/overview


The vote will last for at least 72 hours, following the consensus voting
process.

FLIP wiki:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information

Discussion thread:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html

Thanks,

Yadong

Re: [VOTE] FLIP-100[NEW]: Add Attempt Information

[Gary Yao]

+1 (binding)

Best,
Gary

On Wed, Mar 4, 2020 at 1:18 PM Yadong Xie  wrote:

> Hi all
>
> I want to start the vote for FLIP-100, which proposes to add attempt
> information inside subtask and timeline in web UI.
>
> To help everyone better understand the proposal, we spent some efforts on
> making an online POC
>
> Timeline Attempt (click the vertex timeline to see the differences):
> previous web:
> http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/timeline
> POC web:
>
> http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/timeline
>
> Subtask Attempt (click the vertex and switch to subtask tab to see the
> differences):
> previous web:
> http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/overview
> POC web:
>
> http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/overview
>
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> FLIP wiki:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
>
> Discussion thread:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
>
> Thanks,
>
> Yadong
>

Re: [VOTE] FLIP-100[NEW]: Add Attempt Information

[Kurt Young]

+1

Best,
Kurt


On Wed, Mar 4, 2020 at 8:19 PM Gary Yao  wrote:

> +1 (binding)
>
> Best,
> Gary
>
> On Wed, Mar 4, 2020 at 1:18 PM Yadong Xie  wrote:
>
> > Hi all
> >
> > I want to start the vote for FLIP-100, which proposes to add attempt
> > information inside subtask and timeline in web UI.
> >
> > To help everyone better understand the proposal, we spent some efforts on
> > making an online POC
> >
> > Timeline Attempt (click the vertex timeline to see the differences):
> > previous web:
> >
> http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/timeline
> > POC web:
> >
> >
> http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/timeline
> >
> > Subtask Attempt (click the vertex and switch to subtask tab to see the
> > differences):
> > previous web:
> >
> http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/overview
> > POC web:
> >
> >
> http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/overview
> >
> >
> > The vote will last for at least 72 hours, following the consensus voting
> > process.
> >
> > FLIP wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
> >
> > Discussion thread:
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> >
> > Thanks,
> >
> > Yadong
> >
>

Re: [VOTE] FLIP-100[NEW]: Add Attempt Information

[Jark Wu]

+1

On Wed, 4 Mar 2020 at 20:29, Kurt Young  wrote:

> +1
>
> Best,
> Kurt
>
>
> On Wed, Mar 4, 2020 at 8:19 PM Gary Yao  wrote:
>
> > +1 (binding)
> >
> > Best,
> > Gary
> >
> > On Wed, Mar 4, 2020 at 1:18 PM Yadong Xie  wrote:
> >
> > > Hi all
> > >
> > > I want to start the vote for FLIP-100, which proposes to add attempt
> > > information inside subtask and timeline in web UI.
> > >
> > > To help everyone better understand the proposal, we spent some efforts
> on
> > > making an online POC
> > >
> > > Timeline Attempt (click the vertex timeline to see the differences):
> > > previous web:
> > >
> >
> http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/timeline
> > > POC web:
> > >
> > >
> >
> http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/timeline
> > >
> > > Subtask Attempt (click the vertex and switch to subtask tab to see the
> > > differences):
> > > previous web:
> > >
> >
> http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/overview
> > > POC web:
> > >
> > >
> >
> http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/overview
> > >
> > >
> > > The vote will last for at least 72 hours, following the consensus
> voting
> > > process.
> > >
> > > FLIP wiki:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
> > >
> > > Discussion thread:
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > >
> > > Thanks,
> > >
> > > Yadong
> > >
> >
>

[jira] [Created] (FLINK-16423) test_ha_per_job_cluster_datastream.sh gets stuck

[Robert Metzger (Jira)]

Robert Metzger created FLINK-16423:
--

 Summary: test_ha_per_job_cluster_datastream.sh gets stuck
 Key: FLINK-16423
 URL: https://issues.apache.org/jira/browse/FLINK-16423
 Project: Flink
  Issue Type: Bug
  Components: Tests
Reporter: Robert Metzger


This was seen in 
https://dev.azure.com/rmetzger/Flink/_build/results?buildId=5905&view=logs&j=b1623ac9-0979-5b0d-2e5e-1377d695c991&t=e7804547-1789-5225-2bcf-269eeaa37447
 ... the relevant part of the logs is here:

{code}
2020-03-04T11:27:25.4819486Z 
==
2020-03-04T11:27:25.4820470Z Running 'Running HA per-job cluster (rocks, 
non-incremental) end-to-end test'
2020-03-04T11:27:25.4820922Z 
==
2020-03-04T11:27:25.4840177Z TEST_DATA_DIR: 
/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-25482960156
2020-03-04T11:27:25.6712478Z Flink dist directory: 
/home/vsts/work/1/s/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT
2020-03-04T11:27:25.6830402Z Flink dist directory: 
/home/vsts/work/1/s/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT
2020-03-04T11:27:26.2988914Z Starting zookeeper daemon on host fv-az655.
2020-03-04T11:27:26.3001237Z Running on HA mode: parallelism=4, backend=rocks, 
asyncSnapshots=true, and incremSnapshots=false.
2020-03-04T11:27:27.4206924Z Starting standalonejob daemon on host fv-az655.
2020-03-04T11:27:27.4217066Z Start 1 more task managers
2020-03-04T11:27:30.8412541Z Starting taskexecutor daemon on host fv-az655.
2020-03-04T11:27:38.1779980Z Job () is running.
2020-03-04T11:27:38.1781375Z Running JM watchdog @ 89778
2020-03-04T11:27:38.1781858Z Running TM watchdog @ 89779
2020-03-04T11:27:38.1783272Z Waiting for text Completed checkpoint [1-9]* for 
job  to appear 2 of times in logs...
2020-03-04T13:21:29.9076797Z ##[error]The operation was canceled.
2020-03-04T13:21:29.9094090Z ##[section]Finishing: Run e2e tests
{code}

The last three lines indicate that the test is waiting forever for a checkpoint 
to appear.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16424) Can't verify PGP signatures of Flink 1.9.2 and 1.10.0

[Bob (Jira)]

Bob created FLINK-16424:
---

 Summary: Can't verify PGP signatures of Flink 1.9.2 and 1.10.0
 Key: FLINK-16424
 URL: https://issues.apache.org/jira/browse/FLINK-16424
 Project: Flink
  Issue Type: Improvement
Reporter: Bob


I tried to follow the steps on the download page 
[https://flink.apache.org/downloads.html] and 
[http://www.apache.org/info/verification.html] but i am unable to verify the 
Flink packages with the help of the PGP signatures of Flink 1.9.2 and 1.10.0.

Steps to reproduce:
 # Download Flink via a mirror 
[https://www.apache.org/dyn/closer.lua/flink/flink-1.10.0/flink-1.10.0-bin-scala_2.12.tgz]
 # Download PGP signature file 
[https://www.apache.org/dist/flink/flink-1.10.0/flink-1.10.0-bin-scala_2.12.tgz.asc]
 # Download release-signing keys file [https://www.apache.org/dist/flink/KEYS]

{code:java}
# gpg --import KEYS 
gpg: key 04D9B832: "Alan Gates (No comment) " not changed
gpg: key 0CBAAE9F: "Sean Owen (CODE SIGNING KEY) " not 
changed
gpg: key 0410DA0C: "Ted Dunning (for signing Apache releases) 
" not changed
gpg: key 3592721E: "Henry Saputra (CODE SIGNING KEY) " not 
changed
gpg: key 3D0C92B9: "Owen O'Malley (Code signing) " not 
changed
gpg: key D9839159: "Robert Metzger (CODE SIGNING KEY) " 
not changed
gpg: key 9D403309: "Ufuk Celebi (CODE SIGNING KEY) " not 
changed
gpg: key D675A2E9: "Márton Balassi (CODE SIGNING KEY) " 
not changed
gpg: key C2909CBF: "Maximilian Michels " not changed
gpg: key 34911D5A: "Fabian Hueske (CODE SIGNING KEY) " not 
changed
gpg: key B065B356: "Tzu-Li Tai (CODE SIGNING KEY) " not 
changed
gpg: key 121D7293: "Aljoscha Krettek (CODE SIGNING KEY) " 
not changed
gpg: key 11D464BA: "Chesnay Schepler (CODE SIGNING KEY) " 
not changed
gpg: key 35C33D6A: "Tzu-Li Tai (CODE SIGNING KEY) " not 
changed
gpg: key A96CFFD5: "Till Rohrmann (stsffap) " not changed
gpg: key D920A98C: "Thomas Weise " not changed
gpg: key 3B79EA0E: "jincheng Sun (jincheng) " not changed
gpg: key F7059BA4: "Kurt Young " not changed
gpg: key EFAE3202: "Jark Wu (CODE SIGNING KEY) " not changed
gpg: Total number processed: 19
gpg:  unchanged: 19
{code}
{code:java}
# gpg --verify flink-1.10.0-bin-scala_2.12.tgz.asc 
flink-1.10.0-bin-scala_2.12.tgz
gpg: Signature made Fri 07 Feb 2020 07:36:24 PM CET using RSA key ID 89C115E8
gpg: Can't check signature: No public key
{code}
{code:java}
# gpg --keyserver pgpkeys.mit.edu --recv-key 89C115E8
gpg: requesting key 89C115E8 from hkp server pgpkeys.mit.edu
gpgkeys: key 89C115E8 not found on keyserver
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
{code}
{code:java}
# gpg --verify flink-1.9.2-bin-scala_2.12.tgz.asc2 
flink-1.9.2-bin-scala_2.12.tgz
gpg: Signature made Fri 24 Jan 2020 06:08:33 AM CET using RSA key ID 57B6476C
gpg: Can't check signature: No public key
{code}
{code:java}
# gpg --keyserver pgpkeys.mit.edu --recv-key 57B6476C
gpg: requesting key 57B6476C from hkp server pgpkeys.mit.edu
gpgkeys: key 57B6476C not found on keyserver
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
{code}
Could someone check if a key is missing in the release-signing keys file? Or 
something else is wrong? e.g. for Flink 1.9.1 these steps seem to be fine.


{code:java}
gpg --verify flink-1.9.1-bin-scala_2.12.tgz.asc flink-1.9.1-bin-scala_2.12.tgz
gpg: Signature made Mon 30 Sep 2019 08:57:32 AM CEST using RSA key ID EFAE3202
gpg: Good signature from "Jark Wu (CODE SIGNING KEY) "
gpg: WARNING: This key is not certified with a trusted signature!
gpg:  There is no indication that the signature belongs to the owner.
Primary key fingerprint: E2C4 5417 BED5 C104 154F  3410 85BA CB5A EFAE 3202
 {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16425) Add rate limiting feature for kafka table source

[Zou (Jira)]

Zou created FLINK-16425:
---

 Summary: Add rate limiting feature for kafka table source
 Key: FLINK-16425
 URL: https://issues.apache.org/jira/browse/FLINK-16425
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Reporter: Zou


There is a rate limiting feature in kafka source, but kafka table source cannot 
use this. We could support this in kafka table source.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16426) Add rate limiting feature for kafka table source

[Zou (Jira)]

Zou created FLINK-16426:
---

 Summary: Add rate limiting feature for kafka table source
 Key: FLINK-16426
 URL: https://issues.apache.org/jira/browse/FLINK-16426
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Reporter: Zou


There is a rate limiting feature in kafka source, but kafka table source dose 
not support this. We could add this feature in kafka table source.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16427) Remove directly throw ProgramInvocationExceptions in RemoteStreamEnvironment

[Zili Chen (Jira)]

Zili Chen created FLINK-16427:
-

 Summary: Remove directly throw ProgramInvocationExceptions in 
RemoteStreamEnvironment
 Key: FLINK-16427
 URL: https://issues.apache.org/jira/browse/FLINK-16427
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16428) Fine-grained network buffer management for backpressure

[Zhijiang (Jira)]

Zhijiang created FLINK-16428:


 Summary: Fine-grained network buffer management for backpressure
 Key: FLINK-16428
 URL: https://issues.apache.org/jira/browse/FLINK-16428
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Network
Reporter: Zhijiang
 Fix For: 1.11.0


It is an umbrella ticket for tracing the progress of this improvement.

This is the second ingredient to solve the “checkpoints under backpressure” 
problem (together with unaligned checkpoints). It consists of two steps:
 * See if we can use less network memory in general for streaming jobs (with 
potentially different distribution of floating buffers in the input side)
 * Under backpressure, reduce network memory to have less in-flight data (needs 
specification of algorithm and experiments)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Flink dev blog

[Zhijiang]

Big +1 for this proposal and second Ufuk's feeling!

I guess "Engine room" section in Wiki would attract lots of technical fans.:)

Best,
Zhijiang


--
From:Yu Li 
Send Time:2020 Mar. 4 (Wed.) 14:42
To:dev 
Cc:vthinkxie 
Subject:Re: Flink dev blog

Big +1 on adding a dev blog and starting with wiki. And +1 to promote the
fully polished articles to blog web with a formal process.

The latter one also brings up another good-to-have improvement that adding
categories and navigation in our blog so people could easily find different
topics like release-announcement/events/tech-articles, etc. but I think
we'd better open another thread to keep this one on track (smile).

I'd also like to add one potential topic around in-production practice of
using RocksDB state backend (which seems to be a popular topic in ML
discussions), such as how to enable and monitor RocksDB metrics and do
debugging/perf-tuning with the metrics/logs, and introduce
internals/details around the RocksDB memory management mechanism.

Best Regards,
Yu


On Wed, 4 Mar 2020 at 11:07, Xintong Song  wrote:

> I also like Ufuk's idea.
>
> The wiki allows people to post on their works in a quick and easier way.
> For me and probably many other Chinese folks, writing and polishing a
> formal article in English usually takes a long time, of which a significant
> portion is spent on polishing the language. If the blog does not require
> such formal and high quality languages, I believe it will make things a lot
> easier and encourage more people to share their ideas. Besides, it also
> avoids putting more review workloads on committers.
>
> Regarding promoting wiki post to the main blog, I think the wiki feedbacks
> (comment, likes, etc.) could be a great input. We can also contact the
> original author before promoting posts to the main blog to refine the
> article (responding to the wiki comments, polishing languages, adding
> latest updates, etc.).
>
> Thank you~
>
> Xintong Song
>
>
>
> On Wed, Mar 4, 2020 at 10:25 AM Jark Wu  wrote:
>
> > +1 for this.
> >
> > Regarding to the place to hold blogs. Personally, I prefer to use
> existing
> > blog and separate by tags/categories and title names.
> > Because, the dev blogs are very good learning materials. I believe many
> > users will be interested in these posts. It's just like
> > "Technology Deep Dive" talks in Flink Forward which attracts many
> > audiences. Putting them with main blog together can help
> > to give the dev blogs more exposure.
> >
> > But I also share Robert's concern. So I'm in favor of Ufuk's idea:
> starting
> > with Wiki, and moving good posts to the main blog gradually.
> > We should also improve our current blog web to support tags/categories.
> > Maybe @vthink...@gmail.com  Yadong can help on
> this.
> >
> > Best,
> > Jark
> >
> >
> > On Wed, 4 Mar 2020 at 05:03, Ufuk Celebi  wrote:
> >
> > > +1 on starting with the Wiki. I really like the name "Engine room". Can
> > we
> > > name the section in the Wiki like that? In general, if we think that a
> > post
> > > or a series of posts would be a good fit for the main blog, it would be
> > > pretty straightforward to promote a post from the Engine room to the
> main
> > > blog (including further edits, focus on language, etc.)
> > >
> > > – Ufuk
> > >
> > >
> > > On Tue, Mar 3, 2020 at 5:58 PM Rong Rong  wrote:
> > >
> > > > Big +1 on this. Some of these topics are not only for contributors,
> but
> > > > would also be super useful for advance users.
> > > > One topic I can think of in addition is: Security/Kerberos.
> > > >
> > > > Echo on Both Seth's idea, we could have both wiki and PR submission:
> > > > As Robert mentioned - wiki submission would make the experience more
> > > > frictionless.
> > > > I was having concerns splitting the blog posts in two places, but I
> > also
> > > > think adding the banner/blog-series of "Flink Engine Room" would help
> > > > readers distinct between the two.
> > > >
> > > > --
> > > > Rong
> > > >
> > > > On Tue, Mar 3, 2020 at 8:39 AM Dian Fu 
> wrote:
> > > >
> > > > > Big +1 on this idea. It will benefit both the developers and users
> a
> > > lot.
> > > > >
> > > > > Regarding to the place to hold these blogs, my preference is 3) as
> I
> > > > > notice that there are already a few high quality blogs on flink
> > > > web-site[1]
> > > > > and I guess that may be a good place to start with. We just need to
> > > > figure
> > > > > out a way to let contributors clearly mark the audience of their
> > > articles
> > > > > and also help users to easily determine whether the content is what
> > > they
> > > > > want.
> > > > >
> > > > > Regards,
> > > > > Dian
> > > > >
> > > > > [1] https://flink.apache.org/blog/  >
> > > > > > 在 2020年3月3日，下午11:14，Yadong Xie  写道：
> > > > > >
> > > > > > Hi all
> > > > > >
> > > > > > maybe we can use markdown & GitHub to make the submission easy to
> > > > rev

Re: Building with Hadoop 3

[Stephan Ewen]

Have you tried to just export Hadoop 3's classpath to `HADOOP_CLASSPATH`
and see if that works out of the box?

If the main use case is HDFS access, then there is a fair chance it might
just work, because Flink uses only a small subset of the Hadoop FS API
which is stable between 2.x and 3.x, as far as I tried it out a while back.

On Tue, Mar 3, 2020 at 6:03 PM LINZ, Arnaud 
wrote:

> Hello,
>
> Have you shared it somewhere on the web already?
>
> Best,
>
> Arnaud
>
>
>
> *De :* vino yang 
> *Envoyé :* mercredi 4 décembre 2019 11:55
> *À :* Márton Balassi 
> *Cc :* Chesnay Schepler ; Foster, Craig <
> foscr...@amazon.com>; u...@flink.apache.org; dev@flink.apache.org
> *Objet :* Re: Building with Hadoop 3
>
>
>
> Hi Marton,
>
>
>
> Thanks for your explanation. Personally, I look forward to your
> contribution!
>
>
>
> Best,
>
> Vino
>
>
>
> Márton Balassi  于2019年12月4日周三 下午5:15写道：
>
> Wearing my Cloudera hat I can tell you that we have done this exercise for
> our distros of the  3.0 and 3.1 Hadoop versions. We have not contributed
> these back just yet, but we are open to do so. If the community is
> interested we can contribute those changes back to flink-shaded and suggest
> the necessay changes to flink too. The task was not overly complex, but it
> certainly involved a bit of dependency hell. :-)
>
>
>
> Right now we are focused on internal timelines, but we could invest into
> contributing this back in the end of January timeframe if the community
> deems this a worthwhile effort.
>
>
>
> Best,
>
> Marton
>
>
>
> On Wed, Dec 4, 2019 at 10:00 AM Chesnay Schepler 
> wrote:
>
> There's no JIRA and no one actively working on it. I'm not aware of any
> investigations on the matter; hence the first step would be to just try it
> out.
>
>
>
> A flink-shaded artifact isn't a hard requirement; Flink will work with any
> 2.X hadoop distribution (provided that there aren't any dependency clashes).
>
>
>
> On 03/12/2019 18:22, Foster, Craig wrote:
>
> Hi:
>
> I don’t see a JIRA for Hadoop 3 support. I see a comment on a JIRA here
> from a year ago that no one is looking into Hadoop 3 support [1]. Is there
> a document or JIRA that now exists which would point to what needs to be
> done to support Hadoop 3? Right now builds with Hadoop 3 don’t work
> obviously because there’s no flink-shaded-hadoop-3 artifacts.
>
>
>
> Thanks!
>
> Craig
>
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-11086
>
>
>
>
>
>
> --
>
> L'intégrité de ce message n'étant pas assurée sur internet, la société
> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si
> vous n'êtes pas destinataire de ce message, merci de le détruire et
> d'avertir l'expéditeur.
>
> The integrity of this message cannot be guaranteed on the Internet. The
> company that sent this message cannot therefore be held liable for its
> content nor attachments. Any unauthorized use or dissemination is
> prohibited. If you are not the intended recipient of this message, then
> please delete it and notify the sender.
>

Re: Creating TemporalTable based on Catalog table in SQL Client

[Bowen Li]

you would need to reference the table with fully qualified name with
catalog and database

On Wed, Mar 4, 2020 at 02:17 Gyula Fóra  wrote:

> I guess it will only work now if you specify the catalog name too when
> referencing the table.
>
>
> On Wed, Mar 4, 2020 at 11:15 AM Gyula Fóra  wrote:
>
> > You are right but still if the default catalog is something else and
> > that's the one containing the table then it still wont work currently.
> >
> > Gyula
> >
> > On Wed, Mar 4, 2020 at 5:08 AM Bowen Li  wrote:
> >
> >> Hi Gyula,
> >>
> >> What line 622 (the link you shared) does is not registering catalogs,
> but
> >> setting an already registered catalog as the current one. As you can see
> >> from the method and its comment, catalogs are loaded first before any
> >> tables in yaml are registered, so you should be able to achieve what you
> >> described.
> >>
> >> Bowen
> >>
> >> On Tue, Mar 3, 2020 at 5:16 AM Gyula Fóra  wrote:
> >>
> >> > Hi all!
> >> >
> >> > I was testing the TemporalTable functionality in the SQL client while
> >> using
> >> > the Hive Catalog and I ran into the following problem.
> >> >
> >> > I have a table created in the Hive catalog and I want to create a
> >> temporal
> >> > table over it.
> >> >
> >> > As we cannot create temporal tables in SQL directly I have to define
> it
> >> in
> >> > the environment yaml file. Unfortunately it seems to be impossible to
> >> > reference a table only present in the catalog (not in the yaml) as
> >> catalogs
> >> > are loaded only after creating the temporal table (see
> >> >
> >> >
> >>
> https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/ExecutionContext.java#L622
> >> > )
> >> >
> >> > I am wondering if it would make sense to set the catalogs before all
> >> else
> >> > or if that would cause some other problems.
> >> >
> >> > What do you think?
> >> > Gyula
> >> >
> >>
> >
>

Re: Flink dev blog

[Arvid Heise]

I see that the majority would like to have an uncomplicated process to
publish an article first to gather feedback and then like to have polished
versions on the blog with official review process.

Then, the obvious solution is to have a process that is two-fold:
* First a draft is published and reviewed by peers. The draft could be
polished in smaller increments including proof-reading by native-level
writers.
* Second, when the draft converged enough, we would then make an official
pull request for the dev blog, which would (hopefully) be merged rather
quickly.

For the draft, we would have a wiki subarea "Engine room", which would be
the default location for such drafts. Pages in the wiki would allow for a
gradual polishing and may even live comparably long if the author does not
find the time for polishing. The information is in a semi-published state,
where devs and experts can already find and use it, but it would not
attract as many views as in a blog.

But I'd explicitly also allow drafts to go directly to a PR (with risk of
having many iterations). I'd even say that if someone feels more
comfortable to online editors such as google docs and has enough reviewers
for that, they could go with it. Here, the author needs to ensure a timely
progress or revert to the wiki, since all intermediate versions are
effectively hidden for non-reviewers.

Would the community agree with this approach or do you have concerns? If no
major concerns are raised, I'd start preparation with the wiki on Monday
(03/09/2020).

I'd raise the issue about wiki and blog structure, when we got some
articles to avoid too many concurrent discussions.


On Wed, Mar 4, 2020 at 5:54 PM Zhijiang 
wrote:

> Big +1 for this proposal and second Ufuk's feeling!
>
> I guess "Engine room" section in Wiki would attract lots of technical
> fans.:)
>
> Best,
> Zhijiang
>
>
> --
> From:Yu Li 
> Send Time:2020 Mar. 4 (Wed.) 14:42
> To:dev 
> Cc:vthinkxie 
> Subject:Re: Flink dev blog
>
> Big +1 on adding a dev blog and starting with wiki. And +1 to promote the
> fully polished articles to blog web with a formal process.
>
> The latter one also brings up another good-to-have improvement that adding
> categories and navigation in our blog so people could easily find different
> topics like release-announcement/events/tech-articles, etc. but I think
> we'd better open another thread to keep this one on track (smile).
>
> I'd also like to add one potential topic around in-production practice of
> using RocksDB state backend (which seems to be a popular topic in ML
> discussions), such as how to enable and monitor RocksDB metrics and do
> debugging/perf-tuning with the metrics/logs, and introduce
> internals/details around the RocksDB memory management mechanism.
>
> Best Regards,
> Yu
>
>
> On Wed, 4 Mar 2020 at 11:07, Xintong Song  wrote:
>
> > I also like Ufuk's idea.
> >
> > The wiki allows people to post on their works in a quick and easier way.
> > For me and probably many other Chinese folks, writing and polishing a
> > formal article in English usually takes a long time, of which a
> significant
> > portion is spent on polishing the language. If the blog does not require
> > such formal and high quality languages, I believe it will make things a
> lot
> > easier and encourage more people to share their ideas. Besides, it also
> > avoids putting more review workloads on committers.
> >
> > Regarding promoting wiki post to the main blog, I think the wiki
> feedbacks
> > (comment, likes, etc.) could be a great input. We can also contact the
> > original author before promoting posts to the main blog to refine the
> > article (responding to the wiki comments, polishing languages, adding
> > latest updates, etc.).
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Wed, Mar 4, 2020 at 10:25 AM Jark Wu  wrote:
> >
> > > +1 for this.
> > >
> > > Regarding to the place to hold blogs. Personally, I prefer to use
> > existing
> > > blog and separate by tags/categories and title names.
> > > Because, the dev blogs are very good learning materials. I believe many
> > > users will be interested in these posts. It's just like
> > > "Technology Deep Dive" talks in Flink Forward which attracts many
> > > audiences. Putting them with main blog together can help
> > > to give the dev blogs more exposure.
> > >
> > > But I also share Robert's concern. So I'm in favor of Ufuk's idea:
> > starting
> > > with Wiki, and moving good posts to the main blog gradually.
> > > We should also improve our current blog web to support tags/categories.
> > > Maybe @vthink...@gmail.com  Yadong can help on
> > this.
> > >
> > > Best,
> > > Jark
> > >
> > >
> > > On Wed, 4 Mar 2020 at 05:03, Ufuk Celebi  wrote:
> > >
> > > > +1 on starting with the Wiki. I really like the name "Engine room".
> Can
> > > we
> > > > name the section in the Wiki like that? In general, if we think that
> a
> > > post
> > > > or a

[jira] [Created] (FLINK-16429) failed to restore flink job from checkpoints due to unhandled exceptions

[Yu Yang (Jira)]

Yu Yang created FLINK-16429:
---

 Summary: failed to restore flink job from checkpoints due to 
unhandled exceptions
 Key: FLINK-16429
 URL: https://issues.apache.org/jira/browse/FLINK-16429
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.9.1
Reporter: Yu Yang


We are trying to restore our flink job from check-points, and run into 
AskTimeoutException related failures at a high frequency. Our environment is 
Hadoop 2.7.1 + Yarn + Flink 1.9.1. 

We hit this issue in 9 out of 10 runs, and were able to restore the application 
from given check-points from time to time. As the application can be restored, 
the check-point files shall not be corrupted. It seems that the issue is that 
jobmaster got timeout when it handles job submission request.  

 

Below is the exception stack trace, it is thrown from

[https://github.com/apache/flink/blob/2ec645a5bfd3cfadaf0057412401e91da0b21873/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/AbstractHandler.java#L209]

2020-03-05 00:04:14,360 ERROR 
org.apache.flink.runtime.rest.handler.job.JobSubmitHandler - Unhandled 
exception: httpRequest uri:/v1/jobs, context: 
ChannelHandlerContext(org.apache.flink.runtime.rest.handler.router.RouterHandler_ROUTED_HANDLER,
 [id: 0xc39aca33, L:/10.1.85.22:41000 - R:/10.1.16.251:44]) 
akka.pattern.AskTimeoutException: Ask timed out on 
[Actor[akka://flink/user/dispatcher#-34498396]] after [1 ms]. Message of 
type [org.apache.flink.runtime.rpc.messages.LocalFencedMessage]. A typical 
reason for `AskTimeoutException` is that the recipient actor didn't send a 
reply. at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635) 
at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635) at 
akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:648) at 
akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205) at 
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
 at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) 
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599) 
at 
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
 at 
akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
 at 
akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
 at 
akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
 at java.lang.Thread.run(Thread.java:748)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16430) Pipelined region scheduling

[Zhu Zhu (Jira)]

Zhu Zhu created FLINK-16430:
---

 Summary: Pipelined region scheduling
 Key: FLINK-16430
 URL: https://issues.apache.org/jira/browse/FLINK-16430
 Project: Flink
  Issue Type: Task
  Components: Runtime / Coordination
Affects Versions: 1.11.0
Reporter: Zhu Zhu
 Fix For: 1.11.0


Pipelined region scheduling is targeting to allow batch jobs with PIPELINED 
data exchanges to run without the risk to encounter a resource deadlock.

More details and work items will be added later when the detailed design is 
ready.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[PROPOSAL] Reverse the dependency from flink-streaming-java to flink-client

[tison]

Hi devs,

Here is a proposal to reverse the dependency from flink-streaming-java to
flink-client, for a proper
module dependency graph. Since it changes current structure, it should be
discussed publicly.

The original idea comes from that flink-streaming-java acts as an API only
module just as what
we do in its batch companion flink-java. If a Flink user want to write a
minimum DataStream
program, the only dependency should be flink-streaming java.

However, currently as it is implemented, flink-client and even
flink-runtime are transitively polluted
in when user depends on flink-streaming-java. These dependencies polluted
in as

flink-client:
  - previously, ClusterClient, which is removed by FLIP-73 Executors
  - accidentally, ProgramInvocationException, we just throw in place as it
is accessible.
  - transitively, flink-optimizer, for one utility.
  - transitively, flink-java, for several utilities.
flink-runtime:
  - mainly for JobGraph generating.

With a previous discussion with @Aljoscha Krettek  our
goal is briefly making flink-streaming-java
an API only module. As a first step we can break the dependency from
flink-streaming-java to
flink-client[1][2].

With this first step, continuously we factor out common utilities in
flink-java to
flink-core and eventually eliminate dependencies from streaming to batch;
while
orthogonally, we factor out job compilation logic into
flink-streaming-compiler module and
break the dependency to flink-runtime. The final dependency graph will be:


flink-client -> flink-streaming-compiler -> flink-runtime
 \->
flink-streaming-java

Looking forward to your feedback. Basically whether or not it is in a right
direction, and if so,
how the community integrates this proposal.

Best,
tison.

[1] https://issues.apache.org/jira/browse/FLINK-15090
[2] https://issues.apache.org/jira/browse/FLINK-16427

Re: Flink dev blog

[Xingbo Huang]

Thanks a for this proposal.

As a new contributor to Flink, it would be very helpful to have such blogs
for us to understand the future of Flink and get involved

BTW, I have a question whether the dev blog needs a template like FLIP.

Of course, There is no doubt that dev blogs do not need to be as formal as
FLIP, but templates can be more helpful for developers to understand
articles.

Best,

Xingbo


Arvid Heise  于2020年3月5日周四 上午2:55写道：

> I see that the majority would like to have an uncomplicated process to
> publish an article first to gather feedback and then like to have polished
> versions on the blog with official review process.
>
> Then, the obvious solution is to have a process that is two-fold:
> * First a draft is published and reviewed by peers. The draft could be
> polished in smaller increments including proof-reading by native-level
> writers.
> * Second, when the draft converged enough, we would then make an official
> pull request for the dev blog, which would (hopefully) be merged rather
> quickly.
>
> For the draft, we would have a wiki subarea "Engine room", which would be
> the default location for such drafts. Pages in the wiki would allow for a
> gradual polishing and may even live comparably long if the author does not
> find the time for polishing. The information is in a semi-published state,
> where devs and experts can already find and use it, but it would not
> attract as many views as in a blog.
>
> But I'd explicitly also allow drafts to go directly to a PR (with risk of
> having many iterations). I'd even say that if someone feels more
> comfortable to online editors such as google docs and has enough reviewers
> for that, they could go with it. Here, the author needs to ensure a timely
> progress or revert to the wiki, since all intermediate versions are
> effectively hidden for non-reviewers.
>
> Would the community agree with this approach or do you have concerns? If no
> major concerns are raised, I'd start preparation with the wiki on Monday
> (03/09/2020).
>
> I'd raise the issue about wiki and blog structure, when we got some
> articles to avoid too many concurrent discussions.
>
>
> On Wed, Mar 4, 2020 at 5:54 PM Zhijiang  .invalid>
> wrote:
>
> > Big +1 for this proposal and second Ufuk's feeling!
> >
> > I guess "Engine room" section in Wiki would attract lots of technical
> > fans.:)
> >
> > Best,
> > Zhijiang
> >
> >
> > --
> > From:Yu Li 
> > Send Time:2020 Mar. 4 (Wed.) 14:42
> > To:dev 
> > Cc:vthinkxie 
> > Subject:Re: Flink dev blog
> >
> > Big +1 on adding a dev blog and starting with wiki. And +1 to promote the
> > fully polished articles to blog web with a formal process.
> >
> > The latter one also brings up another good-to-have improvement that
> adding
> > categories and navigation in our blog so people could easily find
> different
> > topics like release-announcement/events/tech-articles, etc. but I think
> > we'd better open another thread to keep this one on track (smile).
> >
> > I'd also like to add one potential topic around in-production practice of
> > using RocksDB state backend (which seems to be a popular topic in ML
> > discussions), such as how to enable and monitor RocksDB metrics and do
> > debugging/perf-tuning with the metrics/logs, and introduce
> > internals/details around the RocksDB memory management mechanism.
> >
> > Best Regards,
> > Yu
> >
> >
> > On Wed, 4 Mar 2020 at 11:07, Xintong Song  wrote:
> >
> > > I also like Ufuk's idea.
> > >
> > > The wiki allows people to post on their works in a quick and easier
> way.
> > > For me and probably many other Chinese folks, writing and polishing a
> > > formal article in English usually takes a long time, of which a
> > significant
> > > portion is spent on polishing the language. If the blog does not
> require
> > > such formal and high quality languages, I believe it will make things a
> > lot
> > > easier and encourage more people to share their ideas. Besides, it also
> > > avoids putting more review workloads on committers.
> > >
> > > Regarding promoting wiki post to the main blog, I think the wiki
> > feedbacks
> > > (comment, likes, etc.) could be a great input. We can also contact the
> > > original author before promoting posts to the main blog to refine the
> > > article (responding to the wiki comments, polishing languages, adding
> > > latest updates, etc.).
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Wed, Mar 4, 2020 at 10:25 AM Jark Wu  wrote:
> > >
> > > > +1 for this.
> > > >
> > > > Regarding to the place to hold blogs. Personally, I prefer to use
> > > existing
> > > > blog and separate by tags/categories and title names.
> > > > Because, the dev blogs are very good learning materials. I believe
> many
> > > > users will be interested in these posts. It's just like
> > > > "Technology Deep Dive" talks in Flink Forward which attracts many
> > > > audiences. Putting them

Re: [DISCUSS] FLIP-110: Support LIKE clause in CREATE TABLE

[Jingsong Li]

Thanks Dawid for starting this discussion.

I like the "LIKE".

1.For "INHERITS", I think this is a good feature too, yes, ALTER TABLE will
propagate any changes in column data definitions and check constraints down
the inheritance hierarchy. A inherits B, A and B share every things, they
have the same kafka topic. If modify schema of B, this means underlying
kafka topic schema changed, so I think it is good to modify A too. If this
for "ConfluentSchemaRegistryCatalog" mention by Jark, I think sometimes
this is just we want.
But "LIKE" also very useful for many cases.

2.For LIKE statement in schema, I know two kinds of like syntax, one is
MySQL/hive/sqlserver, the other is PostgreSQL. I prefer former:
- In the FLIP, there is "OVERWRITING OPTIONS", this will overwrite
properties in "with"? This looks weird, because "LIKE" is in schema, but it
can affect outside properties.

Best,
Jingsong Lee

On Wed, Mar 4, 2020 at 2:05 PM Dawid Wysakowicz 
wrote:

> Hi Jark,
> I did investigate the INHERITS clause, but it has a semantic that in my
> opinion we definitely don't want to support. INHERITS creates a new table
> with a "link" to the original table. Therefore if you e.g change the schema
> of the original table it's also reflected in the child table. It's also
> possible for tables like A inherits B query them like Select * from only A,
> by default it returns results from both tables. I am pretty sure it's not
> what we're looking for.
>
> PostgreSQL implements both the LIKE clause and INHERITS. I am open for
> discussion if we should support multiple LIKE statements or not. Standard
> also allows declaring the clause after the schema part. We can also do it.
> Nevertheless I think including multiple tables might be useful, e.g. when
> you want to union two tables and output to the same Kafka cluster and just
> change the target topic. I know it's not a very common use case but it's
> not a big effort to support it.
>
> Let me know what you think.
>
> Best,
> Dawid
>
> On Wed, 4 Mar 2020, 04:55 Jark Wu,  wrote:
>
> > Hi Dawid,
> >
> > Thanks for starting this discussion. I like the idea.
> > Once we support more intergrated catalogs,
> > e.g. ConfluentSchemaRegistryCatalog, this problem will be more urgent.
> > Because it's very common to adjust existing tables in catalog slightly.
> >
> > My initial thought was introducing INHERITS keyword, which is also
> > supported in PostgreSQL [1].
> > This is also similar to the functionality of Hive CREATE TABLE LIKE [2].
> >
> > CREATE TEMPORARY TABLE MyTable (WATERMARK FOR ts) INHERITS
> > cat.db.KafkoTopic
> > CREATE TEMPORARY TABLE MyTable (WATERMARK FOR ts) INHERITS
> > cat.db.KafkoTopic WITH ('k' = 'v')
> >
> > The INHERITS can inherit an existing table with all columns, watermark,
> and
> > properties, but the properties and watermark and be overwrited
> explicitly.
> >
> > The reason I prefer INHERITS rather than LIKE is the keyword position. We
> > are copying an existing table definition including the properties.
> > However, LIKE appears in the schema part, it sounds like copying
> properties
> > into schema part of DDL.
> >
> > Besides of that, I'm not sure whether the use case stands "merging two
> > tables into a single one with a different connector".
> > From my understanding, most use cases are just slightly adjusting on an
> > existing catalog table with new properties or watermarks.
> > Do we really need to merge two table definitions into a single one? For
> > example, is it possible to merge a Kafka table definition and
> > a Filesystem table definition into a new Kafka table, and the new Kafka
> > table exactly matches the underlying physical data format?
> >
> > Best,
> > Jark
> >
> > [1]: https://www.postgresql.org/docs/9.5/sql-createtable.html
> > [2]:
> >
> >
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTableLike
> >
> >
> > On Tue, 3 Mar 2020 at 21:12, Dawid Wysakowicz 
> > wrote:
> >
> > > Hi devs,
> > >
> > > I wanted to bring another improvement proposal up for a discussion.
> Often
> > > users need to adjust existing tables slightly. This is especially
> useful
> > > when users need to enhance a table created from an external tool (e.g.
> > > HIVE) with Flink's specific information such as e.g watermarks. It can
> > also
> > > be a useful tool for ETL processes, e.g. merging two tables into a
> single
> > > one with a different connector.  My suggestion would be to support an
> > > optional *Feature T171, “LIKE clause in table definition” *of SQL
> > > standard 2008.
> > >
> > > You can see the description of the proposal here:
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-110%3A+Support+LIKE+clause+in+CREATE+TABLE
> > >
> > > Looking forward for your comments.
> > >
> > > Best,
> > >
> > > Dawid
> > >
> >
>


-- 
Best, Jingsong Lee

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

[Peter Huang]

Hi Yang and Kostas,

Thanks for the clarification. It makes more sense to me if the long term
goal is to replace per job mode to application mode
 in the future (at the time that multiple execute can be supported). Before
that, It will be better to keep the concept of
 application mode internally. As Yang suggested, User only need to use a
`-R/-- remote-deploy` cli option to launch
a per job cluster with the main function executed in cluster
entry-point.  +1 for the execution plan.



Best Regards
Peter Huang




On Tue, Mar 3, 2020 at 7:11 AM Yang Wang  wrote:

> Hi Peter,
>
> Having the application mode does not mean we will drop the cluster-deploy
> option. I just want to share some thoughts about “Application Mode”.
>
>
> 1. The application mode could cover the per-job sematic. Its lifecyle is
> bound
> to the user `main()`. And all the jobs in the user main will be executed
> in a same
> Flink cluster. In first phase of FLIP-85 implementation, running user main
> on the
> cluster side could be supported in application mode.
>
> 2. Maybe in the future, we also need to support multiple `execute()` on
> client side
> in a same Flink cluster. Then the per-job mode will evolve to application
> mode.
>
> 3. From user perspective, only a `-R/-- remote-deploy` cli option is
> visible. They
> are not aware of the application mode.
>
> 4. In the first phase, the application mode is working as “per-job”(only
> one job in
> the user main). We just leave more potential for the future.
>
>
> I am not against with calling it “cluster deploy mode” if you all think it
> is clearer for users.
>
>
>
> Best,
> Yang
>
> Kostas Kloudas  于2020年3月3日周二 下午6:49写道：
>
>> Hi Peter,
>>
>> I understand your point. This is why I was also a bit torn about the
>> name and my proposal was a bit aligned with yours (something along the
>> lines of "cluster deploy" mode).
>>
>> But many of the other participants in the discussion suggested the
>> "Application Mode". I think that the reasoning is that now the user's
>> Application is more self-contained.
>> It will be submitted to the cluster and the user can just disconnect.
>> In addition, as discussed briefly in the doc, in the future there may
>> be better support for multi-execute applications which will bring us
>> one step closer to the true "Application Mode". But this is how I
>> interpreted their arguments, of course they can also express their
>> thoughts on the topic :)
>>
>> Cheers,
>> Kostas
>>
>> On Mon, Mar 2, 2020 at 6:15 PM Peter Huang 
>> wrote:
>> >
>> > Hi Kostas,
>> >
>> > Thanks for updating the wiki. We have aligned with the implementations
>> in the doc. But I feel it is still a little bit confusing of the naming
>> from a user's perspective. It is well known that Flink support per job
>> cluster and session cluster. The concept is in the layer of how a job is
>> managed within Flink. The method introduced util now is a kind of mixing
>> job and session cluster to promising the implementation complexity. We
>> probably don't need to label it as Application Model as the same layer of
>> per job cluster and session cluster. Conceptually, I think it is still a
>> cluster mode implementation for per job cluster.
>> >
>> > To minimize the confusion of users, I think it would be better just an
>> option of per job cluster for each type of cluster manager. How do you
>> think?
>> >
>> >
>> > Best Regards
>> > Peter Huang
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Mar 2, 2020 at 7:22 AM Kostas Kloudas 
>> wrote:
>> >>
>> >> Hi Yang,
>> >>
>> >> The difference between per-job and application mode is that, as you
>> >> described, in the per-job mode the main is executed on the client
>> >> while in the application mode, the main is executed on the cluster.
>> >> I do not think we have to offer "application mode" with running the
>> >> main on the client side as this is exactly what the per-job mode does
>> >> currently and, as you described also, it would be redundant.
>> >>
>> >> Sorry if this was not clear in the document.
>> >>
>> >> Cheers,
>> >> Kostas
>> >>
>> >> On Mon, Mar 2, 2020 at 3:17 PM Yang Wang 
>> wrote:
>> >> >
>> >> > Hi Kostas,
>> >> >
>> >> > Thanks a lot for your conclusion and updating the FLIP-85 WIKI.
>> Currently, i have no more
>> >> > questions about motivation, approach, fault tolerance and the first
>> phase implementation.
>> >> >
>> >> > I think the new title "Flink Application Mode" makes a lot senses to
>> me. Especially for the
>> >> > containerized environment, the cluster deploy option will be very
>> useful.
>> >> >
>> >> > Just one concern, how do we introduce this new application mode to
>> our users?
>> >> > Each user program(i.e. `main()`) is an application. Currently, we
>> intend to only support one
>> >> > `execute()`. So what's the difference between per-job and
>> application mode?
>> >> >
>> >> > For per-job, user `main()` is always executed on client side. And
>> For application mode, user
>> >> > `main()` could be exe

Re: [DISCUSS] Introduce flink-connector-hive-xx modules

[Bowen Li]

Thanks, Jingsong, for bringing this up. We've received lots of feedbacks in
the past few months that the complexity involved in different Hive versions
has been quite painful for users to start with. So it's great to step
forward and deal with such issue.

Before getting on a decision, can you please explain:

1) why you proposed segregating hive versions into the 5 ranges above?
2) what different Hive features are supported in the 5 ranges?
3) have you tested that whether the proposed corresponding Flink module
will be fully compatible with each Hive version range?

Thanks,
Bowen



On Wed, Mar 4, 2020 at 1:00 AM Jingsong Lee  wrote:

> Hi all,
>
> I'd like to propose introduce flink-connector-hive-xx modules.
>
> We have documented the dependencies detailed information[2]. But still has
> some inconvenient:
> - Too many versions, users need to pick one version from 8 versions.
> - Too many versions, It's not friendly to our developers either, because
> there's a problem/exception, we need to look at eight different versions of
> hive client code, which are often various.
> - Too many jars, for example, users need to download 4+ jars for Hive 1.x
> from various places.
>
> We have discussed in [1] and [2], but unfortunately, we can not achieve an
> agreement.
>
> For improving this, I'd like to introduce few flink-connector-hive-xx
> modules in flink-connectors, module contains all the dependencies related
> to hive. And only support lower hive metastore versions:
> - "flink-connector-hive-1.2" to support hive 1.0.0 - 1.2.2
> - "flink-connector-hive-2.0" to support hive 2.0.0 - 2.0.1
> - "flink-connector-hive-2.2" to support hive 2.1.0 - 2.2.0
> - "flink-connector-hive-2.3" to support hive 2.3.0 - 2.3.6
> - "flink-connector-hive-3.1" to support hive 3.0.0 - 3.1.2
>
> Users can choose one and download to flink/lib. It includes all hive
> things.
>
> I try to use a single module to deploy multiple versions, but I can not
> find a suitable way, because different modules require different versions
> and different dependencies.
>
> What do you think?
>
> [1]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-have-separate-Flink-distributions-with-built-in-Hive-dependencies-td35918.html
> [2]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-109-Improve-Hive-dependencies-out-of-box-experience-td38290.html
>
> Best,
> Jingsong Lee
>

Re: [VOTE] FLIP-93: JDBC catalog and Postgres catalog

[Bowen Li]

I'm glad to announce that the voting of FLIP-93 has passed, with 7 +1  (3
binding: Jingsong, Kurt, Jark, 4 non-binding: Benchao, zoudan, Terry,
Leonard) and no -1.

Thanks everyone for participating!

Cheers,
Bowen

On Mon, Mar 2, 2020 at 7:33 AM Leonard Xu  wrote:

> +1 (non-binding).
>
>  Very useful feature especially for ETL, It will make  connecting to
> existed DB systems easier.
>
> Best,
> Leonard
>
> > 在 2020年3月2日，21:58，Jark Wu  写道：
> >
> > +1 from my side.
> >
> > Best,
> > Jark
> >
> > On Mon, 2 Mar 2020 at 21:40, Kurt Young  wrote:
> >
> >> +1
> >>
> >> Best,
> >> Kurt
> >>
> >>
> >> On Mon, Mar 2, 2020 at 5:32 PM Jingsong Lee 
> >> wrote:
> >>
> >>> +1 from my side.
> >>>
> >>> Best,
> >>> Jingsong Lee
> >>>
> >>> On Mon, Mar 2, 2020 at 11:06 AM Terry Wang  wrote:
> >>>
>  +1 (non-binding).
>  With this feature, we can more easily interact traditional database in
>  flink.
> 
>  Best,
>  Terry Wang
> 
> 
> 
> > 2020年3月1日 18:33，zoudan  写道：
> >
> > +1 (non-binding)
> >
> > Best,
> > Dan Zou
> >
> >
> >> 在 2020年2月28日，02:38，Bowen Li  写道：
> >>
> >> Hi all,
> >>
> >> I'd like to kick off the vote for FLIP-93 [1] to add JDBC catalog
> >> and
> >> Postgres catalog.
> >>
> >> The vote will last for at least 72 hours, following the consensus
> >>> voting
> >> protocol.
> >>
> >> [1]
> >>
> 
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-93%3A+JDBC+catalog+and+Postgres+catalog
> >>
> >> Discussion thread:
> >>
> 
> >>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-92-JDBC-catalog-and-Postgres-catalog-td36505.html
> >
> 
> 
> >>>
> >>> --
> >>> Best, Jingsong Lee
> >>>
> >>
>
>

Re: [DISCUSS] Introduce flink-connector-hive-xx modules

[Jingsong Li]

Thanks Bowen for involving.

> why you proposed segregating hive versions into the 5 ranges above? &
what different Hive features are supported in the 5 ranges?

For only higher client dependencies version support lower hive metastore
versions:
- Hive 1.0.0 - 1.2.2, thrift change is OK, only hive date column stats, we
can throw exception for the unsupported feature.
- Hive 2.0 and Hive 2.1, primary key support and alter_partition api change.
- Hive 2.2 no thrift change.
- Hive 2.3 change many things, lots of thrift change.
- Hive 3+, not null. unique, timestamp, so many things.

All these things can be found in hive_metastore.thrift.

I think I can try do more effort in implementation to use Hive 2.2 to
support Hive 2.0. So the range size will be 4.

> have you tested that whether the proposed corresponding Flink module will
be fully compatible with each Hive version range?

Yes, I have done some tests, not really for "fully", but it is a technical
judgment.

Best,
Jingsong Lee

On Thu, Mar 5, 2020 at 1:17 PM Bowen Li  wrote:

> Thanks, Jingsong, for bringing this up. We've received lots of feedbacks in
> the past few months that the complexity involved in different Hive versions
> has been quite painful for users to start with. So it's great to step
> forward and deal with such issue.
>
> Before getting on a decision, can you please explain:
>
> 1) why you proposed segregating hive versions into the 5 ranges above?
> 2) what different Hive features are supported in the 5 ranges?
> 3) have you tested that whether the proposed corresponding Flink module
> will be fully compatible with each Hive version range?
>
> Thanks,
> Bowen
>
>
>
> On Wed, Mar 4, 2020 at 1:00 AM Jingsong Lee 
> wrote:
>
> > Hi all,
> >
> > I'd like to propose introduce flink-connector-hive-xx modules.
> >
> > We have documented the dependencies detailed information[2]. But still
> has
> > some inconvenient:
> > - Too many versions, users need to pick one version from 8 versions.
> > - Too many versions, It's not friendly to our developers either, because
> > there's a problem/exception, we need to look at eight different versions
> of
> > hive client code, which are often various.
> > - Too many jars, for example, users need to download 4+ jars for Hive 1.x
> > from various places.
> >
> > We have discussed in [1] and [2], but unfortunately, we can not achieve
> an
> > agreement.
> >
> > For improving this, I'd like to introduce few flink-connector-hive-xx
> > modules in flink-connectors, module contains all the dependencies related
> > to hive. And only support lower hive metastore versions:
> > - "flink-connector-hive-1.2" to support hive 1.0.0 - 1.2.2
> > - "flink-connector-hive-2.0" to support hive 2.0.0 - 2.0.1
> > - "flink-connector-hive-2.2" to support hive 2.1.0 - 2.2.0
> > - "flink-connector-hive-2.3" to support hive 2.3.0 - 2.3.6
> > - "flink-connector-hive-3.1" to support hive 3.0.0 - 3.1.2
> >
> > Users can choose one and download to flink/lib. It includes all hive
> > things.
> >
> > I try to use a single module to deploy multiple versions, but I can not
> > find a suitable way, because different modules require different versions
> > and different dependencies.
> >
> > What do you think?
> >
> > [1]
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-have-separate-Flink-distributions-with-built-in-Hive-dependencies-td35918.html
> > [2]
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-109-Improve-Hive-dependencies-out-of-box-experience-td38290.html
> >
> > Best,
> > Jingsong Lee
> >
>


-- 
Best, Jingsong Lee

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

[Yang Wang]

Hi Peter,
Really thanks for your response.

Hi all @Kostas Kloudas  @Zili Chen
 @Peter Huang  @Rong Rong

It seems that we have reached an agreement. The “application mode”
is regarded as the enhanced “per-job”. It is
orthogonal with “cluster deploy”. Currently, we bind the “per-job” to
`run-user-main-on-client` and “application mode”
to `run-user-main-on-cluster`.

Do you have other concerns to moving FLIP-85 to voting?


Best,
Yang

Peter Huang  于2020年3月5日周四 下午12:48写道：

> Hi Yang and Kostas,
>
> Thanks for the clarification. It makes more sense to me if the long term
> goal is to replace per job mode to application mode
>  in the future (at the time that multiple execute can be supported).
> Before that, It will be better to keep the concept of
>  application mode internally. As Yang suggested, User only need to use a
> `-R/-- remote-deploy` cli option to launch
> a per job cluster with the main function executed in cluster
> entry-point.  +1 for the execution plan.
>
>
>
> Best Regards
> Peter Huang
>
>
>
>
> On Tue, Mar 3, 2020 at 7:11 AM Yang Wang  wrote:
>
>> Hi Peter,
>>
>> Having the application mode does not mean we will drop the cluster-deploy
>> option. I just want to share some thoughts about “Application Mode”.
>>
>>
>> 1. The application mode could cover the per-job sematic. Its lifecyle is
>> bound
>> to the user `main()`. And all the jobs in the user main will be executed
>> in a same
>> Flink cluster. In first phase of FLIP-85 implementation, running user
>> main on the
>> cluster side could be supported in application mode.
>>
>> 2. Maybe in the future, we also need to support multiple `execute()` on
>> client side
>> in a same Flink cluster. Then the per-job mode will evolve to application
>> mode.
>>
>> 3. From user perspective, only a `-R/-- remote-deploy` cli option is
>> visible. They
>> are not aware of the application mode.
>>
>> 4. In the first phase, the application mode is working as “per-job”(only
>> one job in
>> the user main). We just leave more potential for the future.
>>
>>
>> I am not against with calling it “cluster deploy mode” if you all think
>> it is clearer for users.
>>
>>
>>
>> Best,
>> Yang
>>
>> Kostas Kloudas  于2020年3月3日周二 下午6:49写道：
>>
>>> Hi Peter,
>>>
>>> I understand your point. This is why I was also a bit torn about the
>>> name and my proposal was a bit aligned with yours (something along the
>>> lines of "cluster deploy" mode).
>>>
>>> But many of the other participants in the discussion suggested the
>>> "Application Mode". I think that the reasoning is that now the user's
>>> Application is more self-contained.
>>> It will be submitted to the cluster and the user can just disconnect.
>>> In addition, as discussed briefly in the doc, in the future there may
>>> be better support for multi-execute applications which will bring us
>>> one step closer to the true "Application Mode". But this is how I
>>> interpreted their arguments, of course they can also express their
>>> thoughts on the topic :)
>>>
>>> Cheers,
>>> Kostas
>>>
>>> On Mon, Mar 2, 2020 at 6:15 PM Peter Huang 
>>> wrote:
>>> >
>>> > Hi Kostas,
>>> >
>>> > Thanks for updating the wiki. We have aligned with the implementations
>>> in the doc. But I feel it is still a little bit confusing of the naming
>>> from a user's perspective. It is well known that Flink support per job
>>> cluster and session cluster. The concept is in the layer of how a job is
>>> managed within Flink. The method introduced util now is a kind of mixing
>>> job and session cluster to promising the implementation complexity. We
>>> probably don't need to label it as Application Model as the same layer of
>>> per job cluster and session cluster. Conceptually, I think it is still a
>>> cluster mode implementation for per job cluster.
>>> >
>>> > To minimize the confusion of users, I think it would be better just an
>>> option of per job cluster for each type of cluster manager. How do you
>>> think?
>>> >
>>> >
>>> > Best Regards
>>> > Peter Huang
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Mon, Mar 2, 2020 at 7:22 AM Kostas Kloudas 
>>> wrote:
>>> >>
>>> >> Hi Yang,
>>> >>
>>> >> The difference between per-job and application mode is that, as you
>>> >> described, in the per-job mode the main is executed on the client
>>> >> while in the application mode, the main is executed on the cluster.
>>> >> I do not think we have to offer "application mode" with running the
>>> >> main on the client side as this is exactly what the per-job mode does
>>> >> currently and, as you described also, it would be redundant.
>>> >>
>>> >> Sorry if this was not clear in the document.
>>> >>
>>> >> Cheers,
>>> >> Kostas
>>> >>
>>> >> On Mon, Mar 2, 2020 at 3:17 PM Yang Wang 
>>> wrote:
>>> >> >
>>> >> > Hi Kostas,
>>> >> >
>>> >> > Thanks a lot for your conclusion and updating the FLIP-85 WIKI.
>>> Currently, i have no more
>>> >> > questions about motivation, approach, fault tolerance and the first
>>> phase implementation.
>>

Re: [DISCUSS] Introduce flink-connector-hive-xx modules

[Bowen Li]

Thanks Jingsong for your explanation! I'm +1 for this initiative.

According to your description, I think it makes sense to incorporate
support of Hive 2.2 to that of 2.0/2.1 and reducing the number of ranges to
4.

A couple minor followup questions:
1) will there be a base module like "flink-connector-hive-base" which holds
all the common logic of these proposed modules and is compiled into the
uber jar of "flink-connector-hive-xxx"?
2) according to my observation, it's more common to set the version in
module name to be the lowest version that this module supports, e.g. for
Hive 1.0.0 - 1.2.2, the module name can be "flink-connector-hive-1.0"
rather than "flink-connector-hive-1.2"


On Wed, Mar 4, 2020 at 10:20 PM Jingsong Li  wrote:

> Thanks Bowen for involving.
>
> > why you proposed segregating hive versions into the 5 ranges above? &
> what different Hive features are supported in the 5 ranges?
>
> For only higher client dependencies version support lower hive metastore
> versions:
> - Hive 1.0.0 - 1.2.2, thrift change is OK, only hive date column stats, we
> can throw exception for the unsupported feature.
> - Hive 2.0 and Hive 2.1, primary key support and alter_partition api
> change.
> - Hive 2.2 no thrift change.
> - Hive 2.3 change many things, lots of thrift change.
> - Hive 3+, not null. unique, timestamp, so many things.
>
> All these things can be found in hive_metastore.thrift.
>
> I think I can try do more effort in implementation to use Hive 2.2 to
> support Hive 2.0. So the range size will be 4.
>
> > have you tested that whether the proposed corresponding Flink module will
> be fully compatible with each Hive version range?
>
> Yes, I have done some tests, not really for "fully", but it is a technical
> judgment.
>
> Best,
> Jingsong Lee
>
> On Thu, Mar 5, 2020 at 1:17 PM Bowen Li  wrote:
>
> > Thanks, Jingsong, for bringing this up. We've received lots of feedbacks
> in
> > the past few months that the complexity involved in different Hive
> versions
> > has been quite painful for users to start with. So it's great to step
> > forward and deal with such issue.
> >
> > Before getting on a decision, can you please explain:
> >
> > 1) why you proposed segregating hive versions into the 5 ranges above?
> > 2) what different Hive features are supported in the 5 ranges?
> > 3) have you tested that whether the proposed corresponding Flink module
> > will be fully compatible with each Hive version range?
> >
> > Thanks,
> > Bowen
> >
> >
> >
> > On Wed, Mar 4, 2020 at 1:00 AM Jingsong Lee 
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to propose introduce flink-connector-hive-xx modules.
> > >
> > > We have documented the dependencies detailed information[2]. But still
> > has
> > > some inconvenient:
> > > - Too many versions, users need to pick one version from 8 versions.
> > > - Too many versions, It's not friendly to our developers either,
> because
> > > there's a problem/exception, we need to look at eight different
> versions
> > of
> > > hive client code, which are often various.
> > > - Too many jars, for example, users need to download 4+ jars for Hive
> 1.x
> > > from various places.
> > >
> > > We have discussed in [1] and [2], but unfortunately, we can not achieve
> > an
> > > agreement.
> > >
> > > For improving this, I'd like to introduce few flink-connector-hive-xx
> > > modules in flink-connectors, module contains all the dependencies
> related
> > > to hive. And only support lower hive metastore versions:
> > > - "flink-connector-hive-1.2" to support hive 1.0.0 - 1.2.2
> > > - "flink-connector-hive-2.0" to support hive 2.0.0 - 2.0.1
> > > - "flink-connector-hive-2.2" to support hive 2.1.0 - 2.2.0
> > > - "flink-connector-hive-2.3" to support hive 2.3.0 - 2.3.6
> > > - "flink-connector-hive-3.1" to support hive 3.0.0 - 3.1.2
> > >
> > > Users can choose one and download to flink/lib. It includes all hive
> > > things.
> > >
> > > I try to use a single module to deploy multiple versions, but I can not
> > > find a suitable way, because different modules require different
> versions
> > > and different dependencies.
> > >
> > > What do you think?
> > >
> > > [1]
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-have-separate-Flink-distributions-with-built-in-Hive-dependencies-td35918.html
> > > [2]
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-109-Improve-Hive-dependencies-out-of-box-experience-td38290.html
> > >
> > > Best,
> > > Jingsong Lee
> > >
> >
>
>
> --
> Best, Jingsong Lee
>

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

[tison]

+1 to star voting.

Best,
tison.


Yang Wang  于2020年3月5日周四 下午2:29写道：

> Hi Peter,
> Really thanks for your response.
>
> Hi all @Kostas Kloudas  @Zili Chen
>  @Peter Huang  @Rong
> Rong 
> It seems that we have reached an agreement. The “application mode”
> is regarded as the enhanced “per-job”. It is
> orthogonal with “cluster deploy”. Currently, we bind the “per-job” to
> `run-user-main-on-client` and “application mode”
> to `run-user-main-on-cluster`.
>
> Do you have other concerns to moving FLIP-85 to voting?
>
>
> Best,
> Yang
>
> Peter Huang  于2020年3月5日周四 下午12:48写道：
>
>> Hi Yang and Kostas,
>>
>> Thanks for the clarification. It makes more sense to me if the long term
>> goal is to replace per job mode to application mode
>>  in the future (at the time that multiple execute can be supported).
>> Before that, It will be better to keep the concept of
>>  application mode internally. As Yang suggested, User only need to use a
>> `-R/-- remote-deploy` cli option to launch
>> a per job cluster with the main function executed in cluster
>> entry-point.  +1 for the execution plan.
>>
>>
>>
>> Best Regards
>> Peter Huang
>>
>>
>>
>>
>> On Tue, Mar 3, 2020 at 7:11 AM Yang Wang  wrote:
>>
>>> Hi Peter,
>>>
>>> Having the application mode does not mean we will drop the cluster-deploy
>>> option. I just want to share some thoughts about “Application Mode”.
>>>
>>>
>>> 1. The application mode could cover the per-job sematic. Its lifecyle is
>>> bound
>>> to the user `main()`. And all the jobs in the user main will be executed
>>> in a same
>>> Flink cluster. In first phase of FLIP-85 implementation, running user
>>> main on the
>>> cluster side could be supported in application mode.
>>>
>>> 2. Maybe in the future, we also need to support multiple `execute()` on
>>> client side
>>> in a same Flink cluster. Then the per-job mode will evolve to
>>> application mode.
>>>
>>> 3. From user perspective, only a `-R/-- remote-deploy` cli option is
>>> visible. They
>>> are not aware of the application mode.
>>>
>>> 4. In the first phase, the application mode is working as “per-job”(only
>>> one job in
>>> the user main). We just leave more potential for the future.
>>>
>>>
>>> I am not against with calling it “cluster deploy mode” if you all think
>>> it is clearer for users.
>>>
>>>
>>>
>>> Best,
>>> Yang
>>>
>>> Kostas Kloudas  于2020年3月3日周二 下午6:49写道：
>>>
 Hi Peter,

 I understand your point. This is why I was also a bit torn about the
 name and my proposal was a bit aligned with yours (something along the
 lines of "cluster deploy" mode).

 But many of the other participants in the discussion suggested the
 "Application Mode". I think that the reasoning is that now the user's
 Application is more self-contained.
 It will be submitted to the cluster and the user can just disconnect.
 In addition, as discussed briefly in the doc, in the future there may
 be better support for multi-execute applications which will bring us
 one step closer to the true "Application Mode". But this is how I
 interpreted their arguments, of course they can also express their
 thoughts on the topic :)

 Cheers,
 Kostas

 On Mon, Mar 2, 2020 at 6:15 PM Peter Huang 
 wrote:
 >
 > Hi Kostas,
 >
 > Thanks for updating the wiki. We have aligned with the
 implementations in the doc. But I feel it is still a little bit confusing
 of the naming from a user's perspective. It is well known that Flink
 support per job cluster and session cluster. The concept is in the layer of
 how a job is managed within Flink. The method introduced util now is a kind
 of mixing job and session cluster to promising the implementation
 complexity. We probably don't need to label it as Application Model as the
 same layer of per job cluster and session cluster. Conceptually, I think it
 is still a cluster mode implementation for per job cluster.
 >
 > To minimize the confusion of users, I think it would be better just
 an option of per job cluster for each type of cluster manager. How do you
 think?
 >
 >
 > Best Regards
 > Peter Huang
 >
 >
 >
 >
 >
 >
 >
 >
 > On Mon, Mar 2, 2020 at 7:22 AM Kostas Kloudas 
 wrote:
 >>
 >> Hi Yang,
 >>
 >> The difference between per-job and application mode is that, as you
 >> described, in the per-job mode the main is executed on the client
 >> while in the application mode, the main is executed on the cluster.
 >> I do not think we have to offer "application mode" with running the
 >> main on the client side as this is exactly what the per-job mode does
 >> currently and, as you described also, it would be redundant.
 >>
 >> Sorry if this was not clear in the document.
 >>
 >> Cheers,
 >> Kostas
 >>
 >> On Mon, Mar 2, 2020 at 3:17 PM Yang Wang 
 wrote:
 >> >
 

Re: [DISCUSS] FLIP-110: Support LIKE clause in CREATE TABLE

[Jark Wu]

Hi Dawid,

> INHERITS creates a new table with a "link" to the original table.
Yes, INHERITS is a "link" to the original table in PostgreSQL.
But INHERITS is not SQL standard, I think it's fine for vendors to define
theire semantics.

> Standard also allows declaring the clause after the schema part. We can
also do it.
Is that true? I didn't find it in SQL standard. If this is true, I prefer
to put LIKE after the schema part.



Hi Jingsong,

The concern you mentioned in (2) is exactly my concern too. That's why I
suggested INHERITS, or put LIKE after schema part.

Best,
Jark

On Thu, 5 Mar 2020 at 12:05, Jingsong Li  wrote:

> Thanks Dawid for starting this discussion.
>
> I like the "LIKE".
>
> 1.For "INHERITS", I think this is a good feature too, yes, ALTER TABLE will
> propagate any changes in column data definitions and check constraints down
> the inheritance hierarchy. A inherits B, A and B share every things, they
> have the same kafka topic. If modify schema of B, this means underlying
> kafka topic schema changed, so I think it is good to modify A too. If this
> for "ConfluentSchemaRegistryCatalog" mention by Jark, I think sometimes
> this is just we want.
> But "LIKE" also very useful for many cases.
>
> 2.For LIKE statement in schema, I know two kinds of like syntax, one is
> MySQL/hive/sqlserver, the other is PostgreSQL. I prefer former:
> - In the FLIP, there is "OVERWRITING OPTIONS", this will overwrite
> properties in "with"? This looks weird, because "LIKE" is in schema, but it
> can affect outside properties.
>
> Best,
> Jingsong Lee
>
> On Wed, Mar 4, 2020 at 2:05 PM Dawid Wysakowicz 
> wrote:
>
> > Hi Jark,
> > I did investigate the INHERITS clause, but it has a semantic that in my
> > opinion we definitely don't want to support. INHERITS creates a new table
> > with a "link" to the original table. Therefore if you e.g change the
> schema
> > of the original table it's also reflected in the child table. It's also
> > possible for tables like A inherits B query them like Select * from only
> A,
> > by default it returns results from both tables. I am pretty sure it's not
> > what we're looking for.
> >
> > PostgreSQL implements both the LIKE clause and INHERITS. I am open for
> > discussion if we should support multiple LIKE statements or not. Standard
> > also allows declaring the clause after the schema part. We can also do
> it.
> > Nevertheless I think including multiple tables might be useful, e.g. when
> > you want to union two tables and output to the same Kafka cluster and
> just
> > change the target topic. I know it's not a very common use case but it's
> > not a big effort to support it.
> >
> > Let me know what you think.
> >
> > Best,
> > Dawid
> >
> > On Wed, 4 Mar 2020, 04:55 Jark Wu,  wrote:
> >
> > > Hi Dawid,
> > >
> > > Thanks for starting this discussion. I like the idea.
> > > Once we support more intergrated catalogs,
> > > e.g. ConfluentSchemaRegistryCatalog, this problem will be more urgent.
> > > Because it's very common to adjust existing tables in catalog slightly.
> > >
> > > My initial thought was introducing INHERITS keyword, which is also
> > > supported in PostgreSQL [1].
> > > This is also similar to the functionality of Hive CREATE TABLE LIKE
> [2].
> > >
> > > CREATE TEMPORARY TABLE MyTable (WATERMARK FOR ts) INHERITS
> > > cat.db.KafkoTopic
> > > CREATE TEMPORARY TABLE MyTable (WATERMARK FOR ts) INHERITS
> > > cat.db.KafkoTopic WITH ('k' = 'v')
> > >
> > > The INHERITS can inherit an existing table with all columns, watermark,
> > and
> > > properties, but the properties and watermark and be overwrited
> > explicitly.
> > >
> > > The reason I prefer INHERITS rather than LIKE is the keyword position.
> We
> > > are copying an existing table definition including the properties.
> > > However, LIKE appears in the schema part, it sounds like copying
> > properties
> > > into schema part of DDL.
> > >
> > > Besides of that, I'm not sure whether the use case stands "merging two
> > > tables into a single one with a different connector".
> > > From my understanding, most use cases are just slightly adjusting on an
> > > existing catalog table with new properties or watermarks.
> > > Do we really need to merge two table definitions into a single one? For
> > > example, is it possible to merge a Kafka table definition and
> > > a Filesystem table definition into a new Kafka table, and the new Kafka
> > > table exactly matches the underlying physical data format?
> > >
> > > Best,
> > > Jark
> > >
> > > [1]: https://www.postgresql.org/docs/9.5/sql-createtable.html
> > > [2]:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTableLike
> > >
> > >
> > > On Tue, 3 Mar 2020 at 21:12, Dawid Wysakowicz 
> > > wrote:
> > >
> > > > Hi devs,
> > > >
> > > > I wanted to bring another improvement proposal up for a discussion.
> > Often
> > > > users need to adjust existing table

Re: [DISCUSS] Introduce flink-connector-hive-xx modules

[Jingsong Li]

Hi Bowen, thanks for your reply.

> will there be a base module like "flink-connector-hive-base" which holds
all the common logic of these proposed modules

Maybe we don't need, their implementation is only "pom.xml". Different
versions have different dependencies.

> it's more common to set the version in module name to be the lowest
version that this module supports

I have some hesitation, because the actual version number can better
reflect the actual dependency. For example, if the user also knows the
field hiveVersion[1]. He may enter the wrong hiveVersion because of the
name, or he may have the wrong expectation for the hive built-in functions.

[1] https://github.com/apache/flink/pull/11304

Best,
Jingsong Lee

On Thu, Mar 5, 2020 at 2:34 PM Bowen Li  wrote:

> Thanks Jingsong for your explanation! I'm +1 for this initiative.
>
> According to your description, I think it makes sense to incorporate
> support of Hive 2.2 to that of 2.0/2.1 and reducing the number of ranges to
> 4.
>
> A couple minor followup questions:
> 1) will there be a base module like "flink-connector-hive-base" which holds
> all the common logic of these proposed modules and is compiled into the
> uber jar of "flink-connector-hive-xxx"?
> 2) according to my observation, it's more common to set the version in
> module name to be the lowest version that this module supports, e.g. for
> Hive 1.0.0 - 1.2.2, the module name can be "flink-connector-hive-1.0"
> rather than "flink-connector-hive-1.2"
>
>
> On Wed, Mar 4, 2020 at 10:20 PM Jingsong Li 
> wrote:
>
> > Thanks Bowen for involving.
> >
> > > why you proposed segregating hive versions into the 5 ranges above? &
> > what different Hive features are supported in the 5 ranges?
> >
> > For only higher client dependencies version support lower hive metastore
> > versions:
> > - Hive 1.0.0 - 1.2.2, thrift change is OK, only hive date column stats,
> we
> > can throw exception for the unsupported feature.
> > - Hive 2.0 and Hive 2.1, primary key support and alter_partition api
> > change.
> > - Hive 2.2 no thrift change.
> > - Hive 2.3 change many things, lots of thrift change.
> > - Hive 3+, not null. unique, timestamp, so many things.
> >
> > All these things can be found in hive_metastore.thrift.
> >
> > I think I can try do more effort in implementation to use Hive 2.2 to
> > support Hive 2.0. So the range size will be 4.
> >
> > > have you tested that whether the proposed corresponding Flink module
> will
> > be fully compatible with each Hive version range?
> >
> > Yes, I have done some tests, not really for "fully", but it is a
> technical
> > judgment.
> >
> > Best,
> > Jingsong Lee
> >
> > On Thu, Mar 5, 2020 at 1:17 PM Bowen Li  wrote:
> >
> > > Thanks, Jingsong, for bringing this up. We've received lots of
> feedbacks
> > in
> > > the past few months that the complexity involved in different Hive
> > versions
> > > has been quite painful for users to start with. So it's great to step
> > > forward and deal with such issue.
> > >
> > > Before getting on a decision, can you please explain:
> > >
> > > 1) why you proposed segregating hive versions into the 5 ranges above?
> > > 2) what different Hive features are supported in the 5 ranges?
> > > 3) have you tested that whether the proposed corresponding Flink module
> > > will be fully compatible with each Hive version range?
> > >
> > > Thanks,
> > > Bowen
> > >
> > >
> > >
> > > On Wed, Mar 4, 2020 at 1:00 AM Jingsong Lee 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to propose introduce flink-connector-hive-xx modules.
> > > >
> > > > We have documented the dependencies detailed information[2]. But
> still
> > > has
> > > > some inconvenient:
> > > > - Too many versions, users need to pick one version from 8 versions.
> > > > - Too many versions, It's not friendly to our developers either,
> > because
> > > > there's a problem/exception, we need to look at eight different
> > versions
> > > of
> > > > hive client code, which are often various.
> > > > - Too many jars, for example, users need to download 4+ jars for Hive
> > 1.x
> > > > from various places.
> > > >
> > > > We have discussed in [1] and [2], but unfortunately, we can not
> achieve
> > > an
> > > > agreement.
> > > >
> > > > For improving this, I'd like to introduce few flink-connector-hive-xx
> > > > modules in flink-connectors, module contains all the dependencies
> > related
> > > > to hive. And only support lower hive metastore versions:
> > > > - "flink-connector-hive-1.2" to support hive 1.0.0 - 1.2.2
> > > > - "flink-connector-hive-2.0" to support hive 2.0.0 - 2.0.1
> > > > - "flink-connector-hive-2.2" to support hive 2.1.0 - 2.2.0
> > > > - "flink-connector-hive-2.3" to support hive 2.3.0 - 2.3.6
> > > > - "flink-connector-hive-3.1" to support hive 3.0.0 - 3.1.2
> > > >
> > > > Users can choose one and download to flink/lib. It includes all hive
> > > > things.
> > > >
> > > > I try to use a single module to deploy multiple versions, bu