[Discuss] FLIP-26 - SSL Mutual Authentication

Hello, Given that some SSL enhancement bugs have been posted lately, I took some time to revise FLIP-26 which explores how to harden both external and internal communication. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80453255 Some recent related issues: - FLINK-9312 - mutu

Re: [Discuss] FLIP-26 - SSL Mutual Authentication

t; - In YARN we could see if can implement the IP filter via such an > > authorizer. > > > > > > *Hostname Verification* > > > > For internal communication, and especially on dynamic environments like > > Kubernetes, it is very hard to work with certificate

Re: SQL Client Limitations

I'd like to better understand how catalogs will work in SQL Client. I assume we'll be able to reference catalog classes from the environment file (e.g. FLINK-9172). Thanks On Tue, Aug 21, 2018 at 4:56 AM Fabian Hueske wrote: > Hi Dominik, > > The SQL Client supports the same subset of SQL tha

Re: [jira] [Created] (FLINK-10225) Cannot access state from a empty taskmanager

Thanks for posting this issue, I also recently saw this. The cause appears to be that the TaskExecutor's queryable state proxy knows only about jobs that are using slot(s) from that executor. Note FLINK-10117 calls for queryable state to be exposed via the REST endpoint, which may indirectly addr

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

Hi folks, there's clearly some incremental steps to be taken to introduce catalog support to SQL Client, complementary to what is proposed in the Flink-Hive Metastore design doc. I was quietly working on this over the holidays. I posted some new sub-tasks, PRs, and sample code to FLINK-10744. W

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

I propose that the community review and merge the PRs that I posted, and then evolve the design thru 1.8 and beyond. I think having a basic infrastructure in place now will accelerate the effort, do you agree? Thanks again! On Wed, Jan 2, 2019 at 11:20 AM Zhang, Xuefu wrote: > Hi Eron, > > Ha

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

ils of the ExternalCatalog interface; the code would not be obsolete if/when the catalog interface is enhanced as per the design doc. On Wed, Jan 2, 2019 at 1:35 PM Eron Wright wrote: > I propose that the community review and merge the PRs that I posted, and > then evolve the design thru 1.8 and b

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

Thu, Jan 3, 2019 at 9:37 AM Eron Wright wrote: > Would a couple folks raise their hand to make a review pass thru the 6 PRs > listed above? It is a lovely stack of PRs that is 'all green' at the > moment. I would be happy to open follow-on PRs to rapidly align with > other

Kerberos for Streaming & Kafka

, the original keytab is needed to re-authenticate. Spark Streaming solved this problem with `--keytab` on spark-submit (see AMDelegationTokenRenewer.scala).Kafka doesn't support delegation tokens yet (see KIP-48 and KAFKA-1696). Thoughts? Thanks! - Eron Wright

RE: Kerberos for Streaming & Kafka

nuous streaming job to survive, the original keytab is needed to re-authenticate. Spark Streaming solved this problem with `--keytab` on spark-submit (see AMDelegationTokenRenewer.scala). 2. Kafka doesn't support delegation tokens yet (see KIP-48 and KAFKA-1696). Thoughts? Thanks! - Eron Wright

RE: Surprising order of events in union of two streams

aming-scala/src/main/scala/org/numenta/nupic/flink/streaming/connectors/river/RiverSource.scala#L157 -Eron Wright > From: aljos...@apache.org > Date: Mon, 18 Apr 2016 08:20:11 + > Subject: Re: Surprising order of events in union of two streams > To: dev@flink.apache.org > CC: nam

flink-git - an experiment in exactly-once semantics

lores providing exactly-once guarantees as both a source and as a sink. Enjoy,Eron Wright

RE: Data locality and scheduler

My understanding of Flink on YARN is that the task managers are eagerly acquired when the session is started, before any job information (which could be mined for locality hints) is available. This makes sense because the session could be reused for other jobs. But this approach leads to poor

Side-effects of DataSet::count

I was curious as to how the `count` method on DataSet worked, and was surprised to see that it executes the entire program graph. Wouldn’t this cause undesirable side-effects like writing to sinks?Also strange that the graph is mutated with the addition of a sink (that isn’t subsequently r

Re: Side-effects of DataSet::count

they also trigger writing to any previously defined sinks. > The idea behind this behavior is to enable interactive querying (the one > that you are used to get from a shell environment) and it is also a great > debugging tool. > > Best, > > Marton > > On Sun, May 29, 2016

[DISCUSS] Mesos version dependency

Hello, For Flink's initial support for Mesos, coming in Flink 1.2, does anyone have an objection to making the minimum requirement be Mesos 1.0+? There are some functional benefits to 1.0 and it would simplify the test effort. ThanksEron Wright

Re: [jira] [Created] (FLINK-4898) Refactor HTTP handlers and Netty server/client

pipeline. Eron > On Oct 24, 2016, at 4:16 PM, Eron Wright (JIRA) wrote: > > Eron Wright created FLINK-4898: > --- > > Summary: Refactor HTTP handlers and Netty server/client > Key: FLINK-4898 >

CheckpointedFunction applicability?

Hello! The `CheckpointedFunction` interface was introduced to provide user functions with a context for partitioned state storage. This works for user functions that are provided directly to the operator (assuming it extends from `AbstractUdfStreamOperator`). However, I see that some ope

Re: CheckpointedFunction applicability?

not the intended design and should be added. Best, Stefan > Am 05.01.2017 um 04:40 schrieb Eron Wright : > > Hello! > > The `CheckpointedFunction` interface was introduced to provide user functions > with a context for partitioned state storage. This works for user fun

[DISCUSS] Watermark propagation with Sink API

at does nothing. Thoughts? Thanks! Eron Wright StreamNative

Re: [DISCUSS] Watermark propagation with Sink API

hasing out SinkFunction, I'd prefer to only support > > SinkWriter. Having a no-op default sounds good to me. > > > > We have some experimental feature for Kafka [1], which pretty much > reflects > > your idea. Here we have an ugly workaround to be able to proces

Re: [DISCUSS] Watermark propagation with Sink API

Update: opened an issue and a PR. https://issues.apache.org/jira/browse/FLINK-22700 https://github.com/apache/flink/pull/15950 On Tue, May 18, 2021 at 10:03 AM Eron Wright wrote: > Thanks Arvid and David for sharing your ideas on this subject. I'm glad > to hear that you're

Re: [DISCUSS] Watermark propagation with Sink API

ate KafkaShuffle to the new API. But > that's something I can do. > > [1] > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/Flink*Improvement*Proposals__;Kys!!LpKI!2IQYKfnjRuBgkNRxnPbJeFvTdhWjpwN0urN3m0yz_6W11H74kY5dMnp6nc7o$ > [cwiki[.]apa

Re: [DISCUSS] Watermark propagation with Sink API

Does anyone have further comment on FLIP-167? https://cwiki.apache.org/confluence/display/FLINK/FLIP-167%3A+Watermarks+for+Sink+API Thanks, Eron On Thu, May 20, 2021 at 5:02 PM Eron Wright wrote: > Filed FLIP-167: Watermarks for Sink API: > > https://cwiki.apache.org/confluence/disp

Re: [DISCUSS] Watermark propagation with Sink API

is also > work >with ingestion time sufficiently? >6. How do exactly once sinks deal with written watermarks in case of >failure? I guess it's the same as normal records. (Either rollback of >transaction or deduplication on resumption) > > Best, > > Arvi

Re: [DISCUSS] Watermark propagation with Sink API

ter > then. (SinkFunction is already on the exception list) > 5.+6. Awesome, I was also sure but wanted to double check. > > Best, > > Arvid > > > On Wed, May 26, 2021 at 7:29 PM Eron Wright .invalid> > wrote: > > > Arvid, > > > > 1. I assume

[VOTE] Watermark propagation with Sink API

k+API Discussion thread: https://lists.apache.org/thread.html/r5194e1cf157d1fd5ba7ca9b567cb01723bd582aa12dda57d25bca37e%40%3Cdev.flink.apache.org%3E Implementation Issue: https://issues.apache.org/jira/browse/FLINK-22700 Thanks, Eron Wright StreamNative

Re: [DISCUSS] Watermark propagation with Sink API

he.org/confluence/pages/viewpage.action?pageId=120731026#FlinkBylaws-Voting > > [2] > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Deprecating-Mesos-support-td50142.html > > > > On Fri, May 28, 2021 at 8:17 PM Eron Wright > .inval

Re: [DISCUSS] Watermark propagation with Sink API

m pipelines, which may not be what the user > intended. > On the other hand, I don't immediately see a technical reason why the > downstream source wouldn't be able to infer that. > > > On Thu, Jun 3, 2021 at 9:14 PM Eron Wright .invalid> > wrote: > > > Th

Re: [DISCUSS] Watermark propagation with Sink API

amStatus in such a scenario > would add unnecessary false late events. > > In my understanding a StreamStatus makes sense only when talking about > immediately consumed transient channels such as between operators within > a single job. > > Best, > > Dawid > > On 03/06/2021 23:3

Re: [DISCUSS] Watermark propagation with Sink API

afka, are also > >> used > >> > to connect one streaming system with another one, also for an > immediate > >> > consumption. > >> > > >> > You could say the same thing about watermarks (note they are usually > >> > generated in Fli

Re: [VOTE] Watermark propagation with Sink API

ednesday, June 2, 2021 15:44 > > > > To: dev > > > > Subject: Re: [VOTE] Watermark propagation with Sink API > > > > > > > > > > > > [EXTERNAL EMAIL] > > > > > > > > +1 (binding) > > > > > > > &g

Re: [DISCUSS] Watermark propagation with Sink API

gt; With a wall clock, you always run into the issues that I describe since you > are effectively mixing event time and processing time... > > > On Fri, Jun 4, 2021 at 6:28 PM Eron Wright .invalid> > wrote: > > > Dawid, I think you're mischaracterizing the idleness s

Re: [DISCUSS] Watermark propagation with Sink API

bly somewhat safe. We know why no data is coming (no partition) > and as long as we do not have dynamic partition assignment, there will > never be a switch to active without restart (for the foreseeable future). > > On Fri, Jun 4, 2021 at 10:34 PM Eron Wright .invalid> >

Re: [DISCUSS] Watermark propagation with Sink API

fied the > semantics of idleness. We could also cut the scope of the FLIP to exclude > idleness and go ahead without it (there should be enough binding votes > already). > > On Sat, Jun 5, 2021 at 12:09 AM Eron Wright .invalid> > wrote: > > > I understand your scenario

Re: [DISCUSS] Watermark propagation with Sink API

> Exposing idleness could be part of this next/future FLIP that would > define > > idleness in the first place. > > 2. Block FLIP-167, until the idleness is fixed. > > > > I would vote for option number 1. > > > > Piotrek > > > > pon., 7 cze 202

Re: [VOTE] Watermark propagation with Sink API

Voting is re-open for FLIP-167 as-is (without idleness support as was the point of contention). On Fri, Jun 4, 2021 at 10:45 AM Eron Wright wrote: > Little update on this, more good discussion over the last few days, and > the FLIP will probably be amended to incorporate idleness. Votin

Re: [DISCUSS] Definition of idle partitions

It seems to me that idleness was introduced to deal with a very specific issue. In the pipeline, watermarks are aggregated not on a per-split basis but on a per-subtask basis. This works well when each subtask has exactly one split. When a sub-task has multiple splits, various complications occu

Re: [VOTE] Watermark propagation with Sink API

; > > > > > > > Piotr Nowojski 于2021年6月9日周三 下午10:04写道： > > > > > > > > > Thanks for driving this Eron, and sorry for causing the delay. > > > > > > > > > > +1 (binding) from my side > > > > > > > > > > Piotrek >

Re: [DISCUSS] Definition of idle partitions

gt;> > > >> On Wed, Jun 9, 2021 at 7:31 AM Piotr Nowojski < > piotr.nowoj...@gmail.com > > > > > >> wrote: > > >> > > >> > Hi Eron, > > >> > > > >> > Can you elaborate a bit more what do you mean? I don’t

Re: [DISCUSS] Definition of idle partitions

I quickly updated the draft PR that would propagate idleness information to the Sink function, based on the recent improvement provided by FLINK-18934. For illustration purposes. https://github.com/streamnative/flink/pull/2 On Thu, Jun 10, 2021 at 11:34 AM Eron Wright wrote: > Regard

PR: "Propagate watermarks to sink API"

Would someone be willing and able to review the PR which adds watermarks to the sink API? https://github.com/apache/flink/pull/15950 Thanks! Eron

Re: [DISCUSS] FLIP-182: Watermark alignment

The notion of per-split watermarks seems quite interesting. I think the idleness feature could benefit from a per-split approach too, because idleness is typically related to whether any splits are assigned to a given operator instance. On Mon, Jul 12, 2021 at 3:06 AM 刘建刚 wrote: > +1 for the s

Re: [DISCUSS] FLIP-180: Adjust StreamStatus and Idleness definition

This proposal to narrow the definition of idleness to focus on the event-time clock is great. Please mention that the "temporary status toggle" code will be removed. I agree with adding the markActive() functionality, for symmetry. Speaking of symmetry, could we now include the minor enhancement

Re: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage

Jingsong, regarding the LogStore abstraction, I understand that you want to retain some flexibility as the implementation evolves. It makes sense that the abstract interfaces would be @Internal for now. Would you kindly ensure the minimal extensibility is in place, so that the Pulsar dev communit

Mesos component status for 1.3

Hello! Here's a list of related PRs that I'm tracking for 1.3 release: - FLINK-5974 - Mesos-DNS hostname support - FLINK-6336 - Mesos placement constraints - FLINK-5975 - Mesos volume support We need a committer to push these over the finish line. Also, I'd like to nominate myself as a shepherd

Re: [DISCUSS] Feature Freeze

Robert, I'd like to see FLINK-5974 (Mesos DNS support) added to the list of important issues. A PR is ready. On Tue, May 2, 2017 at 4:30 AM, Kostas Kloudas wrote: > The only thing that I want to add in the features to be added for 1.3 > is the

Re: [DISCUSS] Changing Flink's shading model

In my opinion, the ideal approach to mitigating conflicts between application code and Flink itself is to relocate all of Flink's dependencies. Rationale is to avoid placing the burden of relocation on the app developer, and ultimately to eliminate the need for an app uber-jar. For example, imagi

Re: [DISCUSS] Release 1.3.0 RC1 (Non voting, testing release candidate)

Robert, please add FLINK-6606 to the list of JIRAs that you're tracking, thanks. On Tue, May 16, 2017 at 8:30 AM, Robert Metzger wrote: > I totally forgot to post a document with testing tasks in the RC0 thread, > so I'll do it in the RC1 thread. > > Please use this document: > https://docs.goog

Re: [ANNOUNCE] Apache Flink 1.3.0 released

Opened a PR for Homebrew update to `apache-flink` formula: https://github.com/Homebrew/homebrew-core/pull/14151 On Thu, Jun 1, 2017 at 7:48 AM, Robert Metzger wrote: > The Apache Flink community is pleased to announce the release of Apache > Flink 1.3.0. > > Apache Flink® is an open-source strea

Re: Using native library in Flink

The solution mentioned by Timo works well with a standalone Flink cluster but might not work with a YARN or Mesos cluster. An alternative is to have your Java library contain the native library within itself, and to extract it to a temporary directory before calling `System.loadLibrary(...)`. Note

Re: Dropping Java 7 support

Opened FLINK-7242 as an umbrella issue for this. On Wed, Jul 19, 2017 at 3:09 AM, Chesnay Schepler wrote: > Are the specific things we want to change right away? (build profiles > would be one thing) > > Would be neat to collect them in an umbrella issue. > > > On 18.07.2017 16:49, Timo Walther

Re: Dropping Java 7 support

I don't see a ticket for updating Akka to 2.4+, or I'd link it to FLINK-7242. Should I open an umbrella for that? Separate question is whether we'd shoot for 2.4 or 2.5. On Fri, Jul 21, 2017 at 5:22 PM, Eron Wright wrote: > Opened FLINK-7242 as an umbrella issue for this.

[DISCUSS] Service Authorization (redux)

Hello, now might be a good time to revisit an important enhancement to Flink security, so-called service authorization. This means the hardening of a Flink cluster against unauthorized use with some sort of authentication and authorization scheme. Today, Flink relies entirely on network isolati

Re: [DISCUSS] Planning Release 1.4

Aside from the release management aspect, shall we discuss the major themes of 1.4? On Thu, Jun 1, 2017 at 8:26 AM, Robert Metzger wrote: > Hi all, > > Flink 1.2 was released on February 2, Flink 1.3 on June 1, which means > we've managed to release Flink 1.3 in almost exactly 4 months! > > For

Re: [DISCUSS] A more thorough Pull Request check list and template

This seems like a good step in establishing a better PR process. I believe the process could be improved to ensure timely and targeted review by component experts and committers. On Mon, Jul 17, 2017 at 9:36 AM, Stephan Ewen wrote: > Hi all! > > I have reflected a bit on the pull requests and o

Re: [DISCUSS] A more thorough Pull Request check list and template

o bring in a good piece of perspective, at > least, even if the code is not immediately suited for contribution... > > > On Mon, Jul 24, 2017 at 8:18 PM, Eron Wright wrote: > > > This seems like a good step in establishing a better PR process. I > believe > > the p

Re: Class Cache

A Flink program is typically packaged as an 'uber-jar' containing its dependencies. The Flink quickstart project illustrates this (see the use of the shading plugin in pom.xml). Based on your description, the classes of mylib2.jar were copied into mylib1.jar when the latter was built. Try rebui

Re: [DISCUSS] Service Authorization (redux)

I think after Dropping support for Java 7 we will move to Akka 2.4+, so > we > > should be good there. I think quite some users should find a (more) > secure > > Flink interesting. > > > > Best, > > Aljoscha > > > On 24. Jul 2017, at 03:11, Eron Wr

Re: [DISCUSS] Service Authorization (redux)

r Scala 2.10. On the other hand > > people are already using Flink without this feature. > > > > Cheers, > > Till > > > > On Wed, Aug 2, 2017 at 7:21 PM, Eron Wright > wrote: > > > >> Thanks Till and Aljoscha for the feedback. > >> >

Re: [DISCUSS] Flink 1.4 and time based release

I definitely think the conversation must happen in the context of the expected feature-set.JIRA is our friend here. I like the idea of using the "Fix Version" to identify the set from which the leadership can formulate an estimate. I would suggest that folks update their tasks accordingly.

[DISCUSS] Watermark boundary condition

Hello, I think I see a bug in a few places related to determining whether an input is to be considered late. Some components use the logic that (timestamp <= watermark) is considered late. Others use (timestamp < watermark). I think the former is correct according to the definition in Watermark

Re: Using Maven Archetype Plugin 3.0+ to create flink-quickstart-java project

Michael, an alternative is to start your project with the released version of Flink (1.3) rather than 1.4-SNAPSHOT: https://ci.apache.org/projects/flink/flink-docs-release-1.3/quickstart/java_api_quickstart.html#create-project Afterwards you could update your dependencies (and add the Apache snaps

Re: Run apache flink from intellij or eclipse

When you run in the IDE, an embedded ("local") Flink environment is created. It doesn't make use of your Flink cluster as "flink run" would. I believe you can activate the Web UI by using the following API: StreamExecutionEnvironment::createLocalEnvironmentWithWebUI

Re: [DISCUSS] Service Authorization (redux)

hether this feature would be useful to you. Thanks! Eron On Thu, Aug 3, 2017 at 11:11 AM, Eron Wright wrote: > Till, with (c) are you suggesting that we'd use Akka 2.3 for Scala 2.10 > and Akka 2.4+ for Scala 2.11+? Sounds reasonable but I don't know how > feasible it is. I w

Re: [DISCUSS] Releasing Flink 1.4

+1 from our side on this plan. On Mon, Oct 16, 2017 at 3:33 AM, Fabian Hueske wrote: > OK, sounds good to me. > > We have a couple of bugs to fix for the Table API / SQL but have PRs for > most of them. > > There's only one major issue that I'd like to include in 1.4.0 which is a > refactoring o

[DISCUSS] Service Authorization (SSL client authentication)

e have any objections to the technical plan? Thanks! Eron Wright Dell EMC

Re: [VOTE] Release 1.4.0, release candidate #2

On Dell EMC side, we're testing the RC2 on DCOS 1.10.0. Seeing a potential issue with offer acceptance and we'll update the thread with a +1 or with a more concrete issue within 24 hours. Thanks, Eron On Wed, Nov 29, 2017 at 6:54 AM, Chesnay Schepler wrote: > I don't think anyone has taken a

Re: [VOTE] Release 1.4.0, release candidate #2

Unfortunately we've identified a blocker bug for Flink on Mesos - FLINK-8174.We'll have a patch ready on Thursday. Thanks, Eron On Wed, Nov 29, 2017 at 3:40 PM, Eron Wright wrote: > On Dell EMC side, we're testing the RC2 on DCOS 1.10.0. Seeing a > potential issue

Re: [VOTE] Release 1.4.0, release candidate #2

e Hadoop 2.9.0 build problem [1]. > > Best, Fabian > > [1] https://issues.apache.org/jira/browse/FLINK-8177 > > 2017-11-30 4:35 GMT+01:00 Eron Wright : > > > Unfortunately we've identified a blocker bug for Flink on Mesos - > > FLINK-8174. We'll have a pa

Re: [VOTE] Release 1.4.0, release candidate #2

resources. 3 is a moon shot. Striving for 2. Fallback on 1. On Fri, Dec 1, 2017 at 2:10 PM, Aljoscha Krettek wrote: > Thanks for the update! > > Just to be clear, you're proposing going forward with the "simple fix" of > reverting FLINK-7294? > > > On 1. De

Re: [VOTE] Release 1.4.0, release candidate #2

Update: PR for "FLINK-8174 - Mesos RM unable to accept offers for unreserved resources" is ready for review. We were able to solve (3), supporting any combination of reserved and unreserved resources. On Fri, Dec 1, 2017 at 2:24 PM, Eron Wright wrote: > There are three levels

Re: [VOTE] Release 1.4.0, release candidate #3

Just discovered: the removal of Flink's Future (FLINK-7252) causes a breaking change in connectors that use `org.apache.flink.runtime.checkpoint.MasterTriggerRestoreHook`, because `Future` is a type on one of the methods. To my knowledge, this affects only the Pravega connector. Curious to know

Re: TaskManager job lifecycle hooks

Could you speak to whether the lifecycle provided by RichFunction (open/close) would fit the requirement? https://ci.apache.org/projects/flink/flink-docs-release-1.3/api/java/org/apache/flink/api/common/functions/RichFunction.html#open-org.apache.flink.configuration.Configuration- On Thu, Dec 7,

Re: [DISCUSS] Service Authorization (SSL client authentication)

Folks, what is the next step to formally submit a FLIP?i.e. assign a number, drive to 'accepted' state? Given the introduction of new RPC and REST endpoints with FLIP-6, now is a good time to agree on the approach to securing Flink. Thanks! Eron On Mon, Nov 27, 2017 at 11:52 AM, E

Re: Time-stamp of output from a window operator

Each event in the stream has an associated timestamp as metadata. A timestamp assigner simply extracts the timestamp from the user object for that purpose. There is no per-operator watermark, but with `assignTimestampsAndWatermarks` you may insert an operator that overrides upstream watermarks.

Re: Flink-Yarn-Kerberos integration

I agree that it is reasonable to use Hadoop DTs as you describe. That approach is even recommended in YARN's documentation (see Securing Long-lived YARN Services on the YARN Application Security page). But one of the goals of Kerberos integration is to support Kerberized data access for connecto

Re: Time-stamp of output from a window operator

That is correct, the window operator happily accepts late elements into the applicable window(s). Though the element is late (with respect to the current watermark) it might still be included in the main firing, or provoke a late firing. Elements occurring downstream of an operator that uses late

Re: Flink-Yarn-Kerberos integration

ave flink jobs to load it > at > > runtime. > > > > Please let me know your comments. Thanks a lot. > > > > On Fri, Dec 22, 2017 at 3:05 PM, Eron Wright > wrote: > > > >> I agree that it is reasonable to use Hadoop DTs as you describe. That > >>

Re: Timestamp/watermark support in Kinesis consumer

I'd like to know how you envision dealing with resharding in relation to the watermark state. Imagine that a given shard S1 has a watermark of T1, and is then split into two shards S2 and S3. The new shards are assigned to subtasks according to a hash function. The current watermarks of those

Re: Timestamp/watermark support in Kinesis consumer

> working on though. We face issues with restart from older checkpoints where > parent and child shards are consumed in parallel. > > > -- > sent from mobile > > > On Feb 12, 2018 4:36 PM, "Eron Wright" wrote: > > I'd like to know how you envision de

Re: FLIP-6 / Kubernetes

Till, is it possible to package a Flink application as a self-contained deployment on Kubernetes? I mean, run a Flink application using 'flink run' such that it launches its own RM/JM and waits for a sufficient # of TMs to join? Thanks! On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann wrote: >

Re: FLIP-6 / Kubernetes

d not make it for 1.5, but should come > very > > > soon after. > > > > > > @Thomas An implementation of a ResourceManager for k8s should come in > the > > > near future. Would be happy to jump on a joint FLIP, after the 1.5 > > release > >

Re: [DISCUSS] Flink security improvements

Please accept my apologies also. On Mon, Mar 19, 2018 at 2:46 AM, Till Rohrmann wrote: > Hi Shuyi, > > sorry for the unresponsiveness on your proposal. Currently, the community > is strongly focused on fixing and testing Flink 1.5 so that we can release > it soon. My gut feeling is that the comm

[Discuss] FLINK-8533 MasterTriggerRestoreHook state initialization

Hello, We see an issue with the use of `MasterTriggerRestoreHook` to synchronize the state of a source function with that of an external system. I'd like the fix to be considered for 1.5. There's a patch ready: https://github.com/apache/flink/pull/5427 Thanks!

[jira] [Created] (FLINK-11234) ExternalTableCatalogBuilder unable to build a batch-only table

Eron Wright created FLINK-11234: Summary: ExternalTableCatalogBuilder unable to build a batch-only table Key: FLINK-11234 URL: https://issues.apache.org/jira/browse/FLINK-11234 Project: Flink

[jira] [Created] (FLINK-11237) Enhance LocalExecutor to wrap TableEnvironment w/ user classloader

Eron Wright created FLINK-11237: Summary: Enhance LocalExecutor to wrap TableEnvironment w/ user classloader Key: FLINK-11237 URL: https://issues.apache.org/jira/browse/FLINK-11237 Project: Flink

[jira] [Created] (FLINK-11238) Enhance SQL-Client to recursively list tables

Eron Wright created FLINK-11238: Summary: Enhance SQL-Client to recursively list tables Key: FLINK-11238 URL: https://issues.apache.org/jira/browse/FLINK-11238 Project: Flink Issue Type

[jira] [Created] (FLINK-11239) Enhance SQL-Client to recursively list UDFs

Eron Wright created FLINK-11239: Summary: Enhance SQL-Client to recursively list UDFs Key: FLINK-11239 URL: https://issues.apache.org/jira/browse/FLINK-11239 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-11240) Implement external catalog factory and descriptor

Eron Wright created FLINK-11240: Summary: Implement external catalog factory and descriptor Key: FLINK-11240 URL: https://issues.apache.org/jira/browse/FLINK-11240 Project: Flink Issue Type

[jira] [Created] (FLINK-11241) Enhance TableEnvironment to connect to a catalog via a descriptor

Eron Wright created FLINK-11241: Summary: Enhance TableEnvironment to connect to a catalog via a descriptor Key: FLINK-11241 URL: https://issues.apache.org/jira/browse/FLINK-11241 Project: Flink

[jira] [Created] (FLINK-11245) Update documentation for catalogs in SQL-Client

Eron Wright created FLINK-11245: Summary: Update documentation for catalogs in SQL-Client Key: FLINK-11245 URL: https://issues.apache.org/jira/browse/FLINK-11245 Project: Flink Issue Type

[jira] [Created] (FLINK-11247) Fix DESCRIBE command to support catalog tables

Eron Wright created FLINK-11247: Summary: Fix DESCRIBE command to support catalog tables Key: FLINK-11247 URL: https://issues.apache.org/jira/browse/FLINK-11247 Project: Flink Issue Type

[jira] [Created] (FLINK-3883) Rename flink clients for inclusion on system path

Eron Wright created FLINK-3883: --- Summary: Rename flink clients for inclusion on system path Key: FLINK-3883 URL: https://issues.apache.org/jira/browse/FLINK-3883 Project: Flink Issue Type

[jira] [Created] (FLINK-3903) Homebrew Installation

Eron Wright created FLINK-3903: --- Summary: Homebrew Installation Key: FLINK-3903 URL: https://issues.apache.org/jira/browse/FLINK-3903 Project: Flink Issue Type: Task Components

[jira] [Created] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

Eron Wright created FLINK-3929: --- Summary: Support for Kerberos Authentication with Keytab Credential Key: FLINK-3929 URL: https://issues.apache.org/jira/browse/FLINK-3929 Project: Flink Issue

[jira] [Created] (FLINK-3930) Implement Service-Level Authorization

Eron Wright created FLINK-3930: --- Summary: Implement Service-Level Authorization Key: FLINK-3930 URL: https://issues.apache.org/jira/browse/FLINK-3930 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-3931) Implement Transport Encryption (SSL/TLS)

Eron Wright created FLINK-3931: --- Summary: Implement Transport Encryption (SSL/TLS) Key: FLINK-3931 URL: https://issues.apache.org/jira/browse/FLINK-3931 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-3932) Implement State Backend Security

Eron Wright created FLINK-3932: --- Summary: Implement State Backend Security Key: FLINK-3932 URL: https://issues.apache.org/jira/browse/FLINK-3932 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-4897) Implement Dispatcher

Eron Wright created FLINK-4897: --- Summary: Implement Dispatcher Key: FLINK-4897 URL: https://issues.apache.org/jira/browse/FLINK-4897 Project: Flink Issue Type: New Feature

  1   2   >