[jira] [Created] (FLINK-22329) Missing crendentials in jobconf causes repeated authentication in Hive datasources
Junfan Zhang created FLINK-22329: Summary: Missing crendentials in jobconf causes repeated authentication in Hive datasources Key: FLINK-22329 URL: https://issues.apache.org/jira/browse/FLINK-22329 Project: Flink Issue Type: Bug Components: Connectors / Hive Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS]FLIP-150: Introduce Hybrid Source
Thanks, Thomas! @Becket and @Nicholas - would you be ok with that approach? On Thu, Apr 15, 2021 at 6:30 PM Thomas Weise wrote: > Hi Stephan, > > Thanks for the feedback! > > I agree with the approach of starting with a simple implementation > that can address a well understood, significant portion of use cases. > > I'm planning to continue work on the prototype that I had shared. > There is production level usage waiting for it fairly soon. I expect > to open a PR in the coming weeks. > > Thomas > > > > > > On Tue, Apr 13, 2021 at 12:15 PM Stephan Ewen wrote: > > > > Thanks all for this discussion. Looks like there are lots of ideas and > > folks that are eager to do things, so let's see how we can get this > moving. > > > > My take on this is the following: > > > > There will probably not be one Hybrid source, but possibly multiple ones, > > because of different strategies/requirements. > > - One may be very simple, with switching points known up-front. Would > > be good to have this in a very simple implementation. > > - There may be one where the switch is dynamic and the readers need > to > > report back where they left off. > > - There may be one that switches back and forth multiple times > during a > > job, for example Kakfa going to DFS whenever it falls behind retention, > in > > order to catch up again. > > > > This also seems hard to "design on paper"; I expect there are nuances in > a > > production setup that affect some details of the design. So I'd feel most > > comfortable in adding a variant of the hybrid source to Flink that has > been > > used already in a real use case (not necessarily in production, but maybe > > in a testing/staging environment, so it seems to meet all requirements). > > > > > > What do you think about the following approach? > > - If there is a tested PoC, let's try to get it contributed to Flink > > without trying to make it much more general. > > - When we see similar but a bit different requirements for another > hybrid > > source, then let's try to evolve the contributed one. > > - If we see new requirements that are so different that they don't fit > > well with the existing hybrid source, then let us look at building a > second > > hybrid source for those requirements. > > > > We need to make connector contributions in general more easy, and I think > > it is not a bad thing to end up with different approaches and see how > these > > play out against each other when being used by users. For example > switching > > with known boundaries, dynamic switching, back-and-forth-switching, etc. > > (I know some committers are planning to do some work on making > > connector contributions easier, with standardized testing frameworks, > > decoupled CI, etc.) > > > > Best, > > Stephan > > > > > > On Thu, Mar 25, 2021 at 4:41 AM Thomas Weise wrote: > > > > > Hi, > > > > > > As mentioned in my previous email, I had been working on a prototype > for > > > the hybrid source. > > > > > > You can find it at https://github.com/tweise/flink/pull/1 > > > > > > It contains: > > > * Switching with configurable chain of sources > > > * Fixed or dynamic start positions > > > * Test with MockSource and FileSource > > > > > > The purpose of the above PR is to gather feedback and help drive > consensus > > > on the FLIP. > > > > > > * How to support a dynamic start position within the source chain? > > > > > > Relevant in those (few?) cases where start positions are not known > upfront. > > > You can find an example of what that might look like in the tests: > > > > > > > > > > https://github.com/tweise/flink/pull/1/files#diff-8eda4e21a8a53b70c46d30ceecfbfd8ffdb11c14580ca048fa4210564f63ada3R62 > > > > > > > https://github.com/tweise/flink/pull/1/files#diff-3a5658515bb213f9a66db88d45a85971d8c68f64cdc52807622acf27fa703255R132 > > > > > > When switching, the enumerator of the previous source needs to > > > supply information about consumed splits that allows to set the start > > > position for the next source. That could be something like the last > > > processed file, timestamp, etc. (Currently StaticFileSplitEnumerator > > > doesn't track finished splits.) > > > > > > See previous discussion regarding start/end position. The prototype > shows > > > the use of checkpoint state with converter function. > > > > > > * Should readers be deployed dynamically? > > > > > > The prototype assumes a static source chain that is fixed at job > submission > > > time. Conceivably there could be use cases that require more > flexibility. > > > Such as switching one KafkaSource for another. A step in that direction > > > would be to deploy the actual readers dynamically, at the time of > switching > > > source. > > > > > > Looking forward to feedback and suggestions for next steps! > > > > > > Thomas > > > > > > On Sun, Mar 14, 2021 at 11:17 AM Thomas Weise wrote: > > > > > > > Hi Nicholas, > > > > > > > > Thanks for the reply. I had implemented a small PoC. It switches a > > > >
[jira] [Created] (FLINK-22330) Bumps checkstyle, junit, log4j version for security
Brandon Jiang created FLINK-22330: - Summary: Bumps checkstyle, junit, log4j version for security Key: FLINK-22330 URL: https://issues.apache.org/jira/browse/FLINK-22330 Project: Flink Issue Type: Improvement Affects Versions: 1.12.2 Reporter: Brandon Jiang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22331) CLI Frontend (RestClient) doesn't work on Apple M1
Robert Metzger created FLINK-22331:
--
Summary: CLI Frontend (RestClient) doesn't work on Apple M1
Key: FLINK-22331
URL: https://issues.apache.org/jira/browse/FLINK-22331
Project: Flink
Issue Type: Bug
Components: Client / Job Submission
Affects Versions: 1.12.2, 1.13.0
Reporter: Robert Metzger
Attachments: flink-muthmann-client-KlemensMac.local (1).log_native,
flink-muthmann-client-KlemensMac.local.log_rosetta
This issue was first reported by a user:
https://lists.apache.org/thread.html/r50bda40a69688de52c9d6e3489ac2641491387c20fdc1cecedceee76%40%3Cuser.flink.apache.org%3E
See attached logs.
Exception without rosetta:
{code}
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: Failed to execute job 'Streaming WordCount'.
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:366)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219)
at
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
at
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Caused by: org.apache.flink.util.FlinkException: Failed to execute job
'Streaming WordCount'.
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1918)
at
org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:135)
at
org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76)
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1782)
at
org.apache.flink.streaming.examples.wordcount.WordCount.main(WordCount.java:97)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349)
... 8 more
Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to
submit JobGraph.
at
org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$7(RestClusterClient.java:400)
at
java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
at
java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
at
org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:390)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
at
org.apache.flink.runtime.rest.RestClient$ClientHandler.exceptionCaught(RestClient.java:613)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)
at
org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:424)
at
org.apache.flink.shaded.netty4.io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:
[jira] [Created] (FLINK-22332) ConnectedComponentsWithObjectMapITCase.testJobWithoutObjectReuse due to NPE when calling "notifyDataAvailable"
Guowei Ma created FLINK-22332: - Summary: ConnectedComponentsWithObjectMapITCase.testJobWithoutObjectReuse due to NPE when calling "notifyDataAvailable" Key: FLINK-22332 URL: https://issues.apache.org/jira/browse/FLINK-22332 Project: Flink Issue Type: Bug Components: Runtime / Network Affects Versions: 1.11.3 Reporter: Guowei Ma https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16711&view=logs&j=219e462f-e75e-506c-3671-5017d866ccf6&t=94b2a454-a9e3-5226-421d-758b172639ef&l=4476 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22333) Elasticsearch7DynamicSinkITCase.testWritingDocuments fail due to deploy task timeout.
Guowei Ma created FLINK-22333:
-
Summary: Elasticsearch7DynamicSinkITCase.testWritingDocuments fail
due to deploy task timeout.
Key: FLINK-22333
URL: https://issues.apache.org/jira/browse/FLINK-22333
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.13.0
Reporter: Guowei Ma
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16694&view=logs&j=d44f43ce-542c-597d-bf94-b0718c71e5e8&t=03dca39c-73e8-5aaf-601d-328ae5c35f20&l=12329
{code:java}
2021-04-16T23:37:23.5719280Z Apr 16 23:37:23
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2021-04-16T23:37:23.5739250Z Apr 16 23:37:23at
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2021-04-16T23:37:23.5759329Z Apr 16 23:37:23at
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
2021-04-16T23:37:23.5779145Z Apr 16 23:37:23at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
2021-04-16T23:37:23.5799204Z Apr 16 23:37:23at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
2021-04-16T23:37:23.5819302Z Apr 16 23:37:23at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-04-16T23:37:23.5839106Z Apr 16 23:37:23at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-04-16T23:37:23.5859276Z Apr 16 23:37:23at
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
2021-04-16T23:37:23.5868964Z Apr 16 23:37:23at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-04-16T23:37:23.5869925Z Apr 16 23:37:23at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2021-04-16T23:37:23.5919839Z Apr 16 23:37:23at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-04-16T23:37:23.5959562Z Apr 16 23:37:23at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-04-16T23:37:23.5989732Z Apr 16 23:37:23at
org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1081)
2021-04-16T23:37:23.6019422Z Apr 16 23:37:23at
akka.dispatch.OnComplete.internal(Future.scala:264)
2021-04-16T23:37:23.6039067Z Apr 16 23:37:23at
akka.dispatch.OnComplete.internal(Future.scala:261)
2021-04-16T23:37:23.6060126Z Apr 16 23:37:23at
akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
2021-04-16T23:37:23.6089258Z Apr 16 23:37:23at
akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
2021-04-16T23:37:23.6119150Z Apr 16 23:37:23at
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-04-16T23:37:23.6139149Z Apr 16 23:37:23at
org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
2021-04-16T23:37:23.6159077Z Apr 16 23:37:23at
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
2021-04-16T23:37:23.6189432Z Apr 16 23:37:23at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
2021-04-16T23:37:23.6215243Z Apr 16 23:37:23at
akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
2021-04-16T23:37:23.6219148Z Apr 16 23:37:23at
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
2021-04-16T23:37:23.6220221Z Apr 16 23:37:23at
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
2021-04-16T23:37:23.6249411Z Apr 16 23:37:23at
scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
2021-04-16T23:37:23.6259145Z Apr 16 23:37:23at
scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
2021-04-16T23:37:23.6289272Z Apr 16 23:37:23at
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-04-16T23:37:23.6309243Z Apr 16 23:37:23at
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
2021-04-16T23:37:23.6359306Z Apr 16 23:37:23at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
2021-04-16T23:37:23.6369399Z Apr 16 23:37:23at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
2021-04-16T23:37:23.6389444Z Apr 16 23:37:23at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
2021-04-16T23:37:23.6429180Z Apr 16 23:37:23at
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
2021-04-16T23:37:23.6449179Z Apr 16 23:37:23at
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
2021-04-16T23:37:23.6479350Z Apr 16 23:37:23
