[jira] [Created] (FLINK-8385) Fix exceptions in AbstractEventTimeWindowCheckpointingITCase
Stefan Richter created FLINK-8385:
-
Summary: Fix exceptions in
AbstractEventTimeWindowCheckpointingITCase
Key: FLINK-8385
URL: https://issues.apache.org/jira/browse/FLINK-8385
Project: Flink
Issue Type: Bug
Components: State Backends, Checkpointing
Affects Versions: 1.4.0
Reporter: Stefan Richter
Assignee: Stefan Richter
Fix For: 1.5.0
When running all `AbstractEventTimeWindowCheckpointingITCase` with debug
logging, I noticed two types of logged exceptions that should not happen:
1)
{code}
java.util.concurrent.RejectedExecutionException: Task
org.apache.flink.runtime.state.SharedStateRegistry$AsyncDisposalRunnable@49258e6c
rejected from java.util.concurrent.ThreadPoolExecutor@47e27938[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 24]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at
org.apache.flink.runtime.state.SharedStateRegistry.scheduleAsyncDelete(SharedStateRegistry.java:197)
at
org.apache.flink.runtime.state.SharedStateRegistry.unregisterReference(SharedStateRegistry.java:162)
at
org.apache.flink.runtime.state.IncrementalKeyedStateHandle.discardState(IncrementalKeyedStateHandle.java:180)
at
org.apache.flink.util.LambdaUtil.applyToAllWhileSuppressingExceptions(LambdaUtil.java:51)
at
org.apache.flink.runtime.state.StateUtil.bestEffortDiscardAllStateObjects(StateUtil.java:54)
at
org.apache.flink.runtime.checkpoint.OperatorSubtaskState.discardState(OperatorSubtaskState.java:187)
at
org.apache.flink.runtime.checkpoint.OperatorState.discardState(OperatorState.java:108)
at
org.apache.flink.util.LambdaUtil.applyToAllWhileSuppressingExceptions(LambdaUtil.java:51)
at
org.apache.flink.runtime.state.StateUtil.bestEffortDiscardAllStateObjects(StateUtil.java:54)
at
org.apache.flink.runtime.checkpoint.CompletedCheckpoint.doDiscard(CompletedCheckpoint.java:229)
at
org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnShutdown(CompletedCheckpoint.java:198)
at
org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore$2.apply(ZooKeeperCompletedCheckpointStore.java:351)
at
org.apache.flink.runtime.zookeeper.ZooKeeperStateHandleStore$RemoveBackgroundCallback.processResult(ZooKeeperStateHandleStore.java:640)
at
org.apache.curator.framework.imps.Backgrounding$1$1.run(Backgrounding.java:150)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
The cause of this exception is that disposal of an incremental state handle is
triggered in another async thread and the shutdown proceeds to close the I/O
Executor pool. This leads to RejectedExecutionException once the async deletes
are triggered inside the registry. We need to wait for all pending ZK deletes
before closing the I/O Executor pool.
2)
{code}
java.lang.Exception: Could not properly cancel managed keyed state future.
at
org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:91)
at
org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1090)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:685)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:621)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:574)
at
org.apache.flink.streaming.runtime.io.BarrierBuffer.notifyCheckpoint(BarrierBuffer.java:380)
at
org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(BarrierBuffer.java:230)
at
org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:185)
at
org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:225)
at
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:69)
[jira] [Created] (FLINK-8384) Session Window Assigner with Dynamic Gaps
Dyana Rose created FLINK-8384: - Summary: Session Window Assigner with Dynamic Gaps Key: FLINK-8384 URL: https://issues.apache.org/jira/browse/FLINK-8384 Project: Flink Issue Type: Improvement Components: Streaming Reporter: Dyana Rose Priority: Minor *Reason for Improvement* Currently both Session Window assigners only allow a static inactivity gap. Given the following scenario, this is too restrictive: * Given a stream of IoT events from many device types * Assume each device type could have a different inactivity gap * Assume each device type gap could change while sessions are in flight By allowing dynamic inactivity gaps, the correct gap can be determined in the [assignWindows function|https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/assigners/EventTimeSessionWindows.java#L59] by passing the element currently under consideration, the timestamp, and the context to a user defined function. This eliminates the need to create unwieldy work arounds if you only have static gaps. Dynamic Session Window gaps should be available for both Event Time and Processing Time streams. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Queryable State - Count within Time Window
Hi, I want to find the number of events happened so far in last 5 minutes and make that as a Queryable state. Is it possible? It will be of great help if someone provide some sample code for the same. Thanks, Velu.
[jira] [Created] (FLINK-8386) Scala API cannot create type information for Tuple interface
Timo Walther created FLINK-8386:
---
Summary: Scala API cannot create type information for Tuple
interface
Key: FLINK-8386
URL: https://issues.apache.org/jira/browse/FLINK-8386
Project: Flink
Issue Type: Bug
Components: Type Serialization System
Reporter: Timo Walther
Priority: Minor
The Scala API type inference does not work with the
{{org.apache.flink.api.java.tuple.Tuple}} interface.
{code}
val ti: TypeInformation[_] = Types.of[Tuple]
{code}
Leads to:
{code}
Exception in thread "main" java.lang.ClassCastException: [Ljava.lang.Object;
cannot be cast to [Lorg.apache.flink.api.common.typeinfo.TypeInformation;
at
org.apache.flink.table.examples.scala.WordCountTable$.main(WordCountTable.scala:47)
at
org.apache.flink.table.examples.scala.WordCountTable.main(WordCountTable.scala)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8387) Do not allow filter(false)
Timo Walther created FLINK-8387:
---
Summary: Do not allow filter(false)
Key: FLINK-8387
URL: https://issues.apache.org/jira/browse/FLINK-8387
Project: Flink
Issue Type: Improvement
Components: Table API & SQL
Reporter: Timo Walther
Assignee: Timo Walther
Code like the following looks correct but actually produces unintended side
effects:
{code}
left.join(right)
.where('a == 'd)
.select('a)
{code}
{{==}} is not converted into an expression but directly evaluated into an
boolean expression and implicitly converted into an {{false}} literal. Instead
of an exception this leads to an always rejecting table program. We should not
allow filters with {{false}} literals.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
aggregate stream of logs
Hi , i have this scenario of applications that streaming logs to kafka . I want to use flink in order to aggregate the stream and calculate some data i have a guid that i can correlate the stream to single event and i have a field that i can use to understand if is the last stream of the event . lets say the stream is something like that , 92e8366c-8ddd-41ed-a297-f9c28bf38792msg1start 92e8366c-8ddd-41ed-a297-f9c28bf38792msg2start 92e8366c-8ddd-41ed-a297-f9c28bf38792msg3end 92e8366c-8ddd-41ed-a297-f9c28bf3666 msg1start I cant use window trigger because then i can lose some data ? the only thing i can think about is onElement trigger ? what is you suggestion for this use case ?
[jira] [Created] (FLINK-8388) Snapshot documentation redirects to 1.4
Chesnay Schepler created FLINK-8388:
---
Summary: Snapshot documentation redirects to 1.4
Key: FLINK-8388
URL: https://issues.apache.org/jira/browse/FLINK-8388
Project: Flink
Issue Type: Bug
Components: Documentation, Project Website
Affects Versions: 1.5.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Fix For: 1.5.0
The {{baseUrl}} in {{_config.yml}} is still set to 1.4 causing all sections of
the snapshot documentation to redirect to the 1.4 docs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8389) Release all slots upon closing of JobMaster connection
Till Rohrmann created FLINK-8389:
Summary: Release all slots upon closing of JobMaster connection
Key: FLINK-8389
URL: https://issues.apache.org/jira/browse/FLINK-8389
Project: Flink
Issue Type: Improvement
Components: Distributed Coordination
Affects Versions: 1.5.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Fix For: 1.5.0
The {{TaskExecutor}} should release all slots when the {{JobMaster}}
disconnects from it (active disconnect instead of heartbeat timeout initiated).
This will make the slots faster available again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8390) Refactor Hadoop kerberos integration test code
Shuyi Chen created FLINK-8390: - Summary: Refactor Hadoop kerberos integration test code Key: FLINK-8390 URL: https://issues.apache.org/jira/browse/FLINK-8390 Project: Flink Issue Type: Bug Components: Security Affects Versions: 1.5.0 Reporter: Shuyi Chen Assignee: Shuyi Chen As suggested in [Flink-8270| https://issues.apache.org/jira/browse/FLINK-8270] and [Flink-8275|https://issues.apache.org/jira/browse/FLINK-8275|, we want to refactor, possibly remove, the Hadoop-kerberos integration test code from the main code. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
