[jira] [Created] (FLINK-8020) Deadlock found in Flink Streaming job
Weihua Jiang created FLINK-8020: --- Summary: Deadlock found in Flink Streaming job Key: FLINK-8020 URL: https://issues.apache.org/jira/browse/FLINK-8020 Project: Flink Issue Type: Bug Components: Kafka Connector, Streaming, Streaming Connectors Affects Versions: 1.3.2 Environment: Kafka 0.8.2 and Flink 1.3.2 on YARN mode Reporter: Weihua Jiang Priority: Blocker Attachments: jstack67976-2.log Our streaming job run into trouble in these days after a long time smooth running. One issue we found is [#FLINK-8019] and another one is this one. After analyzing the jstack, we believe we found a DEAD LOCK in flink: 1. The thread "cache-process0 -> async-operator0 -> Sink: hbase-sink0 (8/8)" hold lock 0x0007b6aa1788 and is waiting for lock 0x0007b6aa1940. 2. The thread "Time Trigger for cache-process0 -> async-operator0 -> Sink: hbase-sink0 (8/8)" hold lock 0x0007b6aa1940 and is waiting for lock 0x0007b6aa1788. This DEADLOCK made the job fail to proceed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8021) End-to-end tests may not shutdown cluster on failure
Chesnay Schepler created FLINK-8021: --- Summary: End-to-end tests may not shutdown cluster on failure Key: FLINK-8021 URL: https://issues.apache.org/jira/browse/FLINK-8021 Project: Flink Issue Type: Bug Components: Tests Affects Versions: 1.4.0, 1.5.0 Reporter: Chesnay Schepler In this job https://travis-ci.org/zentol/flink/jobs/298656917 the kafka E2E test failed straight away due to a missing class. The subsequent tests failed since they cannot allocate the JM port. It is thus likely that the E2E tests do not shutdown the cluster in all failure cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: Remove the HDFS directory in org.apache.flink.util.FileUtils.deletePathIfEmpty
Hi, You images did not make it through to the mailing list. Best, Aljoscha > On 8. Nov 2017, at 05:25, 马庆祥 wrote: > > Hi,all, > > I enable checkpoint with the configuration in the below figure . > > > it works, but keep getting the below exception: > > > I want to know if the below commit is to resolve the above problem, but the > exception still appears. > [hotfix] [core] Fix FileUtils.deletePathIfEmpty > > Flink version: 1.3.1 > Hadoop version: 1.x > > thanks~
Re: Remove the HDFS directory in org.apache.flink.util.FileUtils.deletePathIfEmpty
For me they showed in user mailing list, but not in dev. (or maybe the reverse, not quite sure...) On 08.11.2017 10:47, Aljoscha Krettek wrote: Hi, You images did not make it through to the mailing list. Best, Aljoscha On 8. Nov 2017, at 05:25, 马庆祥 wrote: Hi,all, I enable checkpoint with the configuration in the below figure . it works, but keep getting the below exception: I want to know if the below commit is to resolve the above problem, but the exception still appears. [hotfix] [core] Fix FileUtils.deletePathIfEmpty Flink version: 1.3.1 Hadoop version: 1.x thanks~
[jira] [Created] (FLINK-8022) Kafka at-least-once tests fail occasionally
Nico Kruber created FLINK-8022:
--
Summary: Kafka at-least-once tests fail occasionally
Key: FLINK-8022
URL: https://issues.apache.org/jira/browse/FLINK-8022
Project: Flink
Issue Type: Bug
Components: Kafka Connector, Tests
Affects Versions: 1.4.0
Reporter: Nico Kruber
Assignee: Tzu-Li (Gordon) Tai
Priority: Critical
{{Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink}}
seems to sporadically fail with missing data, like this execution:
{code}
Test
testOneToOneAtLeastOnceRegularSink(org.apache.flink.streaming.connectors.kafka.Kafka09ProducerITCase)
is running.
17:54:30,195 INFO
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl -
Creating topic oneToOneTopicRegularSink
17:54:30,196 INFO
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl - In
getZKUtils:: zookeeperConnectionString = 127.0.0.1:39436
17:54:30,204 INFO org.I0Itec.zkclient.ZkEventThread
- Starting ZkClient event thread.
17:54:30,204 INFO org.I0Itec.zkclient.ZkClient
- Waiting for keeper state SyncConnected
17:54:30,240 INFO org.I0Itec.zkclient.ZkClient
- zookeeper state changed (SyncConnected)
17:54:30,261 INFO org.I0Itec.zkclient.ZkEventThread
- Terminate ZkClient event thread.
17:54:30,265 INFO
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl - Topic
oneToOneTopicRegularSink create request is successfully posted
17:54:30,366 INFO
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl -
Validating if the topic oneToOneTopicRegularSink has been created or not
17:54:30,373 INFO
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl - In
getZKUtils:: zookeeperConnectionString = 127.0.0.1:39436
17:54:30,374 INFO org.I0Itec.zkclient.ZkEventThread
- Starting ZkClient event thread.
17:54:30,374 INFO org.I0Itec.zkclient.ZkClient
- Waiting for keeper state SyncConnected
17:54:30,404 INFO org.I0Itec.zkclient.ZkClient
- zookeeper state changed (SyncConnected)
17:54:30,420 INFO
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl - topic
oneToOneTopicRegularSink has been created successfully
17:54:30,421 INFO org.I0Itec.zkclient.ZkEventThread
- Terminate ZkClient event thread.
17:54:31,099 INFO
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase - Starting
FlinkKafkaProducer (1/1) to produce into default topic oneToOneTopicRegularSink
17:55:05,229 ERROR
org.apache.flink.streaming.connectors.kafka.Kafka09ProducerITCase -
Test
testOneToOneAtLeastOnceRegularSink(org.apache.flink.streaming.connectors.kafka.Kafka09ProducerITCase)
failed with:
java.lang.AssertionError: Expected to contain all of: <[0, 1, 2, 3, 4, 5, 6, 7,
8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67,
68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87,
88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105,
106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121,
122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137,
138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153,
154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169,
170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185,
186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201,
202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217,
218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233,
234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249,
250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265,
266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281,
282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297,
298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313,
314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329,
330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345,
346, 347, 348, 349, 350, 351, 35
[jira] [Created] (FLINK-8023) User proper AvroSerializer for Avro types
Aljoscha Krettek created FLINK-8023: --- Summary: User proper AvroSerializer for Avro types Key: FLINK-8023 URL: https://issues.apache.org/jira/browse/FLINK-8023 Project: Flink Issue Type: Improvement Components: Type Serialization System Reporter: Aljoscha Krettek Assignee: Stephan Ewen Fix For: 1.4.0 I'm retroactively creating this issue to record that the change happened. It was initially attributed to a similar issue (FLINK-6022). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8024) Let ClusterOverviewHandler directly extend from AbstractRestHandler
Till Rohrmann created FLINK-8024:
Summary: Let ClusterOverviewHandler directly extend from
AbstractRestHandler
Key: FLINK-8024
URL: https://issues.apache.org/jira/browse/FLINK-8024
Project: Flink
Issue Type: Sub-task
Components: REST
Affects Versions: 1.5.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Priority: Minor
Fix For: 1.5.0
In order to get rid of the {{LegacyRestHandler}} we should add a proper
implementation of {{ClusterOverviewHandler}} which extends from
{{AbstractRestHandler}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8025) Let DashboardConfigHandler extend from AbstractRestHandler
Till Rohrmann created FLINK-8025:
Summary: Let DashboardConfigHandler extend from AbstractRestHandler
Key: FLINK-8025
URL: https://issues.apache.org/jira/browse/FLINK-8025
Project: Flink
Issue Type: Sub-task
Components: REST
Affects Versions: 1.5.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Priority: Minor
Fix For: 1.5.0
The {{DashboardConfigHandler}} should directly extend {{AbstractRestHandler}}
to get rid of the {{LegacyRestHandler}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8026) Let ClusterConfigHandler extend from AbstractRestHandler
Till Rohrmann created FLINK-8026:
Summary: Let ClusterConfigHandler extend from AbstractRestHandler
Key: FLINK-8026
URL: https://issues.apache.org/jira/browse/FLINK-8026
Project: Flink
Issue Type: Sub-task
Components: REST
Affects Versions: 1.5.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Priority: Minor
Fix For: 1.5.0
The {{ClusterConfigHandler}} should directly extend from the
{{AbstractRestHandler}} to get rid of the {{LegacyRestHandler}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8027) Generalize existing rest handlers to work with arbitrary RestfulGateway
Till Rohrmann created FLINK-8027:
Summary: Generalize existing rest handlers to work with arbitrary
RestfulGateway
Key: FLINK-8027
URL: https://issues.apache.org/jira/browse/FLINK-8027
Project: Flink
Issue Type: Sub-task
Components: REST
Affects Versions: 1.5.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Fix For: 1.5.0
In order to reuse the existing {{AbstractRestHandler}} we should refactor them
such that they work with arbitrary {{RestfulGateway}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8028) Let JobMaster impelment RestfulGateway
Till Rohrmann created FLINK-8028:
Summary: Let JobMaster impelment RestfulGateway
Key: FLINK-8028
URL: https://issues.apache.org/jira/browse/FLINK-8028
Project: Flink
Issue Type: Sub-task
Components: REST
Affects Versions: 1.5.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Fix For: 1.5.0
In order to use the {{JobMaster}} with the REST handlers, the {{JobMaster}} has
to implement the {{RestfulGateway}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8029) Create common WebMonitorEndpoint
Till Rohrmann created FLINK-8029:
Summary: Create common WebMonitorEndpoint
Key: FLINK-8029
URL: https://issues.apache.org/jira/browse/FLINK-8029
Project: Flink
Issue Type: Sub-task
Components: REST
Affects Versions: 1.5.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Fix For: 1.5.0
In order to reuse the existing the REST handlers, we should create a common
{{WebMonitorEndpoint}} which is shared by the {{Dispatcher}} and the
{{JobMaster}} component.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8030) Start JobMasterRestEndpoint in JobClusterEntrypoint
Till Rohrmann created FLINK-8030:
Summary: Start JobMasterRestEndpoint in JobClusterEntrypoint
Key: FLINK-8030
URL: https://issues.apache.org/jira/browse/FLINK-8030
Project: Flink
Issue Type: Sub-task
Components: REST
Affects Versions: 1.5.0
Reporter: Till Rohrmann
Fix For: 1.5.0
We should launch the {{JobMasterRestEndpoint}} in the {{JobClusterEntrypoint}}
in order to run the web frontend for per-job clusters.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8031) Provide configuration to enable/disable ability to cancel jobs in Web Frontend
Stephen Hesketh created FLINK-8031:
--
Summary: Provide configuration to enable/disable ability to cancel
jobs in Web Frontend
Key: FLINK-8031
URL: https://issues.apache.org/jira/browse/FLINK-8031
Project: Flink
Issue Type: Improvement
Components: flink-contrib
Affects Versions: 1.3.2
Reporter: Stephen Hesketh
The Web API can be configured to prevent job submission by setting
jobmanager.web.submit.enable to false in the Flink YAML configuration file:
{quote}
\# Flag to specify whether job submission is enabled from the web-based
\# runtime monitor. Uncomment to disable.
{color:red}
jobmanager.web.submit.enable: false{color}
{quote}
However, users can still cancel running jobs. We would like to be able to offer
the Web Frontend to teams for monitoring of jobs. However, since this is shared
infrastructure we do not want users to be able to run or cancel jobs through
this interface.
It is proposed that the ability to cancel jobs is also configurable in the
Flink yaml configuration file.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8032) Create job control handler for JobMasterRestHandler
Till Rohrmann created FLINK-8032:
Summary: Create job control handler for JobMasterRestHandler
Key: FLINK-8032
URL: https://issues.apache.org/jira/browse/FLINK-8032
Project: Flink
Issue Type: Sub-task
Components: REST
Affects Versions: 1.5.0
Reporter: Till Rohrmann
In order to work with the {{RestClusterClient}}, the {{JobMasterRestEndpoint}}
has to register handler for job control: cancel, stop, take savepoint and list
jobs, for example.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8033) Build Flink with JDK 9
Hai Zhou UTC+8 created FLINK-8033: - Summary: Build Flink with JDK 9 Key: FLINK-8033 URL: https://issues.apache.org/jira/browse/FLINK-8033 Project: Flink Issue Type: Improvement Components: Build System Affects Versions: 1.4.0 Reporter: Hai Zhou UTC+8 Fix For: 1.5.0 This is a JIRA to track all issues that found to support Flink on Java 9 in the future. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8034) ProcessFailureCancelingITCase.testCancelingOnProcessFailure failing on Travis
Till Rohrmann created FLINK-8034:
Summary:
ProcessFailureCancelingITCase.testCancelingOnProcessFailure failing on Travis
Key: FLINK-8034
URL: https://issues.apache.org/jira/browse/FLINK-8034
Project: Flink
Issue Type: Bug
Components: Tests
Affects Versions: 1.5.0
Reporter: Till Rohrmann
Priority: Critical
The {{ProcessFailureCancelingITCase.testCancelingOnProcessFailure}} is failing
on Travis spuriously.
https://travis-ci.org/tillrohrmann/flink/jobs/299075703
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8035) Unable to submit job when HA is enabled
Robert Metzger created FLINK-8035:
-
Summary: Unable to submit job when HA is enabled
Key: FLINK-8035
URL: https://issues.apache.org/jira/browse/FLINK-8035
Project: Flink
Issue Type: Bug
Components: JobManager
Affects Versions: 1.4.0
Environment: Mac OS X
Reporter: Robert Metzger
Steps to reproduce:
- Get Flink 1.4 (f5a0b4bdfb)
- Get ZK (3.3.6 in this case)
- Put the following flink-conf.yaml:
{code}
high-availability: zookeeper
high-availability.storageDir: file:///tmp/flink-ha
high-availability.zookeeper.quorum: localhost:2181
high-availability.zookeeper.path.cluster-id: /my-namespace
{code}
- Start Flink, submit a job (any streaming example will do)
The job submission will time out. On the JobManager, it seems that the job
submission gets stuck when trying to submit something to Zookeeper.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8036) Consider using gradle to build Flink
Ted Yu created FLINK-8036: - Summary: Consider using gradle to build Flink Key: FLINK-8036 URL: https://issues.apache.org/jira/browse/FLINK-8036 Project: Flink Issue Type: Improvement Reporter: Ted Yu Here is summary from Lukasz over this thread (http://search-hadoop.com/m/Beam/gfKHFVh4NM151XIu1?subj=Re+DISCUSS+Move+away+from+Apache+Maven+as+build+tool) w.r.t. performance boost from using gradle: Maven performs parallelization at the module level, an entire module needs to complete before any dependent modules can start, this means running all the checks like findbugs, checkstyle, tests need to finish. Gradle has task level parallelism between subprojects which means that as soon as the compile and shade steps are done for a project, and dependent subprojects can typically start. This means that we get increased parallelism due to not needing to wait for findbugs, checkstyle, tests to run. I typically see ~20 tasks (at peak) running on my desktop in parallel. Flink should consider using gradle - on Linux with SSD, a clean build takes an hour. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8037) Missing cast in integer arithmetic in TransactionalIdsGenerator#generateIdsToAbort
Ted Yu created FLINK-8037:
-
Summary: Missing cast in integer arithmetic in
TransactionalIdsGenerator#generateIdsToAbort
Key: FLINK-8037
URL: https://issues.apache.org/jira/browse/FLINK-8037
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
public Set generateIdsToAbort() {
Set idsToAbort = new HashSet<>();
for (int i = 0; i < safeScaleDownFactor; i++) {
idsToAbort.addAll(generateIdsToUse(i * poolSize * totalNumberOfSubtasks));
{code}
The operands are integers where generateIdsToUse() expects long parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8038) Support MAP literal
Rong Rong created FLINK-8038:
Summary: Support MAP literal
Key: FLINK-8038
URL: https://issues.apache.org/jira/browse/FLINK-8038
Project: Flink
Issue Type: New Feature
Components: Table API & SQL
Reporter: Rong Rong
Assignee: Rong Rong
Similar to https://issues.apache.org/jira/browse/FLINK-4554
We want to support Map literals which is supported by Calcite:
https://calcite.apache.org/docs/reference.html#value-constructors
{code:sql}
SELECT
MAP['key1', f0, 'key2', f1] AS stringKeyedMap,
MAP['key', 'value'] AS literalMap,
MAP[f0, f1] AS fieldMap
FROM
table
{code}
This should enable users to construct MapTypeInfo, one of the CompositeType.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8039) Support "CREATE TYPE" DDL in Flink SQL
Shuyi Chen created FLINK-8039:
-
Summary: Support "CREATE TYPE" DDL in Flink SQL
Key: FLINK-8039
URL: https://issues.apache.org/jira/browse/FLINK-8039
Project: Flink
Issue Type: New Feature
Components: Table API & SQL
Reporter: Shuyi Chen
Assignee: Shuyi Chen
Allow us to create custom types using DDL, e.g.,
{code:java}
CREATE TYPE myrowtype AS (f1 INTEGER, f2 VARCHAR(10));
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Re: Remove the HDFS directory in org.apache.flink.util.FileUtils.deletePathIfEmpty
That has been solved, Because of the hadoop version issue. Thanks. 2017-11-08 17:54 GMT+08:00 Chesnay Schepler : > For me they showed in user mailing list, but not in dev. (or maybe the > reverse, not quite sure...) > > On 08.11.2017 10:47, Aljoscha Krettek wrote: > >> Hi, >> >> You images did not make it through to the mailing list. >> >> Best, >> Aljoscha >> >> On 8. Nov 2017, at 05:25, 马庆祥 wrote: >>> >>> Hi,all, >>> >>> I enable checkpoint with the configuration in the below figure . >>> >>> >>> it works, but keep getting the below exception: >>> >>> >>> I want to know if the below commit is to resolve the above problem, but >>> the exception still appears. >>> [hotfix] [core] Fix FileUtils.deletePathIfEmpty >>> >>> Flink version: 1.3.1 >>> Hadoop version: 1.x >>> >>> thanks~ >>> >> >> >
