[jira] [Closed] (FLINK-11160) Confluent Avro Serialization Schema
[ https://issues.apache.org/jira/browse/FLINK-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-11160. Resolution: Duplicate > Confluent Avro Serialization Schema > > > Key: FLINK-11160 > URL: https://issues.apache.org/jira/browse/FLINK-11160 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Affects Versions: 1.7.0 >Reporter: Zhenhao Li >Priority: Minor > Labels: newbie, pull-request-available, scala > Original Estimate: 24h > Time Spent: 10m > Remaining Estimate: 23h 50m > > Currently, Flink is missing Serialization Schema to work with the Confluent > Avro format and the Confluent schema registry. > I wrote something that solved this problem for the company I currently work > at. I think it is nice to contribute something back to the community. It has > been used in a Scala project, and the project has been deployed to > production. > The new serialization schemas only serialize GenericRecord and users have to > pass the Avro schema files to the constructors. It might be not flexible > enough to cover a broader set of use cases. The keyed serialization schema > works for only Scala key-value paris. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-11451) Move *QueryConfig and TableDescriptor to flink-table-api-java
[ https://issues.apache.org/jira/browse/FLINK-11451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-11451: --- Labels: pull-request-available (was: ) > Move *QueryConfig and TableDescriptor to flink-table-api-java > - > > Key: FLINK-11451 > URL: https://issues.apache.org/jira/browse/FLINK-11451 > Project: Flink > Issue Type: New Feature > Components: Table API & SQL >Reporter: Timo Walther >Assignee: xueyu >Priority: Major > Labels: pull-request-available > > A more detailed description can be found in > [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions]. > Move QueryConfig, BatchQueryConfig, StreamQueryConfig, TableDescriptor in > flink-table-api-java. > Unblocks TableEnvironment interface task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11409) Make `ProcessFunction`, `ProcessWindowFunction` and etc. pure interfaces
[
https://issues.apache.org/jira/browse/FLINK-11409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757282#comment-16757282
]
Kezhu Wang commented on FLINK-11409:
[~aljoscha] [~dawidwys] I would like to present example code for this
discussion.
{code:java}
public abstract class AbstractFlinkRichFunction extends
AbstractRichFunction implements CheckpointedFunction {
private final OperatorInfo operatorInfo;
protected transient T action;
@Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
// Open target operator action
}
@Override
public void close() throws Exception {
// Close target operator action
super.close();
}
@Override
public void snapshotState(FunctionSnapshotContext snapshotContext) throws
Exception {
// Relay snapshot to target operator action
}
@Override
public void initializeState(FunctionInitializationContext
initializationContext) throws Exception {
// Create operator action base on and operator info
// Relay initializeState to target operator action
}
}
public class FlinkFlatMapFunction extends AbstractFlinkRichFunction
implements FlatMapFunction {
@Override
public void flatMap(Event value, Collector out) throws Exception {
// Relay flatMap to target operator action
}
}
{code}
In above code, `AbstractFlinkRichFunction` focuses on lifecycle management,
while `FlinkXyzFunction` focuses on data processing. This pattern works fine
for `MapFunction`, `FilterFunction`, `SourceFunction` and others. But for
`ProcessFunction` and etc., we have to duplicate `AbstractFlinkRichFunction` as
these function callbacks are implemented as abstract classes. *Due to Java's
single class inheritance, I think exporting _callback like apis_ as classes not
interfaces is intrusive and unfriendly to caller.*
Besides this, from api perspective, I think making `ProcessFunction` and etc.
as subclass of `AbstractRichFunction` mixes up data processing function and
lifecycle management.
> Make `ProcessFunction`, `ProcessWindowFunction` and etc. pure interfaces
>
>
> Key: FLINK-11409
> URL: https://issues.apache.org/jira/browse/FLINK-11409
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
>Reporter: Kezhu Wang
>Priority: Major
> Labels: Breaking-Change
>
> I found these functions express no opinionated demands from implementing
> classes. It would be nice to implement as interfaces not abstract classes as
> abstract class is intrusive and hampers caller user cases. For example,
> client can't write an `AbstractFlinkRichFunction` to unify lifecycle
> management for all data processing functions in easy way.
> I dive history of some of these functions, and find that some functions were
> converted as abstract class from interface due to default method
> implementation, such as `ProcessFunction` and `CoProcessFunction` were
> converted to abstract classes in FLINK-4460 which predate -FLINK-7242-. After
> -FLINK-7242-, [Java 8 default
> method|https://docs.oracle.com/javase/tutorial/java/IandI/defaultmethods.html]
> would be a better solution.
> I notice also that some functions which are introduced after -FLINK-7242-,
> such as `ProcessJoinFunction`, are implemented as abstract classes. I think
> it would be better to establish a well-known principle to guide both api
> authors and callers of data processing functions.
> Personally, I prefer interface for all exported function callbacks for the
> reason I express in first paragraph.
> Besides this, with `AbstractRichFunction` and interfaces for data processing
> functions I think lots of rich data processing functions can be eliminated as
> they are plain classes extending `AbstractRichFunction` and implementing data
> processing interfaces, clients can write this in one line code with clear
> intention of both data processing and lifecycle management.
> Following is a possible incomplete list of data processing functions
> implemented as abstract classes currently:
> * `ProcessFunction`, `KeyedProcessFunction`, `CoProcessFunction` and
> `ProcessJoinFunction`
> * `ProcessWindowFunction` and `ProcessAllWindowFunction`
> * `BaseBroadcastProcessFunction`, `BroadcastProcessFunction` and
> `KeyedBroadcastProcessFunction`
> All above functions are annotated with `@PublicEvolving`, making they
> interfaces won't break Flink's compatibility guarantee but compatibility is
> still a big consideration to evaluate this proposal.
> Any thoughts on this proposal ? Please must comment out.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-11451) Move *QueryConfig and TableDescriptor to flink-table-api-java
[
https://issues.apache.org/jira/browse/FLINK-11451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757275#comment-16757275
]
xueyu edited comment on FLINK-11451 at 1/31/19 2:24 PM:
Hi, [~hequn8128] [~twalthr] I submitted the PR to move QueryConfig,
BatchQueryConfig, StreamQueryConfig and TableDescriptor to
flink-table-api-java. Please have a review when you are free.
I want to ask two questions
1. For {{class QueryConfig private[table] extends Serializable}} how to keep
the semantic of {{private[table]}} in java when migration...
2. Why put TableDescriptor in flink-table-api-java but Descriptor,
FormatDescriptor, ConnectorDescriptor etc. in flink-table-common, they are in
the same package but different module.
Thanks
was (Author: xueyu):
Hi, [~hequn8128] [~twalthr] I submitted the PR to move QueryConfig,
BatchQueryConfig, StreamQueryConfig and TableDescriptor to
flink-table-api-java. Please have a review when you are free.
I want to ask two questions
1. For {{class QueryConfig private[table] extends Serializable }} how to keep
the semantic of {{private[table]}} in java when migration...
2. Why put TableDescriptor in flink-table-api-java but Descriptor,
FormatDescriptor, ConnectorDescriptor etc. in flink-table-common, they are in
the same package but different module.
Thanks
> Move *QueryConfig and TableDescriptor to flink-table-api-java
> -
>
> Key: FLINK-11451
> URL: https://issues.apache.org/jira/browse/FLINK-11451
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: xueyu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Move QueryConfig, BatchQueryConfig, StreamQueryConfig, TableDescriptor in
> flink-table-api-java.
> Unblocks TableEnvironment interface task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11451) Move *QueryConfig and TableDescriptor to flink-table-api-java
[
https://issues.apache.org/jira/browse/FLINK-11451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757275#comment-16757275
]
xueyu commented on FLINK-11451:
---
Hi, [~hequn8128] [~twalthr] I submitted the PR to move QueryConfig,
BatchQueryConfig, StreamQueryConfig and TableDescriptor to
flink-table-api-java. Please have a review when you are free.
I want to ask two questions
1. For {{class QueryConfig private[table] extends Serializable }} how to keep
the semantic of {{private[table]}} in java when migration...
2. Why put TableDescriptor in flink-table-api-java but Descriptor,
FormatDescriptor, ConnectorDescriptor etc. in flink-table-common, they are in
the same package but different module.
Thanks
> Move *QueryConfig and TableDescriptor to flink-table-api-java
> -
>
> Key: FLINK-11451
> URL: https://issues.apache.org/jira/browse/FLINK-11451
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: xueyu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Move QueryConfig, BatchQueryConfig, StreamQueryConfig, TableDescriptor in
> flink-table-api-java.
> Unblocks TableEnvironment interface task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11449) Uncouple the Expression class from RexNodes
[
https://issues.apache.org/jira/browse/FLINK-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757301#comment-16757301
]
Timo Walther commented on FLINK-11449:
--
Thanks for working on this [~sunjincheng121]. I went through all expressions in
{{expressionDsl.scala}} and {{ExpressionParser}} to check how the interfaces
between API and planner could look like. In the long-term vision, we should
treat built-in functions and user-defined functions the same. At the same time,
we should also keep the reworking efforts for the Blink merge low. That's why I
would let the current expression subclasses untouched and let them stay in
{{flink-table-planner}}/{{flink-table-planner-blink}}. I would suggest the
following class structure:
In {{flink-table-api-common}}, because we can even put those expression classes
into flink-table-common for FilterableTableSources in the future:
{code}
public class FunctionDefinition {
public String name;
}
public class UserDefinedFunctionDefinition extends FunctionDefinition {
// the name is ignored for now
public UserDefinedFunction f;
}
public class Expression {
abstract List getChildren();
abstract R accept(ExpressionVisitor visitor);
// do we need more methods here?
}
CallExpression(FunctionDefinition, Expression*) extends Expression
SymbolExpression(TableSymbol ???) extends Expression
FieldReferenceExpression(String) extends Expression
TableReferenceExpression(String, Table) extends Expression
TypeLiteralExpression(TypeInformation) extends Expression
ValueLiteralExpression(Object, TypeInformation) or
ValueLiteralExpression(Object) extends Expression
{code}
In {{flink-table-api-java}}:
{code}
public final class FunctionDefinitions {
// implement just a basic definition (only name) for know
// we can extend this functionality in the future with type inference
logic etc.
public static final FunctionDefinition TRIM = new
FunctionDefinition("trim");
public static final FunctionDefinition CAST = new
FunctionDefinition("cast");
public static List getDefinitions() {
// use reflection for all definitions similar to Calcite's
SqlStdOperatorTable
}
}
{code}
In {{flink-table-planner}}:
{code}
+ visitor pattern in the planner to translate
in Flink: to Flink's case classes
in Blink: to Blink's case classes
visit(call Call) match {
case TRIM =>
}
{code}
What do you think?
> Uncouple the Expression class from RexNodes
> ---
>
> Key: FLINK-11449
> URL: https://issues.apache.org/jira/browse/FLINK-11449
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: sunjincheng
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Calcite will not be part of any API module anymore. Therefore, RexNode
> translation must happen in a different layer. This issue will require a new
> design document.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] Myasuka commented on a change in pull request #7600: [FLINK-11368][tests] Port TaskManagerStartupTest to new code base
Myasuka commented on a change in pull request #7600: [FLINK-11368][tests] Port
TaskManagerStartupTest to new code base
URL: https://github.com/apache/flink/pull/7600#discussion_r252693919
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskManagerRunnerStartupTest.java
##
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.taskexecutor;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.runtime.blob.BlobCacheService;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.heartbeat.HeartbeatServices;
+import org.apache.flink.runtime.highavailability.HighAvailabilityServices;
+import org.apache.flink.runtime.metrics.NoOpMetricRegistry;
+import org.apache.flink.runtime.rpc.FatalErrorHandler;
+import org.apache.flink.runtime.rpc.RpcService;
+import org.apache.flink.util.IOUtils;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.Assume;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.ServerSocket;
+
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+import static org.junit.Assume.assumeNoException;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Tests that check how the {@link TaskManagerRunner} behaves when
encountering startup problems.
+ */
+public class TaskManagerRunnerStartupTest extends TestLogger {
+
+ private static final String LOCAL_HOST = "localhost";
+
+ @Rule
+ public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ private RpcService rpcService = createRpcService();
+
+ /**
+* Tests that the TaskManagerRunner startup fails synchronously when
the I/O
+* directories are not writable.
+*/
+ @Test
+ public void testIODirectoryNotWritable() throws Exception {
+ File nonWritable = tempFolder.newFolder();
+ Assume.assumeTrue("Cannot create non-writable temporary file
directory. Skipping test.",
+ nonWritable.setWritable(false, false));
+
+ try {
+ Configuration cfg = new Configuration();
+ cfg.setString(CoreOptions.TMP_DIRS,
nonWritable.getAbsolutePath());
+
+ try {
+ TaskManagerRunner.startTaskManager(
+ cfg,
+ ResourceID.generate(),
+ rpcService,
+ mock(HighAvailabilityServices.class),
+ mock(HeartbeatServices.class),
+ NoOpMetricRegistry.INSTANCE,
+ mock(BlobCacheService.class),
+ false,
+ mock(FatalErrorHandler.class));
+
+ fail("Should fail synchronously with an
exception");
+ } catch (IOException e) {
+ // splendid!
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ fail(e.getMessage());
+ } finally {
+ // noinspection ResultOfMethodCallIgnored
+ nonWritable.setWritable(true, false);
+ try {
+ FileUtils.deleteDirectory(nonWritable);
+ } catch (IOException e) {
+ // best effort
+ }
+ }
+ }
+
+ /**
+* Tests that the TaskMan
[jira] [Assigned] (FLINK-11409) Make `ProcessFunction`, `ProcessWindowFunction` and etc. pure interfaces
[ https://issues.apache.org/jira/browse/FLINK-11409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kezhu Wang reassigned FLINK-11409: -- Assignee: Kezhu Wang > Make `ProcessFunction`, `ProcessWindowFunction` and etc. pure interfaces > > > Key: FLINK-11409 > URL: https://issues.apache.org/jira/browse/FLINK-11409 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Reporter: Kezhu Wang >Assignee: Kezhu Wang >Priority: Major > Labels: Breaking-Change > > I found these functions express no opinionated demands from implementing > classes. It would be nice to implement as interfaces not abstract classes as > abstract class is intrusive and hampers caller user cases. For example, > client can't write an `AbstractFlinkRichFunction` to unify lifecycle > management for all data processing functions in easy way. > I dive history of some of these functions, and find that some functions were > converted as abstract class from interface due to default method > implementation, such as `ProcessFunction` and `CoProcessFunction` were > converted to abstract classes in FLINK-4460 which predate -FLINK-7242-. After > -FLINK-7242-, [Java 8 default > method|https://docs.oracle.com/javase/tutorial/java/IandI/defaultmethods.html] > would be a better solution. > I notice also that some functions which are introduced after -FLINK-7242-, > such as `ProcessJoinFunction`, are implemented as abstract classes. I think > it would be better to establish a well-known principle to guide both api > authors and callers of data processing functions. > Personally, I prefer interface for all exported function callbacks for the > reason I express in first paragraph. > Besides this, with `AbstractRichFunction` and interfaces for data processing > functions I think lots of rich data processing functions can be eliminated as > they are plain classes extending `AbstractRichFunction` and implementing data > processing interfaces, clients can write this in one line code with clear > intention of both data processing and lifecycle management. > Following is a possible incomplete list of data processing functions > implemented as abstract classes currently: > * `ProcessFunction`, `KeyedProcessFunction`, `CoProcessFunction` and > `ProcessJoinFunction` > * `ProcessWindowFunction` and `ProcessAllWindowFunction` > * `BaseBroadcastProcessFunction`, `BroadcastProcessFunction` and > `KeyedBroadcastProcessFunction` > All above functions are annotated with `@PublicEvolving`, making they > interfaces won't break Flink's compatibility guarantee but compatibility is > still a big consideration to evaluate this proposal. > Any thoughts on this proposal ? Please must comment out. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11496) FlinkS3 FileSysten is not handling multiple local temp directories
Elango Ganesan created FLINK-11496: -- Summary: FlinkS3 FileSysten is not handling multiple local temp directories Key: FLINK-11496 URL: https://issues.apache.org/jira/browse/FLINK-11496 Project: Flink Issue Type: Bug Reporter: Elango Ganesan Fix For: 1.7.1 S3 Flink Filesystem when creating localTemp directory is not splitting and handling the availability of multiple local temp directories . As a result we are seeing exception mentioned below any time we run in a EC2 instance type with more than one ephemeral drive or EBS volume. https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/FlinkS3FileSystem.java#L101 Timestamp: 2019-01-29, 12:42:39 java.nio.file.NoSuchFileException: /mnt/yarn/usercache/hadoop/appcache/application_1548598173158_0004,/mnt1/yarn/usercache/hadoop/appcache/application_1548598173158_0004/.tmp_072167ee-6432-412c-809a-bd0599961cf0 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434) at java.nio.file.Files.newOutputStream(Files.java:216) at org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator.apply(RefCountedTmpFileCreator.java:80) at org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator.apply(RefCountedTmpFileCreator.java:39) at org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream.openNew(RefCountedBufferingFileStream.java:174) at org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.boundedBufferingFileStream(S3RecoverableFsDataOutputStream.java:271) at org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.newStream(S3RecoverableFsDataOutputStream.java:236) at org.apache.flink.fs.s3.common.writer.S3RecoverableWriter.open(S3RecoverableWriter.java:78) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit closed pull request #7532: [FLINK-11389] Fix Incorrectly use job information when call getSerial…
asfgit closed pull request #7532: [FLINK-11389] Fix Incorrectly use job information when call getSerial… URL: https://github.com/apache/flink/pull/7532 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] StefanRRichter commented on a change in pull request #7568: [FLINK-11417] Make access to ExecutionGraph single threaded from JobMaster main thread
StefanRRichter commented on a change in pull request #7568: [FLINK-11417] Make
access to ExecutionGraph single threaded from JobMaster main thread
URL: https://github.com/apache/flink/pull/7568#discussion_r252699265
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##
@@ -478,22 +489,25 @@ public void testEagerSchedulingFailureReturnsSlot()
throws Exception {
slotRequestId);
slotProvider.complete(slotRequestId,
singleLogicalSlot);
},
- executorService);
-
- final CompletableFuture schedulingFuture =
execution.scheduleForExecution(
- slotProvider,
- false,
- LocationPreferenceConstraint.ANY,
- Collections.emptySet());
-
- try {
- schedulingFuture.get();
- // cancel the execution in case we could
schedule the execution
- execution.cancel();
- } catch (ExecutionException ignored) {
- }
-
- assertThat(returnedSlotFuture.get(),
is(equalTo(slotRequestIdFuture.get(;
+ executorService).thenRunAsync(() -> {
+ try {
+ final CompletableFuture
schedulingFuture = execution.scheduleForExecution(
+ slotProvider,
+ false,
+
LocationPreferenceConstraint.ANY,
+ Collections.emptySet());
+
+ try {
+ schedulingFuture.get();
+ // cancel the execution
in case we could schedule the execution
+ execution.cancel();
+ } catch (ExecutionException
ignored) {
+ }
+
+
assertThat(returnedSlotFuture.get(), is(equalTo(slotRequestIdFuture.get(;
+ } catch (Exception ex) {
+ }
+ }, testMainThreadUtil.getMainThreadExecutor());
Review comment:
I think it is not that easy because there is curently no way to complete the
future after scheduling took the failing branch without using another thread.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (FLINK-11389) Incorrectly use job information when call getSerializedTaskInformation in class TaskDeploymentDescriptor
[
https://issues.apache.org/jira/browse/FLINK-11389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann resolved FLINK-11389.
---
Resolution: Fixed
Fixed via
1.8.0:
446145f5ca0cd92c64a6944c19ca1ab53104030f
acf228cc5c88872428ccb10296aa1646837c16c2
1.7.2:
a5c6102bf642a0bfeff4438921aad3ab8d16cf1c
c39192a21098882338f0dcd9636f9241814099ce
1.6.4:
941ed4d816489990cd1a420d90b88f167a89db1a
ea90666b4415d1a7f510053b35101880438eab40
> Incorrectly use job information when call getSerializedTaskInformation in
> class TaskDeploymentDescriptor
>
>
> Key: FLINK-11389
> URL: https://issues.apache.org/jira/browse/FLINK-11389
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination
>Affects Versions: 1.6.3, 1.7.1, 1.8.0
>Reporter: yuqi
>Assignee: yuqi
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.4, 1.7.2, 1.8.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> See TaskDeploymentDescriptor
> {code:java}
> @Nullable
> public SerializedValue getSerializedTaskInformation() {
> if (serializedJobInformation instanceof NonOffloaded) {
> NonOffloaded jobInformation =
> (NonOffloaded)
> serializedTaskInformation;
> return jobInformation.serializedValue;
> } else {
> throw new IllegalStateException(
> "Trying to work with offloaded serialized job
> information.");
> }
> }
> {code}
> the condition serializedJobInformation instanceof NonOffloaded is not
> correctly,
> as serializedJobInformation and serializedTaskInformation are passed from
> ExecutionVertex#createDeploymentDescriptor
> {code:java}
> if (jobInformationOrBlobKey.isLeft()) {
> serializedJobInformation = new
> TaskDeploymentDescriptor.NonOffloaded<>(jobInformationOrBlobKey.left());
> } else {
> serializedJobInformation = new
> TaskDeploymentDescriptor.Offloaded<>(jobInformationOrBlobKey.right());
> }
> final Either,
> PermanentBlobKey> taskInformationOrBlobKey;
> try {
> taskInformationOrBlobKey =
> jobVertex.getTaskInformationOrBlobKey();
> } catch (IOException e) {
> throw new ExecutionGraphException(
> "Could not create a serialized
> JobVertexInformation for " +
> jobVertex.getJobVertexId(), e);
> }
> final TaskDeploymentDescriptor.MaybeOffloaded
> serializedTaskInformation;
> if (taskInformationOrBlobKey.isLeft()) {
> serializedTaskInformation = new
> TaskDeploymentDescriptor.NonOffloaded<>(taskInformationOrBlobKey.left());
> } else {
> serializedTaskInformation = new
> TaskDeploymentDescriptor.Offloaded<>(taskInformationOrBlobKey.right());
> }
> {code}
> serializedJobInformation and serializedTaskInformation are not necessarily
> shared the class NonOffloaded or Offloaded
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11409) Make `ProcessFunction`, `ProcessWindowFunction` and etc. pure interfaces
[
https://issues.apache.org/jira/browse/FLINK-11409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757323#comment-16757323
]
Dawid Wysakowicz commented on FLINK-11409:
--
Ok, now I see your point, I don't entirely agree though this the pattern that
we use/encourage users to use for {{Rich}} versions, as for {{MapFunction}},
{{FilterFunction}} etc. we have their corresponding {{RichMapFunction}},
{{RichFilterFunction}}.
Now I understand your suggestion I would say I like it in general, as it tries
to separate those two (or even more) topics. I'm afraid though the current
policy is that we don't break event the {{@PublicEvolving}} contract...
> Make `ProcessFunction`, `ProcessWindowFunction` and etc. pure interfaces
>
>
> Key: FLINK-11409
> URL: https://issues.apache.org/jira/browse/FLINK-11409
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
>Reporter: Kezhu Wang
>Assignee: Kezhu Wang
>Priority: Major
> Labels: Breaking-Change
>
> I found these functions express no opinionated demands from implementing
> classes. It would be nice to implement as interfaces not abstract classes as
> abstract class is intrusive and hampers caller user cases. For example,
> client can't write an `AbstractFlinkRichFunction` to unify lifecycle
> management for all data processing functions in easy way.
> I dive history of some of these functions, and find that some functions were
> converted as abstract class from interface due to default method
> implementation, such as `ProcessFunction` and `CoProcessFunction` were
> converted to abstract classes in FLINK-4460 which predate -FLINK-7242-. After
> -FLINK-7242-, [Java 8 default
> method|https://docs.oracle.com/javase/tutorial/java/IandI/defaultmethods.html]
> would be a better solution.
> I notice also that some functions which are introduced after -FLINK-7242-,
> such as `ProcessJoinFunction`, are implemented as abstract classes. I think
> it would be better to establish a well-known principle to guide both api
> authors and callers of data processing functions.
> Personally, I prefer interface for all exported function callbacks for the
> reason I express in first paragraph.
> Besides this, with `AbstractRichFunction` and interfaces for data processing
> functions I think lots of rich data processing functions can be eliminated as
> they are plain classes extending `AbstractRichFunction` and implementing data
> processing interfaces, clients can write this in one line code with clear
> intention of both data processing and lifecycle management.
> Following is a possible incomplete list of data processing functions
> implemented as abstract classes currently:
> * `ProcessFunction`, `KeyedProcessFunction`, `CoProcessFunction` and
> `ProcessJoinFunction`
> * `ProcessWindowFunction` and `ProcessAllWindowFunction`
> * `BaseBroadcastProcessFunction`, `BroadcastProcessFunction` and
> `KeyedBroadcastProcessFunction`
> All above functions are annotated with `@PublicEvolving`, making they
> interfaces won't break Flink's compatibility guarantee but compatibility is
> still a big consideration to evaluate this proposal.
> Any thoughts on this proposal ? Please must comment out.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-11496) FlinkS3 FileSysten is not handling multiple local temp directories
[ https://issues.apache.org/jira/browse/FLINK-11496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elango Ganesan updated FLINK-11496: --- Description: We think S3 Flink Filesystem when creating localTemp directory is not splitting and handling the availability of multiple local temp directories . As a result we are seeing exception mentioned below any time we run in a EC2 instance type with more than one ephemeral drive or EBS volume. [https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/FlinkS3FileSystem.java#L101] Timestamp: 2019-01-29, 12:42:39 java.nio.file.NoSuchFileException: /mnt/yarn/usercache/hadoop/appcache/application_1548598173158_0004,/mnt1/yarn/usercache/hadoop/appcache/application_1548598173158_0004/.tmp_072167ee-6432-412c-809a-bd0599961cf0 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434) at java.nio.file.Files.newOutputStream(Files.java:216) at org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator.apply(RefCountedTmpFileCreator.java:80) at org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator.apply(RefCountedTmpFileCreator.java:39) at org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream.openNew(RefCountedBufferingFileStream.java:174) at org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.boundedBufferingFileStream(S3RecoverableFsDataOutputStream.java:271) at org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.newStream(S3RecoverableFsDataOutputStream.java:236) at org.apache.flink.fs.s3.common.writer.S3RecoverableWriter.open(S3RecoverableWriter.java:78) was: S3 Flink Filesystem when creating localTemp directory is not splitting and handling the availability of multiple local temp directories . As a result we are seeing exception mentioned below any time we run in a EC2 instance type with more than one ephemeral drive or EBS volume. https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/FlinkS3FileSystem.java#L101 Timestamp: 2019-01-29, 12:42:39 java.nio.file.NoSuchFileException: /mnt/yarn/usercache/hadoop/appcache/application_1548598173158_0004,/mnt1/yarn/usercache/hadoop/appcache/application_1548598173158_0004/.tmp_072167ee-6432-412c-809a-bd0599961cf0 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434) at java.nio.file.Files.newOutputStream(Files.java:216) at org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator.apply(RefCountedTmpFileCreator.java:80) at org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator.apply(RefCountedTmpFileCreator.java:39) at org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream.openNew(RefCountedBufferingFileStream.java:174) at org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.boundedBufferingFileStream(S3RecoverableFsDataOutputStream.java:271) at org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.newStream(S3RecoverableFsDataOutputStream.java:236) at org.apache.flink.fs.s3.common.writer.S3RecoverableWriter.open(S3RecoverableWriter.java:78) > FlinkS3 FileSysten is not handling multiple local temp directories > -- > > Key: FLINK-11496 > URL: https://issues.apache.org/jira/browse/FLINK-11496 > Project: Flink > Issue Type: Bug >Reporter: Elango Ganesan >Priority: Major > Fix For: 1.7.1 > > > We think S3 Flink Filesystem when creating localTemp directory is not > splitting and handling the availability of multiple local temp directories . > As a result we are seeing exception mentioned below any time we run in a EC2 > instance type with more than one ephemeral drive or EBS volume. > > [https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/FlinkS3FileSystem.java#L101] > > Timestamp: 2019-01-29, 12:42:39 > java.nio.file.NoSuchFileException: > /mnt/yarn/usercache/hadoop/appcache/application_154859817315
[jira] [Comment Edited] (FLINK-11409) Make `ProcessFunction`, `ProcessWindowFunction` and etc. pure interfaces
[
https://issues.apache.org/jira/browse/FLINK-11409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757323#comment-16757323
]
Dawid Wysakowicz edited comment on FLINK-11409 at 1/31/19 3:00 PM:
---
Ok, now I see your point, I don't entirely agree though this is the pattern
that we use/encourage users to use for {{Rich}} versions, as for
{{MapFunction}}, {{FilterFunction}} etc. we have their corresponding
{{RichMapFunction}}, {{RichFilterFunction}}.
Now I understand your suggestion I would say I like it in general, as it tries
to separate those two (or even more) topics. I'm afraid though the current
policy is that we don't break event the {{@PublicEvolving}} contract...
was (Author: dawidwys):
Ok, now I see your point, I don't entirely agree though this the pattern that
we use/encourage users to use for {{Rich}} versions, as for {{MapFunction}},
{{FilterFunction}} etc. we have their corresponding {{RichMapFunction}},
{{RichFilterFunction}}.
Now I understand your suggestion I would say I like it in general, as it tries
to separate those two (or even more) topics. I'm afraid though the current
policy is that we don't break event the {{@PublicEvolving}} contract...
> Make `ProcessFunction`, `ProcessWindowFunction` and etc. pure interfaces
>
>
> Key: FLINK-11409
> URL: https://issues.apache.org/jira/browse/FLINK-11409
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
>Reporter: Kezhu Wang
>Assignee: Kezhu Wang
>Priority: Major
> Labels: Breaking-Change
>
> I found these functions express no opinionated demands from implementing
> classes. It would be nice to implement as interfaces not abstract classes as
> abstract class is intrusive and hampers caller user cases. For example,
> client can't write an `AbstractFlinkRichFunction` to unify lifecycle
> management for all data processing functions in easy way.
> I dive history of some of these functions, and find that some functions were
> converted as abstract class from interface due to default method
> implementation, such as `ProcessFunction` and `CoProcessFunction` were
> converted to abstract classes in FLINK-4460 which predate -FLINK-7242-. After
> -FLINK-7242-, [Java 8 default
> method|https://docs.oracle.com/javase/tutorial/java/IandI/defaultmethods.html]
> would be a better solution.
> I notice also that some functions which are introduced after -FLINK-7242-,
> such as `ProcessJoinFunction`, are implemented as abstract classes. I think
> it would be better to establish a well-known principle to guide both api
> authors and callers of data processing functions.
> Personally, I prefer interface for all exported function callbacks for the
> reason I express in first paragraph.
> Besides this, with `AbstractRichFunction` and interfaces for data processing
> functions I think lots of rich data processing functions can be eliminated as
> they are plain classes extending `AbstractRichFunction` and implementing data
> processing interfaces, clients can write this in one line code with clear
> intention of both data processing and lifecycle management.
> Following is a possible incomplete list of data processing functions
> implemented as abstract classes currently:
> * `ProcessFunction`, `KeyedProcessFunction`, `CoProcessFunction` and
> `ProcessJoinFunction`
> * `ProcessWindowFunction` and `ProcessAllWindowFunction`
> * `BaseBroadcastProcessFunction`, `BroadcastProcessFunction` and
> `KeyedBroadcastProcessFunction`
> All above functions are annotated with `@PublicEvolving`, making they
> interfaces won't break Flink's compatibility guarantee but compatibility is
> still a big consideration to evaluate this proposal.
> Any thoughts on this proposal ? Please must comment out.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] tvielgouarin commented on issue #7604: [Flink 11249] FlinkKafkaProducer011 can not be migrated to FlinkKafkaProducer
tvielgouarin commented on issue #7604: [Flink 11249] FlinkKafkaProducer011 can not be migrated to FlinkKafkaProducer URL: https://github.com/apache/flink/pull/7604#issuecomment-459376829 hi @pnowojski , The Travis check doesn't pass because the migration test of `FlinkKakfkaProducer011` --> `FlinkKafkaProducer` raise an Exception : `org.apache.flink.util.StateMigrationException: The new state serializer for operator state must not be incompatible.` . On master this error seems logic because the bug `FlinkKafkaProducer011` can not be migrated to `FlinkKafkaProducer`isn't fixed yet. However I did try with @yanghua PR and the exception persist. I guess ever we can't initialize the state of a different operator with testHarness or the bug isn't fixed even after the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] tvielgouarin edited a comment on issue #7604: [Flink 11249] FlinkKafkaProducer011 can not be migrated to FlinkKafkaProducer
tvielgouarin edited a comment on issue #7604: [Flink 11249] FlinkKafkaProducer011 can not be migrated to FlinkKafkaProducer URL: https://github.com/apache/flink/pull/7604#issuecomment-459376829 hi @pnowojski , The Travis check doesn't pass because the migration test of `FlinkKakfkaProducer011` --> `FlinkKafkaProducer` raises an Exception : `org.apache.flink.util.StateMigrationException: The new state serializer for operator state must not be incompatible.` . On master this error seems logic because the bug `FlinkKafkaProducer011` can not be migrated to `FlinkKafkaProducer`isn't fixed yet. However I did try with @yanghua PR and the exception persists. I guess ever we can't initialize the state of a different operator with testHarness or the bug isn't fixed even after the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] tillrohrmann commented on a change in pull request #7613: [FLINK-11370][test]Check and port ZooKeeperLeaderElectionITCase to ne…
tillrohrmann commented on a change in pull request #7613:
[FLINK-11370][test]Check and port ZooKeeperLeaderElectionITCase to ne…
URL: https://github.com/apache/flink/pull/7613#discussion_r252704575
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/leaderelection/LeaderChangeClusterComponentsTest.java
##
@@ -68,8 +69,8 @@ public static void setupClass() throws Exception {
miniCluster = new TestingMiniCluster(
new MiniClusterConfiguration.Builder()
+ .setNumTaskManagers(NUM_TMS)
.setNumSlotsPerTaskManager(SLOTS_PER_TM)
- .setNumSlotsPerTaskManager(NUM_TMS)
Review comment:
Good catch. Thanks for fixing it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-11451) Move *QueryConfig and TableDescriptor to flink-table-api-java
[
https://issues.apache.org/jira/browse/FLINK-11451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757322#comment-16757322
]
Timo Walther commented on FLINK-11451:
--
Thank you [~xueyu].
1. In your case, {{private[table]}} means a private constructor in Java. But
sometimes you still need a way of extracting an internal field, so it really
depends on common sense and the use case.
2. Yes, this is a good point. When we port {{ExternalCatalogTable}} we will
also need it in {{flink-table-common}}. Let's put this descriptor there. The
same package is also split in other modules, for example, the Kafka connector
has also a descriptor in this package.
> Move *QueryConfig and TableDescriptor to flink-table-api-java
> -
>
> Key: FLINK-11451
> URL: https://issues.apache.org/jira/browse/FLINK-11451
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: xueyu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Move QueryConfig, BatchQueryConfig, StreamQueryConfig, TableDescriptor in
> flink-table-api-java.
> Unblocks TableEnvironment interface task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] pnowojski merged pull request #7570: [FLINK-11422] Prefer testing class to mock StreamTask in AbstractStre…
pnowojski merged pull request #7570: [FLINK-11422] Prefer testing class to mock StreamTask in AbstractStre… URL: https://github.com/apache/flink/pull/7570 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] pnowojski commented on issue #7570: [FLINK-11422] Prefer testing class to mock StreamTask in AbstractStre…
pnowojski commented on issue #7570: [FLINK-11422] Prefer testing class to mock StreamTask in AbstractStre… URL: https://github.com/apache/flink/pull/7570#issuecomment-459380050 Thanks @TisonKun, merged :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-11422) Prefer testing class to mock StreamTask in AbstractStreamOperatorTestHarness
[ https://issues.apache.org/jira/browse/FLINK-11422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski closed FLINK-11422. -- Resolution: Fixed merged commit e43d554 into apache:master > Prefer testing class to mock StreamTask in AbstractStreamOperatorTestHarness > > > Key: FLINK-11422 > URL: https://issues.apache.org/jira/browse/FLINK-11422 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.8.0 >Reporter: TisonKun >Assignee: TisonKun >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit closed pull request #7527: [FLINK-11106] [yarn] Remove legacy flink-yarn component
asfgit closed pull request #7527: [FLINK-11106] [yarn] Remove legacy flink-yarn component URL: https://github.com/apache/flink/pull/7527 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (FLINK-11106) Remove legacy flink-yarn component
[
https://issues.apache.org/jira/browse/FLINK-11106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann resolved FLINK-11106.
---
Resolution: Fixed
Fixed via
e97967278904cb2706a6514ad0a13d50cc389b20
ef5242405f7977f9c330eb99e1e9b86f3c7d252b
343d84bd35b64540e18a4b811b75c87be06c0332
> Remove legacy flink-yarn component
> --
>
> Key: FLINK-11106
> URL: https://issues.apache.org/jira/browse/FLINK-11106
> Project: Flink
> Issue Type: Sub-task
> Components: Client, Cluster Management, YARN
>Affects Versions: 1.8.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After flink-yarn tests ported to new codebase, we are able to remove legacy
> flink-yarn component, including
> - {{ApplicationClient.scala}}
> - {{YarnJobManager.scala}}
> - {{YarnMessages.scala}}
> - {{YarnTaskManager.scala}}
> - {{org.apache.flink.yarn.messages.*}}
> - {{LegacyYarnClusterDescriptor.java}}
> - {{RegisteredYarnWorkerNode.java}}
> - {{YarnApplicationMasterRunner.java}}
> - {{YarnClusterClient.java}}
> - {{YarnContainerInLaunch.java}}
> - {{YarnFlinkResourceManager.java}}
> - {{YarnResourceManagerCallbackHandler.java}}
> - {{YarnTaskManagerRunnerFactory.java}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11445) Deprecate static methods in TableEnvironments
[
https://issues.apache.org/jira/browse/FLINK-11445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757355#comment-16757355
]
sunjincheng commented on FLINK-11445:
-
Sounds good to me, looking forward the PR.
> Deprecate static methods in TableEnvironments
> -
>
> Key: FLINK-11445
> URL: https://issues.apache.org/jira/browse/FLINK-11445
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Hequn Cheng
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Direct to the {{Batch/StreamTableEnvionrment.create()}} approach. The
> {{create()}} method must not necessarily already perform a planner discovery.
> We can hard-code the target table environment for now.
> {{TableEnvironment.create()}} is not supported yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] tillrohrmann commented on a change in pull request #7613: [FLINK-11370][test]Check and port ZooKeeperLeaderElectionITCase to ne…
tillrohrmann commented on a change in pull request #7613:
[FLINK-11370][test]Check and port ZooKeeperLeaderElectionITCase to ne…
URL: https://github.com/apache/flink/pull/7613#discussion_r252713595
##
File path:
flink-tests/src/test/java/org/apache/flink/test/runtime/leaderelection/ZooKeeperLeaderElectionITCase.java
##
@@ -1,301 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.test.runtime.leaderelection;
-
-import org.apache.flink.api.common.JobExecutionResult;
-import org.apache.flink.configuration.AkkaOptions;
-import org.apache.flink.configuration.ConfigConstants;
-import org.apache.flink.configuration.Configuration;
-import org.apache.flink.configuration.HighAvailabilityOptions;
-import org.apache.flink.configuration.TaskManagerOptions;
-import org.apache.flink.runtime.akka.AkkaUtils;
-import org.apache.flink.runtime.client.JobClient;
-import org.apache.flink.runtime.instance.ActorGateway;
-import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
-import org.apache.flink.runtime.jobgraph.DistributionPattern;
-import org.apache.flink.runtime.jobgraph.JobGraph;
-import org.apache.flink.runtime.jobgraph.JobVertex;
-import org.apache.flink.runtime.jobmanager.Tasks;
-import org.apache.flink.runtime.jobmanager.scheduler.SlotSharingGroup;
-import org.apache.flink.runtime.messages.JobManagerMessages;
-import org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster;
-import org.apache.flink.runtime.testingUtils.TestingCluster;
-import org.apache.flink.runtime.testingUtils.TestingJobManagerMessages;
-import
org.apache.flink.runtime.testingUtils.TestingJobManagerMessages.WaitForAllVerticesToBeRunningOrFinished;
-import org.apache.flink.runtime.testingUtils.TestingUtils;
-import org.apache.flink.runtime.testutils.ZooKeeperTestUtils;
-import org.apache.flink.util.TestLogger;
-
-import akka.actor.ActorSystem;
-import akka.actor.Kill;
-import akka.actor.PoisonPill;
-import org.apache.curator.test.TestingServer;
-import org.junit.AfterClass;
-import org.junit.BeforeClass;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.TemporaryFolder;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.File;
-import java.util.UUID;
-
-import scala.concurrent.Await;
-import scala.concurrent.Future;
-import scala.concurrent.duration.Deadline;
-import scala.concurrent.duration.FiniteDuration;
-import scala.concurrent.impl.Promise;
-
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.fail;
-
-/**
- * Test the election of a new JobManager leader.
- */
-public class ZooKeeperLeaderElectionITCase extends TestLogger {
-
- private static final FiniteDuration timeout =
TestingUtils.TESTING_DURATION();
-
- private static TestingServer zkServer;
-
- @Rule
- public TemporaryFolder tempFolder = new TemporaryFolder();
-
- @BeforeClass
- public static void setup() throws Exception {
- zkServer = new TestingServer(true);
- }
-
- @AfterClass
- public static void tearDown() throws Exception {
- if (zkServer != null) {
- zkServer.close();
- zkServer = null;
- }
- }
-
- /**
-* Tests that the TaskManagers successfully register at the new leader
once the old leader
-* is terminated.
-*/
- @Test
- public void testTaskManagerRegistrationAtReelectedLeader() throws
Exception {
- File rootFolder = tempFolder.getRoot();
-
- Configuration configuration =
ZooKeeperTestUtils.createZooKeeperHAConfig(
- zkServer.getConnectString(),
- rootFolder.getPath());
- configuration.setString(HighAvailabilityOptions.HA_CLUSTER_ID,
UUID.randomUUID().toString());
-
- int numJMs = 10;
- int numTMs = 3;
-
-
configuration.setInteger(ConfigConstants.LOCAL_NUMBER_JOB_MANAGER, numJMs);
-
configuration.setInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER, numTMs);
-
- TestingCluster cluster = new TestingCluster(configuration);
[GitHub] tillrohrmann commented on a change in pull request #7613: [FLINK-11370][test]Check and port ZooKeeperLeaderElectionITCase to ne…
tillrohrmann commented on a change in pull request #7613:
[FLINK-11370][test]Check and port ZooKeeperLeaderElectionITCase to ne…
URL: https://github.com/apache/flink/pull/7613#discussion_r252712333
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/leaderelection/LeaderChangeClusterComponentsTest.java
##
@@ -141,6 +142,16 @@ public void testReelectionOfJobMaster() throws Exception {
assertThat(jobResult.isSuccess(), is(true));
}
+ @Test
+ public void testTaskManagerRegisterReelectionOfResourceManager() throws
Exception {
+
+ assertTrue(miniCluster.requestTaskManagerInfo().get().size() ==
NUM_TMS);
Review comment:
I would suggest to use `assertThat` and Hamcrest matchers for testing
because they give better failure messages.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] tillrohrmann commented on a change in pull request #7613: [FLINK-11370][test]Check and port ZooKeeperLeaderElectionITCase to ne…
tillrohrmann commented on a change in pull request #7613:
[FLINK-11370][test]Check and port ZooKeeperLeaderElectionITCase to ne…
URL: https://github.com/apache/flink/pull/7613#discussion_r252705212
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/leaderelection/LeaderChangeClusterComponentsTest.java
##
@@ -141,6 +142,16 @@ public void testReelectionOfJobMaster() throws Exception {
assertThat(jobResult.isSuccess(), is(true));
}
+ @Test
+ public void testTaskManagerRegisterReelectionOfResourceManager() throws
Exception {
+
+ assertTrue(miniCluster.requestTaskManagerInfo().get().size() ==
NUM_TMS);
+
highAvailabilityServices.revokeResourceManagerLeadership().get();
+ highAvailabilityServices.grantResourceManagerLeadership().get();
+ Thread.sleep(500);
Review comment:
We should try to use sleeps. I would suggest to use a polling loop with a
lower sleep interval to make the test a bit more responsive.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Myasuka commented on issue #7600: [FLINK-11368][tests] Port TaskManagerStartupTest to new code base
Myasuka commented on issue #7600: [FLINK-11368][tests] Port TaskManagerStartupTest to new code base URL: https://github.com/apache/flink/pull/7600#issuecomment-459390193 @zentol The logic of previous `testStartupWhenTaskmanagerActorPortIsUsed` test was a bit strange, because the `port` given to `#runTaskManager()` method was already be occupied by task-manager's server socket. If task-manager could not occupy that port, it would throw `BindException(s"Unable to allocate port for TaskManager.")` before calling `#runTaskManager()`. On the other hand, `TaskManagerRunner` verify port whether available through `#createRpcService`, which is before `#startTaskManager()`. And `BootstrapToolsTest#testActorSystemInstantiationFailureWhenPortOccupied()` should already cover this case. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-11449) Uncouple the Expression class from RexNodes
[
https://issues.apache.org/jira/browse/FLINK-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757410#comment-16757410
]
Dawid Wysakowicz commented on FLINK-11449:
--
In general I like the proposal from [~twalthr], this indeed would allow us in
the future to unify handling of built-in functions and UDFs.
I have just two comments:
* do we need the {{ValueLiteralExpression(Object, TypeInformation)}}? I think
we could express it with {{CallExpression(CAST,
ValueLiteralExpression(Object))}} which would be inline with SQL.
* don't we need to pass {{UserDefinedAggFunctionDefinition}} and
{{UserDefinedTableFunctionDefinition}} to pass through the {{TypeInformation}}
(resultTypeInfo, accTypeInfo) that was derived during function registration?
> Uncouple the Expression class from RexNodes
> ---
>
> Key: FLINK-11449
> URL: https://issues.apache.org/jira/browse/FLINK-11449
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: sunjincheng
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Calcite will not be part of any API module anymore. Therefore, RexNode
> translation must happen in a different layer. This issue will require a new
> design document.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9920) BucketingSinkFaultToleranceITCase fails on travis
[
https://issues.apache.org/jira/browse/FLINK-9920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757409#comment-16757409
]
Yun Tang commented on FLINK-9920:
-
With more detailed messages, I believe this bug happened due to the test file
verifying whether {{truncate}} work did not be deleted.
> BucketingSinkFaultToleranceITCase fails on travis
> -
>
> Key: FLINK-9920
> URL: https://issues.apache.org/jira/browse/FLINK-9920
> Project: Flink
> Issue Type: Bug
> Components: filesystem-connector
>Affects Versions: 1.6.0, 1.7.0
>Reporter: Chesnay Schepler
>Assignee: Aljoscha Krettek
>Priority: Critical
> Labels: test-stability
> Fix For: 1.8.0
>
>
> https://travis-ci.org/zentol/flink/jobs/407021898
> {code}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 13.082 sec
> <<< FAILURE! - in
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSinkFaultToleranceITCase
> runCheckpointedProgram(org.apache.flink.streaming.connectors.fs.bucketing.BucketingSinkFaultToleranceITCase)
> Time elapsed: 5.696 sec <<< FAILURE!
> java.lang.AssertionError: Read line does not match expected pattern.
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSinkFaultToleranceITCase.postSubmit(BucketingSinkFaultToleranceITCase.java:182)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] hequn8128 opened a new pull request #7622: [FLINK-11445][table] Deprecate static methods in TableEnvironments
hequn8128 opened a new pull request #7622: [FLINK-11445][table] Deprecate static methods in TableEnvironments URL: https://github.com/apache/flink/pull/7622 ## What is the purpose of the change This pull request deprecates static methods in TableEnvironments and direct to the java/scala Batch/StreamTableEnvionrment.create() approach. ## Brief change log - Deprecated the getTableEnvironment method in TableEnvironment. - Deprecate the constructors of all table environments. Because of scala, deprecation messages of constructors will only be shown at compile time - Add create method in all 4 TableEnvironments: java/scala Batch/StreamTableEnvionrment. - Replace all old getTableEnvironment methods to the new create methods in all places in Flink. - Adapt document ## Verifying this change This change is already covered by existing tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (docs) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-11445) Deprecate static methods in TableEnvironments
[
https://issues.apache.org/jira/browse/FLINK-11445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-11445:
---
Labels: pull-request-available (was: )
> Deprecate static methods in TableEnvironments
> -
>
> Key: FLINK-11445
> URL: https://issues.apache.org/jira/browse/FLINK-11445
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Hequn Cheng
>Priority: Major
> Labels: pull-request-available
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Direct to the {{Batch/StreamTableEnvionrment.create()}} approach. The
> {{create()}} method must not necessarily already perform a planner discovery.
> We can hard-code the target table environment for now.
> {{TableEnvironment.create()}} is not supported yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-11449) Uncouple the Expression class from RexNodes
[
https://issues.apache.org/jira/browse/FLINK-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757410#comment-16757410
]
Dawid Wysakowicz edited comment on FLINK-11449 at 1/31/19 4:05 PM:
---
In general I like the proposal from [~twalthr], this indeed would allow us in
the future to unify handling of built-in functions and UDFs.
I have just two comments:
* do we need the {{ValueLiteralExpression(Object, TypeInformation)}}? I think
we could express it with {{CallExpression(CAST,
ValueLiteralExpression(Object))}} which would be in line with SQL.
* don't we need to pass {{UserDefinedAggFunctionDefinition}} and
{{UserDefinedTableFunctionDefinition}} to pass through the {{TypeInformation}}
(resultTypeInfo, accTypeInfo) that was derived during function registration?
was (Author: dawidwys):
In general I like the proposal from [~twalthr], this indeed would allow us in
the future to unify handling of built-in functions and UDFs.
I have just two comments:
* do we need the {{ValueLiteralExpression(Object, TypeInformation)}}? I think
we could express it with {{CallExpression(CAST,
ValueLiteralExpression(Object))}} which would be inline with SQL.
* don't we need to pass {{UserDefinedAggFunctionDefinition}} and
{{UserDefinedTableFunctionDefinition}} to pass through the {{TypeInformation}}
(resultTypeInfo, accTypeInfo) that was derived during function registration?
> Uncouple the Expression class from RexNodes
> ---
>
> Key: FLINK-11449
> URL: https://issues.apache.org/jira/browse/FLINK-11449
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: sunjincheng
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Calcite will not be part of any API module anymore. Therefore, RexNode
> translation must happen in a different layer. This issue will require a new
> design document.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-11449) Uncouple the Expression class from RexNodes
[
https://issues.apache.org/jira/browse/FLINK-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757410#comment-16757410
]
Dawid Wysakowicz edited comment on FLINK-11449 at 1/31/19 4:05 PM:
---
In general I like the proposal from [~twalthr], this indeed would allow us in
the future to unify handling of built-in functions and UDFs.
I have just two comments:
* do we need the {{ValueLiteralExpression(Object, TypeInformation)}}? I think
we could express it with {{CallExpression(CAST,
ValueLiteralExpression(Object))}} which would be in line with SQL.
* don't we need {{UserDefinedAggFunctionDefinition}} and
{{UserDefinedTableFunctionDefinition}} to pass through the {{TypeInformation}}
(resultTypeInfo, accTypeInfo) that was derived during function registration?
was (Author: dawidwys):
In general I like the proposal from [~twalthr], this indeed would allow us in
the future to unify handling of built-in functions and UDFs.
I have just two comments:
* do we need the {{ValueLiteralExpression(Object, TypeInformation)}}? I think
we could express it with {{CallExpression(CAST,
ValueLiteralExpression(Object))}} which would be in line with SQL.
* don't we need to pass {{UserDefinedAggFunctionDefinition}} and
{{UserDefinedTableFunctionDefinition}} to pass through the {{TypeInformation}}
(resultTypeInfo, accTypeInfo) that was derived during function registration?
> Uncouple the Expression class from RexNodes
> ---
>
> Key: FLINK-11449
> URL: https://issues.apache.org/jira/browse/FLINK-11449
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: sunjincheng
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Calcite will not be part of any API module anymore. Therefore, RexNode
> translation must happen in a different layer. This issue will require a new
> design document.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-11496) FlinkS3 FileSystem is not handling multiple local temp directories
[ https://issues.apache.org/jira/browse/FLINK-11496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elango Ganesan updated FLINK-11496: --- Summary: FlinkS3 FileSystem is not handling multiple local temp directories (was: FlinkS3 FileSysten is not handling multiple local temp directories) > FlinkS3 FileSystem is not handling multiple local temp directories > -- > > Key: FLINK-11496 > URL: https://issues.apache.org/jira/browse/FLINK-11496 > Project: Flink > Issue Type: Bug >Reporter: Elango Ganesan >Priority: Major > Fix For: 1.7.1 > > > We think S3 Flink Filesystem when creating localTemp directory is not > splitting and handling the availability of multiple local temp directories . > As a result we are seeing exception mentioned below any time we run in a EC2 > instance type with more than one ephemeral drive or EBS volume. > > [https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/FlinkS3FileSystem.java#L101] > > Timestamp: 2019-01-29, 12:42:39 > java.nio.file.NoSuchFileException: > /mnt/yarn/usercache/hadoop/appcache/application_1548598173158_0004,/mnt1/yarn/usercache/hadoop/appcache/application_1548598173158_0004/.tmp_072167ee-6432-412c-809a-bd0599961cf0 > at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) > at > java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434) > at java.nio.file.Files.newOutputStream(Files.java:216) > at > org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator.apply(RefCountedTmpFileCreator.java:80) > at > org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator.apply(RefCountedTmpFileCreator.java:39) > at > org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream.openNew(RefCountedBufferingFileStream.java:174) > at > org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.boundedBufferingFileStream(S3RecoverableFsDataOutputStream.java:271) > at > org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.newStream(S3RecoverableFsDataOutputStream.java:236) > at > org.apache.flink.fs.s3.common.writer.S3RecoverableWriter.open(S3RecoverableWriter.java:78) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-11496) Flink S3 FileSystem is not handling multiple local temp directories
[ https://issues.apache.org/jira/browse/FLINK-11496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elango Ganesan updated FLINK-11496: --- Summary: Flink S3 FileSystem is not handling multiple local temp directories (was: FlinkS3 FileSystem is not handling multiple local temp directories) > Flink S3 FileSystem is not handling multiple local temp directories > --- > > Key: FLINK-11496 > URL: https://issues.apache.org/jira/browse/FLINK-11496 > Project: Flink > Issue Type: Bug >Reporter: Elango Ganesan >Priority: Major > Fix For: 1.7.1 > > > We think S3 Flink Filesystem when creating localTemp directory is not > splitting and handling the availability of multiple local temp directories . > As a result we are seeing exception mentioned below any time we run in a EC2 > instance type with more than one ephemeral drive or EBS volume. > > [https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/FlinkS3FileSystem.java#L101] > > Timestamp: 2019-01-29, 12:42:39 > java.nio.file.NoSuchFileException: > /mnt/yarn/usercache/hadoop/appcache/application_1548598173158_0004,/mnt1/yarn/usercache/hadoop/appcache/application_1548598173158_0004/.tmp_072167ee-6432-412c-809a-bd0599961cf0 > at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) > at > java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434) > at java.nio.file.Files.newOutputStream(Files.java:216) > at > org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator.apply(RefCountedTmpFileCreator.java:80) > at > org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator.apply(RefCountedTmpFileCreator.java:39) > at > org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream.openNew(RefCountedBufferingFileStream.java:174) > at > org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.boundedBufferingFileStream(S3RecoverableFsDataOutputStream.java:271) > at > org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.newStream(S3RecoverableFsDataOutputStream.java:236) > at > org.apache.flink.fs.s3.common.writer.S3RecoverableWriter.open(S3RecoverableWriter.java:78) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] TisonKun opened a new pull request #7623: [FLINK-11386] Remove legacy ContaineredJobManager
TisonKun opened a new pull request #7623: [FLINK-11386] Remove legacy ContaineredJobManager URL: https://github.com/apache/flink/pull/7623 ## What is the purpose of the change Previously, we extend ContaineredJobManager to implement MesosJobManager and YarnJobManager. In FLIP-6 we use ClusterEntrypoint as a new abstraction. After legacy MesosJobManager and YarnJobManager removed, we can now safely drop ContaineredJobManager. ## Brief change log Delete ContaineredJobManager.scala ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) cc @tillrohrmann This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-11386) Remove legacy ContaineredJobManager
[
https://issues.apache.org/jira/browse/FLINK-11386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-11386:
---
Labels: pull-request-available (was: )
> Remove legacy ContaineredJobManager
> ---
>
> Key: FLINK-11386
> URL: https://issues.apache.org/jira/browse/FLINK-11386
> Project: Flink
> Issue Type: Sub-task
> Components: Cluster Management
>Affects Versions: 1.8.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> Previously, we extend {{ContaineredJobManager}} to implement
> {{MesosJobManager}} and {{YarnJobManager}}. In FLIP-6 we use
> {{ClusterEntrypoint}} as a new abstraction.
> After legacy {{MesosJobManager}} and {{YarnJobManager}} removed, we can
> safely drop {{ContaineredJobManager}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-11361) Check and port RecoveryITCase to new code base if necessary
[
https://issues.apache.org/jira/browse/FLINK-11361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun closed FLINK-11361.
Resolution: Duplicate
> Check and port RecoveryITCase to new code base if necessary
> ---
>
> Key: FLINK-11361
> URL: https://issues.apache.org/jira/browse/FLINK-11361
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Till Rohrmann
>Priority: Major
>
> Check and port {{RecoveryITCase}} to new code base if necessary.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] jgrier commented on issue #7099: [FLINK-10887] [jobmaster] Add source watermark tracking to the JobMaster
jgrier commented on issue #7099: [FLINK-10887] [jobmaster] Add source watermark tracking to the JobMaster URL: https://github.com/apache/flink/pull/7099#issuecomment-459420616 @tweise @aljoscha @StephanEwen I think this is in a good state for final review and merge. Take a look when you get the chance please. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-11497) Remove invalid test JobManagerLeaderElectionTest
TisonKun created FLINK-11497: Summary: Remove invalid test JobManagerLeaderElectionTest Key: FLINK-11497 URL: https://issues.apache.org/jira/browse/FLINK-11497 Project: Flink Issue Type: Sub-task Components: Tests Affects Versions: 1.8.0 Reporter: TisonKun Assignee: TisonKun Fix For: 1.8.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] TisonKun opened a new pull request #7624: [FLINK-11497] Remove invalid test JobManagerLeaderElectionTest
TisonKun opened a new pull request #7624: [FLINK-11497] Remove invalid test JobManagerLeaderElectionTest URL: https://github.com/apache/flink/pull/7624 ## What is the purpose of the change LeaderElection functions are tested by `LeaderChangeClusterComponentsTest` and `EmbeddedHaServicesTest#testJobManagerLeaderElection` and so on. We can directly drop invalid test `JobManagerLeaderElectionTest` ## Brief change log Delete `JobManagerLeaderElectionTest.java` ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector:(no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) cc @tillrohrmann This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-11497) Remove invalid test JobManagerLeaderElectionTest
[ https://issues.apache.org/jira/browse/FLINK-11497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-11497: --- Labels: pull-request-available (was: ) > Remove invalid test JobManagerLeaderElectionTest > > > Key: FLINK-11497 > URL: https://issues.apache.org/jira/browse/FLINK-11497 > Project: Flink > Issue Type: Sub-task > Components: Tests >Affects Versions: 1.8.0 >Reporter: TisonKun >Assignee: TisonKun >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-11498) Getting java.lang.InstantiationError when trying to install flink 1.7.1 version through ansible
[
https://issues.apache.org/jira/browse/FLINK-11498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sambi reddy updated FLINK-11498:
Remaining Estimate: 5h (was: 24h)
Original Estimate: 5h (was: 24h)
> Getting java.lang.InstantiationError when trying to install flink 1.7.1
> version through ansible
> ---
>
> Key: FLINK-11498
> URL: https://issues.apache.org/jira/browse/FLINK-11498
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.7.1
> Environment: Openstack
>Reporter: sambi reddy
>Priority: Major
> Fix For: 1.7.1
>
> Original Estimate: 5h
> Remaining Estimate: 5h
>
> We trying to install flink on our cloud with the help ansible automation. we
> were running 1.3.2 since past two years. using the similar kind of automation
> with few config changes and trying to start the cluster but failing to do so.
> Error we are seeing in the logs.
> {noformat}
> Starting the SlotManager.
> Fatal error occurred in the cluster entrypoint.
> java.lang.InstantiationError: org.apache.flink.runtime.blob.BlobKey
> at
> sun.reflect.GeneratedSerializationConstructorAccessor43.newInstance(Unknown
> Source)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:1006)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2011)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
> at java.util.ArrayList.readObject(ArrayList.java:791)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2136)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:503)
> at
> org.apache.flink.runtime.state.RetrievableStreamStateHandle.retrieveState(RetrievableStreamStateHandle.java:58)
> at
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.recoverJobGraph(ZooKeeperSubmittedJobGraphStore.java:202)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.recoverJob(Dispatcher.java:696)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.recoverJobGraphs(Dispatcher.java:681)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.recoverJobs(Dispatcher.java:662)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$null$26(Dispatcher.java:821)
> at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:72)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Shutting down BLOB cache
> Stopped BLOB server at 0.0.0.0:44036{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-11449) Uncouple the Expression class from RexNodes
[
https://issues.apache.org/jira/browse/FLINK-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757523#comment-16757523
]
sunjincheng edited comment on FLINK-11449 at 1/31/19 5:13 PM:
--
Hi, [~twalthr] Thank you for your reply, I feel that the proposal is very
meaningful to me。
Here have several places to confirm:
1. Why should FunctionDefinitions be placed in flink-table-api-java, can it be
put into flink-table-api-common?
2. All the Expression class should using in java, and expressionDsl also keep
using in scala,right?
3. The visitor which in flink-table-planner also defined in scala,right?
4. Why we should defined CAST as a function call? In my opinion, only the
funciton defined in the FunctionCatalog is defined as a funcitoncall. And
others can be defined as other Expression types? Similar to the following
definition in my prototype：
{code:java}
case class Call(functionName: String, args: Seq[Expression]) extends Expression
{
override def children: Seq[Expression] = args
}
case class Cast(child: Expression, resultType: TypeInformation[_]) extends
UnaryExpression{code}
I suggest that is it possible to divide unified built-in functions and UDFs
into multiple commits? In this JIRA we first remove the existing Expresion to
eliminate the dependency on RexNodes, is that makes sense to you?
was (Author: sunjincheng121):
Thank you for your reply, I feel that the proposal is very meaningful to me。
Here have several places to confirm:
1. Why should FunctionDefinitions be placed in flink-table-api-java, can it be
put into flink-table-api-common?
2. All the Expression class should using in java, and expressionDsl also keep
using in scala,right?
3. The visitor which in flink-table-planner also defined in scala,right?
4. Why we should defined CAST as a function call? In my opinion, only the
funciton defined in the FunctionCatalog is defined as a funcitoncall. And
others can be defined as other Expression types? Similar to the following
definition in my prototype：
{code:java}
case class Call(functionName: String, args: Seq[Expression]) extends Expression
{
override def children: Seq[Expression] = args
}
case class Cast(child: Expression, resultType: TypeInformation[_]) extends
UnaryExpression{code}
I suggest that is it possible to divide unified built-in functions and UDFs
into multiple commits? In this JIRA we first remove the existing Expresion to
eliminate the dependency on RexNodes, is that makes sense to you?
> Uncouple the Expression class from RexNodes
> ---
>
> Key: FLINK-11449
> URL: https://issues.apache.org/jira/browse/FLINK-11449
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: sunjincheng
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Calcite will not be part of any API module anymore. Therefore, RexNode
> translation must happen in a different layer. This issue will require a new
> design document.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11449) Uncouple the Expression class from RexNodes
[
https://issues.apache.org/jira/browse/FLINK-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757523#comment-16757523
]
sunjincheng commented on FLINK-11449:
-
Thank you for your reply, I feel that the proposal is very meaningful to me。
Here have several places to confirm:
1. Why should FunctionDefinitions be placed in flink-table-api-java, can it be
put into flink-table-api-common?
2. All the Expression class should using in java, and expressionDsl also keep
using in scala,right?
3. The visitor which in flink-table-planner also defined in scala,right?
4. Why we should defined CAST as a function call? In my opinion, only the
funciton defined in the FunctionCatalog is defined as a funcitoncall. And
others can be defined as other Expression types? Similar to the following
definition in my prototype：
{code:java}
case class Call(functionName: String, args: Seq[Expression]) extends Expression
{
override def children: Seq[Expression] = args
}
case class Cast(child: Expression, resultType: TypeInformation[_]) extends
UnaryExpression{code}
I suggest that is it possible to divide unified built-in functions and UDFs
into multiple commits? In this JIRA we first remove the existing Expresion to
eliminate the dependency on RexNodes, is that makes sense to you?
> Uncouple the Expression class from RexNodes
> ---
>
> Key: FLINK-11449
> URL: https://issues.apache.org/jira/browse/FLINK-11449
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: sunjincheng
>Priority: Major
>
> A more detailed description can be found in
> [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions].
> Calcite will not be part of any API module anymore. Therefore, RexNode
> translation must happen in a different layer. This issue will require a new
> design document.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11498) Getting java.lang.InstantiationError when trying to install flink 1.7.1 version through ansible
sambi reddy created FLINK-11498:
---
Summary: Getting java.lang.InstantiationError when trying to
install flink 1.7.1 version through ansible
Key: FLINK-11498
URL: https://issues.apache.org/jira/browse/FLINK-11498
Project: Flink
Issue Type: Bug
Affects Versions: 1.7.1
Environment: Openstack
Reporter: sambi reddy
Fix For: 1.7.1
We trying to install flink on our cloud with the help ansible automation. we
were running 1.3.2 since past two years. using the similar kind of automation
with few config changes and trying to start the cluster but failing to do so.
Error we are seeing in the logs.
{noformat}
Starting the SlotManager.
Fatal error occurred in the cluster entrypoint.
java.lang.InstantiationError: org.apache.flink.runtime.blob.BlobKey
at sun.reflect.GeneratedSerializationConstructorAccessor43.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:1006)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2011)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at java.util.ArrayList.readObject(ArrayList.java:791)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2136)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:503)
at
org.apache.flink.runtime.state.RetrievableStreamStateHandle.retrieveState(RetrievableStreamStateHandle.java:58)
at
org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.recoverJobGraph(ZooKeeperSubmittedJobGraphStore.java:202)
at
org.apache.flink.runtime.dispatcher.Dispatcher.recoverJob(Dispatcher.java:696)
at
org.apache.flink.runtime.dispatcher.Dispatcher.recoverJobGraphs(Dispatcher.java:681)
at
org.apache.flink.runtime.dispatcher.Dispatcher.recoverJobs(Dispatcher.java:662)
at
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$null$26(Dispatcher.java:821)
at
org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:72)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Shutting down BLOB cache
Stopped BLOB server at 0.0.0.0:44036{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11498) Getting java.lang.InstantiationError when trying to install flink 1.7.1 version through ansible
[
https://issues.apache.org/jira/browse/FLINK-11498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757529#comment-16757529
]
sambi reddy commented on FLINK-11498:
-
First, I tried to install 1.7.1 version on the cluster which is running 1.3.2,
seeing the above error, tried to install it on fresh boxes but still seeing the
same issue.
> Getting java.lang.InstantiationError when trying to install flink 1.7.1
> version through ansible
> ---
>
> Key: FLINK-11498
> URL: https://issues.apache.org/jira/browse/FLINK-11498
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.7.1
> Environment: Openstack
>Reporter: sambi reddy
>Priority: Major
> Fix For: 1.7.1
>
> Original Estimate: 5h
> Remaining Estimate: 5h
>
> We trying to install flink on our cloud with the help ansible automation. we
> were running 1.3.2 since past two years. using the similar kind of automation
> with few config changes and trying to start the cluster but failing to do so.
> Error we are seeing in the logs.
> {noformat}
> Starting the SlotManager.
> Fatal error occurred in the cluster entrypoint.
> java.lang.InstantiationError: org.apache.flink.runtime.blob.BlobKey
> at
> sun.reflect.GeneratedSerializationConstructorAccessor43.newInstance(Unknown
> Source)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:1006)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2011)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
> at java.util.ArrayList.readObject(ArrayList.java:791)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2136)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:503)
> at
> org.apache.flink.runtime.state.RetrievableStreamStateHandle.retrieveState(RetrievableStreamStateHandle.java:58)
> at
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.recoverJobGraph(ZooKeeperSubmittedJobGraphStore.java:202)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.recoverJob(Dispatcher.java:696)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.recoverJobGraphs(Dispatcher.java:681)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.recoverJobs(Dispatcher.java:662)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$null$26(Dispatcher.java:821)
> at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:72)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Shutting down BLOB cache
> Stopped BLOB server at 0.0.0.0:44036{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11499) Extend StreamingFileSink BulkFormats to support arbitrary roll policy's
Seth Wiesman created FLINK-11499: Summary: Extend StreamingFileSink BulkFormats to support arbitrary roll policy's Key: FLINK-11499 URL: https://issues.apache.org/jira/browse/FLINK-11499 Project: Flink Issue Type: Improvement Reporter: Seth Wiesman Currently when using the StreamingFilleSink Bulk-encoding formats can only be combined with the `OnCheckpointRollingPolicy`, which rolls the in-progress part file on every checkpoint. However, many bulk formats such as parquet are most efficient when written as large files; this is not possible when frequent checkpointing is enabled. Currently the only work-around is to have long checkpoint intervals which is not ideal. The StreamingFileSink should be enhanced to support arbitrary roll policy's so users may write large bulk files while retaining frequent checkpoints. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11409) Make `ProcessFunction`, `ProcessWindowFunction` and etc. pure interfaces
[
https://issues.apache.org/jira/browse/FLINK-11409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757541#comment-16757541
]
Kezhu Wang commented on FLINK-11409:
{quote}this is the pattern that we use/encourage users to use for Rich
versions, as for MapFunction, FilterFunction etc. we have their corresponding
RichMapFunction, RichFilterFunction.
{quote}
[~dawidwys] It is perfect ok for {{RichProcessFunction}} to be an abstract
class if there is one as long as we have interface version {{ProcessFunction}}.
Abstract class is not a good candidate in java to encourage callback like api
usage, in this case abstract {{ProcessFunction}} *force* subclass to not
inherit from other branch. The keypoint is that interfaces are composable while
classes are intrusive.
[~aljoscha] I saw this in {{ProcessWindowFunction.Context}} which captures
generic type {{W}} from enclosing class. I think this would not be a big issue
if we have {{ProcessWindowFunction}} as interface from day one. Actually,
{{SourceFunction.Context}} is an interface. I think compatibility plays more
significant than this as I said that "compatibility is still a big
consideration to evaluate this proposal".
Any way, it is perfect ok to not accept this proposal, after all, compatibility
is a big concern.
> Make `ProcessFunction`, `ProcessWindowFunction` and etc. pure interfaces
>
>
> Key: FLINK-11409
> URL: https://issues.apache.org/jira/browse/FLINK-11409
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
>Reporter: Kezhu Wang
>Assignee: Kezhu Wang
>Priority: Major
> Labels: Breaking-Change
>
> I found these functions express no opinionated demands from implementing
> classes. It would be nice to implement as interfaces not abstract classes as
> abstract class is intrusive and hampers caller user cases. For example,
> client can't write an `AbstractFlinkRichFunction` to unify lifecycle
> management for all data processing functions in easy way.
> I dive history of some of these functions, and find that some functions were
> converted as abstract class from interface due to default method
> implementation, such as `ProcessFunction` and `CoProcessFunction` were
> converted to abstract classes in FLINK-4460 which predate -FLINK-7242-. After
> -FLINK-7242-, [Java 8 default
> method|https://docs.oracle.com/javase/tutorial/java/IandI/defaultmethods.html]
> would be a better solution.
> I notice also that some functions which are introduced after -FLINK-7242-,
> such as `ProcessJoinFunction`, are implemented as abstract classes. I think
> it would be better to establish a well-known principle to guide both api
> authors and callers of data processing functions.
> Personally, I prefer interface for all exported function callbacks for the
> reason I express in first paragraph.
> Besides this, with `AbstractRichFunction` and interfaces for data processing
> functions I think lots of rich data processing functions can be eliminated as
> they are plain classes extending `AbstractRichFunction` and implementing data
> processing interfaces, clients can write this in one line code with clear
> intention of both data processing and lifecycle management.
> Following is a possible incomplete list of data processing functions
> implemented as abstract classes currently:
> * `ProcessFunction`, `KeyedProcessFunction`, `CoProcessFunction` and
> `ProcessJoinFunction`
> * `ProcessWindowFunction` and `ProcessAllWindowFunction`
> * `BaseBroadcastProcessFunction`, `BroadcastProcessFunction` and
> `KeyedBroadcastProcessFunction`
> All above functions are annotated with `@PublicEvolving`, making they
> interfaces won't break Flink's compatibility guarantee but compatibility is
> still a big consideration to evaluate this proposal.
> Any thoughts on this proposal ? Please must comment out.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-11498) Getting java.lang.InstantiationError when trying to install flink 1.7.1 version through ansible
[
https://issues.apache.org/jira/browse/FLINK-11498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sambi reddy updated FLINK-11498:
Description:
We are trying to install flink on our cloud with the help of ansible
automation. we were running 1.3.2 since past two years. using the similar kind
of automation with few config changes and trying to start the cluster but
failing to do so.
Error we are seeing in the logs.
{noformat}
Starting the SlotManager.
Fatal error occurred in the cluster entrypoint.
java.lang.InstantiationError: org.apache.flink.runtime.blob.BlobKey
at sun.reflect.GeneratedSerializationConstructorAccessor43.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:1006)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2011)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at java.util.ArrayList.readObject(ArrayList.java:791)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2136)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:503)
at
org.apache.flink.runtime.state.RetrievableStreamStateHandle.retrieveState(RetrievableStreamStateHandle.java:58)
at
org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.recoverJobGraph(ZooKeeperSubmittedJobGraphStore.java:202)
at
org.apache.flink.runtime.dispatcher.Dispatcher.recoverJob(Dispatcher.java:696)
at
org.apache.flink.runtime.dispatcher.Dispatcher.recoverJobGraphs(Dispatcher.java:681)
at
org.apache.flink.runtime.dispatcher.Dispatcher.recoverJobs(Dispatcher.java:662)
at
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$null$26(Dispatcher.java:821)
at
org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:72)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Shutting down BLOB cache
Stopped BLOB server at 0.0.0.0:44036{noformat}
was:
We trying to install flink on our cloud with the help ansible automation. we
were running 1.3.2 since past two years. using the similar kind of automation
with few config changes and trying to start the cluster but failing to do so.
Error we are seeing in the logs.
{noformat}
Starting the SlotManager.
Fatal error occurred in the cluster entrypoint.
java.lang.InstantiationError: org.apache.flink.runtime.blob.BlobKey
at sun.reflect.GeneratedSerializationConstructorAccessor43.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:1006)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2011)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at java.util.ArrayList.readObject(ArrayList.java:791)
at sun.reflect.GeneratedMethodAccessor3.invoke
[jira] [Updated] (FLINK-11498) Getting java.lang.InstantiationError when trying to install flink 1.7.1 version through ansible
[
https://issues.apache.org/jira/browse/FLINK-11498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sambi reddy updated FLINK-11498:
Description:
We are trying to install flink on our cloud with the help of ansible
automation. we were running 1.3.2 since past two years. using the similar kind
of automation with few config changes, deployed the latest version and trying
to install it but failing to start cluster.
Error we are seeing in the logs.
{noformat}
Starting the SlotManager.
Fatal error occurred in the cluster entrypoint.
java.lang.InstantiationError: org.apache.flink.runtime.blob.BlobKey
at sun.reflect.GeneratedSerializationConstructorAccessor43.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:1006)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2011)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at java.util.ArrayList.readObject(ArrayList.java:791)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2136)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:503)
at
org.apache.flink.runtime.state.RetrievableStreamStateHandle.retrieveState(RetrievableStreamStateHandle.java:58)
at
org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.recoverJobGraph(ZooKeeperSubmittedJobGraphStore.java:202)
at
org.apache.flink.runtime.dispatcher.Dispatcher.recoverJob(Dispatcher.java:696)
at
org.apache.flink.runtime.dispatcher.Dispatcher.recoverJobGraphs(Dispatcher.java:681)
at
org.apache.flink.runtime.dispatcher.Dispatcher.recoverJobs(Dispatcher.java:662)
at
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$null$26(Dispatcher.java:821)
at
org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:72)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Shutting down BLOB cache
Stopped BLOB server at 0.0.0.0:44036{noformat}
was:
We are trying to install flink on our cloud with the help of ansible
automation. we were running 1.3.2 since past two years. using the similar kind
of automation with few config changes and trying to start the cluster but
failing to do so.
Error we are seeing in the logs.
{noformat}
Starting the SlotManager.
Fatal error occurred in the cluster entrypoint.
java.lang.InstantiationError: org.apache.flink.runtime.blob.BlobKey
at sun.reflect.GeneratedSerializationConstructorAccessor43.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:1006)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2011)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at java.util.ArrayList.readObject(ArrayList.java:791)
at sun.r
[jira] [Updated] (FLINK-11498) Getting java.lang.InstantiationError when trying to install flink 1.7.1 version
[
https://issues.apache.org/jira/browse/FLINK-11498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sambi reddy updated FLINK-11498:
Summary: Getting java.lang.InstantiationError when trying to install flink
1.7.1 version (was: Getting java.lang.InstantiationError when trying to
install flink 1.7.1 version through ansible)
> Getting java.lang.InstantiationError when trying to install flink 1.7.1
> version
>
>
> Key: FLINK-11498
> URL: https://issues.apache.org/jira/browse/FLINK-11498
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.7.1
> Environment: Openstack
>Reporter: sambi reddy
>Priority: Major
> Fix For: 1.7.1
>
> Original Estimate: 5h
> Remaining Estimate: 5h
>
> We are trying to install flink on our cloud with the help of ansible
> automation. we were running 1.3.2 since past two years. using the similar
> kind of automation with few config changes, deployed the latest version and
> trying to install it but failing to start cluster.
> Error we are seeing in the logs.
> {noformat}
> Starting the SlotManager.
> Fatal error occurred in the cluster entrypoint.
> java.lang.InstantiationError: org.apache.flink.runtime.blob.BlobKey
> at
> sun.reflect.GeneratedSerializationConstructorAccessor43.newInstance(Unknown
> Source)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:1006)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2011)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
> at java.util.ArrayList.readObject(ArrayList.java:791)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2136)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:503)
> at
> org.apache.flink.runtime.state.RetrievableStreamStateHandle.retrieveState(RetrievableStreamStateHandle.java:58)
> at
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.recoverJobGraph(ZooKeeperSubmittedJobGraphStore.java:202)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.recoverJob(Dispatcher.java:696)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.recoverJobGraphs(Dispatcher.java:681)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.recoverJobs(Dispatcher.java:662)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$null$26(Dispatcher.java:821)
> at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:72)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Shutting down BLOB cache
> Stopped BLOB server at 0.0.0.0:44036{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-11499) Extend StreamingFileSink BulkFormats to support arbitrary roll policy's
[ https://issues.apache.org/jira/browse/FLINK-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman updated FLINK-11499: - Component/s: Streaming Connectors filesystem-connector > Extend StreamingFileSink BulkFormats to support arbitrary roll policy's > > > Key: FLINK-11499 > URL: https://issues.apache.org/jira/browse/FLINK-11499 > Project: Flink > Issue Type: Improvement > Components: filesystem-connector, Streaming Connectors >Reporter: Seth Wiesman >Priority: Minor > > Currently when using the StreamingFilleSink Bulk-encoding formats can only be > combined with the `OnCheckpointRollingPolicy`, which rolls the in-progress > part file on every checkpoint. > However, many bulk formats such as parquet are most efficient when written as > large files; this is not possible when frequent checkpointing is enabled. > Currently the only work-around is to have long checkpoint intervals which is > not ideal. > > The StreamingFileSink should be enhanced to support arbitrary roll policy's > so users may write large bulk files while retaining frequent checkpoints. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] stevenzwu commented on issue #7606: [FLINK-10774] Rework lifecycle management of partitionDiscoverer in FlinkKafkaConsumerBase
stevenzwu commented on issue #7606: [FLINK-10774] Rework lifecycle management of partitionDiscoverer in FlinkKafkaConsumerBase URL: https://github.com/apache/flink/pull/7606#issuecomment-459455146 @tillrohrmann sorry for a late question. you mentioned `This implies that all users of this class need to call close() in order to properly clean up resources.` I also noticed your test cases explicitly calls `close` method to clean up resources. In production/distributed environment, when a job restart due to exception from `open` method, only `cancel` method was called. `close` method wasn't invoked in this case. that was my experience with 1.4. is that still true? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] walterddr commented on a change in pull request #7607: [FLINK-10076][table] Upgrade Calcite dependency to 1.18
walterddr commented on a change in pull request #7607: [FLINK-10076][table] Upgrade Calcite dependency to 1.18 URL: https://github.com/apache/flink/pull/7607#discussion_r252766531 ## File path: flink-libraries/flink-table/src/main/java/org/apache/calcite/tools/RelBuilder.java ## @@ -0,0 +1,2445 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.calcite.tools; + +import org.apache.calcite.linq4j.Ord; +import org.apache.calcite.linq4j.function.Experimental; +import org.apache.calcite.plan.Context; +import org.apache.calcite.plan.Contexts; +import org.apache.calcite.plan.RelOptCluster; +import org.apache.calcite.plan.RelOptPredicateList; +import org.apache.calcite.plan.RelOptSchema; +import org.apache.calcite.plan.RelOptTable; +import org.apache.calcite.plan.RelOptUtil; +import org.apache.calcite.rel.RelCollation; +import org.apache.calcite.rel.RelCollations; +import org.apache.calcite.rel.RelDistribution; +import org.apache.calcite.rel.RelFieldCollation; +import org.apache.calcite.rel.RelNode; +import org.apache.calcite.rel.core.Aggregate; +import org.apache.calcite.rel.core.AggregateCall; +import org.apache.calcite.rel.core.CorrelationId; +import org.apache.calcite.rel.core.Filter; +import org.apache.calcite.rel.core.Intersect; +import org.apache.calcite.rel.core.Join; +import org.apache.calcite.rel.core.JoinRelType; +import org.apache.calcite.rel.core.Match; +import org.apache.calcite.rel.core.Minus; +import org.apache.calcite.rel.core.Project; +import org.apache.calcite.rel.core.RelFactories; +import org.apache.calcite.rel.core.SemiJoin; +import org.apache.calcite.rel.core.Sort; +import org.apache.calcite.rel.core.TableScan; +import org.apache.calcite.rel.core.Union; +import org.apache.calcite.rel.core.Values; +import org.apache.calcite.rel.logical.LogicalFilter; +import org.apache.calcite.rel.logical.LogicalProject; +import org.apache.calcite.rel.metadata.RelMetadataQuery; +import org.apache.calcite.rel.type.RelDataType; +import org.apache.calcite.rel.type.RelDataTypeFactory; +import org.apache.calcite.rel.type.RelDataTypeField; +import org.apache.calcite.rel.type.RelDataTypeFieldImpl; +import org.apache.calcite.rex.RexBuilder; +import org.apache.calcite.rex.RexCall; +import org.apache.calcite.rex.RexCorrelVariable; +import org.apache.calcite.rex.RexExecutor; +import org.apache.calcite.rex.RexInputRef; +import org.apache.calcite.rex.RexLiteral; +import org.apache.calcite.rex.RexNode; +import org.apache.calcite.rex.RexShuttle; +import org.apache.calcite.rex.RexSimplify; +import org.apache.calcite.rex.RexUtil; +import org.apache.calcite.runtime.Hook; +import org.apache.calcite.schema.SchemaPlus; +import org.apache.calcite.server.CalciteServerStatement; +import org.apache.calcite.sql.SemiJoinType; +import org.apache.calcite.sql.SqlAggFunction; +import org.apache.calcite.sql.SqlKind; +import org.apache.calcite.sql.SqlOperator; +import org.apache.calcite.sql.fun.SqlStdOperatorTable; +import org.apache.calcite.sql.type.SqlTypeName; +import org.apache.calcite.sql.validate.SqlValidatorUtil; +import org.apache.calcite.util.Holder; +import org.apache.calcite.util.ImmutableBitSet; +import org.apache.calcite.util.ImmutableIntList; +import org.apache.calcite.util.ImmutableNullableList; +import org.apache.calcite.util.Litmus; +import org.apache.calcite.util.NlsString; +import org.apache.calcite.util.Pair; +import org.apache.calcite.util.Util; +import org.apache.calcite.util.mapping.Mapping; +import org.apache.calcite.util.mapping.Mappings; + +import com.google.common.base.Preconditions; +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import com.google.common.collect.Iterables; +import com.google.common.collect.Lists; + +import java.math.BigDecimal; +import java.util.AbstractList; +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Deque; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Objects; +import java.util.Set; +import java.util.SortedSet; +import java.util.TreeSet;
[GitHub] walterddr commented on a change in pull request #7607: [FLINK-10076][table] Upgrade Calcite dependency to 1.18
walterddr commented on a change in pull request #7607: [FLINK-10076][table]
Upgrade Calcite dependency to 1.18
URL: https://github.com/apache/flink/pull/7607#discussion_r252780180
##
File path:
flink-libraries/flink-table/src/test/scala/org/apache/flink/table/api/batch/sql/SetOperatorsTest.scala
##
@@ -47,6 +47,7 @@ class SetOperatorsTest extends TableTestBase {
}
@Test
+ // unknown issue here, EXISTS should not cause b_long to be renamed as
b_long3 though. fixed test
Review comment:
this line was suppose to be removed. will fix
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] walterddr commented on a change in pull request #7607: [FLINK-10076][table] Upgrade Calcite dependency to 1.18
walterddr commented on a change in pull request #7607: [FLINK-10076][table] Upgrade Calcite dependency to 1.18 URL: https://github.com/apache/flink/pull/7607#discussion_r252764493 ## File path: flink-libraries/flink-table/pom.xml ## @@ -96,25 +96,12 @@ under the License. org.apache.calcite calcite-core - 1.17.0 + 1.18.0 - org.apache.calcite.avatica avatica-metrics - Review comment: I did some experiments and seems like Calcite 1.18 has dependencies on `com.fastjackson.core` and `core.fastjackson.databind` that cannot be ignored. I will try to push a fix. but in general, do you think we should somehow shade calcite to avoid pulling in jackson directly into flink-table? (The answer might be different if we first pull in #7587 like @zentol suggested. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] walterddr commented on a change in pull request #7607: [FLINK-10076][table] Upgrade Calcite dependency to 1.18
walterddr commented on a change in pull request #7607: [FLINK-10076][table]
Upgrade Calcite dependency to 1.18
URL: https://github.com/apache/flink/pull/7607#discussion_r252786140
##
File path:
flink-libraries/flink-table/src/test/scala/org/apache/flink/table/api/stream/table/OverWindowTest.scala
##
@@ -53,20 +54,20 @@ class OverWindowTest extends TableTestBase {
unaryNode(
"DataStreamCalc",
streamTableNode(0),
-term("select", "a", "b", "c", "proctime")
+term("select", "a", "b", "c", "proctime", "1 AS $4")
Review comment:
due to "[CALCITE-2726] RexSimplify does not simplify enough for some of the
nested operations, doesn't do nested visitCall". this looks like something we
have to fix on Calcite side. do you think we can proceed with this test change
and later fix it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] walterddr commented on a change in pull request #7607: [FLINK-10076][table] Upgrade Calcite dependency to 1.18
walterddr commented on a change in pull request #7607: [FLINK-10076][table] Upgrade Calcite dependency to 1.18 URL: https://github.com/apache/flink/pull/7607#discussion_r252764596 ## File path: flink-libraries/flink-table/pom.xml ## @@ -267,6 +250,8 @@ under the License. com.google.guava:guava net.hydromatic:* com.esri.geometry:* + com.jayway.jsonpath:* + com.fasterxml.jackson.dataformat:* Review comment: 👍 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] walterddr commented on a change in pull request #7607: [FLINK-10076][table] Upgrade Calcite dependency to 1.18
walterddr commented on a change in pull request #7607: [FLINK-10076][table]
Upgrade Calcite dependency to 1.18
URL: https://github.com/apache/flink/pull/7607#discussion_r252765823
##
File path:
flink-libraries/flink-table/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java
##
@@ -5129,6 +5129,11 @@ private SqlNode navigationInDefine(SqlNode node, String
alpha) {
public void validateAggregateParams(SqlCall aggCall, SqlNode filter,
SqlValidatorScope scope) {
+ validateAggregateParams(aggCall, filter, null, scope);
Review comment:
there are more changes in this class. I will recopy the file from new
release. thanks for the check, I wasn't aware we always need to keep them in
sync.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] walterddr commented on a change in pull request #7607: [FLINK-10076][table] Upgrade Calcite dependency to 1.18
walterddr commented on a change in pull request #7607: [FLINK-10076][table] Upgrade Calcite dependency to 1.18 URL: https://github.com/apache/flink/pull/7607#discussion_r252766675 ## File path: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/AggSqlFunction.scala ## @@ -63,6 +64,7 @@ class AggSqlFunction( null, false, requiresOver, +Optionality.IGNORED, Review comment: 👍 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] walterddr commented on a change in pull request #7607: [FLINK-10076][table] Upgrade Calcite dependency to 1.18
walterddr commented on a change in pull request #7607: [FLINK-10076][table]
Upgrade Calcite dependency to 1.18
URL: https://github.com/apache/flink/pull/7607#discussion_r252766737
##
File path:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/schema/TimeIndicatorRelDataType.scala
##
@@ -34,6 +34,8 @@ class TimeIndicatorRelDataType(
originalType.getSqlTypeName,
originalType.getPrecision) {
+ digest = toString
Review comment:
👍
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] bowenli86 edited a comment on issue #7587: [FLINK-11064] [table] Setup a new flink-table module structure
bowenli86 edited a comment on issue #7587: [FLINK-11064] [table] Setup a new flink-table module structure URL: https://github.com/apache/flink/pull/7587#issuecomment-459476927 nice work! @twalthr This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] bowenli86 commented on issue #7587: [FLINK-11064] [table] Setup a new flink-table module structure
bowenli86 commented on issue #7587: [FLINK-11064] [table] Setup a new flink-table module structure URL: https://github.com/apache/flink/pull/7587#issuecomment-459476927 nice work! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-10796) Add a default external catalog (as FlinkInMemoryCatalog) to CatalogManager
[ https://issues.apache.org/jira/browse/FLINK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li closed FLINK-10796. Resolution: Duplicate > Add a default external catalog (as FlinkInMemoryCatalog) to CatalogManager > -- > > Key: FLINK-10796 > URL: https://issues.apache.org/jira/browse/FLINK-10796 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.8.0 > > > CatalogManager has a default catalog (as FlinkInMemoryCatalog), which will be > selected when a catalog name isn’t given in a request. > Users can also specify an external catalog as their preferred default one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-10698) Create CatalogManager class manages all external catalogs and temporary meta objects
[
https://issues.apache.org/jira/browse/FLINK-10698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bowen Li closed FLINK-10698.
Resolution: Duplicate
> Create CatalogManager class manages all external catalogs and temporary meta
> objects
>
>
> Key: FLINK-10698
> URL: https://issues.apache.org/jira/browse/FLINK-10698
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.6.1
>Reporter: Xuefu Zhang
>Assignee: Bowen Li
>Priority: Major
> Fix For: 1.8.0
>
>
> Currently {{TableEnvironment}} manages a list of registered external catalogs
> as well as in-memory meta objects, and interacts with Calcite schema. It
> would be cleaner to delegate all those responsibilities to a dedicate class,
> especially when Flink's meta objects are also stored in a catalog.
> {{CatalogManager}} is responsible to manage all meta objects, including
> external catalogs, temporary meta objects, and Calcite schema.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-10696) Add APIs to ExternalCatalog for views and UDFs
[ https://issues.apache.org/jira/browse/FLINK-10696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li closed FLINK-10696. Resolution: Won't Fix Release Note: existing catalog API will be replaced, thus no need to modify existing ones anymore > Add APIs to ExternalCatalog for views and UDFs > -- > > Key: FLINK-10696 > URL: https://issues.apache.org/jira/browse/FLINK-10696 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Affects Versions: 1.6.1 >Reporter: Xuefu Zhang >Assignee: Bowen Li >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently there are APIs for tables only. However, views and UDFs are also > common objects in a catalog. > This is required when we store Flink tables/views/UDFs in an external > persistent storage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-10768) Move external catalog related code from TableEnvironment to CatalogManager
[ https://issues.apache.org/jira/browse/FLINK-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li closed FLINK-10768. Resolution: Duplicate > Move external catalog related code from TableEnvironment to CatalogManager > -- > > Key: FLINK-10768 > URL: https://issues.apache.org/jira/browse/FLINK-10768 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Add a new CatalogManager class and port existing Calcite-directly-related > code from TableEnvironment into CatalogManager. > Background: there are two parallel efforts going on right now - FLINK-10686, > driven by Timo, includes moving external catalogs APIs from flink-table to > flink-table-common, also from Scala to Java; FLINK-10744 I'm working on right > now to integrate Flink with Hive and enhance external catalog functionality. > As discussed with @twalthr in FLINK-10689, we'd better parallelize these > efforts while introducing minimal overhead for integrating them later. Our > agreed way is to writing new code/feature related to external catalogs/hive > in Java in flink-table first then move to other module like > flink-table-common, this way we can minimize migration efforts. If existing > classes are modified for a feature we can start migrating them to Java in a > separate commit first and then perform the actual feature changes, and > migrated classes can be placed in flink-table/src/main/java until we find a > better module structure. > Thus, this is NOT a feature, but purely refactor, thus no new functions > should be introduced. It acts the pre-requisite for FLINK-10698 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11500) 1.8
Chesnay Schepler created FLINK-11500: Summary: 1.8 Key: FLINK-11500 URL: https://issues.apache.org/jira/browse/FLINK-11500 Project: Flink Issue Type: Bug Reporter: Chesnay Schepler -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] zentol merged pull request #7535: [FLINK-11382] Disable MetricFetcher if interval is configured to 0 an…
zentol merged pull request #7535: [FLINK-11382] Disable MetricFetcher if interval is configured to 0 an… URL: https://github.com/apache/flink/pull/7535 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-11382) Disable MetricFetcher if interval is configured to 0
[ https://issues.apache.org/jira/browse/FLINK-11382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757702#comment-16757702 ] Chesnay Schepler commented on FLINK-11382: -- [~xleesf] Could you open a follow-up to update the documentation? > Disable MetricFetcher if interval is configured to 0 > > > Key: FLINK-11382 > URL: https://issues.apache.org/jira/browse/FLINK-11382 > Project: Flink > Issue Type: New Feature > Components: Metrics >Reporter: Chesnay Schepler >Assignee: leesf >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Follow-up for FLINK-10822 to disable the MetricFetcher completely if the > interval is configured to 0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] jgrier opened a new pull request #7625: Backport [FLINK-10887] Add JobMaster RPC endpoint that is used to share information across source subtasks.
jgrier opened a new pull request #7625: Backport [FLINK-10887] Add JobMaster RPC endpoint that is used to share information across source subtasks. URL: https://github.com/apache/flink/pull/7625 Backporting this Flink PR to our Lyft internal branch: https://github.com/apache/flink/pull/7099 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jgrier commented on issue #7625: Backport [FLINK-10887] Add JobMaster RPC endpoint that is used to share information across source subtasks.
jgrier commented on issue #7625: Backport [FLINK-10887] Add JobMaster RPC endpoint that is used to share information across source subtasks. URL: https://github.com/apache/flink/pull/7625#issuecomment-459500385 Mistake. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jgrier closed pull request #7625: Backport [FLINK-10887] Add JobMaster RPC endpoint that is used to share information across source subtasks.
jgrier closed pull request #7625: Backport [FLINK-10887] Add JobMaster RPC endpoint that is used to share information across source subtasks. URL: https://github.com/apache/flink/pull/7625 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-10887) Add source watermark tracking to the JobMaster
[ https://issues.apache.org/jira/browse/FLINK-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated FLINK-10887: - Fix Version/s: 1.8.0 > Add source watermark tracking to the JobMaster > -- > > Key: FLINK-10887 > URL: https://issues.apache.org/jira/browse/FLINK-10887 > Project: Flink > Issue Type: Sub-task > Components: JobManager >Reporter: Jamie Grier >Assignee: Jamie Grier >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > Original Estimate: 24h > Time Spent: 20m > Remaining Estimate: 23h 40m > > We need to add a new RPC to the JobMaster such that the current watermark for > every source sub-task can be reported and the current global minimum/maximum > watermark can be retrieved so that each source can adjust their partition > read rates in an attempt to keep sources roughly aligned in event time. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-11501) Add a ratelimiting feature to the FlinkKafkaConsumer
[ https://issues.apache.org/jira/browse/FLINK-11501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lakshmi Rao updated FLINK-11501: Description: There are instances when a Flink job that reads from Kafka can read at a significantly high throughput (particularly while processing a backlog) and degrade the underlying Kafka cluster. While Kafka quotas are perhaps the best way to enforce this ratelimiting, there are cases where such a setup is not available or easily enabled. In such a scenario, ratelimiting on the FlinkKafkaConsumer is useful feature. The approach is essentially involves using Guava's [RateLimiter|https://google.github.io/guava/releases/19.0/api/docs/index.html?com/google/common/util/concurrent/RateLimiter.html] to ratelimit the bytes read from Kafka (in the [KafkaConsumerThread|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java]) More discussion here: [https://lists.apache.org/thread.html/8140b759ba83f33a22d809887fd2d711f5ffe7069c888eb9b1142272@%3Cdev.flink.apache.org%3E] was: There are instances when a Flink job that reads from Kafka can read at a significantly high throughput (particularly while processing a backlog) and degrade the underlying Kafka cluster. While Kafka quotas are perhaps the best way to enforce this ratelimiting, there are cases where such a setup is not available or easily enabled. The approach is essentially to use Guava's rate limiter to ratelimit the bytes read from Kafka (in the [KafkaConsumerThread|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java]) More discussion here: [https://lists.apache.org/thread.html/8140b759ba83f33a22d809887fd2d711f5ffe7069c888eb9b1142272@%3Cdev.flink.apache.org%3E] > Add a ratelimiting feature to the FlinkKafkaConsumer > > > Key: FLINK-11501 > URL: https://issues.apache.org/jira/browse/FLINK-11501 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Reporter: Lakshmi Rao >Assignee: Lakshmi Rao >Priority: Major > > There are instances when a Flink job that reads from Kafka can read at a > significantly high throughput (particularly while processing a backlog) and > degrade the underlying Kafka cluster. > While Kafka quotas are perhaps the best way to enforce this ratelimiting, > there are cases where such a setup is not available or easily enabled. In > such a scenario, ratelimiting on the FlinkKafkaConsumer is useful feature. > The approach is essentially involves using Guava's > [RateLimiter|https://google.github.io/guava/releases/19.0/api/docs/index.html?com/google/common/util/concurrent/RateLimiter.html] > to ratelimit the bytes read from Kafka (in the > [KafkaConsumerThread|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java]) > More discussion here: > [https://lists.apache.org/thread.html/8140b759ba83f33a22d809887fd2d711f5ffe7069c888eb9b1142272@%3Cdev.flink.apache.org%3E] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-11501) Add a ratelimiting feature to the FlinkKafkaConsumer
[ https://issues.apache.org/jira/browse/FLINK-11501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lakshmi Rao updated FLINK-11501: Description: There are instances when a Flink job that reads from Kafka can read at a significantly high throughput (particularly while processing a backlog) and degrade the underlying Kafka cluster. While Kafka quotas are perhaps the best way to enforce this ratelimiting, there are cases where such a setup is not available or easily enabled. The approach is essentially to use Guava's rate limiter to ratelimit the bytes read from Kafka (in the [KafkaConsumerThread|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java]) More discussion here: [https://lists.apache.org/thread.html/8140b759ba83f33a22d809887fd2d711f5ffe7069c888eb9b1142272@%3Cdev.flink.apache.org%3E] was: There are instances when a Flink job that reads from Kafka can read at a significantly high throughput (particularly while processing a backlog) and degrade the underlying Kafka cluster. While Kafka quotas are perhaps the best way to enforce this ratelimiting, there are cases where such a setup is not available or easily enabled. The approach is essentially to use Guava's rate limiter to ratelimit the bytes read from Kafka. More discussion here: https://lists.apache.org/thread.html/8140b759ba83f33a22d809887fd2d711f5ffe7069c888eb9b1142272@%3Cdev.flink.apache.org%3E > Add a ratelimiting feature to the FlinkKafkaConsumer > > > Key: FLINK-11501 > URL: https://issues.apache.org/jira/browse/FLINK-11501 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Reporter: Lakshmi Rao >Assignee: Lakshmi Rao >Priority: Major > > There are instances when a Flink job that reads from Kafka can read at a > significantly high throughput (particularly while processing a backlog) and > degrade the underlying Kafka cluster. > While Kafka quotas are perhaps the best way to enforce this ratelimiting, > there are cases where such a setup is not available or easily enabled. > The approach is essentially to use Guava's rate limiter to ratelimit the > bytes read from Kafka (in the > [KafkaConsumerThread|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java]) > More discussion here: > [https://lists.apache.org/thread.html/8140b759ba83f33a22d809887fd2d711f5ffe7069c888eb9b1142272@%3Cdev.flink.apache.org%3E] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11501) Add a ratelimiting feature to the FlinkKafkaConsumer
Lakshmi Rao created FLINK-11501: --- Summary: Add a ratelimiting feature to the FlinkKafkaConsumer Key: FLINK-11501 URL: https://issues.apache.org/jira/browse/FLINK-11501 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Lakshmi Rao Assignee: Lakshmi Rao There are instances when a Flink job that reads from Kafka can read at a significantly high throughput (particularly while processing a backlog) and degrade the underlying Kafka cluster. While Kafka quotas are perhaps the best way to enforce this ratelimiting, there are cases where such a setup is not available or easily enabled. The approach is essentially to use Guava's rate limiter to ratelimit the bytes read from Kafka. More discussion here: https://lists.apache.org/thread.html/8140b759ba83f33a22d809887fd2d711f5ffe7069c888eb9b1142272@%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] leesf commented on issue #7555: [FLINK-11373] CliFrontend cuts off reason for error messages
leesf commented on issue #7555: [FLINK-11373] CliFrontend cuts off reason for error messages URL: https://github.com/apache/flink/pull/7555#issuecomment-459555295 cc @zentol @dawidwys This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] leesf commented on issue #7536: [hotfix][docs] Remove redundant symbols
leesf commented on issue #7536: [hotfix][docs] Remove redundant symbols URL: https://github.com/apache/flink/pull/7536#issuecomment-459555384 cc @zentol This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jgrier closed pull request #7099: [FLINK-10887] [jobmaster] Add source watermark tracking to the JobMaster
jgrier closed pull request #7099: [FLINK-10887] [jobmaster] Add source watermark tracking to the JobMaster URL: https://github.com/apache/flink/pull/7099 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jgrier opened a new pull request #7099: [FLINK-10887] [jobmaster] Add source watermark tracking to the JobMaster
jgrier opened a new pull request #7099: [FLINK-10887] [jobmaster] Add source watermark tracking to the JobMaster URL: https://github.com/apache/flink/pull/7099 ## What is the purpose of the change This commit adds a JobMaster RPC endpoint that is used to for global information sharing. One use case will be event time source synchronization where it will be used to share watermarks but there are others. It takes the form of a set of named aggregates that can be updated by a client-supplied AggregateFunction. Note that the RPC endpoint accepts a serialized AggregateFunction in the form of a byte array. We need to do this so that we can deserialize this using the UserCodeClassLoader. The normal RpcService path does not use the UserCodeClassLoader nor is there any easy way to make it do so. This PR also includes the code/wiring neccessary to expose this functionality to user functions via the `StreamingRuntimeEnvironment`. The PR seems large but it is mostly wiring. To quickly assess the changes I suggest looking at the following classes: - `GlobalAggregateManager` (to understand the API) - `RpcGlobalAggregateManager` (to see the client-side RPC with the JobMaster) - `JobMaster / JobMasterGateway` (server side implementation of the above) - `GlobalAggregateManagerITCase` (for typical usage from user code) Most of the rest of the PR is just wiring it all up. ## Brief change log - New RPC endpoint on JobMaster to create, update, and retrieve named aggregates. - Updated JobMaster Tests - Client side exposure of above endpoint via the StreamingRuntimeEnvironment and GlobalAggregateManager classes. - Integration test exercising typical usage from user code. ## Verifying this change This change added tests and can be verified as follows: - JobMasterTest - GlobalAggregateManagerITCase ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): No - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: Yes - The serializers: No - The runtime per-record code paths (performance sensitive): No - Anything that affects deployment or recovery: No - The S3 file system connector: No ## Documentation - Does this pull request introduce a new feature? No - If yes, how is the feature documented? not applicable This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-11487) Support for writing data to Apache Flume
[ https://issues.apache.org/jira/browse/FLINK-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ambition closed FLINK-11487. Resolution: Fixed Release Note: [FLINK-4446] Remove Flume connector (now in Bahir) > Support for writing data to Apache Flume > > > Key: FLINK-11487 > URL: https://issues.apache.org/jira/browse/FLINK-11487 > Project: Flink > Issue Type: New Feature > Components: Streaming Connectors >Affects Versions: 1.7.1 > Environment: JDK 1.8 > Scala 2.11 > Flink 1.7.1 > Apache Flume 1.6.0 >Reporter: ambition >Priority: Major > Labels: pull-request-available > Fix For: 1.7.2, 1.8.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Flume is a distributed, reliable, and available service for efficiently > collecting, aggregating, and moving large amounts of data, has many users. > Unfortunately, Flink does not currently support with data to Flume. > The following is the official website of flume and github source address： > [Apache Flume website|http://flume.apache.org/index.html] > [Apache Flume github|https://github.com/apache/flume] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ambition119 closed pull request #7618: [FLINK-11487] Support for writing data to Apache Flume
ambition119 closed pull request #7618: [FLINK-11487] Support for writing data to Apache Flume URL: https://github.com/apache/flink/pull/7618 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ambition119 commented on issue #7618: [FLINK-11487] Support for writing data to Apache Flume
ambition119 commented on issue #7618: [FLINK-11487] Support for writing data to Apache Flume URL: https://github.com/apache/flink/pull/7618#issuecomment-459574423 @sjwiesman Thank you for your comment, let me know Apache Bahir. At first I thought that the code I wrote was not very good, but after reading the code of Apache Bahir, I found my write code is very good. @dawidwys I followed the history and found that it was moved from Flink to Bahir, so I will PR to Bahir again.  This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-11484) Blink java.util.concurrent.TimeoutException
[
https://issues.apache.org/jira/browse/FLINK-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
pj updated FLINK-11484:
---
Attachment: 1.png
> Blink java.util.concurrent.TimeoutException
> ---
>
> Key: FLINK-11484
> URL: https://issues.apache.org/jira/browse/FLINK-11484
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.5.5
> Environment: The link of blink source code:
> [github.com/apache/flink/tree/blink|https://github.com/apache/flink/tree/blink]
>Reporter: pj
>Priority: Major
> Labels: blink
> Attachments: 1.png
>
>
> *If I run blink application on yarn and the parallelism number larger than 1.*
> *Following is the command :*
> ./flink run -m yarn-cluster -ynm FLINK_NG_ENGINE_1 -ys 4 -yn 10 -ytm 5120 -p
> 40 -c XXMain ~/xx.jar
> *Following is the code:*
> {{DataStream outputStream = tableEnv.toAppendStream(curTable, Row.class);
> outputStream.print();}}
> *{{The whole subtask of application will hang a long time and finally the
> }}{{toAppendStream()}}{{ function will throw an exception like below:}}*
> {{org.apache.flink.client.program.ProgramInvocationException: Job failed.
> (JobID: f5e4f7243d06035202e8fa250c364304) at
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:276)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:482)
> at
> org.apache.flink.streaming.api.environment.StreamContextEnvironment.executeInternal(StreamContextEnvironment.java:85)
> at
> org.apache.flink.streaming.api.environment.StreamContextEnvironment.executeInternal(StreamContextEnvironment.java:37)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1893)
> at com.ngengine.main.KafkaMergeMain.startApp(KafkaMergeMain.java:352) at
> com.ngengine.main.KafkaMergeMain.main(KafkaMergeMain.java:94) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:561)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:445)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:419)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:786)
> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:280)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:215) at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1029)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1105)
> at java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1105)
> Caused by: java.util.concurrent.TimeoutException at
> org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:834)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)}}{{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11484) Blink java.util.concurrent.TimeoutException
[
https://issues.apache.org/jira/browse/FLINK-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757871#comment-16757871
]
pj commented on FLINK-11484:
Thanks for your response. I notice that the all tasks in my job hang a long
time and then throw the concurrent.TimeoutException. !1.png!
My job has 5 unchain tasks and the parallelism number is 40. Whether does it
mean that there need at least 200 task slots?
> Blink java.util.concurrent.TimeoutException
> ---
>
> Key: FLINK-11484
> URL: https://issues.apache.org/jira/browse/FLINK-11484
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.5.5
> Environment: The link of blink source code:
> [github.com/apache/flink/tree/blink|https://github.com/apache/flink/tree/blink]
>Reporter: pj
>Priority: Major
> Labels: blink
> Attachments: 1.png
>
>
> *If I run blink application on yarn and the parallelism number larger than 1.*
> *Following is the command :*
> ./flink run -m yarn-cluster -ynm FLINK_NG_ENGINE_1 -ys 4 -yn 10 -ytm 5120 -p
> 40 -c XXMain ~/xx.jar
> *Following is the code:*
> {{DataStream outputStream = tableEnv.toAppendStream(curTable, Row.class);
> outputStream.print();}}
> *{{The whole subtask of application will hang a long time and finally the
> }}{{toAppendStream()}}{{ function will throw an exception like below:}}*
> {{org.apache.flink.client.program.ProgramInvocationException: Job failed.
> (JobID: f5e4f7243d06035202e8fa250c364304) at
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:276)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:482)
> at
> org.apache.flink.streaming.api.environment.StreamContextEnvironment.executeInternal(StreamContextEnvironment.java:85)
> at
> org.apache.flink.streaming.api.environment.StreamContextEnvironment.executeInternal(StreamContextEnvironment.java:37)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1893)
> at com.ngengine.main.KafkaMergeMain.startApp(KafkaMergeMain.java:352) at
> com.ngengine.main.KafkaMergeMain.main(KafkaMergeMain.java:94) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:561)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:445)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:419)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:786)
> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:280)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:215) at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1029)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1105)
> at java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1105)
> Caused by: java.util.concurrent.TimeoutException at
> org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:834)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)}}{{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] tzulitai commented on issue #7601: [FLINK-11164] Check for sentinel values when creating new Kinesis ShardIterator
tzulitai commented on issue #7601: [FLINK-11164] Check for sentinel values when creating new Kinesis ShardIterator URL: https://github.com/apache/flink/pull/7601#issuecomment-459578559 @aljoscha LGTM 👍 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-11071) Dynamic proxy classes cannot be resolved when deserializing job graph
[ https://issues.apache.org/jira/browse/FLINK-11071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BoWang closed FLINK-11071. -- > Dynamic proxy classes cannot be resolved when deserializing job graph > - > > Key: FLINK-11071 > URL: https://issues.apache.org/jira/browse/FLINK-11071 > Project: Flink > Issue Type: Bug > Components: Core >Affects Versions: 1.6.2, 1.7.0, 1.8.0 >Reporter: Oleg Zhukov >Assignee: BoWang >Priority: Major > Labels: pull-request-available > Fix For: 1.6.4, 1.7.2, 1.8.0 > > Attachments: SocketWindowWordCount.java > > Time Spent: 20m > Remaining Estimate: 0h > > It turns impossible to use Java dynamic proxy objects in the job definition > (for example, as a MapFunction implementation). > During deserialization of the job graph, the default implementation of > ObjectInputStream.resolveProxyClass(..) is used, which is not using the > custom class loader (to look into the submitted jar) and therefore throws > ClassNotFoundException. > Looks like in order to address this, > InstantiationUtil.ClassLoaderObjectInputStream needs to provide custom > implementation of resolveProxyClass(..) method as well (in addition to > resolveClass(..)). > In order to reproduce the issue, run the attached SocketWindowWordCount Flink > app. It's a slight variation of the canonical [SocketWindowWordCount > > example|https://ci.apache.org/projects/flink/flink-docs-master/tutorials/local_setup.html] > with a dynamic proxy implementation of the flat map transformation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-11453) Support SliceStream with forwardable pane info
[
https://issues.apache.org/jira/browse/FLINK-11453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rong Rong reassigned FLINK-11453:
-
Assignee: Rong Rong
> Support SliceStream with forwardable pane info
> ---
>
> Key: FLINK-11453
> URL: https://issues.apache.org/jira/browse/FLINK-11453
> Project: Flink
> Issue Type: Sub-task
> Components: DataStream API
>Reporter: Rong Rong
>Assignee: Rong Rong
>Priority: Major
>
> Support slicing operation that produces slicing:
> {code:java}
> val slicedStream: SlicedStream = inputStream
> .keyBy("key")
> .sliceWindow(Time.seconds(5L)) // new “slice window” concept: to
> combine
>// tumble results based on discrete
>// non-overlapping windows.
> .aggregate(aggFunc)
> {code}
> {{SlicedStream}} will produce results that exposes current {{WindowOperator}}
> internal state {{InternalAppendingState}}, which can be
> later applied with {{WindowFunction}} separately in another operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-11454) Support MergedStream operation
[
https://issues.apache.org/jira/browse/FLINK-11454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rong Rong reassigned FLINK-11454:
-
Assignee: Rong Rong
> Support MergedStream operation
> --
>
> Key: FLINK-11454
> URL: https://issues.apache.org/jira/browse/FLINK-11454
> Project: Flink
> Issue Type: Sub-task
> Components: DataStream API
>Reporter: Rong Rong
>Assignee: Rong Rong
>Priority: Major
>
> Following SlicedStream, the mergedStream operator merges results from sliced
> stream and produces windowing results.
> {code:java}
> val slicedStream: SlicedStream = inputStream
> .keyBy("key")
> .sliceWindow(Time.seconds(5L)) // new “slice window” concept: to
> combine
>// tumble results based on discrete
>// non-overlapping windows.
> .aggregate(aggFunc)
> val mergedStream1: MergedStream = slicedStream
> .slideOver(Time.second(10L)) // combine slice results with same
>
>// windowing function, equivalent to
>// WindowOperator with an aggregate
> state
>// and derived aggregate function.
> val mergedStream2: MergedStream = slicedStream
> .slideOver(Count.of(5))
> .apply(windowFunction) // apply a different window function
> over
>// the sliced results.{code}
> MergedStream are produced by MergeOperator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11451) Move *QueryConfig and TableDescriptor to flink-table-api-java
[ https://issues.apache.org/jira/browse/FLINK-11451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757923#comment-16757923 ] Hequn Cheng commented on FLINK-11451: - Hi, [~xueyu] I think [~twalthr]'s suggestion is good. I don't have any other suggestions. Many thanks to the quick PR! > Move *QueryConfig and TableDescriptor to flink-table-api-java > - > > Key: FLINK-11451 > URL: https://issues.apache.org/jira/browse/FLINK-11451 > Project: Flink > Issue Type: New Feature > Components: Table API & SQL >Reporter: Timo Walther >Assignee: xueyu >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > A more detailed description can be found in > [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions]. > Move QueryConfig, BatchQueryConfig, StreamQueryConfig, TableDescriptor in > flink-table-api-java. > Unblocks TableEnvironment interface task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] dianfu opened a new pull request #7626: [FLINK-11450][table] Port and move TableSource and TableSink to flink-table-common
dianfu opened a new pull request #7626: [FLINK-11450][table] Port and move TableSource and TableSink to flink-table-common URL: https://github.com/apache/flink/pull/7626 ## What is the purpose of the change *This pull request ported the following classes to flink-table-common: TableSource, TableSink, TableSinkBase, DefinedFieldMapping, NestedFieldsProjectableTableSource, ProjectableTableSource, TableConnectorUtil, ExternalCatalog, ExternalCatalogTable, Metadata, MetadataValidator, Schema, SchematicDescriptor, Statistics, StatisticsValidator, StreamTableDescriptorValidator, StreamableDescriptor, TableDescriptor, ColumnStats, TableStats* ## Brief change log - *Ported source/sink related classes to flink-table-common: TableSource, TableSink, TableSinkBase, DefinedFieldMapping, NestedFieldsProjectableTableSource, ProjectableTableSource, TableConnectorUtil* - *Ported ExternalCatalog to flink-table-common and also the classes it depends on: ExternalCatalogTable, Metadata, MetadataValidator, Schema, SchematicDescriptor, Statistics, StatisticsValidator, StreamTableDescriptorValidator, StreamableDescriptor, TableDescriptor, ColumnStats, TableStats, TableNotExistException, CatalogNotExistException* - *Ported the tests to flink-table-common* The following things should be noted: - *The package name of TableConnectorUtil is changed from util to utils as there is a utils package in flink-table-common. Just create a util package for backward compatibility. Not sure if this is needed?* - *The interface of Schema.rowtime is changed as it depends on Rowtime which cannot be moved to flink-table-common for the time being as Rowtime depends on Expression* - *The constants variables are copied from SchemeValidator to Scheme as they are needed by Scheme, but SchemeValidator cannot be moved to flink-table-common for the time being. Have not moved these constants directly to Schema as I noticed all the constants are defined in Validator classes* - *StreamableDescriptor don't extends TableDescriptor anymore as otherwise it cannot be defined as an interface any more. * ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (docs) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-11450) Port and move TableSource and TableSink to flink-table-common
[ https://issues.apache.org/jira/browse/FLINK-11450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-11450: --- Labels: pull-request-available (was: ) > Port and move TableSource and TableSink to flink-table-common > - > > Key: FLINK-11450 > URL: https://issues.apache.org/jira/browse/FLINK-11450 > Project: Flink > Issue Type: New Feature > Components: Table API & SQL >Reporter: Timo Walther >Assignee: Dian Fu >Priority: Major > Labels: pull-request-available > > A more detailed description can be found in > [FLIP-32|https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions]. > This step only unblockes the TableEnvironment interfaces task. > Stream/BatchTableSouce/Sink remain in flink-table-api-java-bridge for now > until they have been reworked. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] liukaiyi commented on issue #7583: Blink init
liukaiyi commented on issue #7583: Blink init URL: https://github.com/apache/flink/pull/7583#issuecomment-459601393 > @ajliualiyun @KurtYoung org.apache.flink.sql.parser.impl.FlinkSqlParserImpl is missing in blink branch I also have this problem. in blink branch flink-sql-parser packageTesting is problematic This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-11502) Remove invalid test FlinkActorTest
TisonKun created FLINK-11502: Summary: Remove invalid test FlinkActorTest Key: FLINK-11502 URL: https://issues.apache.org/jira/browse/FLINK-11502 Project: Flink Issue Type: Sub-task Components: Tests Affects Versions: 1.8.0 Reporter: TisonKun Assignee: TisonKun Fix For: 1.8.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] TisonKun opened a new pull request #7627: [FLINK-11502] Remove invalid test FlinkActorTest
TisonKun opened a new pull request #7627: [FLINK-11502] Remove invalid test FlinkActorTest URL: https://github.com/apache/flink/pull/7627 ## What is the purpose of the change `FlinkActorTest` test `FlinkActor` class, which is no longer used in FLIP-6 codebase. ## Brief change log Delete `FlinkActorTest.scala`. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector:(no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) cc @tillrohrmann This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-11502) Remove invalid test FlinkActorTest
[ https://issues.apache.org/jira/browse/FLINK-11502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-11502: --- Labels: pull-request-available (was: ) > Remove invalid test FlinkActorTest > -- > > Key: FLINK-11502 > URL: https://issues.apache.org/jira/browse/FLINK-11502 > Project: Flink > Issue Type: Sub-task > Components: Tests >Affects Versions: 1.8.0 >Reporter: TisonKun >Assignee: TisonKun >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
