Re: [DISCUSS] GitBox

+1 for using Gitbox Timo Am 16.05.18 um 17:43 schrieb Kenneth Knowles: Actually, GitHub has a feature so you do not require picture-perfect commits: https://help.github.com/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork/ If the owner of the PR checks the box, it will gi

Re: [DISCUSS] GitBox

I couldn't find any such setting in one of my repos :( On 16.05.2018 21:03, Kenneth Knowles wrote: When I open a pull request to Beam, it is on by default. I have just run an experiment to see if it is remembering the last option I checked and it is not. Even after I disable it for one pull requ

Re: CloudWatch Metrics Reporter

To @Chesnay 's question on AWS library: yes, it would require using aws-java-sdk-core and aws-java-sdk-cloudwatch . Both are of Apache License 2.0. This is d

Re: [DISCUSS] GitBox

Rong Rong, see my reply. It can be enabled by default. I think it may be already. Kenn On Wed, May 16, 2018 at 4:24 PM Rong Rong wrote: > +1 > > One question regarding "This however has to be enabled by the contributor, > separately for each PR." > can it be by default enable when creating PR?

Re: [DISCUSS] GitBox

+1 One question regarding "This however has to be enabled by the contributor, separately for each PR." can it be by default enable when creating PR? On Wed, May 16, 2018 at 2:08 PM, Ted Yu wrote: > +1 > Original message From: Shuyi Chen > Date: 5/16/18 1:12 PM (GMT-08:00) To

Re: [DISCUSS] GitBox

+1 Original message From: Shuyi Chen Date: 5/16/18 1:12 PM (GMT-08:00) To: dev@flink.apache.org Subject: Re: [DISCUSS] GitBox +1 :) A lot of projects are already using it. On Wed, May 16, 2018 at 3:40 AM, Chesnay Schepler wrote: > Hell

Re: [DISCUSS] GitBox

+1 :) A lot of projects are already using it. On Wed, May 16, 2018 at 3:40 AM, Chesnay Schepler wrote: > Hello, > > during the discussion about how to better manage pull requests [1] the > topic of GitBox integration came up again. > > This seems like a good

Re: [DISCUSS] GitBox

+1 Kenneth Knowles schrieb am Mi., 16. Mai 2018, 21:04: > When I open a pull request to Beam, it is on by default. I have just run an > experiment to see if it is remembering the last option I checked and it is > not. Even after I disable it for one pull request, the next one has it > checked ag

Re: [DISCUSS] Adding new interfaces in [Stream]ExecutionEnvironment

Hi Aljoscha, Fabian, Rong, Ted and Timo, Thanks a lot for the feedback. Let me clarify the usage scenario in a bit more detail. The context is that we want to add support for SQL DDL to load UDF from external JARs located either in local filesystem or HDFS or a HTTP endpoint in Flink SQL. The loca

Re: [VOTE] Release 1.5.0, release candidate #3

Testing the RC has surfaced a problem with the release of blobs of finished jobs [1]. This is a release blocker and, thus, I have to cancel the RC 3. I'll prepare a new RC once the problem has been fixed. Thanks to all for testing the release candidate! [1] https://issues.apache.org/jira/browse/F

Re: [DISCUSS] GitBox

When I open a pull request to Beam, it is on by default. I have just run an experiment to see if it is remembering the last option I checked and it is not. Even after I disable it for one pull request, the next one has it checked again. So it may be a repository-level setting that you can set up.

Re: [DISCUSS] GitBox

This however has to be enabled by the contributor, separately for each PR. We'll see how often we get the opportunity to use it. On 16.05.2018 17:43, Kenneth Knowles wrote: Actually, GitHub has a feature so you do not require picture-perfect commits: https://help.github.com/articles/allowing-cha

Re: [DISCUSS] GitBox

+1 On Wed, May 16, 2018 at 2:09 PM, Thomas Weise wrote: > +1 > > > On Wed, May 16, 2018 at 8:31 AM, Stefan Richter < > s.rich...@data-artisans.com > > wrote: > > > +1 > > > > > Am 16.05.2018 um 12:40 schrieb Chesnay Schepler : > > > > > > Hello, > > > > > > during the discussion about how to bet

Re: [DISCUSS] GitBox

+1 Regards JB Le 16 mai 2018 à 20:09, à 20:09, Thomas Weise a écrit: >+1 > > >On Wed, May 16, 2018 at 8:31 AM, Stefan Richter >> wrote: > >> +1 >> >> > Am 16.05.2018 um 12:40 schrieb Chesnay Schepler >: >> > >> > Hello, >> > >> > during the discussion about how to better manage pull requests [1]

Re: [DISCUSS] GitBox

+1 On Wed, May 16, 2018 at 8:31 AM, Stefan Richter wrote: > +1 > > > Am 16.05.2018 um 12:40 schrieb Chesnay Schepler : > > > > Hello, > > > > during the discussion about how to better manage pull requests [1] the > topic of GitBox integration came up again. > > > > This seems like a good opport

Re: [DISCUSS] Adding new interfaces in [Stream]ExecutionEnvironment

I think the question here is whether registering Jar files (or other executable files) during job submission is sufficient for @shuyi's use case. If I understand correctly regarding the part of dynamic distribution of the external libraries in runtime. This is used to deal with DDL/DSL such as:

[jira] [Created] (FLINK-9386) Remove netty-router dependency

Piotr Nowojski created FLINK-9386: - Summary: Remove netty-router dependency Key: FLINK-9386 URL: https://issues.apache.org/jira/browse/FLINK-9386 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-9385) Operators with two inputs should show "Records Received" in Web UI separately, rather than added together

Josh Lemer created FLINK-9385: - Summary: Operators with two inputs should show "Records Received" in Web UI separately, rather than added together Key: FLINK-9385 URL: https://issues.apache.org/jira/browse/FLINK-9385

Re: [DISCUSS] GitBox

Actually, GitHub has a feature so you do not require picture-perfect commits: https://help.github.com/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork/ If the owner of the PR checks the box, it will give committers write access to their branch (on their fork). A nice bonus is

Re: [DISCUSS] GitBox

+1 > Am 16.05.2018 um 12:40 schrieb Chesnay Schepler : > > Hello, > > during the discussion about how to better manage pull requests [1] the topic > of GitBox integration came up again. > > This seems like a good opportunity to restart this discussion that we had > about a year ago [2]. > >

[jira] [Created] (FLINK-9384) KafkaAvroTableSource failed to work due to type mismatch

Jun Zhang created FLINK-9384: Summary: KafkaAvroTableSource failed to work due to type mismatch Key: FLINK-9384 URL: https://issues.apache.org/jira/browse/FLINK-9384 Project: Flink Issue Type: Bu

Re: [DISCUSS] GitBox

+1 > On 16 May 2018, at 14:24, Aljoscha Krettek wrote: > > +1 > > On Beam, we gradually enabled this, first for the website repo and then for > the main repo and we didn't run into problems. > >> On 16. May 2018, at 12:45, Chesnay Schepler wrote: >> >> Forget an important feature: It would

Re: Closing (automatically?) inactive pull requests

The question is what would such tool offer on top of over a Github’s view of PR sorted by “least recently updated”: https://github.com/apache/flink/pulls?q=is%3Apr+is%3Aopen+sort%3Aupdated-asc ? Maybe it would be good enou

[jira] [Created] (FLINK-9383) Extend DistributedCache E2E test to cover directories

Chesnay Schepler created FLINK-9383: --- Summary: Extend DistributedCache E2E test to cover directories Key: FLINK-9383 URL: https://issues.apache.org/jira/browse/FLINK-9383 Project: Flink Iss

[jira] [Created] (FLINK-9382) Inconsistent file storage behavior in FileCache

Chesnay Schepler created FLINK-9382: --- Summary: Inconsistent file storage behavior in FileCache Key: FLINK-9382 URL: https://issues.apache.org/jira/browse/FLINK-9382 Project: Flink Issue Typ

Re: [DISCUSS] Adding new interfaces in [Stream]ExecutionEnvironment

Yes, we are using the addJar functionionality of the JobGraph as well for the SQL Client. I think the execution environment is not the right place to specify jars. The location of the jars depends on the submission method. If a local path is specified in the main() method of a packaged Flink j

Re: [DISCUSS] Adding new interfaces in [Stream]ExecutionEnvironment

I think this functionality is already there, we just have to expose it in the right places: ClusterClient.submitJob() takes a JobGraph, JobGraph has method addJar() for adding jars that need to be in the classloader for executing a user program. > On 16. May 2018, at 12:34, Fabian Hueske wrote

Re: Errors checkpointing to S3 for high-scale jobs

For posterity: Here is the Jira Issue that tracks this: https://issues.apache.org/jira/browse/FLINK-9061 On Thu, Mar 22, 2018 at 11:46 PM, Jamie Grier wrote: > I think we need to modify the way we write checkpoints to S3 for high-scale > jobs (those with many total tasks). The issue is that we

Re: [DISCUSS] GitBox

+1 On Beam, we gradually enabled this, first for the website repo and then for the main repo and we didn't run into problems. > On 16. May 2018, at 12:45, Chesnay Schepler wrote: > > Forget an important feature: It would allow committers to close pull requests. > > On 16.05.2018 12:40, Chesna

[jira] [Created] (FLINK-9381) BlobServer data for a job is not getting cleaned up at JM

Amit Jain created FLINK-9381: Summary: BlobServer data for a job is not getting cleaned up at JM Key: FLINK-9381 URL: https://issues.apache.org/jira/browse/FLINK-9381 Project: Flink Issue Type: B

[jira] [Created] (FLINK-9380) Failing end-to-end tests should not clean up logs

Till Rohrmann created FLINK-9380: Summary: Failing end-to-end tests should not clean up logs Key: FLINK-9380 URL: https://issues.apache.org/jira/browse/FLINK-9380 Project: Flink Issue Type: B

[jira] [Created] (FLINK-9379) HA end-to-end test failing locally

Till Rohrmann created FLINK-9379: Summary: HA end-to-end test failing locally Key: FLINK-9379 URL: https://issues.apache.org/jira/browse/FLINK-9379 Project: Flink Issue Type: Bug Co

Re: [DISCUSS] GitBox

Forget an important feature: It would allow committers to close pull requests. On 16.05.2018 12:40, Chesnay Schepler wrote: Hello, during the discussion about how to better manage pull requests [1] the topic of GitBox integration came up again. This seems like a good opportunity to restart

[DISCUSS] GitBox

Hello, during the discussion about how to better manage pull requests [1] the topic of GitBox integration came up again. This seems like a good opportunity to restart this discussion that we had about a year ago [2]. * What is GitBox Essentially, GitBox allow us to use GitHub features.

Re: [DISCUSS] Adding new interfaces in [Stream]ExecutionEnvironment

Hi Ted, The design doc is in late draft status and proposes support for SQL DDL statements (CREATE TABLE, CREATE FUNCTION, etc.). The question about registering JARs came up because we need a way to distribute JAR files that contain the code of user-defined functions. The design doc will soon be

Re: Elasticsearch Sink

Good to know! Thanks a lot for pushing this Christophe. Please ping me when the new PR is opened, and we can continue the discussion there. Ideally we have this in early in the 1.6 release cycle, so that the Elasticsearch e2e tests (will be merging a PR for that soon) can catching anything unex

[jira] [Created] (FLINK-9378) Improve TableException message with TypeName usage

Sergey Nuyanzin created FLINK-9378: -- Summary: Improve TableException message with TypeName usage Key: FLINK-9378 URL: https://issues.apache.org/jira/browse/FLINK-9378 Project: Flink Issue Ty

[jira] [Created] (FLINK-9377) Remove writing serializers as part of the checkpoint meta information

Tzu-Li (Gordon) Tai created FLINK-9377: -- Summary: Remove writing serializers as part of the checkpoint meta information Key: FLINK-9377 URL: https://issues.apache.org/jira/browse/FLINK-9377 Proje

[jira] [Created] (FLINK-9376) Allow upgrading to incompatible state serializers (state schema evolution)

Tzu-Li (Gordon) Tai created FLINK-9376: -- Summary: Allow upgrading to incompatible state serializers (state schema evolution) Key: FLINK-9376 URL: https://issues.apache.org/jira/browse/FLINK-9376

Re: [DISCUSS] Drop "canEqual" from TypeInformation, TypeSerializer, etc.

+1 TypeInformation has too many methods that need to be implemented but provide little benefit for Flink. Am 16.05.18 um 10:55 schrieb Ted Yu: +1 from me as well. I checked a few serializer classes. The `equals` method on serializers contains the logic of `canEqual` method whose existence se

Re: Rewriting a new file instead of writing a ".valid-length" file inBucketSink when restoring

Hi Till Thanks for your suggestion. A small tool can work lightly and asynchronously. However, I don't know when others will use the data, so I should use the tool to check and truncate the finished file once a valid-length file is found. I think it's hard to maintain it and it shouldn't be ma

Re: Closing (automatically?) inactive pull requests

Hi, I'm not objecting closing stale PRs. We have quite a few PRs with very little chance of being merged and I would certainly appreciate cleaning up those. However, I think we should not automate closing PRs for the reasons I gave before. A tool that reminds us of state PRs as proposed by Till s

Re: [DISCUSS] Drop "canEqual" from TypeInformation, TypeSerializer, etc.

+1 from me as well. I checked a few serializer classes. The `equals` method on serializers contains the logic of `canEqual` method whose existence seems redundant. On Wed, May 16, 2018 at 1:49 AM, Tzu-Li (Gordon) Tai wrote: > +1. > > Looking at the implementations of the `canEqual` method in se

Re: [DISCUSS] Drop "canEqual" from TypeInformation, TypeSerializer, etc.

+1. Looking at the implementations of the `canEqual` method in several serializers, it seems like all that is done is a check whether the object is of the same serializer class. We’ll have to be careful and double check all `equals` method on serializers that may have relied on the `canEqual` m

Re: [DISCUSS] Adding new interfaces in [Stream]ExecutionEnvironment

bq. In a design document, Timo mentioned that we can ship multiple JAR files Mind telling us where the design doc can be retrieved ? Thanks On Wed, May 16, 2018 at 1:29 AM, Fabian Hueske wrote: > Hi, > > I'm not sure if we need to modify the existing method. > What we need is a bit different f

Re: CloudWatch Metrics Reporter

I've written a cloud watch reporter for our own use. It's not pretty to crack out the metrics correctly for cloudwatch as the current metrics don't all set the metric names in a good hierarchy and then they aren't all added to the metric variables either. If someone opens the Jira I can see about

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi, Yes. IMO it makes sense to put the logic into the abstract base classes to share the implementation across different state backends and state primitives. The overhead of storing the key twice is a valid concern, but I'm not sure about the approach to add a timestamp to each value. How would we

[DISCUSS] Drop "canEqual" from TypeInformation, TypeSerializer, etc.

Hi all! As part of an attempt to simplify some code in the TypeInfo and TypeSerializer area, I would like to drop the "canEqual" methods for the following reason: "canEqual()" is necessary to make proper equality checks across hierarchies of types. This is for example useful in a collection API,

Re: [DISCUSS] Adding new interfaces in [Stream]ExecutionEnvironment

Hi, I'm not sure if we need to modify the existing method. What we need is a bit different from what registerCachedFile() provides. The method ensures that a file is copied to each TaskManager and can be locally accessed from a function's RuntimeContext. In our case, we don't need to access the fi

[jira] [Created] (FLINK-9375) Introduce AbortCheckpoint message from JM to TMs

Stefan Richter created FLINK-9375: - Summary: Introduce AbortCheckpoint message from JM to TMs Key: FLINK-9375 URL: https://issues.apache.org/jira/browse/FLINK-9375 Project: Flink Issue Type:

Re: Elasticsearch Sink

Ok thanks for the feedback. > I agree. IIRC, the ES PRs that were opened also did this by changing the return type from Client to AutoClosable, as well as letting the call bridge also handle creation of BulkProcessors, correct? Correct. > Instead, we maintain our own request class to abstract th

Re: CloudWatch Metrics Reporter

Hello, there was no demand for a CloudWatch reporter so far. I only quickly skimmed the API docs, but it appears that the data is inserted via REST. Would the reporter require the usage of any aws library, or could be use an arbitrary http client? If it is the latter there shouldn't be a lice

CloudWatch Metrics Reporter

Hi, In my team we use CloudWatch as our monitoring & alerting system. I noticed that CloudWatch does not appear in the list of supported Reporters. I was wondering why is that. Was there no demand from the community? Is it related to licensing issue with AWS? Was it a technical concern? Would you

[jira] [Created] (FLINK-9374) Flink Kinesis Producer does not backpressure

Franz Thoma created FLINK-9374: -- Summary: Flink Kinesis Producer does not backpressure Key: FLINK-9374 URL: https://issues.apache.org/jira/browse/FLINK-9374 Project: Flink Issue Type: Bug

Re: [VOTE] Release 1.5.0, release candidate #2

Same error on 3.5.2 ... Let me check rc3 also. -- Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/