[jira] [Created] (FLINK-9373) Always call RocksIterator.status() to check the internal error of RocksDB

Sihua Zhou created FLINK-9373: - Summary: Always call RocksIterator.status() to check the internal error of RocksDB Key: FLINK-9373 URL: https://issues.apache.org/jira/browse/FLINK-9373 Project: Flink

Re: Elasticsearch Sink

Hi, What if the user in a ES5.3+ case is calling the deprecated method? You  agree it will fail? I'm not necessarily against that. I just want to make  it clear that we don't have a perfect solution here either? I think what we could do here is at least in the deprecated `add(ActionRequest)` met

?????? Rewriting a new file instead of writing a ".valid-length" file inBucketSink when restoring

Hi gary Our Hadoop version is very low and does not support truncate function. Some people are working on upgrading it. I just think the ".valid-length" file is not friendly enough for users. Maybe it should be configurable to choose whether using the ".valid-length" file at least. Regards,

Re: Closing (automatically?) inactive pull requests

I have questions in this regard (you guys might have addressed it in this email chain): how PRs get assigned to a reviewer apart of a contributor tag someone? what if PR never gets a reviewer attention and it became in-active due to long review respond? should Bot assign a reviewer to a PR base

Re: Support more intelligent function lookup in FunctionCatalog for UDF

Thanks Fabian & Timo for the comments and suggestions. I agree we should go step-by-step to enable these functionalities. I will start to consider & fill in the implementation part and create umbrella tickets. Best, Rong On Tue, May 15, 2018 at 6:54 AM, Timo Walther wrote: > I added some comme

[jira] [Created] (FLINK-9372) Typo on Elasticsearch website link (elastic.io --> elastic.co)

Yazdan Shirvany created FLINK-9372: -- Summary: Typo on Elasticsearch website link (elastic.io --> elastic.co) Key: FLINK-9372 URL: https://issues.apache.org/jira/browse/FLINK-9372 Project: Flink

Re: Elasticsearch Sink

Hi Gordon, On Tue, May 15, 2018 at 6:16 AM, Tzu-Li (Gordon) Tai wrote: > Hi, > > Let me first clarify a few things so that we are on the same page here: > > 1. The reason that we are looking into having a new base-module for > versions 5.3+ is that > new Elasticsearch BulkProcessor APIs are brea

[jira] [Created] (FLINK-9371) High Availability JobManager Registration Failure

Jason Kania created FLINK-9371: -- Summary: High Availability JobManager Registration Failure Key: FLINK-9371 URL: https://issues.apache.org/jira/browse/FLINK-9371 Project: Flink Issue Type: Bug

Re: Closing (automatically?) inactive pull requests

I like Till's proposal to notify the participants on the PR to PTAL. But I would also suggest to auto-close when no action is taken, with a friendly reminder that PRs can be reopened anytime. The current situation with 350 open PRs may send a signal to contributors that it may actually be too much

Re: [VOTE] Release 1.5.0, release candidate #2

Hi, Regarding the Kinesis consumer, fwiw we backported most of the changes to 1.4.x in our fork and running our pipelines with those. Thanks, Thomas On Tue, May 15, 2018 at 5:50 AM, Till Rohrmann wrote: > Thanks for the feedback. I actually have to cancel this RC because of the > following is

[VOTE] Release 1.5.0, release candidate #3

Hi everyone, Please review and vote on the release candidate #3 for the version 1.5.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], *

Re: 回复： Rewriting a new file instead of writing a ".valid-length" file inBucketSink when restoring

Hi Xinyu, would it help to have a small tool which can truncate the finished files which have a valid-length file associated? That way, one could use this tool before others are using the data farther down stream. Cheers, Till On Tue, May 15, 2018 at 3:05 PM, Xinyu Zhang <342689...@qq.com> wrote

Re: Closing (automatically?) inactive pull requests

How does the bot decide whether the PR is waiting for reviews or is being abandoned by contributor ? How about letting the bot count the number of times contributor pings committer(s) for review ? When unanswered ping count crosses some threshold, say 3, the bot publishes the JIRA and PR somewhere

Re: Closing (automatically?) inactive pull requests

I'm a bit torn here because I can see the pros and cons for both sides. Maybe a compromise could be to not have a closing but a monitoring bot which notifies us about inactive PRs. This could then trigger an investigation of the underlying problem and ultimately lead to a conscious decision to clo

Re: Support more intelligent function lookup in FunctionCatalog for UDF

I added some comments to your documents. I think we should work on these limitations step by step. A first step could be to support Map?> by considering only the raw types. Another step would be to allow eval(Object) as a wild card for operands. Regards, Timo Am 14.05.18 um 18:23 schrieb Rong

Re: Closing (automatically?) inactive pull requests

/So far I did it twice for older PRs. In both cases I didn’t get any response and I even forgot in which PRs I had asked this question, so now I can not even close them :S/ To be honest this sounds more like an issue with how your organize your work. No amount of closing PRs can fix that. With

?????? ?????? Rewriting a new file instead of writing a ".valid-length" file inBucketSink when restoring

Yes, I'm glad to do it. but I'm not sure writing a new file is a good solution. So I want to discuss it here. Do you have any ideas? @Kostas -- -- ??: "twalthr"; : 2018??5??15??(??) 8:21 ??: "Xinyu Zhang"<342689...@qq.com>;

[jira] [Created] (FLINK-9370) Activate distributed cache end-to-end test

Chesnay Schepler created FLINK-9370: --- Summary: Activate distributed cache end-to-end test Key: FLINK-9370 URL: https://issues.apache.org/jira/browse/FLINK-9370 Project: Flink Issue Type: Bu

Re: [VOTE] Release 1.5.0, release candidate #2

We've had the enforcer-plugin issue before and neither Timo nor me could reproduce it locally. See FLINK-9091 and FLINK-9145 . On 15.05.2018 14:50, Till Rohrmann wrote: Thanks for the feedbac

Re: [VOTE] Release 1.5.0, release candidate #2

Thanks for the feedback. I actually have to cancel this RC because of the following issues [1, 2, 3, 4]. The issues are already fixed and I will create a new RC asap. @Ted, thanks for reporting the problem with the BindException. We should look into why this happens but as Stephan said it should n

Re: Closing (automatically?) inactive pull requests

I agree that we have other, even more important, problems with reviewing PR and community, but that shouldn’t block us from trying to clean up things a little bit and minimise the effort needed for reviewing PRs. Now before reviewing/picking older PRs I had to ask this “Hey, are you still intere

Re: 回复： Rewriting a new file instead of writing a ".valid-length" file inBucketSink when restoring

As far as I know, the bucketing sink is currenlty also limited by relying on Hadoops file system abstraction. It is planned to switch to Flink's file system abstraction which might also improve this situation. Kostas (in CC) might know more about it. But I think we can discuss if an other beha

Re: Rewriting a new file instead of writing a ".valid-length" file in BucketSink when restoring

Hi Timo, Should not we delegate the recovery option to the user? I think we can ask the user to provide Reader to respective Writer class and save valid-length info in operator state apart from the current flow. According to user chosen recovery option, we can stream the Reader output to Writer cl

[jira] [Created] (FLINK-9369) DistributedCache returns parent directory for directories

Chesnay Schepler created FLINK-9369: --- Summary: DistributedCache returns parent directory for directories Key: FLINK-9369 URL: https://issues.apache.org/jira/browse/FLINK-9369 Project: Flink

[jira] [Created] (FLINK-9368) End-to-end test: Python API

Chesnay Schepler created FLINK-9368: --- Summary: End-to-end test: Python API Key: FLINK-9368 URL: https://issues.apache.org/jira/browse/FLINK-9368 Project: Flink Issue Type: Sub-task

?????? Rewriting a new file instead of writing a ".valid-length" file inBucketSink when restoring

Thanks for your reply. Indeed, if a file is very large, it will take a long time. However, the the ??.valid-length?? file is not convenient for others who use the data in HDFS. Maybe we should provide a configuration for users to choose which strategy they prefer. Do you have any ideas? -

Re: Rewriting a new file instead of writing a ".valid-length" file in BucketSink when restoring

I guess writing a new file would take much longer than just using the .valid-length file, especially if the files are very large. The restoring time should be as minimal as possible to ensure little downtime on restarts. Regards, Timo Am 15.05.18 um 09:31 schrieb Gary Yao: Hi, The Bucketin

[jira] [Created] (FLINK-9367) Truncate() in BucketingSink is only allowed after hadoop2.7

zhangxinyu created FLINK-9367: - Summary: Truncate() in BucketingSink is only allowed after hadoop2.7 Key: FLINK-9367 URL: https://issues.apache.org/jira/browse/FLINK-9367 Project: Flink Issue Ty

[jira] [Created] (FLINK-9366) Distribute Cache only works for client-accessible files

Chesnay Schepler created FLINK-9366: --- Summary: Distribute Cache only works for client-accessible files Key: FLINK-9366 URL: https://issues.apache.org/jira/browse/FLINK-9366 Project: Flink I

[jira] [Created] (FLINK-9365) Add version information to remote rpc messages

Till Rohrmann created FLINK-9365: Summary: Add version information to remote rpc messages Key: FLINK-9365 URL: https://issues.apache.org/jira/browse/FLINK-9365 Project: Flink Issue Type: Sub-

Re: Closing (automatically?) inactive pull requests

I'm with Chesnay on this issue. Stale PRs, i.e., a PR where a contributor becomes inactive, are one of our smallest issues, IMO. There are more reasons for the high number of PRs. * Lack of timely reviews. * Not eagerly closing PRs that have no or very little chance of being merged. Common reasons

Re: Rewriting a new file instead of writing a ".valid-length" file in BucketSink when restoring

Hi, The BucketingSink truncates the file if the Hadoop FileSystem supports this operation (Hadoop 2.7 and above) [1]. What version of Hadoop are you using? Best, Gary [1] https://github.com/apache/flink/blob/bcd028d75b0e5c5c691e24640a2196b2fdaf85e0/flink-connectors/flink-connector-filesystem/src

Re: Closing (automatically?) inactive pull requests

-1 For clarification (since the original mail indicates otherwise), the number of pull requests that this would affect is fairly small. Only about 25-30% of all open PRs are blocked on the contributor, the remaining ones are actually blocked on the review. Thus is reject the premise that one ha