Re: Flip 6 mesos support

Hi, Till: Issued opened here, https://issues.apache.org/jira/browse/FLINK-9077 On Tue, Mar 27, 2018 at 11:31 PM Till Rohrmann wrote: > Hi Renjie, > > we couldn't take a look at your configuration yet, because the ML filters > attachments out. You could upload your log to https://gist.github.com/

Compressing files with the Bucketing Sink

I want to upload a compressed file (gzip preferrably) using the Bucketing Sink. What is the best way to do this? Would I have to implement my own Writer that does the compression? Has anyone done something similar?

Re: [DISCUSS] Implement end-to-end tests in Java

+1 on Timo's proposal. Bash script can be difficult to extend, debug and maintain as the end-to-end tests become larger and more complex. I would favor Java comparing to Bash script. Another option would be to use scala, it's JVM based and work with Java, but at the same time much more concise. O

Re: [DISCUSS] Implement end-to-end tests in Java

+1 using Java for implementing e2e testing. I have already seem many bash tests reuse logics written in Java from example modules. However creating and maintaining the bash script to extend upon these example cases for testing is kind of painful. Maybe we can start extending the logics for e2e tes

[jira] [Created] (FLINK-9100) Shadow/Hide password from configuration that is logged to log file

Szymon Szczypiński created FLINK-9100: - Summary: Shadow/Hide password from configuration that is logged to log file Key: FLINK-9100 URL: https://issues.apache.org/jira/browse/FLINK-9100 Project: F

[ANNOUNCE] Weekly community update #13

Dear squirrels, here is the next weekly community update thread. Please post any news and updates you want to share with the community here. # Release 1.5 The community is still focused on hardening the Flink 1.5 release and adding more automated end-to-end tests. If you want to help with the ef

Re: [DISCUSS] Implement end-to-end tests in Java

+1 for Timo's proposal. Bash scripts can be difficult to maintain and more complex logic can be hard to express with bash. Thus, implementing logic in Java or Python is a good idea and will help to reuse code. I would also favour Java at the moment, because the rest of Flink is also written in Java

Re: Flip 6 mesos support

Hi Renjie, we couldn't take a look at your configuration yet, because the ML filters attachments out. You could upload your log to https://gist.github.com/ or open a JIRA to which you attach the log. Cheers, Till On Sun, Mar 25, 2018 at 9:04 AM, Renjie Liu wrote: > Hi: > Could you please help

Re: [DISCUSS] Implement end-to-end tests in Java

At this moment, +1 from my side for maintaining bash scripts. Mingleizhang > 在 2018年3月27日，下午9:42，Piotr Nowojski 写道： > > +1 > > I would personally go with Python, but I see a reason to Kostas’s arguments > in favour of Java. Regardless, based on my experience with maintaining bash > scripts,

Re: [DISCUSS] Implement end-to-end tests in Java

Hi Timo, Thank you for providing so many information. - I have some my thoughts on end to end tests. I think such as Kafka, and elastic search . We can not run end to end tests in the IDE for debugging and set breakpoints. For those, we still need implement its logic in bash scripts like dow

SSL config on Kubernetes - Dynamic IP

Hi all, Currently I have a Flink 1.4 cluster running on kubernetes and with SSL configuration based on https://ci.apache.org/projects/flink/flink-docs-master/ops/security-ssl.html . However, as the IP of the nodes are dynamic (from the nature of kubernetes), we are using only the DNS which we can

Re: [DISCUSS] Implement end-to-end tests in Java

+1 I would personally go with Python, but I see a reason to Kostas’s arguments in favour of Java. Regardless, based on my experience with maintaining bash scripts, both of those options (Java/python) are much better in the long run. Piotrek > On 27 Mar 2018, at 15:38, Kostas Kloudas wrote:

Re: [DISCUSS] Implement end-to-end tests in Java

Hi Timo, Thanks for opening this. I agree that bash is not the best tool for all the reasons that you mention plus: 1) it is difficult to write re-usable code 2) there are a lot of ways to express the same thing and difficult to build “best practices” (as it should be in a community proj

[jira] [Created] (FLINK-9099) Failing allocated slots not noticed

Till Rohrmann created FLINK-9099: Summary: Failing allocated slots not noticed Key: FLINK-9099 URL: https://issues.apache.org/jira/browse/FLINK-9099 Project: Flink Issue Type: Bug C

[jira] [Created] (FLINK-9098) ClassLoaderITCase unstable

Stephan Ewen created FLINK-9098: --- Summary: ClassLoaderITCase unstable Key: FLINK-9098 URL: https://issues.apache.org/jira/browse/FLINK-9098 Project: Flink Issue Type: Bug Components:

[DISCUSS] Implement end-to-end tests in Java

Hi everyone, after reviewing a bunch of end-to-end tests, I'm wondering if we should really continue implementing everything in bash scripts. Wouldn't it be nicer to implement them in Java code that just calls the interfaces of Flink (e.g. "./bin/flink run" or REST API)? Here are some though

[jira] [Created] (FLINK-9097) Jobs can be dropped in HA when job submission fails

Till Rohrmann created FLINK-9097: Summary: Jobs can be dropped in HA when job submission fails Key: FLINK-9097 URL: https://issues.apache.org/jira/browse/FLINK-9097 Project: Flink Issue Type:

[jira] [Created] (FLINK-9096) flip 6 allocates different resources for different jobs

Renjie Liu created FLINK-9096: - Summary: flip 6 allocates different resources for different jobs Key: FLINK-9096 URL: https://issues.apache.org/jira/browse/FLINK-9096 Project: Flink Issue Type: I