[jira] [Created] (FLINK-4503) Remove with method from CoGroupedStream and JoinedStream, and change apply method return type

2016-08-25 Thread Jark Wu (JIRA)
Jark Wu created FLINK-4503: -- Summary: Remove with method from CoGroupedStream and JoinedStream, and change apply method return type Key: FLINK-4503 URL: https://issues.apache.org/jira/browse/FLINK-4503 Proje

Re: Additional project downloads

2016-08-25 Thread Greg Hogan
Also the metrics reporters. The circumstances for this request are that I wanted to use the metrics reporters for 1.1.1 and had to go looking on Maven Central (also had to download dependencies, which may be an issue with packaging). I'm also looking to update the Gelly documentation to walkthroug

[jira] [Created] (FLINK-4502) Cassandra connector documentation has misleading consistency guarantees

2016-08-25 Thread Elias Levy (JIRA)
Elias Levy created FLINK-4502: - Summary: Cassandra connector documentation has misleading consistency guarantees Key: FLINK-4502 URL: https://issues.apache.org/jira/browse/FLINK-4502 Project: Flink

[jira] [Created] (FLINK-4501) Cassandra sink can lose messages

2016-08-25 Thread Elias Levy (JIRA)
Elias Levy created FLINK-4501: - Summary: Cassandra sink can lose messages Key: FLINK-4501 URL: https://issues.apache.org/jira/browse/FLINK-4501 Project: Flink Issue Type: Bug Components

[jira] [Created] (FLINK-4500) Cassandra sink can lose messages

2016-08-25 Thread Elias Levy (JIRA)
Elias Levy created FLINK-4500: - Summary: Cassandra sink can lose messages Key: FLINK-4500 URL: https://issues.apache.org/jira/browse/FLINK-4500 Project: Flink Issue Type: Bug Components

Re: Task manager processes crashing one after the other

2016-08-25 Thread Gyula Fóra
Stephan, I ported the fix for the concurrency issue from the Flink commit so now that should be fine. I ran some fail/restore tests and that specific issue hasn't appeared again. However I now get many segfaults in the initializeForJob method where the RocksDb instance is opened. Just for the rec

Fwd: Enabling Encryption between slaves in Flink

2016-08-25 Thread Vinay Patil
Hi, I have a requirement that all the data flowing between the task managers should be encrypted, is there a way in Flink to do that. Can we use the configuration file to enable this as follows : http://doc.akka.io/docs/akka/snapshot/scala/remoting.html#Remoting_Sample or do we need to add the a

Re: Additional project downloads

2016-08-25 Thread Stephan Ewen
The downloads would be just the components' jar files, or everything? At some point, someone suggested to add the jars of all libraries (gelly, ml, ...) and connectors into the download tarball: - bin/ - conf/ - lib/ (core flink runtime and apis) - libraries/ +-gelly/ +-CEP/

[jira] [Created] (FLINK-4499) Introduce findbugs maven plugin

2016-08-25 Thread Ted Yu (JIRA)
Ted Yu created FLINK-4499: - Summary: Introduce findbugs maven plugin Key: FLINK-4499 URL: https://issues.apache.org/jira/browse/FLINK-4499 Project: Flink Issue Type: Improvement Reporter:

Re: Task manager processes crashing one after the other

2016-08-25 Thread Gyula Fóra
Yes seems like that, I remember the fix in Flink. I apparently made a mistake somewhere in our code :) Thanks, Gyula On Thu, Aug 25, 2016, 18:59 Stephan Ewen wrote: > We saw some crashes in earlier versions when native handles in RocksDB > (even for config option objects) were manually and too

Re: Task manager processes crashing one after the other

2016-08-25 Thread Stephan Ewen
We saw some crashes in earlier versions when native handles in RocksDB (even for config option objects) were manually and too eagerly released. Maybe you have a similar issue here? On Thu, Aug 25, 2016 at 6:27 PM, Gyula Fóra wrote: > Hi, > This seems to be a sneaky concurrency issue in our cust

[jira] [Created] (FLINK-4498) Better Cassandra sink documentation

2016-08-25 Thread Elias Levy (JIRA)
Elias Levy created FLINK-4498: - Summary: Better Cassandra sink documentation Key: FLINK-4498 URL: https://issues.apache.org/jira/browse/FLINK-4498 Project: Flink Issue Type: Improvement

Re: Task manager processes crashing one after the other

2016-08-25 Thread Gyula Fóra
Hi, This seems to be a sneaky concurrency issue in our custom statebackend implementation. I made some changes, will keep you posted. Cheers, Gyula On Thu, Aug 25, 2016, 10:54 Gyula Fóra wrote: > Hi, > > Sure I am sending the TM logs in priv. > > Currently what I did was to bump the Rocks vers

[jira] [Created] (FLINK-4497) Add support for Scala tuples and case classes to Cassandra sink

2016-08-25 Thread Elias Levy (JIRA)
Elias Levy created FLINK-4497: - Summary: Add support for Scala tuples and case classes to Cassandra sink Key: FLINK-4497 URL: https://issues.apache.org/jira/browse/FLINK-4497 Project: Flink Issu

[jira] [Created] (FLINK-4496) Refactor the TimeServiceProvider to take a Trigerable instead of a Runnable.

2016-08-25 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-4496: - Summary: Refactor the TimeServiceProvider to take a Trigerable instead of a Runnable. Key: FLINK-4496 URL: https://issues.apache.org/jira/browse/FLINK-4496 Project:

[jira] [Created] (FLINK-4495) Running multiple jobs on yarn (without yarn-session)

2016-08-25 Thread Niels Basjes (JIRA)
Niels Basjes created FLINK-4495: --- Summary: Running multiple jobs on yarn (without yarn-session) Key: FLINK-4495 URL: https://issues.apache.org/jira/browse/FLINK-4495 Project: Flink Issue Type:

[jira] [Created] (FLINK-4494) Expose the TimeServiceProvider from the Task to each Operator.

2016-08-25 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-4494: - Summary: Expose the TimeServiceProvider from the Task to each Operator. Key: FLINK-4494 URL: https://issues.apache.org/jira/browse/FLINK-4494 Project: Flink

[jira] [Created] (FLINK-4493) Unify the snapshot output format for keyed-state backends

2016-08-25 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-4493: - Summary: Unify the snapshot output format for keyed-state backends Key: FLINK-4493 URL: https://issues.apache.org/jira/browse/FLINK-4493 Project: Flink Iss

[jira] [Created] (FLINK-4490) Decouple Slot and Instance

2016-08-25 Thread Kurt Young (JIRA)
Kurt Young created FLINK-4490: - Summary: Decouple Slot and Instance Key: FLINK-4490 URL: https://issues.apache.org/jira/browse/FLINK-4490 Project: Flink Issue Type: Sub-task Reporter:

[jira] [Created] (FLINK-4491) Handle index.number_of_shards in the ES connector

2016-08-25 Thread Flavio Pompermaier (JIRA)
Flavio Pompermaier created FLINK-4491: - Summary: Handle index.number_of_shards in the ES connector Key: FLINK-4491 URL: https://issues.apache.org/jira/browse/FLINK-4491 Project: Flink Iss

[jira] [Created] (FLINK-4492) Cleanup files from canceled snapshots

2016-08-25 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-4492: - Summary: Cleanup files from canceled snapshots Key: FLINK-4492 URL: https://issues.apache.org/jira/browse/FLINK-4492 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-4489) Implement TaskManager's SlotManager

2016-08-25 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-4489: Summary: Implement TaskManager's SlotManager Key: FLINK-4489 URL: https://issues.apache.org/jira/browse/FLINK-4489 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-4488) Prevent cluster shutdown after job execution for non-detached jobs

2016-08-25 Thread Maximilian Michels (JIRA)
Maximilian Michels created FLINK-4488: - Summary: Prevent cluster shutdown after job execution for non-detached jobs Key: FLINK-4488 URL: https://issues.apache.org/jira/browse/FLINK-4488 Project: F

[jira] [Created] (FLINK-4487) Need tools for managing the yarn-session better

2016-08-25 Thread Niels Basjes (JIRA)
Niels Basjes created FLINK-4487: --- Summary: Need tools for managing the yarn-session better Key: FLINK-4487 URL: https://issues.apache.org/jira/browse/FLINK-4487 Project: Flink Issue Type: Impro

[jira] [Created] (FLINK-4486) JobManager not fully running when yarn-session.sh finishes

2016-08-25 Thread Niels Basjes (JIRA)
Niels Basjes created FLINK-4486: --- Summary: JobManager not fully running when yarn-session.sh finishes Key: FLINK-4486 URL: https://issues.apache.org/jira/browse/FLINK-4486 Project: Flink Issue

[jira] [Created] (FLINK-4485) Finished jobs in yarn session fill /tmp filesystem

2016-08-25 Thread Niels Basjes (JIRA)
Niels Basjes created FLINK-4485: --- Summary: Finished jobs in yarn session fill /tmp filesystem Key: FLINK-4485 URL: https://issues.apache.org/jira/browse/FLINK-4485 Project: Flink Issue Type: Bu

[jira] [Created] (FLINK-4484) FLIP-10: Unify Savepoints and Checkpoints

2016-08-25 Thread Ufuk Celebi (JIRA)
Ufuk Celebi created FLINK-4484: -- Summary: FLIP-10: Unify Savepoints and Checkpoints Key: FLINK-4484 URL: https://issues.apache.org/jira/browse/FLINK-4484 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Python API for Fllink libraries

2016-08-25 Thread Chesnay Schepler
Helli Ivan, i don't know why it is the way it is. Regarding issues to work on: You should be able to go through the transformations documentation and see which are not supported. Regards, Chesnay On 21.08.2016 01:11, Ivan Mushketyk wrote: Hi Chesnay, Thank you for you repply. Out of curios

Re: Task manager processes crashing one after the other

2016-08-25 Thread Gyula Fóra
Hi, Sure I am sending the TM logs in priv. Currently what I did was to bump the Rocks version to 4.9.0 let's see if that helps. Cheers, Gyula Till Rohrmann ezt írta (időpont: 2016. aug. 25., Cs, 10:35): > Hi Gyula, > > I haven't seen this problem before. Do you have the logs of the failed TMs

Re: Task manager processes crashing one after the other

2016-08-25 Thread Till Rohrmann
Hi Gyula, I haven't seen this problem before. Do you have the logs of the failed TMs so that we have some more context what was going on? Cheers, Till On Thu, Aug 25, 2016 at 9:40 AM, Gyula Fóra wrote: > Hi guys, > > For quite some time now we fairly frequently experience a task manager > cras

Task manager processes crashing one after the other

2016-08-25 Thread Gyula Fóra
Hi guys, For quite some time now we fairly frequently experience a task manager crashes around the time new streaming jobs are deployed. We use RocksDB backend so this might be related. We tried changing the GC from G1 to CMS that didnt help. Yesterday for instance 6 task managers crashed one of