[jira] [Created] (FLINK-4053) Return value from Connection should be checked against null

2016-06-09 Thread Ted Yu (JIRA)
Ted Yu created FLINK-4053: - Summary: Return value from Connection should be checked against null Key: FLINK-4053 URL: https://issues.apache.org/jira/browse/FLINK-4053 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-4052) Unstable test

2016-06-09 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-4052: --- Summary: Unstable test Key: FLINK-4052 URL: https://issues.apache.org/jira/browse/FLINK-4052 Project: Flink Issue Type: Bug Components: Tests Aff

Re: incremental Checkpointing , Rocksdb HA

2016-06-09 Thread Nick Dimiduk
IIRC, all the above support data locality from back in the MR days. Not sure how much data you're planning to checkpoint though -- is locality really that important for transient processor state? On Thu, Jun 9, 2016 at 11:06 AM, CPC wrote: > Cassandra backend would be interesting especially if

Re: incremental Checkpointing , Rocksdb HA

2016-06-09 Thread CPC
Cassandra backend would be interesting especially if flink could benefit from cassandra data locality. Cassandra/spark integration is using this for information to schedule spark tasks. On 9 June 2016 at 19:55, Nick Dimiduk wrote: > You might also consider support for a Bigtable > backend: HBas

Re: incremental Checkpointing , Rocksdb HA

2016-06-09 Thread Nick Dimiduk
You might also consider support for a Bigtable backend: HBase/Accumulo/Cassandra. The data model should be similar (identical?) to RocksDB and you get HA, recoverability, and support for really large state "for free". On Thursday, June 9, 2016, Chen Qin wrote: > Hi there, > > What is progress on

incremental Checkpointing , Rocksdb HA

2016-06-09 Thread Chen Qin
Hi there, What is progress on incremental checkpointing? Does flink dev has plan to work on this or JIRA to track this? super interested to know. I also research and consider use rocksdbstatebackend without running HDFS cluster nor talk to S3. Some primitive idea is to use ZK to store / notify st

[jira] [Created] (FLINK-4051) RabbitMQ Source might not react to cancel signal

2016-06-09 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-4051: - Summary: RabbitMQ Source might not react to cancel signal Key: FLINK-4051 URL: https://issues.apache.org/jira/browse/FLINK-4051 Project: Flink Issue Type:

[jira] [Created] (FLINK-4050) FlinkKafkaProducer API Refactor

2016-06-09 Thread Elias Levy (JIRA)
Elias Levy created FLINK-4050: - Summary: FlinkKafkaProducer API Refactor Key: FLINK-4050 URL: https://issues.apache.org/jira/browse/FLINK-4050 Project: Flink Issue Type: Improvement Com

[jira] [Created] (FLINK-4049) Mark RichInputFormat.openInputFormat and closeInputFormat as @PublicEvolving

2016-06-09 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-4049: - Summary: Mark RichInputFormat.openInputFormat and closeInputFormat as @PublicEvolving Key: FLINK-4049 URL: https://issues.apache.org/jira/browse/FLINK-4049 Project:

[jira] [Created] (FLINK-4048) Remove Hadoop dependencies from ExecutionEnvironment

2016-06-09 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-4048: - Summary: Remove Hadoop dependencies from ExecutionEnvironment Key: FLINK-4048 URL: https://issues.apache.org/jira/browse/FLINK-4048 Project: Flink Issue Ty

[jira] [Created] (FLINK-4047) Fix documentation about determinism of KeySelectors

2016-06-09 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-4047: Summary: Fix documentation about determinism of KeySelectors Key: FLINK-4047 URL: https://issues.apache.org/jira/browse/FLINK-4047 Project: Flink Issue Type:

[jira] [Created] (FLINK-4046) Failing a restarting job can get stuck in JobStatus.FAILING

2016-06-09 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-4046: Summary: Failing a restarting job can get stuck in JobStatus.FAILING Key: FLINK-4046 URL: https://issues.apache.org/jira/browse/FLINK-4046 Project: Flink Is

[jira] [Created] (FLINK-4045) Test AMQP connector using Apache ActiveMQ

2016-06-09 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-4045: - Summary: Test AMQP connector using Apache ActiveMQ Key: FLINK-4045 URL: https://issues.apache.org/jira/browse/FLINK-4045 Project: Flink Issue Type: Sub-tas

[jira] [Created] (FLINK-4044) Rename RabbitMQ connector to AMQP connector

2016-06-09 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-4044: - Summary: Rename RabbitMQ connector to AMQP connector Key: FLINK-4044 URL: https://issues.apache.org/jira/browse/FLINK-4044 Project: Flink Issue Type: Sub-t

[jira] [Created] (FLINK-4043) Generalize RabbitMQ connector into AMQP connector

2016-06-09 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-4043: - Summary: Generalize RabbitMQ connector into AMQP connector Key: FLINK-4043 URL: https://issues.apache.org/jira/browse/FLINK-4043 Project: Flink Issue Type:

[jira] [Created] (FLINK-4042) TwitterStream example does not work

2016-06-09 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-4042: Summary: TwitterStream example does not work Key: FLINK-4042 URL: https://issues.apache.org/jira/browse/FLINK-4042 Project: Flink Issue Type: Bug C

[jira] [Created] (FLINK-4041) Failure while asking ResourceManager for RegisterResource

2016-06-09 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-4041: - Summary: Failure while asking ResourceManager for RegisterResource Key: FLINK-4041 URL: https://issues.apache.org/jira/browse/FLINK-4041 Project: Flink Iss

Re: Broadcast data sent increases with # slots per TM

2016-06-09 Thread Felix Neutatz
Hi everybody, could we use the org.apache.flink.api.common.cache.DistributedCache to work around this Broadcast issue for the moment, until we fixed it? Or do you think it won't scale either? Best regards, Felix 2016-06-09 10:57 GMT+02:00 Stephan Ewen : > Till is right. Broadcast joins currentl

[jira] [Created] (FLINK-4040) Same env.java.opts is applied for TM , JM and ZK

2016-06-09 Thread Gyula Fora (JIRA)
Gyula Fora created FLINK-4040: - Summary: Same env.java.opts is applied for TM , JM and ZK Key: FLINK-4040 URL: https://issues.apache.org/jira/browse/FLINK-4040 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-4039) G1 garbage collector only set if no custom jvm arguments

2016-06-09 Thread Gyula Fora (JIRA)
Gyula Fora created FLINK-4039: - Summary: G1 garbage collector only set if no custom jvm arguments Key: FLINK-4039 URL: https://issues.apache.org/jira/browse/FLINK-4039 Project: Flink Issue Type:

[jira] [Created] (FLINK-4038) Impossible to set more than 1 JVM argument in env.java.opts

2016-06-09 Thread Gyula Fora (JIRA)
Gyula Fora created FLINK-4038: - Summary: Impossible to set more than 1 JVM argument in env.java.opts Key: FLINK-4038 URL: https://issues.apache.org/jira/browse/FLINK-4038 Project: Flink Issue Ty

Re: [PROPOSAL] Structure the Flink Open Source Development

2016-06-09 Thread Till Rohrmann
I agree. I could be the third backup if you need help with the component. On Thu, Jun 9, 2016 at 11:33 AM, Aljoscha Krettek wrote: > Should probably, yes. > > On Thu, 9 Jun 2016 at 10:53 Stephan Ewen wrote: > > > Should state bakends and checkpointing go together? > > > > The two of us could be

Re: [PROPOSAL] Structure the Flink Open Source Development

2016-06-09 Thread Aljoscha Krettek
Should probably, yes. On Thu, 9 Jun 2016 at 10:53 Stephan Ewen wrote: > Should state bakends and checkpointing go together? > > The two of us could be shepherds for that. Till would be another person > (but he has a lot of components already). > > On Wed, Jun 8, 2016 at 9:22 PM, Aljoscha Krettek

[jira] [Created] (FLINK-4037) Introduce ArchivedExecutionGraph without any user classes

2016-06-09 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-4037: - Summary: Introduce ArchivedExecutionGraph without any user classes Key: FLINK-4037 URL: https://issues.apache.org/jira/browse/FLINK-4037 Project: Flink Iss

Re: Broadcast data sent increases with # slots per TM

2016-06-09 Thread Stephan Ewen
Till is right. Broadcast joins currently materialize once per slot. Originally, the purely push based runtime was not good enough to handle it differently. By now, we could definitely handle BC Vars differently (only one slot per TM requests). For BC Joins, the hash tables do not coordinate spilli

Re: [PROPOSAL] Structure the Flink Open Source Development

2016-06-09 Thread Stephan Ewen
Should state bakends and checkpointing go together? The two of us could be shepherds for that. Till would be another person (but he has a lot of components already). On Wed, Jun 8, 2016 at 9:22 PM, Aljoscha Krettek wrote: > I think it would make sense to also move "State Backends" out from > "R

[jira] [Created] (FLINK-4036) Potential deadlock in StreamIterationSink

2016-06-09 Thread Ufuk Celebi (JIRA)
Ufuk Celebi created FLINK-4036: -- Summary: Potential deadlock in StreamIterationSink Key: FLINK-4036 URL: https://issues.apache.org/jira/browse/FLINK-4036 Project: Flink Issue Type: Bug

Re: Broadcast data sent increases with # slots per TM

2016-06-09 Thread Till Rohrmann
If I'm not mistaken, then broadcast variables and broadcast inputs of joins follow different code paths. Broadcast variables use additional input channels and are read before the actual driver code runs. In contrast to that, a join operation is a two input operator where the join driver decides how