[jira] [Commented] (FLINK-4253) Rename "recovery.mode" config key to "high-availability"

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414673#comment-15414673 ] ASF GitHub Bot commented on FLINK-4253: --- Github user ramkrish86 commented on the iss

[GitHub] flink issue #2342: FLINK-4253 - Rename "recovery.mode" config key to "high-a...

2016-08-09 Thread ramkrish86
Github user ramkrish86 commented on the issue: https://github.com/apache/flink/pull/2342 Looks like only this build failed `https://travis-ci.org/apache/flink/jobs/150992610` and that too due to a cassandra-connector test case. Should be unrelated to this PR. --- If your project

[jira] [Updated] (FLINK-4347) Implement SlotManager for new ResourceManager

2016-08-09 Thread Kurt Young (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-4347: -- Description: The slot manager is responsible to maintain the list of slot requests and slot allocations.

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread wenlong88
Github user wenlong88 commented on the issue: https://github.com/apache/flink/pull/2345 @aljoscha I am curious at what problem and incompatibility between semi-aync snapshot and key-group, can you explain some background information? --- If your project is set up for it, you can repl

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414640#comment-15414640 ] ASF GitHub Bot commented on FLINK-4340: --- Github user wenlong88 commented on the issu

[jira] [Created] (FLINK-4347) Implement SlotManager for new ResourceManager

2016-08-09 Thread Kurt Young (JIRA)
Kurt Young created FLINK-4347: - Summary: Implement SlotManager for new ResourceManager Key: FLINK-4347 URL: https://issues.apache.org/jira/browse/FLINK-4347 Project: Flink Issue Type: Sub-task

[jira] [Commented] (FLINK-4341) Checkpoint state size grows unbounded when task parallelism not uniform

2016-08-09 Thread Scott Kidder (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414599#comment-15414599 ] Scott Kidder commented on FLINK-4341: - Unfortunately I don't have the time to develop

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414591#comment-15414591 ] ASF GitHub Bot commented on FLINK-4340: --- Github user wenlong88 commented on the issu

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread wenlong88
Github user wenlong88 commented on the issue: https://github.com/apache/flink/pull/2345 @StephanEwen http://rocksdb.org/blog/2609/use-checkpoints-for-efficient-snapshots/ since sst files is immutable once created, in when doing checkpoint, rocksdb creates hard link for all live s

[jira] [Created] (FLINK-4346) Implement basic RPC abstraction

2016-08-09 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-4346: --- Summary: Implement basic RPC abstraction Key: FLINK-4346 URL: https://issues.apache.org/jira/browse/FLINK-4346 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-4345) Implement new ResourceManager

2016-08-09 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-4345: --- Summary: Implement new ResourceManager Key: FLINK-4345 URL: https://issues.apache.org/jira/browse/FLINK-4345 Project: Flink Issue Type: New Feature C

[jira] [Created] (FLINK-4344) Implement new JobManager

2016-08-09 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-4344: --- Summary: Implement new JobManager Key: FLINK-4344 URL: https://issues.apache.org/jira/browse/FLINK-4344 Project: Flink Issue Type: New Feature Compon

[jira] [Commented] (FLINK-4341) Checkpoint state size grows unbounded when task parallelism not uniform

2016-08-09 Thread Stephan Ewen (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414334#comment-15414334 ] Stephan Ewen commented on FLINK-4341: - Thanks for reporting this issue. Sounds pretty

[jira] [Updated] (FLINK-4341) Checkpoint state size grows unbounded when task parallelism not uniform

2016-08-09 Thread Stephan Ewen (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen updated FLINK-4341: Priority: Critical (was: Major) > Checkpoint state size grows unbounded when task parallelism not u

[jira] [Updated] (FLINK-4319) Rework Cluster Management (FLIP-6)

2016-08-09 Thread Stephan Ewen (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen updated FLINK-4319: Description: This is the root issue to track progress of the rework of cluster management (FLIP-6)

[jira] [Created] (FLINK-4343) Implement new TaskManager

2016-08-09 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-4343: --- Summary: Implement new TaskManager Key: FLINK-4343 URL: https://issues.apache.org/jira/browse/FLINK-4343 Project: Flink Issue Type: New Feature Compo

[jira] [Commented] (FLINK-4341) Checkpoint state size grows unbounded when task parallelism not uniform

2016-08-09 Thread Scott Kidder (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414301#comment-15414301 ] Scott Kidder commented on FLINK-4341: - I also noticed that when checkpointing is enabl

[GitHub] flink pull request #2346: [FLINK-4342] [build] Fix dependencies of flink-con...

2016-08-09 Thread StephanEwen
GitHub user StephanEwen opened a pull request: https://github.com/apache/flink/pull/2346 [FLINK-4342] [build] Fix dependencies of flink-connector-filesystem This fixes dependencies of the `flink-connector-filesystem` project - Remove unneeded Guava dependency - Set h

[jira] [Commented] (FLINK-4342) Fix dependencies of flink-connector-filesystem

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414219#comment-15414219 ] ASF GitHub Bot commented on FLINK-4342: --- GitHub user StephanEwen opened a pull reque

[jira] [Created] (FLINK-4342) Fix dependencies of flink-connector-filesystem

2016-08-09 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-4342: --- Summary: Fix dependencies of flink-connector-filesystem Key: FLINK-4342 URL: https://issues.apache.org/jira/browse/FLINK-4342 Project: Flink Issue Type: Bug

[jira] [Commented] (FLINK-4326) Flink start-up scripts should optionally start services on the foreground

2016-08-09 Thread JIRA
[ https://issues.apache.org/jira/browse/FLINK-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414182#comment-15414182 ] Ismaël Mejía commented on FLINK-4326: - Well that's the question I was wondering before

[jira] [Created] (FLINK-4341) Checkpoint state size grows unbounded when task parallelism not uniform

2016-08-09 Thread Scott Kidder (JIRA)
Scott Kidder created FLINK-4341: --- Summary: Checkpoint state size grows unbounded when task parallelism not uniform Key: FLINK-4341 URL: https://issues.apache.org/jira/browse/FLINK-4341 Project: Flink

[jira] [Commented] (FLINK-4326) Flink start-up scripts should optionally start services on the foreground

2016-08-09 Thread Greg Hogan (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414167#comment-15414167 ] Greg Hogan commented on FLINK-4326: --- The configuration parsing in {{jobmanager.sh}}, {{t

[jira] [Commented] (FLINK-3623) Adjust MurmurHash algorithm

2016-08-09 Thread Stefan Richter (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414110#comment-15414110 ] Stefan Richter commented on FLINK-3623: --- In my opinion, our implementation of Murmur

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414086#comment-15414086 ] ASF GitHub Bot commented on FLINK-4340: --- Github user StephanEwen commented on the is

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread StephanEwen
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2345 @wenlong88 Can you explain this a bit more? How exactly are you creating the local checkpoint? --- If your project is set up for it, you can reply to this email and have your reply appear on Gi

[jira] [Updated] (FLINK-4309) Potential null pointer dereference in DelegatingConfiguration#keySet()

2016-08-09 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated FLINK-4309: -- Description: {code} final int prefixLen = this.prefix == null ? 0 : this.prefix.length(); for (String k

[GitHub] flink pull request #2343: [FLINK-4337] [build] Remove unnecessary Scala Suff...

2016-08-09 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2343 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (FLINK-4337) Remove unnecessary Scala suffix from Hadoop1 artifact

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413919#comment-15413919 ] ASF GitHub Bot commented on FLINK-4337: --- Github user asfgit closed the pull request

[jira] [Closed] (FLINK-4337) Remove unnecessary Scala suffix from Hadoop1 artifact

2016-08-09 Thread Stephan Ewen (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-4337. --- > Remove unnecessary Scala suffix from Hadoop1 artifact >

[jira] [Resolved] (FLINK-4337) Remove unnecessary Scala suffix from Hadoop1 artifact

2016-08-09 Thread Stephan Ewen (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-4337. - Resolution: Fixed Fixed via aed7a2872a475f1d69b0fa11d837d0d56c06e825 > Remove unnecessary Scala s

[jira] [Commented] (FLINK-4253) Rename "recovery.mode" config key to "high-availability"

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413886#comment-15413886 ] ASF GitHub Bot commented on FLINK-4253: --- Github user ramkrish86 commented on a diff

[GitHub] flink pull request #2342: FLINK-4253 - Rename "recovery.mode" config key to ...

2016-08-09 Thread ramkrish86
Github user ramkrish86 commented on a diff in the pull request: https://github.com/apache/flink/pull/2342#discussion_r74105158 --- Diff: flink-runtime/src/test/scala/org/apache/flink/runtime/testingUtils/TestingUtils.scala --- @@ -502,7 +503,8 @@ object TestingUtils { c

[jira] [Updated] (FLINK-4329) Fixes Streaming File Source Timestamps/Watermarks Handling

2016-08-09 Thread Kostas Kloudas (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kostas Kloudas updated FLINK-4329: -- Summary: Fixes Streaming File Source Timestamps/Watermarks Handling (was: Streaming File Source

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413789#comment-15413789 ] ASF GitHub Bot commented on FLINK-3929: --- Github user vijikarthi commented on the iss

[jira] [Commented] (FLINK-4326) Flink start-up scripts should optionally start services on the foreground

2016-08-09 Thread Milosz Tanski (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413790#comment-15413790 ] Milosz Tanski commented on FLINK-4326: -- [~greghogan] A lot of the traditional things

[GitHub] flink issue #2275: FLINK-3929 Support for Kerberos Authentication with Keyta...

2016-08-09 Thread vijikarthi
Github user vijikarthi commented on the issue: https://github.com/apache/flink/pull/2275 @mxm - Could you please let me know if you are okay with the modifications to the integration test case scenarios that I have mentioned. I am open to keep just 3 classes for each scenarios (HDFS,

[GitHub] flink issue #2002: Support for bz2 compression in flink-core

2016-08-09 Thread mtanski
Github user mtanski commented on the issue: https://github.com/apache/flink/pull/2002 @StephanEwen would you like me to use the same version as the one being current include (transitive) or bump it to a recent version that include Snappy? --- If your project is set up for it, you can

[jira] [Commented] (FLINK-4282) Add Offset Parameter to WindowAssigners

2016-08-09 Thread Aditi Viswanathan (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413733#comment-15413733 ] Aditi Viswanathan commented on FLINK-4282: -- Yes - exactly. So it's just the windo

[GitHub] flink issue #2289: [FLINK-3866] StringArraySerializer type should be mutable

2016-08-09 Thread mushketyk
Github user mushketyk commented on the issue: https://github.com/apache/flink/pull/2289 Hi, Could someone please review this small bug-fix? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[jira] [Commented] (FLINK-3866) StringArraySerializer claims type is immutable; shouldn't

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413711#comment-15413711 ] ASF GitHub Bot commented on FLINK-3866: --- Github user mushketyk commented on the issu

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413665#comment-15413665 ] ASF GitHub Bot commented on FLINK-4271: --- Github user StephanEwen commented on the is

[GitHub] flink issue #2305: [FLINK-4271] [DataStreamAPI] Enable CoGroupedStreams and ...

2016-08-09 Thread StephanEwen
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2305 We could "deprecate" the `apply()` method and redirect to `with(...)`. That would be easier to find than a comment about casting. --- If your project is set up for it, you can reply to this emai

[jira] [Closed] (FLINK-3779) Add support for queryable state

2016-08-09 Thread Ufuk Celebi (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ufuk Celebi closed FLINK-3779. -- Resolution: Implemented Fix Version/s: 1.2.0 Implemented in a909adb~1..490e7eb (master). > Add s

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413649#comment-15413649 ] ASF GitHub Bot commented on FLINK-4271: --- Github user aljoscha commented on the issue

[GitHub] flink issue #2305: [FLINK-4271] [DataStreamAPI] Enable CoGroupedStreams and ...

2016-08-09 Thread aljoscha
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/2305 Wouldn't this suffer from the same problem as the "casting solution"? People would use `apply` and then wonder why there is no `setParallelism`, not bothering to read the Javadoc to find out that th

[jira] [Commented] (FLINK-3779) Add support for queryable state

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413647#comment-15413647 ] ASF GitHub Bot commented on FLINK-3779: --- Github user asfgit closed the pull request

[GitHub] flink pull request #2051: [FLINK-3779] Add support for queryable state

2016-08-09 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2051 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (FLINK-4282) Add Offset Parameter to WindowAssigners

2016-08-09 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413645#comment-15413645 ] Aljoscha Krettek commented on FLINK-4282: - The {{WindowAssigner}} should not try t

[GitHub] flink pull request #2344: [hotfix][build] Remove Scala suffix from Hadoop1 s...

2016-08-09 Thread rmetzger
Github user rmetzger closed the pull request at: https://github.com/apache/flink/pull/2344 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is en

[GitHub] flink issue #2344: [hotfix][build] Remove Scala suffix from Hadoop1 shading ...

2016-08-09 Thread rmetzger
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2344 True. Thank you, I'll close this one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabl

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413641#comment-15413641 ] ASF GitHub Bot commented on FLINK-4271: --- Github user StephanEwen commented on the is

[GitHub] flink issue #2344: [hotfix][build] Remove Scala suffix from Hadoop1 shading ...

2016-08-09 Thread uce
Github user uce commented on the issue: https://github.com/apache/flink/pull/2344 Duplicate of #2343? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] flink issue #2305: [FLINK-4271] [DataStreamAPI] Enable CoGroupedStreams and ...

2016-08-09 Thread StephanEwen
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2305 I just thought of a different trick: We could add a second variant of the `apply(..)` function (for example called `with(...)` as in the DataSet API) and have the proper return type there (callin

[jira] [Commented] (FLINK-4334) Shaded Hadoop1 jar not fully excluded in Quickstart

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413637#comment-15413637 ] ASF GitHub Bot commented on FLINK-4334: --- Github user rehevkor5 commented on the issu

[GitHub] flink issue #2341: [FLINK-4334] [quickstarts] Correctly exclude hadoop1 in q...

2016-08-09 Thread rehevkor5
Github user rehevkor5 commented on the issue: https://github.com/apache/flink/pull/2341 You got it @StephanEwen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishe

[jira] [Commented] (FLINK-4334) Shaded Hadoop1 jar not fully excluded in Quickstart

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413638#comment-15413638 ] ASF GitHub Bot commented on FLINK-4334: --- Github user rehevkor5 closed the pull reque

[GitHub] flink pull request #2341: [FLINK-4334] [quickstarts] Correctly exclude hadoo...

2016-08-09 Thread rehevkor5
Github user rehevkor5 closed the pull request at: https://github.com/apache/flink/pull/2341 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is e

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413596#comment-15413596 ] ASF GitHub Bot commented on FLINK-4340: --- Github user wenlong88 commented on the issu

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread wenlong88
Github user wenlong88 commented on the issue: https://github.com/apache/flink/pull/2345 @aljoscha we use rocksdb checkpoint machanism to do the semi-async checkpoint, which use hard link to make checkpoint, cost quite a little IO and time in synchronized phrase. This works well even w

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread gyfora
Github user gyfora commented on the issue: https://github.com/apache/flink/pull/2345 Hi, Isn't this way of checkpointing is much much slower then the semi async version? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413579#comment-15413579 ] ASF GitHub Bot commented on FLINK-4340: --- Github user gyfora commented on the issue:

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread gyfora
Github user gyfora commented on the issue: https://github.com/apache/flink/pull/2345 But I wonder what would happen in a scenario with a lot of states: Semi async: short local copy time at every snapshot + very fast restore Fully async: no copy time + very slow restore (put

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread gyfora
Github user gyfora commented on the issue: https://github.com/apache/flink/pull/2345 Good thing about the way fully async checkpoints are restored though is that it is very trivial to insert some state adaptor code :) --- If your project is set up for it, you can reply to this email

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413575#comment-15413575 ] ASF GitHub Bot commented on FLINK-4340: --- Github user aljoscha commented on the issue

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread aljoscha
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/2345 Jip, that's also good. 😃 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wi

[jira] [Commented] (FLINK-4326) Flink start-up scripts should optionally start services on the foreground

2016-08-09 Thread Greg Hogan (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413551#comment-15413551 ] Greg Hogan commented on FLINK-4326: --- Using {{exec}} solves the problem of not knowing th

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413563#comment-15413563 ] ASF GitHub Bot commented on FLINK-4340: --- Github user gyfora commented on the issue:

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413536#comment-15413536 ] ASF GitHub Bot commented on FLINK-4340: --- Github user StephanEwen commented on the is

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread StephanEwen
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2345 +1 for this There seems to be an issue with the RocksDB backup engine, to we should probably discourage that mode even in current releases. I would also remove the `HDFSCopyFromL

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413548#comment-15413548 ] ASF GitHub Bot commented on FLINK-4340: --- Github user gyfora commented on the issue:

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread gyfora
Github user gyfora commented on the issue: https://github.com/apache/flink/pull/2345 But you are right it is probably more important to keep the latency down for the running programs, and for that the fully async seems to be strictly better --- If your project is set up for it, you

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413547#comment-15413547 ] ASF GitHub Bot commented on FLINK-4340: --- Github user gyfora commented on the issue:

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread gyfora
Github user gyfora commented on the issue: https://github.com/apache/flink/pull/2345 Some of the benefits we lose on restore. Especially for very large states this can be pretty serious. Maybe this is required for the sharding to some extent but I don't see this as completely

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413542#comment-15413542 ] ASF GitHub Bot commented on FLINK-4340: --- Github user StephanEwen commented on the is

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413537#comment-15413537 ] ASF GitHub Bot commented on FLINK-4340: --- Github user gyfora commented on the issue:

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread StephanEwen
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2345 The "full async" takes more time but runs completely in the background, so performs better in most cases than "semi async". --- If your project is set up for it, you can reply to this email and

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413544#comment-15413544 ] ASF GitHub Bot commented on FLINK-4340: --- Github user StephanEwen commented on the is

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread StephanEwen
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2345 @gyfora You mean if the "full async" is slower than the "semi async"? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your proje

[GitHub] flink issue #2002: Support for bz2 compression in flink-core

2016-08-09 Thread StephanEwen
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2002 I think this looks good. +1 to merge. As a followup, we can upgrade the `commons-compression` version via dependency management (if it is backwards compatible, which apache commons libs

[jira] [Commented] (FLINK-4282) Add Offset Parameter to WindowAssigners

2016-08-09 Thread Aditi Viswanathan (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413519#comment-15413519 ] Aditi Viswanathan commented on FLINK-4282: -- Also when we want to assign time zone

[jira] [Commented] (FLINK-2090) toString of CollectionInputFormat takes long time when the collection is huge

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413521#comment-15413521 ] ASF GitHub Bot commented on FLINK-2090: --- Github user asfgit closed the pull request

[jira] [Closed] (FLINK-2090) toString of CollectionInputFormat takes long time when the collection is huge

2016-08-09 Thread Stephan Ewen (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-2090. --- > toString of CollectionInputFormat takes long time when the collection is huge >

[jira] [Commented] (FLINK-4035) Bump Kafka producer in Kafka sink to Kafka 0.10.0.0

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413518#comment-15413518 ] ASF GitHub Bot commented on FLINK-4035: --- Github user rmetzger commented on the issue

[GitHub] flink issue #2231: [FLINK-4035] Bump Kafka producer in Kafka sink to Kafka 0...

2016-08-09 Thread rmetzger
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2231 Sorry for joining this discussion late. I've been on vacation. I also stumbled across the code duplicates. I'll check out the code from this pull request and see if there's a good way of re-using

[GitHub] flink pull request #2323: [FLINK-2090] toString of CollectionInputFormat tak...

2016-08-09 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2323 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Resolved] (FLINK-2090) toString of CollectionInputFormat takes long time when the collection is huge

2016-08-09 Thread Stephan Ewen (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-2090. - Resolution: Fixed Fix Version/s: 1.2.0 Fixed via b5d58934d7124e0076e588e74485a60e7c1f484b

[jira] [Closed] (FLINK-4212) Lock PID file when starting daemons

2016-08-09 Thread Greg Hogan (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Hogan closed FLINK-4212. - Resolution: Implemented Implemented in 46b427fac9cfceca7839fc93f06ba758101f4fee > Lock PID file when star

[jira] [Commented] (FLINK-4212) Lock PID file when starting daemons

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413516#comment-15413516 ] ASF GitHub Bot commented on FLINK-4212: --- Github user greghogan closed the pull reque

[GitHub] flink pull request #2251: [FLINK-4212] [scripts] Lock PID file when starting...

2016-08-09 Thread greghogan
Github user greghogan closed the pull request at: https://github.com/apache/flink/pull/2251 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is e

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413498#comment-15413498 ] ASF GitHub Bot commented on FLINK-4340: --- GitHub user aljoscha reopened a pull reques

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413497#comment-15413497 ] ASF GitHub Bot commented on FLINK-4340: --- Github user aljoscha closed the pull reques

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413496#comment-15413496 ] ASF GitHub Bot commented on FLINK-4340: --- Github user aljoscha commented on the issue

[GitHub] flink pull request #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint ...

2016-08-09 Thread aljoscha
Github user aljoscha closed the pull request at: https://github.com/apache/flink/pull/2345 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is en

[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread aljoscha
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/2345 Technically, `HDFSCopyFromLocal` and `HDFSCopyToLocal` are now unused. Should we remove them? They might be useful for some stuff in the future. --- If your project is set up for it, you can reply

[GitHub] flink pull request #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint ...

2016-08-09 Thread aljoscha
GitHub user aljoscha reopened a pull request: https://github.com/apache/flink/pull/2345 [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode This was causing to many problems and is also incompatible with the upcoming key-group/sharding changes. R: @StephanEwen for rev

[jira] [Commented] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413493#comment-15413493 ] ASF GitHub Bot commented on FLINK-4340: --- GitHub user aljoscha opened a pull request:

[GitHub] flink pull request #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint ...

2016-08-09 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/2345 [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode This was causing to many problems and is also incompatible with the upcoming key-group/sharding changes. R: @StephanEwen for revie

[GitHub] flink issue #2002: Support for bz2 compression in flink-core

2016-08-09 Thread mtanski
Github user mtanski commented on the issue: https://github.com/apache/flink/pull/2002 Now that 1.1 one is out, is it possible to get this in? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[jira] [Created] (FLINK-4340) Remove RocksDB Semi-Async Checkpoint Mode

2016-08-09 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-4340: --- Summary: Remove RocksDB Semi-Async Checkpoint Mode Key: FLINK-4340 URL: https://issues.apache.org/jira/browse/FLINK-4340 Project: Flink Issue Type: Imp

  1   2   >