Client-Bookie Protocol Enhancements.

As we are moving towards mega store, which can house 10s of millions(even 100s) of ledgers, and 1000s of bookies, there can be a huge overhead on some of the operations we are performing right now. 1. Compaction/GC A deletion is just a metadata operation. Deletes the zk node. (currently it just de

Build failed in Jenkins: bookkeeper-master-git-pullrequest #382

See Changes: [mmerli] BOOKKEEPER-1069: If client uses V2 proto, set the connection to always -- [...truncated 33.22 KB...] Running org.apache.bookkeeper.ut

Build failed in Jenkins: bookkeeper-master-git-pullrequest #381

See Changes: [mmerli] BOOKKEEPER-1068: Expose ByteBuf in LedgerEntry to avoid data copy -- [...truncated 28.54 KB...] Running org.apache.bookkeeper.zookeep

Build failed in Jenkins: bookkeeper-master-git-pullrequest #380

See Changes: [mmerli] BOOKKEEPER-1055: Optimize handling of masterKey in case it is empty [mmerli] Removed unneeded check for all zeroes master key -- [..

[jira] [Commented] (BOOKKEEPER-1066) Introduce GrowableArrayBlockingQueue

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011535#comment-16011535 ] ASF GitHub Bot commented on BOOKKEEPER-1066: Github user merlimat comment

[GitHub] bookkeeper issue #153: BOOKKEEPER-1066: Introduce GrowableArrayBlockingQueue

Github user merlimat commented on the issue: https://github.com/apache/bookkeeper/pull/153 @jvrao I don't have a benchmark for this specific change, though I agree it might make sense to add one. The points here are: * In some places we are using `LinkedBlockingQueue` whi

[GitHub] bookkeeper issue #153: BOOKKEEPER-1066: Introduce GrowableArrayBlockingQueue

Github user jvrao commented on the issue: https://github.com/apache/bookkeeper/pull/153 Interesting and on the paper sounds great. But do we have any perf testing or benchmark? I believe we need to have a benchmark / ref to run and compare any perf improvements. I have seen extremely

[jira] [Commented] (BOOKKEEPER-1066) Introduce GrowableArrayBlockingQueue

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011529#comment-16011529 ] ASF GitHub Bot commented on BOOKKEEPER-1066: Github user jvrao commented

[jira] [Commented] (BOOKKEEPER-1069) If client uses V2 proto, set the connection to always decode V2 messages

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011518#comment-16011518 ] ASF GitHub Bot commented on BOOKKEEPER-1069: GitHub user merlimat opened

[GitHub] bookkeeper pull request #157: BOOKKEEPER-1069: If client uses V2 proto, set ...

GitHub user merlimat opened a pull request: https://github.com/apache/bookkeeper/pull/157 BOOKKEEPER-1069: If client uses V2 proto, set the connection to always decode V2 messages Avoid handling parsing exception for each request and instead adapt to what the client is sending. Y

[jira] [Created] (BOOKKEEPER-1069) If client uses V2 proto, set the connection to always decode V2 messages

Matteo Merli created BOOKKEEPER-1069: Summary: If client uses V2 proto, set the connection to always decode V2 messages Key: BOOKKEEPER-1069 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1069

Build failed in Jenkins: bookkeeper-master-git-pullrequest #379

See Changes: [mmerli] BOOKKEEPER-1068: Expose ByteBuf in LedgerEntry to avoid data copy -- [...truncated 26.26 KB...] [INFO] Copying 3 resources [INFO] Cop

[GitHub] bookkeeper pull request #116: BOOKKEEPER-1008 Move to Netty4.1

Github user kishorekasi closed the pull request at: https://github.com/apache/bookkeeper/pull/116 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featur

[jira] [Commented] (BOOKKEEPER-1008) Move to netty4

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011502#comment-16011502 ] ASF GitHub Bot commented on BOOKKEEPER-1008: Github user kishorekasi clos

[jira] [Commented] (BOOKKEEPER-1055) Optimize handling of masterKey in case it is empty

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011470#comment-16011470 ] ASF GitHub Bot commented on BOOKKEEPER-1055: GitHub user merlimat opened

[GitHub] bookkeeper pull request #156: BOOKKEEPER-1055: Optimize handling of masterKe...

GitHub user merlimat opened a pull request: https://github.com/apache/bookkeeper/pull/156 BOOKKEEPER-1055: Optimize handling of masterKey in case it is empty On each request client and bookies are exchanging the ledger masterKey, which is a 20 bytes MAC digest of the ledger password

[GitHub] bookkeeper pull request #155: Expose ByteBuf in LedgerEntry to avoid data co...

GitHub user merlimat opened a pull request: https://github.com/apache/bookkeeper/pull/155 Expose ByteBuf in LedgerEntry to avoid data copy To avoid copying the entries payloads when writing/reading on a ledger and having to allocate a lot of `byte[]` on the JVM heap, we need to acce

[jira] [Created] (BOOKKEEPER-1068) Expose ByteBuf in LedgerEntry to avoid data copy

Matteo Merli created BOOKKEEPER-1068: Summary: Expose ByteBuf in LedgerEntry to avoid data copy Key: BOOKKEEPER-1068 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1068 Project: Bookkeeper

[jira] [Commented] (BOOKKEEPER-1048) Use ByteBuf in LedgerStorageInterface

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011441#comment-16011441 ] ASF GitHub Bot commented on BOOKKEEPER-1048: Github user asfgit closed th

[jira] [Resolved] (BOOKKEEPER-1048) Use ByteBuf in LedgerStorageInterface

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Merli resolved BOOKKEEPER-1048. -- Resolution: Fixed Issue resolved by merging pull request 139 [https://g

[GitHub] bookkeeper pull request #139: BOOKKEEPER-1048: Use ByteBuf in LedgerStorage ...

Github user asfgit closed the pull request at: https://github.com/apache/bookkeeper/pull/139 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (BOOKKEEPER-1067) Add Prometheus stats provider

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011358#comment-16011358 ] ASF GitHub Bot commented on BOOKKEEPER-1067: Github user jvrao commented

[GitHub] bookkeeper issue #154: BOOKKEEPER-1067: Add Prometheus stats provider

Github user jvrao commented on the issue: https://github.com/apache/bookkeeper/pull/154 Awesome!! @kishorekasi --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[jira] [Resolved] (BOOKKEEPER-887) Allow to use multiple bookie journals

[ https://issues.apache.org/jira/browse/BOOKKEEPER-887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Merli resolved BOOKKEEPER-887. - Resolution: Duplicate > Allow to use multiple bookie journals > --

[jira] [Resolved] (BOOKKEEPER-1047) Add missing error code in ZK setData return path

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Merli resolved BOOKKEEPER-1047. -- Resolution: Fixed > Add missing error code in ZK setData return path >

[jira] [Resolved] (BOOKKEEPER-1063) Use executure.execute() instead of submit() to avoid creation of unused FutureTask

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Merli resolved BOOKKEEPER-1063. -- Resolution: Fixed > Use executure.execute() instead of submit() to avoid creation o

Build failed in Jenkins: bookkeeper-master-git-pullrequest #377

See Changes: [mmerli] BOOKKEEPER-1048: Use ByteBuf in LedgerStorage interface -- [...truncated 28.33 KB...] Running org.apache.bookkeeper.conf.SystemProper

[jira] [Commented] (BOOKKEEPER-588) SSL support

[ https://issues.apache.org/jira/browse/BOOKKEEPER-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011235#comment-16011235 ] ASF GitHub Bot commented on BOOKKEEPER-588: --- Github user sijie commented on

[GitHub] bookkeeper issue #97: BOOKKEEPER-588 SSL Support

Github user sijie commented on the issue: https://github.com/apache/bookkeeper/pull/97 @kishorekasi @jvrao ^^ any thoughts? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enable

[jira] [Commented] (BOOKKEEPER-588) SSL support

[ https://issues.apache.org/jira/browse/BOOKKEEPER-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011233#comment-16011233 ] ASF GitHub Bot commented on BOOKKEEPER-588: --- Github user eolivelli commented

[GitHub] bookkeeper issue #97: BOOKKEEPER-588 SSL Support

Github user eolivelli commented on the issue: https://github.com/apache/bookkeeper/pull/97 @sijie I can rebase, but @kishorekasi told he would have picked this issue up in order to implement more security features. I am ok for rebasing, getting this merged and then let Kishore or @

Re: Netty 4

Awesome!! On Mon, May 15, 2017 at 11:11 AM, Enrico Olivelli wrote: > Great! > I will update public snapshots asap > > Enrico > > Il lun 15 mag 2017, 19:54 Sijie Guo ha scritto: > > > I merged BOOKKEEPER-1008. It lands on master now! It is a great milestone > > for 4.5.0. > > > > Thank you every

[jira] [Commented] (BOOKKEEPER-1028) inc/excl opts listunderreplicate

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011223#comment-16011223 ] ASF GitHub Bot commented on BOOKKEEPER-1028: Github user sijie commented

[GitHub] bookkeeper issue #127: BOOKKEEPER-1028 and BOOKKEEPER-1029

Github user sijie commented on the issue: https://github.com/apache/bookkeeper/pull/127 @reddycharan do you mind addressing the findbugs error? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[jira] [Commented] (BOOKKEEPER-753) Bookie should run garbage collection before startup when all directories became full

[ https://issues.apache.org/jira/browse/BOOKKEEPER-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011218#comment-16011218 ] ASF GitHub Bot commented on BOOKKEEPER-753: --- Github user sijie commented on

[GitHub] bookkeeper issue #81: BOOKKEEPER-753: Bookie should run garbage collection b...

Github user sijie commented on the issue: https://github.com/apache/bookkeeper/pull/81 @jvrao even you enforce force compaction or allow entry log creation, it doesn't really address the issue that this pull request addresses. the change is useful when all your disks are full (full he

[jira] [Commented] (BOOKKEEPER-1008) Move to netty4

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011204#comment-16011204 ] ASF GitHub Bot commented on BOOKKEEPER-1008: Github user sijie commented

[GitHub] bookkeeper issue #116: BOOKKEEPER-1008 Move to Netty4.1

Github user sijie commented on the issue: https://github.com/apache/bookkeeper/pull/116 @kishorekasi I merged #138 . do you mind closing this pull request? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[jira] [Resolved] (BOOKKEEPER-1061) BookieWatcher should not do ZK blocking operations from ZK async callback thread

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sijie Guo resolved BOOKKEEPER-1061. --- Resolution: Fixed Issue resolved by merging pull request 149 [https://github.

[jira] [Commented] (BOOKKEEPER-1061) BookieWatcher should not do ZK blocking operations from ZK async callback thread

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011194#comment-16011194 ] ASF GitHub Bot commented on BOOKKEEPER-1061: Github user asfgit closed th

[GitHub] bookkeeper pull request #149: BOOKKEEPER-1061: BookieWatcher should not do Z...

Github user asfgit closed the pull request at: https://github.com/apache/bookkeeper/pull/149 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (BOOKKEEPER-1067) Add Prometheus stats provider

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sijie Guo resolved BOOKKEEPER-1067. --- Resolution: Fixed Issue resolved by merging pull request 154 [https://github.

[jira] [Commented] (BOOKKEEPER-1067) Add Prometheus stats provider

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011175#comment-16011175 ] ASF GitHub Bot commented on BOOKKEEPER-1067: Github user asfgit closed th

[GitHub] bookkeeper pull request #154: BOOKKEEPER-1067: Add Prometheus stats provider

Github user asfgit closed the pull request at: https://github.com/apache/bookkeeper/pull/154 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (BOOKKEEPER-588) SSL support

[ https://issues.apache.org/jira/browse/BOOKKEEPER-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011168#comment-16011168 ] ASF GitHub Bot commented on BOOKKEEPER-588: --- Github user sijie commented on

[GitHub] bookkeeper issue #97: BOOKKEEPER-588 SSL Support

Github user sijie commented on the issue: https://github.com/apache/bookkeeper/pull/97 @eolivelli since netty 4 is in, do you mind rebasing this pull request? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

Re: Netty 4

Great! I will update public snapshots asap Enrico Il lun 15 mag 2017, 19:54 Sijie Guo ha scritto: > I merged BOOKKEEPER-1008. It lands on master now! It is a great milestone > for 4.5.0. > > Thank you everyone for making this happen! > > - Sijie > -- -- Enrico Olivelli

[jira] [Assigned] (BOOKKEEPER-1008) Move to netty4

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sijie Guo reassigned BOOKKEEPER-1008: - Assignee: Matteo Merli (was: Robert Joseph Evans) > Move to netty4 > --

[jira] [Resolved] (BOOKKEEPER-1008) Move to netty4

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sijie Guo resolved BOOKKEEPER-1008. --- Resolution: Fixed Fix Version/s: 4.5.0 Issue resolved by merging pull request 138

Netty 4

I merged BOOKKEEPER-1008. It lands on master now! It is a great milestone for 4.5.0. Thank you everyone for making this happen! - Sijie

[jira] [Commented] (BOOKKEEPER-1008) Move to netty4

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16010988#comment-16010988 ] ASF GitHub Bot commented on BOOKKEEPER-1008: Github user asfgit closed th

[GitHub] bookkeeper pull request #138: BOOKKEEPER-1008: Netty 4.1

Github user asfgit closed the pull request at: https://github.com/apache/bookkeeper/pull/138 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (BOOKKEEPER-1010) Bump up Guava version to 20.0

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16010972#comment-16010972 ] ASF GitHub Bot commented on BOOKKEEPER-1010: Github user sijie commented

[GitHub] bookkeeper issue #117: BOOKKEEPER-1010 Moving Guava to 20.0

Github user sijie commented on the issue: https://github.com/apache/bookkeeper/pull/117 @arvindkandhare any updates? Do you want to merge @eolivelli 's patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[jira] [Commented] (BOOKKEEPER-1058) Ignore already deleted ledger on replication audit

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16010954#comment-16010954 ] ASF GitHub Bot commented on BOOKKEEPER-1058: Github user asfgit closed th

[jira] [Resolved] (BOOKKEEPER-1058) Ignore already deleted ledger on replication audit

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sijie Guo resolved BOOKKEEPER-1058. --- Resolution: Fixed Issue resolved by merging pull request 146 [https://github.

[GitHub] bookkeeper pull request #146: BOOKKEEPER-1058: Ignore already deleted ledger...

Github user asfgit closed the pull request at: https://github.com/apache/bookkeeper/pull/146 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] bookkeeper pull request #154: BOOKKEEPER-1067: Add Prometheus stats provider

GitHub user merlimat opened a pull request: https://github.com/apache/bookkeeper/pull/154 BOOKKEEPER-1067: Add Prometheus stats provider Prometheus (https://prometheus.io) is a metrics collection system, similar but much more flexible than graphite. It would be good to expo

[jira] [Commented] (BOOKKEEPER-1067) Add Prometheus stats provider

[ https://issues.apache.org/jira/browse/BOOKKEEPER-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16010947#comment-16010947 ] ASF GitHub Bot commented on BOOKKEEPER-1067: GitHub user merlimat opened

[jira] [Created] (BOOKKEEPER-1067) Add Prometheus stats provider

Matteo Merli created BOOKKEEPER-1067: Summary: Add Prometheus stats provider Key: BOOKKEEPER-1067 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1067 Project: Bookkeeper Issue Type

Jenkins build is back to normal : bookkeeper-master #1749

See