Re: Leveled and Anti-Entropy

[Heinz N. Gies]

Have you taken a look at the changes here 
https://github.com/Kyorai/riak_core/pull/24 


It pulls the AAE work for riak_kv into riak_core.


> On 21. Jul 2017, at 16:06, Martin Sumner  wrote:
> 
> 
> I've added some anti-entropy features to Leveled (the pure-Erlang KV store 
> designed as a Riak backend).  These features are in-part an experiment in how 
> to approach both anti-entropy and full-sync multi-data centre replication in 
> the future.
> 
> There's a long write-up, including some history of AAE in Riak:
> 
> https://github.com/martinsumner/leveled/blob/master/docs/ANTI_ENTROPY.md 
> 
> 
> In summary, Riak's current AAE is based on cryptographically strong Merkle 
> trees, and this experiment is based on removing that security strength, as it 
> isn't relevant to the context in which is used.  Instead Leveled now has 
> Merkle Trees that can be merged and also can be built incrementally (i.e. be 
> built key by key even when the keys are not in segment order).
> 
> Using these new trees (coined TicTac trees to fit into Leveled's terrible 
> naming convention), we can build AAE trees in folds incrementally and hence 
> at a lower cost, but also merge trees across independent stores.  In the 
> future trees can be built from folds using Riak coverage queries, across 
> either indexes or objects in the store - and compared between different 
> database clusters even where those clusters are partitioned differently e.g. 
> different ring-sizes.
> 
> The expectation is that there will be more flexibility of choice in what we 
> can decide to compare at run time - not just are the objects consistent, are 
> the indexes consistent.  Also split from partition constraints there will be 
> improved flexibility in what we can decide to compare between - e.g. make it 
> easier to compare with a different database.
> 
> Coupled with this there's a demonstration of using temporary indexes in 
> Leveled, index entries that auto-expire at a TTL, and we've shown how this 
> can be used with tree-creating folds to compare recent changes between stores 
> at a lower cost than comparing the whole database state: with the added 
> advantage that the long-term footprint of the database is not extended by 
> maintaining a separate copy of all the keys and hashes.
> 
> Concurrently to this, we now have some other work ongoing in the space of 
> replication and anti-entropy:
> 
> - @russelldb is continuing to test and improve his open source real-time 
> replication solution (rabl) which uses RabbitMQ.  He's hoping to be able to 
> talk further on progress with this by the end of August.
> - I'm working on implementing in riak_core a core_node_worker_pool, which is 
> intended to compliment the core_vnode_worker_pool but allow for coverage 
> queries where snapshots are taken on a covering set of vnodes, but folds are 
> then scheduled to run one-at-a-time on each node.  This can then be used to 
> regulate the impact of anti-entropy folds.
> 
> Our current target is to have a release candidate of open-source replication 
> (both real-time and full-sync) by the end of September.  This will initially 
> be focused only on replication between two Riak clusters.
> 
> Regards
> 
> Martin (@masleeds)
> 
> P.S. Hopefully next Friday we should also be able to report back on the 
> improvements and test enhancements that followed up the work on riak_core 
> claim.
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



signature.asc
Description: Message signed with OpenPGP
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Lets come together - how riak(_core) should be developed in the future?

[Heinz N. Gies]

Heya everyone,
I figured I’ll pick up the initiative here and see if we all can come together 
before fragmentation hits. Annette mentioned this today and I think she’s right.

There are many people that want to see risk(_core) continue to evolve and 
improve, out of the top of my head there are ES - who picked up support for 
riak, NHS/UK - which is using it and seem to keep doing so, there is Gordon, 
Annette, Chris, Tristan, Mariano and me who all have an interest to keep things 
going. I’m pretty sure I forgot a good few people (sorry! It doesn’t mean I 
love you any less, I just have a brain like a sieve).

Amongst all of us I’m sure we’ve quite some forks. Tristan, Chris, Mariano and 
me started https://github.com/Kyorai  to consolidate 
the git repos that correspond to the current hex packages as an example, NHS 
runs a fork of core at https://github.com/ramensen/riak_core 
 and I guess ES will have one too by 
now. (Again apologies if I forgot someone).

I am quite sure that all of us have the same goal and probably are interested 
in the same things (keeping riak(_core) strong). So what I’d like to suggest is 
we find a time to sit together and chat about how we pool resources instead of 
trying to cross merge in N different places. Perhaps late next week EU 
afternoon US morning? Please raise a hand if interested, discuss and share 
times that would work for you :)

Cheers,
Heinz


signature.asc
Description: Message signed with OpenPGP
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Lets come together - how riak(_core) should be developed in the future?

[Heinz N. Gies]

Gokhan has made a doodle thingy (I didn’t even knew that existed) to track 
availability:

https://beta.doodle.com/poll/ywf778xanbegqbyq 
<https://beta.doodle.com/poll/ywf778xanbegqbyq>

Lets see when we can do it and then find a medium.


> On 18. Aug 2017, at 05:36, DeadZen  wrote:
> 
> I'm down for a chat on how to finally get our collective stuff together. 
> Maybe a google hangout?
> 
> On Thu, Aug 17, 2017 at 2:14 PM Gokhan Boranalp  <mailto:gok...@zetaops.io>> wrote:
> Hi all,
> 
> I have created a Slack channel. Here is the invitation link[1]
> I think it is better to have a quick talk beforehand in Slack to
> organize big meeting with a rest of Riak funs/lovers.
> We are ready to provide some resources and i believe lots of folks
> would like to do the same.
> It would be very efficient to organize, as fast as possible.
> For anyone missed, there is an alternate mail group[2] if this one
> ripped of suddenly.
> 
> 
> [1] 
> https://join.slack.com/t/postriak/shared_invite/MjI4MDU2NTI0NDM1LTE1MDI5OTMxMDAtMTQzNTM3MzAwZA
>  
> <https://join.slack.com/t/postriak/shared_invite/MjI4MDU2NTI0NDM1LTE1MDI5OTMxMDAtMTQzNTM3MzAwZA>
> [2] https://groups.google.com/forum/#!forum/riak-users 
> <https://groups.google.com/forum/#!forum/riak-users>
> 
> On Thu, Aug 17, 2017 at 6:51 PM, Russell Brown  <mailto:russell.br...@icloud.com>> wrote:
> > Great idea Heinz, I think speaking together is the right start.
> >
> > I’m free next week, but a key member of our team is not. I know we’re never 
> > going to get everyone to agree on a time, but if we could do it early the 
> > following week it would suit more. Otherwise, I will attend and pass info 
> > along, so don’t worry. I think bet365 should be involved too. They probably 
> > have the largest team anywhere working on Riak.
> >
> > Maybe a doodle http://doodle.com/en_GB/ <http://doodle.com/en_GB/> would be 
> > a good thing?
> >
> > Russell
> >
> > On 17 Aug 2017, at 15:47, Heinz N. Gies  > <mailto:he...@licenser.net>> wrote:
> >
> >> Heya everyone,
> >> I figured I’ll pick up the initiative here and see if we all can come 
> >> together before fragmentation hits. Annette mentioned this today and I 
> >> think she’s right.
> >>
> >> There are many people that want to see risk(_core) continue to evolve and 
> >> improve, out of the top of my head there are ES - who picked up support 
> >> for riak, NHS/UK - which is using it and seem to keep doing so, there is 
> >> Gordon, Annette, Chris, Tristan, Mariano and me who all have an interest 
> >> to keep things going. I’m pretty sure I forgot a good few people (sorry! 
> >> It doesn’t mean I love you any less, I just have a brain like a sieve).
> >>
> >> Amongst all of us I’m sure we’ve quite some forks. Tristan, Chris, Mariano 
> >> and me started https://github.com/Kyorai <https://github.com/Kyorai> to 
> >> consolidate the git repos that correspond to the current hex packages as 
> >> an example, NHS runs a fork of core at 
> >> https://github.com/ramensen/riak_core 
> >> <https://github.com/ramensen/riak_core> and I guess ES will have one too 
> >> by now. (Again apologies if I forgot someone).
> >>
> >> I am quite sure that all of us have the same goal and probably are 
> >> interested in the same things (keeping riak(_core) strong). So what I’d 
> >> like to suggest is we find a time to sit together and chat about how we 
> >> pool resources instead of trying to cross merge in N different places. 
> >> Perhaps late next week EU afternoon US morning? Please raise a hand if 
> >> interested, discuss and share times that would work for you :)
> >>
> >> Cheers,
> >> Heinz
> >> ___
> >> riak-users mailing list
> >> riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
> >> <http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
> >
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
> > <http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
> 
> 
> 
> --
> Iyi calismalar dilerim.
> -
> Gokhan Boranalp
> +90 555 54

Re: Riak and the demise of Basho

[Heinz N. Gies]

+1 For Apache 2 :)
> On 6. Sep 2017, at 20:27, Jordan West  wrote:
> 
> Really excited to see this news. Thanks Bet365! +1 to apache2.
> 
> Jordan
> 
> --
> Sent from my iPhone
> On September 6, 2017 at 10:46:50 AM, AJAX DoneBy Jack (ajaxd...@gmail.com 
> ) wrote:
> 
>> +2 Apache 2
>> 
>> On Wed, Sep 6, 2017 at 1:45 PM Bryan Hunt > > wrote:
>> +1 Apache 2
>> 
>> 
>> On Wed, 6 Sep 2017, 18:09 Tom Santero > > wrote:
>> +1 Apache 2
>> 
>> On Wed, Sep 6, 2017 at 11:10 AM, Gokhan Boranalp > > wrote:
>> +1 for Apache 2
>> i wish we could use GPL 3 :(
>> 
>> 
>> On Sep 6, 2017 4:55 PM, "Drew Kerrigan" > > wrote:
>> +1 to Apache 2 for MDC and other relevant EE repos.
>> 
>> On Wed, Sep 6, 2017 at 9:02 AM DeadZen > > wrote:
>> Sept 16 is software freedom day, a release that day might be a nice idea.
>> 
>> On Wed, Sep 6, 2017 at 8:37 AM, Christopher Meiklejohn
>> mailto:christopher.meiklej...@gmail.com>> 
>> wrote:
>> > Given that all of the code except for MDC (and maybe JMX) is under
>> > Apache 2, I would assume that those components would follow the
>> > already existing license of the other components.
>> >
>> > Is there a plan to change the original license on the open source
>> > product from Apache 2?
>> >
>> > Thanks,
>> > Christopher
>> >
>> > On Wed, Sep 6, 2017 at 2:13 PM,  > > > wrote:
>> >> Hi,
>> >>
>> >> A quick status update. Today bet365 have signed the agreement to purchase 
>> >> all Basho IP. We expect the agreement to be ratified in the US courts 
>> >> next week. Once cleared, our intention is to open source all code, help 
>> >> rebuild the community and collaboratively take the development of RIAK 
>> >> forward.
>> >>
>> >> In the coming weeks we will hopefully answer the questions people have 
>> >> and will be calling on the community to help forge the initial RIAK 
>> >> Roadmap.
>> >>
>> >> One of the initial questions we have for the community is which OSS 
>> >> license would people like applied to the code? Our thought is the most 
>> >> open and permissive.
>> >>
>> >> Andy.
>> >>
>> >> Andrew Deane
>> >> Systems Development Manager - Middleware
>> >> Hillside (Technology) Limited
>> >> andrew.de...@bet365.com 
>> >> bet365.com 
>> >> -Original Message-
>> >> From: riak-users [mailto:riak-users-boun...@lists.basho.com 
>> >> ] On Behalf Of 
>> >> martin@bet365.com 
>> >> Sent: 24 August 2017 16:18
>> >> To: riak-users@lists.basho.com 
>> >> Cc: Martin Davies
>> >> Subject: FW: Riak and the demise of Basho
>> >>
>> >> Hi
>> >>
>> >> I have been asked to forward the below message from Martin Davies, the 
>> >> CEO of Technology for bet365.
>> >>
>> >> Kind Regards
>> >>
>> >> Martin Cox
>> >> 
>> >> From: Martin Davies
>> >> Sent: 24 August 2017 16:11
>> >> To: Martin Cox
>> >> Subject: Riak and the demise of Basho
>> >>
>> >> Hi,
>> >> I have been wanting to make you aware for a few weeks now that we 
>> >> have reached an agreement, in principle, to buy all of Basho's remaining 
>> >> assets (except support contracts) from the receiver. Up until this 
>> >> afternoon, I was constrained by some confidentiality needs of the 
>> >> receiver and was unable to speak.
>> >>
>> >> We have agreed a price for the assets and are almost at the end of 
>> >> sorting out the legal agreement. Once this is complete, this will then 
>> >> need to be processed through the courts which, I am advised, should take 
>> >> a week or so.
>> >>
>> >> It is our intention to open source all of Basho's products and all of 
>> >> the source code that they have been working on. We'll do this as quickly 
>> >> as we are able to organise it, and we would appreciate some input from 
>> >> the community on how you would like this done.
>> >>
>> >>
>> >> Martin Davies
>> >> Chief Executive Officer - Technology
>> >> Hillside (Technology) Limited
>> >> bet365.com 
>> >>
>> >> This email and any files transmitted with it are confidential and contain 
>> >> information which may be privileged or confidential and are intended 
>> >> solely to be for the use of the individual(s) or entity to which they are 
>> >> addressed. If you are not the intended recipient be aware that any 
>> >> disclosure, copying, distribution or use of the contents of this 
>> >> information is strictly prohibited and may be illegal. If you have 
>> >> received this email in error, please notify us by telephone or email 
>> >> immediately and delete it from your system. Activity and use of our email 
>> >> system is monitored to secure its effective operation and for other 
>> >> lawful business purposes. Comm

Re: A new home for basho/cuttlefish

[Heinz N. Gies]

Hi Luke,
The Kyorai org is a collection of (ex)basho projects that have been forked and 
maintained outside of basho for a while now. While it was a bit of a scattered 
effort in the beginning the demise of basho ended up causing us to collect them 
in a central place.

How things go forward really depends a bit on how things go forward with riak 
and the basho PI, and the community around it.

Cheers,
Heinz

> On 27. Sep 2017, at 20:44, Luke Bakken  wrote:
> 
> Hi again everyone -
> 
> Apologies for not setting a subject in my first email.
> 
> Of course this is Bet365's decision. The purpose of this message
> thread is to let people know that the RabbitMQ team is willing to
> maintain cuttlefish. I also found the following more-up-to-date fork
> and will be looking at the changes there -
> 
> https://github.com/Kyorai/cuttlefish
> 
> Thanks again!
> Luke
> 
> On Wed, Sep 27, 2017 at 11:27 AM, Bryan Hunt
>  wrote:
>> Hi Luke,
>> 
>> It’s for bet365 to make that decision.
>> 
>> For the good of the general population the following would be nice:
>> 
>> a) Upgrade to rebar3
>> b) Upgrade Erlang version support so it compiles under Erlang 20 (if it 
>> doesn’t already)
>> c) Package uploaded to hex.pm
>> 
>> Bryan
>> 
>> 
>>> On 27 Sep 2017, at 18:59, Luke Bakken  wrote:
>>> 
>>> Hello Riak users -
>>> 
>>> The next RabbitMQ release (3.7.0) will use cuttlefish for its
>>> configuration. I'm writing to express interest on behalf of the
>>> RabbitMQ team in taking over maintenance of the project. At the
>>> moment, we forked cuttlefish to the RabbitMQ organization [0] to fix a
>>> couple pressing issues. After that, it would be great if the
>>> repository could be transferred to either its own, new organization or
>>> to the RabbitMQ organization entirely. Basho transferred both
>>> Webmachine and Lager to their own independent organizations, for
>>> instance (github.com/webmachine, github.com/erlang-lager)
>>> 
>>> Once transferred, GitHub will take care of all the necessary
>>> redirections from the Basho organization to cuttlefish's new home.
>>> 
>>> Thanks,
>>> Luke Bakken
>>> 
>>> [0] - https://github.com/rabbitmq/cuttlefish
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



signature.asc
Description: Message signed with OpenPGP
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: A new home for basho/cuttlefish

[Heinz N. Gies]

Martin,
I fully agree. We started the Kyorai org log before it was public knowledge 
that bet365 (or anyone) would buy the IP of basho and done under the assumption 
that the basho repository would either be deleted or end up rotting there. The 
goal was/is to prevent fragmentation and give the community still using the 
repos a home where they can come together.

That said I think it’ll take some time getting everything back together. The 
repos there are currently the mirrors of what is on hex, and due to how hard it 
was to get changes merged to the basho repositories have all diverged to some 
degree or another. So there is work to be done to get it all back in sync. I 
can’t speak for the other people who maintain repos on Kyorai (so I suspect 
they feel the same) but I’d be eager to get it all back together and kill of 
forks.

Cheers,
Heinz

> On 28. Sep 2017, at 09:50, martin@bet365.com wrote:
> 
> Hello
> 
> Whilst bet365 did buy the Basho IP, they have emphasised that the entire 
> catalogue of repositories is to be handed over to the community and that they 
> are not intending to be an overarching authority on decision making, but will 
> steer the project just as much as any other involved parties would. So, from 
> my perspective, it is not a decision for bet365 to take, but for the 
> community as a whole.
> 
> Personally, I don't see a lot of merit in moving the repo away from its 
> current home at this point in time. Once the legal processes have concluded 
> and people have the facility to manage the repository configurations, anyone 
> interested in contributing and maintaining can be added - enabling the 
> RabbitMQ team to work on Cuttlefish without the upheaval of moving 
> repositories around. When the dust has settled, then revisit the question of 
> spinning the project out into its own org.
> 
> What's everyone else's thoughts on this?
> 
> Martin Cox
> Software Developer
> Hillside (Technology) Limited
> m: +44(0) 7585 990614
> t: +44(0) 8456 000 365
> e: martin@bet365.com
> bet365.com
> 
> ________
> From: riak-users [riak-users-boun...@lists.basho.com] on behalf of Heinz N. 
> Gies [he...@licenser.net]
> Sent: 27 September 2017 20:45
> To: Luke Bakken
> Cc: riak-users@lists.basho.com
> Subject: Re: A new home for basho/cuttlefish
> 
> Hi Luke,
> The Kyorai org is a collection of (ex)basho projects that have been forked 
> and maintained outside of basho for a while now. While it was a bit of a 
> scattered effort in the beginning the demise of basho ended up causing us to 
> collect them in a central place.
> 
> How things go forward really depends a bit on how things go forward with riak 
> and the basho PI, and the community around it.
> 
> Cheers,
> Heinz
> 
>> On 27. Sep 2017, at 20:44, Luke Bakken  wrote:
>> 
>> Hi again everyone -
>> 
>> Apologies for not setting a subject in my first email.
>> 
>> Of course this is Bet365's decision. The purpose of this message
>> thread is to let people know that the RabbitMQ team is willing to
>> maintain cuttlefish. I also found the following more-up-to-date fork
>> and will be looking at the changes there -
>> 
>> https://github.com/Kyorai/cuttlefish
>> 
>> Thanks again!
>> Luke
>> 
>> On Wed, Sep 27, 2017 at 11:27 AM, Bryan Hunt
>>  wrote:
>>> Hi Luke,
>>> 
>>> It’s for bet365 to make that decision.
>>> 
>>> For the good of the general population the following would be nice:
>>> 
>>> a) Upgrade to rebar3
>>> b) Upgrade Erlang version support so it compiles under Erlang 20 (if it 
>>> doesn’t already)
>>> c) Package uploaded to hex.pm
>>> 
>>> Bryan
>>> 
>>> 
>>>> On 27 Sep 2017, at 18:59, Luke Bakken  wrote:
>>>> 
>>>> Hello Riak users -
>>>> 
>>>> The next RabbitMQ release (3.7.0) will use cuttlefish for its
>>>> configuration. I'm writing to express interest on behalf of the
>>>> RabbitMQ team in taking over maintenance of the project. At the
>>>> moment, we forked cuttlefish to the RabbitMQ organization [0] to fix a
>>>> couple pressing issues. After that, it would be great if the
>>>> repository could be transferred to either its own, new organization or
>>>> to the RabbitMQ organization entirely. Basho transferred both
>>>> Webmachine and Lager to their own independent organizations, for
>>>> instance (github.com/webmachine, github.com/erlang-lager)
>>>> 
>>>> Once transferred, GitHub will take care of all t