There is now a third update available for 2.9.0: https://github.com/basho/riak/tree/riak-2.9.0p3.
Again, the fixes are related to memory management in leveled, and specifically related to references to sub-binaries. This main issue was related to a lazy-load of file metadata which occurs following a riak restart, plus also an issue with managing memory use during journal compaction following many days of repeated compactions. Release notes ( https://github.com/basho/riak/blob/riak-2.9.0p3/RELEASE-NOTES.md) contain some more details and links. I would recommend updating from any previous release of 2.9.0 if you have enabled either the leveled backend, or Tictac AAE. Updated packages are available (thanks to Nick Adams at TI Tokyo) - https://files.tiot.jp/riak/kv/2.9/2.9.0p3/. Thanks again to the testing team at the NHS Spine project, Aaron Gibbon (BJSS) and Ramen Sen, for their continued efforts to stress Riak 2.9.0 in different environments and scenarios and uncover these problems. On a more general note, there are ongoing tests of a pre-release of 2.9.1 that have been happening over the past month, so we continue to make progress towards that release. No major issues have been highlighted so far. Work on Riak 3.0 has slowed over the summer, but I hope we can pick up the pace again and make further progress in September. Regards Martin On Fri, 28 Jun 2019 at 09:34, Martin Sumner <martin.sum...@adaptip.co.uk> wrote: > There is now a second update available for 2.9.0: > https://github.com/basho/riak/tree/riak-2.9.0p2. > > This patch, like the patch before, resolves a memory management issue in > leveled, which this time could be triggered by sending many large objects > in a short period of time. The underlying problem is described a bit > further here https://github.com/martinsumner/leveled/issues/285, and is > resolved by leveled working more sympathetically with the beam binary > memory management. > > Switching to the patched version is not urgent unless you are using the > leveled backend, and may send a large number of large objects in a burst. > > Updated packages are available (thanks to Nick Adams at TI Tokyo) - > https://files.tiot.jp/riak/kv/2.9/2.9.0p2/ > > Thanks again to the testing team at the NHS Spine project, Aaron Gibbon > (BJSS) and Ramen Sen, who discovered the problem. The issue was discovered > in a handoff scenario where there were a tens of thousands of 2MB objects > stored in a portion of the keyspace at the end of the handoff - which led > to memory issues until either more PUTs were received (to force a persist > to disk) or a restart occurred.. > > Regards > > > On Sat, 25 May 2019 at 09:35, Martin Sumner <martin.sum...@adaptip.co.uk> > wrote: > >> Unfortunately, Riak 2.9.0 was released with an issue whereby a race >> condition in heavy-PUT scenarios (e.g. handoffs), could cause a leak of >> file descriptors. >> >> The issue is described here - >> https://github.com/basho/riak_kv/issues/1699, and the underlying issue >> here - https://github.com/martinsumner/leveled/issues/278. >> >> There is a new patched version of the release available (2.9.0p1) at >> https://github.com/basho/riak/tree/riak-2.9.0p1. This should be used in >> preference to the original release of 2.9.0. >> >> Updated packages are available (thanks to Nick Adams at TI Tokyo) - >> https://files.tiot.jp/riak/kv/2.9/2.9.0p1/ >> >> Thanks also to the testing team at the NHS Spine project, Aaron Gibbon >> (BJSS) and Ramen Sen, who discovered the problem. >> >> Regards >> >> Martin >> >> >> >> >>
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
