On Tue, Nov 10, 2020 at 2:09 AM Ivan Kelly <iv...@apache.org> wrote: > Hi folks, > > It's been about a year since Streamlio joined Splunk and since then > we've had a bit of forking with our BK branch. > It has gotten to a stage where it's starting to be a problem for us, > so we'd like to start to get things back in sync. > > +1 Very cool! Looking forward to them!
> There are a couple of big chunks of work to come back. > We've added a data integrity checker that replaces a lot of the > functionality of autorecovery and allows us to run without a journal. > We refactored the bookie to allow dependency injection. > We've rewritten the entry logger to use direct I/O (allowing 2GBps > writes per bookie). > One other thing we've done is to change the build system to use gradle. > The major driver for this was that maven is just slow, even before you > start running tests. > "mvn clean package -DskipTests" takes 4m30 on my laptop. "./gradlew > clean jar" takes 40s. > Subsequent builds on gradle are much much faster, as it does > incremental building. > Incremental building exists in maven, but it doesn't work. > Gradle also handle multimodule projects better. If I make a change in > bookkeeper-common, > "./gradlew :bookkeeper-server:test" will pick up the change without > having to explicitly > "mvn install" the bookkeeper-common. In my opinion it's just a much > nicer build system > to work with. Even the poms it generates are better as they avoid > dependency pollution. > > What are peoples opinions on moving BookKeeper to gradle (assuming > I/splunk do the legwork)? > If people are open to it, I'll submit a BP. > > Another thing that BK (and the whole ecosystem) is missing is > structured logging. > We also plan to add structured logging to BK in soon. This is a major > motivator for converging the branches, > as it touches a lot of places. > There might be slight overlap with BP-40 which I have in the works for Audit logging for BKShell and Bkctl. We also have some more work in this area that we're looking to contribute back soon as well. Would like to have a discussion around this when we come to it. Nonetheless, I'm very interested in this. > Anyhow, any feedback appreciated. > > -Ivan > -- Anup Ghatage www.ghatage.com
