Hi Michael, Yes, testing is always a problem, and that is exactly why we would like to release our code as a plugin, outside of the main source tree, so that the project won't need to test the hardware-dependent code. The pluggable CommitLog will allow this approach.
Actually, we have already released another plugin for CAPI-Flash-based RowCache, which takes advantage of the pluggable RowCache mechanism. https://github.com/ppc64le/capi-rowcache We would just like to repeat this approach in CommitLog. Thanks, Rei Odaira 2017-11-01 16:30 GMT-05:00 Michael Kjellman <mkjell...@internalcircle.com>: > Rei: > > One thing that comes up when these type of conversations occur is how the > project can test hardware dependent code. In the case of the PPC64 stuff, > hardware actually got donated to the ASF so Jenkins runs could be done to > check that things work. Any thoughts on this aspect? Might be a bit > pre-mature, but I thought I'd at least mention it... On the flip side: if > CommitLog becomes pluggable enough, shipping an implementation compatible > with the hardware out of tree might also be viable too. > > best, > kjellman > > > On Nov 1, 2017, at 2:25 PM, 大平怜 <rei.oda...@gmail.com> wrote: > > > > Hi Ariel, > > > > CommitLogSegment assumes commit log files stored on a regular file > system. > > Our CAPI Flash system bypasses OS and directly accesses flash, > > so we cannot use the current framework of CommitLogSegment as it is. > > Intel's SPDK also bypasses a file system, so we think this kind of > > requirement > > is not uncommon. > > > > It would not be easy to reuse AbstractCommitLogSegmentManager, either, > > because the archiving and synchronization logics have to be decoupled. > > It would require major rework, and we don't think we should affect > > the existing implementation so much. > > > > We do not change any existing format of CommitLog. Our plugin will use > > its own format, as it must manage commit logs on the 4KB-block-oriented > > address spaces of flash devices. > > > > > > Regards, > > Rei Odaira > > > > > > 2017-10-31 15:38 GMT-05:00 Ariel Weisberg <ar...@weisberg.ws>: > > > >> Hi, > >> > >> There are pluggable elements to the commit log such as those used to > >> support mmap or compressed. > >> > >> Can you describe at a high level what a new implementation would look > >> like and why it can't be a mode of the existing implementation? > >> > >> You are not proposing changing the format correct? > >> > >> Regards, > >> Ariel > >> > >> On Tue, Oct 31, 2017, at 04:09 PM, 大平怜 wrote: > >>> Hello, > >>> > >>> We are developing a Cassandra plugin to store CommitLog on our > >>> low-latency > >>> Flash device (CAPI-Flash). To do that, the original CommitLog > interface > >>> must be changed to allow plugins. Anyone has any thoughts about it? > We > >>> have our codebase ready, but we think we should start with high-level > >>> discussion. > >>> > >>> The runtime overhead will be minimal. The only overhead will be > changing > >>> method invocations to CommitLog#add(), CommitLog#getCurrentPosition(), > >>> etc. > >>> into interface invocations. > >>> > >>> Synching to CommitLog is one of the performance bottlenecks in > Cassandra > >>> especially with batch commit. I think the pluggable CommitLog will > allow > >>> other interesting alternatives, such as one using SPDK. Appreciate any > >>> comments. > >>> > >>> > >>> Regards, > >>> Rei Odaira > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > >> > >> > >
