Rei: One thing that comes up when these type of conversations occur is how the project can test hardware dependent code. In the case of the PPC64 stuff, hardware actually got donated to the ASF so Jenkins runs could be done to check that things work. Any thoughts on this aspect? Might be a bit pre-mature, but I thought I'd at least mention it... On the flip side: if CommitLog becomes pluggable enough, shipping an implementation compatible with the hardware out of tree might also be viable too.
best, kjellman > On Nov 1, 2017, at 2:25 PM, 大平怜 <rei.oda...@gmail.com> wrote: > > Hi Ariel, > > CommitLogSegment assumes commit log files stored on a regular file system. > Our CAPI Flash system bypasses OS and directly accesses flash, > so we cannot use the current framework of CommitLogSegment as it is. > Intel's SPDK also bypasses a file system, so we think this kind of > requirement > is not uncommon. > > It would not be easy to reuse AbstractCommitLogSegmentManager, either, > because the archiving and synchronization logics have to be decoupled. > It would require major rework, and we don't think we should affect > the existing implementation so much. > > We do not change any existing format of CommitLog. Our plugin will use > its own format, as it must manage commit logs on the 4KB-block-oriented > address spaces of flash devices. > > > Regards, > Rei Odaira > > > 2017-10-31 15:38 GMT-05:00 Ariel Weisberg <ar...@weisberg.ws>: > >> Hi, >> >> There are pluggable elements to the commit log such as those used to >> support mmap or compressed. >> >> Can you describe at a high level what a new implementation would look >> like and why it can't be a mode of the existing implementation? >> >> You are not proposing changing the format correct? >> >> Regards, >> Ariel >> >> On Tue, Oct 31, 2017, at 04:09 PM, 大平怜 wrote: >>> Hello, >>> >>> We are developing a Cassandra plugin to store CommitLog on our >>> low-latency >>> Flash device (CAPI-Flash). To do that, the original CommitLog interface >>> must be changed to allow plugins. Anyone has any thoughts about it? We >>> have our codebase ready, but we think we should start with high-level >>> discussion. >>> >>> The runtime overhead will be minimal. The only overhead will be changing >>> method invocations to CommitLog#add(), CommitLog#getCurrentPosition(), >>> etc. >>> into interface invocations. >>> >>> Synching to CommitLog is one of the performance bottlenecks in Cassandra >>> especially with batch commit. I think the pluggable CommitLog will allow >>> other interesting alternatives, such as one using SPDK. Appreciate any >>> comments. >>> >>> >>> Regards, >>> Rei Odaira >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: dev-h...@cassandra.apache.org >> >>
