BP-46: Pluggable journal file channel Motivation:
Currently, the Journal file channel implementation is based on the basic File API which can meet the vast majority of application scenarios. Persistent memory is a new generation technology which allows us to get lower latency and higher throughput and here is a blog https://www.verizonmedia.com/technology/blog/apache-pulsar-overview <https://www.verizonmedia.com/technology/blog/apache-pulsar-overview> comparing the performance for the different devices. If you want to know more about the pmem you can refer to https://software.intel.com/content/www/us/en/develop/articles/introduction-to-programming-with-persistent-memory-from-intel.html <https://software.intel.com/content/www/us/en/develop/articles/introduction-to-programming-with-persistent-memory-from-intel.html> and https://docs.pmem.io/persistent-memory/getting-started-guide/introduction <https://docs.pmem.io/persistent-memory/getting-started-guide/introduction>. The pmem providers also provide the SDK for the pmem device which can achieve lower latency than using the File API such as https://pmem.io/pmdk/ <https://pmem.io/pmdk/>. And the pmdk also provide libpmemlog lib which can easily to implement the Journal file. So the proposal is to allow users to use diversified Journal File implementations through plug-ins. And here is the in-progress PR https://github.com/apache/bookkeeper/pull/2742 Proposed Change: API Changes: We need to abstract the `FileChannel` in the JournalChannel https://github.com/apache/bookkeeper/blob/d03b046c2526a3b89305da6460b601656ecd0700/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/JournalChannel.java#L50 <https://github.com/apache/bookkeeper/blob/d03b046c2526a3b89305da6460b601656ecd0700/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/JournalChannel.java#L50> This proposal we will introduce a new configuration `journalChannelProvider` to choose which journal channel we are used for save journal files. By default it will keep the current way to write journal files. ``` protected static final String JOURNAL_CHANNEL_PROVIDER = "journalChannelProvider"; ``` ``` /** * A FileChannel for the JournalChannel read and write, we can use this interface to extend the FileChannel * which we use in the JournalChannel. */ interface JournalFileChannel { /** * An interface for get the FileChannel from the provider. * @return */ FileChannel getFileChannel() throws FileNotFoundException; /** * Check the given file if exists. * * @param file * @return */ boolean fileExists(File file); /** * Get the file descriptor of the opened file. * * @return * @throws IOException */ FileDescriptor getFD() throws IOException; } ``` ``` /** * An interface of the FileChannelProvider. */ public interface FileChannelProvider { static FileChannelProvider newProvider(String providerClassName) throws IOException { try { Class<?> providerClass = Class.forName(providerClassName); Object obj = providerClass.newInstance(); return (FileChannelProvider) obj; } catch (Exception e) { throw new IOException(e); } } /** * Get the journalFileChannel with the given file and configuration. * * @param file * @param configuration * @return * @throws IOException */ JournalFileChannel open(File file, ServerConfiguration configuration) throws IOException; } ``` Configuration Changes: Users are expected to define `journalChannelProvider` in `conf/bk_server.conf`. journalChannelProvider=org.apache.bookkeeper.bookie.DefaultFileChannelProvider BookieAdmin: Because we will support different way to read journal, we also need to change the `ReadJournal` command to make it can work with the new journal provider. https://github.com/apache/bookkeeper/blob/d03b046c2526a3b89305da6460b601656ecd0700/bookkeeper-server/src/main/java/org/apache/bookkeeper/tools/cli/commands/bookie/ReadJournalCommand.java#L127 Migration Plan and Compatibility: Because we abstract the write process, we can make the current implementation as the default provider. That will not introduce any compatibility issue.
