I've worked on the TokuDB storage engine for quite a while now. I have had many experiences over the years, so I guess it's hard to know where to begin. I guess I will start small, and if the conversation evolves, I can contribute more thoughts. I think the current API is really good, as evidenced by the fact that many storage engines have used it to plug into MySQL. The two areas that I see we can really benefit from are the following: - documentation of the API, discussing clearly what the contracts for the functions are. There are still several functions I am not fully clear on, but am glad that they "just work". This includes documentation on the locking mechanisms. For instance, clear documentation on what external_lock, store_lock, and other related functions do would be great. - general API cleanup. In too many places, the functions just assume things, such as the buffer passed into write_row is the same as table->record[0]. In other instances, the well defined way to answer certain questions is not obvious. For instance: - If I want information on the fields, do I use the table object or table_share? - There seem to be several ways to determine if a field is NULL - There is redundant information in the KEY* structure and the fields that make up the KEY. I recall when working on MySQL 5.1, this information was inconsistent and it led to bugs, such as MySQL bug 37292 - I am sure there are plenty of other instances of this that I can find. - Transactions. The API seems to not be designed with transactions taken into account. Transactions are started in some cases of external_lock, or in start_stmt. Transactions are committed either in the handlerton's commit function or in some cases in external_lock. A clear API that takes transactions into consideration would be quite helpful.
Those are my early thoughts. I am happy to share more if people are interested. -Zardosht On Tue, Aug 13, 2013 at 6:41 PM, MARK CALLAGHAN <mdcal...@gmail.com> wrote: > The storage engine API has never been my favorite part of MySQL. I haven't > written a storage engine but I used to maintain a custom storage engine and > migrated that from 4.0 to 5.0 so I had some experience with the API. > > Is there a different API that would make it easier for innovation to come to > MySQL/MariaDB? For example there is a lot going on in the write-optimized > database space. Until recently we had nothing for MySQL open-source. Now we > have open source TokuDB (great news). I haven't read their source but I have > been told they use something that was derived from the BerkeleyDB API. > > Would the BerkeleyDB API make it easier to get new storage engines into > MySQL? For example, we don't have an engine optimized for in-memory > workloads. And while InnoDB will have great cache hit rates when all data is > in-memory. It might write much more data to disk than required. > > Do we need a better API or is the current code good enough? I hope to hear > from people who have done serious storage engine work. > > -- > Mark Callaghan > mdcal...@gmail.com > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-developers > Post to : maria-developers@lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-developers > More help : https://help.launchpad.net/ListHelp > _______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
