On Tuesday, May 22, 2012 8:26:22 AM UTC-7, Brice Figureau wrote: > > On Mon, 2012-05-21 at 15:39 -0600, Deepak Giridharagopal wrote: > > On Mon, May 21, 2012 at 2:04 PM, Marc Zampetti > > <marc.zampe...@gmail.com> wrote: > > > Why wouldn't a DB-agnostic model be used? > > > > > > The short answer is performance. To effectively > > implement things we've > > got on our roadmap, we need things that (current) > > MySQL doesn't > > support: array types are critical for efficiently > > supporting things > > like parameter values, recursive query support is > > critical for fast > > graph traversal operations, things like INTERSECT are > > handy for query > > generation, and we rely on fast joins (MySQL's nested > > loop joins don't > > always cut it). It's much easier for us to support > > databases with > > these features than those that don't. For fairly > > divergent database > > targets, it becomes really hard to get the performance > > we want while > > simultaneously keeping our codebase manageable. > > > > > > I understand the need to not support everything. Having > > designed a number of systems that require some of the features > > you say you need, I can say with confidence that most of those > > issues can be handled without having an RDBMS that has all > > those advanced features. So I will respectfully disagree that > > you need features you listed. Yes, you may not be able to use > > something like ActiveRecord or Hibernate, and have to > > hand-code your SQL more often, but there are a number of > > techniques that can be used to at least achieve similar > > performance characteristics. I think it is a bit dangerous to > > assume that your user base can easily and quickly switch out > > their RDBMS systems as easy as this announcement seems to > > suggest. I'm happy to be wrong if the overall community thinks > > that is true, but for something that is as core to one's > > infrastructure as Puppet, making such a big change seems > > concerning. > > > > > > We aren't using ActiveRecord or Hibernate, and we are using hand-coded > > SQL where necessary to wring maximum speed out of the underlying data > > store. I'm happy to go into much greater detail about why the features > > I listed are important, but I think that's better suited to puppet-dev > > than puppet-users. We certainly didn't make this decision cavalierly; > > it was made after around a month of benchmarking various solutions > > ranging from traditional databases like PostgreSQL to document stores > > like MongoDB to KV stores such as Riak to graph databases like Neo4J. > > For Puppet's particular type of workload, with Puppet's volume of > > data, with Puppet's required durability and safety requirements...I > > maintain this was the best choice. > > > > While I don't doubt that given a large enough amount of time and > > enough engineers we could get PuppetDB working fast enough on > > arbitrary backing stores (MySQL included), we have limited time and > > resources. From a pragmatic standpoint, we felt that supporting a > > database that was available on all platforms Puppet supports, that > > costs nothing, that has plenty of modules on the Puppet Forge to help > > set it up, that has a great reliability record, that meets our > > performance needs, and that in the worst case has free/cheap hosted > > offerings (such as Heroku) was a reasonable compromise. > > I didn't had a look to the code itself, but is the postgresql code > isolated in its own module? > > If yes, then that'd definitely help if someone (not saying I'm > volunteering :) wants to port the code to MySQL. > > On a side note, that'd be terriffic Deepak if you would start a thread > on the puppet-dev explaining how the postgresql storage has been done to > achieve the speed :) >
I'm working on putting together an in-depth look into the technology inside PuppetDB, as well as everything we've done to make it fast. That should be coming soon. > -- > Brice Figureau > Follow the latest Puppet Community evolutions on www.planetpuppet.org! > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/y9AAD02ZVYwJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
