Re: Salutations
Steve Greenland <[EMAIL PROTECTED]> writes: > I'm not trying to start a religious war, but at the moment, MySQL > doesn't support transactions, which I see as one of the big gains in > moving from (or supporting in addition to) the current file based > system. Yes, one can implement transactions "on-top-of", but then you > might as well stick with the file system. > > Of course, the Right Thing is to be DB independent, but you probably > need to have some sort of "pre-test" to make sure the configured DB > supports all the necessary functionality. Well, my little knowledge tends to make me favor PostreSQL too, but I'd be happy to defer to people with more knowledge, and as you point out, one question we need to address is whether we want to be LCD (Least Common Denominator), or whether or not we are willing, if even for the short term, to commit ourselves to a particular DB in exchange for enhanced functionality... As long as it's a free DB, it's not as huge a deal to exclusively support just that one, since anyone can install it (and we can include it if needed), but it's still not as flexible as allowing the user to use the DB they've already deployed, and as we move into the small-business arena, allowing people to stick data in *their* DB will probably become more critical. Though, even there, as long as we have ways to interface with other DBs (i.e. batch push/pull) that may make most people happy. Anyway, definitely a discussion we need to have. -- Rob Browning <[EMAIL PROTECTED]> PGP=E80E0D04F521A094 532B97F5D64E3930 ___ gnucash-devel mailing list [EMAIL PROTECTED] http://www.gnumatic.com/cgi-bin/mailman/listinfo/gnucash-devel
Re: Salutations
David Merrill <[EMAIL PROTECTED]> writes: > Oh, is that similar to ODBC in that it abstracts the database behind > an API through which you pass SQL? Maybe that answers my questions > above? Presuming my nebulous knowledge of what ODBC is is right, then yes. Gnome DBA is an abstract interface to the SQL databases, and it supports various backends. > What would be the benefit? With relatively small datasets, there > wouldn't be much, if any, speed gains, but only lots of memory > use. It seems like overkill for many people, and since gnucash > already has a working local storage mechanism... Reasons in favor (off the top of my head): - one set of code to maintain. - possibly better memory use (since right now we read in the *entire* file into memory). - we're going to need transactions/logging/journaling/whatever (i.e. safety mechanisms) anyway, I suspect implementing that once with the help of a good DB would be a lot easier than writing a DB implementation *and* a hokey local version. - it's probably easier to solve the multi-user stuff once too, and this may be easier if we only have one storage model. - also it's not clear to me (though I'm no expert) that it has to be a lot slower than what we've been doing. Especially if either MySQL or PostgreSQL get around to optimizing the embedded case. > Plus, it adds another point of failure, along with more complexity, to > the system. When the power is needed, fine, you have to accept the > complexity, but when it's not needed it should be avoided. Depending on how we implement it, I could also see it decreasing the code complexity in some critical areas, but of course it's hard to know what the overall effect would be. As an example, much of the file IO code would go away -- it would be much harder to get that wrong. Also, we're already doing "hand-hacked" stuff internally that a DB would probably be much better at, and as our data-model gets richer (especially as we move to support more consulting/small-business/financial-analysis stuff) this is only going to get uglier as we have to hand-hack GLib indices, and push around GLists of all the data whenever we don't have a DB. I'm not saying you're wrong, I'm just saying that I can see things that a DB *might* make better across the board. > Now, I don't know MySQL or PostgreSQL (yet), so there could be gotchas > I don't know about, but I doubt it. I just read an article online > today where a guy converted his MySQL database to PostgreSQL in a day > with minimal coding required. Sounds good. > Definitely designing the database schema is a good place to start. > That is database-independent work and isn't affected by other design > decisions much if at all. I'll take a look at the XML and see how it > might be mapped into a database. OK, feel free to ask any questions you might have. I suspect that gnc_numeric and kvp_frame are going to be the two ugliest bits. As I've mentioned before, we may have to rethink these while considering an SQL database. > I have a feeling that the multi-user aspects are going to be the real > problem areas. Designing the database won't be hard (I've done many > financial apps in my time!), and writing SQL will be pretty > straightforward (the engine does all the work, really). I agree. > That's the idea, anyway. :-D Thanks much for the help. -- Rob Browning <[EMAIL PROTECTED]> PGP=E80E0D04F521A094 532B97F5D64E3930 ___ gnucash-devel mailing list [EMAIL PROTECTED] http://www.gnumatic.com/cgi-bin/mailman/listinfo/gnucash-devel
Re: Salutations
On 08-Dec-00, 17:37 (CST), Rob Browning <[EMAIL PROTECTED]> wrote: > > Just off the top of my head, here are some issues: > > * Should we prefer PostgreSQL or MySQL? I'm not trying to start a religious war, but at the moment, MySQL doesn't support transactions, which I see as one of the big gains in moving from (or supporting in addition to) the current file based system. Yes, one can implement transactions "on-top-of", but then you might as well stick with the file system. Of course, the Right Thing is to be DB independent, but you probably need to have some sort of "pre-test" to make sure the configured DB supports all the necessary functionality. Steve -- [EMAIL PROTECTED] ___ gnucash-devel mailing list [EMAIL PROTECTED] http://www.gnumatic.com/cgi-bin/mailman/listinfo/gnucash-devel
Re: Performance improvement for xml loads (+comments)
Before I continue, let me state for the record, that dealing
gracefully with arbitrary hardware/os-failures was never a stated goal
of the file format, so I'm going to ignore the points relating to
that.
I also maintain that all of this talk is essentially wasted, so I'm
probably going to quit responding to "kill the XML" arguments soon.
We have a format, it's almost certainly a very temporary format -
presuming that the SQL stuff goes as we expect, so it's not worth the
time it takes to type to hassle about it now. And the XML code (or
some other similar thing) *will* still be used, most likely, as a text
import/export format.
Al Snell <[EMAIL PROTECTED]> writes:
> Gzipping and ungzipping are actually quite memory and CPU intensive
> operations.
You assert this as if it's fact, yet I've seen actual tests of our
code showing that gzipping it has a negligable (less than 5 percent as
I recall) effect on write speed.
I've also seen work here done at UT that shows that for real-cases,
sometimes compression can actually be a performance win, depending on
the algorithm and circumstances -- hence the research into compressed
page VM systems. Disks are still far slower than CPU's and RAM, this
means that compression can increase your effective bandwidth to disk
in some circumstances, so I won't take it as a given that compression
is *always* slower, though it may well be in this case.
> An XDR version would take no time at all; I've got much of it already
> written in a CVS repository. From the .x files I've written, rpcgen
> will create the C type defintiions for GnuCash data structures, and
> C code to load and save them. Easy peasy!
If I had known anything about XDR (what is it?) when we were trying to
figure out what do to with the dead binary format, I would certainly
have considered it. I didn't, and I don't recall anyone else bringing
it up as a trivial solution then either.
But again, if we're going to SQL, and that's the current plan, at
least, then this point is moot. (Though I am still interested in
knowing more about XDR.)
> There's a basic set you can depend on, but it gets hairy above that. I
> design RDBMS schemas for a living... in cross-DBMS
> ("heterogenous") environments (MySQL and PostgreSQL a speciality -
> www.upmystreet.com runs on my schemas, and since the introduction of the
> classified ads systems, it's taking more load than the GnuCash databases
> of a pretty large organisation will ever take :-)
>
> Using an embedded SQL "server" within GnuCash will be a Good Thing. From
> an outside perspective, it will just mean that GnuCash uses a file of some
> wierd binary format (that nobody has a hope in hell of hand-tweaking); but
> it means that with little more than the flick of acompile-time switch, it
> could also use a "live" RDBMS server, sharing access with other users and
> all that.
Well it looks like David Merrill may be interested in starting work on
the SQL stuff. We'd love to have your help/input if you're
interested.
Thanks
--
Rob Browning <[EMAIL PROTECTED]> PGP=E80E0D04F521A094 532B97F5D64E3930
___
gnucash-devel mailing list
[EMAIL PROTECTED]
http://www.gnumatic.com/cgi-bin/mailman/listinfo/gnucash-devel
/src/engine/sql/*
There is some code in /src/engine/sql that says it is broken. How bad is it? Should it be thrown away? Is there something salvageable? -- Dr. David C. Merrill http://www.lupercalia.net Linux Documentation Project[EMAIL PROTECTED] Collection Editor & Coordinatorhttp://www.linuxdoc.org Finger me for my public key When some people decide it's time for everyone to make big changes, it means that they want you to change first. ___ gnucash-devel mailing list [EMAIL PROTECTED] http://www.gnumatic.com/cgi-bin/mailman/listinfo/gnucash-devel
Re: Salutations
On Sat, Dec 09, 2000 at 05:14:47PM -0600, Rob Browning wrote: > David Merrill <[EMAIL PROTECTED]> writes: > > > Oh, is that similar to ODBC in that it abstracts the database behind > > an API through which you pass SQL? Maybe that answers my questions > > above? > > Presuming my nebulous knowledge of what ODBC is is right, then yes. > Gnome DBA is an abstract interface to the SQL databases, and it > supports various backends. Yep, that's ODBC in a nutshell. > > What would be the benefit? With relatively small datasets, there > > wouldn't be much, if any, speed gains, but only lots of memory > > use. It seems like overkill for many people, and since gnucash > > already has a working local storage mechanism... > > Reasons in favor (off the top of my head): Okay, okay, sounds like you've thought through the implications pretty well. Can you, if you haven't already, write up a list of some of the major design points you have already considered? I'll work on some of my own tomorrow. -- Dr. David C. Merrill http://www.lupercalia.net Linux Documentation Project[EMAIL PROTECTED] Collection Editor & Coordinatorhttp://www.linuxdoc.org Finger me for my public key Q: How many DEC repairman does it take to fix a flat? A: Five; four to hold the car up and one to swap tires. Q: How long does it take? A: It's indeterminate. It will depend upon how many flats they've brought with them. Q: What happens if you've got TWO flats? A: They replace your generator. ___ gnucash-devel mailing list [EMAIL PROTECTED] http://www.gnumatic.com/cgi-bin/mailman/listinfo/gnucash-devel
Re: Salutations
On Sat, Dec 09, 2000 at 05:00:20PM -0600, Rob Browning wrote: > Steve Greenland <[EMAIL PROTECTED]> writes: > > > I'm not trying to start a religious war, but at the moment, MySQL > > doesn't support transactions, which I see as one of the big gains in > > moving from (or supporting in addition to) the current file based > > system. Yes, one can implement transactions "on-top-of", but then you > > might as well stick with the file system. > > > > Of course, the Right Thing is to be DB independent, but you probably > > need to have some sort of "pre-test" to make sure the configured DB > > supports all the necessary functionality. > > Well, my little knowledge tends to make me favor PostreSQL too, but > I'd be happy to defer to people with more knowledge, and as you point > out, one question we need to address is whether we want to be LCD > (Least Common Denominator), or whether or not we are willing, if even > for the short term, to commit ourselves to a particular DB in exchange > for enhanced functionality... We *are* going to wind up with some kind of Least Common Denominator based on our requirements, and I suspect that support for transactions will be among them. If that happens, MySQL is out. If we can find ways to do what we need to do without support for transactions, that would be good, though. I'd hate to not support MySQL since so many people use it. I also am not interested in religious wars. We should keep in mind that it may take some time to get a production release out with SQL support, and MySQL is working on transaction support right now. I tried checking out gnome-db, but their site was down this afternoon. Has anyone here actually worked with gnome-db, or any other database abstraction library? > As long as it's a free DB, it's not as huge a deal to exclusively > support just that one, since anyone can install it (and we can include > it if needed), but it's still not as flexible as allowing the user to > use the DB they've already deployed, and as we move into the > small-business arena, allowing people to stick data in *their* DB will > probably become more critical. Though, even there, as long as we have > ways to interface with other DBs (i.e. batch push/pull) that may make > most people happy. True, taking the expense of installing the db changes the equation. The project I last worked on that required extensive multiple database support was for exactly that reason. -- Dr. David C. Merrill http://www.lupercalia.net Linux Documentation Project[EMAIL PROTECTED] Collection Editor & Coordinatorhttp://www.linuxdoc.org Finger me for my public key Harrisberger's Fourth Law of the Lab: Experience is directly proportional to the amount of equipment ruined. ___ gnucash-devel mailing list [EMAIL PROTECTED] http://www.gnumatic.com/cgi-bin/mailman/listinfo/gnucash-devel
