> -----Original Message----- > From: Tom Lane [mailto:[EMAIL PROTECTED] > Sent: Friday, June 20, 2003 11:47 PM > To: Dann Corbit > Cc: Jason Earl; PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > "Dann Corbit" <[EMAIL PROTECTED]> writes: > > Look at this: > > > http://www.mysql.com/information/crash-me.php?mysql_4_1=on&postgres=on > > This looks a little cleaner than the last time I looked at it > (more than three years ago), but it's still fundamentally a > marketing effort. It is not an exercise in spec compliance > measurement, because there are hundreds of bullet points that > all look exactly alike, whether they are measuring > spec-required elements, random vendor extensions to the spec, > or spec violations. To take just one example of the latter, > "Calculate 1--1" is still shown with a green star for MySQL > and a failure for Postgres, when a more correct reading would > be "Fails to recognize SQL-standard -- comment syntax" for > MySQL. And yes, they were called out on this three years > ago, and no they haven't fixed the entry since then. I > should believe that there is any good faith on their part? > > For another example, take a close look at the "Quoting" > section, which purports to measure compliance to the spec's > ideas about how to quote an identifier. Postgres accepts > double-quoted identifiers per spec, including doubled double > quotes per spec, and rejects bracketed or backquoted > identifiers per spec. MySQL is apparently spec compliant on > just one of those four points. Curious that they manage to > end up with a better looking display than us in this section; > in particular note that Postgres is specifically claimed > *not* to handle double-quoted identifiers. (Memory is fuzzy > after three years, but IIRC when you look at the actual test > code being used, it tests more than whether double quoted > identifiers are allowed, and really is failing us on some > quite unrelated detail.) > > Another point worth mentioning is that most of the numerical > limits shown in the table have nothing to do with actual > server limits, but with random limitations of their test > process. For instance, I'm not sure what "max index part > length 235328" really means, but I am pretty sure it's got > nothing to do with the Postgres server. Or look at "constant > string size in SELECT 16777207" ... nope, there's no such > limit. (If they'd put a "+" in there then it'd be okay, but > no.) I still remember watching crash-me trying to measure the > max query length of Postgres 7.0: the crashme client process > dumped core before Postgres did, after which the controlling > script announced that we weren't crash-safe. > > > So far, I have seen three problems pointed out (out of 600+ tests). > > These are the high spots from three-year-old memories. Do > you really want a detailed analysis? A quick look at their > table recalls plenty of bogosity to my mind. > > A last point is that this table is comparing MySQL 4.1 > (bleeding edge alpha release) against PG 7.2 (one full major > release behind the times). While I cannot really blame the > MySQL guys for not being up-to-the- minute on everyone else's > releases, this does emphasize the key point, namely that this > isn't a fair comparison run by disinterested parties but a > marketing effort of, by, and for MySQL.
It seems pretty clear that there are warts on the Crashme test. Perhaps 70% or so is truly useful. Maybe the useful subset could be approximated or modified to be useful as a general tool set. Not too surprising that a commercial enterprise tries to bend the facts in their favor a bit. Some other stuff worth note: http://osdb.sourceforge.net/ http://sourceforge.net/projects/osdldbt (looks like someone has put a bunch of PostgreSQL effort into it. http://sourceforge.net/projects/ltp/ (DOTS) http://www.mysql.com/portal/software/item-222.html (I won't mention where it's from) ftp://ftp.cs.wisc.edu/OO7/ Win32 specific, but has source code: http://www.mipt.sw.ru/en/install/ots/ (ODBC testing) http://www.mipt.sw.ru/en/install/ats/ (ADO testing) Some other interesting stuff is found there too... Test tools links: http://www.softwareqatest.com/qattls1.html http://www.aptest.com/resources.html ---------------------------(end of broadcast)--------------------------- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
