Re: [HACKERS] bitmap AM design

2005-03-05 Thread Pailloncy Jean-Gerard
Pailloncy Jean-Gerard wrote: You should have a look to this thread http://archives.postgresql.org/pgsql-hackers/2005-02/msg00263.php Take a look at this paper about "lock-free parallel hash table" http://www.cs.rug.nl/~wim/mechver/hashtable/ Is this relevant? Hash indexes are on-disk data structure

Re: [HACKERS] bitmap AM design

2005-03-04 Thread Victor Y. Yegorov
Some more thoughts and questions. The main idea above bitmaps is narrowing some data sets' possible values to "yes" and "no" (i.e. 1 and 0) and then organize the data in the series of bits, where bit's position determines values to consider. In the cases, where several indexed attributes are used

Re: [HACKERS] bitmap AM design

2005-03-04 Thread pgsql
> [EMAIL PROTECTED] writes: >> Anyway, IMHO, hash indexes would be dramatically improved if you could >> specify your own hashing function > > That's called a custom operator class. Would I also be able to query the bucket size and all that? > >> and declare initial table size. > > It would be in

Re: [HACKERS] bitmap AM design

2005-03-04 Thread Tom Lane
[EMAIL PROTECTED] writes: > Anyway, IMHO, hash indexes would be dramatically improved if you could > specify your own hashing function That's called a custom operator class. > and declare initial table size. It would be interesting to see if setting up the hashtable with about the right number o

Re: [HACKERS] bitmap AM design

2005-03-04 Thread Tom Lane
Neil Conway <[EMAIL PROTECTED]> writes: > (BTW, is poor concurrency really the biggest issue with hash indexes? If > so, there is some low-hanging fruit that I noticed a few years ago, but > never got around to fixing: _hash_doinsert() write-locks the hash > metapage on every insertion merely to

Re: [HACKERS] bitmap AM design

2005-03-04 Thread pgsql
> Pailloncy Jean-Gerard wrote: >> You should have a look to this thread >> http://archives.postgresql.org/pgsql-hackers/2005-02/msg00263.php >> >> Take a look at this paper about "lock-free parallel hash table" >> http://www.cs.rug.nl/~wim/mechver/hashtable/ > > Is this relevant? Hash indexes are o

Re: [HACKERS] bitmap AM design

2005-03-04 Thread Neil Conway
Pailloncy Jean-Gerard wrote: You should have a look to this thread http://archives.postgresql.org/pgsql-hackers/2005-02/msg00263.php Take a look at this paper about "lock-free parallel hash table" http://www.cs.rug.nl/~wim/mechver/hashtable/ Is this relevant? Hash indexes are on-disk data structure

Re: [HACKERS] bitmap AM design

2005-03-04 Thread Pailloncy Jean-Gerard
Le 2 mars 05, à 21:17, Hannu Krosing a écrit : Ühel kenal päeval (teisipäev, 1. märts 2005, 14:54-0500), kirjutas [EMAIL PROTECTED]: Now, it occurs to me that if my document reference table can refer to something other than an indexed primary key, I can save a lot of index processing time in Postgr

Re: [HACKERS] bitmap AM design

2005-03-03 Thread pgsql
> Ühel kenal päeval (teisipäev, 1. märts 2005, 14:54-0500), kirjutas > [EMAIL PROTECTED]: >> Now, it occurs to me that if my document reference table can refer to >> something other than an indexed primary key, I can save a lot of index >> processing time in PostgreSQL if I can have a "safe" analog

Re: [HACKERS] bitmap AM design

2005-03-03 Thread Hannu Krosing
Ühel kenal päeval (teisipäev, 1. märts 2005, 14:54-0500), kirjutas [EMAIL PROTECTED]: > Now, it occurs to me that if my document reference table can refer to > something other than an indexed primary key, I can save a lot of index > processing time in PostgreSQL if I can have a "safe" analogy to CT

Re: [HACKERS] bitmap AM design

2005-03-01 Thread pgsql
OK, lets step back a bit and see if there is a solution that fits what we think we need and PostgreSQL. Lets talk about FTSS, its something I can discuss easily. It is a two stage system with an indexer and a server. Only the data to be indexed is in the database, all the FTSS data structures are

Re: [HACKERS] bitmap AM design

2005-03-01 Thread Victor Y. Yegorov
* Tom Lane <[EMAIL PROTECTED]> [01.03.2005 21:07]: > Hmm, you seem to be envisioning that these are actually lists, not > arrays --- that is, to find the N'th page in a list requires traversing > list links starting at the first page. That doesn't sound workable. > If you can't access all the entr

Re: [HACKERS] bitmap AM design

2005-03-01 Thread Tom Lane
"Victor Y. Yegorov" <[EMAIL PROTECTED]> writes: > All lists (list of ctids, bitmaps) will only grow, no data will be > deleted, as deletes will require relocation and possibly exclusive > lock on the index. > Extending lists will need only a short-term exclusive locks on the pages in > the tails of

Re: [HACKERS] bitmap AM design

2005-03-01 Thread Tom Lane
[EMAIL PROTECTED] writes: > A "persistent reference" index would be like almost any other index except > that as new items are added to a table a new entry is added to the end of > the index. When a row is updated, its CTID is updated in the table. This *does not work* under MVCC: it can't cope wi

Re: [HACKERS] bitmap AM design

2005-03-01 Thread pgsql
> [EMAIL PROTECTED] writes: >> Tom, I posted a message about a week ago (I forget the name) about a >> persistent reference index, sort of like CTID, but basically a table >> lookup. The idea is to simulate a structure that ISAM sort of techniques >> can work in PostgreSQL. > >> Eliminating the bit

Re: [HACKERS] bitmap AM design

2005-03-01 Thread Tom Lane
[EMAIL PROTECTED] writes: > Tom, I posted a message about a week ago (I forget the name) about a > persistent reference index, sort of like CTID, but basically a table > lookup. The idea is to simulate a structure that ISAM sort of techniques > can work in PostgreSQL. > Eliminating the bitmap inde

Re: [HACKERS] bitmap AM design

2005-03-01 Thread Victor Y. Yegorov
* Tom Lane <[EMAIL PROTECTED]> [01.03.2005 09:37]: > The other stuff you mentioned implies that an insertion therefore > requires exclusive lock on the index (because you may have to relocate > everything in sight to add one more CTID slot). No, exclusive lock on index is worst thing to do. All l

Re: [HACKERS] bitmap AM design

2005-03-01 Thread pgsql
> I don't think we really need any more fundamentally nonconcurrent index > types :-( > Tom, I posted a message about a week ago (I forget the name) about a persistent reference index, sort of like CTID, but basically a table lookup. The idea is to simulate a structure that ISAM sort of technique

Re: [HACKERS] bitmap AM design

2005-02-28 Thread Tom Lane
"Victor Y. Yegorov" <[EMAIL PROTECTED]> writes: > Neil suggested a very good way how to handle updates. Of course, it's not > necessary to strictly follow tuple layout in the table's heap, as I wanted > to do initially. All that's needed, is that bit positions in bitmaps would > be tied with CTID p

[HACKERS] bitmap AM design

2005-02-28 Thread Victor Y. Yegorov
Here's the design of bitmap AM I'm planning to implement. I've discussed it with Neil, but I'm willing to get more feedback on it. There are going to be 1 metapage, 1 list of CTIDs (LOC), one list of attribute values (LOV, including attributes for multi-column indexes) and a bitmap for each entry