Re: [HACKERS] Pluggable storage

On Wed, Oct 25, 2017 at 1:59 PM, Amit Kapila wrote: >> Another thing to consider is that, if we could replace satisfiesfunc, >> it would probably break some existing code. There are multiple places >> in the code that compare snapshot->satisfies to >> HeapTupleSatisfiesHistoricMVCC and HeapTupleS

Re: [HACKERS] Pluggable storage

On Wed, Oct 25, 2017 at 11:37 AM, Robert Haas wrote: > On Fri, Oct 20, 2017 at 5:47 AM, Amit Kapila wrote: >> I think what we need here is a way to register satisfies function >> (SnapshotSatisfiesFunc) in SnapshotData for different storage engines. > > I don't see how that helps very much. Snap

Re: [HACKERS] Pluggable storage

On Fri, Oct 20, 2017 at 5:47 AM, Amit Kapila wrote: > I think what we need here is a way to register satisfies function > (SnapshotSatisfiesFunc) in SnapshotData for different storage engines. I don't see how that helps very much. SnapshotSatisfiesFunc takes a HeapTuple as an argument, and it ca

Re: [HACKERS] Pluggable storage

On Sat, Oct 14, 2017 at 1:09 AM, Alexander Korotkov wrote: > On Fri, Oct 13, 2017 at 9:41 PM, Robert Haas wrote: >> >> On Fri, Oct 13, 2017 at 1:59 PM, Peter Geoghegan wrote: >> >> Fully agreed. >> > >> > If we implement that interface, where does that leave EvalPlanQual()? > > > From the first

Re: [HACKERS] Pluggable storage

On Fri, Oct 13, 2017 at 1:58 PM, Haribabu Kommi wrote: > > On Fri, Oct 13, 2017 at 11:55 AM, Robert Haas wrote: >> >> On Thu, Oct 12, 2017 at 8:00 PM, Haribabu Kommi >> wrote: >> > Currently I added a snapshot_satisfies API to find out whether the tuple >> > satisfies the visibility or not with

Re: [HACKERS] Pluggable storage

On Fri, Oct 13, 2017 at 9:41 PM, Robert Haas wrote: > On Fri, Oct 13, 2017 at 1:59 PM, Peter Geoghegan wrote: > >> Fully agreed. > > > > If we implement that interface, where does that leave EvalPlanQual()? > >From the first glance, it seems that pluggable storage should override EvalPlanQualFe

Re: [HACKERS] Pluggable storage

On Fri, Oct 13, 2017 at 9:37 PM, Robert Haas wrote: > On Fri, Oct 13, 2017 at 5:25 AM, Kuntal Ghosh > wrote: > > For some other > > storage engine, if we maintain the older version in different storage, > > undo for example, and don't require a new index entry, should we still > > call it HOT-ch

Re: [HACKERS] Pluggable storage

On Fri, Oct 13, 2017 at 1:59 PM, Peter Geoghegan wrote: >> Fully agreed. > > If we implement that interface, where does that leave EvalPlanQual()? > Do those semantics have to be preserved? For a general-purpose heap storage format, I would say yes. I mean, we don't really have control over how

Re: [HACKERS] Pluggable storage

On Fri, Oct 13, 2017 at 5:25 AM, Kuntal Ghosh wrote: > For some other > storage engine, if we maintain the older version in different storage, > undo for example, and don't require a new index entry, should we still > call it HOT-chain? I would say, emphatically, no. HOT is a creature of the exi

Re: [HACKERS] Pluggable storage

On Thu, Oct 12, 2017 at 2:23 PM, Robert Haas wrote: > On Thu, Oct 12, 2017 at 4:38 PM, Alexander Korotkov >> However I imply that alternative storage would share our "MVCC model". So, >> it >> should share our transactional model including transactions, >> subtransactions, snapshots etc. >> There

Re: [HACKERS] Pluggable storage

On Fri, Oct 13, 2017 at 1:58 PM, Haribabu Kommi wrote: > > > On Fri, Oct 13, 2017 at 11:55 AM, Robert Haas wrote: >> >> On Thu, Oct 12, 2017 at 8:00 PM, Haribabu Kommi >> wrote: >> >> That seems like a strange choice of API. I think it should more >> integrated with the scan logic. For example

Re: [HACKERS] Pluggable storage

On Fri, Oct 13, 2017 at 11:55 AM, Robert Haas wrote: > On Thu, Oct 12, 2017 at 8:00 PM, Haribabu Kommi > wrote: > > Currently I added a snapshot_satisfies API to find out whether the tuple > > satisfies the visibility or not with different types of visibility > routines. > > I feel these > > are

Re: [HACKERS] Pluggable storage

On Thu, Oct 12, 2017 at 8:00 PM, Haribabu Kommi wrote: > Currently I added a snapshot_satisfies API to find out whether the tuple > satisfies the visibility or not with different types of visibility routines. > I feel these > are some how enough to develop a different storage methods like UNDO. >

Re: [HACKERS] Pluggable storage

On Fri, Oct 13, 2017 at 8:23 AM, Robert Haas wrote: > On Thu, Oct 12, 2017 at 4:38 PM, Alexander Korotkov > wrote: > > It's probably that we imply different meaning to "MVCC implementation". > > While writing "MVCC implementation" I meant that, for instance, > alternative > > storage > > may imp

Re: [HACKERS] Pluggable storage

On Thu, Oct 12, 2017 at 4:38 PM, Alexander Korotkov wrote: > It's probably that we imply different meaning to "MVCC implementation". > While writing "MVCC implementation" I meant that, for instance, alternative > storage > may implement UNDO chains to store versions of same row. Correspondingly,

Re: [HACKERS] Pluggable storage

On Wed, Oct 11, 2017 at 11:08 PM, Robert Haas wrote: > On Mon, Oct 9, 2017 at 10:22 AM, Alexander Korotkov > wrote: > > For me, it's crucial point that pluggable storages should be able to have > > different MVCC implementation, and correspondingly have full control over > > its interactions wit

Re: [HACKERS] Pluggable storage

On Wed, Oct 11, 2017 at 1:08 PM, Robert Haas wrote: > On Mon, Oct 9, 2017 at 10:22 AM, Alexander Korotkov > wrote: >> For me, it's crucial point that pluggable storages should be able to have >> different MVCC implementation, and correspondingly have full control over >> its interactions with ind

Re: [HACKERS] Pluggable storage

On Mon, Oct 9, 2017 at 10:22 AM, Alexander Korotkov wrote: > For me, it's crucial point that pluggable storages should be able to have > different MVCC implementation, and correspondingly have full control over > its interactions with indexes. > Thus, it would be good if we would get consensus on

Re: [HACKERS] Pluggable storage

On Mon, Oct 9, 2017 at 5:32 PM, Tom Lane wrote: > Alexander Korotkov writes: > > For me, it's crucial point that pluggable storages should be able to have > > different MVCC implementation, and correspondingly have full control over > > its interactions with indexes. > > Thus, it would be good i

Re: [HACKERS] Pluggable storage

Alexander Korotkov writes: > For me, it's crucial point that pluggable storages should be able to have > different MVCC implementation, and correspondingly have full control over > its interactions with indexes. > Thus, it would be good if we would get consensus on that point. I'd like > other di

Re: [HACKERS] Pluggable storage

On Wed, Sep 27, 2017 at 7:51 PM, Alexander Korotkov < a.korot...@postgrespro.ru> wrote: > I took a look on this patch. I've following notes for now. > > tuple_insert_hook tuple_insert; /* heap_insert */ >> tuple_update_hook tuple_update; /* heap_update */ >> tuple_delete_hook tuple_delete; /* hea

Re: [HACKERS] Pluggable storage

Hi! I took a look on this patch. I've following notes for now. tuple_insert_hook tuple_insert; /* heap_insert */ > tuple_update_hook tuple_update; /* heap_update */ > tuple_delete_hook tuple_delete; /* heap_delete */ I don't think this set of functions provides good enough level of abstraction

Re: [HACKERS] Pluggable storage

On Wed, Aug 23, 2017 at 8:26 AM, Haribabu Kommi wrote: > - Minor: don't think the _function suffix for Storis necessary, just >> makes things long, and every member has it. Besides that, it's also >> easy to misunderstand - for a second I understood >> scan_getnext_function to be about gett

Re: [HACKERS] Pluggable storage

Hi, Thank you for review on this subject. I think it's extremely important for PostgreSQL to eventually get pluggable storage API. In general I agree with all your points. But I'd like to make couple of comments. On Tue, Aug 15, 2017 at 9:53 AM, Andres Freund wrote: > - I don't think we shoul

Re: [HACKERS] Pluggable storage

On Thu, Sep 14, 2017 at 8:17 AM, Haribabu Kommi wrote: > Instead of modifying the Bitmap Heap and Sample scan's to avoid referring > the internal members of the HeapScanDesc, I divided the HeapScanDesc > into two parts. > > 1. StorageScanDesc > 2. HeapPageScanDesc > > The StorageScanDesc contains

Re: [HACKERS] Pluggable storage

On Sat, Sep 9, 2017 at 1:23 PM, Haribabu Kommi wrote: > > I rebased the patch to the latest master and also fixed the duplicate OID > and some slot fixes. Updated patches are attached. > While analyzing the removal of HeapScanDesc usage other than heap modules, The mostly used member is "*rs_cb

Re: [HACKERS] Pluggable storage

On Fri, Sep 1, 2017 at 1:51 PM, Haribabu Kommi wrote: > Here I attached new set of patches that are rebased to the latest master. Hi Haribabu, Could you please post a new rebased version? 0007-Scan-functions-are-added-to-storage-AM.patch conflicts with recent changes. Thanks! -- Thomas Munro

Re: [HACKERS] Pluggable storage

On Wed, Aug 23, 2017 at 11:59 PM, Amit Kapila wrote: > On Wed, Aug 23, 2017 at 11:05 AM, Haribabu Kommi > wrote: > > > > > > On Mon, Aug 21, 2017 at 7:25 PM, Amit Kapila > > wrote: > >> > >> On Mon, Aug 21, 2017 at 12:58 PM, Haribabu Kommi > >> wrote: > >> > > >> > On Sun, Aug 13, 2017 at 5:17

Re: [HACKERS] Pluggable storage

On Wed, Aug 23, 2017 at 11:05 AM, Haribabu Kommi wrote: > > > On Mon, Aug 21, 2017 at 7:25 PM, Amit Kapila > wrote: >> >> On Mon, Aug 21, 2017 at 12:58 PM, Haribabu Kommi >> wrote: >> > >> > On Sun, Aug 13, 2017 at 5:17 PM, Amit Kapila >> > wrote: >> >> >> >> >> >> Also, it is quite possible th

Re: [HACKERS] Pluggable storage

On Mon, Aug 21, 2017 at 7:25 PM, Amit Kapila wrote: > On Mon, Aug 21, 2017 at 12:58 PM, Haribabu Kommi > wrote: > > > > On Sun, Aug 13, 2017 at 5:17 PM, Amit Kapila > > wrote: > >> > >> > >> Also, it is quite possible that some of the storage Am's don't even > >> want to return bool as a parame

Re: [HACKERS] Pluggable storage

On Tue, Aug 15, 2017 at 4:53 PM, Andres Freund wrote: > Hi, > > > On 2017-06-13 11:50:27 +1000, Haribabu Kommi wrote: > > Here I attached WIP patches to support pluggable storage. The patch > series > > are may not work individually. Still so many things are under > development. > > These patches

Re: [HACKERS] Pluggable storage

On Mon, Aug 21, 2017 at 12:58 PM, Haribabu Kommi wrote: > > On Sun, Aug 13, 2017 at 5:17 PM, Amit Kapila > wrote: >> >> >> Also, it is quite possible that some of the storage Am's don't even >> want to return bool as a parameter from HeapTupleSatisfies* API's. I >> guess what we need here is to

Re: [HACKERS] Pluggable storage

On Sun, Aug 13, 2017 at 5:17 PM, Amit Kapila wrote: > On Sat, Aug 12, 2017 at 10:31 AM, Haribabu Kommi > wrote: > >> > >> Why do we need to store handler function in TupleDesc? As of now, the > >> above patch series has it available in RelationData and > >> TupleTableSlot, I am not sure if inst

Re: [HACKERS] Pluggable storage

Hi, On 2017-06-13 11:50:27 +1000, Haribabu Kommi wrote: > Here I attached WIP patches to support pluggable storage. The patch series > are may not work individually. Still so many things are under development. > These patches are just to share the approach of the current development. Making a pa

Re: [HACKERS] Pluggable storage

On Sun, Aug 13, 2017 at 5:21 PM, Amit Kapila wrote: > On Sat, Aug 12, 2017 at 10:34 AM, Haribabu Kommi > wrote: > > > > On Tue, Aug 8, 2017 at 2:21 PM, Amit Kapila > wrote: > >> > >> +typedef struct StorageAmRoutine > >> +{ > >> > >> In this structure, you have already covered most of the API's

Re: [HACKERS] Pluggable storage

On Sat, Aug 12, 2017 at 10:34 AM, Haribabu Kommi wrote: > > On Tue, Aug 8, 2017 at 2:21 PM, Amit Kapila wrote: >> >> On Tue, Jun 13, 2017 at 7:20 AM, Haribabu Kommi >> wrote: >> > >> > >> > On Fri, Oct 14, 2016 at 7:26 AM, Alvaro Herrera >> > >> > wrote: >> >> >> >> I have sent the partial patc

Re: [HACKERS] Pluggable storage

On Sat, Aug 12, 2017 at 10:31 AM, Haribabu Kommi wrote: >> >> Why do we need to store handler function in TupleDesc? As of now, the >> above patch series has it available in RelationData and >> TupleTableSlot, I am not sure if instead of that keeping it in >> TupleDesc is a good idea. Which all

Re: [HACKERS] Pluggable storage

On Tue, Aug 8, 2017 at 2:21 PM, Amit Kapila wrote: > On Tue, Jun 13, 2017 at 7:20 AM, Haribabu Kommi > wrote: > > > > > > On Fri, Oct 14, 2016 at 7:26 AM, Alvaro Herrera < > alvhe...@2ndquadrant.com> > > wrote: > >> > >> I have sent the partial patch I have to Hari Babu Kommi. We expect that >

Re: [HACKERS] Pluggable storage

On Mon, Aug 7, 2017 at 11:12 PM, Amit Kapila wrote: > On Tue, Aug 1, 2017 at 1:56 PM, Haribabu Kommi > wrote: > > > > > > On Sun, Jul 23, 2017 at 4:10 PM, Amit Kapila > > wrote: > >> > > > >> > >> > 1. Design an API that returns values/nulls array and convert that > into a > >> > HeapTuple when

Re: [HACKERS] Pluggable storage

On Tue, Jun 13, 2017 at 7:20 AM, Haribabu Kommi wrote: > > > On Fri, Oct 14, 2016 at 7:26 AM, Alvaro Herrera > wrote: >> >> I have sent the partial patch I have to Hari Babu Kommi. We expect that >> he will be able to further this goal some more. > > > Thanks Alvaro for sharing your development

Re: [HACKERS] Pluggable storage

On Tue, Aug 1, 2017 at 1:56 PM, Haribabu Kommi wrote: > > > On Sun, Jul 23, 2017 at 4:10 PM, Amit Kapila > wrote: >> > >> >> > 1. Design an API that returns values/nulls array and convert that into a >> > HeapTuple whenever it is required in the upper layers. All the existing >> > heap form/defor

Re: [HACKERS] Pluggable storage

On Sun, Jul 23, 2017 at 4:10 PM, Amit Kapila wrote: > On Wed, Jul 19, 2017 at 11:33 AM, Haribabu Kommi > wrote: > > > > I am finding out that eliminating the HeapTuple usage in the upper layers > > needs some major changes, How about not changing anything in the upper > > layers of storage curre

Re: [HACKERS] Pluggable storage

On Wed, Jul 19, 2017 at 11:33 AM, Haribabu Kommi wrote: > > > On Sat, Jul 15, 2017 at 12:30 PM, Robert Haas wrote: >> > > I am finding out that eliminating the HeapTuple usage in the upper layers > needs some major changes, How about not changing anything in the upper > layers of storage currentl

Re: [HACKERS] Pluggable storage

On Wed, Jul 19, 2017 at 10:56 AM, Robert Haas wrote: >> I strongly agree. I simply don't understand how you can adopt UNDO for >> MVCC, and yet expect to get a benefit commensurate with the effort >> without also implementing "retail index tuple deletion" first. > > I agree that we need retail ind

Re: [HACKERS] Pluggable storage

On Sat, Jul 15, 2017 at 8:58 PM, Peter Geoghegan wrote: > To repeat myself, for emphasis: *Not all bloat is equal*. +1. > I strongly agree. I simply don't understand how you can adopt UNDO for > MVCC, and yet expect to get a benefit commensurate with the effort > without also implementing "retai

Re: [HACKERS] Pluggable storage

On Sat, Jul 15, 2017 at 6:36 PM, Alexander Korotkov wrote: > I think in general there are two ways dealing with out index AM API > limitation. One of them is to extend index AM API. That's pretty much what I have in mind. I think it's likely that if we end up with, say, 3 kinds of heap and 12 k

Re: [HACKERS] Pluggable storage

On Sat, Jul 15, 2017 at 12:30 PM, Robert Haas wrote: > On Fri, Jul 14, 2017 at 8:35 AM, Haribabu Kommi > wrote: > > To replace tuple with slot, I took trigger and SPI calls as the first > step > > in modifying > > from tuple to slot, Here I attached a WIP patch. The notable changes are, > > > >

Re: [HACKERS] Pluggable storage

On Mon, Jul 17, 2017 at 1:24 PM, Alexander Korotkov wrote: > It's probably depends on particular storage (once we have pluggable > storages). Some storages would have additional level of indirection while > others wouldn't. Agreed. Like kill_prior_tuple, it's an optional capability, and where im

Re: [HACKERS] Pluggable storage

On Mon, Jul 17, 2017 at 7:51 PM, Peter Geoghegan wrote: > On Mon, Jul 17, 2017 at 3:22 AM, Alexander Korotkov > wrote: > > I think that "retail index tuple deletion" is the feature which could > give > > us some advantages even independently from pluggable storages. For > example, > > imagine v

Re: [HACKERS] Pluggable storage

On Mon, Jul 17, 2017 at 3:22 AM, Alexander Korotkov wrote: > I think that "retail index tuple deletion" is the feature which could give > us some advantages even independently from pluggable storages. For example, > imagine very large table with only small amount of dead tuples. In this > case,

Re: [HACKERS] Pluggable storage

On Sun, Jul 16, 2017 at 3:58 AM, Peter Geoghegan wrote: > I strongly agree. I simply don't understand how you can adopt UNDO for > MVCC, and yet expect to get a benefit commensurate with the effort > without also implementing "retail index tuple deletion" first. > Pursuing UNDO this way has the s

Re: [HACKERS] Pluggable storage

On Sat, Jul 15, 2017 at 3:36 PM, Alexander Korotkov wrote: > I think that pruning and vacuum are artifacts of not only current heap > formats, but they are also artifacts of current index AM API. And this is > more significant circumstance given that we're going to preserve > compatibility of new

Re: [HACKERS] Pluggable storage

On Sat, Jul 15, 2017 at 5:14 AM, Robert Haas wrote: > On Thu, Jun 22, 2017 at 9:30 AM, Alexander Korotkov > wrote: > > If #1 is only about changing tuple and page formats, then could be much > > simpler than the patch upthread? We can implement "page format access > > methods" with routines for

Re: [HACKERS] Pluggable storage

On Fri, Jul 14, 2017 at 8:35 AM, Haribabu Kommi wrote: > To replace tuple with slot, I took trigger and SPI calls as the first step > in modifying > from tuple to slot, Here I attached a WIP patch. The notable changes are, > > 1. Replace most of the HeapTuple with Slot in SPI interface functions.

Re: [HACKERS] Pluggable storage

On Thu, Jun 22, 2017 at 4:27 PM, Tomas Vondra wrote: > Can you elaborate a bit more about this TID bit pattern issues? I do > remember that not all TIDs are valid due to safeguards on individual fields, > like for example > > Assert(iptr->ip_posid < (1 << MaxHeapTuplesPerPageBits)) > > But per

Re: [HACKERS] Pluggable storage

On Thu, Jun 22, 2017 at 9:30 AM, Alexander Korotkov wrote: > If #1 is only about changing tuple and page formats, then could be much > simpler than the patch upthread? We can implement "page format access > methods" with routines for insertion, update, pruning and deletion of tuples > *in particu

Re: [HACKERS] Pluggable storage

On Wed, Jun 28, 2017 at 7:43 AM, Haribabu Kommi wrote: > > On Wed, Jun 28, 2017 at 12:00 AM, Alexander Korotkov > wrote: >> >> On Tue, Jun 27, 2017 at 4:19 PM, Amit Kapila >> wrote: >>> >>> On Thu, Jun 22, 2017 at 5:46 PM, Alexander Korotkov >>> wrote: >>> >>> Probably, some other algorithm for

Re: [HACKERS] Pluggable storage

On Tue, Jun 27, 2017 at 7:23 PM, Alexander Korotkov wrote: > On Tue, Jun 27, 2017 at 4:08 PM, Amit Kapila > wrote: >> >> Similarly, >> if the page format is changed you need to consider all page scan API's >> like heapgettup_pagemode/heapgetpage. > > > If page format is changed, then heapgettup_p

Re: [HACKERS] Pluggable storage

On Tue, Jun 27, 2017 at 11:53 PM, Alexander Korotkov < a.korot...@postgrespro.ru> wrote: > On Tue, Jun 27, 2017 at 4:08 PM, Amit Kapila > wrote: > >> On Thu, Jun 22, 2017 at 8:00 PM, Alexander Korotkov >> wrote: >> > On Wed, Jun 21, 2017 at 10:47 PM, Robert Haas >> wrote: >> >> >> >> On Mon, Ju

Re: [HACKERS] Pluggable storage

On Wed, Jun 28, 2017 at 12:00 AM, Alexander Korotkov < a.korot...@postgrespro.ru> wrote: > On Tue, Jun 27, 2017 at 4:19 PM, Amit Kapila > wrote: > >> On Thu, Jun 22, 2017 at 5:46 PM, Alexander Korotkov >> wrote: >> > On Tue, Jun 13, 2017 at 4:50 AM, Haribabu Kommi < >> kommi.harib...@gmail.com>

Re: [HACKERS] Pluggable storage

On Thu, Jun 22, 2017 at 5:47 AM, Robert Haas wrote: > On Mon, Jun 12, 2017 at 9:50 PM, Haribabu Kommi > wrote: > > Open Items: > > > > 1. The BitmapHeapScan and TableSampleScan are tightly coupled with > > HeapTuple and HeapScanDesc, So these scans are directly operating > > on those structures

Re: [HACKERS] Pluggable storage

On Tue, Jun 27, 2017 at 4:19 PM, Amit Kapila wrote: > On Thu, Jun 22, 2017 at 5:46 PM, Alexander Korotkov > wrote: > > On Tue, Jun 13, 2017 at 4:50 AM, Haribabu Kommi < > kommi.harib...@gmail.com> > > wrote: > >> > >> On Fri, Oct 14, 2016 at 7:26 AM, Alvaro Herrera < > alvhe...@2ndquadrant.com>

Re: [HACKERS] Pluggable storage

On Tue, Jun 27, 2017 at 4:08 PM, Amit Kapila wrote: > On Thu, Jun 22, 2017 at 8:00 PM, Alexander Korotkov > wrote: > > On Wed, Jun 21, 2017 at 10:47 PM, Robert Haas > wrote: > >> > >> On Mon, Jun 12, 2017 at 9:50 PM, Haribabu Kommi > >> wrote: > >> > Open Items: > >> > > >> > 1. The BitmapHeap

Re: [HACKERS] Pluggable storage

On Thu, Jun 22, 2017 at 5:46 PM, Alexander Korotkov wrote: > On Tue, Jun 13, 2017 at 4:50 AM, Haribabu Kommi > wrote: >> >> On Fri, Oct 14, 2016 at 7:26 AM, Alvaro Herrera >> wrote: >>> >>> I have sent the partial patch I have to Hari Babu Kommi. We expect that >>> he will be able to further th

Re: [HACKERS] Pluggable storage

On Thu, Jun 22, 2017 at 8:00 PM, Alexander Korotkov wrote: > On Wed, Jun 21, 2017 at 10:47 PM, Robert Haas wrote: >> >> On Mon, Jun 12, 2017 at 9:50 PM, Haribabu Kommi >> wrote: >> > Open Items: >> > >> > 1. The BitmapHeapScan and TableSampleScan are tightly coupled with >> > HeapTuple and HeapS

Re: [HACKERS] Pluggable storage

On Mon, Jun 26, 2017 at 10:55 PM, Tomas Vondra wrote: > On 06/26/2017 05:18 PM, Alexander Korotkov wrote: > >> I see that design question for PostgreSQL pluggable storages is very hard. >> > > IMHO it's mostly expected to be hard. > > Firstly, PostgreSQL is a mature product with many advanced fea

Re: [HACKERS] Pluggable storage

Hi, On 06/26/2017 05:18 PM, Alexander Korotkov wrote: Hackers, I see that design question for PostgreSQL pluggable storages is very hard. IMHO it's mostly expected to be hard. Firstly, PostgreSQL is a mature product with many advanced features, and reworking a low-level feature without bre

Re: [HACKERS] Pluggable storage

Hackers, I see that design question for PostgreSQL pluggable storages is very hard. BTW, I think it worth analyzing existing use-cases of pluggable storages. I think that the most famous DBMS with pluggable storage API is MySQL. This why I decided to start with it. I've added MySQL/MariaDB sectio

Re: [HACKERS] Pluggable storage

On 23/06/2017 16:07, Tomas Vondra wrote: > > I think you're probably right - GIN does compress the posting lists by > exploiting the TID redundancy (that it's page/offset structure), and I > don't think there are other index AMs doing that. > > But I'm not sure we can simply rely on that - it's p

Re: [HACKERS] Pluggable storage

Tomas Vondra writes: > It would be really great if you could explain why BitmapScans are > dubious, instead of just labeling them as dubious. (I guess you mean > Bitmap Heap Scans, right?) The two things I'm aware of are (a) the assumption you noted, that fetching tuples in TID sort order is a

Re: [HACKERS] Pluggable storage

Hi, On 6/23/17 10:38 AM, Teodor Sigaev wrote: 1. Table AM with a 6-byte TID. 2. Table AM with a custom locator format, which could be TID-like. 3. Table AM with no locators. Currently TID has its own type in system catalog. Seems, it's possible that storage claims type of TID which it uses. T

Re: [HACKERS] Pluggable storage

1. Table AM with a 6-byte TID. 2. Table AM with a custom locator format, which could be TID-like. 3. Table AM with no locators. Currently TID has its own type in system catalog. Seems, it's possible that storage claims type of TID which it uses. Then, AM could claim it too, so the core based o

Re: [HACKERS] Pluggable storage

On Thu, Jun 22, 2017 at 11:27 PM, Robert Haas wrote: > On Thu, Jun 22, 2017 at 8:32 AM, Alexander Korotkov > wrote: >> On Thu, Jun 22, 2017 at 4:01 AM, Michael Paquier >> wrote: >>> Putting that in a couple of words. >>> 1. Table AM with a 6-byte TID. >>> 2. Table AM with a custom locator format

Re: [HACKERS] Pluggable storage

Hi, On 6/22/17 4:36 PM, Alexander Korotkov wrote: On Thu, Jun 22, 2017 at 5:27 PM, Robert Haas > wrote: On Thu, Jun 22, 2017 at 8:32 AM, Alexander Korotkov mailto:a.korot...@postgrespro.ru>> wrote: > On Thu, Jun 22, 2017 at 4:01 AM, Michael Paquier mail

Re: [HACKERS] Pluggable storage

Hi, On 6/21/17 9:47 PM, Robert Haas wrote: ... > like int8 or numeric, it won't work at all. Even for other things, it's going to cause problems because the bit patterns won't be what the code is expecting; e.g. bitmap scans care about the structure of the TID, not just how many bits it is. (D

Re: [HACKERS] Pluggable storage

On Thu, Jun 22, 2017 at 5:27 PM, Robert Haas wrote: > On Thu, Jun 22, 2017 at 8:32 AM, Alexander Korotkov > wrote: > > On Thu, Jun 22, 2017 at 4:01 AM, Michael Paquier < > michael.paqu...@gmail.com> > > wrote: > >> Putting that in a couple of words. > >> 1. Table AM with a 6-byte TID. > >> 2. Ta

Re: [HACKERS] Pluggable storage

On Wed, Jun 21, 2017 at 10:47 PM, Robert Haas wrote: > On Mon, Jun 12, 2017 at 9:50 PM, Haribabu Kommi > wrote: > > Open Items: > > > > 1. The BitmapHeapScan and TableSampleScan are tightly coupled with > > HeapTuple and HeapScanDesc, So these scans are directly operating > > on those structures

Re: [HACKERS] Pluggable storage

On Thu, Jun 22, 2017 at 8:32 AM, Alexander Korotkov wrote: > On Thu, Jun 22, 2017 at 4:01 AM, Michael Paquier > wrote: >> Putting that in a couple of words. >> 1. Table AM with a 6-byte TID. >> 2. Table AM with a custom locator format, which could be TID-like. >> 3. Table AM with no locators. >>

Re: [HACKERS] Pluggable storage

On Thu, Jun 22, 2017 at 4:01 AM, Michael Paquier wrote: > Putting that in a couple of words. > 1. Table AM with a 6-byte TID. > 2. Table AM with a custom locator format, which could be TID-like. > 3. Table AM with no locators. > > Getting into having #1 first to work out would already be really >

Re: [HACKERS] Pluggable storage

On Tue, Jun 13, 2017 at 4:50 AM, Haribabu Kommi wrote: > On Fri, Oct 14, 2016 at 7:26 AM, Alvaro Herrera > wrote: > >> I have sent the partial patch I have to Hari Babu Kommi. We expect that >> he will be able to further this goal some more. > > > Thanks Alvaro for sharing your development patc

Re: [HACKERS] Pluggable storage

On Thu, Jun 22, 2017 at 11:12 AM, Amit Langote wrote: > On 2017/06/22 10:01, Michael Paquier wrote: >> #3 implies that the index AM logic is implemented in the table >> AM. Not saying that it is not useful, but it does not feel natural to >> have the planner request for a sequential scan, just to

Re: [HACKERS] Pluggable storage

On 2017/06/22 10:01, Michael Paquier wrote: > #3 implies that the index AM logic is implemented in the table > AM. Not saying that it is not useful, but it does not feel natural to > have the planner request for a sequential scan, just to have the table > AM secretly do some kind of index/skipping

Re: [HACKERS] Pluggable storage

On Thu, Jun 22, 2017 at 4:47 AM, Robert Haas wrote: > I think that BitmapHeapScan, at least, is applicable to any table AM > that has TIDs. It seems to me that in general we can imagine three > kinds of table AMs: > > 1. Table AMs where a tuple can be efficiently located by a real TID. > By a re

Re: [HACKERS] Pluggable storage

On Mon, Jun 12, 2017 at 9:50 PM, Haribabu Kommi wrote: > Open Items: > > 1. The BitmapHeapScan and TableSampleScan are tightly coupled with > HeapTuple and HeapScanDesc, So these scans are directly operating > on those structures and providing the result. > > These scan types may not be applicable

Re: [HACKERS] Pluggable storage

I have sent the partial patch I have to Hari Babu Kommi. We expect that he will be able to further this goal some more. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (p

Re: [HACKERS] Pluggable storage

Alvaro Herrera wrote: > Many have expressed their interest in this topic, but I haven't seen any > design of how it should work. Here's my attempt; I've been playing with > this for some time now and I think what I propose here is a good initial > plan. I regret to announce that I'll have to stay

Re: [HACKERS] Pluggable storage

On 2016-08-18 08:58:11 +0100, Simon Riggs wrote: > On 16 August 2016 at 19:46, Andres Freund wrote: > > On 2016-08-15 12:02:18 -0400, Robert Haas wrote: > >> Thanks for taking a stab at this. I'd like to throw out a few concerns. > >> > >> One, I'm worried that adding an additional layer of point

Re: [HACKERS] Pluggable storage

On August 18, 2016 7:44:50 AM PDT, Ants Aasma wrote: >On Tue, Aug 16, 2016 at 9:46 PM, Andres Freund >wrote: >> On 2016-08-15 12:02:18 -0400, Robert Haas wrote: >>> I am somewhat inclined to >>> believe that we need to restructure the executor in a bigger way so >>> that it passes around datums

Re: [HACKERS] Pluggable storage

On Tue, Aug 16, 2016 at 9:46 PM, Andres Freund wrote: > On 2016-08-15 12:02:18 -0400, Robert Haas wrote: >> I am somewhat inclined to >> believe that we need to restructure the executor in a bigger way so >> that it passes around datums instead of tuples; I'm inclined to >> believe that the curren

Re: [HACKERS] Pluggable storage

On Wed, Aug 17, 2016 at 10:33 PM, Alvaro Herrera wrote: > Anastasia Lubennikova wrote: >> >> Except these, there are some pretty strange and unrelated functions in >> src/backend/catalog. >> I'm willing to fix them, but I'd like to synchronize our efforts. > > I very much would like to stay away f

Re: [HACKERS] Pluggable storage

On Thu, Aug 18, 2016 at 10:58 AM, Simon Riggs wrote: > On 16 August 2016 at 19:46, Andres Freund wrote: > > On 2016-08-15 12:02:18 -0400, Robert Haas wrote: > >> Thanks for taking a stab at this. I'd like to throw out a few concerns. > >> > >> One, I'm worried that adding an additional layer of

Re: [HACKERS] Pluggable storage

On 16 August 2016 at 19:46, Andres Freund wrote: > On 2016-08-15 12:02:18 -0400, Robert Haas wrote: >> Thanks for taking a stab at this. I'd like to throw out a few concerns. >> >> One, I'm worried that adding an additional layer of pointer-jumping is >> going to slow things down and make Andres'

Re: [HACKERS] Pluggable storage

Anastasia Lubennikova wrote: > 13.08.2016 02:15, Alvaro Herrera: > >To support this, we introduce StorageTuple and StorageScanDesc. > >StorageTuples represent a physical tuple coming from some storage AM. > >It is necessary to have a pointer to a StorageAmRoutine in order to > >manipulate the tupl

Re: [HACKERS] Pluggable storage

13.08.2016 02:15, Alvaro Herrera: Many have expressed their interest in this topic, but I haven't seen any design of how it should work. Here's my attempt; I've been playing with this for some time now and I think what I propose here is a good initial plan. This will allow us to write permanent

Re: [HACKERS] Pluggable storage

On 2016-08-15 12:02:18 -0400, Robert Haas wrote: > Thanks for taking a stab at this. I'd like to throw out a few concerns. > > One, I'm worried that adding an additional layer of pointer-jumping is > going to slow things down and make Andres' work to speed up the > executor more difficult. I don

Re: [HACKERS] Pluggable storage

13.08.2016 02:15, Alvaro Herrera: Many have expressed their interest in this topic, but I haven't seen any design of how it should work. Here's my attempt; I've been playing with this for some time now and I think what I propose here is a good initial plan. This will allow us to write permanent

Re: [HACKERS] Pluggable storage

On Fri, Aug 12, 2016 at 7:15 PM, Alvaro Herrera wrote: > Many have expressed their interest in this topic, but I haven't seen any > design of how it should work. Here's my attempt; I've been playing with > this for some time now and I think what I propose here is a good initial > plan. This will

Re: [HACKERS] Pluggable storage

On Sat, Aug 13, 2016 at 2:15 AM, Alvaro Herrera wrote: > Many have expressed their interest in this topic, but I haven't seen any > design of how it should work. That's it. My design presented at PGCon was very sketchy, and I didn't deliver any prototype yet. > Here's my attempt; I've been p

[HACKERS] Pluggable storage

Many have expressed their interest in this topic, but I haven't seen any design of how it should work. Here's my attempt; I've been playing with this for some time now and I think what I propose here is a good initial plan. This will allow us to write permanent table storage that works differentl