Re: [HACKERS] Extending varlena

2008-08-20 Thread Tom Lane
> Peter Eisentraut wrote: >> If you replace the third point by "maybe partition TOAST tables", replace >> large object handle by TOAST pointer, and create an API to work on TOAST >> pointers, how are the two so much different? And why should they be? The reason they should be different is that

Re: [HACKERS] Extending varlena

2008-08-20 Thread Stephen R. van den Berg
Peter Eisentraut wrote: >If you replace the third point by "maybe partition TOAST tables", replace >large object handle by TOAST pointer, and create an API to work on TOAST >pointers, how are the two so much different? And why should they be? I can >see that there are going to be needs to acce

Re: [HACKERS] Extending varlena

2008-08-19 Thread Kevin Grittner
>>> Tom Lane <[EMAIL PROTECTED]> wrote: > I was kinda wondering about something closer to the TOAST model, where > a blob is only referenceable from a value that's in a table field; > and that value encapsulates the "name" of the blob in some way that > needn't even be user-visible. This'd grea

Re: [HACKERS] Extending varlena

2008-08-19 Thread Peter Eisentraut
Am Monday, 18. August 2008 schrieb Tom Lane: > - permissions features (more than "none" anyway) > - better management of orphaned objects (obsoleting vacuumlo) > - support > 16TB of large objects (maybe partition pg_largeobject?) > - dump and restore probably need improvement to be practical for su

Re: [HACKERS] Extending varlena

2008-08-19 Thread Magnus Hagander
Simon Riggs wrote: > On Mon, 2008-08-18 at 16:22 -0400, Tom Lane wrote: >> David Fetter <[EMAIL PROTECTED]> writes: >>> What would need to happen for the next jump up from where varlena is >>> now, to 8 bytes? >> Dealing with upwards-of-4GB blobs as single Datums isn't remotely sane, >> and won't b

Re: [HACKERS] Extending varlena

2008-08-19 Thread Magnus Hagander
Josh Berkus wrote: > Andrew, > >> I always find these requests puzzling. Is it really useful to store the >> data for a jpeg, video file or a 10GB tar ball in a database column? > > Some people find it useful. Because LOs are actually easier to manage in > PG than in most other DBMSes, right n

Re: [HACKERS] Extending varlena

2008-08-19 Thread Jeremy Drake
On Mon, 18 Aug 2008, Tom Lane wrote: > What would make more sense is to redesign the large-object stuff to be > somewhat modern and featureful, and provide stream-access APIs (think > lo_read, lo_seek, etc) that allow offsets wider than 32 bits. A few years ago, I was working on such a project fo

Re: [HACKERS] Extending varlena

2008-08-18 Thread Simon Riggs
On Mon, 2008-08-18 at 23:43 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > Probably also using a separate Sequence to allocate numbers rather than > > using up all the Oids on LOs would be a good plan. > > Well, assuming that your Large Objects are actually Large, you aren't

Re: [HACKERS] Extending varlena

2008-08-18 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > Probably also using a separate Sequence to allocate numbers rather than > using up all the Oids on LOs would be a good plan. Well, assuming that your Large Objects are actually Large, you aren't going to need as many OIDs as all that ;-) However: I was ch

Re: [HACKERS] Extending varlena

2008-08-18 Thread Simon Riggs
On Mon, 2008-08-18 at 16:22 -0400, Tom Lane wrote: > David Fetter <[EMAIL PROTECTED]> writes: > > What would need to happen for the next jump up from where varlena is > > now, to 8 bytes? > > Dealing with upwards-of-4GB blobs as single Datums isn't remotely sane, > and won't become so in the near

Re: [HACKERS] Extending varlena

2008-08-18 Thread Gregory Williamson
David Fetter wrote" <...> > > > This'd greatly simplify the > > cleanup-dead-objects problem, and we could avoid addressing the > > permissions problem at all, since regular SQL permissions on the table > > would serve fine. But it's not clear what regular SQL fetch and update > > behaviors shou

Re: [HACKERS] Extending varlena

2008-08-18 Thread David Fetter
On Mon, Aug 18, 2008 at 07:31:04PM -0400, Tom Lane wrote: > David Fetter <[EMAIL PROTECTED]> writes: > > On Mon, Aug 18, 2008 at 04:22:56PM -0400, Tom Lane wrote: > >> The main things I think we'd need to consider besides just the > >> access API are > >> > >> - permissions features (more than "no

Re: [HACKERS] Extending varlena

2008-08-18 Thread Tom Lane
David Fetter <[EMAIL PROTECTED]> writes: > On Mon, Aug 18, 2008 at 06:09:13PM -0400, Andrew Chernow wrote: >> I always find these requests puzzling. Is it really useful to store >> the data for a jpeg, video file or a 10GB tar ball in a database >> column? > It is if you need transaction seman

Re: [HACKERS] Extending varlena

2008-08-18 Thread Tom Lane
David Fetter <[EMAIL PROTECTED]> writes: > On Mon, Aug 18, 2008 at 04:22:56PM -0400, Tom Lane wrote: >> The main things I think we'd need to consider besides just the >> access API are >> >> - permissions features (more than "none" anyway) > Would ROLEs work, or are you thinking of the per-row an

Re: [HACKERS] Extending varlena

2008-08-18 Thread Andrew Chernow
Jeff Davis wrote: I always find these requests puzzling. Is it really useful to store the data for a jpeg, video file or a 10GB tar ball in a database column? One use case is that it can use the existing postgresql protocol, So can what I am suggesting. How about a user-defined C funct

Re: [HACKERS] Extending varlena

2008-08-18 Thread Andrew Chernow
Tom Lane wrote: Andrew Chernow <[EMAIL PROTECTED]> writes: Anyways (back on topic), I am in favor of removing limits from any section of the database ... not just your suggestion. The end-user application should impose limits. That's nice as an abstract principle, but there are only so many

Re: [HACKERS] Extending varlena

2008-08-18 Thread Tom Lane
Andrew Chernow <[EMAIL PROTECTED]> writes: > Anyways (back on topic), I am in favor of removing limits from any > section of the database ... not just your suggestion. The end-user > application should impose limits. That's nice as an abstract principle, but there are only so many hours in the

Re: [HACKERS] Extending varlena

2008-08-18 Thread Jeff Davis
> I always find these requests puzzling. Is it really useful to store the > data for a jpeg, video file or a 10GB tar ball in a database column? One use case is that it can use the existing postgresql protocol, and does not require extra filesystem mounts, extra error handling, and other comple

Re: [HACKERS] Extending varlena

2008-08-18 Thread Josh Berkus
Andrew, > I always find these requests puzzling. Is it really useful to store the > data for a jpeg, video file or a 10GB tar ball in a database column? Some people find it useful. Because LOs are actually easier to manage in PG than in most other DBMSes, right now that's a significant source

Re: [HACKERS] Extending varlena

2008-08-18 Thread Andrew Chernow
David Fetter wrote: On Mon, Aug 18, 2008 at 06:09:13PM -0400, Andrew Chernow wrote: David Fetter wrote: Folks, As the things stored in databases grow, we're going to start needing to think about database objects that 4 bytes of size can't describe. People are already storing video in lo and b

Re: [HACKERS] Extending varlena

2008-08-18 Thread David Fetter
On Mon, Aug 18, 2008 at 06:09:13PM -0400, Andrew Chernow wrote: > David Fetter wrote: >> Folks, >> >> As the things stored in databases grow, we're going to start >> needing to think about database objects that 4 bytes of size can't >> describe. People are already storing video in lo and bytea fie

Re: [HACKERS] Extending varlena

2008-08-18 Thread Andrew Chernow
David Fetter wrote: Folks, As the things stored in databases grow, we're going to start needing to think about database objects that 4 bytes of size can't describe. People are already storing video in lo and bytea fields. To date, the sizes of media files have never trended downward. I alway

Re: [HACKERS] Extending varlena

2008-08-18 Thread David Fetter
On Mon, Aug 18, 2008 at 04:22:56PM -0400, Tom Lane wrote: > David Fetter <[EMAIL PROTECTED]> writes: > > What would need to happen for the next jump up from where varlena > > is now, to 8 bytes? > > Dealing with upwards-of-4GB blobs as single Datums isn't remotely > sane, and won't become so in th

Re: [HACKERS] Extending varlena

2008-08-18 Thread Tom Lane
David Fetter <[EMAIL PROTECTED]> writes: > What would need to happen for the next jump up from where varlena is > now, to 8 bytes? Dealing with upwards-of-4GB blobs as single Datums isn't remotely sane, and won't become so in the near (or even medium) future. So I don't see the point of doing all

[HACKERS] Extending varlena

2008-08-18 Thread David Fetter
Folks, As the things stored in databases grow, we're going to start needing to think about database objects that 4 bytes of size can't describe. People are already storing video in lo and bytea fields. To date, the sizes of media files have never trended downward. What would need to happen for t