Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Tom Lane
Andrew Dunstan writes: > On 05/07/2014 04:13 PM, Tom Lane wrote: >> A README file would be better, >> perhaps, but there's not a specific directory associated with the jsonb >> code; so I think this sort of info belongs either in jsonb.h or in the >> file header comment for jsonb_gin.c. > Is ther

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Andrew Dunstan
On 05/07/2014 04:13 PM, Tom Lane wrote: A README file would be better, perhaps, but there's not a specific directory associated with the jsonb code; so I think this sort of info belongs either in jsonb.h or in the file header comment for jsonb_gin.c. Is there any reason we couldn't have a R

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Tom Lane
Peter Geoghegan writes: > On Wed, May 7, 2014 at 12:27 PM, Tom Lane wrote: >> The jsonb_ops storage format for values is inherently lossy, because it >> cannot distinguish the string values "n", "t", "f" from JSON null or >> boolean true, false respectively; nor does it know the difference betwee

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Tom Lane
Peter Geoghegan writes: > On Wed, May 7, 2014 at 12:08 PM, Tom Lane wrote: >> Is this an obsolete speculation about writing jsonb_hash_ops, or is there >> still something worth preserving there? If so, what? > This is not obsolete. It would equally apply to a GiST opclass that > wanted to suppo

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 12:27 PM, Tom Lane wrote: > The jsonb_ops storage format for values is inherently lossy, because it > cannot distinguish the string values "n", "t", "f" from JSON null or > boolean true, false respectively; nor does it know the difference between > a number and a string cont

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Tom Lane
And while I'm looking at it ... The jsonb_ops storage format for values is inherently lossy, because it cannot distinguish the string values "n", "t", "f" from JSON null or boolean true, false respectively; nor does it know the difference between a number and a string containing digits. This appe

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 12:08 PM, Tom Lane wrote: > * Jsonb Keys and string array elements are treated equivalently when > * serialized to text index storage. One day we may wish to create an opclass > * that only indexes values, but for now keys and values are stored in GIN > * indexes in a w

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Tom Lane
btw ... in jsonb.h there is this comment: * Jsonb Keys and string array elements are treated equivalently when * serialized to text index storage. One day we may wish to create an opclass * that only indexes values, but for now keys and values are stored in GIN * indexes in a way that doesn't

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Andrew Dunstan
On 05/07/2014 02:52 PM, Andres Freund wrote: On 2014-05-07 14:48:51 -0400, Tom Lane wrote: Andres Freund writes: * The jentry representation should be changed so it's possible to get the type of a entry without checking individual types. That'll make code like findJsonbValueFromSuperHea

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Andres Freund
On 2014-05-07 14:48:51 -0400, Tom Lane wrote: > Andres Freund writes: > > * The jentry representation should be changed so it's possible to get the > > type > > of a entry without checking individual types. That'll make code like > > findJsonbValueFromSuperHeader() (well, whatever you've rena

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Tom Lane
Andres Freund writes: > * The jentry representation should be changed so it's possible to get the type > of a entry without checking individual types. That'll make code like > findJsonbValueFromSuperHeader() (well, whatever you've renamed it to) > much easier to read. Preferrably so it an ha

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Andres Freund
On 2014-05-07 10:55:28 -0700, Peter Geoghegan wrote: > On Wed, May 7, 2014 at 4:40 AM, Andres Freund wrote: > > * The jentry representation should be changed so it's possible to get the > > type > > of a entry without checking individual types. That'll make code like > > findJsonbValueFromSup

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Peter Geoghegan
On Wed, May 7, 2014 at 4:40 AM, Andres Freund wrote: > * Imo we need space in jsonb ondisk values to indicate a format > version. We won't fully get this right. That would be nice. > * The jentry representation should be changed so it's possible to get the type > of a entry without checking

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Andres Freund
On 2014-05-07 09:44:36 -0400, Tom Lane wrote: > Magnus Hagander writes: > > On Wed, May 7, 2014 at 2:56 PM, Andres Freund wrote: > >> Well, I guess it depends on what we define 'beta1' to be. Imo evaluating > >> problematic pieces of new code, locating unfinished pieces is part of > >> that. I don

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Tom Lane
Magnus Hagander writes: > On Wed, May 7, 2014 at 2:56 PM, Andres Freund wrote: >> Well, I guess it depends on what we define 'beta1' to be. Imo evaluating >> problematic pieces of new code, locating unfinished pieces is part of >> that. I don't see much point in forbidding incompatible changes in

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Magnus Hagander
On Wed, May 7, 2014 at 3:04 PM, Andres Freund wrote: > On 2014-05-07 15:00:01 +0200, Magnus Hagander wrote: > > On Wed, May 7, 2014 at 2:56 PM, Andres Freund >wrote: > > > > > On 2014-05-07 08:50:33 -0400, Robert Haas wrote: > > > > On Wed, May 7, 2014 at 7:40 AM, Andres Freund > > > wrote: > >

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Andres Freund
On 2014-05-07 15:00:01 +0200, Magnus Hagander wrote: > On Wed, May 7, 2014 at 2:56 PM, Andres Freund wrote: > > > On 2014-05-07 08:50:33 -0400, Robert Haas wrote: > > > On Wed, May 7, 2014 at 7:40 AM, Andres Freund > > wrote: > > > > I don't think it's likely that beta1 will be binary compatible

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Magnus Hagander
On Wed, May 7, 2014 at 2:56 PM, Andres Freund wrote: > On 2014-05-07 08:50:33 -0400, Robert Haas wrote: > > On Wed, May 7, 2014 at 7:40 AM, Andres Freund > wrote: > > > I don't think it's likely that beta1 will be binary compatible with the > > > final version at this point. > > > > I rather thin

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Andres Freund
On 2014-05-07 08:50:33 -0400, Robert Haas wrote: > On Wed, May 7, 2014 at 7:40 AM, Andres Freund wrote: > > I don't think it's likely that beta1 will be binary compatible with the > > final version at this point. > > I rather think we're not ready for beta1 at this point (but I expect > to lose t

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Magnus Hagander
On Wed, May 7, 2014 at 1:20 PM, Heikki Linnakangas wrote: > So, apart from cleaning up the code, we really need to take a close look > at the on-disk format now. The code can be cleaned up later, too, but we're > going to be stuck with the on-disk format forever, so it's critical to get > that rig

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Robert Haas
On Wed, May 7, 2014 at 7:40 AM, Andres Freund wrote: >> I'm going to proceed refactoring those things, which will change the on-disk >> format. It's late in the release cycle - these things really should've been >> cleaned up earlier - but it's important to get the on-disk format right. >> Shout i

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Andres Freund
Hi, On 2014-05-07 14:20:19 +0300, Heikki Linnakangas wrote: > So, apart from cleaning up the code, we really need to take a close look at > the on-disk format now. The code can be cleaned up later, too, but we're > going to be stuck with the on-disk format forever, so it's critical to get > that r

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Michael Paquier
On Wed, May 7, 2014 at 8:20 PM, Heikki Linnakangas wrote: > So, apart from cleaning up the code, we really need to take a close look at > the on-disk format now. The code can be cleaned up later, too, but we're > going to be stuck with the on-disk format forever, so it's critical to get > that rig

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Heikki Linnakangas
So, apart from cleaning up the code, we really need to take a close look at the on-disk format now. The code can be cleaned up later, too, but we're going to be stuck with the on-disk format forever, so it's critical to get that right. First, a few observations: * JENTRY_ISFIRST is redundant.

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-07 Thread Heikki Linnakangas
On 05/06/2014 11:30 PM, Peter Geoghegan wrote: On Tue, May 6, 2014 at 12:48 PM, Andres Freund wrote: Enthusiatically seconded. I've asked for that about three times without much success. If it had been my decision the patch wouldn't have been merged without that and other adjustments. I'm a

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-06 Thread Peter Geoghegan
On Tue, May 6, 2014 at 5:13 PM, Andres Freund wrote: > If you think the following is a solution of essential complexity in > *new* code for navigating one level down a relatively simple *new* > datastructure - then we have a disconnect that's larger than I am > willing to argue about You omitted

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-06 Thread Andres Freund
On 2014-05-06 15:45:39 -0700, Peter Geoghegan wrote: > I don't really know what to say to that. Lots of code in Postgres is > complicated, especially if you look at one particular function without > some wider context. > Is your objection that the complexity is incidental rather than > essential?

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-06 Thread Peter Geoghegan
On Tue, May 6, 2014 at 3:37 PM, Andres Freund wrote: > That might or might not be true. I don't really remember. Documentation > about the on-disk format is the one thing I am sure about that's not > done. I think it would be best to do that with reference to a concrete example. As I said, I'll w

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-06 Thread Andres Freund
On 2014-05-06 13:30:26 -0700, Peter Geoghegan wrote: > On Tue, May 6, 2014 at 12:48 PM, Andres Freund wrote: > > Enthusiatically seconded. I've asked for that about three times without > > much success. If it had been my decision the patch wouldn't have been > > merged without that and other adj

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-06 Thread Oleg Bartunov
FYI, http://obartunov.livejournal.com/178495.html This is hash based gin opclass for hstore with all operators support. It's pity we had no time to do the same for jsonb, but we may include it and couple of other opclasses to contrib/jsonx. Oleg On Wed, May 7, 2014 at 12:18 AM, Peter Geoghegan

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-06 Thread Peter Geoghegan
On Tue, May 6, 2014 at 12:48 PM, Andres Freund wrote: > Enthusiatically seconded. I've asked for that about three times without much > success. If it had been my decision the patch wouldn't have been merged > without that and other adjustments. I'm almost certain that the only feedback of yours

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-06 Thread Peter Geoghegan
On Tue, May 6, 2014 at 1:06 PM, Bruce Momjian wrote: > I also would like to know what the index-everything hash ops does? Does > it index the keys, values, or both? It indexes both, but it isn't possible to test existence (of a key) with the hash GIN opclass. -- Peter Geoghegan -- Sent via

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-06 Thread Bruce Momjian
On Tue, May 6, 2014 at 09:48:04PM +0200, Andres Freund wrote: > On May 6, 2014 9:30:15 PM CEST, Heikki Linnakangas > wrote: > >I'm reading the new jsonb code, trying to understand the on-disk > >representation. And I cannot make heads or tails of it. > > > >My first entry point was jsonb.h. Jso

Re: [HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-06 Thread Andres Freund
On May 6, 2014 9:30:15 PM CEST, Heikki Linnakangas wrote: >I'm reading the new jsonb code, trying to understand the on-disk >representation. And I cannot make heads or tails of it. > >My first entry point was jsonb.h. Jsonb struct is the on-disk >representation, so I looked at the comments abov

[HACKERS] Wanted: jsonb on-disk representation documentation

2014-05-06 Thread Heikki Linnakangas
I'm reading the new jsonb code, trying to understand the on-disk representation. And I cannot make heads or tails of it. My first entry point was jsonb.h. Jsonb struct is the on-disk representation, so I looked at the comments above that. No help, the comments are useless for getting an overvi