Re: Detoasting optionally to make Explain-Analyze less misleading

2024-07-08 Thread Michael Christofides
> I'm not sure there's a need for it. When a query runs under > auto_explain, the output values will be sent to the client, > so those cycles should be accounted for anyway, no? > Yes, great point, the total duration reported by auto_explain includes it. Explicit serialization stats might still b

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-07-08 Thread Tom Lane
Michael Christofides writes: > While looking to add support for SERIALIZE in an explain visualisation tool > I work on, I realised there isn't yet an equivalent auto_explain parameter > for SERIALIZE. I'm not sure if this is a deliberate omission (perhaps for a > similar reason planning time is no

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-07-08 Thread Michael Christofides
> > So thanks again! and this will really help a lot of people. I'd like to echo this thanks to you all. While looking to add support for SERIALIZE in an explain visualisation tool I work on, I realised there isn't yet an equivalent auto_explain parameter for SERIALIZE. I'm not sure if this is a

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-04-10 Thread stepan rutz
First of all thanks for bringing this Feature to PostgreSQL. From a regular-user perspective (not everyone is a Pro) it is very misleading that ANALYZE doesn't do what it suggests it does. To run the query into some kind of /dev/null type of destination is feasible and that is what people end up d

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-04-10 Thread Tom Lane
Matthias van de Meent writes: > Upthread at [0], Stepan mentioned that we should default to SERIALIZE > when ANALYZE is enabled. I suspect a patch in that direction would > primarily contain updates in the test plan outputs, but I've not yet > worked on that. > Does anyone else have a strong opin

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-04-10 Thread Matthias van de Meent
On Wed, 3 Apr 2024 at 23:50, Tom Lane wrote: > I've pushed this after a good deal of cosmetic polishing -- for > example, I spent some effort on making serializeAnalyzeReceive > look as much like printtup as possible, in hopes of making it > easier to keep the two functions in sync in future. Upt

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-04-04 Thread Matthias van de Meent
On Wed, 3 Apr 2024 at 23:50, Tom Lane wrote: > > Matthias van de Meent writes: >> On Tue, 2 Apr 2024 at 17:47, Tom Lane wrote: >>> IIUC, it's not possible to use the SERIALIZE option when explaining >>> CREATE TABLE AS, because you can't install the instrumentation tuple >>> receiver when the In

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-04-03 Thread Tom Lane
Matthias van de Meent writes: > On Tue, 2 Apr 2024 at 17:47, Tom Lane wrote: >> IIUC, it's not possible to use the SERIALIZE option when explaining >> CREATE TABLE AS, because you can't install the instrumentation tuple >> receiver when the IntoRel one is needed. I think that's fine because >> n

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-04-03 Thread Matthias van de Meent
On Tue, 2 Apr 2024 at 17:47, Tom Lane wrote: > > Matthias van de Meent writes: > > Attached is v9, which is rebased on master to handle 24eebc65's > > changed signature of pq_sendcountedtext. > > It now also includes autocompletion, and a second patch which adds > > documentation to give users in

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-04-02 Thread Tom Lane
Matthias van de Meent writes: > Attached is v9, which is rebased on master to handle 24eebc65's > changed signature of pq_sendcountedtext. > It now also includes autocompletion, and a second patch which adds > documentation to give users insights into this new addition to > EXPLAIN. I took a quic

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-03-12 Thread Matthias van de Meent
On Mon, 26 Feb 2024 at 21:54, stepan rutz wrote: > > Hi Matthias, thanks for picking it up. I still believe this is valuable > to a lot of people out there. Thanks for dealing with my proposal. > Matthias, Tom, Tomas everyone. > > Two (more or less) controversial remarks from side. > > 1. Actually

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-02-26 Thread stepan rutz
Hi Matthias, thanks for picking it up. I still believe this is valuable to a lot of people out there. Thanks for dealing with my proposal. Matthias, Tom, Tomas everyone. Two (more or less) controversial remarks from side. 1. Actually serialization should be the default for "analyze" in explain,

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-02-26 Thread Matthias van de Meent
Hi, I've taken the liberty to update this patch, and register it in the commitfest app to not lose track of progress [0]. The attached v8 patch measures scratch memory allocations (with MEMORY option), total time spent in serialization (with TIMING on, measures are inclusive of unseparated memcpy

Re: Detoasting optionally to make Explain-Analyze less misleading

2023-11-02 Thread Tomas Vondra
On 11/2/23 22:33, Matthias van de Meent wrote: > On Thu, 2 Nov 2023 at 22:25, Tomas Vondra > wrote: >> >> >> >> On 11/2/23 21:02, Matthias van de Meent wrote: >>> On Thu, 2 Nov 2023 at 20:32, Tomas Vondra >>> wrote: On 11/2/23 20:09, stepan rutz wrote: > db1=# explain (analyze, seri

Re: Detoasting optionally to make Explain-Analyze less misleading

2023-11-02 Thread Matthias van de Meent
On Thu, 2 Nov 2023 at 22:25, Tomas Vondra wrote: > > > > On 11/2/23 21:02, Matthias van de Meent wrote: > > On Thu, 2 Nov 2023 at 20:32, Tomas Vondra > > wrote: > >> On 11/2/23 20:09, stepan rutz wrote: > >>> db1=# explain (analyze, serialize) select * from test; > >>>

Re: Detoasting optionally to make Explain-Analyze less misleading

2023-11-02 Thread Tomas Vondra
On 11/2/23 21:02, Matthias van de Meent wrote: > On Thu, 2 Nov 2023 at 20:32, Tomas Vondra > wrote: >> On 11/2/23 20:09, stepan rutz wrote: >>> db1=# explain (analyze, serialize) select * from test; >>> QUERY PLAN >>>

Re: Detoasting optionally to make Explain-Analyze less misleading

2023-11-02 Thread Matthias van de Meent
On Thu, 2 Nov 2023 at 20:32, Tomas Vondra wrote: > On 11/2/23 20:09, stepan rutz wrote: > > db1=# explain (analyze, serialize) select * from test; > > QUERY PLAN > >

Re: Detoasting optionally to make Explain-Analyze less misleading

2023-11-02 Thread stepan rutz
Hi Thomas, indeed by doing less the code also becomes trivial and ExplainPropertyInteger can be used as a oneliner. My intention was to actually get the realistic payload-bytes from the wire-protocol counted by the serialization option. I am also adding the protocol bits and the length of the da

Re: Detoasting optionally to make Explain-Analyze less misleading

2023-11-02 Thread Tomas Vondra
On 11/2/23 20:09, stepan rutz wrote: > Hi Thomas, > > you are right of course. Thanks! > > I have attached a new version of the patch that supports the syntax like > suggested. The previous patch was insonsistent in style indeed. > > explain (analyze, serialize) > > and > > explain (analyze

Re: Detoasting optionally to make Explain-Analyze less misleading

2023-11-02 Thread stepan rutz
Hi Thomas, you are right of course. Thanks! I have attached a new version of the patch that supports the syntax like suggested. The previous patch was insonsistent in style indeed. explain (analyze, serialize) and explain (analyze, serialize binary) That doesn't make too much of a difference

Re: Detoasting optionally to make Explain-Analyze less misleading

2023-11-02 Thread Tomas Vondra
Hi, On 9/15/23 22:09, stepan rutz wrote: > Hi, > > please see a revised version yesterday's mail. The patch attached now > provides the following: > > EXPLAIN(ANALYZE,SERIALIZE) > > and > > EXPLAIN(ANALYZE,SERIALIZEBINARY) > I haven't looked at the patch in detail yet, but this option name l

Re: Detoasting optionally to make Explain-Analyze less misleading

2023-09-15 Thread stepan rutz
Hi, please see a revised version yesterday's mail. The patch attached now provides the following: EXPLAIN(ANALYZE,SERIALIZE) and EXPLAIN(ANALYZE,SERIALIZEBINARY) and timing output. Both options perform the serialization during analyze and provide an additional output in the plan like this:

Re: Detoasting optionally to make Explain-Analyze less misleading

2023-09-14 Thread stepan rutz
Hi Tom, Hi Matthias, you are right of course. I have looked at the code from printtup.c and made a new version of the patch. Thanks for the MemoryContextReset hint too (@Matthias) This time is called  EXPLAIN(ANALYZE,SERIALIZE) (hey, it also sounds nicer phonetically) If the option SERIALIZE i

Re: Detoasting optionally to make Explain-Analyze less misleading

2023-09-13 Thread Jehan-Guillaume de Rorthais
Hi Stepan & all, On Tue, 12 Sep 2023 17:16:00 +0200 stepan rutz wrote: ... > Attached a new patch. Hoping for feedback, Nice addition to EXPLAIN! On the feature front, what about adding the actual detoasting/serializing time in the explain output? That could be: => explain (analyze,serial

Re: Detoasting optionally to make Explain-Analyze less misleading

2023-09-12 Thread Tom Lane
Matthias van de Meent writes: > Hmm, maybe we should measure the overhead of serializing the tuples instead. > The difference between your patch and "serializing the tuples, but not > sending them" is that serializing also does the detoasting, but also > includes any time spent in the serializatio

Re: Detoasting optionally to make Explain-Analyze less misleading

2023-09-12 Thread stepan rutz
Hi Matthias, thanks for your feedback. I wasn't sure on the memory-contexts. I was actually also unsure on whether the array   TupleTableSlot.tts_isnull is also set up correctly by the previous call to slot_getallattrs(slot). This would allow to get rid of even more code from this patch, which

Re: Detoasting optionally to make Explain-Analyze less misleading

2023-09-12 Thread Matthias van de Meent
On Tue, 12 Sept 2023 at 12:56, stepan rutz wrote: > > Hi, > > I have fallen into this trap and others have too. If you run > EXPLAIN(ANALYZE) no de-toasting happens. This makes query-runtimes > differ a lot. The bigger point is that the average user expects more > from EXPLAIN(ANALYZE) than what i

Detoasting optionally to make Explain-Analyze less misleading

2023-09-12 Thread stepan rutz
Hi, I have fallen into this trap and others have too. If you run EXPLAIN(ANALYZE) no de-toasting happens. This makes query-runtimes differ a lot. The bigger point is that the average user expects more from EXPLAIN(ANALYZE) than what it provides. This can be suprising. You can force detoasting dur