Re: [PR] Enable reading `StringViewArray` by default from Parquet [datafusion]

Mon, 16 Sep 2024 13:59:55 -0700 


alamb commented on PR #12092:
URL: https://github.com/apache/datafusion/pull/12092#issuecomment-2354023193

   I figured out what is going on (different than I thought). I believe 
`StringView::slice()` is quite a bit slower than `StringArray::slice` due to 
the fact it has a buffers field
   
   The query is
   ```sql
   SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\\.)?([^/]+)/.*$', '\\1') 
AS k, AVG(length("Referer")) AS l, COUNT(*) AS c, MIN("Referer")
   FROM hits_partitioned
   WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC 
LIMIT 25;
   ```
   
   Some flamegraphs:
   
![flamegraph-main](https://github.com/user-attachments/assets/740c4dcc-5d0f-4ef1-8dd7-f61ad5459133)
   
![flamegraph-string-view](https://github.com/user-attachments/assets/1dc311d8-55e9-46d3-bdd6-12890fdfd6cd)
   
   ![Screenshot 2024-09-16 at 4 44 50 
PM](https://github.com/user-attachments/assets/551582fa-0744-4c7a-9075-b5b4102c6861)
   
   I will think about the best way to proceed here


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to