Optimize object lookup in REST catalog

2024-12-04 Thread Vladimir Ozerov
Hi, Consider the query “SELECT * FROM t”. The query engine needs to resolve the object “t” during semantic analysis. In Iceberg, this could be a table, a view, a materialized view (soon). Currently, the engine has to guess object type via multiple REST calls, e.g loadTable -> loadView. This incr

Re: Overwrite old properties on table replace with REST catalog

2024-12-04 Thread Vladimir Ozerov
Hi, Haizhou. Thanks for bringing this up. Yes, this problem appears similar to what I explained but with a slightly different manifestation. What is funny is that you observed it when doing REST integration tests for Spark, and I was doing the same for Trino (the fully-fledged integration tests ar

Re: [Proposal] Automating the PyIceberg Release Process

2024-12-04 Thread Fokko Driesprong
Hey Kevin, I agree with your reasoning. Having reproducible builds would be nice, but it doesn't seem very popular in Pythonland. For example, cibuildwheel that we use, doesn't mention anything about reproducibility. The steps that your suggestion make sense