Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-06-10 Thread via GitHub
alamb commented on PR #16086: URL: https://github.com/apache/datafusion/pull/16086#issuecomment-2961173895 FWIW @phillipleblanc also hit this as well in SpiceAI. See this for more details - https://github.com/spiceai/spiceai/pull/6178 -- This is an automated message from the Apache Gi

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-06-10 Thread via GitHub
xudong963 commented on PR #16086: URL: https://github.com/apache/datafusion/pull/16086#issuecomment-2961055915 > Great to hear! Sorry for the inconvenience... No problem! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-06-10 Thread via GitHub
adriangb commented on PR #16086: URL: https://github.com/apache/datafusion/pull/16086#issuecomment-2961049931 Great to hear! Sorry for the inconvenience... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-06-10 Thread via GitHub
xudong963 commented on PR #16086: URL: https://github.com/apache/datafusion/pull/16086#issuecomment-2961047647 Fyi, we've upgraded to DF47 with the fix successfully. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-22 Thread via GitHub
alamb commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2103114864 ## datafusion/core/src/datasource/physical_plan/parquet.rs: ## @@ -848,6 +864,88 @@ mod tests { assert_eq!(read.len(), 0); } +#[tokio::test] +

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-21 Thread via GitHub
alamb commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2100929304 ## datafusion/datasource-parquet/src/opener.rs: ## @@ -178,7 +182,7 @@ impl FileOpener for ParquetOpener { // Build predicates for this specific file

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-21 Thread via GitHub
etseidl commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2100927019 ## datafusion/datasource-parquet/src/opener.rs: ## @@ -178,7 +182,7 @@ impl FileOpener for ParquetOpener { // Build predicates for this specific file

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-21 Thread via GitHub
adriangb commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2100914890 ## datafusion/datasource-parquet/src/opener.rs: ## @@ -178,7 +182,7 @@ impl FileOpener for ParquetOpener { // Build predicates for this specific file

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-21 Thread via GitHub
alamb commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2100907909 ## datafusion/datasource-parquet/src/opener.rs: ## @@ -178,7 +182,7 @@ impl FileOpener for ParquetOpener { // Build predicates for this specific file

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-21 Thread via GitHub
adriangb commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2100900247 ## datafusion/datasource-parquet/src/opener.rs: ## @@ -178,7 +182,7 @@ impl FileOpener for ParquetOpener { // Build predicates for this specific file

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-21 Thread via GitHub
etseidl commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2100826502 ## datafusion/datasource-parquet/src/opener.rs: ## @@ -178,7 +182,7 @@ impl FileOpener for ParquetOpener { // Build predicates for this specific file

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-21 Thread via GitHub
alamb merged PR #16086: URL: https://github.com/apache/datafusion/pull/16086 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-21 Thread via GitHub
alamb commented on PR #16086: URL: https://github.com/apache/datafusion/pull/16086#issuecomment-2897860575 Thanks again @xudong963 and @adriangb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-21 Thread via GitHub
xudong963 commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2099949463 ## datafusion/core/src/datasource/physical_plan/parquet.rs: ## @@ -848,6 +864,88 @@ mod tests { assert_eq!(read.len(), 0); } +#[tokio::test]

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-21 Thread via GitHub
xudong963 commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2099919691 ## datafusion/datasource-parquet/src/opener.rs: ## @@ -55,8 +55,9 @@ pub(super) struct ParquetOpener { pub limit: Option, /// Optional predicate to app

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-21 Thread via GitHub
xudong963 commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2099919691 ## datafusion/datasource-parquet/src/opener.rs: ## @@ -55,8 +55,9 @@ pub(super) struct ParquetOpener { pub limit: Option, /// Optional predicate to app

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-20 Thread via GitHub
alamb commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2098220180 ## datafusion/datasource-parquet/src/opener.rs: ## @@ -178,7 +182,7 @@ impl FileOpener for ParquetOpener { // Build predicates for this specific file

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-20 Thread via GitHub
adriangb commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2098086027 ## datafusion/datasource-parquet/src/opener.rs: ## @@ -178,7 +182,7 @@ impl FileOpener for ParquetOpener { // Build predicates for this specific file

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-20 Thread via GitHub
alamb commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2098034125 ## datafusion/datasource-parquet/src/opener.rs: ## @@ -178,7 +182,7 @@ impl FileOpener for ParquetOpener { // Build predicates for this specific file

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-19 Thread via GitHub
adriangb commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2095856518 ## datafusion/core/src/datasource/physical_plan/parquet.rs: ## @@ -156,10 +162,20 @@ mod tests { source = source .with_pushd

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-19 Thread via GitHub
adriangb commented on PR #16086: URL: https://github.com/apache/datafusion/pull/16086#issuecomment-2891268458 > Thank you @adriangb , I'll review the PR tomorrow carefully and try to verify if this is related to the issue we encountered during upgrading DF47 Thank you 🙏🏻 and sorry for

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-19 Thread via GitHub
adriangb commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2095854477 ## datafusion/core/src/datasource/physical_plan/parquet.rs: ## @@ -156,10 +162,20 @@ mod tests { source = source .with_pushd

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-19 Thread via GitHub
xudong963 commented on PR #16086: URL: https://github.com/apache/datafusion/pull/16086#issuecomment-2891263368 Thank you @adriangb , I'll review the PR tomorrow carefully and try to verify if this is related to the issue we encountered during upgrading DF47 -- This is an automated message

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-19 Thread via GitHub
adriangb commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2094817127 ## datafusion/core/src/datasource/physical_plan/parquet.rs: ## @@ -848,6 +848,68 @@ mod tests { assert_eq!(read.len(), 0); } +#[tokio::test] +

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-18 Thread via GitHub
adriangb commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2094817127 ## datafusion/core/src/datasource/physical_plan/parquet.rs: ## @@ -848,6 +848,68 @@ mod tests { assert_eq!(read.len(), 0); } +#[tokio::test] +

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-18 Thread via GitHub
adriangb commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2094868000 ## datafusion/core/src/datasource/physical_plan/parquet.rs: ## @@ -848,6 +848,68 @@ mod tests { assert_eq!(read.len(), 0); } +#[tokio::test] +

Re: [PR] Revert use file schema in parquet pruning [datafusion]

2025-05-18 Thread via GitHub
adriangb commented on code in PR #16086: URL: https://github.com/apache/datafusion/pull/16086#discussion_r2094817127 ## datafusion/core/src/datasource/physical_plan/parquet.rs: ## @@ -848,6 +848,68 @@ mod tests { assert_eq!(read.len(), 0); } +#[tokio::test] +