2010YOUY01 commented on PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#issuecomment-2990228423
> > 'here is the API changes that might break your system during upgrades',
and this PR is like a new feature you might want to try in the new release --
do we have a separate pla
xudong963 commented on PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#issuecomment-2990161090
> 'here is the API changes that might break your system during upgrades',
and this PR is like a new feature you might want to try in the new release --
do we have a separate place
2010YOUY01 merged PR #16268:
URL: https://github.com/apache/datafusion/pull/16268
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@dat
2010YOUY01 commented on PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#issuecomment-2988025104
> It's better to mention the option in DF49 upgrade doc
Thank you for the reminder @xudong963 .
I got a question: as I understand it, upgrade guide is like 'here is th
ding-young commented on code in PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#discussion_r2156147905
##
datafusion/physical-plan/src/sorts/sort.rs:
##
@@ -258,6 +259,8 @@ impl ExternalSorter {
batch_size: usize,
sort_spill_reservation_bytes: u
ding-young commented on code in PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#discussion_r2156147684
##
datafusion/physical-plan/src/joins/sort_merge_join.rs:
##
@@ -1324,6 +1326,8 @@ impl Stream for SortMergeJoinStream {
impl SortMergeJoinStream {
#[allo
2010YOUY01 commented on code in PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#discussion_r2154095345
##
datafusion/physical-plan/src/joins/sort_merge_join.rs:
##
@@ -1324,6 +1326,7 @@ impl Stream for SortMergeJoinStream {
impl SortMergeJoinStream {
#[allo
2010YOUY01 closed pull request #16268: Add compression option to SpillManager
URL: https://github.com/apache/datafusion/pull/16268
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment
2010YOUY01 commented on PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#issuecomment-2983191181
close and reopen to trigger CI again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go t
2010YOUY01 closed pull request #16268: Add compression option to SpillManager
URL: https://github.com/apache/datafusion/pull/16268
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment
ding-young commented on PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#issuecomment-2982326469
Currently CI fails, but I think that is due to change introduced in another
pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please l
ding-young commented on code in PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#discussion_r2153450196
##
datafusion/common/src/config.rs:
##
@@ -274,6 +276,61 @@ config_namespace! {
}
}
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
+pub enum Spi
ding-young commented on code in PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#discussion_r2153447729
##
datafusion/physical-plan/src/spill/spill_manager.rs:
##
@@ -44,16 +44,23 @@ pub struct SpillManager {
schema: SchemaRef,
/// Number of batches to b
ding-young commented on code in PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#discussion_r2153442938
##
datafusion/core/tests/memory_limit/mod.rs:
##
@@ -630,6 +635,77 @@ async fn test_disk_spill_limit_not_reached() -> Result<()>
{
Ok(())
}
+/// Extern
ding-young commented on code in PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#discussion_r2151623165
##
datafusion/common/src/config.rs:
##
@@ -274,6 +276,60 @@ config_namespace! {
}
}
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum SpillCompres
ding-young commented on code in PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#discussion_r2149653552
##
datafusion/common/src/config.rs:
##
@@ -330,6 +386,13 @@ config_namespace! {
/// the new schema verification step.
pub skip_physical_aggreg
2010YOUY01 commented on code in PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#discussion_r2141401180
##
datafusion/common/src/config.rs:
##
@@ -274,6 +276,60 @@ config_namespace! {
}
}
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum SpillCompres
xudong963 commented on code in PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#discussion_r2139649288
##
datafusion/common/src/config.rs:
##
@@ -330,6 +386,13 @@ config_namespace! {
/// the new schema verification step.
pub skip_physical_aggrega
ding-young commented on PR #16268:
URL: https://github.com/apache/datafusion/pull/16268#issuecomment-2961345444
This is ready for review :) @2010YOUY01
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
ding-young opened a new pull request, #16268:
URL: https://github.com/apache/datafusion/pull/16268
## Which issue does this PR close?
- Closes #16130 .
## TODO
- [ ] add test for compression in spill file
- [ ] refine arg names
- [ ] check config docs
20 matches
Mail list logo
