alamb commented on code in PR #14745:
URL: https://github.com/apache/datafusion/pull/14745#discussion_r1963434459
##########
.github/workflows/extended.yml:
##########
@@ -39,43 +39,54 @@ jobs:
linux-build-lib:
name: linux build test
runs-on: ubuntu-latest
- container:
- image: amd64/rust
steps:
- uses: actions/checkout@v4
- - name: Setup Rust toolchain
- uses: ./.github/actions/setup-builder
with:
- rust-version: stable
+ submodules: true
+ fetch-depth: 1
+ - name: Install Rust
+ run: |
+ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s --
-y
+ source $HOME/.cargo/env
+ rustup default stable
+ - name: Install Protobuf Compiler
+ run: sudo apt-get install -y protobuf-compiler
- name: Prepare cargo build
run: |
cargo check --profile ci --all-targets
cargo clean
-# # Run extended tests (with feature 'extended_tests')
-# # Disabling as it is running out of disk space
-# # see https://github.com/apache/datafusion/issues/14576
-# linux-test-extended:
-# name: cargo test 'extended_tests' (amd64)
-# needs: linux-build-lib
-# runs-on: ubuntu-latest
-# container:
-# image: amd64/rust
-# steps:
-# - uses: actions/checkout@v4
-# with:
-# submodules: true
-# fetch-depth: 1
-# - name: Setup Rust toolchain
-# uses: ./.github/actions/setup-builder
-# with:
-# rust-version: stable
-# - name: Run tests (excluding doctests)
-# run: cargo test --profile ci --exclude datafusion-examples --exclude
datafusion-benchmarks --workspace --lib --tests --bins --features
avro,json,backtrace,extended_tests
-# - name: Verify Working Directory Clean
-# run: git diff --exit-code
-# - name: Cleanup
-# run: cargo clean
+ # Run extended tests (with feature 'extended_tests')
+ linux-test-extended:
+ name: cargo test 'extended_tests' (amd64)
+ needs: linux-build-lib
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ submodules: true
+ fetch-depth: 1
+ - name: Free Disk Space (Ubuntu)
+ uses:
jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be
Review Comment:
I am a little worried about using a third-party action
It seems from the soure we could put a few shell command and get the same
effect:
https://github.com/jlumbroso/free-disk-space/blob/54081f138730dfa15788a46383842cd2f914a1be/action.yml#L149
https://github.com/jlumbroso/free-disk-space/blob/54081f138730dfa15788a46383842cd2f914a1be/action.yml#L161-L162
https://github.com/jlumbroso/free-disk-space/blob/54081f138730dfa15788a46383842cd2f914a1be/action.yml#L175-L185
##########
datafusion/core/tests/memory_limit/memory_limit_validation/sort_mem_validation.rs:
##########
@@ -67,10 +69,35 @@ fn sort_with_mem_limit_2_cols_2_runner() {
spawn_test_process("sort_with_mem_limit_2_cols_2");
}
+/// `spawn_test_process` might trigger multiple recompilations and the test
binary
+/// size might grow indefinitely. This initializer ensures recompilation is
only done
+/// once and the target size is bounded.
+///
+/// TODO: This is a hack, can be cleaned up if we have a better way to let
multiple
+/// test cases run in different processes (instead of different threads by
default)
+fn init_once() {
Review Comment:
I don't undersrtand how this avoids recompilation
It seems like recompilation would happen if the options / features were
different
##########
.github/workflows/extended.yml:
##########
@@ -95,7 +106,7 @@ jobs:
- name: Run tests
run: |
cd datafusion
- cargo test --profile ci --exclude datafusion-examples --exclude
datafusion-benchmarks --exclude datafusion-sqllogictest --workspace --lib
--tests --features=force_hash_collisions,avro,extended_tests
+ cargo test --profile ci --exclude datafusion-examples --exclude
datafusion-benchmarks --exclude datafusion-sqllogictest --workspace --lib
--tests --features=force_hash_collisions,avro
Review Comment:
I agree the fact that the flag `extended_tests` and the workflow is named
`extended` is quite confusing
Maybe as a follow on PR we can rename the `extended_test` flag somthing
different like `extra_tests` or `extended_suite` 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
