Hi all, The Apache Auron (Incubating) community is pleased to announce the new release of Apache Auron (Incubating) 7.0.0.
Auron is dedicated to improving the efficiency and elasticity of different map-reduce engines and provides an elastic, highly efficient service for intermediate data including shuffle data, spilled data, result data, etc. Currently Auron supports Spark full-featured and improves Spark job's performance, stability and elasticity. Download Link: ● https://auron.apache.org/archives/v7.0.0-incubating.html (Starting from v6.0.0, Auron only provides source download because there are many new build options. Users can build with their own environment and options, or build on GitHub Actions by forking the repository.) GitHub Release Tag: ● https://github.com/apache/auron/releases/tag/v7.0.0 Release Notes: ● https://auron.apache.org/archives/v7.0.0-incubating.html Website: ● https://auron.apache.org/ Auron Resources: ● Issue: https://github.com/apache/auron/issues ● Mailing list: [email protected] Best regards, Ruilei Ma On behalf of the Apache Auron(incubating) community Release notes: ## New Features * **AuronConfiguration**: Introduced new configuration system with dynamic default values * **New Modules**: Added auron-core, auron-flink-extension, auron-memmgr, auron-iceberg, and auron-spark-tests modules for better modularity * **Spark UI Integration**: Introduced comprehensive Spark UI support with build info display and metrics * **Extended Native Functions**: Implemented native functions for round, pow/power, lpad, rpad, reverse, initcap, levenshtein, quarter, hour, minute, second, find_in_set, nvl, nvl2, least, greatest, isnan, bround * **Enhanced Expressions**: Native support for EqualNullSafe expression, CollectLimit, monotonically_increasing_id(), and spark_partition_id() * **Columnar Aggregation**: Implemented columnar aggregate buffers for improved performance * **JniBridge & AuronAdaptor**: Introduced for better Java-Rust interoperability ## Improvements * **Performance Optimization**: - Improved SortMergeJoin by moving has_multi_equal out of loop - Optimized join operation to flush in time on duplicated keys * **Enhanced Compatibility**: - Support for Spark 3.5.8 and Spark 4.0.2 - Initial compatibility support for Spark 4.1 - Support for JDK 21 - Upgraded from JUnit 4 to JUnit 5 * **Build & CI**: - Unified GitHub Actions runner image to Ubuntu 24.04 - Standardized Maven build summary output to align with Apache formatting standards - Introduced auron-it for enhanced CI integration testing - Enabled Clippy lint with -D warnings enforcement - Extracted Rust test logic to standalone CI workflow * **Observability**: - Improved Spark UI integration with build info and metrics - Support for DataSourceScan metrics - Enhanced memory logging readability * **Configuration Management**: - Added spark.auron.enable.broadcastexchange and spark.auron.enable.shuffleExchange configs - Added spark.auron.expression.singleChildFallback.enabled config - Introduced SparkAuronConfiguration and deprecated AuronConf * **Dependency Updates**: - Upgraded Spark 3.5.7 to 3.5.8 - Bumped Uniffle from 0.9.2 to 0.10.0 - Bumped Celeborn from 0.6.0 to 0.6.2 - Bumped Maven from 3.9.11 to 3.9.12 * **Memory Management**: - Introduced SparkOnHeapSpillManager for better memory control - Fixed possible deadlock in OnHeapSpillManager * **Code Quality**: - Added Maven checkstyle and scalastyle plugins - Removed "Exec" suffix for physical operators for cleaner naming - Eliminated unwrap usage and enforced Clippy lint - Cleaned up Clippy format/string/condition/logic lints ## Bug Fixes * **Data Correctness**: - Fixed concat_ws null handling - Fixed Casting from Struct and Map to Utf8 not supported - Fixed NULL handling for NOT IN subquery - Fixed isNan semantics aligned with Spark - Fixed initCap semantics aligned with Spark - Fixed StringSplit conversion - Fixed ORC incorrect results from RLEv2 patched base - Fixed ORC check offset overflow - Fixed Generate index out of bounds - Fixed unexpected invocation of UDF * **Type System**: - Fixed type mismatch: cast lpad/rpad len to LongType - Fixed string_lower only supports literal utf8 - Fixed Casting from Array/Map/Struct to Utf8 not supported - Fixed FileScanConfig.projection conversion * **Execution Engine**: - Fixed disable convert BroadcastExchange to native does not take effect - Fixed disable convert ShuffleExchange to native does not take effect - Fixed NativeEmptyExec should fallback on unsupported types - Fixed IllegalArgumentException in buildTimePartExt - Fixed MD5 output back to utf8 - Fixed java.lang.NoClassDefFoundError with shaded-spark * **Performance**: - Fixed SortMergeJoin operation on duplicated keys - Fixed possible deadlock in OnHeapSpillManager * **Ecosystem Integration**: - Fixed data size calculation in Celeborn shuffle writer - Fixed UDAF fallback bug when handling DeclarativeAggregator - Fixed spark_normalize_nan_and_zero None * **Testing**: - Fixed FileNotFoundException in BuildinfoInSparkUISuite - Fixed NativeConvertersSuite cast trim-disable test failures - Fixed and enhanced nvl2 unit test * **Documentation**: - Clarified Docker build instructions - Improved README formatting and grammar - Added contributing guidelines ## Notable Changes * **Module Restructuring**: Refactored JniBridge and AuronConf, moved to auron-core module * **Testing Enhancement**: Added comprehensive correctness testing for Spark 3.3 (Aggregate, Sort, Parquet, Functions, Expressions) * **Docker Support**: Expanded build support to modern Linux distributions and added Azure Linux 3 support * **License Compliance**: Added Apache-2.0 license declaration to all native-engine Cargo.toml files NOTE: This release includes a significant number of new features, performance optimizations, bug fixes, and improvements across modularity, compatibility, observability, and testing. The project continues to enhance its ecosystem integration with Uniffle and Celeborn, while maintaining strong backward compatibility with Spark 3.x and forward compatibility with Spark 4.x. Some minor changes are not included in the above list, please see the commit list for more details. ## What's Changed * [RELEASE] Bump version 7.0.0-SNAPSHOT by @richox in https://github.com/apache/auron/pull/1343 * bump spark-3.5 version to 3.5.7 by @XorSum in https://github.com/apache/auron/pull/1353 * [AURON #1351] Unify GitHub Actions runner image to Ubuntu 24.04 by @merrily01 in https://github.com/apache/auron/pull/1352 * [AURON #1344] Delete unused profile in pom.xml by @guixiaowen in https://github.com/apache/auron/pull/1345 * [AURON #1355] Upgrade actions/cache from v3 to v4 in `build-ce7-release.yml` by @merrily01 in https://github.com/apache/auron/pull/1356 * [AURON-1339] Introduce AuronConfiguration by @Tartarus0zm in https://github.com/apache/auron/pull/1340 * [AURON #1341] Eliminate hardcoded values in the AuronSparkSessionExtension class by @guixiaowen in https://github.com/apache/auron/pull/1342 * [AURON #1358] Rename package name for hadoop-shim module by @Tartarus0zm in https://github.com/apache/auron/pull/1359 * bump maven version to 3.9.11 by @cxzl25 in https://github.com/apache/auron/pull/1350 * Bump serde from 1.0.226 to 1.0.227 by @dependabot[bot] in https://github.com/apache/auron/pull/1357 * Remove the "Exec" suffix for physical operators by @XorSum in https://github.com/apache/auron/pull/1349 * [AURON #1365] Fix artifact name formatting issue with conditional version parameters by @merrily01 in https://github.com/apache/auron/pull/1366 * [AURON #1367] Bump Uniffle from 0.9.2 to 0.10.0 by @merrily01 in https://github.com/apache/auron/pull/1354 * [AURON #1362]Add function description() in ConfigOption by @guixiaowen in https://github.com/apache/auron/pull/1363 * [AURON-1332] Introduce JniBridge and AuronAdaptor for auron-core module by @Tartarus0zm in https://github.com/apache/auron/pull/1333 * fix UDAF fallback bug when handling DeclarativeAggregator by @richox in https://github.com/apache/auron/pull/1369 * Bump sonic-rs from 0.5.4 to 0.5.5 by @dependabot[bot] in https://github.com/apache/auron/pull/1377 * Bump serde from 1.0.227 to 1.0.228 by @dependabot[bot] in https://github.com/apache/auron/pull/1376 * [AURON #1374] Introduce AuronUDFWrapperContext by @Tartarus0zm in https://github.com/apache/auron/pull/1375 * [AURON #1383] fix UT fail by @Tartarus0zm in https://github.com/apache/auron/pull/1384 * [AURON #1385] Move Auron Native lib from spark-extension to auron-core by @Tartarus0zm in https://github.com/apache/auron/pull/1386 * [AURON #1387] Introduce auron-flink-extension module by @Tartarus0zm in https://github.com/apache/auron/pull/1389 * Fix typo in AuronConverters by @guixiaowen in https://github.com/apache/auron/pull/1380 * [AURON #1381] Modify Auron run unit tests by default by @Tartarus0zm in https://github.com/apache/auron/pull/1382 * [AURON #1423]Refactor convertBroadcastExchangeExec to use direct type parameter instead of pattern matching by @guixiaowen in https://github.com/apache/auron/pull/1424 * [AURON #1415]Add version check in auron-build.sh by @guixiaowen in https://github.com/apache/auron/pull/1416 * [AURON #1419]Add neverConvertReasonTag to record the reason for non-conversion. by @guixiaowen in https://github.com/apache/auron/pull/1420 * [AURON#1411] Doc: Update the Uniffle build message in auron-build.sh. by @slfan1989 in https://github.com/apache/auron/pull/1412 * Fix typo in README by @guixiaowen in https://github.com/apache/auron/pull/1425 * [AURON #1421]add flink in reformat by @guixiaowen in https://github.com/apache/auron/pull/1422 * [AURON#1407] Fix CentOS 7 Dockerfile build failure. by @slfan1989 in https://github.com/apache/auron/pull/1408 * [AURON#1396] Upgrade from JUnit 4 to JUnit 5. by @slfan1989 in https://github.com/apache/auron/pull/1397 * Bump Maven Surefire from 3.3.0 to 3.5.3. by @slfan1989 in https://github.com/apache/auron/pull/1400 * [AURON #1378]add spark.auron.enable.broadcastexchange to control BroadcastExchangeExec into Native or not by @guixiaowen in https://github.com/apache/auron/pull/1379 * Bump parking_lot from 0.12.4 to 0.12.5 by @dependabot[bot] in https://github.com/apache/auron/pull/1406 * [AURON#1413] Standardize Maven build summary output to align with Apache formatting standards. by @slfan1989 in https://github.com/apache/auron/pull/1414 * [AURON#1398] Support for Building on JDK 21. by @slfan1989 in https://github.com/apache/auron/pull/1399 * [AURON#1327] Implement native function of `round` by @slfan1989 in https://github.com/apache/auron/pull/1426 * [AURON #1428]Follow #1413 in auron-flink-extension by @guixiaowen in https://github.com/apache/auron/pull/1429 * [AURON #1372]add spark.auron.enable.shuffleExchange to control ShuffleExchangeExec into Native or not by @guixiaowen in https://github.com/apache/auron/pull/1373 * [AURON #1409] Introduce Spark UI for auron by @guixiaowen in https://github.com/apache/auron/pull/1410 * [AURON #1314] Speed up CI by using Apache `closer.lua` syntax for dependency downloads by @merrily01 in https://github.com/apache/auron/pull/1315 * [AURON #1440]Follow #1413 in spark-ui by @guixiaowen in https://github.com/apache/auron/pull/1441 * [AURON #1417] Expand build support to modern Linux distributions. by @slfan1989 in https://github.com/apache/auron/pull/1418 * [AURON #1431] Add EmptyNativeRDD extending NativeRDD by @guixiaowen in https://github.com/apache/auron/pull/1430 * [AURON #1437] Introduce AuronArrowFFIExporter by @Tartarus0zm in https://github.com/apache/auron/pull/1439 * [AURON #1445] Add license check for flink module by @Tartarus0zm in https://github.com/apache/auron/pull/1446 * [AURON #1447] Fix auron-flink-extension module junit dependency conflict with flink by @Tartarus0zm in https://github.com/apache/auron/pull/1448 * [AURON #1449] introduce flink test workflow by @Tartarus0zm in https://github.com/apache/auron/pull/1450 * Bump tokio from 1.47.1 to 1.48.0 by @dependabot[bot] in https://github.com/apache/auron/pull/1453 * [AURON #1454] Fix redundant hyphen in log filename construction by @merrily01 in https://github.com/apache/auron/pull/1455 * [AURON #1434] Use Maven-native version extraction instead of xmllint for better portability by @merrily01 in https://github.com/apache/auron/pull/1435 * [AURON #1456] Introduce SparkAuronConfiguration by @Tartarus0zm in https://github.com/apache/auron/pull/1457 * [AURON #1409] followup to always get active context by @zuston in https://github.com/apache/auron/pull/1433 * [AURON #1461] Align Auron tab style with Spark UI naming conventions by @merrily01 in https://github.com/apache/auron/pull/1462 * [AURON #1458] Centralize supported versions and simplify validation with unified build info output by @merrily01 in https://github.com/apache/auron/pull/1459 * [AURON #1468] Introduce SparkMetricNode by @Tartarus0zm in https://github.com/apache/auron/pull/1469 * [AURON #1476] Remove extra parenthesis in assignment causing invalid Docker build context by @merrily01 in https://github.com/apache/auron/pull/1477 * [AURON #1482] Support native avg result cast to return type by @Tartarus0zm in https://github.com/apache/auron/pull/1483 * Bump tikv-jemalloc-ctl from 0.6.0 to 0.6.1 by @dependabot[bot] in https://github.com/apache/auron/pull/1467 * Bump foldhash from 0.1.5 to 0.2.0 by @dependabot[bot] in https://github.com/apache/auron/pull/1466 * Bump procfs from 0.17.0 to 0.18.0 by @dependabot[bot] in https://github.com/apache/auron/pull/1465 * [AURON #1488] Remove isDriverSide method from JniBridge by @Tartarus0zm in https://github.com/apache/auron/pull/1489 * [AURON #1493] Remove unused getSparkEnvConfAsString method from JniBridge by @Tartarus0zm in https://github.com/apache/auron/pull/1494 * Bump bigdecimal from 0.4.8 to 0.4.9 by @dependabot[bot] in https://github.com/apache/auron/pull/1475 * [AURON #1496] Introduce SparkOnHeapSpillManager by @Tartarus0zm in https://github.com/apache/auron/pull/1497 * Bump tikv-jemallocator from 0.6.0 to 0.6.1 by @dependabot[bot] in https://github.com/apache/auron/pull/1495 * [AURON #1473] Rename module spark-extension-shims-spark3 to spark-extension-shims-spark. by @slfan1989 in https://github.com/apache/auron/pull/1474 * [AURON #1491] Add workflow for paimon #1491 by @guixiaowen in https://github.com/apache/auron/pull/1490 * split memory manager to module auron-memmgr by @richox in https://github.com/apache/auron/pull/1506 * Bump byte-buddy from 1.12.10 to 1.14.11. by @slfan1989 in https://github.com/apache/auron/pull/1505 * [AURON #1498] Enable the JniBridge in the auron-core module by @Tartarus0zm in https://github.com/apache/auron/pull/1499 * [AURON #1507] Enable AuronAdaptor#getAuronUDFWrapperContext for spark by @Tartarus0zm in https://github.com/apache/auron/pull/1508 * [AURON #1509] Deprecated JniBridge and AuronCallNativeWrapper under the spark-extension module by @Tartarus0zm in https://github.com/apache/auron/pull/1510 * [AURON #1525] Fallback outputs the exception stack when the log level is debug by @cxzl25 in https://github.com/apache/auron/pull/1525 * [AURON #1523] Fix CheckStyle Violations in Auron Codebase. by @slfan1989 in https://github.com/apache/auron/pull/1523 * [AURON #1516] Add work flow for jdk 21 #1516 by @guixiaowen in https://github.com/apache/auron/pull/1517 * [AURON #1513] Implement native function of pow/power. by @slfan1989 in https://github.com/apache/auron/pull/1514 * [AURON #1480] Add auron-iceberg project #1480 by @guixiaowen in https://github.com/apache/auron/pull/1486 * [AURON #1526] Add `auron-build.sh` to release workflow trigger paths by @merrily01 in https://github.com/apache/auron/pull/1527 * [AURON #1528] Add `auron-build-info.properties` to `.gitignore` by @cxzl25 in https://github.com/apache/auron/pull/1528 * [AURON #1533] Implement native function of lpad, rpad. by @slfan1989 in https://github.com/apache/auron/pull/1534 * [AURON #1529] Remove obsolete `version` attribute to prevent confusion by @merrily01 in https://github.com/apache/auron/pull/1530 * [AURON #1538] Implement native function of `reverse`. by @slfan1989 in https://github.com/apache/auron/pull/1540 * [AURON #1531] Enable SparkAuronConfiguration and deprecated AuronConf by @Tartarus0zm in https://github.com/apache/auron/pull/1532 * [AURON #1545] Clarify Docker build instructions to reflect supported OS images by @merrily01 in https://github.com/apache/auron/pull/1545 * [AURON #1544] Remove outdated `RELEASES.md` file by @merrily01 in https://github.com/apache/auron/pull/1544 * [AURON #1543] Add Docker build support for Azure Linux 3 by @adityavaish in https://github.com/apache/auron/pull/1542 * [AURON #1549] Implement native function of `initcap`. by @slfan1989 in https://github.com/apache/auron/pull/1550 * [AURON #1571]Implement native function of levenshtein(80) #1571 by @guixiaowen in https://github.com/apache/auron/pull/1573 * [AURON #1518] Implement native function of quarter. by @slfan1989 in https://github.com/apache/auron/pull/1519 * [AURON #1593] Add Spark prefix for log option key by @Tartarus0zm in https://github.com/apache/auron/pull/1594 * [AURON #1520] Implement native function of hour, minute, second. by @slfan1989 in https://github.com/apache/auron/pull/1522 * [AURON #1588]Implement native function of find_in_set(81)，nvl(82), nv2(83) ,least(84), greatest(85), isnan(87) #1588 by @guixiaowen in https://github.com/apache/auron/pull/1585 * [AURON #1442] Fix `BuildinfoInSparkUISuite.scala` failure due to missing `/tmp/spark-events/` directory by @merrily01 in https://github.com/apache/auron/pull/1444 * [AURON #1597] Fix java.lang.NoClassDefFoundError: io/netty/buffer/Unpooled when running with shaded-spark by @ShreyeshArangath in https://github.com/apache/auron/pull/1598 * [AURON #1442] Fix FileNotFoundException in BuildinfoInSparkUISuite by properly configuring temp event log dir by @yew1eb in https://github.com/apache/auron/pull/1609 * [AURON #1613] Introduce AuronExtFunctions by @Tartarus0zm in https://github.com/apache/auron/pull/1614 * chore: Clean up redundant NOTICE entry by @CalvinKirs in https://github.com/apache/auron/pull/1616 * Bump sonic-rs from 0.5.5 to 0.5.6 by @dependabot[bot] in https://github.com/apache/auron/pull/1603 * [AURON #1516][FOLLOWUP] Use JDK 21 for CentOS 7 release by @cxzl25 in https://github.com/apache/auron/pull/1608 * [AURON #1621] Introduces dynamic default value for ConfigOption by @Tartarus0zm in https://github.com/apache/auron/pull/1622 * Bump bytesize from 2.1.0 to 2.2.0 by @dependabot[bot] in https://github.com/apache/auron/pull/1619 * [AURON #1604] Make MD5 output back to utf8 by @Flyangz in https://github.com/apache/auron/pull/1605 * [AURON #1612] Filter out empty build-info values to improve Spark UI display by @merrily01 in https://github.com/apache/auron/pull/1615 * [AURON #1629] Fix out-of-date module name in labeler.yml by @yew1eb in https://github.com/apache/auron/pull/1629 * Bump bytes from 1.10.1 to 1.11.0 by @dependabot[bot] in https://github.com/apache/auron/pull/1634 * [AURON #1631] Updates the labeler workflow to cover the newly added modules in the Auron project by @merrily01 in https://github.com/apache/auron/pull/1632 * [AURON #1602] Implement AuronAdaptor SPI discovery with Spark provider by @yew1eb in https://github.com/apache/auron/pull/1620 * [AURON #1650] Override nodeName for NativeEmptyExec by @yew1eb in https://github.com/apache/auron/pull/1651 * Bump bytesize from 2.2.0 to 2.3.0 by @dependabot[bot] in https://github.com/apache/auron/pull/1648 * Bump lz4_flex from 0.11.5 to 0.12.0 by @dependabot[bot] in https://github.com/apache/auron/pull/1623 * [AURON #1579] Fix and enhance nvl2 unit test by @yew1eb in https://github.com/apache/auron/pull/1635 * [AURON #1617] Fix NativeConvertersSuite cast trim-disable test failures by @yew1eb in https://github.com/apache/auron/pull/1618 * [AURON #1652] Fix: add Spark_ prefix for Hashing Functions in Auron ext functions by @yew1eb in https://github.com/apache/auron/pull/1653 * [AURON #1654] Add AuronQueryTest base class for SQL testing by @yew1eb in https://github.com/apache/auron/pull/1655 * [AURON #1644] Fix build-native.sh ensure native lib cache takes effect by @yew1eb in https://github.com/apache/auron/pull/1645 * [AURON #1537] Fix incorrect FileScanConfig.projection conversion by @richox in https://github.com/apache/auron/pull/1591 * [AURON #1636] Bump orc-rust 0.7.0 by @cxzl25 in https://github.com/apache/auron/pull/1661 * [AURON #1663] Fix: NativeEmptyExec should fallback on unsupported types by @yew1eb in https://github.com/apache/auron/pull/1664 * [AURON #1659] Remove ubuntu-22.04-arm runner by @cxzl25 in https://github.com/apache/auron/pull/1660 * [AURON #1656] Support to disable the scan timestamp for Parquet and ORC formats by @cxzl25 in https://github.com/apache/auron/pull/1657 * Bump bytesize from 2.3.0 to 2.3.1 by @dependabot[bot] in https://github.com/apache/auron/pull/1669 * [AURON #1674] Fix: route Upper to Spark_StringUpper by @yew1eb in https://github.com/apache/auron/pull/1675 * [AURON #1595] Enforce skiptest=false to not miss failing UT by @ShreyeshArangath in https://github.com/apache/auron/pull/1596 * [AURON #1665] Override verboseStringWithOperatorId in NativeFileSourceScanBase by @yew1eb in https://github.com/apache/auron/pull/1643 * [AURON #1668] Remove the usage of BASH associative array for bash 3.x version by @xumingming in https://github.com/apache/auron/pull/1672 * [AURON #1665] Override verboseStringWithOperatorId in NativeFilterBase by @yew1eb in https://github.com/apache/auron/pull/1667 * [AURON #1665] Override verboseStringWithOperatorId in NativeProjectBase by @yew1eb in https://github.com/apache/auron/pull/1666 * [AURON #1625] Fix disable convert BroadcastExchange to native does not take effect by @yew1eb in https://github.com/apache/auron/pull/1627 * [AURON #1625] Fix disable convert ShuffleExchange to native does not take effect by @yew1eb in https://github.com/apache/auron/pull/1628 * Bump log from 0.4.28 to 0.4.29 by @dependabot[bot] in https://github.com/apache/auron/pull/1692 * [AURON #1690] Improving build-native.sh by @yew1eb in https://github.com/apache/auron/pull/1691 * [AURON #1646] `isNan` semantics are aligned with Spark by @ShreyeshArangath in https://github.com/apache/auron/pull/1647 * [AURON #1686]Fix IllegalArgumentException in buildTimePartExt leading to fallback to Spark by @yew1eb in https://github.com/apache/auron/pull/1687 * [AURON #1702] Fix typo in join_hash_map.rs by @xumingming in https://github.com/apache/auron/pull/1703 * [AURON #1720] Exclude temp file: auron-build-info.properties from apache-rat plugin by @xumingming in https://github.com/apache/auron/pull/1721 * [AURON #1727] Fix concat_ws null by @cxzl25 in https://github.com/apache/auron/pull/1728 * [AURON #1713] `dev/reformat` format Rust early by @cxzl25 in https://github.com/apache/auron/pull/1719 * [AURON #1722] Use PR title and description for commit message by @cxzl25 in https://github.com/apache/auron/pull/1723 * [AURON #1708] Fix spark_normalize_nan_and_zero None by @cxzl25 in https://github.com/apache/auron/pull/1709 * [AURON #1730] Fix data size calculation in Celeborn shuffle writer by @DDDominik in https://github.com/apache/auron/pull/1731 * Revert "[AURON #1730] Fix data size calculation in Celeborn shuffle writer" by @richox in https://github.com/apache/auron/pull/1736 * [AURON #1688] Fix type mismatch: cast lpad/rpad len (2nd arg) to LongType by @yew1eb in https://github.com/apache/auron/pull/1689 * [AURON #1711] Fix Casting from Struct to Utf8 not supported by @cxzl25 in https://github.com/apache/auron/pull/1718 * [AURON #1707] Bump Celeborn 0.6.2 by @XorSum in https://github.com/apache/auron/pull/1741 * [AURON #1751] rename scalarTypeSupported -> isTypeSupported by @xumingming in https://github.com/apache/auron/pull/1753 * [AURON #1742] Disable Spark UI in test by @XorSum in https://github.com/apache/auron/pull/1742 * [AURON #1755] Use test-specific Log4j2 config in Maven test runs by @yew1eb in https://github.com/apache/auron/pull/1756 * [AURON #1748] ORC check offset overflow by @cxzl25 in https://github.com/apache/auron/pull/1749 * [AURON #1638] Support scan ORC data using microsecond precision by @cxzl25 in https://github.com/apache/auron/pull/1684 * [AURON #1695] Improve SortMergeJoin: move has_multi_equal out of loop by @xumingming in https://github.com/apache/auron/pull/1696 * [AURON #1680] initCap semantics are aligned with Spark by @yew1eb in https://github.com/apache/auron/pull/1681 * [AURON #1710] Fix string_lower only supports literal utf8 by @cxzl25 in https://github.com/apache/auron/pull/1714 * [AURON #1639] Avoid ByteBuddyAgent inject when class not found by @cxzl25 in https://github.com/apache/auron/pull/1640 * [AURON #1732] Add maven repository by @cxzl25 in https://github.com/apache/auron/pull/1733 * [AURON #1737] fix Generate index out of bounds by @cxzl25 in https://github.com/apache/auron/pull/1738 * [AURON #1712] Fix Casting from Map to Utf8 not supported by @cxzl25 in https://github.com/apache/auron/pull/1744 * [AURON #1725] Support Native CollectLimit by @yew1eb in https://github.com/apache/auron/pull/1726 * [AURON #1764] Fix Javadoc typo in AuronConfiguration: "mapper" → "mapped" by @yew1eb in https://github.com/apache/auron/pull/1765 * [AURON #1766] Improve memory log readability in NativeHelper by @yew1eb in https://github.com/apache/auron/pull/1767 * [AURON #1762] Fix ORC incorrect results from RLEv2 patched base by @dh20 in https://github.com/apache/auron/pull/1768 * [AURON #1771] build-native.sh set -e by @cxzl25 in https://github.com/apache/auron/pull/1772 * [AURON #1785] Rename NativeParquetSinkExec NativeRDD name to NativeRDD.ParquetSink by @yew1eb in https://github.com/apache/auron/pull/1786 * [AURON #1776] Implement native support for EqualNullSafe expression by @yew1eb in https://github.com/apache/auron/pull/1777 * [AURON #1763] Fix unexpected invocation of UDF by @cxzl25 in https://github.com/apache/auron/pull/1787 * [AURON #1794] Fix IDEA import error by using fixed shim module name by @yew1eb in https://github.com/apache/auron/pull/1795 * Bump bigdecimal from 0.4.9 to 0.4.10 by @dependabot[bot] in https://github.com/apache/auron/pull/1798 * [AURON #1745] Introduce auron-spark-tests submodule for correctness testing by @xumingming in https://github.com/apache/auron/pull/1752 * [AURON #1484] Change test-jar dependencies scope to 'test' in spark-extension-shims-spark module by @cxzl25 in https://github.com/apache/auron/pull/1796 * [AURON #1792] Keep the null result in the reverse connection result by @dh20 in https://github.com/apache/auron/pull/1793 * [AURON #1780] Fix the ORC table written by Hive to read null uppercase fields by @dh20 in https://github.com/apache/auron/pull/1781 * [AURON #1803] fix possible deadlock in OnHeapSpillManager by @richox in https://github.com/apache/auron/pull/1804 * [AURON #1822] Add label for auron-spark-tests PRs by @ShreyeshArangath in https://github.com/apache/auron/pull/1823 * [AURON #1760] Enable Dependabot updates for GitHub Actions by @yew1eb in https://github.com/apache/auron/pull/1761 * Bump actions/stale from 5 to 10 by @dependabot[bot] in https://github.com/apache/auron/pull/1824 * Bump actions/upload-artifact from 4 to 6 by @dependabot[bot] in https://github.com/apache/auron/pull/1829 * [AURON #1805] Add contributing guidelines by @ShreyeshArangath in https://github.com/apache/auron/pull/1806 * Bump arduino/setup-protoc from 2 to 3 by @dependabot[bot] in https://github.com/apache/auron/pull/1826 * [AURON #1790] Rename workflow build-ce-releases.yml by @yew1eb in https://github.com/apache/auron/pull/1791 * [AURON #1790] Fix: correct build workflow badge link in README by @yew1eb in https://github.com/apache/auron/pull/1836 * Bump actions/labeler from 4 to 6 by @dependabot[bot] in https://github.com/apache/auron/pull/1830 * Bump actions/cache from 4 to 5 by @dependabot[bot] in https://github.com/apache/auron/pull/1825 * Bump actions/setup-java from 4 to 5 by @dependabot[bot] in https://github.com/apache/auron/pull/1827 * Bump actions/checkout from 4 to 6 by @dependabot[bot] in https://github.com/apache/auron/pull/1831 * Bump tokio from 1.48.0 to 1.49.0 by @dependabot[bot] in https://github.com/apache/auron/pull/1832 * [AURON #1841] fix(ci): adapt labeler.yml to action/labeler v6 by @yew1eb in https://github.com/apache/auron/pull/1842 * [AURON #1834] Support DataSourceScan metrics by @XorSum in https://github.com/apache/auron/pull/1835 * [AURON #1837] ci: Clean up injected JARs to keep Spark bin cache pure by @yew1eb in https://github.com/apache/auron/pull/1838 * Bump actions/download-artifact from 4 to 7 by @dependabot[bot] in https://github.com/apache/auron/pull/1828 * [AURON #1877] Fix spelling errors via the typos tool by @yew1eb in https://github.com/apache/auron/pull/1878 * [AURON #1792][FOLLOWUP] Broadcast isNullAwareAntiJoin flag by @cxzl25 in https://github.com/apache/auron/pull/1866 * Bump prost from 0.14.1 to 0.14.3 by @dependabot[bot] in https://github.com/apache/auron/pull/1884 * [AURON #1883] Bump maven version to 3.9.12 by @cxzl25 in https://github.com/apache/auron/pull/1885 * [AURON #1812][Correctness Testing] Spark 3.3 - Add Aggregate operator related tests by @ShreyeshArangath in https://github.com/apache/auron/pull/1819 * [AURON #1833] Refactor from_proto.rs to planner.rs by @yew1eb in https://github.com/apache/auron/pull/1843 * [AURON #1852] Init flink module by @Tartarus0zm in https://github.com/apache/auron/pull/1887 * [AURON #1895] Disable auto-close for stale issues by @ShreyeshArangath in https://github.com/apache/auron/pull/1896 * [AURON #1898] Fix NULL handling for NOT IN subquery by @yew1eb in https://github.com/apache/auron/pull/1899 * [AURON #1624] Support the pushdown of ORC predicates by @dh20 in https://github.com/apache/auron/pull/1886 * [AURON #1900] Upgrade Spark 3.5.7 to 3.5.8 by @yew1eb in https://github.com/apache/auron/pull/1901 * Bump chrono from 0.4.42 to 0.4.43 by @dependabot[bot] in https://github.com/apache/auron/pull/1894 * [AURON #1845] Fix StringSplit conversion by @Flyangz in https://github.com/apache/auron/pull/1846 * [AURON #1873] Add unified join BuildSide abstraction for cross-spark-version compatibility by @yew1eb in https://github.com/apache/auron/pull/1874 * [AURON #1502] Implement native function of bround. by @slfan1989 in https://github.com/apache/auron/pull/1706 * [AURON #1799][ci] Fix Spark cache key with hadoop-profile by @yew1eb in https://github.com/apache/auron/pull/1800 * [AURON #1769][AIP-3]: Introduce auron-it for Enhanced CI Integration Testing by @yew1eb in https://github.com/apache/auron/pull/1802 * [AURON #1637] Enforce Clippy lint and Eliminate unwrap by @yew1eb in https://github.com/apache/auron/pull/1649 * [AURON #1610] Handle case-insensitive columns in NativeHiveTableScanBase by @Flyangz in https://github.com/apache/auron/pull/1611 * [AURON #1693] join operation should flush in time on duplicated keys by @XorSum in https://github.com/apache/auron/pull/1701 * [AURON #1904] Fix CI pipeline failure by @yew1eb in https://github.com/apache/auron/pull/1905 * [AURON #1867][BUILD] Add scalastyle-maven-plugin by @yew1eb in https://github.com/apache/auron/pull/1868 * [AURON #1879] Add maven checkstyle plugin by @yew1eb in https://github.com/apache/auron/pull/1880 * [AURON #1907][CI] Fix YAML syntax error and restore TPCDS CI execution by @yew1eb in https://github.com/apache/auron/pull/1908 * Bump actions/upload-artifact from 4 to 6 by @dependabot[bot] in https://github.com/apache/auron/pull/1934 * [AURON #1911][clippy] Enable -D warnings and resolve related issues by @yew1eb in https://github.com/apache/auron/pull/1922 * [AURON #1814][Correctness Testing] Spark 3.3 - Add Sort operator related tests by @ShreyeshArangath in https://github.com/apache/auron/pull/1820 * [AURON #1926] ci: Remove opt-level=0 sed command to improve TPC-DS performance by @yew1eb in https://github.com/apache/auron/pull/1927 * [AURON #1769] Remove unused legacy files by @yew1eb in https://github.com/apache/auron/pull/1931 * [AURON #1937][CI] Skip TPCDS test q14b by @yew1eb in https://github.com/apache/auron/pull/1939 * [AURON #1888] Implement spark_partition_id() function by @ShreyeshArangath in https://github.com/apache/auron/pull/1928 * [AURON #1944] fix(clippy): fix panic error in spark_partition_id test by @yew1eb in https://github.com/apache/auron/pull/1945 * [AURON #1917] Clean up Clippy format/string lints by @lemorage in https://github.com/apache/auron/pull/1948 * [AURON #1816][Correctness Testing] Spark 3.3 - Add Parquet related tests by @ShreyeshArangath in https://github.com/apache/auron/pull/1954 * [AURON #1937][CI] Enable TPCDS test q14b by @cxzl25 in https://github.com/apache/auron/pull/1946 * [AURON #1920] Clean up Clippy condition/logic lints by @lemorage in https://github.com/apache/auron/pull/1960 * [AURON #1942] Implement columnar aggregate buffers by @richox in https://github.com/apache/auron/pull/1943 * [AURON #1778] Add spark.auron.expression.singleChildFallback.enabled config by @yew1eb in https://github.com/apache/auron/pull/1779 * [AURON #1769][CI] Improve auron-it build by @yew1eb in https://github.com/apache/auron/pull/1932 * [AURON #1875] Minor refactor: move legacy spark version compatibility methods to Shims.scala by @yew1eb in https://github.com/apache/auron/pull/1876 * [AURON #1739] Support LIMIT with OFFSET by @yew1eb in https://github.com/apache/auron/pull/1740 * [AURON #1757] Add cleanup logic to prevent flaky tests caused by leftover locations by @yew1eb in https://github.com/apache/auron/pull/1758 * [AURON #1940] Extract Rust test logic to standalone CI workflow by @yew1eb in https://github.com/apache/auron/pull/1941 * [AURON #1870][BUILD] Add `-Xfatal-warnings` to scala-maven-plugin and fix related compilation failures by @yew1eb in https://github.com/apache/auron/pull/1872 * [AURON #1963] Refactor SparkAuronConfiguration and remove deprecated AuronConf classes by @richox in https://github.com/apache/auron/pull/1964 * [AURON #1817][Correctness Testing] Spark 3.3 - Add Functions related tests by @ShreyeshArangath in https://github.com/apache/auron/pull/1966 * [AURON #1956] Add initial compatibility support for Spark 4.1 (UT/CI Pass) by @yew1eb in https://github.com/apache/auron/pull/1958 * [AURON #1818][Correctness Testing] Spark 3.3 - Add Expressions related tests by @ShreyeshArangath in https://github.com/apache/auron/pull/1965 * [AURON #1961] Fix Spark 4.0+: unit test catalyst codegen failure due to session artifact isolation by @yew1eb in https://github.com/apache/auron/pull/1971 * [AURON #1974] ci: add missing q30 q32 to tpcds queries by @yew1eb in https://github.com/apache/auron/pull/1973 * [AURON #1969] Disable auron.udf.singleChildFallback.enabled in unit tests by @yew1eb in https://github.com/apache/auron/pull/1970 * [AURON #1889] Implement monotonically_increasing_id() function by @ShreyeshArangath in https://github.com/apache/auron/pull/1955 * [AURON #1318][FOLLOWUP] Add test for `common_prefix_len` by @cxzl25 in https://github.com/apache/auron/pull/1897 * [AURON #1850] Add FlinkArrowUtils for Flink-Arrow type conversion by @x-tong in https://github.com/apache/auron/pull/1959 * [AURON #1988] Fix empty log message in AuronUniffleShuffleReader. by @slfan1989 in https://github.com/apache/auron/pull/1989 * [AURON#1991] Improve README formatting and grammar. by @slfan1989 in https://github.com/apache/auron/pull/1992 * [AURON #1404] Support for Spark 4.0.2 Compatibility in Auron. by @slfan1989 in https://github.com/apache/auron/pull/1990 * fix(bloom_filter): skip 1-byte marker in unfreeze_from_arrays to fix version mismatch by @BrytonLee in https://github.com/apache/auron/pull/1979 * Bump jemalloc_pprof from 0.8.1 to 0.8.2 by @dependabot[bot] in https://github.com/apache/auron/pull/1998 * [AURON #1999] Add Apache-2.0 license declaration to all native-engine Cargo.toml files. by @slfan1989 in https://github.com/apache/auron/pull/2000 * [RELEASE] Bump version 7.0.0-incubating by @merrily01 in https://github.com/apache/auron/pull/2007 ## New Contributors * @guixiaowen made their first contribution in https://github.com/apache/auron/pull/1345 * @adityavaish made their first contribution in https://github.com/apache/auron/pull/1542 * @ShreyeshArangath made their first contribution in https://github.com/apache/auron/pull/1598 * @yew1eb made their first contribution in https://github.com/apache/auron/pull/1609 * @CalvinKirs made their first contribution in https://github.com/apache/auron/pull/1616 * @xumingming made their first contribution in https://github.com/apache/auron/pull/1672 * @dh20 made their first contribution in https://github.com/apache/auron/pull/1768 * @lemorage made their first contribution in https://github.com/apache/auron/pull/1948 * @x-tong made their first contribution in https://github.com/apache/auron/pull/1959 * @BrytonLee made their first contribution in https://github.com/apache/auron/pull/1979 **Full Changelog**: https://github.com/apache/auron/compare/v6.0.0...v7.0.0
