This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 467af6ea1d1 [improvement](deps) Upgrade librdkafka to 2.16.0 (#69235)
467af6ea1d1 is described below
commit 467af6ea1d16858b805367acbcdd6ec56e7140e6
Author: Shekhar Rajak <[email protected]>
AuthorDate: Wed Oct 21 09:02:33 2026 +0530
[improvement](deps) Upgrade librdkafka to 2.16.0 (#69235)
### What problem does this PR solve?
Issue Number: close #69234
Upgrade librdkafka from 2.11.0 to 2.16.0 as a dependency prerequisite
for exploring Kafka share-group ingestion (#63190). This change does not
enable share consumers in Routine Load.
### Release note
Upgrade the librdkafka dependency to 2.16.0.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (details below)
- Behavior changed:
- [x] Yes: upgrades the underlying Kafka client library; no new Doris
ingestion mode or configuration.
- Does this need documentation?
- [x] No.
Validation:
- `bash -n thirdparty/vars.sh thirdparty/download-thirdparty.sh
thirdparty/build-thirdparty.sh` passed.
- `git diff --check` passed.
- `bash thirdparty/build-thirdparty.sh librdkafka` successfully
downloaded, checksum-verified, extracted, and patched 2.16.0.
Compilation was blocked by the local prerequisite check: `byacc: command
not found`.
- Built the patched upstream C and C++ libraries on macOS using
`./configure --enable-static --disable-c11threads --disable-ssl
--disable-gssapi --disable-curl --disable-zstd --disable-zlib
--disable-lz4-ext` followed by `make -j8`.
- `make -C tests -j8 build` and `make -C tests unit` passed
(`0000_unittests`).
Full Doris third-party/BE compilation, SASL/SSL linking, and
broker-backed Routine Load ingestion/recovery remain unvalidated.
Keeping this PR in draft pending that coverage.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---------
Co-authored-by: Shekhar Rajak <[email protected]>
---
thirdparty/download-thirdparty.sh | 4 +--
...dkafka-2.11.0.patch => librdkafka-2.16.0.patch} | 29 +---------------------
thirdparty/vars.sh | 10 ++++----
3 files changed, 8 insertions(+), 35 deletions(-)
diff --git a/thirdparty/download-thirdparty.sh
b/thirdparty/download-thirdparty.sh
index c65ea6dfd7d..5af6d4a96fa 100755
--- a/thirdparty/download-thirdparty.sh
+++ b/thirdparty/download-thirdparty.sh
@@ -499,10 +499,10 @@ fi
# patch librdkafka to avoid crash
if [[ " ${TP_ARCHIVES[*]} " =~ " LIBRDKAFKA " ]]; then
- if [[ "${LIBRDKAFKA_SOURCE}" == "librdkafka-2.11.0" ]]; then
+ if [[ "${LIBRDKAFKA_SOURCE}" == "librdkafka-2.16.0" ]]; then
cd "${TP_SOURCE_DIR}/${LIBRDKAFKA_SOURCE}"
if [[ ! -f "${PATCHED_MARK}" ]]; then
- patch -p0 <"${TP_PATCH_DIR}/librdkafka-2.11.0.patch"
+ patch -p0 <"${TP_PATCH_DIR}/librdkafka-2.16.0.patch"
touch "${PATCHED_MARK}"
fi
cd -
diff --git a/thirdparty/patches/librdkafka-2.11.0.patch
b/thirdparty/patches/librdkafka-2.16.0.patch
similarity index 61%
rename from thirdparty/patches/librdkafka-2.11.0.patch
rename to thirdparty/patches/librdkafka-2.16.0.patch
index 21792174073..cf33d2d9b82 100644
--- a/thirdparty/patches/librdkafka-2.11.0.patch
+++ b/thirdparty/patches/librdkafka-2.16.0.patch
@@ -34,36 +34,9 @@
# Clear define name ($2): caller may have additional checks
mkl_check_failed "$cname" "" "$3" "pkg-config --libs failed"
return 1
---- src/rdkafka.c
-+++ src/rdkafka.c
-@@ -3750,6 +3750,7 @@ rd_kafka_resp_err_t
rd_kafka_query_watermark_offsets(rd_kafka_t *rk,
- struct rd_kafka_partition_leader *leader;
- rd_list_t leaders;
- rd_kafka_resp_err_t err;
-+ int tmout;
-
- partitions = rd_kafka_topic_partition_list_new(1);
- rktpar =
-@@ -3796,9 +3797,13 @@ rd_kafka_resp_err_t
rd_kafka_query_watermark_offsets(rd_kafka_t *rk,
-
- /* Wait for reply (or timeout) */
- while (state.err == RD_KAFKA_RESP_ERR__IN_PROGRESS) {
-- rd_kafka_q_serve(rkq, RD_POLL_INFINITE, 0,
-- RD_KAFKA_Q_CB_CALLBACK, rd_kafka_poll_cb,
-- NULL);
-+ tmout = rd_timeout_remains(ts_end);
-+ if (rd_timeout_expired(tmout)) {
-+ state.err = RD_KAFKA_RESP_ERR__TIMED_OUT;
-+ break;
-+ }
-+ rd_kafka_q_serve(rkq, tmout, 0, RD_KAFKA_Q_CB_CALLBACK,
-+ rd_kafka_poll_cb, NULL);
- }
-
- rd_kafka_q_destroy_owner(rkq);
--- src/rdkafka_broker.c
+++ src/rdkafka_broker.c
-@@ -4759,7 +4759,9 @@ static int rd_kafka_broker_thread_main(void *arg) {
+@@ -5131,7 +5131,9 @@ void rd_kafka_broker_destroy_final(rd_kafka_broker_t
*rkb) {
*/
void rd_kafka_broker_destroy_final(rd_kafka_broker_t *rkb) {
diff --git a/thirdparty/vars.sh b/thirdparty/vars.sh
index 45782688dd4..347f2b0fd7d 100644
--- a/thirdparty/vars.sh
+++ b/thirdparty/vars.sh
@@ -222,11 +222,11 @@ CYRUS_SASL_NAME=cyrus-sasl-2.1.27.tar.gz
CYRUS_SASL_SOURCE=cyrus-sasl-2.1.27
CYRUS_SASL_MD5SUM="a33820c66e0622222c5aefafa1581083"
-# librdkafka-2.11.0
-LIBRDKAFKA_DOWNLOAD="https://github.com/confluentinc/librdkafka/archive/refs/tags/v2.11.0.tar.gz";
-LIBRDKAFKA_NAME=librdkafka-2.11.0.tar.gz
-LIBRDKAFKA_SOURCE=librdkafka-2.11.0
-LIBRDKAFKA_MD5SUM="bc611d0340e269abaa8886d42ff9c558"
+# librdkafka-2.16.0
+LIBRDKAFKA_DOWNLOAD="https://github.com/confluentinc/librdkafka/archive/refs/tags/v2.16.0.tar.gz";
+LIBRDKAFKA_NAME=librdkafka-2.16.0.tar.gz
+LIBRDKAFKA_SOURCE=librdkafka-2.16.0
+LIBRDKAFKA_MD5SUM="6d2c62921241863d2bc95c58be27af6f"
# zstd
ZSTD_DOWNLOAD="https://github.com/facebook/zstd/releases/download/v1.5.7/zstd-1.5.7.tar.gz";
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
