morningman opened a new pull request, #66837:
URL: https://github.com/apache/doris/pull/66837
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
The tree carried two libhdfs builds:
- `build_hadoop_libs` built the **3.3.6.6** fork into
`installed/{include,lib}/hadoop_hdfs/`
- `build_hadoop_libs_3_4` built the **3.4.2.4** fork into
`installed/{include,lib}/hadoop_hdfs_3_4/`
On Linux both ran. In the latest scheduled `apache/doris-thirdparty` build
that
is ~10 extra minutes of every third-party build (11m38s for the 3.3.6.6 fork
on
the x86_64 runner, 10m00s on the arm64 one) for a library that only one
module
still consumed.
The two prefixes were also mixed up on the consumer side:
| consumer | header | library |
|---|---|---|
| `be/src/io/fs/hdfs.h` | `hadoop_hdfs_3_4/hdfs.h` (3.4.2.4) | — |
| `be/src/io/hdfs_builder.cpp` | `hadoop_hdfs/hdfs.h` (**3.3.6.6**) | — |
| `be/CMakeLists.txt` | — | `hadoop_hdfs_3_4/native/libhdfs.a` (3.4.2.4) |
| `cloud/CMakeLists.txt` | `hadoop_hdfs/hdfs.h` (**3.3.6.6**) |
`hadoop_hdfs/native/libhdfs.a` (**3.3.6.6**) |
`hdfs_builder.cpp` pulled in both headers in a single translation unit (only
the
first one won, because they share the `LIBHDFS_HDFS_H` include guard), and BE
compiled part of its HDFS code against the 3.3.6.6 header while linking the
3.4.2.4 archive.
This PR keeps exactly one libhdfs on every platform:
- `thirdparty/vars.sh`: `HADOOP_LIBS_*` now points at
`hadoop-3.4.2.4-for-doris`;
the `HADOOP_LIBS_3_4_*` block and its `TP_ARCHIVES` entry are gone.
- `thirdparty/build-thirdparty.sh`: `build_hadoop_libs_3_4` is folded into
`build_hadoop_libs`, which installs 3.4.2.4 under the existing
`hadoop_hdfs/`
prefix. The package is appended on Linux and macOS alike, so the platform
branch around it goes away.
- `be/CMakeLists.txt`, `be/src/io/fs/hdfs.h`: `hadoop_hdfs_3_4/` ->
`hadoop_hdfs/`.
- `be/src/io/hdfs_builder.cpp`: drop the now-duplicate `hadoop_hdfs/hdfs.h`
include - `io/fs/hdfs.h`, included two lines below, already re-exports it.
- `cloud/CMakeLists.txt`, `build.sh`, `run-be-ut.sh`, `run-cloud-ut.sh` need
no
change: they already reference `hadoop_hdfs/`, which now holds 3.4.2.4.
Net effect: BE and the cloud meta-service compile and link against the same
libhdfs, and every third-party build drops one full Hadoop native build.
### Release note
None
### Check List (For Author)
- Test
- [x] No need to test or manual test. Explain why:
- [x] Other reason: this only changes which Hadoop fork the
third-party
build produces and where it is installed. It is covered by the
existing
BE/cloud compilation plus any HDFS regression case, since a wrong
path
fails the link outright.
- Behavior changed:
- [x] Yes. `installed/lib/hadoop_hdfs_3_4/` and
`installed/include/hadoop_hdfs_3_4/`
no longer exist in the prebuilt third-party archive; both now live
under
`hadoop_hdfs/`, and their contents are 3.4.2.4 rather than 3.3.6.6.
Anything
outside this repository that hardcodes either path needs updating, and
the
third-party prebuilt has to be rebuilt before this lands in a
build-env image.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
