[
https://issues.apache.org/jira/browse/HDDS-17275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17275:
------------------------------
Attachment: HDDS-17275.001.diff
> Make release tarballs reproducible across build environments
> ------------------------------------------------------------
>
> Key: HDDS-17275
> URL: https://issues.apache.org/jira/browse/HDDS-17275
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Attachments: HDDS-17275.001.diff
>
>
> h3. Problem
> The {{repro}} check runs {{artifact:compare}} in the same job, on the same
> runner, path, JDK, user, locale and umask as the reference build. It covers
> attached Maven artifacts, but not the release tarballs ({{ozone-X.tar.gz}} is
> produced by {{dist-tar-stitching}}, {{ozone-X-src.tar.gz}} has
> {{attach=false}}), and it cannot detect differences caused by the build
> environment. Verifying a release independently (for example with Apache
> Trusted Releases, see HADOOP-19976) needs the tarballs to be identical when
> built elsewhere.
> h3. Observation
> Two runs of {{mvn clean package -DskipTests -DskipDocs -Pdist -Psrc}} on the
> same machine, the second with {{TZ=Asia/Tokyo LC_ALL=de_DE.UTF-8}} and
> {{umask 002}}:
> * all jars (including javadoc, sources and shaded jars), {{bom.json}},
> {{bom.xml}} and the source tarball were identical
> * the binary tarball had identical file contents and entry order, but
> different archive metadata: entry mtimes, permissions (following umask),
> owner and group names of the builder, and the mtime and file name in the gzip
> header
> h3. Gaps
> # *Binary tarball metadata.* {{dist-tar-stitching}} runs plain {{tar}} and
> {{gzip}}. Entry order also depends on the directory listing order of the
> filesystem (sorted on APFS, not on ext4).
> # *Unpinned inputs.* The {{hadoop-native-lib}} profile downloads libhadoop
> from the {{master}} branch of {{ozone-thirdparty}} without a checksum. The
> RocksDB source archive for {{hdds-rocks-native}} is downloaded without a
> checksum.
> # *Optional content depending on the environment.* Docs are included only if
> Hugo is available, and {{--enableGitInfo}} is used only in a git checkout, so
> building from the source tarball gives different docs than building from git.
> libhadoop is included only if it was downloaded. The Hugo docs are being
> removed by HDDS-15923 ([PR
> #10829|https://github.com/apache/ozone/pull/10829]), which resolves the docs
> part.
> # *Build environment not pinned or recorded.* The release guide uses the
> floating {{rockylinux:9}} image and unpinned JDK, cmake and gcc packages. No
> {{.buildinfo}} is produced, so a verifier cannot know which toolchain to use.
> # *Native library build path.* {{libozone_rocksdb_tools.so}} is built without
> {{-ffile-prefix-map}}, so it may embed the build directory. Not verified.
> # *rpm and deb packages* (HDDS-13439) may embed build host and build time.
> Not verified; only relevant if they become release artifacts.
> h3. Proposed changes
> * Create the binary tarball with sorted entries, mtime from
> {{project.build.outputTimestamp}}, numeric owner and group 0, normalized
> permissions, and {{gzip -n}}.
> * Pin downloaded inputs to a tag or commit and verify them with {{sha256}}.
> * Make tarball content independent of optional downloads such as libhadoop
> (include it consistently or fail).
> * Generate {{.buildinfo}} ({{artifact:buildinfo}}) and document a pinned
> build environment for releases.
> * Extend the {{repro}} check (or add a separate one) to compare both tarballs
> from an independent build with a different path, user, TZ, locale and umask.
> These can be split into separate subtasks if preferred. Nondeterminism within
> a single build is tracked separately in HDDS-16309 and HDDS-12862.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
