alamb commented on code in PR #24792: URL: https://github.com/apache/datafusion/pull/24792#discussion_r4126808375
########## docs/scripts/validate_site.py: ########## @@ -0,0 +1,420 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Validate current and immutable full-site documentation output.""" Review Comment: What needs to be validated? I don't undertstand why we need a 400 line python script here ########## docs/scripts/assemble_site.py: ########## @@ -0,0 +1,132 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Replace the current site while retaining immutable full-site snapshots.""" Review Comment: Why do we need to replace the current site? This is pretty confusing to me ########## .github/workflows/docs.yaml: ########## @@ -20,12 +20,19 @@ on: branches: - main paths: + - .gitattributes Review Comment: i think these changes are unrelated to this PR (to trigger on changes to pyproject/uv) -- can you please make a separate PR to add them to main (it will be easier to review / merge) along with the rationale? ########## docs/scripts/snapshot_site.py: ########## @@ -0,0 +1,286 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Build one complete documentation snapshot from an exact release tag.""" Review Comment: don't we already have a build.sh script to do this? It seems like a lot of python code to call a few scripts 🤔 ########## docs/README.md: ########## @@ -19,63 +19,142 @@ # DataFusion Documentation -This folder contains the source content of the [User Guide](./source/user-guide) -and [Contributor Guide](./source/contributor-guide). These are both published to -https://datafusion.apache.org/ as part of the release process. +This folder contains the sources for https://datafusion.apache.org/. The root +site is built continuously from `main`. Complete, immutable release sites are +published under `/versions/<version>/`. ## Dependencies -Install build dependencies and build the documentation using +From the repository root, install the documentation dependencies using [uv](https://docs.astral.sh/uv/): ```sh -uv sync -uv run bash build.sh +uv sync --package datafusion-docs ``` The docs build regenerates the workspace dependency graph via `docs/scripts/generate_dependency_graph.sh`, so ensure `cargo`, `cargo-depgraph` (`cargo install cargo-depgraph --version ^1.6 --locked`), and Graphviz `dot` (`brew install graphviz` or `sudo apt-get install -y graphviz`) are available. +`.gitattributes` keeps documentation shell scripts LF-terminated so the same Review Comment: this seems like a somewhat irrelevant detail 🤔 ########## docs/README.md: ########## @@ -19,63 +19,142 @@ # DataFusion Documentation -This folder contains the source content of the [User Guide](./source/user-guide) -and [Contributor Guide](./source/contributor-guide). These are both published to -https://datafusion.apache.org/ as part of the release process. +This folder contains the sources for https://datafusion.apache.org/. The root +site is built continuously from `main`. Complete, immutable release sites are +published under `/versions/<version>/`. ## Dependencies -Install build dependencies and build the documentation using +From the repository root, install the documentation dependencies using [uv](https://docs.astral.sh/uv/): ```sh -uv sync -uv run bash build.sh +uv sync --package datafusion-docs ``` The docs build regenerates the workspace dependency graph via `docs/scripts/generate_dependency_graph.sh`, so ensure `cargo`, `cargo-depgraph` (`cargo install cargo-depgraph --version ^1.6 --locked`), and Graphviz `dot` (`brew install graphviz` or `sudo apt-get install -y graphviz`) are available. +`.gitattributes` keeps documentation shell scripts LF-terminated so the same +scripts run from Linux and WSL checkouts. -## Build & Preview +## Build and Preview -Run the provided script to build the HTML pages. +Build the current complete site from the repository root: ```bash -# If using venv, ensure you have activated it -./build.sh +uv run --package datafusion-docs docs/build.sh ``` -The HTML will be generated into a `build` directory. Open `build/html/index.html` -in your preferred browser, e.g. +The HTML is generated in `docs/build/html`. Serve it over HTTP because browsers +do not load the version manifest from `file:` URLs: -Preview the site on Linux by running this command. +```bash +python3 -m http.server --directory docs/build/html 8000 +``` + +Then open http://localhost:8000/. + +The public and assembled layouts are: + +```text +https://datafusion.apache.org/ +|-- user-guide/ +|-- library-user-guide/ +|-- contributor-guide/ +|-- _static/versions.json +`-- versions/55.0.0/ + |-- user-guide/ + |-- library-user-guide/ + |-- contributor-guide/ + |-- download.html + |-- search.html + |-- _sources/ + |-- _static/ + `-- sitemap.xml +``` + +## Release Snapshots + +`docs/source/_static/versions.json` is both the PyData version-picker manifest +and the release catalog. It contains `Development` at the site root and records +each release's semantic version, exact tag, and exact 40-character commit. + +The first snapshot is the lightweight tag `55.0.0`, which peels to Review Comment: "peels to"? ########## .gitattributes: ########## @@ -1,4 +1,6 @@ .github/ export-ignore +docs/*.sh text eol=lf Review Comment: why is this needed? ########## docs/README.md: ########## @@ -19,63 +19,142 @@ # DataFusion Documentation -This folder contains the source content of the [User Guide](./source/user-guide) -and [Contributor Guide](./source/contributor-guide). These are both published to -https://datafusion.apache.org/ as part of the release process. +This folder contains the sources for https://datafusion.apache.org/. The root +site is built continuously from `main`. Complete, immutable release sites are Review Comment: what does 'immutable' mean in this context? Maybe we should just say "released versions"? ########## docs/tests/test_versioned_docs.py: ########## @@ -0,0 +1,347 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review Comment: what does this test? ########## docs/README.md: ########## @@ -19,63 +19,142 @@ # DataFusion Documentation -This folder contains the source content of the [User Guide](./source/user-guide) -and [Contributor Guide](./source/contributor-guide). These are both published to -https://datafusion.apache.org/ as part of the release process. +This folder contains the sources for https://datafusion.apache.org/. The root +site is built continuously from `main`. Complete, immutable release sites are +published under `/versions/<version>/`. ## Dependencies -Install build dependencies and build the documentation using +From the repository root, install the documentation dependencies using [uv](https://docs.astral.sh/uv/): ```sh -uv sync -uv run bash build.sh +uv sync --package datafusion-docs ``` The docs build regenerates the workspace dependency graph via `docs/scripts/generate_dependency_graph.sh`, so ensure `cargo`, `cargo-depgraph` (`cargo install cargo-depgraph --version ^1.6 --locked`), and Graphviz `dot` (`brew install graphviz` or `sudo apt-get install -y graphviz`) are available. +`.gitattributes` keeps documentation shell scripts LF-terminated so the same +scripts run from Linux and WSL checkouts. -## Build & Preview +## Build and Preview -Run the provided script to build the HTML pages. +Build the current complete site from the repository root: ```bash -# If using venv, ensure you have activated it -./build.sh +uv run --package datafusion-docs docs/build.sh ``` -The HTML will be generated into a `build` directory. Open `build/html/index.html` -in your preferred browser, e.g. +The HTML is generated in `docs/build/html`. Serve it over HTTP because browsers +do not load the version manifest from `file:` URLs: -Preview the site on Linux by running this command. +```bash +python3 -m http.server --directory docs/build/html 8000 +``` + +Then open http://localhost:8000/. + +The public and assembled layouts are: Review Comment: is this list necessary? It seems like it will just get out of date over time ########## docs/README.md: ########## @@ -19,63 +19,142 @@ # DataFusion Documentation -This folder contains the source content of the [User Guide](./source/user-guide) -and [Contributor Guide](./source/contributor-guide). These are both published to -https://datafusion.apache.org/ as part of the release process. +This folder contains the sources for https://datafusion.apache.org/. The root +site is built continuously from `main`. Complete, immutable release sites are +published under `/versions/<version>/`. ## Dependencies -Install build dependencies and build the documentation using +From the repository root, install the documentation dependencies using Review Comment: if this needs changing, perhaps you can make another standalone PR to update the build docs ########## docs/source/_static/versions.json: ########## @@ -0,0 +1,14 @@ +[ Review Comment: Yes, this looks good -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
