Hey all, Thanks all for your patience on the documentation refactor. We're nearing the completion of the first (and most time-consuming) phase. I'd like to reiterate why we're doing yet another docs refactor.
Issues addressed by the first documentation refactor from the first docs: - We don't want the versioned docs or javadoc files to be tracked in the main branch to avoid multiple copies of the docs being indexed in GitHub or IDEs. - We need a top level (non-versioned) Iceberg website that holds versioned docs. Issues addressed by this refactor along with the original issues are addressed: - The current docs release process is cumbersome and the code lives across multiple repositories making it difficult to know where to contribute for documentation: https://github.com/apache/iceberg-docs/blob/main/README.md. - We wanted there to be an easy way to apply retroactive fixes to older doc versions. - A simple release process that should just be pushing a button and reviewing a PR. - Restyle Mkdocs default theme to look like the existing Iceberg theme. - Fix broken links (there were a lot). More design details: https://docs.google.com/document/d/1WJXzcwC6isfoywcLY2lZ9gZN6i0JU1I2SIZmCkumbZc/edit?usp=sharing Phase 1 Solution: We've moved to an mkdocs-material theme, which has a rich ecosystem of plugins that enable us to do the versioning under a single build process. Code: https://github.com/apache/iceberg/tree/main/site Preview: https://apache.github.io/iceberg/ We're currently running the preview of the new docs site on the main repo's github pages link (no worries, https://iceberg.apache.org/ is currently hosted on the ASF infra site in the iceberg-docs repo https://github.com/apache/iceberg-docs/blob/main/.asf.yaml#L38-L39) Improved Release process: There's some questions I have to the community about how the release process should go. I want to automate it as much as we can, while keeping a control in place. My suggested approach would be to automate the first three steps and have a 4th manual step for the release manager: 1. Invoke docs release workflow after main release (Unless we decide the steps below shouldn't be automated): 1. Create a copy of the versioned documentation and build the Javadocs. 2. Automerge these builds into the headless [docs branch]( https://github.com/apache/iceberg/tree/docs) and the [javadoc branch]( https://github.com/apache/iceberg/tree/javadoc) which are independent from the main branch. I don't see much benefit in reviewing the docs branch that has already been reviewed going into main, or reviewing the 1k+ change for the javadocs, hence those would be auto-merged and easy to roll back if something rendered improperly before the final PR. 3. Create a pull request with an offline build of the documentation to verify everything renders correctly. 2. The release manager validates the site/docs and merges that PR to finalize the docs release. For more information see the README: https://github.com/apache/iceberg/blob/main/site/README.md The final merge to swap is here: https://github.com/apache/iceberg/pull/9520 I want to leave some time and capture a vote on when we'll be ready to move forward, and address any concerns before we swap the sites. Along with your vote, could you also let me know if you're in favor of automating the release steps except for the final merge into the main branch that updates the site version to use in the static pages. Thanks all! Bits On Fri, Sep 29, 2023 at 12:20 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > > Hi Brian > > Thanks for the update. I will take a look. > > Regards > JB > > Le ven. 29 sept. 2023 à 07:05, Brian Olsen <bitsondata...@gmail.com> a écrit : >> >> Hey All, >> >> I know it's been a while but the first phase of the docs refactor has landed. I think it's at a decent point for everyone to take a look. To be clear, this is not going to replace the existing website yet, but get the first large landing of new docs to provide the initial proof of concept for the build and make incremental changes until we are comfortable making the swap. Once this is in and 1.4.0 goes out, I'll have to retroactively create tags for each prior version of the documentation. While that's happening, we can have someone else work on the look and feel of the website, to look closer to our current site. >> >> https://github.com/apache/iceberg/pull/8659 >> >> Thanks! Let me know if you have any questions! >> >> - Bits >> >> On Thu, Jul 27, 2023 at 4:10 PM Szehon Ho <szehon.apa...@gmail.com> wrote: >>> >>> Hi >>> >>> I'm ok with putting things back in Iceberg repo, it gets more visbility on prs. I guess it used to be a bit distracting, but now with more projects in Iceberg (pyiceberg, rust) we have to anyway use tags to filter through all the mails. >>> >>> Just wanted to +1 on Fokko/Ryan suggestion to avoid versioned doc directories, I had a lot of difficulties in this part doing the last release: https://github.com/apache/iceberg/issues/8151 , as did Anton when I consulted him offline. >>> >>> For me, replacing the 'latest' branch with a tag would be the biggest win as it caused me the most trouble. If we can avoid versioned docs and use tags across the board, that would be even better, I do think all the versions are already tagged in Github on every release, if that is your question? >>> >>> Thanks, >>> Szehon >>> >>> On Thu, Jul 27, 2023 at 2:31 AM Brian Olsen <bitsondata...@gmail.com> wrote: >>>> >>>> Thanks Fokko, >>>> >>>> Yeah, I think tío address that we would need to switch to a tagging that prefixes the different project name as a namespace within the tags space (e.g. pyIceberg-0.4.0, rust-0.0.1, etc…). But certainly this would result in an explosion of tags as we continue to introduce more projects. I’m not sure if this makes it difficult to find things as long as you start to search the prefix in GitHub it should be easy enough to find. Has anyone else worked on a project where this type of tagging is applied? Are their any performance, searching, or other implications we are missing? >>>> >>>> Bits >>>> >>>> On Thu, Jul 27, 2023 at 4:18 AM Fokko Driesprong <fo...@apache.org> wrote: >>>>> >>>>> Hey Brian, >>>>> >>>>> Thanks for raising this. As a release manager, I can confirm that the current structure is confusing, and I can also see the community struggling with this because they are willing to contribute to the docs, but cannot always find the place where to do this. I think the complexity of the current website mostly comes from the versioned docs. It would be great if we can find a way to make this easier. Instead of using the branches, we could also use the release tags and build the docs for those versions. >>>>> >>>>> I think switching to mkdocs-material is a great idea. We currently also use this for PyIceberg, and it works really well. My main concern is around merging everything together. Should we combine Java and Python in the same documentation? They have a different versioning scheme, so that would create a matrix of versions. Go and Rust is also in the making, so that would explode at some point. >>>>> >>>>> Cheers, Fokko >>>>> >>>>> Ps. Currently, PyIceberg uses the gh-pages branch for publishing the docs. >>>>> >>>>> >>>>> Op do 27 jul 2023 om 00:04 schreef Brian Olsen < bitsondata...@gmail.com>: >>>>>> >>>>>> Hey all, >>>>>> >>>>>> I have some proposals I'd like to make to fixing the docs. I would want to do this in two phases. >>>>>> >>>>>> The first phase I'm proposing that we locate all the documentation (reference docs, website, and pyIceberg) back into the apache/iceberg repository. I explain my reasoning in the attached document. This phase would also update us from Hugo to MkDocs but keep all the content the same. >>>>>> >>>>>> The second phase, is focused on iteratively building out the content that we've marked missing in some the proposal that Sam R. created along with a recent community member, Mahfuza. We will also restructure the content to following the diátaxis method (https://diataxis.fr/). >>>>>> >>>>>> https://docs.google.com/document/d/1WJXzcwC6isfoywcLY2lZ9gZN6i0JU1I2SIZmCkumbZc/edit#heading=h.gli9mc2ghfz1 >>>>>> >>>>>> Let me know what you think and bring on the questions and criticisms please! :) >>>>>> >>>>>> Bits
