I have been working on a similar process for Apache RAT and Commons CLI (help module). The help option in RAT 0.17-SNAPSHOT generates the text for the help options partly from the new commons-cli help module and partly from Enums and similar in the code.
For the RAT documentation we produce fragments, for example a markdown fragment that lists every value in an enum as well as its description (a variable in the enum). We then use the Maven site plugin so it can run velocity to combine fragments together using an `insert` statement in a master document. In this way the master markdown says something like # All about X The options for X are: --insert X.txt and the result is the document specifies all the options and their descriptions. Add another enum value and it appears in the documentation. The commons-cli 1.10-SNAPSHOT help system uses the concept of a "scribe" that writes a document in an agnostic way. Basically it has an interface with methods for inserting section titles, paragraphs, lists, and tables. Specific implementations write specific formats. So the same scribe can write markdown, HTML, APT, or any number of formats. I have spent the last week or so working out how to do this for the Aiven Kafka Connectors. The work is still in an early stage but is visible currently at https://claudenw.github.io/cloud-storage-connectors-for-apache-kafka/ The S3 Source -> Configuration link on that page is created by wrapping a markdown file generated by writing the ConfigDef values. There is also a yaml version ( https://claudenw.github.io/cloud-storage-connectors-for-apache-kafka/s3-source-connector/S3SourceConfig.yml) that Aiven's inhouse documentation team is going to use to generate in house documentation. In any case I support this goal and am willing to rework my contributions for RAT, Commons-cli, and Aiven Kafka Connectors in support of this goal, or whatever else needs to be done. Given that the 3 projects mentioned already do something similar, and I did come across another package that did something similar last week, perhaps we should think about a component for Apache Commons that handles generating files from code. Claude On Mon, Feb 24, 2025 at 3:19 PM Mickael Maison <mickael.mai...@gmail.com> wrote: > Hi Harish, > > This is definitively something we want to do! > > MM1. One aspect to keep in mind in terms of compatibility is whether > existing links will still work. Thousands of online resources link to > the Apache Kafka website. Is it possible to keep all links working? > > MM2. A bunch of sections are generated. To do so we have a number of > classes in the public API (for example 0, 1) with toHtml() or main() > methods that output HTML. Should we update these to output Markdown > directly instead? > > Converting the website has been discussed several times, I'd suggest > checking at least https://issues.apache.org/jira/browse/KAFKA-2967 to > see what was previously said. > > 0: > https://kafka.apache.org/39/javadoc/org/apache/kafka/common/config/ConfigDef.html#toHtml() > 1: > https://kafka.apache.org/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > Thanks, > Mickael > > On Mon, Feb 24, 2025 at 3:33 PM Bruno Cadonna <cado...@apache.org> wrote: > > > > Hi Harish, > > > > Thanks for the KIP! Great initiative! > > > > Do you plan to have a development version of the website online in > > parallel to the old HTML version? > > > > I understand that locally with the markdown files and an markdown > > editor, we see the rough structure of the docs (which is already an > > improvement), but we will not see the end product, because for that we > > again need a web server. > > > > I assume that after you migrated all docs to markdown, we want to see > > the complete website, review the content, and make some refinements. > > > > Best, > > Bruno > > > > On 23.02.25 07:52, Harish Vishwanath wrote: > > > Thank you David. Appreciate your feedback. My comments below: > > > > > > DA1) I think this is a very good suggestion. While building the > prototype, > > > I did spend additional effort for older versions of the doc since > there are > > > fairly significant differences in the structure and format of the > documents > > > as compared to 3.x versions. I have written the automation which can do > > > this, but I do think hosting the older versions (<3.x) in its current > > > format and moving the rest to markdown will reduce the overall > > > testing/validation effort. That said, if we foresee making relatively > > > frequent updates to older versions, it might be worth converting > everything > > > to markdown. > > > > > > DA2) My hunch is that this can be quite involved. Rendering a usable > HTML > > > version requires all the supporting artifacts (css, js, images, hugo > > > shortcodes, partials etc.,) which will only be present in the website > > > documentation. Further, the hyper links themselves will not work > without > > > a webserver. To serve a directory structure based HTML, it will > require us > > > to have different templates for these docs where dependencies on the > > > artifacts I mentioned above will need to be removed, links will need > to be > > > converted to use relative directory locations etc., > > > > > > DA3) This is definitely possible. Using tools such as `pandoc` we > should be > > > able to render plain text versions of markdown files. For ex, the below > > > would create a plain text version of all docs for 39/ branch., which > can be > > > packaged with the binary distribution. > > > > > > find docs/ -name "*.md" -type f | while read -r file; do > > > dir=$(dirname "$file") > > > filename=$(basename "$file" .md) > > > mkdir -p "39-ascii/$dir" > > > pandoc -f markdown -t plain "$file" -o > "39-ascii/$dir/$filename.txt" > > > done > > > > > > DA4) Makes sense. I updated the wiki to reflect this. > > > > > > > > > Regards, > > > Harish > > > > > > > > > On Fri, Feb 21, 2025 at 8:56 AM David Arthur <mum...@gmail.com> wrote: > > > > > >> Thanks for the KIP, Harish! > > >> > > >> I am +100 just on the title alone :) > > >> > > >> The motivations in the KIP are sound. All of these are very real > annoyances > > >> when it comes to maintaining our docs. I suspect we have not done > anything > > >> to improve them since the system we have worked well enough in the > past. A > > >> better way to say that is: it hasn't been bad enough to motivate us to > > >> change it. > > >> > > >> With efforts like this, there is always some question about which > tool or > > >> method we should use. I like that your proposal includes these > specifics as > > >> well as a complete and functional demo. > > >> > > >> Few questions: > > >> > > >> DA1) Do we really need to convert all of our docs to the new thing? It > > >> might be reasonable to draw a line somewhere (maybe 3.0?) and have > docs > > >> prior to that be hosted elsewhere (kafka.apache.org/old or > something). > > >> This > > >> may or may not be more work than just converting everything. WDYT? > > >> > > >> DA2) For our packaged docs (those that go in the binaries we > distribute), > > >> do you think it's possible to render HTML that can be viewed without a > > >> webserver? Currently our packaged docs are kind of useless IMO. It > would be > > >> better if they could be viewed without any extra effort by an > operator. > > >> > > >> DA3) As a possible alternative to DA2, could we render the markdown > docs as > > >> ascii text files? This is arguably better for operators since they > don't > > >> always have a GUI or web browser handy. > > >> > > >> DA4) Just a nitpick, could you reduce the "Sample directory layout" > down to > > >> one or two examples? It's difficult to see the versioned vs > non-versioned > > >> docs in the current block > > >> > > >> Thanks! > > >> David A > > >> > > >> > > >> On Sun, Feb 16, 2025 at 2:10 PM Harish Vishwanath < > > >> harish.shas...@gmail.com> > > >> wrote: > > >> > > >>> Hello, > > >>> > > >>> Following up from my previous email regarding moving AK > documentation to > > >>> markdown, I wrote KIP-1133 > > >>> < > > >>> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown > > >>>> > > >>> detailing > > >>> the proposal and would like to start discussions. Please take a look. > > >>> > > >>> Regards, > > >>> Harish > > >>> > > >> > > >> > > >> -- > > >> David Arthur > > >> > > > > > >
