Hi Harish, Thanks for this KIP.
Is it possible to add dev website from trunk for developers? It could help developer to browser the web without local build and check recent documentation change quickly. I think it can improve our documentation a lot since developers can easy to check it. Best, TaiJuWu Harish Vishwanath <harish.shas...@gmail.com> 於 2025年3月11日 週二 上午11:53寫道： > Thanks Chia-Ping. I have updated > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown#KIP1133:AKDocumentationandWebsiteinMarkdown-DevelopmentWorkflow > > > the KIP to include the above information. > > Regards, > Harish > > > On Sun, Mar 9, 2025 at 10:09 PM Chia-Ping Tsai <chia7...@gmail.com> wrote: > > > hi Harish > > > > > Hugo supports live-reload. The docsy theme has dependencies on golang > and > > a > > couple of nodeJS libraries (autoprefixer, postcss etc.,) and as such it > > will be simpler to use docker based workflows to avoid having to install > > any dependencies locally for development (ref: > > > > > https://github.com/hvishwanath/kafka-site-md/blob/main/README.md#local-development-server > > ). We can set up CI pipelines to build and push production images serving > > the site with nginx. > > > > Thank you for the explanation. Would you please consider including this > > information in the KIP? Documenting the new workflow for reviewing > > documentation would be very beneficial for future reference. In fact, we > > recently resolved a bug where the local site's behavior differed from the > > deployed version on Apache ... > > > > Best, > > Chia-Ping > > > > Harish Vishwanath <harish.shas...@gmail.com> 於 2025年3月10日 週一 下午12:51寫道： > > > > > Thank you Chia-Ping. > > > > > > Conversion is mostly automated ( https://github.com/hvishwanath/ak2md > ) > > > and > > > as such the prototype > > > <https://kafka-site-md-711970345036.us-central1.run.app/> supports all > > > previous versions. However, not having to support all previous versions > > > will definitely be simpler. > > > > > > Hugo supports live-reload. The docsy theme has dependencies on golang > > and a > > > couple of nodeJS libraries (autoprefixer, postcss etc.,) and as such it > > > will be simpler to use docker based workflows to avoid having to > install > > > any dependencies locally for development (ref: > > > > > > > > > https://github.com/hvishwanath/kafka-site-md/blob/main/README.md#local-development-server > > > ). We can set up CI pipelines to build and push production images > serving > > > the site with nginx. > > > > > > Regards, > > > Harish > > > > > > > > > On Sun, Mar 9, 2025 at 9:08 PM Chia-Ping Tsai <chia7...@gmail.com> > > wrote: > > > > > > > hi Harish > > > > > > > > I have to say, this KIP is one of the best ideas I've seen recently, > > and > > > I > > > > agree with Matthias that we don't need to translate any older > > versions. I > > > > truly hope this KIP can be included in 4.0. > > > > > > > > I have a quick question: what is the recommended procedure for > testing > > > the > > > > site locally after migrating to Markdown? Previously, we used a > > container > > > > to host the site locally for previewing changes. Could you please > > provide > > > > guidance on the updated workflow? > > > > > > > > Best, > > > > Chia-Ping > > > > > > > > Harish Vishwanath <harish.shas...@gmail.com> 於 2025年3月10日 週一 > > 上午11:53寫道： > > > > > > > > > Hello all > > > > > > > > > > Sorry for my delayed response. Thanks for your feedback and > comments. > > > > > Please see below: > > > > > > > > > > > Do you plan to have a development version of the website online > in > > > > > parallel to the old HTML version? > > > > > Yes. The current prototype which supports all versions until AK 3.9 > > is > > > > > running at https://kafka-site-md-711970345036.us-central1.run.app/ > , > > > the > > > > > prototype > > > > > < > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown#KIP1133:AKDocumentationandWebsiteinMarkdown-Prototype > > > > > > > > > > > section of the KIP has more details. We should be able to host this > > in > > > > ASF > > > > > infra. Further, as described in the KIP > > > > > > > > > > >I understand that locally with the markdown files and an markdown > > > > editor, > > > > > we see the rough structure of the docs (which is already an > > > improvement), > > > > > but we will not see the end product, because for that we again > need a > > > web > > > > > server. > > > > > Hugo supports live reload making local development easier (ref: > > > > > > https://github.com/hvishwanath/kafka-site-md/blob/main/Makefile#L25) > > > > > > > > > > > > > > > [MM1] Yes., I have ensured that all existing auto generated > > > > documentations > > > > > (html files under javadoc, generated folders) are available at the > > same > > > > > relative location as they are in the current website. You can see > > this > > > in > > > > > the prototype. > > > > > For example: > > > > > > > > > > > > > > > > > > > > https://kafka.apache.org/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > > > > is > > > > > available at > > > > > > > > > > > > > > > > > > > > https://kafka-site-md-711970345036.us-central1.run.app/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > > > > < > > > > > > > > > > > > > > > https://kafka-site-md-711970345036.us-central1.run.app/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > > > > > > > > > > Furthermore, for production, we can serve this with a full blown > web > > > > server > > > > > such as nginx allowing us to set up any required redirects. > > > > > > > > > > [MM2] I recommend that we leave any auto generated documentation to > > > > > continue to be in HTML format. We can serve them as static assets > > (ref: > > > > > https://github.com/hvishwanath/kafka-site-md/tree/main/static) > > > > > > > > > > >I am also in favor of having easy access to `trunk / SNAPSHOT` > > version > > > > of > > > > > the docs if possible. > > > > > Yes, this is definitely possible. > > > > > > > > > > > > > > > Regards, > > > > > Harish > > > > > > > > > > > > > > > On Sat, Mar 1, 2025 at 3:01 PM Matthias J. Sax <mj...@apache.org> > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I am totally in favor to make this happen, and I was disappointed > > > each > > > > > > time in the past, when it was proposed but we did not do it. > > > > > > > > > > > > > > > > > > > > > > > > Given that it is a huge effort, I am wondering if it would be > > > > sufficient > > > > > > to just to this change for 4.0 release though, and not translate > > any > > > > > > older versions at all? > > > > > > > > > > > > > > > > > > For including the docs in the release, I am actually wondering > what > > > the > > > > > > value is? Does anybody actually use these artifact? Of course, we > > > need > > > > > > to publish JavaDocs, but the content of the web page seems to > have > > > very > > > > > > low value to the community? -- I did also check ASF guidelines > > about > > > it > > > > > > and the webpage [1] say: > > > > > > > > > > > > > This material may include documentation concerning the release > > > [...] > > > > > > > > > > > > So it is optional, but not required to publish docs artifacts > from > > my > > > > > > understanding. I have no objections to keep publishing the web > page > > > > > > docs, just saying, if the effort is too large and the value to > low, > > > we > > > > > > could consider just dropping it. > > > > > > > > > > > > > > > > > > I am also in favor of having easy access to `trunk / SNAPSHOT` > > > version > > > > > > of the docs if possible. Some other ASF project do this already, > > for > > > > > > example Flink [2]. It's much easier to work with, and allows to > > spot > > > > > > issues during development. While it does not help on a PR > directly, > > > it > > > > > > is still a good way to verify a PR after it was merged, so any > > errors > > > > > > can be detected and corrected more easily. > > > > > > > > > > > > > > > > > > Also agree to the other raised points about generated content and > > > > > > permalinks. For permalinks, if effort is reasonable, I would > rather > > > try > > > > > > to redirect as many as possible. > > > > > > > > > > > > > > > > > > > > > > > > -Matthias > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > https://www.apache.org/legal/release-policy.html#distribute-other-artifacts > > > > > > [2] https://nightlies.apache.org/flink/flink-docs-master/ > > > > > > > > > > > > > > > > > > On 2/25/25 8:43 AM, David Arthur wrote: > > > > > > > MM1: I suspect Hugo has some way of doing url redirection, but > at > > > the > > > > > > very > > > > > > > least we can add redirects in our htaccess file > > > > > > > https://github.com/apache/kafka-site/blob/asf-site/.htaccess > > > > > > > > > > > > > > Excluding the "front matter" (intro, quickstart, powered by, > > etc) I > > > > > think > > > > > > > the main links we need to worry about are the configuration > docs > > > > pages. > > > > > > In > > > > > > > the demo, it looks like we are using the same anchors for > > specific > > > > > config > > > > > > > sections. > > > > > > > > > > > > > > E.g., > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kafka-site-md-711970345036.us-central1.run.app/39/configuration/configuration/#topicconfigs_compression.gzip.level > > > > > > > is the same content as > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kafka.apache.org/documentation/#topicconfigs_compression.zstd.level > > > > > > > > > > > > > > So maybe it shouldn't be too hard? > > > > > > > > > > > > > > Personally, I think the only links we should consider to be > > > > > "permalinks" > > > > > > > are the front matter and the documentation pages with the > version > > > in > > > > > the > > > > > > > URL. The unversioned documentation pages (like > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kafka.apache.org/documentation/#topicconfigs_compression.zstd.level > > > > > > ) > > > > > > > can change from release to release, so they are not expected to > > be > > > > > > > permanent. > > > > > > > > > > > > > > Do we have any data on inbound links to the site? Maybe from > > Google > > > > > > > Analytics or similar? > > > > > > > > > > > > > > -David A > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 24, 2025 at 10:17 AM Mickael Maison < > > > > > > mickael.mai...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > >> Hi Harish, > > > > > > >> > > > > > > >> This is definitively something we want to do! > > > > > > >> > > > > > > >> MM1. One aspect to keep in mind in terms of compatibility is > > > whether > > > > > > >> existing links will still work. Thousands of online resources > > link > > > > to > > > > > > >> the Apache Kafka website. Is it possible to keep all links > > > working? > > > > > > >> > > > > > > >> MM2. A bunch of sections are generated. To do so we have a > > number > > > of > > > > > > >> classes in the public API (for example 0, 1) with toHtml() or > > > main() > > > > > > >> methods that output HTML. Should we update these to output > > > Markdown > > > > > > >> directly instead? > > > > > > >> > > > > > > >> Converting the website has been discussed several times, I'd > > > suggest > > > > > > >> checking at least > > > https://issues.apache.org/jira/browse/KAFKA-2967 > > > > to > > > > > > >> see what was previously said. > > > > > > >> > > > > > > >> 0: > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://kafka.apache.org/39/javadoc/org/apache/kafka/common/config/ConfigDef.html#toHtml() > > > > > > >> 1: > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://kafka.apache.org/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > > > > > >> > > > > > > >> Thanks, > > > > > > >> Mickael > > > > > > >> > > > > > > >> On Mon, Feb 24, 2025 at 3:33 PM Bruno Cadonna < > > cado...@apache.org > > > > > > > > > > wrote: > > > > > > >>> > > > > > > >>> Hi Harish, > > > > > > >>> > > > > > > >>> Thanks for the KIP! Great initiative! > > > > > > >>> > > > > > > >>> Do you plan to have a development version of the website > online > > > in > > > > > > >>> parallel to the old HTML version? > > > > > > >>> > > > > > > >>> I understand that locally with the markdown files and an > > markdown > > > > > > >>> editor, we see the rough structure of the docs (which is > > already > > > an > > > > > > >>> improvement), but we will not see the end product, because > for > > > that > > > > > we > > > > > > >>> again need a web server. > > > > > > >>> > > > > > > >>> I assume that after you migrated all docs to markdown, we > want > > to > > > > see > > > > > > >>> the complete website, review the content, and make some > > > > refinements. > > > > > > >>> > > > > > > >>> Best, > > > > > > >>> Bruno > > > > > > >>> > > > > > > >>> On 23.02.25 07:52, Harish Vishwanath wrote: > > > > > > >>>> Thank you David. Appreciate your feedback. My comments > below: > > > > > > >>>> > > > > > > >>>> DA1) I think this is a very good suggestion. While building > > the > > > > > > >> prototype, > > > > > > >>>> I did spend additional effort for older versions of the doc > > > since > > > > > > >> there are > > > > > > >>>> fairly significant differences in the structure and format > of > > > the > > > > > > >> documents > > > > > > >>>> as compared to 3.x versions. I have written the automation > > which > > > > can > > > > > > do > > > > > > >>>> this, but I do think hosting the older versions (<3.x) in > its > > > > > current > > > > > > >>>> format and moving the rest to markdown will reduce the > overall > > > > > > >>>> testing/validation effort. That said, if we foresee making > > > > > relatively > > > > > > >>>> frequent updates to older versions, it might be worth > > converting > > > > > > >> everything > > > > > > >>>> to markdown. > > > > > > >>>> > > > > > > >>>> DA2) My hunch is that this can be quite involved. Rendering > a > > > > usable > > > > > > >> HTML > > > > > > >>>> version requires all the supporting artifacts (css, js, > > images, > > > > hugo > > > > > > >>>> shortcodes, partials etc.,) which will only be present in > the > > > > > website > > > > > > >>>> documentation. Further, the hyper links themselves will not > > work > > > > > > >> without > > > > > > >>>> a webserver. To serve a directory structure based HTML, it > > will > > > > > > >> require us > > > > > > >>>> to have different templates for these docs where > dependencies > > on > > > > the > > > > > > >>>> artifacts I mentioned above will need to be removed, links > > will > > > > need > > > > > > >> to be > > > > > > >>>> converted to use relative directory locations etc., > > > > > > >>>> > > > > > > >>>> DA3) This is definitely possible. Using tools such as > `pandoc` > > > we > > > > > > >> should be > > > > > > >>>> able to render plain text versions of markdown files. For > ex, > > > the > > > > > > below > > > > > > >>>> would create a plain text version of all docs for 39/ > branch., > > > > which > > > > > > >> can be > > > > > > >>>> packaged with the binary distribution. > > > > > > >>>> > > > > > > >>>> find docs/ -name "*.md" -type f | while read -r file; do > > > > > > >>>> dir=$(dirname "$file") > > > > > > >>>> filename=$(basename "$file" .md) > > > > > > >>>> mkdir -p "39-ascii/$dir" > > > > > > >>>> pandoc -f markdown -t plain "$file" -o > > > > > > >> "39-ascii/$dir/$filename.txt" > > > > > > >>>> done > > > > > > >>>> > > > > > > >>>> DA4) Makes sense. I updated the wiki to reflect this. > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> Regards, > > > > > > >>>> Harish > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> On Fri, Feb 21, 2025 at 8:56 AM David Arthur < > > mum...@gmail.com> > > > > > > wrote: > > > > > > >>>> > > > > > > >>>>> Thanks for the KIP, Harish! > > > > > > >>>>> > > > > > > >>>>> I am +100 just on the title alone :) > > > > > > >>>>> > > > > > > >>>>> The motivations in the KIP are sound. All of these are very > > > real > > > > > > >> annoyances > > > > > > >>>>> when it comes to maintaining our docs. I suspect we have > not > > > done > > > > > > >> anything > > > > > > >>>>> to improve them since the system we have worked well enough > > in > > > > the > > > > > > >> past. A > > > > > > >>>>> better way to say that is: it hasn't been bad enough to > > > motivate > > > > us > > > > > > to > > > > > > >>>>> change it. > > > > > > >>>>> > > > > > > >>>>> With efforts like this, there is always some question about > > > which > > > > > > >> tool or > > > > > > >>>>> method we should use. I like that your proposal includes > > these > > > > > > >> specifics as > > > > > > >>>>> well as a complete and functional demo. > > > > > > >>>>> > > > > > > >>>>> Few questions: > > > > > > >>>>> > > > > > > >>>>> DA1) Do we really need to convert all of our docs to the > new > > > > thing? > > > > > > It > > > > > > >>>>> might be reasonable to draw a line somewhere (maybe 3.0?) > and > > > > have > > > > > > >> docs > > > > > > >>>>> prior to that be hosted elsewhere (kafka.apache.org/old or > > > > > > >> something). > > > > > > >>>>> This > > > > > > >>>>> may or may not be more work than just converting > everything. > > > > WDYT? > > > > > > >>>>> > > > > > > >>>>> DA2) For our packaged docs (those that go in the binaries > we > > > > > > >> distribute), > > > > > > >>>>> do you think it's possible to render HTML that can be > viewed > > > > > without > > > > > > a > > > > > > >>>>> webserver? Currently our packaged docs are kind of useless > > IMO. > > > > It > > > > > > >> would be > > > > > > >>>>> better if they could be viewed without any extra effort by > an > > > > > > >> operator. > > > > > > >>>>> > > > > > > >>>>> DA3) As a possible alternative to DA2, could we render the > > > > markdown > > > > > > >> docs as > > > > > > >>>>> ascii text files? This is arguably better for operators > since > > > > they > > > > > > >> don't > > > > > > >>>>> always have a GUI or web browser handy. > > > > > > >>>>> > > > > > > >>>>> DA4) Just a nitpick, could you reduce the "Sample directory > > > > layout" > > > > > > >> down to > > > > > > >>>>> one or two examples? It's difficult to see the versioned vs > > > > > > >> non-versioned > > > > > > >>>>> docs in the current block > > > > > > >>>>> > > > > > > >>>>> Thanks! > > > > > > >>>>> David A > > > > > > >>>>> > > > > > > >>>>> > > > > > > >>>>> On Sun, Feb 16, 2025 at 2:10 PM Harish Vishwanath < > > > > > > >>>>> harish.shas...@gmail.com> > > > > > > >>>>> wrote: > > > > > > >>>>> > > > > > > >>>>>> Hello, > > > > > > >>>>>> > > > > > > >>>>>> Following up from my previous email regarding moving AK > > > > > > >> documentation to > > > > > > >>>>>> markdown, I wrote KIP-1133 > > > > > > >>>>>> < > > > > > > >>>>>> > > > > > > >>>>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown > > > > > > >>>>>>> > > > > > > >>>>>> detailing > > > > > > >>>>>> the proposal and would like to start discussions. Please > > take > > > a > > > > > > look. > > > > > > >>>>>> > > > > > > >>>>>> Regards, > > > > > > >>>>>> Harish > > > > > > >>>>>> > > > > > > >>>>> > > > > > > >>>>> > > > > > > >>>>> -- > > > > > > >>>>> David Arthur > > > > > > >>>>> > > > > > > >>>> > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
