On Sun, 2020-04-05 at 21:33 +0200, Roland Clobus wrote: > Thank you for your efforts to make the live-manual better. > I'm adding the mailing list to the list of recipients, because the > work > is done by the live team (of which I'm not a member, just a person > with > interest in the project) and the team can decide on the future > direction > of the project.
Yes, I'm aware. I was just curious about the nature of the update to live-manual you were tackling. I was not ready to initiate a proper discussion/proposal of format changes, which would have occurred openly later if I was still then interested in actually carrying it out. I just wanted at this time to merely gain an understanding of whether this was something you were already doing, and if not then some idea of when perhaps your work might be available to build such a change upon. > > I believe you were the person who spoke of being in the middle of a > > big> update to live-manual? > Well big... I am working (with the little time that I have) on > updating > the live-manual, primarily such that all references to Alioth get > removed. While doing so, I got to know the live building process and > updated parts of the manual that got out-of-date over time. Yes, I understood that your work essentially was focussed on bringing it up to date and my impression was that this was more than just a few minor tweaks, especially after noticing previously a question being asked about live-wrapper which the manual does not currently cover at all, and since if it was a very minor change you'd have probably submitted it by now. (Though of course the response indicated that live-wrapper is effectively dead so no longer needs covering anyway...) Alioth references - Aside from translation files, which I've ignored, all I can find is three old links in searching for "alioth"; one of which I missed in my previous "live-systems" update, I've just submitted an MR for that, the other two I'd previously spotted but not known what to do with only require minor corrections. So I don't understand what it is you're actually doing with respect to "removing alioth references" if it's not just that... (not that I need to know) (and not that I've really looked at the manual in quite some time). Perhaps you could publish a preview (WIP) copy of your changes so far, even if it's currently an incomplete mess? (I looked at your salsa repo on Saturday but saw no sign of such work, so you must have it locally only I suppose). btw, I've noticed that the manual is currently missing discussion of injecting environment variables via the user config, if fixing that fits within your scope of improvements and want to tackle it. It would be good to have that covered. I can provide some details if needed. > > Does your work involve at all: > > b. any change to the markup language and respective "build" tool > > used > > for "building" the documentation (generation of HTML pages and > > PDFs). > > No, the markup language SiSU is not a common markup language, but for > me > it suits its purpose. ok. > > I ask because I made a relatively small change the other day to the > > coding style section (submitted an MR which is pending review); > > I assume you are referring to > https://salsa.debian.org/live-team/live-manual/-/merge_requests/22 yes. > > The build tool SiSU was my biggest gripe. I was not impressed with > > the > > hundreds of megabytes of dozens of packages required for sisu- > > complete > > installation, that it seemed to be generating a postgresql database > > on > > installation, that it pulls in ruby and such, and the manual having > > to > > give advice on speeding up a supposedly very slow build process (I > > followed the tips to limit the scope of building, which was > > reasonably > > quick, I did not explore how slow the worst case supposedly is). > > As far as I can see, SiSU was a nice tool at the time the live-manual > was started. It apparently didn't catch on, as nowadays there are > hardly > any reverse-(build)dependencies on SiSU. Yes, I believe that live-manual is the only package build depending on sisu-complete, and unless I've made a mistake in how I've searched the archive, it is the only package in the entire archive actually using SiSU. A good few dozen on the other hand depend upon markdown or pandoc. Hell, I just looked at the SiSU website and found that all of their source code links to be dead (everything at http://git.sisudoc.org: http://git.sisudoc.org/software/sisu; http://git.sisudoc.org/git/code/sisu.git; http://git.sisudoc.org/gitweb/?p=code/sisu.git;a=summary) which is not at all a good sign, not a state expected of a healthy project. > However, SiSU allows the author to focus on the content without being > bothered by layout decisions and markup. I don't exactly agree, and I do not think that the separation is a significant concern here. >From what little of SiSU I know (primarily from noting the difference in ssi vs. ssm files), yes, SiSU separates content from layout, but it certainly does not stop authors being bothered by markup. Surely you did not mean to suggest that? The SSI content files are riddled with SiSU markup artefacts, as to be expected in anything that is not just plaintext. You cannot escape from having to deal with markup of one form or another unless writing only plaintext. Furthermore the uncommon nature of SiSU markup is itself a hindrance to authoring changes. I only just about got by without having to find SiSU documentation by copying what I was seeing elsewhere in the files. Markdown/commonmark and HTML on the other hand are widely understood. The layout components of live-manual are _very_ minimal. I would not expect anyone to reasonably consider them to be getting in the way of authoring changes if we were to move to one of the two proposed alternate formats. If we consider markdown as an alternate to SiSU, then essentially all of the ssi files will translate to equivalent markdown/commonmark markup. They could perhaps work stand alone without any template, with the all-in-one version being created from a simple bit of script mostly just appending each page into a single file. For HTML as an alternative, you'd have a relatively small block of HTML before and after the chunk that is the content. The HTML within the content would largely just be equivalent markup, with the major different just being <p></p> tags surrounding paragraphs and similar for headings. Or, if really wanting to separate out the starting and ending HTML from the chunk that is the content, you could have that in a "template" HTML file that has a placeholder for the content, and then have the build script inject the file content of pages using the `sed` tool or similar. You thus split out this small amount of non-content HTML, at the expense of having to "build" pages to "test"/review changes in their non-source form. Leaving it in each file though I really don't consider worthy of being said to be a bother to authors. If ever wanting to modify this starting/ending portion of HTML though, having a template has benefits over having to change multiple files and across multiple translations, but how often are changes ever going to be made that this becomes a concern? Is it really enough of a concern to make templating worthwhile vs. the trade off of needing to "build" pages. Generating the all-in-one copy of the documentation can quite trivially be achieved from both markdown and HTML forms, in theory, you're just essentially copying all of the individual pages into one file effectively. If using HTML and no template, then it's still a trivial task to extract the content portion to do this, so that's no bother. Another factor is that my editor of choice - gedit - has syntax highlighting for many formats, but this does not include SiSU. > For the translators, the translation framework is also present, using > the well-known po files. I would argue that for documentation files like the manual, having a translation framework is over the top and inappropriate. The po translation system is well suited to code files because it separates the text strings from the mass of code they are sparsely located within, allowing the translator to focus on the data they need - the strings. It also makes sense considering the use of the translations - the translations are not going into translation specific binaries, they are just translating the string data, for which a single common compiled binary is to make use of when it dynamically loads the right set of language strings at runtime (except when just using the embedded English of course with the gettext system). Compare this to the manual, where a different copy is made for each language, and the content of relevance to translations forms almost the entirety of the source for generating "built" artifacts, thus bringing into question the model of trying to split the content out in the same way, when it might be must more simple to be able to just generate them directly from the files containing translated content, if only permitted to contain a tiny portion of layout if necessary. Of course PO is not perfect. You may be interested by Project Fluent, if you've not come across it before: https://projectfluent.org/ With regard to the manual, breaking up the content of the pages in terms of individual headings, bullet points, paragraphs, etc, in POT/PO files, really does not add much over just having a copy of the original files, very little of which is not pure content. Compare about_manual.ssi and about_manual.ssi.pot. There is no benefit to PO here in terms of focussing on the job at hand; all of the small bits of markup within sentences/paragraphs are carried across, even things like "code{" get grabbed as possibly translatable strings. Compare user_basics.ssi and user_basics.ssi.pot. The first thing in there is "code{", which, since the PO file is trying to avoid duplicate copies of, means that it appears nowhere else in the PO file, meaning that you cannot just go through the PO file, translating, understanding context as perfectly as with the original. PO thus can potentially even be a hindrance to translation of these files. For instance, take this string from the POT: " # apt-get install xorriso\n". It is not necessarily immediately obvious to a translator that this is a shell command and that the word "install" if not all words it contains should not be translated. The surrounding code block markers having been removed potentially damages understanding of context. Of course I'm forgetting thus far that po files have each original string alongside each translation string as an "ID", which may be very helpful for comparison. An alternate that might work perfectly well for these documentation files might be for translators to view the original and translation side by side, or in a split 2-file view in an editor. Of course perhaps po4a actually works with HTML and markdown files, I'm not sure. I just did a brief google search and did not get a clear answer. If so then the issue of moving away from po4a is rather redundant is it not, if it is considered worth keeping for the translation workflow, compared to side-by-side / split-dual-view comparison based work as just described. Since, as I repeat, these files are almost entirely pure content, and po damages the view of that actual content in cases and thus damages understanding important context. The other issue of course is translators identifying portions of text that have changed where updating translation is necessary. If keeping po then this aspect is irrelevant. If moving away from it, then translators would either reply upon side-by-side / split-dual-view comparison, going through line by line; review the commits to the repo and update following the changes made in them; or get a diff of the English version from when the translation was last updated up to present, and work from that. > > Just at the minute I'm taking a brief pause in my other work to > > consider the possibility of whether there is a better, more modern, > > lightweight, etc tool that could replace use of SiSU. Something > > markdown/commonmark based perhaps. Or maybe if we just used plain > > HTML > > directly, with a conversion tool to produce PDFs (if we want to be > > able > > to make them). > > > > I don't think we need care about the database backed "search" > > mechanism > > SiSU provides. > > Back when Alioth was still running, the database backend functioned > as a > kind of search engine. I'm not sure that I follow. I presumed that the SiSU search component was a feature of generating a search facility for searching within the content of a SiSU formatted set of documents. So the database generates and stores a pre-compiled set of data with which to respond to searches with. I am familiar with a similar thing being a part of documentation generated for Rust projects via `cargo doc`, only I believe it just compiles information into javascript, rather than use a DBMS. I do not consider such a search facility to be of value to the live-build manual. It does not consist of many pages and you can easily (a) use the in-page search facility of your web browser either in multi-page or single-page view (b) use the search facility in your PDF viewer if using the PDF (c) use search engine to find the right page for multi page view. > > And I expect we don't need the same translation stuff we > > use for code; translations can just be copies of the english files, > > translated... > > I disagree here. Having a standard translation framework in place is > important to me. For me, translated files should take over the > structure > and as much of the markup from the original file as possible, and > preferably only replace the English content. Again, as above, _very_ little of the content of the files, now and under the alternate formats, is not content or markup that naturally and already finds its way into the translation files. I do not expect any notable difference under markdown/HTML if continuing to use po. > > Anyway, I just wanted to know whether you were already working on > > anything in this area or just rewriting the text. Also I was > > wondering > > how near completion you might be, as I'd not want to start any > > effort > > (if I was to do it myself) to convert to a different markup and > > such > > until your substantial rewrite was available to do it on, otherwise > > it > > would just create a lot of extra work for one of us to rework > > things on > > top of the other's changes obviously. > > I would rather ask whether any of the team members would be willing > to > do a review if all files will be touched. As in thoroughly compare old and new side by side word for word? Personally I would not be overly concerned about review, provided a discussion and agreement takes place on the direction of the work beforehand regarding selection of format, and use of templating and translation. Considering all the benefits, not least moving away from relying upon an essentially obsolete build tool, then I think there's good chance they'd agree. In terms of what the reviewer might do in their review of such a change, I would not necessarily expect them to bother to look all that closely at such a change, at least if it were me submitting it rather than someone new here. I have to some degree built up a working relationship with Raphaël and Luca. I do not know whether that relationship will ever extend to team membership, but I've been working with them over the past few weeks getting a substantial amount of work merged into live-build. Furthermore (1) we're just talking about chunks of documentation here, not code where bugs including security vulnerabilities can be easily introduced by the smallest flaw (unintentional or deliberate), (2) it's not like they'd expect it likely that I, having established a good working relationship with them, would suddenly jeopardise that by playing some prank submitting a vandalised work, or that it likely that I'd screw things up royally performing such a format conversion, such as loosing a paragraph. I would thus not expect them to be spending time doing a word by word comparison. I'd expect little more than a cursory look at how I'd implemented the new format, and a quick check that the build/translation functionality works as before; trusting me that the content is unchanged. Of course I do appreciate that they already have their work cut out reviewing all of my live-build contribution work as it is. But then this would not necessarily need to get reviewed and merged immediately, it could sit in salsa pending merge for a little while until their time frees up enough (or they give me access and freedom to do it myself, assuming they agree with the nature of the change in principle). > My proposal: > * First gather consensus from the team whether a change is needed > * When so, decide on a solution that matches the requirements Of course. As I said above, I was just seeking out some info on the nature of your work at this stage, prior to making any possible enquiry/exploration of format change at a later date. > Let me try to summarise the current state: > > As-is state: > * The documentation is written in SiSU > * The output is available in PDF (A4 and letter, each both in > portrait > and landscape), HTML, epub, odf and plain text `markdown` can convert from markdown to HTML. I'm not sure if it can do more. `pandoc` can convert from markdown to HTML, PDF (using LaTeX), ODT, docx, RTF, plaintext, EPUB, and others. I don't have any experience of using these, they're just a couple of tools I came across doing a brief bit of research yesterday. So thus I do not know about portrait vs. landscape and A4 vs. letter, though I don't see why we'd care about anything other than portrait A4... > * Translations in all document formats are generated for 9 languages > (ca, de, es, fr, it, ja, pl, pt_BR, ro) po4a may well work just fine for markdown/HTML, so no difference if so; Just perhaps requires a bit of effort to migrate existing translations. (If we care to, perhaps they're so out of date by now the existing copies should just be ditched?) > * Dead links can be found using linkchecker on the html output We're getting HTML output in any case; Whether we go with markdown or actual HTML, we will always want HTML output for the web hosted copy. > * sisu-complete pulls in many packages > * The latest build from the git branch master is available on > https://live-team.pages.debian.net/live-manual/ > * live-manual is packaged in Debian as live-manual, live-manual-epub, > live-manual-html, live-manual-odf and live-manual-pdf Interesting, I actually had not noticed the existence of all of those actual packages. I'm not quite sure whether I'd agree that this is the ideal means of distributing the manual. I mean in terms of the PDF form, live-manual-pdf dumps 40 files into /usr/share/doc/live- manual/pdf/, with different orientations, and different languages, and compressed into archives. Is that really a suitable/desirable/useful way to distribute the manual? > Positive notes: > * Sisu is well-supported with syntax highlighting As mentioned above, no highlighting in my editor - gedit. Being little used is not going to encourage package maintenance. > * Proof-reading the English text is done by 'make PROOF=1', which > takes > only about 8 seconds on my computer. > > Negative notes: > * Sisu has a few reverse build dependencies > * sisu-complete brings a large list of dependencies, but can be > replaced > by a smaller list > > Personally, I see no immediate need to have this large > transition/rewrite right now. > A task within the live team, that I see will be more pressing, will > be > the generation of the standard live images (about which I shortly > wrote > on 2020-03-21T17:27). Current Debian Stable images are built with > live-wrapper, which uses vmdebootstrap under the hood. vmdebootstrap > depends on Python 2, and will not be present in the next version of > Debian. I am not aware of something that provides a 1-to-1 > replacement > that will work on Debian Testing (and therefore the next release of > Debian). If you mean bullseye, I expect that it won't be until some time next year that that gets released, so I don't see a pressing concern for focussing on a move away fro live-wrapper there. Also, the live images were previously produced with live-build until live-wrapper came into existence a few years back and they moved to that, as far as I understand it. With live-wrapper and vmdebootstrap being abandoned, I would expect that it should not be a big deal for them to just move back to live-build... Furthermore, live-manual is currently live-build focussed only, and although live-wrapper and live-build both have the maintainer listed as the Debian live team, effectively they have different groups of people working on them. Since the original author of live-build left, live-build has been in low maintenance mode, with Raphaël Hertzog holding ownership, and sharing upload responsibility with Luca Boccassi, as I understand it. It is these two who have been reviewing and merging my contributions these past few weeks. Effectively I personally have been working full time just now on live-build improving many deficiencies in the codebase and fixing various bugs. Though I am not on the team myself. With live-wrapper, if you checkout debian/control and review the commit history, it is Iain Learmonth and Steve McIntyre who are the main two who have been developing it, and the two of them along with Jonathan Carter who hold upload responsibility. So although all of them are grouped under the banner of the Debian live team, and are members of owner or maintainer status in its salsa project, it's two different sets of people developing and maintaining the two different tools. Furthermore the original author of live-build was just the developer of that tool, I do not believe he was ever a member of the Debian team responsible for releasing official images. I've lost touch with who the team members are, if I ever knew. Perhaps those involved in live- wrapper belong to it? Perhaps not. I recall Steve being a notable member of the project from past contribution work I did for Debian, but I don't know if releasing images is one of his areas of responsibility. So, live-manual is currently live-build oriented only. If a discussion of expanding it to cover live-wrapper ever took place, I'm not aware of it, and in any case it would be pointless now that live-wrapper is looking dead. With live-manual being live-build only, and with the live-build, live-wrapper, and debian-image-team (whatever we should call them) being separate groups of people (or at least the live-build people being separate), I don't expect that the effort of which you speak regarding migrating away from live-wrapper for official images is of any concern to the workload of those working on live-build and its documentation in live-manual.
