Hi Ihor, thanks a lot for the patches and explanations. Your assessment makes sense and I agree that we should stick to the current design as much as possible. I will go ahead and adapt my ox-html.el code the way you propose and use your patched ox.el for this. It shouldn't take too much effort and I will get back as soon as I have results (or questions ;-).
-- Orm Am Mittwoch, den 24. Juli 2024 um 10:20:16 Uhr (+0000) schrieb Ihor Radchenko: > Orm Finnendahl <orm.finnend...@selma.hfmdk-frankfurt.de> writes: > > > To recapitulate: In my code, org-export-as calls process-multipage in > > the backend. This function: > > > > - collects and adds information necessary for org-multipage to do its > > job, splitting the document into different parts, etc. and > > > > - then calls org-export-data on the subtrees and exports each returned > > string to an individual file. > > > > - It finally issues a done string and executes a browser open/visit > > file or simply exits nil. > > Currently, org-export-as does the following: > > 1. Compute global export attributes, according to the selected export backend > 2. Copy original buffer into working copy > 3. Process and parse the copy, generating AST > 4. Do the actual export > > You plugged your multipage processing into (4), but what it actually > does involves (3), (4), and also a new kind of post-processing. > I do not think that it is a good design from the point of view of ox.el. > I prefer to reuse or extend the existing mechanisms if at all possible - > this makes new features less confusing for users and backend developers. > > > - collects and adds information necessary for org-multipage to do its > > job, splitting the document into different parts, etc. and > > What you describe here is more or less what :filter-parse-tree filters > do - they can rearrange the parse tree before passing it to the > transcoders. Why not reusing it for multipage export? > > > - then calls org-export-data on the subtrees and exports each returned > > string to an individual file. > > And you simply call `org-export-transcode-page' for this, followed by > writing the returned string to file. > > The first part can fit within `org-export-as', but writing to file is > going a step beyond, duplicating what `org-export-to-file' does. > > > - It finally issues a done string and executes a browser open/visit > > file or simply exits nil. > > ... which again steps beyond `org-export-as' scope - post-processing is > currently done as a part of `org-export-to-file'/`org-export-to-buffer'. > > ---- > > Let me propose the following changes to ox.el: > > 1. org-data will be transcoded using `org-export-transcode-org-data', > which can be overridden by setting org-data transcoders in the > individual backends. > > 2. org-export-as will understand transcoded output to be a list of > strings and will transfer INFO plist as text property in the return > values > > 3. org-export-to-file will make use of the text properties to retrieve > the file name to write. This way, export backend itself can assign > the file names where each exporter string should go. > > I believe that my changes should allow you to implement multipage export > in the following way: > > 1. You can use :filter-parse-tree in ox-html backend to replace the > original (org-data ...) AST with a list of > ((org-page ...) (org-page ...) ...) pseudo-elements and populate INFO > channel with auxiliary information you now compute in > `org-html-process-multipage' > > 2. You can define org-page transcoder to render individual pages as > needed > > 3. You can assign :output-file text property to the returned org-page > strings and use org-export-to-file to generate the multipage output > on disk > > 4. You can handle opening exported files by augmenting POST-PROCESS > argument in `org-html-export-to-multipage-html' and calling > `org-export-file' instead of `org-export-as'. > > The tentative patches (against Org mode main branch) implementing my > changes are attached. > > From 540c8ef21c26df79cf48f58afb4e88130985e2f7 Mon Sep 17 00:00:00 2001 > Message-ID: > <540c8ef21c26df79cf48f58afb4e88130985e2f7.1721815865.git.yanta...@posteo.net> > From: Ihor Radchenko <yanta...@posteo.net> > Date: Wed, 24 Jul 2024 11:40:57 +0200 > Subject: [PATCH 1/3] ox: Factor out org-data transcoding into dedicated > overrideable transcoder > > * lisp/ox.el (org-export-transcode-org-data): New function serving as > the default transcoder for org-data export. > (org-export-transcoder): Use `org-export-transcode-org-data' when no > org-data transcoder is defined. > (org-export-as): Rely upon org-data transcoder to do its job. > --- > lisp/ox.el | 55 +++++++++++++++++++++++++++++------------------------- > 1 file changed, 30 insertions(+), 25 deletions(-) > > diff --git a/lisp/ox.el b/lisp/ox.el > index fbd9bb0df..bdee71082 100644 > --- a/lisp/ox.el > +++ b/lisp/ox.el > @@ -1883,9 +1883,11 @@ (defun org-export-transcoder (blob info) > INFO is a plist containing export directives." > (let ((type (org-element-type blob))) > ;; Return contents only for complete parse trees. > - (if (eq type 'org-data) (lambda (_datum contents _info) contents) > - (let ((transcoder (cdr (assq type (plist-get info :translate-alist))))) > - (and (functionp transcoder) transcoder))))) > + (let ((transcoder (cdr (assq type (plist-get info :translate-alist))))) > + (cond > + ((functionp transcoder) transcoder) > + ;; Use default org-data transcoder unless specified. > + ((eq type 'org-data) #'org-export-transcode-org-data))))) > > (defun org-export--keep-spaces (data info) > "Non-nil, when post-blank spaces after removing DATA should be preserved. > @@ -3004,31 +3006,34 @@ (defun org-export-as > backend info subtreep visible-only ext-plist)) > ;; Eventually transcode TREE. Wrap the resulting string into > ;; a template. > - (let* ((body (org-element-normalize-string > - (or (org-export-data (plist-get info :parse-tree) info) > - ""))) > - (inner-template (cdr (assq 'inner-template > - (plist-get info > :translate-alist)))) > - (full-body (org-export-filter-apply-functions > - (plist-get info :filter-body) > - (if (not (functionp inner-template)) body > - (funcall inner-template body info)) > - info)) > - (template (cdr (assq 'template > - (plist-get info :translate-alist)))) > - (output > - (if (or (not (functionp template)) body-only) full-body > - (funcall template full-body info)))) > + (let ((output > + (or (org-export-data (plist-get info :parse-tree) info) > + ""))) > ;; Call citation export finalizer. > (when (plist-get info :with-cite-processors) > (setq output (org-cite-finalize-export output info))) > - ;; Remove all text properties since they cannot be > - ;; retrieved from an external process. Finally call > - ;; final-output filter and return result. > - (org-no-properties > - (org-export-filter-apply-functions > - (plist-get info :filter-final-output) > - output info))))))))) > + (let ((filters (plist-get info :filter-final-output))) > + ;; Remove all text properties since they cannot be > + ;; retrieved from an external process. Finally call > + ;; final-output filter and return result. > + (org-no-properties > + (org-export-filter-apply-functions filters output > info)))))))))) > + > +(defun org-export-transcode-org-data (_ body info) > + "Transcode `org-data' node with BODY. Return transcoded string. > +INFO is the communication channel plist." > + (let* ((inner-template (cdr (assq 'inner-template > + (plist-get info :translate-alist)))) > + (full-body (org-export-filter-apply-functions > + (plist-get info :filter-body) > + (if (not (functionp inner-template)) body > + (funcall inner-template body info)) > + info)) > + (template (cdr (assq 'template > + (plist-get info :translate-alist)))) > + (body-only (memq 'body-only (plist-get info :export-options)))) > + (if (or (not (functionp template)) body-only) full-body > + (funcall template full-body info)))) > > (defun org-export--annotate-info (backend info &optional subtreep > visible-only ext-plist) > "Annotate the INFO plist according to the BACKEND. > -- > 2.45.2 > > From 1b0b331f92abc1ca7e04f71fe7ff60da57c719b8 Mon Sep 17 00:00:00 2001 > Message-ID: > <1b0b331f92abc1ca7e04f71fe7ff60da57c719b8.1721815865.git.yanta...@posteo.net> > In-Reply-To: > <540c8ef21c26df79cf48f58afb4e88130985e2f7.1721815865.git.yanta...@posteo.net> > References: > <540c8ef21c26df79cf48f58afb4e88130985e2f7.1721815865.git.yanta...@posteo.net> > From: Ihor Radchenko <yanta...@posteo.net> > Date: Wed, 24 Jul 2024 11:51:21 +0200 > Subject: [PATCH 2/3] org-export-as: Allow the return value to be a list of > strings; add INFO > > * lisp/ox.el (org-export-as): Allow the transcoders to return list of > strings and return it. When returning a string, put INFO plist as > text property. Do not remove text properties assigned by the > transcoders. > (org-export-data): Document that list of strings may be returned. > --- > lisp/ox.el | 28 ++++++++++++++++++++-------- > 1 file changed, 20 insertions(+), 8 deletions(-) > > diff --git a/lisp/ox.el b/lisp/ox.el > index bdee71082..a76b3b353 100644 > --- a/lisp/ox.el > +++ b/lisp/ox.el > @@ -1930,7 +1930,7 @@ (defun org-export-data (data info) > > The `:filter-parse-tree' filters are not applied. > > -Return a string." > +Return a string or a list of strings." > (or (gethash data (plist-get info :exported-data)) > ;; Handle broken links according to > ;; `org-export-with-broken-links'. > @@ -2969,7 +2969,9 @@ (defun org-export-as > with external parameters overriding Org default settings, but > still inferior to file-local settings. > > -Return code as a string." > +Return code as a string or a list of strings. > +The returned strings will have their `org-export-info' property set to > +export information channel." > (when (symbolp backend) (setq backend (org-export-get-backend backend))) > (org-export-barf-if-invalid-backend backend) > (org-fold-core-ignore-modifications > @@ -3009,15 +3011,25 @@ (defun org-export-as > (let ((output > (or (org-export-data (plist-get info :parse-tree) info) > ""))) > + (setq output (ensure-list output)) > ;; Call citation export finalizer. > (when (plist-get info :with-cite-processors) > - (setq output (org-cite-finalize-export output info))) > + (setq output > + (mapcar > + (lambda (o) (org-cite-finalize-export o info)) > + output))) > (let ((filters (plist-get info :filter-final-output))) > - ;; Remove all text properties since they cannot be > - ;; retrieved from an external process. Finally call > - ;; final-output filter and return result. > - (org-no-properties > - (org-export-filter-apply-functions filters output > info)))))))))) > + ;; Call final-output filter and return result. > + (setq output > + (mapcar > + (lambda (o) (org-export-filter-apply-functions filters > o info)) > + output))) > + ;; Apply org-export-info property. > + (setq output > + (mapcar > + (lambda (o) (org-add-props o nil 'org-export-info info)) > + output)) > + (if (length= output 1) (car output) output)))))))) > > (defun org-export-transcode-org-data (_ body info) > "Transcode `org-data' node with BODY. Return transcoded string. > -- > 2.45.2 > > From 6fa2efadd229a667fba1b18aecc9d1ead5f284ac Mon Sep 17 00:00:00 2001 > Message-ID: > <6fa2efadd229a667fba1b18aecc9d1ead5f284ac.1721815865.git.yanta...@posteo.net> > In-Reply-To: > <540c8ef21c26df79cf48f58afb4e88130985e2f7.1721815865.git.yanta...@posteo.net> > References: > <540c8ef21c26df79cf48f58afb4e88130985e2f7.1721815865.git.yanta...@posteo.net> > From: Ihor Radchenko <yanta...@posteo.net> > Date: Wed, 24 Jul 2024 12:09:36 +0200 > Subject: [PATCH 3/3] org-export-to-file: Derive file name to write from export > output > > * lisp/ox.el (org-export--write-output): New helper function > performing writing an export output or a list of outputs to file. It > derives the file name from :output-file property in the output string > or INFO plist stored in the output string. > (org-export-to-file): Handle export output being a list of strings. > Use `org-export--write-output'. > --- > lisp/ox.el | 61 ++++++++++++++++++++++++++++++++++-------------------- > 1 file changed, 38 insertions(+), 23 deletions(-) > > diff --git a/lisp/ox.el b/lisp/ox.el > index a76b3b353..d78c04998 100644 > --- a/lisp/ox.el > +++ b/lisp/ox.el > @@ -6830,6 +6830,31 @@ (defun org-latex-export-as-latex > (switch-to-buffer-other-window buffer)) > buffer))) > > +(defun org-export--write-output (output encoding) > + "Write OUTPUT to file with ENCODING. > +OUTPUT may be a string or a list of strings. > +The target file is retrieved from :output-file OUTPUT property or > +:output-file property in plist stored in `org-export-info' property of > +each string. > + > +Return the file name or a list of file names." > + (if (listp output) (mapcar #'org-export--write-output output) > + (let ((file (or > + (get-text-property 0 :output-file output) > + (plist-get > + (get-text-property 0 'org-export-info output) > + :output-file)))) > + (with-temp-buffer > + (insert output) > + ;; Ensure final newline. This is what was done > + ;; historically, when we used `write-file'. > + ;; Note that adding a newline is only safe for > + ;; non-binary data. > + (unless (bolp) (insert "\n")) > + (let ((coding-system-for-write encoding)) > + (write-region nil nil file)) > + file)))) > + > ;;;###autoload > (defun org-export-to-file > (backend file &optional async subtreep visible-only body-only ext-plist > @@ -6878,33 +6903,23 @@ (defun org-latex-export-to-latex > `(let ((output > (org-export-as > ',backend ,subtreep ,visible-only ,body-only > - ',ext-plist))) > - (with-temp-buffer > - (insert output) > - ;; Ensure final newline. This is what was done > - ;; historically, when we used `write-file'. > - ;; Note that adding a newline is only safe for > - ;; non-binary data. > - (unless (bolp) (insert "\n")) > - (let ((coding-system-for-write ',encoding)) > - (write-region nil nil ,file))) > - (or (ignore-errors (funcall ',post-process ,file)) ,file))) > + ',ext-plist)) > + file) > + (setq file (org-export--write-output output ',encoding)) > + (let ((post (lambda (f) (or (ignore-errors (funcall > ',post-process f)) f)))) > + (if (listp file) (mapcar post file) (funcall post file))))) > (let ((output (org-export-as > - backend subtreep visible-only body-only ext-plist))) > - (with-temp-buffer > - (insert output) > - ;; Ensure final newline. This is what was done > - ;; historically, when we used `write-file'. > - ;; Note that adding a newline is only safe for > - ;; non-binary data. > - (unless (bolp) (insert "\n")) > - (let ((coding-system-for-write encoding)) > - (write-region nil nil file))) > + backend subtreep visible-only body-only ext-plist)) > + file) > + (setq file (org-export--write-output output encoding)) > (when (and (org-export--copy-to-kill-ring-p) (org-string-nw-p > output)) > (org-kill-new output)) > ;; Get proper return value. > - (or (and (functionp post-process) (funcall post-process file)) > - file)))))) > + (let ((post (lambda (f) > + (or (and (functionp post-process) > + (funcall post-process f)) > + f)))) > + (if (listp file) (mapcar post file) (funcall post file)))))))) > > (defun org-export-output-file-name (extension &optional subtreep pub-dir) > "Return output file's name according to buffer specifications. > -- > 2.45.2 > > > -- > Ihor Radchenko // yantar92, > Org mode contributor, > Learn more about Org mode at <https://orgmode.org/>. > Support Org development at <https://liberapay.com/org-mode>, > or support my work at <https://liberapay.com/yantar92>
