On 25-07-20 05:23, John Wiegley wrote:
After writing the loeb function in Emacs Lisp recently, it occurs to me that
it could be used to write an evaluation function for Org spreadsheet tables
that is both maximally efficient (in that no cell is calculated twice) and
always converges on a stable end result. The steps are as follows:
1. Use =org-table-to-lisp= to parse the Org table into a Lisp matrix (list of
lists).
Make sense. org-table-to-lisp is lightning fast, even for tables with
1000's of rows.
Beware that the current code distinguishes between small and large
recalculations:
- C-c * single row recomputing
- C-u C-c * full table recomputing
2. Map through all of the cells in this matrix, converting each into a closure
that maps from a matrix of strings to a string.
The work of each such closure is to calculate the formula for that
particular cell using =org-table-eval-formula= — which may involve looking
up the relevant cell, row or column formula from the table’s formula field
— and resolving any cell references by looking up those values in the
matrix argument.
Make sense too. It is easier to work on a Lisp matrix than on the Org
Mode buffer. And less bug-prone.
3. Apply the relevant [[https://github.com/jwiegley/emacs-loeb][loeb]]
function to this matrix. This does the work of resolving all references, so
long as the table represents a directed, acyclic graph of data
dependencies. If it does not, infinite recursion will result in a Lisp
evaluation exception.
Wow! I read the motivation for LOEB here:
https://github.com/quchen/articles/blob/master/loeb-moeb.md
I didn't understood a word. Therefore I do agree, this is the way to go :)
- If I understand, a cell is either a string or a lazy closure able to
compute it.
- As soon as the value of a cell is needed, the closure is computed.
- The closure is discarded and replaced by the sound string value,
ensuring any closure is computed only once.
4. Render the final matrix return by =loeb=, which is a matrix of strings,
back into the Org-table, honoring all formatting directives.
I do precisely that in my orgtbl-aggregate package, with the function
orgtbl-aggregate--elisp-table-to-string
See: https://github.com/tbanel/orgaggregate
This function renders a Lisp matrix into a big Lisp string, ready to be
inserted back into the Org Mode buffer, with correct alignments (left or
right justification) and column width.
(I discovered that it is much, much faster to insert a big string
representing the full table, than inserting row after row, or worse,
cell after cell. The insert function is quite slow).
There is also the column shrinking (when the actual content of a column
is wider than the requested width) to take into account.
The current code also does an intricate handling of possibly folded
regions of the table.
Quite a piece of work. Unless we decide that we do not need such a
sophistication.
When I went looking at org-table.el, I find that it’s very much tailored to
work on the state of the buffer and its “work space”, and may not be so easy
to convert to a functional style, where instead we convert the table to a Lisp
data structure, operate on the data structure, and then render it back out to
a table.
I did the same observation.
Are there any org-table.el masters who might be interested in helping with
such a reworking of the calculation engine?
I'm not a master, far from that. But I bumped into the complexities of
Org Mode tables quite often writing my orgtbl-aggregate package.
For instance, see my mail of July 24 on the mailing list:
"[BUG] org-table-recalculate not fully completing its task"
It would be nice to have a simpler table-Lisp-API, with more
understandable expectations.
So yes, your idea, John, is quite appealing. If implemented, we must
expect to re-discover zillions of fine details coded deep inside the
intricacies of org-table.el. There will be tight links with the
mysterious org-element.el file as well. A huge work ahead!
So probably it would be wise not to attempt to replace all of
org-table.el at once. Instead, I guess the first step should be to
delimit a (small) sub-set of table features to work on.
Maybe table-realignment (bound to TAB) might be a simpler first step (no
loeb). Current realignment tend to be slow. Your loeb idea could then be
built upon this first work, which would have sorted-out issues like
formatting, folding and relationships with org-elements?
For some context:
https://newartisans.com/2025/05/implementing-loeb-emacs-lisp/
From Wikipedia:
Löb's theorem states that in Peano arithmetic (PA) (or any formal system
including PA), for any formula P, if it is provable in PA that "if P is
provable in PA then P is true", then P is provable in PA.
Well, well, well…
