Hi Tim, David, and Gustav,
I am fairly certain that with only a few exceptions it is possible
to specify a context free grammar for org syntax, followed by a second
pass that deals specifically with markup and a few other forms,
notably the reassembly of things like plain lists. The fact that this
is possible because most org constructs are line oriented.
Just a note that the linked parser.rkt [0] is indeed a BNF describing org
syntax in the same style as a bison/yacc grammar. One of the reasons
why I set out to work on this was precisely so that there could be a
reference that could be consulted by the community when questions
about extended org come up.
There are proposals for new syntax that appear on this list with
terrifying frequency, and they are routinely shot down or simply
ignored for good reason, however it is hard to communicate that to
enthusiastic contributors who have an immediate use case that they
want to solve and share and are unlikely to be aware of side effects.
Having a grammar where such issues can be tested empirically should
provide a significant safeguard while also making it easier for
contributors to play with the grammar and see the issues.
In all my work on the grammar I have found maybe 2 or 3 places where
the grammar could be "extended" but it isn't so much extended as it is
regularized, where some parts of org already parse a more complex
grammar while other very similar parts choose not to. Overall the cost
of not parsing certain forms in certain situations adds complexity
rather than reducing it.
The situation for contribution is also further complicated by the fact
that the elisp implementation of org mode is internally inconsistent
in its behavior with regard to the syntax, so great care has to be
taken if someone tries to make and argument based on the behavior of
one component.
All this to say that the need for a conservative approach to changes
and extensions combined with the internally inconsistent behavior of
different parts of the elisp implementation means that the
introduction of new features is extremely difficult because it is hard
to predict the consequences on other parts of org.
Overcoming this is why I started working on the grammar, because
in the absence of a formal spec for what org should do, it is very hard
to make changes to what it is currently doing without having nasty
side effects.
Best!
Tom
0. https://github.com/tgbugs/laundry/blob/next/laundry/parser.rkt note
the upcoming path change (which I will note in the original thread when
it happens).
PS I'm planning to reply to the main thread as well. My short take is
finding a dedicated and responsive maintainer that can take over from
Bastien is a high priority. The only other thing that might help is to
have some way to track outstanding and closed patches, issues, etc.
that is more accessible than trolling through years worth of posts on
this mailing list, but that is a can of worms that has already been
shot down multiple times.
