Yue Yi <[email protected]> writes: >> I am not sure what is the problem here. Could you please elaborate? > > Sure. > > With the patch applied, the value of `org-element-emphasis-post-re' is: > > "[[:space:]]\\|\\c}\\|\\c]\\|\\c-\\|\\c,\\|\\c|\\|$" > > As you can see, this explicitly includes the line-end anchor "$". > > However, in `org-set-emph-re', the template string used in concat ends > with "\\([%s]\\|$\\)". This means the post parameter is injected into a > bracket expression [...].
Got it. It was a mistake. I replaced the opening part, but forgot the closing. See the attached updated patch. >> > Apart from that, your code works great. I look forward to using this in >> > Emacs 31 to get rid of the annoying ZWS. Though we'll still need them >> > for English text (like a*b*c), that's a separate discussion. >> >> We are far from there. I am mostly toying around whether this syntax is >> going to break Org or not. >> >> There are still problems with the proposed approach. In particular, >> using Po Unicode character category might be problematic. >> "!?.," are all Po, but we currently just allow them as right boundary. >> This makes sense since !* is probably intentional - in English, ! is >> end of sentence and should be followed by a space. So, it is unlikely to >> be expected as a left boundary of emphasis. >> 、 。 ！ ， ． ？ are also Po, but I am not sure whether one may expect >> to write 您好。*我*叫Ihor。 > > Yes, this is exactly what we expect. > > Unlike English, CJK languages do not use spaces to separate sentences or > phrases. The punctuation marks themselves act as the delimiters. What about other Po characters like in https://www.compart.com/en/unicode/category/Po ? > If we were forced to add a space (e.g., 您好。 *我*...), it would create > an unnaturally wide gap in the text, which is considered a typographic > error in Chinese. Therefore, allowing punctuation as a left boundary is good. > > I'd like to provide or do some tests if needed. Tests would be helpful, yes. But before that, I need to think hard about what to do with .,!? and similar English punctuation. We might possibly consult https://www.unicodecharacter.org/property/Line_Break/CL (Line_Break property = close punctuation), but 。 is also CL. But then ⁈ has Line_Break = Nonstarter, and we might want to allow ⁈ after emphasis. So, Line_Break is a shaky metrics. Maybe "Terminal Punctuation" property.
>From 19126b3e0978141a494b9c542d02734533bed147 Mon Sep 17 00:00:00 2001 Message-ID: <19126b3e0978141a494b9c542d02734533bed147.1766686671.git.yanta...@posteo.net> From: Ihor Radchenko <[email protected]> Date: Sat, 20 Dec 2025 11:58:16 +0100 Subject: [PATCH v2] WIP: Org markup: Allow Unicode punctuation and breakable symbols around emphasis * lisp/org-element.el (org-element-category-table): Define custom category table adding opening/closing punctuation, opening/closing quotes, dashes, and auxiliary punctuation. (org-element--parse-generic-emphasis): Extend allowed characters around emphasis to generic opening/closing punctuation, quote punctuation, dash-likes, and auxiliary ,-like punctuation. Also, allow breakable characters, like Chinese/Japanese symbols for languages that do not use spaces. * lisp/org.el (org-element-emphasis-pre-re): (org-element-emphasis-post-re): Add new regexps defining emphasis boundaries. * lisp/org.el (org-mode): Setup category table. (org-emphasis-regexp-components): Allow pre/post to be nil to follow the new defaults. Change the default values of pre/past to nil. (org-set-emph-re): (org-do-emphasis-faces): (org-emphasize): Fall back to parser defaults when pre/past in `org-emphasis-regexp-components' is nil. --- lisp/org-element.el | 37 ++++++++++++++++-- lisp/org.el | 93 ++++++++++++++++++++++++++++++--------------- 2 files changed, 97 insertions(+), 33 deletions(-) diff --git a/lisp/org-element.el b/lisp/org-element.el index 6abccd001..c22b68695 100644 --- a/lisp/org-element.el +++ b/lisp/org-element.el @@ -3318,6 +3318,38 @@ ;;; Objects ;;;; Bold +(defvar org-element-category-table + (let ((category-table (copy-category-table)) + (uniprop-table (unicode-property-table-internal 'general-category))) + ;; Define categories + (define-category ?{ "Opening punctuation" category-table) + (define-category ?} "Closing punctuation" category-table) + (define-category ?\[ "Initial quote" category-table) + (define-category ?\] "Final quote" category-table) + (define-category ?- "Dash" category-table) + (define-category ?, "Other punctuation" category-table) + ;; Map characters to categories according to their general-category + (map-char-table + (lambda (key val) + (pcase val + ('Ps (modify-category-entry key ?{ category-table)) + ('Pe (modify-category-entry key ?} category-table)) + ('Pi (modify-category-entry key ?\[ category-table)) + ('Pf (modify-category-entry key ?\] category-table)) + ('Pd (modify-category-entry key ?- category-table)) + ('Po (modify-category-entry key ?, category-table)))) + uniprop-table) + category-table) + "Category table for Org buffers. +The table defines additional Unicode categories: +- ?{ for opening punctuation +- ?} for closing punctuation +- ?[ for opening quote +- ?] for closing quote +- ?- for dash-like +- ?, for other punctuation. +These categories are necessary for parsing emphasis.") + (defun org-element--parse-generic-emphasis (mark type) "Parse emphasis object at point, if any. @@ -3331,7 +3363,7 @@ (defun org-element--parse-generic-emphasis (mark type) (unless (bolp) (forward-char -1)) (let ((opening-re (rx-to-string - `(seq (or line-start (any space ?- ?\( ?' ?\" ?\{)) + `(seq (regexp ,org-element-emphasis-pre-re) ,mark (not space))))) (when (looking-at-p opening-re) @@ -3341,8 +3373,7 @@ (defun org-element--parse-generic-emphasis (mark type) `(seq (not space) (group ,mark) - (or (any space ?- ?. ?, ?\; ?: ?! ?? ?' ?\" ?\) ?\} ?\\ ?\[) - line-end))))) + (regexp ,org-element-emphasis-post-re))))) (when (re-search-forward closing-re nil t) (let ((closing (match-end 1))) (goto-char closing) diff --git a/lisp/org.el b/lisp/org.el index 67d9679fe..eee0e25ca 100644 --- a/lisp/org.el +++ b/lisp/org.el @@ -3823,6 +3823,31 @@ (defcustom org-pretty-entities-include-sub-superscripts t :version "24.1" :type 'boolean) +(defconst org-element-emphasis-pre-re + (rx (or line-start space + ;; opening punctuation + (category ?{) (category ?\[) + ;; dashes, other punctuation + (category ?-) (category ?,) + ;; Chinese, Japanese, and other breakable + ;; characters + (category ?|))) + "Regexp matching character before opening emphasis marker. +Assumes that `org-element-category-table' is active.") + +(defconst org-element-emphasis-post-re + (rx (or space + ;; closing punctuation + (category ?}) (category ?\]) + ;; dashes, other punctuation + (category ?-) (category ?,) + ;; Chinese, Japanese, and other breakable + ;; characters + (category ?|) + line-end)) + "Regexp matching character after opening emphasis marker. +Assumes that `org-element-category-table' is active.") + (defvar org-emph-re nil "Regular expression for matching emphasis. After a match, the match groups contain these elements: @@ -3850,10 +3875,13 @@ (defun org-set-emph-re (var val) (body (if (<= nl 0) body (format "%s*?\\(?:\n%s*?\\)\\{0,%d\\}" body body nl))) (template - (format (concat "\\([%s]\\|^\\)" ;before markers + ;; See `org-element--parse-generic-emphasis' + (format (concat "\\(%s\\)" ;before markers "\\(\\([%%s]\\)\\([^%s]\\|[^%s]%s[^%s]\\)\\3\\)" - "\\([%s]\\|$\\)") ;after markers - pre border border body border post))) + "\\(%s\\)") ;after markers + (if pre (format "[%s]\\|^" pre) org-element-emphasis-pre-re) + border border body border + (if post (format "[%s]\\|$" post) org-element-emphasis-post-re)))) (setq org-emph-re (format template "*/_+")) (setq org-verbatim-re (format template "=~"))))) @@ -3861,7 +3889,7 @@ (defun org-set-emph-re (var val) ;; set this option proved cumbersome. See this message/thread: ;; https://orgmode.org/list/[email protected] (defvar org-emphasis-regexp-components - '("-[:space:]('\"{" "-[:space:].,:!?;'\")}\\[" "[:space:]" "." 1) + '(nil nil "[:space:]" "." 1) "Components used to build the regular expression for FONTIFYING emphasis. WARNING: This variable only affects visual fontification, but does not change Org markup. For example, it does not affect how emphasis markup @@ -3874,7 +3902,9 @@ (defvar org-emphasis-regexp-components specify what is allowed/forbidden in each part: pre Chars allowed as prematch. Beginning of line will be allowed too. + nil means use parser defaults. post Chars allowed as postmatch. End of line will be allowed too. + nil means use parser defaults. border The chars *forbidden* as border characters. body-regexp A regexp like \".\" to match a body character. Don't use non-shy groups here, and don't allow newline here. @@ -5106,6 +5136,9 @@ (define-derived-mode org-mode outline-mode "Org" (org-install-agenda-files-menu)) (setq-local outline-regexp org-outline-regexp) (setq-local outline-level 'org-outline-level) + (require 'org-element) + (defvar org-element-category-table) ; org-element.el + (set-category-table org-element-category-table) ;; Initialize cache. (org-element-cache-reset) (when (and org-element-cache-persistent @@ -5385,35 +5418,27 @@ (defsubst org-rear-nonsticky-at (pos) (defun org-do-emphasis-faces (limit) "Run through the buffer and emphasize strings." - (let ((quick-re (format "\\([%s]\\|^\\)\\([~=*/_+]\\)" - (car org-emphasis-regexp-components)))) + (let ((quick-re (format "\\(%s\\)\\([~=*/_+]\\)" + (if (car org-emphasis-regexp-components) + (format "[%s]\\|^" (car org-emphasis-regexp-components)) + org-element-emphasis-pre-re)))) (catch :exit (while (re-search-forward quick-re limit t) (let* ((marker (match-string 2)) - (verbatim? (member marker '("~" "=")))) + (verbatim? (member marker '("~" "="))) + (context (save-match-data + (save-excursion + (goto-char (match-beginning 2)) + (org-element-context))))) (when (save-excursion (goto-char (match-beginning 0)) (and - ;; Do not match table hlines. - (not (and (equal marker "+") - (org-match-line - "[ \t]*\\(|[-+]+|?\\|\\+[-+]+\\+\\)[ \t]*$"))) - ;; Do not match headline stars. Do not consider - ;; stars of a headline as closing marker for bold - ;; markup either. - (not (and (equal marker "*") - (save-excursion - (forward-char) - (skip-chars-backward "*") - (looking-at-p org-outline-regexp-bol)))) + (org-element-type-p + context + '(bold code italic strike-through underline verbatim)) + (equal (match-beginning 2) (org-element-begin context)) ;; Match full emphasis markup regexp. - (looking-at (if verbatim? org-verbatim-re org-emph-re)) - ;; Do not span over paragraph boundaries. - (not (string-match-p org-element-paragraph-separate - (match-string 2))) - ;; Do not span over cells in table rows. - (not (and (save-match-data (org-match-line "[ \t]*|")) - (string-match-p "|" (match-string 4)))))) + (looking-at (if verbatim? org-verbatim-re org-emph-re)))) (pcase-let ((`(,_ ,face ,_) (assoc marker org-emphasis-alist)) (m (if org-hide-emphasis-markers 4 2))) (font-lock-prepend-text-property @@ -5478,12 +5503,20 @@ (defun org-emphasize (&optional char) (setq string (concat s string s)) (when beg (delete-region beg end)) (unless (or (bolp) - (string-match (concat "[" (nth 0 erc) "\n]") - (char-to-string (char-before (point))))) + (string-match + (if (nth 0 erc) (concat "[" (nth 0 erc) "\n]") + ;; See `org-element--parse-generic-emphasis' + (rx (or (regexp org-element-emphasis-pre-re) + "\n"))) + (char-to-string (char-before (point))))) (insert " ")) (unless (or (eobp) - (string-match (concat "[" (nth 1 erc) "\n]") - (char-to-string (char-after (point))))) + (string-match + ;; See `org-element--parse-generic-emphasis' + (if (nth 1 erc) (concat "[" (nth 1 erc) "\n]") + (rx (or (regexp org-element-emphasis-post-re) + "\n"))) + (char-to-string (char-after (point))))) (insert " ") (backward-char 1)) (insert string) (and move (backward-char 1)))) -- 2.52.0
-- Ihor Radchenko // yantar92, Org mode maintainer, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92>
