This is an automated email from the git hooks/post-receive script.
git pushed a commit to branch master
in repository terminology.
View the commit online.
commit 29752822f6056ca4a8b405768ea47ea806100b30
Author: Cedric BAIL <[email protected]>
AuthorDate: Mon Aug 3 13:27:46 2026 -0600
utf8: widen runs of plain ASCII a run at a time
utf8_to_codepoints() walked byte by byte through text that is overwhelmingly
plain printable ASCII, paying the lead-byte dispatch for every character of it.
Scan for the end of the run instead, then widen the whole run in one call.
Compiled in only where there is a vector kernel to call. The scalar kernel walks
the run exactly as the byte loop below it does, so it can only pay for the same
bytes twice: on x86_64 that measured 4-6% slower than not doing it at all on the
SGR and unicode corpora. Gating it at runtime on simd_enabled() is worse still,
since the run path then stays in the loop to be branched over -- that alone cost
~8% on plain ASCII -- so the test is #if, with simd_enabled() kept inside it for
hardware that has the kernels.
The remaining kernel calls are guarded by an inline test on the first byte and a
minimum run length. Without the guard, input that is mostly non-ASCII pays a
call per character to be told the run is empty, which measured as a net loss.
The escape-code suite and the unit tests are registered a second time with
TERMINOLOGY_SIMD_DISABLE set. Where there is a vector kernel that gives the
scalar path -- what every other target runs -- the same end-to-end coverage;
where there is no kernel to disable, the second run is a duplicate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
---
src/bin/utf8.c | 33 +++++++++++++++++++++++++++++++++
tests/meson.build | 15 +++++++++++++++
2 files changed, 48 insertions(+)
diff --git a/src/bin/utf8.c b/src/bin/utf8.c
index 0a1a1610..b6eec7a5 100644
--- a/src/bin/utf8.c
+++ b/src/bin/utf8.c
@@ -1,5 +1,6 @@
#include "private.h"
#include "utf8.h"
+#include "simd/simd.h"
/* How many bytes the sequence introduced by this lead byte occupies, or 0 if it
* cannot start one (a continuation byte, or a length this decoder will not
@@ -54,10 +55,42 @@ utf8_to_codepoints(const char *buf, int len, Eina_Unicode *codepoints,
int *consumed)
{
int i = 0, j = 0;
+#if defined(TERMINOLOGY_HAVE_NEON)
+ /* Hoisted out of the loop: simd_enabled() is a call, and what it reports
+ * cannot change mid-buffer. */
+ Eina_Bool fast = simd_enabled();
+#endif
while (i < len)
{
Eina_Unicode g;
+#if defined(TERMINOLOGY_HAVE_NEON)
+ size_t run;
+
+ /* Plain printable ASCII needs none of the per-character work below, so
+ * widen it a run at a time. The inline test on the first byte keeps
+ * mostly-non-ASCII input from calling the scanner per character.
+ *
+ * Compiled in only where there is a vector kernel to call. The scalar
+ * kernel walks the run exactly as the loop below does, so it can only
+ * pay for the same bytes twice -- and merely leaving this in the loop
+ * to be branched over costs the decoder ~8% on a non-NEON target. */
+ if (fast &&
+ ((unsigned char)buf[i] >= 0x20) && ((unsigned char)buf[i] < 0x7f))
+ run = simd_scan_plain_ascii((const unsigned char *)buf + i,
+ (size_t)(len - i));
+ else
+ run = 0;
+ /* Below this the scan-then-widen pair costs more than it saves. */
+ if (run >= 4)
+ {
+ simd_widen_ascii((const unsigned char *)buf + i, run,
+ codepoints + j);
+ i += (int)run;
+ j += (int)run;
+ continue;
+ }
+#endif
if (buf[i])
{
diff --git a/tests/meson.build b/tests/meson.build
index 25ec8267..257e1ba4 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -28,8 +28,23 @@ if tests
timeout: 600)
endforeach
+ # Again with the vector kernels off, so the scalar path -- what non-NEON
+ # targets run -- has end-to-end coverage too.
+ test('escape-codes-scalar',
+ run_tests,
+ args: ['-v',
+ '-t', tytest.full_path(),
+ '-r', meson.current_source_dir() / 'tests.results',
+ '-d', meson.current_source_dir()],
+ env: {'TERMINOLOGY_SIMD_DISABLE': '1'},
+ depends: tytest,
+ workdir: meson.current_source_dir(),
+ timeout: 300)
+
# The in-process C unit tests compiled into tytest itself.
test('unit', tytest, args: ['all'], timeout: 120)
+ test('unit-scalar', tytest, args: ['all'],
+ env: {'TERMINOLOGY_SIMD_DISABLE': '1'}, timeout: 120)
endif
if benchmarks
--
To stop receiving notification emails like this one, please contact
the administrator of this repository.
