Prefilter rune-mode input

asciiFuzzyIndex gave up on non-ASCII lines, so every item ran the full
score matrix. A []rune is a fixed 4-byte stride, so the SIMD byte
scanners can run over it directly: find the low byte, then confirm
4-byte alignment and three zero bytes.

Case folding and normalization can turn a non-ASCII rune into the ASCII
char being searched, which the scan cannot see. ToChars now flags lines
holding such a rune and those keep the old path. Normalization is
Latin-only, so Hangul, CJK, Cyrillic, Greek, Hebrew, Arabic, Thai, kana
and emoji never set the flag.

Chars had no spare padding, so inBytes moves into a flags byte.

Measured on 1.4M-line corpora:

- Mostly-ASCII paths behind a Hangul prefix: 'conf' 1.8x, 'binutils'
  2.9x, 'ltversion' 4.0x, no-match 8.4x
- Every line CJK: 17x on both matching and non-matching queries
- ASCII input unchanged, non-ASCII patterns not covered yet
This commit is contained in:
Junegunn Choi
2026-08-08 19:48:04 +09:00
parent 793e58b558
commit a650900eda
8 changed files with 641 additions and 18 deletions
+1 -1
View File
@@ -102,7 +102,7 @@ itest:
# FUZZTIME (e.g. make fuzz FUZZTIME=5m).
FUZZTIME ?= 30s
fuzz:
@for t in FuzzFuzzyMatchV2Single FuzzFuzzyMatchV2Two; do \
@for t in FuzzFuzzyMatchV2Single FuzzFuzzyMatchV2Two FuzzRunePrefilter; do \
echo "== $$t =="; \
$(GO) test -run '^$$' -fuzz "^$$t$$" -fuzztime $(FUZZTIME) ./src/algo || exit 1; \
done