The scan only ran for ASCII patterns, so searching CJK text with a CJK
query still built the full score matrix. Scan for one byte of the pattern
rune and verify all four.
Which byte matters. Every ASCII rune contributes three zero bytes, so
U+AE00 scanned by its zero low byte hits on nearly every character of an
ASCII-heavy line. Pick a byte that cannot occur in an ASCII rune, else
any non-zero one.
A non-ASCII pattern rune is safe only when no other rune lowercases onto
it. Uncased is not sufficient: U+00DF has no simple uppercase, yet U+1E9E
lowercases to it, so the foldable set is excluded too.
Measured on 1.4M-line corpora, with a non-ASCII query:
- Every line CJK: 5.1x to 10.2x
- Mostly-ASCII paths behind a Hangul prefix: 5.6x to 6.0x, and 1.2x
where every line matches so nothing can be rejected
- ASCII queries unchanged, kept off the non-inlinable guard
asciiFuzzyIndex gave up on non-ASCII lines, so every item ran the full
score matrix. A []rune is a fixed 4-byte stride, so the SIMD byte
scanners can run over it directly: find the low byte, then confirm
4-byte alignment and three zero bytes.
Case folding and normalization can turn a non-ASCII rune into the ASCII
char being searched, which the scan cannot see. ToChars now flags lines
holding such a rune and those keep the old path. Normalization is
Latin-only, so Hangul, CJK, Cyrillic, Greek, Hebrew, Arabic, Thai, kana
and emoji never set the flag.
Chars had no spare padding, so inBytes moves into a flags byte.
Measured on 1.4M-line corpora:
- Mostly-ASCII paths behind a Hangul prefix: 'conf' 1.8x, 'binutils'
2.9x, 'ltversion' 4.0x, no-match 8.4x
- Every line CJK: 17x on both matching and non-matching queries
- ASCII input unchanged, non-ASCII patterns not covered yet
Junegunn Choi