Prefilter rune-mode input for non-ASCII patterns

The scan only ran for ASCII patterns, so searching CJK text with a CJK
query still built the full score matrix. Scan for one byte of the pattern
rune and verify all four.

Which byte matters. Every ASCII rune contributes three zero bytes, so
U+AE00 scanned by its zero low byte hits on nearly every character of an
ASCII-heavy line. Pick a byte that cannot occur in an ASCII rune, else
any non-zero one.

A non-ASCII pattern rune is safe only when no other rune lowercases onto
it. Uncased is not sufficient: U+00DF has no simple uppercase, yet U+1E9E
lowercases to it, so the foldable set is excluded too.

Measured on 1.4M-line corpora, with a non-ASCII query:

- Every line CJK: 5.1x to 10.2x
- Mostly-ASCII paths behind a Hangul prefix: 5.6x to 6.0x, and 1.2x
  where every line matches so nothing can be rejected
- ASCII queries unchanged, kept off the non-inlinable guard
This commit is contained in:
Junegunn Choi committed 2026-08-08 19:48:04 +09:00
1 parent 465837f3ad
commit 9dfdba41f5
5 files changed
+274 -18

No files matched your search

+18
View File
@@ -35,3 +35,21 @@ func lastIndexAsciiRuneRef(runes []rune, caseSensitive bool, b byte, from int) i
}
return -1
}
func indexRuneRef(runes []rune, r rune, from int) int {
for i := from; i < len(runes); i++ {
if runes[i] == r {
return i
}
}
return -1
}
func lastIndexRuneRef(runes []rune, r rune, from int) int {
for i := len(runes) - 1; i >= from; i-- {
if runes[i] == r {
return i
}
}
return -1
}