increase SPH_MAX_WORD_LEN and derived constants to make searchable token length support up to 32k length
enhancementest::size_L
### Proposal:
Manticore currently truncates long tokens during tokenization and dictionary
construction. This makes it hard to search data that often contains long exact
values, for example:
- email addresses with long local parts or domains;
- hashes and signatures;
- generated IDs;
- security, mail, and audit log identifiers;
- other machine-generated tokens that are meaningful as a whole.
The current hardcoded token/dictionary limits are too small for these cases.
Users can insert the data, but the indexed token may be truncated, so exact
searches and wildcard searches can miss the value or match a truncated form
instead of the original token.
## Requested behavior
Add support for long keyword tokens up to 32 KiB.
This should be available through a new opt-in dictionary format, for example:
```sql
dict = keywords_v2
```
The existing `dict = keywords` format should stay unchanged for compatibility and performance.
For the new format:
- support long tokens in both plain and RT tables;
- keep the full token for exact lookup;
- support prefix and infix lookup for long tokens;
- keep `CALL KEYWORDS()` useful for inspecting long indexed terms;
- preserve the token across RT flush, optimize/merge, restart, and disk chunk
reload;
- avoid silent truncation for tokens above the new cap. If a token is larger
than 32 KiB, skip it or report it explicitly instead of indexing a misleading
prefix.
## Example
With a table configured for stored keywords and infix search:
```sql
CREATE TABLE mail_log(
id bigint,
sender text,
recipient text,
headers text,
body text,
attachment_hash text
)
dict='keywords_v2'
index_exact_words='1'
min_infix_len='2';
```
A token longer than the current limit should be indexed and searchable:
```sql
INSERT INTO mail_log VALUES (
1,
'alessandro.very-long-generated-local-part@example-corporate-domain.test',
'security@example.test',
'message-id=<very-long-generated-message-id@example.test>',
'...',
'sha256:<long-hash-or-signature-value>'
);
SELECT * FROM mail_log
WHERE MATCH('"alessandro.very-long-generated-local-part@example-corporate-domain.test"');
SELECT * FROM mail_log
WHERE MATCH('*generated-local-part*');
```
Both queries should be able to find the row when the indexed token is within the
32 KiB limit.
## Notes
- Long-token morphology can be skipped. Stemmers and morphology dictionaries
are designed for normal words, not 32 KiB hashes or generated IDs. The
original token should still be indexed.
- The feature should not require increasing the legacy hardcoded constants in a
way that breaks existing dictionary encoding. A versioned keyword dictionary
format is safer.
- The new format should keep old indexes readable and should fail cleanly if an
older binary tries to open an unsupported v2 dictionary.
### Checklist:
<sup>To be completed by the assignee. Check off tasks that have been completed or are not applicable.</sup>
<details>
- [x] Implementation completed
- [x] Tests developed
- [x] Documentation updated
- [x] Documentation reviewed
- [ ] Changelog updated
- [x] OpenAPI YAML updated and issue created to rebuild clients
</details>
5 条评论