Added structural cue chunking strategy based on JinaAI's tokenizer ch…
Inspired by https://jina.ai/tokenizer/#chunking which leverage common structural cues and build a set of rules and heuristics which should perform exceptionally well across diverse types of content, including Markdown, HTML, LaTeX, and more, ensuring accurate segmentation of text into meaningful chunks.
Reference: https://gist.github.com/JeremiahZhang/2f8ae87dad836b25f40c02b8c43d16ec
Original x post: https://x.com/JinaAI_/status/1823756993108304135
合并状态:未合并 0 条评论