ITADN

Added structural cue chunking strategy based on JinaAI's tokenizer ch…

#92Pull RequestAswanthManoj 创建于 2024-08-23
A
AswanthManojcommented
Inspired by https://jina.ai/tokenizer/#chunking which leverage common structural cues and build a set of rules and heuristics which should perform exceptionally well across diverse types of content, including Markdown, HTML, LaTeX, and more, ensuring accurate segmentation of text into meaningful chunks. Reference: https://gist.github.com/JeremiahZhang/2f8ae87dad836b25f40c02b8c43d16ec Original x post: https://x.com/JinaAI_/status/1823756993108304135
合并状态:未合并 0 条评论