Text API¶

kavalai.text turns HTML and markdown into the chunks a RAG index stores. It uses the standard library only, so it also runs under Pyodide, and it is not imported by import kavalai. For the parameters and a worked example, read Parsing and chunking text.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Text for a RAG index: HTML or markdown in, chunks to embed out.

parse_html() reduces a page to its title, its visible text as Block objects that each carry the heading path above them, its links and its robots directives, in one pass of the standard-library HTML parser. chunk_blocks() packs blocks into Chunk objects of about target_chars and never more than max_chars, splitting an oversized block at line breaks, then at sentence ends, then between words, and cutting only a single over-long word. chunk_markdown() reads markdown into the same blocks and packs them with the same function.

The module uses the standard library only, so it runs under Pyodide, and every step is linear in its input: no regular expression is involved, and the parser’s bookkeeping is amortised constant work per tag.

The boilerplate rules are parameters of parse_html(); the defaults are the module constants DROP_TAGS, CHROME_TAGS, DROP_ROLES, CONTENT_TAGS and ROBOTS_NAMES. Fetching a page, reading PDF and detecting the language are outside its scope.

kavalai.text.CHROME_TAGS = frozenset({'aside', 'footer', 'header'})¶: Elements that are page chrome outside a content element and part of it inside one: an article’s own <header> holds its title and byline, the page’s holds the logo and the menu.

kavalai.text.CONTENT_TAGS = frozenset({'article', 'main'})¶: Elements that mark a page’s content. An element whose role names one of them counts as that element.

kavalai.text.DROP_ROLES = frozenset({'alertdialog', 'banner', 'complementary', 'contentinfo', 'dialog', 'menu', 'menubar', 'navigation', 'search'})¶: ARIA landmark roles that mark boilerplate on any element.

kavalai.text.DROP_TAGS = frozenset({'button', 'canvas', 'dialog', 'iframe', 'nav', 'noscript', 'object', 'script', 'select', 'style', 'svg', 'template', 'textarea'})¶

code, fallbacks, navigation and form controls. Their links are still collected.

Type:¶: Elements whose text is never content

kavalai.text.HEADING_SEPARATOR = ' › '¶

"Products › Pricing".

Type:¶: Joins the headings of a heading path

kavalai.text.MAX_CHARS = 2000¶: The chunk size chunk_blocks() never exceeds. Small embedding models truncate at 512 tokens, so a much longer chunk would be embedded only in part.

kavalai.text.ROBOTS_NAMES = frozenset({'robots'})¶: <meta name> values whose content is read as robots directives.

kavalai.text.TARGET_CHARS = 1200¶: The chunk size chunk_blocks() packs towards.

class kavalai.text.Block(text: str, heading: str = '', kind: str = 'paragraph', level: int = 0)[source]¶

Bases: object

One paragraph-sized run of visible text and the headings above it.

Variables:¶

text : str¶: The text, whitespace collapsed; a <br> is kept as a line break, and a code block keeps its whitespace.
heading : str¶: The heading path above the block, joined by HEADING_SEPARATOR. For a heading block, the path ends in the heading itself.
kind : str¶: "paragraph", "heading", "item" (a list item, definition term or description), "row" (a table row, its cells separated by " | "), "code" or "quote".
level : int¶: The heading level, 1 to 6, for a heading block; 0 otherwise.

text : str¶

heading : str = ''¶

kind : str = 'paragraph'¶

level : int = 0¶

class kavalai.text.Chunk(text: str, heading: str, position: int)[source]¶

Bases: object

One piece of a document to embed.

Variables:¶

text : str¶: What is embedded: the prefix (title and heading path) when one was asked for, a blank line, then the body.
heading : str¶: The heading path the chunk’s blocks sit under.
position : int¶: The chunk’s index in the list it was returned in.

text : str¶

heading : str¶

position : int¶

class kavalai.text.ParsedPage(title: str, blocks: list[Block], links: list[str], noindex: bool = False, nofollow: bool = False)[source]¶

Bases: object

What one HTML document reduces to.

Variables:¶

title : str¶: The <title>, whitespace collapsed; the first heading when the page has no title.
blocks : list[kavalai.text.Block]¶: The visible text in document order, heading blocks included.
links : list[str]¶: The href of every <a> and <area>, boilerplate included, in document order without repeats; absolute when a base URL was given or the page declares <base href>.
noindex : bool¶: Whether a robots <meta> tag says noindex or none.
nofollow : bool¶: Whether a robots <meta> tag says nofollow or none.

title : str¶

blocks : list[Block]¶

links : list[str]¶

noindex : bool = False¶

nofollow : bool = False¶

property markdown : str¶

# headings, - items, fenced code.

The rendering serves RAG chunking and a readable archive, not fidelity: inline formatting and link targets are not reproduced.

Type:¶: The blocks as markdown

kavalai.text.chunk_blocks(blocks: Iterable[Block], *, title: str = '', target_chars: int | None = 1200, max_chars: int | None = 2000, prefix: bool = True) → list[Chunk][source]¶

Pack blocks into chunks of about target_chars, at most max_chars.

Consecutive blocks under the same heading path are joined by a blank line until the next one would pass target_chars; a new heading path always starts a new chunk. A block longer than the space left under max_chars is split at line breaks, then at sentence ends, then between words, and a single word longer than that is cut. Heading blocks contribute through the heading path, not as body text; blocks with no text are skipped.

With prefix, every chunk begins with the title and its heading path ("Acme › Products › Pricing") and a blank line, so a chunk retrieved on its own still says what it is about. A heading path that already begins with the title does not repeat it. max_chars counts the prefix, which is shortened to at most half of it.

Parameters:¶

blocks: Iterable[Block]¶: The blocks, in document order.
title: str = ''¶: The document title for the prefix.
target_chars: int | None = 1200¶: The size chunks are packed towards; None packs up to max_chars. A target above max_chars is lowered to it.
max_chars: int | None = 2000¶: The size no chunk exceeds; None never splits a block, so each heading section packs to target_chars or, with both None, stays one chunk.
prefix: bool = True¶: Whether to begin each chunk with the title and heading path.

Returns:¶

The chunks, position numbered from 0.

Raises:¶

ValueError – A size below 1.

kavalai.text.chunk_markdown(markdown: str, *, title: str = '', target_chars: int | None = 1200, max_chars: int | None = 2000, prefix: bool = True) → list[Chunk][source]¶

Chunk markdown with chunk_blocks().

ATX headings (# to ######) form the heading path; a blank line ends a paragraph block, and a fenced code block (```` or ~~~) is one block whose # lines are not headings. The keyword arguments are those of chunk_blocks().

kavalai.text.parse_html(html: str, *, base_url: str | None = None, content_tags: Iterable[str] = frozenset({'article', 'main'}), drop_tags: Iterable[str] = frozenset({'button', 'canvas', 'dialog', 'iframe', 'nav', 'noscript', 'object', 'script', 'select', 'style', 'svg', 'template', 'textarea'}), chrome_tags: Iterable[str] = frozenset({'aside', 'footer', 'header'}), drop_roles: Iterable[str] = frozenset({'alertdialog', 'banner', 'complementary', 'contentinfo', 'dialog', 'menu', 'menubar', 'navigation', 'search'}), drop_hidden: bool = True, strip_chars: str = '¶', robots_names: Iterable[str] = frozenset({'robots'})) → ParsedPage[source]¶

Reduce an HTML document to its title, text blocks, links and robots meta.

When the page has text inside a content element (content_tags), only that text is kept and only the headings inside content elements form the heading paths; otherwise the whole visible page is kept.

Parameters:¶

html: str¶: The document.
base_url: str | None = None¶: The URL the document was fetched from; relative links are resolved against it, or against <base href> when the page declares one.
content_tags: Iterable[str] = frozenset({'article', 'main'})¶: Elements that mark the page’s content.
drop_tags: Iterable[str] = frozenset({'button', 'canvas', 'dialog', 'iframe', 'nav', 'noscript', 'object', 'script', 'select', 'style', 'svg', 'template', 'textarea'})¶: Elements whose text is dropped wherever they occur.
chrome_tags: Iterable[str] = frozenset({'aside', 'footer', 'header'})¶: Elements whose text is dropped outside a content element and kept inside one.
drop_roles: Iterable[str] = frozenset({'alertdialog', 'banner', 'complementary', 'contentinfo', 'dialog', 'menu', 'menubar', 'navigation', 'search'})¶: role values whose element’s text is dropped.
drop_hidden: bool = True¶: Whether to drop elements that are not rendered: the hidden attribute (except until-found), aria-hidden="true" and an inline display: none or visibility: hidden.
strip_chars: str = '¶'¶: Characters removed from all text.
robots_names: Iterable[str] = frozenset({'robots'})¶: <meta name> values whose directives are read; add a crawler’s own token ("mybot") to honour directives addressed to it.

Returns:¶

The parsed page.