LLM Clients API¶

kavalai.llm_clients provides a unified, observable interface over LLM and embedding providers. Every call returns a ModelCallStat with token usage and timing, and structured output is validated against a Pydantic response_model.

Providers¶

make_client() builds a client from a "provider/model" id and reads the matching credential from the environment:

Prefix	Class	Credential
`openai/`	`OpenAIClient`	`OPENAI_API_KEY`
`gemini/`	`GeminiClient`	`GEMINI_API_KEY`
`anthropic/`	`AnthropicClient`	`ANTHROPIC_API_KEY`
`ollama/`	`OllamaClient`	`OLLAMA_HOST` (no key)
`browser/`	`BrowserLLMClient`	none — runs client-side over WebGPU

make_embedding_client() mirrors it for embeddings, adding the fastembed/ prefix for local, key-free embedding.

The browser/ provider needs a WebGPU-capable page rather than a Python process; see Running in the browser for what it can do and how models are downloaded and cached.

Registering your own¶

The provider names above are the built-ins, not the whole set: register_llm_provider() and register_embedding_provider() add more, so mycorp/model-x resolves anywhere a model name is accepted — including workflow YAML.

A provider that speaks the OpenAI wire format needs no code at all: register OpenAIClient under a new name with its base_url and key bound to it. One with a protocol of its own means implementing BaseLlmClient (for chat) or BaseEmbeddingClient (for embeddings) — one method each. LLM clients works through both against live APIs, and Cookbook has the condensed recipe.

Name-to-backend registries for LLM, embedding and RAG backends.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

A workflow names its backends with strings – llm_model: openai/gpt-5.6-luna in YAML, a rag_query node naming a service – so resolution has to happen from a name. Until this module existed that name could only be one of five values hard-coded in an if chain, which meant adding a backend required editing Kaval.AI itself.

Three registries live here, all instances of the same Registry:

llm_providers – resolves provider/model to a BaseLlmClient
embedding_providers – resolves provider/model to a BaseEmbeddingClient
rag_services – resolves a plain name to a BaseRagService

Registration is deliberately explicit – there is no scanning and no plugin path. The set of available backends is discovered; the behaviour of any given run is still passed, since an engine’s client_factory and rag_services arguments both outrank these registries.

A registration Target is a class, a dotted path to one, or a factory callable. DEFAULT_NAME is the name looked up when a workflow does not name a backend explicitly; it matches BaseRagService’s own default collection and source identifiers. BUILTIN_LLM_PACKAGES names the PyPI distribution behind each built-in LLM provider, so an import failure can say what to install.

exception kavalai.llm_clients.registry.RegistryError[source]¶

Bases: ValueError

Raised for duplicate registrations, unknown names and bad dotted paths.

Subclasses ValueError because that is what make_client and make_embedding_client raised for an unusable name before the registries existed, and callers catch it.

class kavalai.llm_clients.registry.Registry(kind: str, register_hint: str)[source]¶

Bases: object

A name-to-target registry with lazy dotted-path resolution.

Names may be a bare provider ("openai") or a full provider/model pair ("mycorp/mymodel"). Lookup tries the exact string first and then the part before the first /, so a model-specific registration wins over a provider-wide one without either having to know about the other.

kind is the noun used in error messages (“LLM provider”) and register_hint the public function that registers into this registry, so an error can name the call that fixes it. Names shipped with Kaval.AI are marked as built-ins; verify() skips them, because their SDKs are optional extras that a given install need not have.

register(name: str, target: type | str | Callable[[...], Any], *, replace: bool = False, validate: bool = False, **defaults: Any) → None[source]¶

Bind name to target.

Parameters:¶

name: str¶: Provider name ("mycorp") or provider/model pair.
target: type | str | Callable[[...], Any]¶: A class, a dotted path to one, or a factory callable.
replace: bool = False¶: Overwrite an existing registration. Without it a duplicate raises, matching FunctionKernel.register_python_tool.
validate: bool = False¶: Import a dotted path immediately instead of on first use.
**defaults: Any¶: Bound now, forwarded to the target at construction.

Raises:¶

RegistryError – The name is empty, starts or ends with /, or is already registered and replace is not set.

unregister(name: str) → None[source]¶: Remove a registration. Unknown names are ignored.

names() → list[str][source]¶: Every registered name, sorted. Resolves nothing.

lookup(name: str) → str | None[source]¶

Return the registered key serving name, or None.

Tries the exact name first, then the provider part before the first /. Nothing is imported.

build(name: str, *args: Any, **kwargs: Any) → Any[source]¶

Construct the backend registered for name.

Classes are built through from_model; anything else is called directly. Registration defaults are merged in beneath the caller’s keyword arguments, so a caller can always override them.

Raises:¶: RegistryError – Nothing is registered for name.

unknown_message(name: str) → str[source]¶: The error text for a name nothing is registered for.

verify() → None[source]¶

Import every non-built-in dotted registration.

Raises on the first that cannot be imported. Entry points call this after loading operator-supplied modules, so a mistyped path fails at start-up rather than at the first request that happens to reach that node.

Built-ins are skipped deliberately: their SDKs are optional extras, and an install with only openai should not fail start-up because anthropic is absent.

mark_builtin(name: str) → None[source]¶: Record name as shipped with Kaval.AI, so verify skips it.

kavalai.llm_clients.registry.register_llm_provider(name: str, target: type | str | Callable[[...], Any], *, replace: bool = False, validate: bool = False, **defaults: Any) → None[source]¶

name may also be a full provider/model pair, which pins one model to one class; lookup prefers it over a provider-wide registration.

register_llm_provider("mycorp", MyClient)
register_llm_provider("mycorp", "mycorp.llm.MyClient")
register_llm_provider("mycorp/fast", MyClient, base_url="https://...")

The target receives (model, parameters, stats_receiver) plus any registration defaults, where model is the part after the first /. Forwarding stats_receiver to BaseLlmClient is what makes a custom client’s usage show up in the backoffice alongside the built-ins.

kavalai.llm_clients.registry.register_embedding_provider(name: str, target: type | str | Callable[[...], Any], *, replace: bool = False, validate: bool = False, **defaults: Any) → None[source]¶

The target receives (model,) plus any registration defaults.

kavalai.llm_clients.registry.register_rag_service(name: str, target: type | str | Callable[[...], Any], *, replace: bool = False, validate: bool = False, **defaults: Any) → None[source]¶

Unlike models, a RAG service is named on its own – there is no provider/model split. The name is what a rag_query node’s service field and a workflow’s rag_service default refer to; "default" is used when neither names one.

The target receives only the registration defaults, so everything the backend needs (a URI, a filename, an embedding model) is bound here:

register_rag_service(
    "default", SqliteRagService,
    filename="handbook.db", model="fastembed/BAAI/bge-small-en-v1.5",
)

kavalai.llm_clients.registry.make_rag_service(name: str = 'default', **kwargs: Any) → Any[source]¶

Construct the RAG service registered under name.

Parameters:¶

name: str = 'default'¶: A registered service name. Defaults to "default", which is what a workflow uses when neither a node nor the graph names one.
**kwargs: Any¶: Extra constructor arguments, overriding any bound at registration.

Raises:¶

RegistryError – No service is registered under that name.

kavalai.llm_clients.registry.registered_llm_providers() → list[str][source]¶: Every registered LLM provider name, sorted.

kavalai.llm_clients.registry.registered_embedding_providers() → list[str][source]¶: Every registered embedding provider name, sorted.

kavalai.llm_clients.registry.registered_rag_services() → list[str][source]¶: Every registered RAG service name, sorted.

kavalai.llm_clients.registry.verify_registrations() → None[source]¶

Import every dotted registration in every registry.

Raises RegistryError on the first that cannot be imported. Called by entry points after loading operator-supplied provider modules, so a mistyped path fails at start-up instead of mid-run.

Built-ins are skipped – their SDKs are optional extras, so importing them all would make start-up depend on packages the install may not need.

Base client and models¶

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

class kavalai.llm_clients.base_client.LlmClientParameters(*, temperature: float | None = None, top_p: float | None = None, reasoning_effort: str | None = None, service_tier: str | None = None, max_output_tokens: Annotated[int | None, Gt(gt=0)] = None, timeout_seconds: float | None = 30.0, stream_timeout_seconds: float | None = None)[source]¶

Bases: BaseModel

Optional per-call LLM parameters.

Sampling parameters default to None and are only sent to a provider when explicitly set, so each provider’s own defaults apply otherwise (some models, e.g. recent Claude models, reject sampling params outright).

max_output_tokens caps the tokens a call may generate. Each client sends it under its provider’s own name (max_output_tokens for OpenAI and Gemini, max_tokens for Anthropic and WebLLM, num_predict for Ollama). On reasoning models the cap includes the reasoning tokens. A call that reaches the cap raises OutputTruncatedError rather than returning the partial answer.

temperature : float | None¶

top_p : float | None¶

reasoning_effort : str | None¶

service_tier : str | None¶

max_output_tokens : int | None¶

timeout_seconds : float | None¶

stream_timeout_seconds : float | None¶

model_config : ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class kavalai.llm_clients.base_client.ChatMessage(*, role: str | None = None, type: str | None = None, content: str | None = None)[source]¶

Bases: BaseModel

Standard chat completion message.

role : str | None¶

type : str | None¶

content : str | None¶

model_config : ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class kavalai.llm_clients.base_client.ChatHistory(*, messages: list[ChatMessage])[source]¶

Bases: BaseModel

messages : list[ChatMessage]¶

model_config : ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

kavalai.llm_clients.base_client.ensure_user_turn(messages: list[ChatMessage]) → list[ChatMessage][source]¶

Guarantee the history contains a turn a model can answer.

Workflow llm nodes render their whole prompt into a single system message, and BaseLlmClient.prompt() does the same. A chat request with no user turn is unusual and providers disagree about it: Anthropic’s Messages API rejects an empty messages list outright, while small local models answer it literally — llama3.2 prefixes its reply with a literal “assistant” line.

So the convention is: if nothing but system content is present, that content becomes the user turn. Histories that already contain a user or assistant message are returned untouched.

Bases: BaseModel

call_type : Literal['llm', 'embedding']¶

model : str | None¶

request_data : str | None¶

response_data : str | None¶

response_code : int | None¶

prompt_tokens : int | None¶

completion_tokens : int | None¶

total_tokens : int | None¶

cached_prompt_tokens : int | None¶

reasoning_tokens : int | None¶

batch_size : int | None¶

duration_seconds : float | None¶

model_config : ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class kavalai.llm_clients.base_client.ModelStatsReceiver[source]¶

Bases: object

receive_model_stats(stats: ModelCallStat)[source]¶

class kavalai.llm_clients.base_client.ModelStatsLogger(format_str: str | None = None)[source]¶

Bases: ModelStatsReceiver

Logs model call statistics using a configurable format.

receive_model_stats(stats: ModelCallStat)[source]¶

kavalai.llm_clients.base_client.error_status_code(error: Exception) → int | None[source]¶

Best-effort HTTP status from a provider SDK exception.

Every provider raises its own error type, but all of them carry the status somewhere: status_code (OpenAI, Anthropic, Ollama), code (Gemini), or a nested response object.

exception kavalai.llm_clients.base_client.LlmClientException[source]¶

Bases: RuntimeError

An error the LLM clients raise themselves, as opposed to a provider SDK.

exception kavalai.llm_clients.base_client.OutputTruncatedError(*, model: str, reason: str, max_output_tokens: int | None = None, partial_output: str = '', request_data: Any = None, prompt_tokens: int | None = None, completion_tokens: int | None = None, cached_prompt_tokens: int | None = None, reasoning_tokens: int | None = None)[source]¶

Bases: LlmClientException

The provider stopped generating because the output cap was reached.

Raised in place of the partial answer. For a structured call that answer is JSON cut off mid-value, which either fails to parse or, after lenient repair, validates into a model with content missing. The retry loop does not retry it: the same cap produces the same cut, and is billed again.

max_output_tokens is the cap that was sent, or None when the provider’s own limit applied. reason is the stop reason as the provider reported it (max_output_tokens, MAX_TOKENS, max_tokens, length). partial_output is the text generated before the cut, and the token counts are those of the truncated call, which is recorded as a model call like any other because it was billed.

class kavalai.llm_clients.base_client.BaseLlmClient(llm_client_parameters: LlmClientParameters | None = None, model_stats_receiver: ModelStatsReceiver | None = None)[source]¶

Bases: object

Common interface of the LLM clients.

provider is the prefix a subclass sets so that recorded model calls are named provider/model (openai/gpt-4o).

provider : str = ''¶

property timeout_seconds : float¶: Per-request timeout, 30 s unless LlmClientParameters sets one.

async stream_chat_completions(*, chat_history: ChatHistory, response_model: type[BaseModel] | None = None, stream_delta: bool = False) → Streamer[source]¶

Execute a chat completion and return a Streamer.

Parameters:¶

chat_history: ChatHistory¶: The history of messages.
response_model: type[BaseModel] | None = None¶: Optional Pydantic model for structured output.
stream_delta: bool = False¶: When True, partial chunks carry only the newly generated text and consumers reassemble; otherwise each partial carries the full accumulated (safe-parsed) content so far.

Returns:¶

A Streamer instance that will yield the completion events.

async chat_completions(*, chat_history: ChatHistory, response_model: type[BaseModel] | None = None)[source]¶

async stream_prompt(system_message: str, response_model: type[BaseModel] | None = None) → Streamer[source]¶

async prompt(system_message: str, response_model: type[BaseModel] | None = None)[source]¶

classmethod from_model(model: str, parameters: LlmClientParameters | None = None, stats_receiver: ModelStatsReceiver | None = None, **defaults) → BaseLlmClient[source]¶

Construct this client for a resolved model name.

The registry calls this rather than the constructor, so a client whose __init__ does not take the shape below can say so once here instead of making every registration wrap it in a lambda. defaults are the keyword arguments bound at registration (a base URL, a host, a key).

Parameters:¶

model: str¶: Model name with the provider prefix already removed.
parameters: LlmClientParameters | None = None¶: Optional per-call sampling/timeout parameters.
stats_receiver: ModelStatsReceiver | None = None¶: Where model-call statistics are reported. Passing it through is what puts a custom client’s usage in the backoffice alongside the built-in providers.
**defaults¶: Extra constructor arguments bound at registration.

stat_model_name() → str[source]¶: The provider/model name this client records its calls under.

Provider clients¶

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

class kavalai.llm_clients.openai_client.OpenAIClient(model: str, llm_client_parameters: LlmClientParameters | None = None, model_stats_receiver: ModelStatsReceiver | None = None, api_key: str | None = None, base_url: str | None = None)[source]¶

Bases: BaseLlmClient

OpenAI LLM client implementation using the Responses API and Streamer.

provider : str = 'openai'¶

kavalai.llm_clients.openai_client.usage_counts(usage: Any) → dict[source]¶

The token counts of a Responses API usage, named as a model call stat.

Cached input is billed at a fraction of fresh input, and reasoning tokens are output the caller never sees — both are subsets of the two main counts.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

class kavalai.llm_clients.gemini_client.GeminiClient(model: str, llm_client_parameters: LlmClientParameters | None = None, model_stats_receiver: ModelStatsReceiver | None = None, api_key: str | None = None)[source]¶

Bases: BaseLlmClient

Gemini LLM client implementation using the Streamer.

provider : str = 'gemini'¶

kavalai.llm_clients.gemini_client.convert_messages(messages: list[dict[str, Any]]) → tuple[str | None, list[Content]][source]¶

kavalai.llm_clients.gemini_client.remove_additional_properties(schema: dict[str, Any]) → None[source]¶

Strip additionalProperties from every object schema, in place.

Gemini’s response schema does not accept the field.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

class kavalai.llm_clients.anthropic_client.AnthropicClient(model: str, llm_client_parameters: LlmClientParameters | None = None, model_stats_receiver: ModelStatsReceiver | None = None, api_key: str | None = None, base_url: str | None = None)[source]¶

Bases: BaseLlmClient

Anthropic (Claude) LLM client implementation using the Messages API and Streamer.

provider : str = 'anthropic'¶

kavalai.llm_clients.anthropic_client.convert_messages(messages: list[dict[str, Any]]) → tuple[str | None, list[dict[str, Any]]][source]¶

Split chat messages into the system prompt and Anthropic message turns.

The Messages API takes the system prompt as a top-level system parameter and requires a non-empty messages list, so a system-only history (e.g. from BaseLlmClient.prompt()) is sent as the single user turn instead.

kavalai.llm_clients.anthropic_client.forbid_additional_properties(schema: dict[str, Any]) → None[source]¶

Set additionalProperties: false on every object schema, in place.

Anthropic structured outputs require closed object schemas.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

class kavalai.llm_clients.ollama_client.OllamaClient(model: str, llm_client_parameters: LlmClientParameters | None = None, model_stats_receiver: ModelStatsReceiver | None = None, host: str | None = None)[source]¶

Bases: BaseLlmClient

Ollama LLM client implementation using the Streamer.

provider : str = 'ollama'¶

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

kavalai.llm_clients.browser_client.get_browser_bridge()[source]¶

Return the page’s JS bridge object (window.kavalBrowserLLM).

Shared by the browser LLM client (.chat) and the browser embedding client (.embed). Raises a helpful LlmClientException when not running under Pyodide, or when the page has not loaded a bridge.

class kavalai.llm_clients.browser_client.BrowserLLMClient(model: str, llm_client_parameters: LlmClientParameters | None = None, model_stats_receiver: ModelStatsReceiver | None = None)[source]¶

Bases: BaseLlmClient

LLM client that runs entirely in the browser, with no network calls.

Inference happens inside the page through a tiny JavaScript bridge exposed on window.kavalBrowserLLM, typically backed by a WebGPU engine such as WebLLM. This makes Kaval.AI’s LLM nodes usable inside Pyodide with no API key, no provider account and no CORS constraints — the model is downloaded once and cached by the browser.

Use it through make_client("browser/<model-id>") or construct it directly. <model-id> is passed verbatim to the bridge (e.g. a WebLLM model id like Llama-3.2-1B-Instruct-q4f32_1-MLC).

The bridge contract is a single async function:

window.kavalBrowserLLM.chat(requestJson) -> Promise<resultJson>

where requestJson is a JSON string of {model, messages, temperature?, top_p?, max_tokens?, response_format?} and resultJson is a JSON string of either {content, usage, finish_reason?} or {error}. Exchanging plain JSON strings keeps the Python<->JS boundary free of proxy-conversion surprises. A finish_reason of "length" raises OutputTruncatedError; a bridge that does not report the field cannot signal truncation.

provider : str = 'browser'¶

Embeddings¶

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

class kavalai.llm_clients.embeddings.BaseEmbeddingClient(model: str)[source]¶

Bases: object

Common interface for v2 embedding clients.

The model name is bound at construction (the factory splits the provider/model string), so compute_embeddings only takes the texts. Implementations return the embeddings plus a database-ready ModelCallStat (the ORM row) so callers such as PostgresRagService can persist usage directly.

classmethod from_model(model: str, **defaults) → BaseEmbeddingClient[source]¶

Construct this client for a resolved model name.

The registry calls this rather than the constructor, because embedding clients do not share one: they variously take api_key, host, base_url, cache_dir or nothing at all. defaults are the keyword arguments bound at registration.

Parameters:¶

model: str¶: Model name with the provider prefix already removed.
**defaults¶: Extra constructor arguments bound at registration.

async compute_embeddings(texts: list[str], normalize: bool = False, normalizer: Normalizer | None = None, **kwargs) → tuple[list[list[float]], ModelCallStat][source]¶

class kavalai.llm_clients.embeddings.OpenAIEmbeddingClient(model: str, api_key: str | None = None, base_url: str | None = None, timeout: float = 30.0, dimensions: int | None = None)[source]¶

Bases: BaseEmbeddingClient

OpenAI embeddings (e.g. text-embedding-3-small).

dimensions shortens the returned vectors on models that support it (text-embedding-3-*); None keeps the model’s native size. Bound at registration, it makes a provider name stand for the reduced model:

register_embedding_provider(
    "openai-512", OpenAIEmbeddingClient, dimensions=512
)
make_embedding_client("openai-512/text-embedding-3-small")

The recorded model call keeps the response’s model and usage only. The vectors are the call’s result, returned to the caller; stored in the statistics row as well, they duplicated every indexed vector.

async compute_embeddings(texts: list[str], normalize: bool = False, normalizer: Normalizer | None = None, **kwargs) → tuple[list[list[float]], ModelCallStat][source]¶

class kavalai.llm_clients.embeddings.GeminiEmbeddingClient(model: str, api_key: str | None = None, dimensions: int | None = None)[source]¶

Bases: BaseEmbeddingClient

Google Gemini embeddings.

dimensions is sent as output_dimensionality, which shortens the vectors on models that support it; None keeps the native size. As with OpenAIEmbeddingClient, binding it at registration gives the reduced model a name of its own.

async compute_embeddings(texts: list[str], normalize: bool = False, normalizer: Normalizer | None = None, **kwargs) → tuple[list[list[float]], ModelCallStat][source]¶

class kavalai.llm_clients.embeddings.OllamaEmbeddingClient(model: str, host: str | None = None, timeout: float = 30.0)[source]¶

Bases: BaseEmbeddingClient

Ollama (local) embeddings.

async compute_embeddings(texts: list[str], normalize: bool = False, normalizer: Normalizer | None = None, **kwargs) → tuple[list[list[float]], ModelCallStat][source]¶

class kavalai.llm_clients.embeddings.FastEmbedClient(model: str, cache_dir: str | None = None, threads: int | None = None, **kwargs)[source]¶

Bases: BaseEmbeddingClient

Local embeddings via FastEmbed / ONNX Runtime (no API key).

classmethod from_model(model: str, **defaults) → FastEmbedClient[source]¶

Fall back to FASTEMBED_CACHE_DIR / FASTEMBED_THREADS.

The other embedding clients read their credential inside __init__; FastEmbed needs no credential but does take two environment-tunable knobs, so the same fallback lives here instead.

async compute_embeddings(texts: list[str], normalize: bool = False, normalizer: Normalizer | None = None, **kwargs) → tuple[list[list[float]], ModelCallStat][source]¶

class kavalai.llm_clients.embeddings.BrowserEmbeddingClient(model: str)[source]¶

Bases: BaseEmbeddingClient

In-browser embeddings via the WebLLM bridge (Pyodide only, no API key).

Mirrors BrowserLLMClient: inference happens inside the page through window.kavalBrowserLLM, here via its async embed function:

window.kavalBrowserLLM.embed(requestJson) -> Promise<resultJson>

where requestJson is a JSON string of {model, input} (input is the list of texts) and resultJson is a JSON string of either {embeddings, usage} or {error}. The model is downloaded once and cached by the browser — no API key, no provider account, no CORS.

Use it through make_embedding_client("browser/<model-id>"); <model-id> is passed verbatim to the bridge (e.g. a WebLLM embedding id like snowflake-arctic-embed-m-q0f32-MLC-b4).

async compute_embeddings(texts: list[str], normalize: bool = False, normalizer: Normalizer | None = None, **kwargs) → tuple[list[list[float]], ModelCallStat][source]¶

kavalai.llm_clients.embeddings.make_embedding_client(model: str, **kwargs) → BaseEmbeddingClient[source]¶

Construct an embedding client from a provider/model string.

The provider is the part before the first /; the remainder is the model name and may itself contain slashes (fastembed/BAAI/bge-small-en-v1.5). Built-in providers are openai, gemini, ollama, fastembed and browser; browser runs client-side via a WebLLM bridge (Pyodide only) and needs no API key.

Any provider registered with register_embedding_provider() resolves here too, and an exact provider/model registration takes precedence over the provider-wide one.

Parameters:¶

model: str¶: The provider/model identifier.
**kwargs¶: Extra constructor arguments, overriding any bound at registration.

Raises:¶

ValueError – model has no provider prefix.
RegistryError – No provider is registered for the prefix.

Streaming¶

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

exception kavalai.llm_clients.streamer.StreamerTimeoutException(names: list[str], timeout_seconds: float)[source]¶

Bases: Exception

Raised when no stream chunk arrives within the configured timeout.

Reported by Streamer while waiting on its queue when a timeout_seconds is set; names lists the streamers still active when the timeout elapsed.

class kavalai.llm_clients.streamer.StreamContent(*, type: str, name: str, value: str | None = None)[source]¶

Bases: BaseModel

StreamContent represents a streamed message from a Streamer.

Variables:¶

type : str¶: The type of stream message (e.g., ‘partial’, ‘complete’).
name : str¶: The identifier for the stream source or target.
value : str | None¶: The actual content string.

type : str¶

name : str¶

value : str | None¶

model_config : ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class kavalai.llm_clients.streamer.ValueStreamer(name: str, queue: Queue, response_model: type[BaseModel] | None = None, stream_delta: bool = False, on_complete_callback: Callable[[], None] | None = None)[source]¶

Bases: object

A helper class to manage and push streaming content to an asyncio queue.

Variables:¶

name: str¶: Default name for the stream chunks.
queue: Queue¶: The asyncio.Queue where messages are placed.

get_safe_value() → str[source]¶: Safely parse and return the buffered content as JSON string if response_model is set, otherwise return as string.

async stream_partial(value: str)[source]¶

Push a ‘partial’ chunk to the queue.

Parameters:¶

value: str¶: The partial content to stream.

async stream_complete()[source]¶

Push a ‘complete’ chunk to the queue, indicating the stream has finished.

In delta mode the chunk carries no value; otherwise it carries the full accumulated (safe-parsed) content.

class kavalai.llm_clients.streamer.Streamer(stream_delta: bool = False, timeout_seconds: float | None = None)[source]¶

Bases: object

property queue : Queue¶

get_value_streamer(name: str, stream_delta: bool | None = None, response_model: type[BaseModel] | None = None) → ValueStreamer[source]¶

async stream_error(error: Exception)[source]¶

Push an ‘error’ chunk to the queue.

The consumer’s iteration then raises error itself when it is a RuntimeError, so an LLM client’s own exception types (such as OutputTruncatedError) reach the caller intact, and otherwise a RuntimeError carrying its message.

reset_active() → None[source]¶

Forget all active value streamers.

Called between retry attempts: the failed attempt’s value streamers never complete, and without a reset their names would keep the completion accounting from ever reaching zero.

async stream_restart(value: str | None = None)[source]¶

Push a ‘restart’ chunk to the queue.

Signals that the in-flight completion is starting over (e.g. a retry after a transient error): consumers should discard any partial content accumulated so far; it will be re-sent.