Skip to main content

​
LLM Clients

TrustTest provides a flexible abstraction layer for working with different LLM providers through its LLMClient interface. This architecture allows for seamless integration with various LLM services while maintaining a consistent interface for generating questions, evaluations, and other LLM-powered features.

​
Architecture

The core of this system is the LLMClient abstract base class, which defines two main methods:
  • complete(instructions, system_prompt): For single-prompt completions
  • complete_chat(messages): For multi-turn conversations
Each implementation handles provider-specific details while exposing a unified interface.

​
Supported Providers

TrustTest currently supports the following LLM providers:
  • OpenAI
uv add "trusttest[openai]"
  • Anthropic
uv add "trusttest[anthropic]"
  • Google
uv add "trusttest[google]"
  • Ollama
uv add "trusttest[ollama]"
  • vLLM
uv add "trusttest[vllm]"
  • Azure OpenAI
uv add "trusttest[openai]"
  • Deepseek
uv add "trusttest[deepseek]"

​
Usage Example

import asyncio
import os
from trusttest.llm_clients import get_llm_client

# Set up environment variables for the provider
os.environ["DEEPSEEK_BASE_URL"] = "https://api.deepseek.com"
os.environ["DEEPSEEK_API_KEY"] = "<your-api-key>"

# Initialize the client
llm = get_llm_client(model="deepseek-chat", provider="deepseek")

# Make a completion request
response = asyncio.run(llm.complete("Return a json saying hello"))
print(response)
The abstraction allows for easy switching between providers while maintaining consistent behavior across the application.

​
Embeddings Clients

TrustTest provides a flexible abstraction layer for working with different embedding providers through its EmbeddingsModel interface. This architecture allows for seamless integration with various embedding services while maintaining a consistent interface for generating vector representations of text.

​
Architecture

The core of this system is the EmbeddingsModel abstract base class, which defines the main method:
  • embed(texts): Converts a sequence of texts into numerical vector representations
Each implementation handles provider-specific details while exposing a unified interface.

​
Supported Providers

TrustTest currently supports the following embedding providers:
  • OpenAI
  • Google
  • Ollama

​
Usage Example

import os

from trusttest.embeddings import get_embeddings_model

os.environ["OPENAI_API_KEY"] = "<your-api-key>"

embeddings = get_embeddings_model(
    provider="openai",
    model="text-embedding-3-small",
)

texts = ["Hello world", "TrustTest is great"]
vectors = embeddings.embed(texts)
print(vectors.shape)
The abstraction allows for easy switching between providers while maintaining consistent behavior across the application.

​
Global Configuration

TrustTest provides a global configuration system to manage LLM and embeddings settings across your application. The configuration can be set using the set_config() function, which accepts a dictionary with settings for different components: 
import trusttest

trusttest.set_config({
    "evaluator": {
        "provider": "openai",
        "model": "gpt-4",
        "temperature": 0.2
    },
    "question_generator": {
        "provider": "openai",
        "model": "gpt-4",
        "temperature": 0.5
    },
    "embeddings": {
        "provider": "openai",
        "model": "text-embedding-3-small"
    },
    "topic_summarizer": {
        "provider": "openai",
        "model": "gpt-4",
        "temperature": 0.2
    }
})
The configuration supports the following components:
  • evaluator: LLM settings for evaluation tasks
  • question_generator: LLM settings for generating test questions
  • embeddings: Settings for the embeddings model
  • topic_summarizer: LLM settings for topic summarization
Each component accepts:
  • provider: One of “openai”, “azure”, “google”, “anthropic”, “ollama” (for LLMs) or “openai”, “azure”, “google”, “ollama” (for embeddings)
  • model: The specific model name for the chosen provider
  • temperature: (LLMs only) Controls randomness in model outputs (0.0 to 1.0)

​
Implementing Custom Clients

Both LLM and Embeddings clients can be easily extended by implementing custom providers. The base classes provide a clear interface that you need to implement.

​
Custom LLM Client

To create a custom LLM client, inherit from LLMClient and implement the required methods: 
from trusttest.llm_clients.base import LLMClient, ChatMessage, BaseLLMResponse

class CustomLLMClient(LLMClient):
    async def complete(
        self,
        instructions: str,
        system_prompt: Optional[str] = None,
        response_schema: Type[BaseModel] = BaseLLMResponse,
    ) -> Dict[str, Any]:
        # implement your custom logic here
        raise NotImplementedError

    async def complete_chat(
        self,
        messages: Sequence[ChatMessage],
        response_schema: Type[BaseModel] = BaseLLMResponse,
    ) -> Dict[str, Any]:
        # implement your custom logic here
        raise NotImplementedError
The LLMClient expects to define the response schema, this is a pydantic model that will be used to parse the response from the LLM. Once implemented, you are ready to use them: 
custom_llm = CustomLLMClient()
evaluator = CorrectnessEvaluator(llm_client=custom_llm)

​
Custom Embeddings Client

To create a custom embeddings client, inherit from EmbeddingsModel and implement the required method: 
from trusttest.embeddings import EmbeddingsModel
import numpy as np

class CustomEmbeddingsModel(EmbeddingsModel):
    def embed(self, texts: list[str]) -> np.ndarray:
        # Implement your custom embedding logic
        pass