Response Generator

The Response Generator module is designed to handle the generation of responses and images using various models. It includes two main components: lm_response for language model responses and t2i_response for text-to-image responses.

Quick Start

generate_responses

The generate_responses function processes all data files in a specified directory to generate responses using specified models.

Definition:

generate_responses(
    data_folder: str,
    request_type: List[str],
    async_list: List[str],
    sync_list: Optional[List[str]] = [],
    prompt_key: List[str] = 'prompt_key',
    result_key: List[str] = 'result_key',
    file_name_extension: str = '_responses',
    image_key: Optional[List[Optional[str]]] = None,
    **kwargs
) -> None
Parameters:

  • data_folder (str) Path to the folder containing data files

  • request_type (List[str]) List of request types (‘llm’ or ‘vlm’)

  • async_list (List[str]) List of asynchronous model names

  • sync_list (List[str], optional) List of synchronous model names

  • prompt_key (List[str], optional) Keys to extract prompts from data

  • result_key (List[str], optional) Keys to store responses in data

  • file_name_extension (str, optional) Extension for output files

  • image_key (List[Optional[str]], optional) Keys for image paths in VLM tasks

  • kwargs Additional model parameters (e.g., temperature, max_new_tokens)

Examples:

For LLM Usage:

import trusteval
await trusteval.generate_responses(
    data_folder='path/to/data',
    request_type=['llm'],
    async_list=['model1', 'model2'],
    sync_list=['model3'],
    prompt_key=['prompt_key'],
    result_key=['result_key'],
    temperature=0.7  # Optional: model parameter
)

For VLM Usage:

import trusteval
await trusteval.generate_responses(
    data_folder='path/to/data',
    request_type=['vlm'],
    async_list=['model1', 'model2'],
    prompt_key=['prompt_key'],
    result_key=['result_key'],
    image_key=['image_path'],
    max_new_tokens=100 # Optional: model parameter
)

generate_images

Definition:

generate_images(
    base_dir: str,
    aspect: str,
    local_models: Optional[List[str]] = None,
    api_models: Optional[List[str]] = None
) -> None
Parameters:

  • base_dir (str) Base directory for data and output

  • aspect (str) Evaluation aspect (‘robustness’, ‘fairness’, etc.)

  • local_models (List[str], optional) List of local model names

  • api_models (List[str], optional) List of API model names

Example Usage:

import trusteval

trusteval.generate_images(
    base_dir=base_dir = 'path/to/base_dir',
    aspect='robustness_t2i',
    local_models=['model1', 'model2'],
    api_models=['model3']
)

lm_response

The lm_response module processes responses from different language models, handling both asynchronous and synchronous services.

Classes

ResponseProcessor

The ResponseProcessor class processes responses from different models, handling both asynchronous and synchronous services.

Parameters:

  • request_type (str) The type of request (‘llm’ or ‘vlm’)

  • async_list (List[str], optional) List of asynchronous model names

  • sync_list (List[str], optional) List of synchronous model names

  • save_path (str, optional) Path to save the processed responses

Functions

process_async_service

Definition:

process_async_service(
    service: ModelService,
    model_name: str,
    prompt: str,
    image_urls: Optional[List[str]],
    **kwargs
) -> Dict[str, Any]
Parameters:

  • service (ModelService) The ModelService instance

  • model_name (str) Name of the model

  • prompt (str) The input prompt

  • image_urls (List[str], optional) List of image URLs for VLM

  • kwargs Additional model parameters

process_sync_service

Definition:

process_sync_service(
    service: ModelService,
    model_name: str,
    prompt: str,
    image_urls: Optional[List[str]]
) -> Dict[str, Any]
Parameters:

  • service (ModelService) The ModelService instance

  • model_name (str) Name of the model

  • prompt (str) The input prompt

  • image_urls (List[str], optional) List of image URLs for VLM

t2i_response

The t2i_response module generates images based on text prompts using various models.

Functions

generate_images_local

Definition:

generate_images_local(
    prompts: List[str],
    output_paths: List[str],
    model_name: Optional[str] = None
) -> None
Parameters:

  • prompts (List[str]) List of prompt strings

  • output_paths (List[str]) List of output file paths

  • model_name (str, optional) Name of the model

generate_images_api

Definition:

generate_images_api(
    prompts: List[str],
    output_paths: List[str],
    model_name: Optional[str] = None
) -> None
Parameters:

  • prompts (List[str]) List of prompt strings

  • output_paths (List[str]) List of output file paths

  • model_name (str, optional) Name of the model

process_data

Definition:

process_data(
    data_path: Optional[str] = None,
    model_name: Optional[str] = None,
    base_dir: Optional[str] = None,
    process_type: str = 'local',
    aspect: Optional[str] = None
) -> None
Parameters:

  • data_path (str, optional) Path to the JSON data file

  • model_name (str, optional) Name of the model

  • base_dir (str, optional) Base directory for output

  • process_type (str) Type of processing (‘local’ or ‘api’)

  • aspect (str, optional) Evaluation aspect