Promptfoo

What is Promptfoo

Promptfoo a library designed for evaluating and testing the quality of Large Language Model (LLM) prompts. It provides tools for creating test datasets, setting up evaluation metrics, and selecting the best prompts and models. Aimed at developers, it helps in measuring LLM quality improvements and catching regressions, offering both a web viewer and a command-line interface for integration into existing workflows. Promptfoo is utilized by LLM applications serving millions of users, streamlining the process of prompt evaluation and model selection.

Key Features

• Test Dataset Creation: Allows users to use a sample of user inputs for objective prompt tuning.
• Evaluation Metrics Setup: Offers built-in metrics and custom metric options for evaluating prompts.
• Prompt & Model Selection: Enables side-by-side comparison of prompts and model outputs for optimal selection.
• Web Viewer & Command Line Interface: Provides tools for easy integration into test/CI workflows.