Promptfoo

Run Promptfoo evals on 1600+ LLMs with observability and cost tracking via Portkey.

Promptfoo is an open source library for evaluating LLM output quality.

With Portkey, you can:

• Manage prompts with versioning and call them in Promptfoo
• Run evals on 1600+ LLMs including private/local models
• Track costs and metrics for all eval runs
• Avoid rate limits with load balancing and caching

1. Reference Portkey Prompts

Use prompts stored in Portkey directly in Promptfoo:

1. Set PORTKEY_API_KEY environment variable
2. Use portkey:// prefix with your prompt ID:

prompts:
  - portkey://pp-test-prompt-669f48
providers:
  - openai:gpt-3.5-turbo-0613
tests:
  - vars:
      topic: ...

Variables from test cases are automatically plugged into the Portkey prompt.

Promptfoo doesn’t follow temperature, model, and other parameters set in Portkey. Set them in the providers configuration.

2. Route to Any Provider

Set PORTKEY_API_KEY and configure providers with the portkey: prefix:

OpenAI

providers:
  id: portkey:gpt-4o
  config:
    portkeyProvider: openai

Cloud Providers (Azure, Bedrock, Vertex)

Azure OpenAI

providers:
  id: portkey:xxx
  config:
    portkeyVirtualKey: "@azure-openai-prod"

3. Track Costs & Metrics

Add metadata to segment requests and track costs per team/project:

providers:
  id: portkey:claude-3-opus-20240229
  config:
    portkeyVirtualKey: "@anthropic-prod"
    portkeyMetadata:
      team: alpha9
      prompt: classification
    portkeyTraceId: run_1

Filter and group by these keys in Portkey dashboards.

4. Avoid Rate Limits with Caching

Create a config with load balancing and caching for high-volume evals:

{
  "cache": { "mode": "simple" },
  "strategy": { "mode": "loadbalance" },
  "targets": [
    { "provider": "@openai-key-1" },
    { "provider": "@openai-key-2" },
    { "provider": "@openai-key-3" }
  ]
}

Reference the config in your YAML:

providers:
  id: portkey:claude-3-opus-20240229
  config:
    portkeyConfig: pc-your-config-id

---

Roadmap

🚧 Coming soon: View Promptfoo eval results in Portkey’s feedback section.

Next Steps

Model Catalog

Set up providers

Prompt Management

Version and manage prompts