litellm-e2e · build #718

889 passed, 0 failed, 57 skipped

1.106.0stable

Oct 18, 2026, 3:47 AM UTC ·955bdaa ↗· build #718 ↗· 1.106.0 ↗· run history

Before a LiteLLM release ships, the full end-to-end test suite runs against a real deployed proxy. This page shows how every run went — nightlies, release candidates, and stables. Click a module to drill into its tests, and open a test to see the steps it ran.

All 946 tests

  1. ✓test_anthropic_chat_returns_thinking_content TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • reasoning
    1. Add a deployment named e2e-anthropic-thinking-e39b4533e154 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-thinking-e39b4533e154
    View test on GitHub ↗
  2. ✓test_anthropic_chat_returns_tool_call TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • function_calling
    1. Add a deployment named e2e-anthropic-tool-fbb66b469217 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-tool-fbb66b469217
    View test on GitHub ↗
  3. ✓test_anthropic_chat_streams_real_content TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-anthropic-stream-21284659f235 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-anthropic-stream-21284659f235
    View test on GitHub ↗
  4. ✓test_anthropic_chat_streams_tool_call TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • function_calling
    1. Add a deployment named e2e-anthropic-tool-stream-ada489244468 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-anthropic-tool-stream-ada489244468
    View test on GitHub ↗
  5. ✓test_anthropic_chat_structured_output_conforms_to_schema TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • response_schema
    1. Add a deployment named e2e-anthropic-schema-fbc1fa59b84e that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-schema-fbc1fa59b84e
    View test on GitHub ↗
  6. ✓test_anthropic_chat_vision_describes_image TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • vision
    1. Add a deployment named e2e-anthropic-vision-02b1161221dd that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-vision-02b1161221dd
    View test on GitHub ↗
  7. ✓test_anthropic_passthrough_nonstreaming_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to claude-haiku-4-5 through /anthropic with streaming set to False
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  8. ✓test_anthropic_passthrough_streaming_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to claude-haiku-4-5 through /anthropic with streaming set to True
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  9. ✓test_anthropic_passthrough_tool_call_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to claude-haiku-4-5 through /anthropic with streaming set to False
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  10. ✓test_anthropic_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • prompt_caching
    1. Add a deployment named e2e-anthropic-cache-f6f47ccd7f73 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /chat/completions
    View test on GitHub ↗
  11. ✓test_anthropic_with_extended_thinking TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    • reasoning
    1. Add a deployment named e2e-chat-tool-loop-8c07e8e16697 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-8c07e8e16697
    View test on GitHub ↗
  12. ✓test_array_input_returns_vectors TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • openai
    • openai/text-embedding-3-small
    1. Add a deployment named e2e-embeddings-array-50c862dbfdf3 that calls openai/text-embedding-3-small
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  13. ✓test_audio_speech_returns_audio TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-tts
    1. Add a deployment named e2e-speech-ff750937901b that calls openai/gpt-4o-mini-tts
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  14. ✓test_audio_speech_streams_audio_chunks TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • stream
    • openai
    • openai/gpt-4o-mini-tts
    1. Add a deployment named e2e-speech-1e68526b1432 that calls openai/gpt-4o-mini-tts
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  15. ✓test_audio_transcriptions_returns_text TestAudioTranscriptionstest_audio_transcriptions_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-transcribe
    1. Add a deployment named e2e-transcribe-5680399aef07 that calls openai/gpt-4o-mini-transcribe
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  16. ✓test_azure_foundry_chat_returns_content TestAzureFoundryChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    1. Add a deployment named e2e-azure-foundry-chat-348014543cb4 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-azure-foundry-chat-348014543cb4
    View test on GitHub ↗
  17. ✓test_azure_openai_chat_returns_content TestAzureOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-azure-openai-chat-b84c6a42cb40 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-azure-openai-chat-b84c6a42cb40
    View test on GitHub ↗
  18. ✓test_azure_openai_chat_returns_tool_call TestAzureOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    • function_calling
    1. Add a deployment named e2e-azure-openai-tool-01e59c6ff1bf that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-azure-openai-tool-01e59c6ff1bf
    View test on GitHub ↗
  19. ✓test_basic_nonstream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    1. Add a deployment named e2e-azure-foundry-messages-586574c1cb8b that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-586574c1cb8b and user id: e2e-test-user
    View test on GitHub ↗
  20. ✓test_basic_stream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    • llm-translation
    • messages
    • stream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    1. Add a deployment named e2e-azure-foundry-messages-d23a775b0633 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-d23a775b0633 and user id: e2e-test-user
    View test on GitHub ↗
  21. ✓test_batch_s3_keys_do_not_break_chat TestBedrockBatchDeploymentServesChattest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-batch-chat-05b172ca7b3b that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  22. ✓test_bedrock_converse TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/converse/us.anthropic.claude-sonnet-5-5
    • function_calling
    1. Add a deployment named e2e-chat-tool-loop-c89ee9652ccf that calls bedrock/converse/us.anthropic.claude-sonnet-5-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-c89ee9652ccf
    View test on GitHub ↗
  23. ✓test_bedrock_converse_chat_returns_content TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-chat-b9dfde19078f that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-chat-b9dfde19078f
    View test on GitHub ↗
  24. ✓test_bedrock_converse_chat_returns_reasoning TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • reasoning
    1. Add a deployment named e2e-bedrock-thinking-21e513eaf5c1 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-thinking-21e513eaf5c1
    View test on GitHub ↗
  25. ✓test_bedrock_converse_chat_returns_tool_call TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • function_calling
    1. Add a deployment named e2e-bedrock-tool-cfcadb700460 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-tool-cfcadb700460
    View test on GitHub ↗
  26. ✓test_bedrock_converse_chat_streams_real_content TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-stream-fbbfd1251ff0 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bedrock-stream-fbbfd1251ff0
    View test on GitHub ↗
  27. ✓test_bedrock_converse_chat_vision_describes_image TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • vision
    1. Add a deployment named e2e-bedrock-vision-ad6ea6726b32 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-vision-ad6ea6726b32
    View test on GitHub ↗
  28. ✓test_bedrock_converse_reads_a_pdf_sent_by_url TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.amazon.nova-2-lite-v1:0
    • pdf_input
    1. Add a deployment named e2e-bedrock-document-59ae4b0b017d that calls bedrock/us.amazon.nova-2-lite-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-document-59ae4b0b017d
    View test on GitHub ↗
  29. ✓test_bedrock_converse_with_extended_thinking TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/converse/us.anthropic.claude-sonnet-4-6
    • function_calling
    • reasoning
    1. Add a deployment named e2e-chat-tool-loop-05854f19b66e that calls bedrock/converse/us.anthropic.claude-sonnet-4-6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-05854f19b66e
    View test on GitHub ↗
  30. ✓test_bedrock_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • bedrock
    • bedrock/amazon.titan-embed-text-v2:0
    1. Add a deployment named e2e-embeddings-bedrock-792b96b466e3 that calls bedrock/amazon.titan-embed-text-v2:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  31. ✓test_bedrock_image_generation_returns_image TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images
    • nonstream
    • bedrock
    • bedrock/stability.stable-image-core-v1:1
    1. Add a deployment named e2e-bedrock-image-c2c2aa8caf1f that calls bedrock/stability.stable-image-core-v1:1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  32. ✓test_bedrock_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • prompt_caching
    1. Add a deployment named e2e-bedrock-cache-a95e6ca60f9d that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /chat/completions
    View test on GitHub ↗
  33. ✓test_bedrock_request_id_header_surfaces TestBedrockResponseHeaderstest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-headers-050952fd0296 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  34. ✓test_bedrock_request_id_header_surfaces_on_stream TestBedrockResponseHeaderstest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-headers-stream-2beafbb2124c that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bedrock-headers-stream-2beafbb2124c
    View test on GitHub ↗
  35. ✓test_bedrock_rerank_scores_top_n TestReranktest_rerank_e2e.py›
    • llm-translation
    • rerank
    • nonstream
    • bedrock
    • bedrock/arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0
    1. Add a deployment named e2e-bedrock-rerank-93423437bfcb that calls bedrock/arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/rerank request to e2e-bedrock-rerank-93423437bfcb for "What is the capital of the United States?"
    View test on GitHub ↗
  36. ✓test_bedrock_titan_embeds_token_array_input_as_its_decoded_text TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • bedrock
    • bedrock/amazon.titan-embed-text-v2:0
    1. Add a deployment named e2e-embeddings-titan-tokens-b1cce3913594 that calls bedrock/amazon.titan-embed-text-v2:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  37. ✓test_bridged_stream_delivers_content_finish_reason_and_done TestResponsesBridgeChatCompletionsStreamingtest_responses_bridge_streaming_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.3-codex
    1. Add a deployment named e2e-bridge-stream-f6c103b993ce that calls openai/gpt-5.3-codex
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bridge-stream-f6c103b993ce
    View test on GitHub ↗
  38. ✓test_bridged_stream_reassembles_tool_call TestResponsesBridgeChatCompletionsStreamingtest_responses_bridge_streaming_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.3-codex
    • function_calling
    1. Add a deployment named e2e-bridge-stream-7bbc8ea83210 that calls openai/gpt-5.3-codex
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bridge-stream-7bbc8ea83210
    View test on GitHub ↗
  39. ✓test_bridged_stream_shares_one_chunk_id TestResponsesBridgeChatCompletionsStreamingtest_responses_bridge_streaming_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.3-codex
    1. Add a deployment named e2e-bridge-stream-f31e15bb9d37 that calls openai/gpt-5.3-codex
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bridge-stream-f31e15bb9d37
    View test on GitHub ↗
  40. ✓test_cache_read_tokens_bill_at_the_cache_read_rate TestTogetherChatCompletionstest_together_ai_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • together_ai
    • prompt_caching
    1. Add a deployment named e2e-together-9663a11a8f0c that calls together_ai/deepseek-ai/DeepSeek-V4-Flash-0731
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-9663a11a8f0c
    4. Send a request to /chat/completions
    5. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  41. ✓test_caller_completion_window_bills_its_rates TestSailResponsestest_sail_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-ec10d5abb348 that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  42. ✓test_calls_without_auth_is_denied TestRealtimeHttptest_realtime_http_e2e.py›
    • proxy-auth
    • realtime
    1. Send a request to /v1/realtime/calls
    View test on GitHub ↗
  43. ✓test_cancel_background_response[azure] TestBackgroundResponseCanceltest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-resp-cancel-1a3ebeec9b13 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  44. ✓test_cancel_background_response[openai] TestBackgroundResponseCanceltest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-resp-cancel-1c9c2e455e35 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  45. ✓test_cancel_background_streaming_response_by_streamed_id TestBackgroundResponseCanceltest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-resp-cancel-stream-952c0545af17 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  46. ✓test_chat_returns_real_completion[claude-haiku-4-5-anthropic] TestChatCompletionsRegressiontest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to claude-haiku-4-5
    View test on GitHub ↗
  47. ✓test_chat_returns_real_completion[gemini-2.5-flash-gemini] TestChatCompletionsRegressiontest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  48. ✓test_chat_returns_real_completion[gpt-5.5-openai] TestChatCompletionsRegressiontest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  49. ✓test_chat_stream_is_sse_and_ends_with_done TestChatStreamContracttest_chat_stream_contract_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.6
    1. Add a deployment named e2e-chat-stream-838b17e60167 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-chat-stream-838b17e60167
    View test on GitHub ↗
  50. ✓test_chat_template_kwargs_reach_together TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • together_ai/Qwen/Qwen3.5-9B
    • reasoning
    1. Add a deployment named e2e-together-034079936f07 that calls together_ai/Qwen/Qwen3.5-9B
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-034079936f07
    View test on GitHub ↗
  51. ✓test_client_secret_missing_auth_is_denied TestRealtimeHttptest_realtime_http_e2e.py›
    • proxy-auth
    • realtime
    1. Add a deployment named e2e-realtime-http-1082b7659570 that calls openai/gpt-realtime
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/realtime/client_secrets
    View test on GitHub ↗
  52. ✓test_client_tool_call_marks_work_without_injection_points[azure-foundry] TestCacheControlInjectionToolCallstest_cache_control_injection_tool_calls_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    • function_calling
    • prompt_caching
    1. Add a deployment named e2e-cache-control-tool-calls-azure_foundry-7c781bba94a4 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  53. ✓test_client_tool_call_marks_work_without_injection_points[vertex] TestCacheControlInjectionToolCallstest_cache_control_injection_tool_calls_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/claude-sonnet-5
    • function_calling
    • prompt_caching
    1. Add a deployment named e2e-cache-control-tool-calls-vertex-6a2722e96929 that calls vertex_ai/claude-sonnet-5
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  54. ✓test_cohere_chat_returns_content TestCohereChattest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • cohere
    • cohere/command-r-08-2024
    1. Add a deployment named e2e-cohere-chat-704e96380c60 that calls cohere/command-r-08-2024
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-cohere-chat-704e96380c60
    View test on GitHub ↗
  55. ✓test_cohere_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • cohere
    • cohere/embed-v4.0
    1. Add a deployment named e2e-embeddings-cohere-39b7373c965c that calls cohere/embed-v4.0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  56. ✓test_container_file_lifecycle_through_the_gateway TestOpenAIContainerFilestest_containers_e2e.py›
    • llm-translation
    • containers
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  57. ✓test_contents_are_counted_by_the_provider_endpoint[gemini-2.5-flash-vertex] TestGeminiContentsTokenCountingtest_token_counter_gemini_contents_e2e.py›
    • llm-translation
    • count_tokens
    • vertex_ai
    • gemini-2.5-flash-vertex
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /utils/token_counter
    View test on GitHub ↗
  58. ✓test_contents_are_counted_by_the_provider_endpoint[gemini-2.5-flash] TestGeminiContentsTokenCountingtest_token_counter_gemini_contents_e2e.py›
    • llm-translation
    • count_tokens
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /utils/token_counter
    View test on GitHub ↗
  59. ✓test_converse_empty_messages_returns_client_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-3a908deef2e1 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-3a908deef2e1/converse
    View test on GitHub ↗
  60. ✓test_converse_invalid_model_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    1. Add a deployment named e2e-bedrock-native-e2cdea32f606 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/does-not-exist/converse
    View test on GitHub ↗
  61. ✓test_converse_missing_messages_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-443eaf29a68e that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-443eaf29a68e/converse
    View test on GitHub ↗
  62. ✓test_converse_output_format_returns_schema_json_text TestBedrockMessagestest_messages_bedrock_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • response_schema
    1. Add a deployment named e2e-messages-bedrock-5550b7709ffe that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  63. ✓test_converse_returns_assistant TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-f7fe7aedd688 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-f7fe7aedd688/converse
    View test on GitHub ↗
  64. ✓test_converse_stream_returns_chunks TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • stream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-45e5b0c8397b that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-45e5b0c8397b/converse-stream
    View test on GitHub ↗
  65. ✓test_cost_header_and_spend_row_match_the_registry_price TestTogetherChatCompletionstest_together_ai_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • together_ai
    1. Add a deployment named e2e-together-f90b1f80fe87 that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  66. ✓test_cost_header_matches_the_spend_log[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 with the prompt "Say hello. 680a63afb1c7" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  67. ✓test_cost_header_matches_the_spend_log[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 with the prompt "Say hello. 4eac27a26140" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  68. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 with the prompt "Say hello. 451a35cdbb04" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  69. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d with the prompt "Say hello. edc1faf9ab33" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  70. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 with the prompt "Say hello. 1747d1d870b3" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  71. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 with the prompt "Say hello. b8b55135f05d" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  72. ✓test_cost_header_matches_the_spend_log[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 with the prompt "Say hello. 480b4f0cd6f6" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  73. ✓test_cost_header_matches_the_spend_log[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 with the prompt "Say hello. f7d3e08bd69f" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  74. ✓test_cost_header_matches_the_spend_log[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 with the prompt "Say hello. 03089220a32c" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  75. ✓test_cost_header_matches_the_spend_log[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d with the prompt "Say hello. 9995e5f5ac9b" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  76. ✓test_cost_header_matches_the_spend_log[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 with the prompt "Say hello. 89892fe66187" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  77. ✓test_cost_header_matches_the_spend_log[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 with the prompt "Say hello. 13af680ff753" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  78. ✓test_cost_header_matches_the_spend_log[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 with the prompt "Say hello. bd92e5c88722" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  79. ✓test_cost_header_matches_the_spend_log[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 with the prompt "Say hello. d14fe663ed47" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  80. ✓test_cost_header_matches_the_spend_log[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 with the prompt "Say hello. 363f978fa547" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  81. ✓test_cost_header_matches_the_spend_log[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d with the prompt "Say hello. 993a27d57933" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  82. ✓test_cost_header_matches_the_spend_log[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 with the prompt "Say hello. 1dab7789dbeb" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  83. ✓test_cost_header_matches_the_spend_log[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 with the prompt "Say hello. 8a428c7d2d1d" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  84. ✓test_count_matches_billed_input_tokens_for_a_model_bedrock_runtime_cannot_count TestBedrockMessagesCountTokenstest_messages_count_tokens_bedrock_e2e.py›
    • llm-translation
    • count_tokens
    • bedrock
    • bedrock/global.anthropic.claude-opus-4-8
    1. Add a deployment named e2e-count-tokens-bedrock-9397c8fd95eb that calls bedrock/global.anthropic.claude-opus-4-8
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  85. ✓test_create_batch_missing_input_file_id_returns_error TestFilesBatchesContracttest_files_batches_contract_e2e.py›
    • llm-translation
    • batches
    • batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /v1/batches
    View test on GitHub ↗
  86. ✓test_create_client_secret TestRealtimeHttptest_realtime_http_e2e.py›
    • llm-translation
    • realtime
    • openai
    • openai/gpt-realtime
    1. Add a deployment named e2e-realtime-http-6e79e144bfae that calls openai/gpt-realtime
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /v1/realtime/client_secrets
    View test on GitHub ↗
  87. ✓test_create_list_retrieve_delete_lifecycle TestVectorStorestest_vector_stores_e2e.py›
    • llm-translation
    • vector_stores
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. POST /v1/vector_stores
    3. GET /v1/vector_stores/vs_6ad44605948c8191926f23ba4d2563e7
    4. GET /v1/vector_stores
    5. DELETE /v1/vector_stores/vs_6ad44605948c8191926f23ba4d2563e7
    View test on GitHub ↗
  88. ✓test_credential_backed_messages TestCredentialBackedMessagestest_credential_messages_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Save the provider credential e2e-cred-ed8d0c3492a7
    2. Add a deployment named e2e-cred-messages-ed8d0c3492a7 that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  89. ✓test_custom_pricing_is_billed_at_configured_rate TestCustomPricingtest_custom_pricing_e2e.py›
    • cost-map
    • spend_reporting
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named custom-priced-flash-7be4251eb58d that calls gemini/gemini-2.5-flash
    3. Send a /chat/completions request to custom-priced-flash-7be4251eb58d
    4. GET /spend/logs
    View test on GitHub ↗
  90. ✓test_custom_pricing_is_isolated_from_sibling_deployment TestCustomPricingtest_custom_pricing_e2e.py›
    • cost-map
    • model_management
    1. Add a deployment named custom-priced-flash-fe1c6de1430e that calls gemini/gemini-2.5-flash
    2. Add a deployment named base-flash-4543e21f6ba4 that calls gemini/gemini-2.5-flash
    3. List the deployments from /model/info
    View test on GitHub ↗
  91. ✓test_deleted_response_is_no_longer_retrievable[azure] TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-resp-delete-a0a083daeb02 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  92. ✓test_deleted_response_is_no_longer_retrievable[openai] TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-resp-delete-aac2bbb6aa87 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  93. ✓test_embeddings_call_logs_its_cost TestOpenAIPassthroughSpendtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • openai
    • text-embedding-3-small
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/embeddings request to text-embedding-3-small through /openai_passthrough for "cost this sentence 6def653c26ad"
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  94. ✓test_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • openai
    • openai/text-embedding-3-small
    1. Add a deployment named e2e-embeddings-a1f796e75e98 that calls openai/text-embedding-3-small
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  95. ✓test_empty_image_returns_error TestImageEdittest_image_edits_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1
    1. Add a deployment named e2e-image-edit-8cbf904adf84 that calls openai/gpt-image-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  96. ✓test_empty_input_returns_client_error TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-responses-val-6c30fd47c15c that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/responses
    View test on GitHub ↗
  97. ✓test_empty_messages_returns_client_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-a9ffe3601d3c that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  98. ✓test_empty_prompt_returns_client_error TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1-mini
    1. Add a deployment named e2e-image-26bc8c6e7344 that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/images/generations
    View test on GitHub ↗
  99. ✓test_empty_prompt_returns_error TestImageEdittest_image_edits_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1
    1. Add a deployment named e2e-image-edit-a7206bd88647 that calls openai/gpt-image-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  100. ✓test_file_attach_poll_and_search TestVectorStorestest_vector_stores_e2e.py›
    • llm-translation
    • vector_stores
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload vs_doc.txt to /v1/files
    3. POST /v1/vector_stores
    4. POST /v1/vector_stores/vs_6ad4461127a881919c87e690cef1afc1/files
    5. GET /v1/vector_stores/vs_6ad4461127a881919c87e690cef1afc1/files/file-22uc9Hq8aBrXgUwSwygrnV
    6. POST /v1/vector_stores/vs_6ad4461127a881919c87e690cef1afc1/search
    7. DELETE /v1/vector_stores/vs_6ad4461127a881919c87e690cef1afc1/files/file-22uc9Hq8aBrXgUwSwygrnV
    View test on GitHub ↗
  101. ✓test_flagged_model_keeps_prompt_cache_across_system_reminder TestAnthropicChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-opus-4-8
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-chat-midsys-e86d595b7f89 that calls anthropic/claude-opus-4-8
    2. Generate a virtual key with models: e2e-chat-midsys-e86d595b7f89 and user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  102. ✓test_flagged_model_keeps_prompt_cache_across_system_reminder TestBedrockInvokeChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-sonnet-5
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-chat-midsys-c3efcb76e183 that calls bedrock/invoke/us.anthropic.claude-sonnet-5
    2. Generate a virtual key with models: e2e-chat-midsys-c3efcb76e183 and user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  103. ✓test_gemini TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • gemini
    • gemini/gemini-3.5-flash-lite
    • function_calling
    1. Add a deployment named e2e-chat-tool-loop-67a8472fb5e2 that calls gemini/gemini-3.5-flash-lite
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-67a8472fb5e2
    View test on GitHub ↗
  104. ✓test_gemini_chat_returns_content_and_logs_cost TestGeminiChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Add a deployment named e2e-gemini-chat-72adf9a0d64a that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-gemini-chat-72adf9a0d64a
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  105. ✓test_gemini_passthrough_nonstreaming_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a Gemini generateContent request to gemini-2.5-flash through /gemini
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  106. ✓test_gemini_passthrough_streaming_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a Gemini streamGenerateContent request to gemini-2.5-flash through /gemini
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  107. ✓test_gemini_passthrough_tool_call_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • gemini
    • gemini-2.5-flash
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a Gemini generateContent request to gemini-2.5-flash through /gemini
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  108. ✓test_generate_content_returns_response_cost_header TestGoogleNativeGenerateContenttest_google_native_e2e.py›
    • llm-translation
    • google_genai
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-google-native-2586559f6af6 that calls gemini/gemini-2.5-flash
    3. Send a request to /v1beta/models/e2e-google-native-2586559f6af6:generateContent
    View test on GitHub ↗
  109. ✓test_image_edit_returns_image TestImageEdittest_image_edits_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1
    1. Add a deployment named e2e-image-edit-2c33a114534a that calls openai/gpt-image-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  110. ✓test_image_generation_returns_image TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1-mini
    1. Add a deployment named e2e-image-85a7cf46277c that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  111. ✓test_injection_points_respect_cap_with_tool_call_marks[azure-foundry] TestCacheControlInjectionToolCallstest_cache_control_injection_tool_calls_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    • function_calling
    • prompt_caching
    1. Add a deployment named e2e-cache-control-tool-calls-azure_foundry-372a7dcb9819 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  112. ✓test_injection_points_respect_cap_with_tool_call_marks[vertex] TestCacheControlInjectionToolCallstest_cache_control_injection_tool_calls_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/claude-sonnet-5
    • function_calling
    • prompt_caching
    1. Add a deployment named e2e-cache-control-tool-calls-vertex-a37a807885a8 that calls vertex_ai/claude-sonnet-5
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  113. ✓test_input_items_list_the_stored_prompt TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-resp-items-111128b2707d that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  114. ✓test_invalid_chunking_returns_error TestVectorStorestest_vector_stores_e2e.py›
    • llm-translation
    • vector_stores
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /v1/vector_stores
    View test on GitHub ↗
  115. ✓test_invalid_max_completion_tokens_return_client_errors TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-9e3a61546a98 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  116. ✓test_invalid_n_returns_client_error TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1-mini
    1. Add a deployment named e2e-image-c14444e2c8ff that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/images/generations
    View test on GitHub ↗
  117. ✓test_invalid_role_returns_client_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-2fcc3a0101b9 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  118. ✓test_invalid_size_returns_client_error TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1-mini
    1. Add a deployment named e2e-image-460c7b6ec61e that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/images/generations
    View test on GitHub ↗
  119. ✓test_invalid_temperatures_return_client_errors TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-9e4d65545531 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  120. ✓test_invoke_invalid_temperature_returns_client_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-a7741739d08d that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-a7741739d08d/invoke
    View test on GitHub ↗
  121. ✓test_invoke_missing_max_tokens_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-0111194d1c53 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-0111194d1c53/invoke
    View test on GitHub ↗
  122. ✓test_invoke_missing_messages_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-ed82867b9d8f that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-ed82867b9d8f/invoke
    View test on GitHub ↗
  123. ✓test_invoke_regional_id_completes TestBedrockInvokeRegionalModelIdstest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-invoke-27a54af381f2 that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-invoke-27a54af381f2
    View test on GitHub ↗
  124. ✓test_invoke_regional_id_streams TestBedrockInvokeRegionalModelIdstest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-invoke-stream-3fa36fc5694a that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bedrock-invoke-stream-3fa36fc5694a
    View test on GitHub ↗
  125. ✓test_invoke_returns_message TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-419b28745495 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-419b28745495/invoke
    View test on GitHub ↗
  126. ✓test_invoke_stream_returns_chunks TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • stream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-8a912d5a5d6b that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-8a912d5a5d6b/invoke-with-response-stream
    View test on GitHub ↗
  127. ✓test_messages_logs_cost_matching_the_response_header TestAnthropicMessagestest_messages_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-messages-cost-fb43ea624ffb that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  128. ✓test_messages_output_format_returns_schema_json TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • response_schema
    1. Add a deployment named e2e-messages-7a6fe16c1d89 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  129. ✓test_messages_returns_completion TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-messages-6e042d52f360 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  130. ✓test_messages_streams_completion TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-messages-dde6af10c339 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  131. ✓test_messages_tool_use TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-messages-04a1b2e0d8c8 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  132. ✓test_missing_file_returns_error TestAudioTranscriptionstest_audio_transcriptions_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-transcribe
    1. Add a deployment named e2e-transcribe-d7e7f3d26957 that calls openai/gpt-4o-mini-transcribe
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload empty.wav to /v1/audio/transcriptions
    View test on GitHub ↗
  133. ✓test_missing_input_returns_error TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    1. Add a deployment named e2e-embeddings-missin-f32bc667c212 that calls openai/text-embedding-3-small
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /embeddings
    View test on GitHub ↗
  134. ✓test_missing_messages_returns_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-0db557bddc04 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  135. ✓test_missing_model_returns_client_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    1. Add a deployment named e2e-chat-sec-1350d5aa1650 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  136. ✓test_missing_model_returns_client_error TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /embeddings
    View test on GitHub ↗
  137. ✓test_missing_model_returns_client_error TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /v1/responses
    View test on GitHub ↗
  138. ✓test_missing_model_returns_error TestAudioTranscriptionstest_audio_transcriptions_e2e.py›
    • llm-translation
    • audio
    1. Add a deployment named e2e-transcribe-6710471ab984 that calls openai/gpt-4o-mini-transcribe
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload weather_question_24k.wav to /v1/audio/transcriptions
    View test on GitHub ↗
  139. ✓test_missing_model_returns_error TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages
    1. Add a deployment named e2e-messages-cb388d2bb95d that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/messages
    View test on GitHub ↗
  140. ✓test_mistral TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • mistral
    • mistral/mistral-medium-3.5
    • function_calling
    1. Add a deployment named e2e-chat-tool-loop-1e4ddda7d3a0 that calls mistral/mistral-medium-3.5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-1e4ddda7d3a0
    View test on GitHub ↗
  141. ✓test_mistral_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • mistral
    • mistral/mistral-embed
    1. Add a deployment named e2e-embeddings-mistral-d39954532084 that calls mistral/mistral-embed
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  142. ✓test_model_info_reports_custom_pricing TestCustomPricingtest_custom_pricing_e2e.py›
    • cost-map
    • model_management
    1. Add a deployment named custom-priced-flash-e0a005d0cf7a that calls gemini/gemini-2.5-flash
    2. List the deployments from /model/info
    View test on GitHub ↗
  143. ✓test_moderations_flags_violent_content TestModerationstest_moderations_e2e.py›
    • llm-translation
    • moderations
    • nonstream
    • openai
    • openai/omni-moderation-latest
    1. Add a deployment named e2e-moderation-a4524e5cacb2 that calls openai/omni-moderation-latest
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  144. ✓test_moderations_passes_benign_content TestModerationstest_moderations_e2e.py›
    • llm-translation
    • moderations
    • nonstream
    • openai
    • openai/omni-moderation-latest
    1. Add a deployment named e2e-moderation-452e8293a3a4 that calls openai/omni-moderation-latest
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  145. ✓test_multi_turn_history_is_honored TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-26d91886913b that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-sec-26d91886913b
    View test on GitHub ↗
  146. ✓test_nova_sonic_response_create_completes TestNovaSonicRealtimerealtime/test_realtime_bedrock_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • bedrock
    • bedrock/amazon.nova-2-sonic-v1:0
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-nova-sonic-1e30af012873 that calls bedrock/amazon.nova-2-sonic-v1:0
    3. Open a /v1/realtime websocket session to e2e-nova-sonic-1e30af012873
    4. Wait for a session.created event on the realtime websocket
    5. Send the realtime event session.update over the websocket
    6. Wait for a session.updated event on the realtime websocket
    7. Send the realtime event conversation.item.create over the websocket
    8. Send the realtime event response.create over the websocket
    9. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  147. ✓test_nova_sonic_voice_turn_shows_each_sentence_once TestNovaSonicAssistantTextrealtime/test_realtime_bedrock_duplicate_text_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • bedrock
    • bedrock/amazon.nova-2-sonic-v1:0
    • audio_input
    • audio_output
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-nova-sonic-dupe-226264958bbd that calls bedrock/amazon.nova-2-sonic-v1:0
    3. Open a /v1/realtime websocket session to e2e-nova-sonic-dupe-226264958bbd
    4. Wait for a session.created event on the realtime websocket
    5. Send the realtime event session.update over the websocket
    6. Wait for a session.updated event on the realtime websocket
    7. Stream the spoken question as mic chunks, then silence until the turn goes idle
    View test on GitHub ↗
  148. ✓test_nova_stream_relays_text_usage_and_stop TestBedrockMessagestest_messages_bedrock_e2e.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • bedrock/us.amazon.nova-2-lite-v1:0
    1. Add a deployment named e2e-messages-bedrock-d79e9d389b24 that calls bedrock/us.amazon.nova-2-lite-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  149. ✓test_ocr_reads_the_document_and_bills_its_pages[azure-ai-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • azure_ai
    • azure_ai/mistral-document-ai-2512
    1. Add a deployment named ocr-content-azure-ai-image-0415d2548b42 that calls azure_ai/mistral-document-ai-2512
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/ocr
    View test on GitHub ↗
  150. ✓test_ocr_reads_the_document_and_bills_its_pages[cohere-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • cohere
    • cohere/parse-v5.0
    1. Add a deployment named ocr-content-cohere-image-a591b941be0b that calls cohere/parse-v5.0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/ocr
    View test on GitHub ↗
  151. ✓test_ocr_reads_the_document_and_bills_its_pages[mistral-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • mistral
    • mistral/mistral-ocr-latest
    1. Add a deployment named ocr-content-mistral-image-d7ee67212baf that calls mistral/mistral-ocr-latest
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/ocr
    View test on GitHub ↗
  152. ✓test_ocr_reads_the_document_and_bills_its_pages[mistral-pdf] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • mistral
    • mistral/mistral-ocr-latest
    1. Add a deployment named ocr-content-mistral-pdf-a17ab9e35b83 that calls mistral/mistral-ocr-latest
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/ocr
    View test on GitHub ↗
  153. ✓test_ocr_reads_the_document_and_bills_its_pages[vertex-mistral-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • vertex_ai
    • vertex_ai/mistral-ocr-2505
    1. Add a deployment named ocr-content-vertex-mistral-image-59aecbaec657 that calls vertex_ai/mistral-ocr-2505
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/ocr
    View test on GitHub ↗
  154. ✓test_openai_chat_logs_cost TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    1. Add a deployment named e2e-openai-cost-2b8a2e7602b9 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-cost-2b8a2e7602b9
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  155. ✓test_openai_chat_reasoning_reports_reasoning_tokens TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • reasoning
    1. Add a deployment named e2e-openai-reasoning-4e4a883d5624 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-reasoning-4e4a883d5624
    View test on GitHub ↗
  156. ✓test_openai_chat_returns_tool_call TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • function_calling
    1. Add a deployment named e2e-openai-tool-2ac02800b27f that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-tool-2ac02800b27f
    View test on GitHub ↗
  157. ✓test_openai_chat_streams_real_content TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.6
    1. Add a deployment named e2e-openai-chat-2d2d60d7c6b3 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-openai-chat-2d2d60d7c6b3
    View test on GitHub ↗
  158. ✓test_openai_chat_streams_tool_call TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.6
    • function_calling
    1. Add a deployment named e2e-openai-tool-stream-ef15d0d5da08 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-openai-tool-stream-ef15d0d5da08
    View test on GitHub ↗
  159. ✓test_openai_chat_structured_output_conforms_to_schema TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • response_schema
    1. Add a deployment named e2e-openai-schema-bd16d05253e9 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-schema-bd16d05253e9
    View test on GitHub ↗
  160. ✓test_openai_chat_vision_describes_image TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o
    • vision
    1. Add a deployment named e2e-openai-vision-12ac78605543 that calls openai/gpt-4o
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-vision-12ac78605543
    View test on GitHub ↗
  161. ✓test_openai_family_model_id_completes_with_max_tokens TestBedrockOpenAIFamilyDefaultRoutetest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/global.openai.gpt-6-sol
    1. Add a deployment named e2e-bedrock-openai-family-1474de4f94a5 that calls bedrock/global.openai.gpt-6-sol
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-openai-family-1474de4f94a5
    View test on GitHub ↗
  162. ✓test_openai_prefix_chat_returns_completion_and_logs_its_cost TestOpenAIProviderPrefixChattest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/chat/completions request to gpt-5.5 through /openai
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  163. ✓test_openai_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • prompt_caching
    1. Add a deployment named e2e-openai-cache-4b39e96c6cb4 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-cache-4b39e96c6cb4
    View test on GitHub ↗
  164. ✓test_openai_service_tier_is_echoed TestServiceTiertest_provider_features_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-service-tier-7d505dc93621 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-service-tier-7d505dc93621
    View test on GitHub ↗
  165. ✓test_output_format_returns_schema_json TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    • response_schema
    1. Add a deployment named e2e-azure-foundry-messages-1b540edd48e0 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-1b540edd48e0 and user id: e2e-test-user
    View test on GitHub ↗
  166. ✓test_passthrough_denies_model_outside_key_allowlist TestPassthroughModelAllowlisttest_passthrough_e2e.py›
    • proxy-auth
    • passthrough
    1. Generate a virtual key with models: gemini-2.5-flash
    2. Send a /v1/messages request to claude-haiku-4-5 through /anthropic with streaming set to False
    View test on GitHub ↗
  167. ✓test_passthrough_prefix_lists_batches_from_openai TestOpenAIPassthroughPrefixtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. List batches from /openai_passthrough/v1/batches
    View test on GitHub ↗
  168. ✓test_passthrough_prefix_uploads_a_file_to_openai TestOpenAIPassthroughPrefixtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload e2e-passthrough-batch.jsonl to /openai_passthrough/v1/files
    View test on GitHub ↗
  169. ✓test_pipecat_tool_smoke[azure]realtime/test_realtime_pipecat_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • azure
    • azure/gpt-realtime
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  170. ✓test_pipecat_tool_smoke[gemini]realtime/test_realtime_pipecat_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • gemini
    • gemini/gemini-3.1-flash-live-preview
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  171. ✓test_pipecat_tool_smoke[openai]realtime/test_realtime_pipecat_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • openai
    • openai/gpt-realtime-2
    • function_calling
    1. Add a realtime deployment that calls openai/gpt-realtime-2
    2. Add a realtime deployment that calls azure/gpt-realtime
    3. Add a realtime deployment that calls gemini/gemini-3.1-flash-live-preview
    4. Add a realtime deployment that calls vertex_ai/gemini-live-2.5-flash-native-audio
    5. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  172. ✓test_plain_call_returns_a_message TestSailMessagestest_sail_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-b29ed1917283 that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  173. ✓test_polly_generative_voice_returns_mp3 TestAwsPollySpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • aws_polly
    • aws_polly/generative
    1. Add a deployment named e2e-speech-polly-a4b962238ff0 that calls aws_polly/generative
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  174. ✓test_realtime_upgrade_reaches_openai_through_the_passthrough_prefix TestOpenAIPassthroughWebsockettest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • websocket
    • openai
    • gpt-realtime-2
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a websocket to /openai_passthrough/v1/realtime and wait for its first event
    View test on GitHub ↗
  175. ✓test_reasoner_returns_reasoning_by_default TestDeepSeekReasoningDisabletest_deepseek_reasoning_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • deepseek
    • deepseek/deepseek-v4-pro
    • reasoning
    1. Add a deployment named e2e-deepseek-reasoner-119ad51d9a8f that calls deepseek/deepseek-v4-pro
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-deepseek-reasoner-119ad51d9a8f
    View test on GitHub ↗
  176. ✓test_reasoning_effort_none_disables_reasoning TestDeepSeekReasoningDisabletest_deepseek_reasoning_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • deepseek
    • deepseek/deepseek-v4-pro
    • reasoning
    1. Add a deployment named e2e-deepseek-reasoner-4a71386bc646 that calls deepseek/deepseek-v4-pro
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-deepseek-reasoner-4a71386bc646
    View test on GitHub ↗
  177. ✓test_reasoning_effort_none_reaches_together TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • together_ai/Qwen/Qwen3.5-9B
    • reasoning
    1. Add a deployment named e2e-together-4ec166da4bba that calls together_ai/Qwen/Qwen3.5-9B
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-4ec166da4bba
    View test on GitHub ↗
  178. ✓test_reasoning_items_replay_into_structured_output_after_tool_call TestResponsesOpenAIHostedFeaturestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    • reasoning
    • response_schema
    1. Add a deployment named e2e-responses-reasoning-a69070e1da2a that calls openai/gpt-5.4-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  179. ✓test_reasoning_streams_as_reasoning_content_deltas TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • together_ai
    • reasoning
    1. Add a deployment named e2e-together-774d6c9c0db6 that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-together-774d6c9c0db6
    View test on GitHub ↗
  180. ✓test_reasoning_surfaces_as_reasoning_content TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • reasoning
    1. Read the model cost map
    2. Add a deployment named e2e-together-20857154ff4c that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    3. Generate a virtual key with user id: e2e-test-user
    4. Send a /chat/completions request to e2e-together-20857154ff4c
    View test on GitHub ↗
  181. ✓test_replayed_reasoning_content_reaches_together TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • together_ai/MiniMaxAI/MiniMax-M3
    • reasoning
    1. Add a deployment named e2e-together-33cac20a7677 that calls together_ai/MiniMaxAI/MiniMax-M3
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-33cac20a7677
    View test on GitHub ↗
  182. ✓test_reply_carries_assistant_text_and_usage[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  183. ✓test_reply_carries_assistant_text_and_usage[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  184. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Start a provider edge server
    2. Add a deployment named e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 that calls openai/gpt-4o-mini
    3. Save the provider credential e2e-matrix-gpt-4o-mini-bf2cbf121c6d
    4. Add a deployment named e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d that calls openai/gpt-4o-mini
    5. Add a deployment named e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 that calls openai/gpt-5.4-mini
    6. Save the provider credential e2e-matrix-gpt-5.4-mini-ded258805787
    7. Add a deployment named e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 that calls openai/gpt-5.4-mini
    8. Add a deployment named e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 that calls anthropic/claude-haiku-4-5
    9. Save the provider credential e2e-matrix-claude-haiku-4-5-6e5e03796d71
    10. Add a deployment named e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 that calls anthropic/claude-haiku-4-5
    11. Generate a virtual key with user id: e2e-test-user
    12. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  185. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  186. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  187. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  188. ✓test_reply_carries_assistant_text_and_usage[chat_completions-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Add a deployment named e2e-ollama-ollama_chat-68a32c9c8050 that calls ollama_chat/gemma4:31b
    2. Add a deployment named e2e-ollama-ollama-9d6596ad7e4b that calls ollama/gemma4:31b
    3. Generate a virtual key with user id: e2e-test-user
    4. Send a /chat/completions request to e2e-ollama-ollama_chat-68a32c9c8050 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  189. ✓test_reply_carries_assistant_text_and_usage[chat_completions-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ollama-ollama-9d6596ad7e4b with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  190. ✓test_reply_carries_assistant_text_and_usage[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  191. ✓test_reply_carries_assistant_text_and_usage[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  192. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  193. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  194. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  195. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  196. ✓test_reply_carries_assistant_text_and_usage[messages-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama_chat-68a32c9c8050 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  197. ✓test_reply_carries_assistant_text_and_usage[messages-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama-9d6596ad7e4b with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  198. ✓test_reply_carries_assistant_text_and_usage[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  199. ✓test_reply_carries_assistant_text_and_usage[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  200. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  201. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  202. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  203. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  204. ✓test_reply_carries_assistant_text_and_usage[responses-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama_chat-68a32c9c8050 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  205. ✓test_reply_carries_assistant_text_and_usage[responses-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama-9d6596ad7e4b with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  206. ✓test_required_tool_arguments_and_correlated_result[nonstream] TestOpenAIMessagesToolContinuationtest_messages_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.6
    • function_calling
    1. Add a deployment named e2e-bridge-tool-abfe7742c770 that calls openai/gpt-5.6
    2. Generate a virtual key with models: e2e-bridge-tool-abfe7742c770 and user id: e2e-test-user
    View test on GitHub ↗
  207. ✓test_required_tool_arguments_and_correlated_result[stream] TestOpenAIMessagesToolContinuationtest_messages_e2e.py›
    • llm-translation
    • messages
    • stream
    • openai
    • openai/gpt-5.6
    • function_calling
    1. Add a deployment named e2e-bridge-tool-b5860eea4163 that calls openai/gpt-5.6
    2. Generate a virtual key with models: e2e-bridge-tool-b5860eea4163 and user id: e2e-test-user
    View test on GitHub ↗
  208. ✓test_rerank_scores_top_n TestReranktest_rerank_e2e.py›
    • llm-translation
    • rerank
    • nonstream
    • cohere
    • cohere/rerank-v3.5
    1. Add a deployment named e2e-rerank-9a6b721af5ec that calls cohere/rerank-v3.5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/rerank request to e2e-rerank-9a6b721af5ec for "What is the capital of the United States?"
    View test on GitHub ↗
  209. ✓test_response_format_json_schema_shapes_the_reply TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • together_ai/Qwen/Qwen3.5-9B
    • response_schema
    1. Add a deployment named e2e-together-f28225d07897 that calls together_ai/Qwen/Qwen3.5-9B
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-f28225d07897
    View test on GitHub ↗
  210. ✓test_responses_anthropic_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-responses-bcf0d1eae4b2 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  211. ✓test_responses_anthropic_returns_function_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-responses-e0a4543c0b0c that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  212. ✓test_responses_anthropic_strict_array_schema_tool_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-responses-ed5ec7650237 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  213. ✓test_responses_anthropic_tool_output_continues_with_previous_response_id TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-responses-6fae30094dea that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  214. ✓test_responses_azure_openai_preview_api_version_accepts_truncation TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-responses-azure-preview-20604c66df20 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  215. ✓test_responses_azure_openai_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-responses-azure-openai-3851c3485d69 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  216. ✓test_responses_azure_openai_returns_function_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    • function_calling
    1. Add a deployment named e2e-responses-azure-openai-tool-2e4bd0c9f130 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  217. ✓test_responses_azure_openai_streaming_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • stream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-responses-azure-stream-2249957ad1f3 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  218. ✓test_responses_bedrock_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-responses-cf93a5688744 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  219. ✓test_responses_bedrock_returns_function_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • function_calling
    1. Add a deployment named e2e-responses-93ac03a684eb that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  220. ✓test_responses_compact_returns_compacted_conversation TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-responses-compact-698529f1b7df that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  221. ✓test_responses_context_management_compacts_server_side TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-responses-context-compaction-a37d7181f7d9 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  222. ✓test_responses_gemini_replays_legacy_function_call_output TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    • function_calling
    1. Add a deployment named e2e-responses-gemini-tool-45e25ae6925b that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  223. ✓test_responses_gemini_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Add a deployment named e2e-responses-gemini-0ab149835c13 that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  224. ✓test_responses_gemini_streaming_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • stream
    • gemini
    • gemini/gemini-2.5-flash
    1. Add a deployment named e2e-responses-gemini-f943842be5b1 that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  225. ✓test_responses_logs_cost TestResponsestest_responses_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-responses-f55f95a4f8ed that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  226. ✓test_responses_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-responses-45d4d5b7d8db that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  227. ✓test_responses_returns_function_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Add a deployment named e2e-responses-5db656188635 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  228. ✓test_responses_streaming_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-responses-ff41ebe303e5 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  229. ✓test_responses_upgrade_is_accepted_on_the_openai_prefix TestOpenAIPassthroughWebsockettest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • websocket
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a websocket to /openai/v1/responses and wait for its first event
    View test on GitHub ↗
  230. ✓test_responses_vertex_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    1. Add a deployment named e2e-responses-vertex-44e6e5dae023 that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  231. ✓test_responses_vertex_returns_function_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    • function_calling
    1. Add a deployment named e2e-responses-vertex-tool-95e063aca28c that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  232. ✓test_responses_vision_describes_image TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o
    • vision
    1. Add a deployment named e2e-responses-b8c211cecba9 that calls openai/gpt-4o
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  233. ✓test_retrieve_invalid_batch_id_returns_error TestFilesBatchesContracttest_files_batches_contract_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. GET /v1/batches/invalid-batch-id
    View test on GitHub ↗
  234. ✓test_rust_ocr_response[azure-ai] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • azure_ai
    • azure_ai/mistral-document-ai-2512
    1. Add a deployment named rust-ocr-azure-ai-8a7a7885239f that calls azure_ai/mistral-document-ai-2512
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-azure-ai-8a7a7885239f
    View test on GitHub ↗
  235. ✓test_rust_ocr_response[azure-document-intelligence] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • azure_ai
    • azure_ai/doc-intelligence/prebuilt-layout
    1. Add a deployment named rust-ocr-azure-document-intelligence-918ac23f8b3a that calls azure_ai/doc-intelligence/prebuilt-layout
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-azure-document-intelligence-918ac23f8b3a
    View test on GitHub ↗
  236. ✓test_rust_ocr_response[mistral] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • mistral
    • mistral/mistral-ocr-latest
    1. Add a deployment named rust-ocr-mistral-fca94e488196 that calls mistral/mistral-ocr-latest
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-mistral-fca94e488196
    View test on GitHub ↗
  237. ✓test_rust_ocr_response[vertex-mistral] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • vertex_ai
    • vertex_ai/mistral-ocr-2505
    1. Add a deployment named rust-ocr-vertex-mistral-feb6e9abccd2 that calls vertex_ai/mistral-ocr-2505
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-vertex-mistral-feb6e9abccd2
    View test on GitHub ↗
  238. ✓test_service_account_key_reads_container_file_by_native_id TestAzureContainerFilestest_containers_e2e.py›
    • llm-translation
    • containers
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-containers-decoy-4f66d0c3f664 that calls azure/gpt-5.4-nano
    2. Add a deployment named e2e-containers-4f66d0c3f664 that calls azure/gpt-5.4-nano
    3. Create a team with team alias: e2e-containers-4f66d0c3f664 and models: e2e-containers-4f66d0c3f664
    4. Create an internal user with user email: e2e-containers-4f66d0c3f664@example.com and user role: internal_user
    5. Add a user to the team with /team/member_add
    6. POST /key/service-account/generate
    View test on GitHub ↗
  239. ✓test_service_account_key_reads_container_file_created_by_a_streamed_response TestAzureContainerFilestest_containers_e2e.py›
    • llm-translation
    • containers
    • stream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-containers-decoy-c28cdb3f18f9 that calls azure/gpt-5.4-nano
    2. Add a deployment named e2e-containers-c28cdb3f18f9 that calls azure/gpt-5.4-nano
    3. Create a team with team alias: e2e-containers-c28cdb3f18f9 and models: e2e-containers-c28cdb3f18f9
    4. Create an internal user with user email: e2e-containers-c28cdb3f18f9@example.com and user role: internal_user
    5. Add a user to the team with /team/member_add
    6. POST /key/service-account/generate
    View test on GitHub ↗
  240. ✓test_service_tier_bills_the_matching_completion_window[auto-base] TestSailChatCompletionstest_sail_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-53282b3ce57a that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  241. ✓test_service_tier_bills_the_matching_completion_window[balanced-balanced] TestSailChatCompletionstest_sail_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-872563185cda that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  242. ✓test_shell_tool_stream_surfaces_shell_call_and_its_output TestResponsesOpenAIHostedFeaturestest_responses_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-responses-shell-00b98aa366ef that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  243. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 with the prompt "Say hello."
    View test on GitHub ↗
  244. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 with the prompt "Say hello."
    View test on GitHub ↗
  245. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 with the prompt "Say hello."
    View test on GitHub ↗
  246. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d with the prompt "Say hello."
    View test on GitHub ↗
  247. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 with the prompt "Say hello."
    View test on GitHub ↗
  248. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 with the prompt "Say hello."
    View test on GitHub ↗
  249. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-ollama-ollama_chat-68a32c9c8050 with the prompt "Say hello."
    View test on GitHub ↗
  250. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-ollama-ollama-9d6596ad7e4b with the prompt "Say hello."
    View test on GitHub ↗
  251. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 with the prompt "Say hello."
    View test on GitHub ↗
  252. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 with the prompt "Say hello."
    View test on GitHub ↗
  253. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 with the prompt "Say hello."
    View test on GitHub ↗
  254. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d with the prompt "Say hello."
    View test on GitHub ↗
  255. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 with the prompt "Say hello."
    View test on GitHub ↗
  256. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 with the prompt "Say hello."
    View test on GitHub ↗
  257. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-ollama-ollama_chat-68a32c9c8050 with the prompt "Say hello."
    View test on GitHub ↗
  258. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-ollama-ollama-9d6596ad7e4b with the prompt "Say hello."
    View test on GitHub ↗
  259. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 with the prompt "Say hello."
    View test on GitHub ↗
  260. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 with the prompt "Say hello."
    View test on GitHub ↗
  261. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 with the prompt "Say hello."
    View test on GitHub ↗
  262. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d with the prompt "Say hello."
    View test on GitHub ↗
  263. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 with the prompt "Say hello."
    View test on GitHub ↗
  264. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 with the prompt "Say hello."
    View test on GitHub ↗
  265. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-ollama-ollama_chat-68a32c9c8050 with the prompt "Say hello."
    View test on GitHub ↗
  266. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-ollama-ollama-9d6596ad7e4b with the prompt "Say hello."
    View test on GitHub ↗
  267. ✓test_stream_generate_content_frames_sse_the_way_google_sdks_expect TestGoogleNativeGenerateContenttest_google_native_e2e.py›
    • llm-translation
    • google_genai
    • stream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-google-native-0da89f69b98c that calls gemini/gemini-2.5-flash
    3. Send a request to /v1beta/models/e2e-google-native-0da89f69b98c:streamGenerateContent
    View test on GitHub ↗
  268. ✓test_streamed_response_can_be_deleted[azure] TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • stream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-resp-delete-stream-b0fc4584d76d that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  269. ✓test_streamed_response_can_be_deleted[openai] TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-resp-delete-stream-8d24f3dce6b7 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  270. ✓test_streamed_responses_call_logs_its_cost TestOpenAIPassthroughSpendtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • stream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to gpt-5.5 through /openai_passthrough with streaming set to True
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  271. ✓test_streams_text_deltas TestTogetherMessagestest_together_ai_e2e.py›
    • llm-translation
    • messages
    • stream
    • together_ai
    1. Add a deployment named e2e-together-b3653a40c464 that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /v1/messages request to e2e-together-b3653a40c464
    View test on GitHub ↗
  272. ✓test_success_response_matches_chat_completion_contract TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-c52fc1c7f1fc that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  273. ✓test_temperature_boundaries_succeed TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-db3cdd221db3 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  274. ✓test_text_completion_returns_text TestCompletionsEndpointtest_completions_endpoint_e2e.py›
    • llm-translation
    • completions
    • nonstream
    • openai
    • openai/gpt-5.4-nano
    1. Add a deployment named e2e-completions-7779d087a5b6 that calls openai/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  275. ✓test_text_conversation[azure]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • azure
    • azure/gpt-realtime
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to azure-realtime-261a10691a8e
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  276. ✓test_text_conversation[gemini]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • gemini
    • gemini/gemini-3.1-flash-live-preview
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to gemini-realtime-ef7a585f92c0
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  277. ✓test_text_conversation[openai]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • openai
    • openai/gpt-realtime-2
    1. Add a realtime deployment that calls openai/gpt-realtime-2
    2. Add a realtime deployment that calls azure/gpt-realtime
    3. Add a realtime deployment that calls gemini/gemini-3.1-flash-live-preview
    4. Add a realtime deployment that calls vertex_ai/gemini-live-2.5-flash-native-audio
    5. Generate a virtual key with user id: e2e-test-user
    6. Open a /v1/realtime websocket session to openai-realtime-b165dc74f922
    7. Wait for a session.created event on the realtime websocket
    8. Send the realtime event session.update over the websocket
    9. Wait for a session.updated event on the realtime websocket
    10. Send the realtime event conversation.item.create over the websocket
    11. Send the realtime event response.create over the websocket
    12. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  278. ✓test_text_conversation[vertex_ai]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • vertex_ai
    • vertex_ai/gemini-live-2.5-flash-native-audio
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to vertex-realtime-fdba0438cdcf
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  279. ✓test_thinking_disabled_disables_reasoning TestDeepSeekReasoningDisabletest_deepseek_reasoning_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • deepseek
    • deepseek/deepseek-v4-pro
    • reasoning
    1. Add a deployment named e2e-deepseek-reasoner-b3a9e765dd63 that calls deepseek/deepseek-v4-pro
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-deepseek-reasoner-b3a9e765dd63
    View test on GitHub ↗
  280. ✓test_tool_call_is_returned TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • function_calling
    1. Add a deployment named e2e-together-6f73f263fce6 that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-6f73f263fce6
    View test on GitHub ↗
  281. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set
    View test on GitHub ↗
  282. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather to
    View test on GitHub ↗
  283. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to T
    View test on GitHub ↗
  284. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool us
    View test on GitHub ↗
  285. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to
    View test on GitHub ↗
  286. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool u
    View test on GitHub ↗
  287. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ollama-ollama_chat-68a32c9c8050 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  288. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ollama-ollama-9d6596ad7e4b with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  289. ✓test_tool_call_is_returned_named_and_addressable[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to T
    View test on GitHub ↗
  290. ✓test_tool_call_is_returned_named_and_addressable[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool us
    View test on GitHub ↗
  291. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  292. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set
    View test on GitHub ↗
  293. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  294. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use se
    View test on GitHub ↗
  295. ✓test_tool_call_is_returned_named_and_addressable[messages-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama_chat-68a32c9c8050 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  296. ✓test_tool_call_is_returned_named_and_addressable[messages-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama-9d6596ad7e4b with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  297. ✓test_tool_call_is_returned_named_and_addressable[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to
    View test on GitHub ↗
  298. ✓test_tool_call_is_returned_named_and_addressable[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool u
    View test on GitHub ↗
  299. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  300. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use se
    View test on GitHub ↗
  301. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  302. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use s
    View test on GitHub ↗
  303. ✓test_tool_call_is_returned_named_and_addressable[responses-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama_chat-68a32c9c8050 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  304. ✓test_tool_call_is_returned_named_and_addressable[responses-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama-9d6596ad7e4b with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  305. ✓test_tool_call_is_streamed TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • together_ai
    • function_calling
    1. Add a deployment named e2e-together-ab46c357d03d that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-together-ab46c357d03d
    View test on GitHub ↗
  306. ✓test_tool_call_round_trip[azure]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • azure
    • azure/gpt-realtime
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to azure-realtime-261a10691a8e
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    9. Send the realtime event conversation.item.create over the websocket
    10. Send the realtime event response.create over the websocket
    11. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  307. ✓test_tool_call_round_trip[gemini]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • gemini
    • gemini/gemini-3.1-flash-live-preview
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to gemini-realtime-ef7a585f92c0
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    9. Send the realtime event conversation.item.create over the websocket
    10. Send the realtime event response.create over the websocket
    11. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  308. ✓test_tool_call_round_trip[openai]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • openai
    • openai/gpt-realtime-2
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to openai-realtime-b165dc74f922
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    9. Send the realtime event conversation.item.create over the websocket
    10. Send the realtime event response.create over the websocket
    11. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  309. ✓test_tool_call_round_trip[vertex_ai]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • vertex_ai
    • vertex_ai/gemini-live-2.5-flash-native-audio
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to vertex-realtime-fdba0438cdcf
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    9. Send the realtime event conversation.item.create over the websocket
    10. Send the realtime event response.create over the websocket
    11. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  310. ✓test_tool_call_streams_as_tool_call_deltas[ollama_chat] TestOllamaStreamedToolCalltest_ollama_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  311. ✓test_tool_call_streams_as_tool_call_deltas[ollama] TestOllamaStreamedToolCalltest_ollama_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  312. ✓test_tool_result_round_trip TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • function_calling
    1. Add a deployment named e2e-together-4204cb25cbd6 that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-4204cb25cbd6
    View test on GitHub ↗
  313. ✓test_tool_result_round_trip TestTogetherMessagestest_together_ai_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • together_ai
    • function_calling
    1. Add a deployment named e2e-together-0486c487d932 that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/messages request to e2e-together-0486c487d932
    View test on GitHub ↗
  314. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 over /chat/completions
    View test on GitHub ↗
  315. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather to
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 over /chat/completions
    View test on GitHub ↗
  316. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to T
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 over /chat/completions
    View test on GitHub ↗
  317. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool us
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d over /chat/completions
    View test on GitHub ↗
  318. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 over /chat/completions
    View test on GitHub ↗
  319. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool u
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 over /chat/completions
    View test on GitHub ↗
  320. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ollama-ollama_chat-68a32c9c8050 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama_chat-68a32c9c8050 over /chat/completions
    View test on GitHub ↗
  321. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ollama-ollama-9d6596ad7e4b with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama-9d6596ad7e4b over /chat/completions
    View test on GitHub ↗
  322. ✓test_tool_result_round_trip_reaches_the_model[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to T
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 over /v1/messages
    View test on GitHub ↗
  323. ✓test_tool_result_round_trip_reaches_the_model[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool us
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 over /v1/messages
    View test on GitHub ↗
  324. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 over /v1/messages
    View test on GitHub ↗
  325. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d over /v1/messages
    View test on GitHub ↗
  326. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 over /v1/messages
    View test on GitHub ↗
  327. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use se
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 over /v1/messages
    View test on GitHub ↗
  328. ✓test_tool_result_round_trip_reaches_the_model[messages-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama_chat-68a32c9c8050 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama_chat-68a32c9c8050 over /v1/messages
    View test on GitHub ↗
  329. ✓test_tool_result_round_trip_reaches_the_model[messages-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama-9d6596ad7e4b with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama-9d6596ad7e4b over /v1/messages
    View test on GitHub ↗
  330. ✓test_tool_result_round_trip_reaches_the_model[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-env_ref-b5e6a1f4be46 over /v1/responses
    View test on GitHub ↗
  331. ✓test_tool_result_round_trip_reaches_the_model[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool u
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-stored_credential-6e5e03796d71 over /v1/responses
    View test on GitHub ↗
  332. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-env_ref-925b291094e0 over /v1/responses
    View test on GitHub ↗
  333. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use se
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-stored_credential-bf2cbf121c6d over /v1/responses
    View test on GitHub ↗
  334. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-env_ref-769d104ab902 over /v1/responses
    View test on GitHub ↗
  335. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use s
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-stored_credential-ded258805787 over /v1/responses
    View test on GitHub ↗
  336. ✓test_tool_result_round_trip_reaches_the_model[responses-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama_chat-68a32c9c8050 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama_chat-68a32c9c8050 over /v1/responses
    View test on GitHub ↗
  337. ✓test_tool_result_round_trip_reaches_the_model[responses-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama-9d6596ad7e4b with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama-9d6596ad7e4b over /v1/responses
    View test on GitHub ↗
  338. ✓test_tool_use_block_is_returned TestTogetherMessagestest_together_ai_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • together_ai
    • function_calling
    1. Add a deployment named e2e-together-9754e7ada40a that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/messages request to e2e-together-9754e7ada40a
    View test on GitHub ↗
  339. ✓test_tool_use_nonstream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-azure-foundry-messages-45e3ad4b96cf that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-45e3ad4b96cf and user id: e2e-test-user
    View test on GitHub ↗
  340. ✓test_tool_use_stream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    • llm-translation
    • messages
    • stream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-azure-foundry-messages-978e3c7eee50 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-978e3c7eee50 and user id: e2e-test-user
    View test on GitHub ↗
  341. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestAnthropicChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-chat-midsys-cc139fbb3112 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-chat-midsys-cc139fbb3112 and user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  342. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestBedrockInvokeChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-chat-midsys-140c3517ef02 that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with models: e2e-chat-midsys-140c3517ef02 and user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  343. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestBedrockInvokeMidConversationSystemtest_messages_mid_conversation_system_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-midsys-54d6308eaa3b that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with models: e2e-midsys-54d6308eaa3b and user id: e2e-test-user
    View test on GitHub ↗
  344. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestAzureFoundryMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • azure_ai/claude-opus-4-7
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-midsys-0fb0515b744d that calls azure_ai/claude-opus-4-7
    2. Generate a virtual key with models: e2e-midsys-0fb0515b744d and user id: e2e-test-user
    View test on GitHub ↗
  345. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestVertexMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • vertex_ai/claude-sonnet-4-6
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-midsys-4f8d96927695 that calls vertex_ai/claude-sonnet-4-6
    2. Generate a virtual key with models: e2e-midsys-4f8d96927695 and user id: e2e-test-user
    View test on GitHub ↗
  346. ✓test_unknown_service_tier_is_dropped_and_billed_asap[5] TestSailChatCompletionstest_sail_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-9203fa3e7d06 that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  347. ✓test_unknown_service_tier_is_dropped_and_billed_asap[bogus] TestSailChatCompletionstest_sail_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-056ab9b24ea3 that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  348. ✓test_upload_without_purpose_returns_error TestFilesBatchesContracttest_files_batches_contract_e2e.py›
    • llm-translation
    • files
    • batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload batch_input.jsonl to /v1/files
    View test on GitHub ↗
  349. ✓test_upstream_handshake_refusal_is_an_error_event_and_policy_close[azure-bad-key]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • azure
    • azure/gpt-realtime
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a realtime deployment that calls azure/gpt-realtime
    3. Open a /v1/realtime websocket session to azure-realtime-refused-5f186d93fbe0
    View test on GitHub ↗
  350. ✓test_verbose_json_returns_word_timestamps TestWhisperTranscriptionFormatstest_audio_transcriptions_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/whisper-1
    1. Add a deployment named e2e-whisper-verbose-fb0b6a8b596a that calls openai/whisper-1
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload weather_question_24k.wav to /v1/audio/transcriptions
    View test on GitHub ↗
  351. ✓test_vertex_chat_returns_content TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    1. Add a deployment named e2e-vertex-chat-18a699acc73e that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-vertex-chat-18a699acc73e
    View test on GitHub ↗
  352. ✓test_vertex_chat_returns_tool_call TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    • function_calling
    1. Add a deployment named e2e-vertex-tool-30bc0a678e8f that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-vertex-tool-30bc0a678e8f
    View test on GitHub ↗
  353. ✓test_vertex_chat_streams_real_content TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    1. Add a deployment named e2e-vertex-stream-cdfef44be888 that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-vertex-stream-cdfef44be888
    View test on GitHub ↗
  354. ✓test_vertex_chat_vision_describes_image TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    • vision
    1. Add a deployment named e2e-vertex-vision-fb2d0a295c4a that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-vertex-vision-fb2d0a295c4a
    View test on GitHub ↗
  355. ✓test_vertex_embeddings_honor_requested_dimensions TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • vertex_ai
    • vertex_ai/text-embedding-005
    1. Add a deployment named e2e-embeddings-vertex-dims-cea7ef09a25d that calls vertex_ai/text-embedding-005
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  356. ✓test_vertex_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • vertex_ai
    • vertex_ai/text-embedding-005
    1. Add a deployment named e2e-embeddings-vertex-e58259d21ba8 that calls vertex_ai/text-embedding-005
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  357. ✓test_vertex_multimodal_embeddings_honor_dimensions_and_are_costed TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • vertex_ai
    • vertex_ai/multimodalembedding@001
    1. Add a deployment named e2e-embeddings-vertex-mm-564a28d9ae01 that calls vertex_ai/multimodalembedding@001
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  358. ✓test_vertex_passthrough_via_managed_model_logs_cost TestVertexPassthroughSpendTrackingtest_vertex_passthrough_e2e.py›
    • spend-budgets
    • passthrough
    • nonstream
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. POST /model/new
    3. Wait for the last control-plane write to reach every proxy replica
    4. Send a Vertex AI generateContent request to gemini-2.5-flash in us-central1 through /vertex_ai
    5. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  359. ✓test_vertex_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    • prompt_caching
    1. Add a deployment named e2e-vertex-cache-0d4252c0a3c2 that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /chat/completions
    4. Wait for the request's spend log in /spend/logs
    5. List the deployments from /model/info
    View test on GitHub ↗
  360. ✓test_vtt_format_returns_webvtt_transcript TestWhisperTranscriptionFormatstest_audio_transcriptions_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/whisper-1
    1. Add a deployment named e2e-whisper-vtt-34fbee4c5878 that calls openai/whisper-1
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload weather_question_24k.wav to /v1/audio/transcriptions
    View test on GitHub ↗
  361. −test_empty_input_returns_error TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-tts

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on empty input instead of surfacing the provider 4xx

    View test on GitHub ↗
  362. −test_flagged_model_keeps_prompt_cache_across_system_reminder TestBedrockInvokeMidConversationSystemtest_messages_mid_conversation_system_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-sonnet-5
    • mid_conversation_system
    • prompt_caching

    No steps recorded

    LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

    View test on GitHub ↗
  363. −test_flagged_model_keeps_prompt_cache_across_system_reminder TestAzureFoundryMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • azure_ai/claude-opus-4-8
    • mid_conversation_system
    • prompt_caching

    No steps recorded

    LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

    View test on GitHub ↗
  364. −test_flagged_model_keeps_prompt_cache_across_system_reminder TestVertexMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • vertex_ai/claude-opus-4-8
    • mid_conversation_system
    • prompt_caching

    No steps recorded

    LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

    View test on GitHub ↗
  365. −test_gemini_passthrough_returns_the_same_header_contract_as_the_managed_routetest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • gemini
    • gemini-2.5-flash

    No steps recorded

    stage red: product gap, native passthrough returns no x-litellm-response-cost or x-ratelimit-* headers

    View test on GitHub ↗
  366. −test_hosted_vllm_chat_returns_content TestHostedVllmChattest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • hosted_vllm

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  367. −test_invalid_response_id_returns_error TestResponsesRetrievetest_responses_retrieve_e2e.py›
    • llm-translation
    • responses

    No steps recorded

    stage red: product gap (LIT-5447), retrieving an unknown response id returns 400 (model=None) instead of 404

    View test on GitHub ↗
  368. −test_invalid_voice_returns_error TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-tts

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on invalid voice instead of surfacing the provider 4xx

    View test on GitHub ↗
  369. −test_missing_document_returns_error TestRustOcrGatewaytest_ocr_rust_e2e.py›
    • llm-translation
    • ocr

    No steps recorded

    stage red: product gap, /v1/ocr 500s (aocr TypeError) on missing document instead of 400

    View test on GitHub ↗
  370. −test_missing_input_returns_error TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-tts

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on missing input instead of 400

    View test on GitHub ↗
  371. −test_missing_input_returns_error TestModerationstest_moderations_e2e.py›
    • llm-translation
    • moderations

    No steps recorded

    stage red: product gap, /v1/moderations 500s (KeyError 'input') on missing input instead of 400

    View test on GitHub ↗
  372. −test_missing_input_returns_error TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • openai
    • openai/gpt-4o-mini

    No steps recorded

    stage red: product gap, /v1/responses 500s (aresponses TypeError) on missing input instead of 400

    View test on GitHub ↗
  373. −test_missing_max_tokens_returns_error TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages

    No steps recorded

    stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing max_tokens instead of 400

    View test on GitHub ↗
  374. −test_missing_messages_returns_error TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages

    No steps recorded

    stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing messages instead of 400

    View test on GitHub ↗
  375. −test_missing_model_returns_error TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on missing model instead of 400

    View test on GitHub ↗
  376. −test_missing_prompt_returns_error TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images

    No steps recorded

    stage red: product gap, /v1/images/generations 500s (aimage_generation TypeError) on missing prompt instead of 400

    View test on GitHub ↗
  377. −test_retrieve_invalid_id_returns_error TestVectorStorestest_vector_stores_e2e.py›
    • llm-translation
    • vector_stores
    • openai

    No steps recorded

    stage red: product gap, retrieving a nonexistent vector store returns 2xx with an error envelope in the body instead of 404

    View test on GitHub ↗
  378. −test_search_missing_query_returns_error TestVectorStorestest_vector_stores_e2e.py›
    • llm-translation
    • vector_stores
    • openai

    No steps recorded

    stage red: product gap, vector store search 500s (asearch TypeError) on missing query instead of 400

    View test on GitHub ↗
  379. −test_static_and_x_pass_headers_reach_upstream TestPassthroughHeaderstest_passthrough_headers_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • anthropic
    • claude-haiku-4-5-20251001

    No steps recorded

    stage red: ALB sends custom passthrough paths to backend pods, which lack ANTHROPIC_API_KEY, so the os.environ header forwards unresolved and Anthropic 401s; routing user-defined paths to the gateway needs a product/infra decision (fixed path namespace or dynamic ingress)

    View test on GitHub ↗
  380. −test_store_and_retrieve_by_id TestResponsesRetrievetest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini

    No steps recorded

    stage red: product gap (LIT-5446), retrieve returns a different id than the stored response (non-idempotent response-id re-encryption)

    View test on GitHub ↗
  381. −test_vertex_partner_model_streams_openai_shaped_chunks[vertex_ai/mistral-small-2503] TestVertexPartnerChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • vertex_ai
    • vertex_ai/mistral-small-2503

    No steps recorded

    the e2e Vertex project has no access to mistral-small-2503 (404 publisher model not found)

    View test on GitHub ↗
  382. −test_vertex_partner_model_streams_openai_shaped_chunks[vertex_ai/openai/gpt-oss-120b-maas] TestVertexPartnerChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • vertex_ai
    • vertex_ai/openai/gpt-oss-120b-maas

    No steps recorded

    never served by the e2e Vertex project (60s read timeout, no headers)

    View test on GitHub ↗
  383. −test_web_search_server_tool_is_served TestBedrockWebSearchServerTooltest_bedrock_web_search_server_tool_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • web_search

    No steps recorded

    stage red: environment gap, the e2e stack neither enables the websearch_interception callback nor declares a search backend, so the request reaches bedrock's transformation and takes its by-design 400. Unskip once the ephemeral stack ships the config in this module's docstring.

    View test on GitHub ↗
  384. ✓test_a_drained_group_does_not_block_a_different_group TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-mag-free-0a1b3d48afce and user id: e2e-test-user
    2. Send a /chat/completions request to e2e-mag-unbudgeted-0a1b3d48afce with the prompt "hi 1e6f8f91f1a1"
    View test on GitHub ↗
  385. ✓test_a_key_that_spent_nothing_is_blocked_by_the_shared_pool TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-mag-budget-0a1b3d48afce and user id: e2e-test-user
    2. Send a /chat/completions request to e2e-mag-budgeted-0a1b3d48afce with the prompt "hi b5074fc53a63"
    View test on GitHub ↗
  386. ✓test_alias_shares_rpm_bucket_with_model_group TestModelGroupAliasRateLimitratelimit/test_model_group_alias_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • e2e-alias-rl-alias
    • e2e-alias-rl-target
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word ba585835cc3c"
    3. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word 24775318381e"
    4. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word caa8173011d1"
    5. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word 5fef9f64f5a6"
    View test on GitHub ↗
  387. ✓test_bare_key_blocks_over_its_own_budget TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 2d10b14df458"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 692f52af7255"
    View test on GitHub ↗
  388. ✓test_bare_key_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 94ee4de783f8"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 19712e2356c9"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 6b4c5445de68"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 80f484d72c30"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset dcf6f4c75501"
    View test on GitHub ↗
  389. ✓test_blocked_member_does_not_block_peer TestTeamMemberBudgetIsolationbudgets/test_team_member_budget_isolation_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Create the team e2e-member-iso-2249ad9cc80a and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Generate a virtual key with user id: 55e95b1e-22b3-4bb2-ae55-89134c9e99bb and team id: 9e9d7258-e232-4362-88bf-531a8c9ad06f
    5. Generate a virtual key with user id: 7bff5574-a979-4c79-8782-cbe166ffb872 and team id: 9e9d7258-e232-4362-88bf-531a8c9ad06f
    6. Send a /chat/completions request to gpt-5.5 with the prompt "tight c57b24392daf"
    7. Send a /chat/completions request to gpt-5.5 with the prompt "tight 6cdecd278206"
    8. Send a /chat/completions request to gpt-5.5
    9. Send a /chat/completions request to gpt-5.5 with the prompt "tight b1e479d8a9dc"
    10. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  390. ✓test_budget_crud_roundtripbudgets/test_budget_crud_e2e.py›
    • spend-budgets
    • budget_management
    1. Create a budget with /budget/new
    2. Read the budget from /budget/info
    3. Generate a virtual key with budget id: 4763780f-97a0-46dd-9f79-728786af1cfa
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  391. ✓test_budget_delete_removes_itbudgets/test_budget_crud_e2e.py›
    • spend-budgets
    • budget_management
    1. Create a budget with /budget/new
    2. Delete the budget
    3. Read the budget from /budget/info
    View test on GitHub ↗
  392. ✓test_budget_duration_schedules_reset_on_keybudgets/test_budget_crud_e2e.py›
    • spend-budgets
    • key_management
    1. Generate a virtual key with max budget: 10.0 and budget duration: 30d
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  393. ✓test_budget_fallback_reroutes_anthropic_messages_to_openaibudgets/test_budget_fallback_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • anthropic
    • openai
    • claude-haiku-4-5
    • gpt-5.5
    1. Generate a virtual key with model max budget: claude-haiku-4-5: budget limit: 1e-06 and time period: 30d and budget fallbacks: claude-haiku-4-5: gpt-5.5
    2. Send a /v1/messages request to claude-haiku-4-5 with the prompt "hi 4a1c4a40b534"
    3. Send a /v1/messages request to claude-haiku-4-5 with the prompt "hi bf2fe8debef1"
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  394. ✓test_burst_of_concurrent_calls_loses_no_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.6-luna
    1. Add a priced deployment, then create two teams with one key each and send 7 /chat/completions requests per key, 6 of them at once
    2. Send a /chat/completions request to e2e-reconciliation-39d5177e2794
    3. Check that the /spend/logs rows of team b8fc6e0f-4f85-45fa-81cb-3ceb9eaca9f7 match each response's tokens and cost
    4. Wait for the key's spend in /key/info to reach the expected minimum
    5. Check that the /spend/logs rows of team b0d36151-e0f2-4d85-a31d-96c0c423d65d match each response's tokens and cost
    6. Wait for the key's spend in /key/info to reach the expected minimum
    View test on GitHub ↗
  395. ✓test_burst_rate_limit_does_not_freeze_fresh_key TestRedisCircuitBreakerPathratelimit/test_redis_circuit_breaker_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-cb-model-032162450b4c that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-cb-model-032162450b4c, key alias: e2e-cb-hot-8e94f41b83c0 and rpm limit: 1
    3. Generate a virtual key with models: e2e-cb-model-032162450b4c and key alias: e2e-cb-cool-c7e912d795a2
    4. Send a /chat/completions request to e2e-cb-model-032162450b4c with the prompt "burst 77b5daa41d88"
    5. Send a /chat/completions request to e2e-cb-model-032162450b4c with the prompt "burst d9e54de2141b"
    6. Send a /chat/completions request to e2e-cb-model-032162450b4c with the prompt "burst 88bc39653c04"
    7. Send a /chat/completions request to e2e-cb-model-032162450b4c with the prompt "burst 3aba8af5fee0"
    8. Send a /chat/completions request to e2e-cb-model-032162450b4c with the prompt "burst daaa3fbcbeb6"
    9. Send a /chat/completions request to e2e-cb-model-032162450b4c with the prompt "burst 99bcb9c5a17e"
    10. Send a /chat/completions request to e2e-cb-model-032162450b4c with the prompt "burst 9dc76b1eb158"
    11. Send a /chat/completions request to e2e-cb-model-032162450b4c with the prompt "burst 271cea29c814"
    12. Send a /chat/completions request to e2e-cb-model-032162450b4c with the prompt "burst eb950a72592e"
    13. Send a /chat/completions request to e2e-cb-model-032162450b4c with the prompt "burst aa7c54b3b4a4"
    14. Send a /chat/completions request to e2e-cb-model-032162450b4c with the prompt "burst e88a04f73a95"
    15. Send a /chat/completions request to e2e-cb-model-032162450b4c with the prompt "burst 7aa30af1cc10"
    16. Send a /chat/completions request to e2e-cb-model-032162450b4c with the prompt "fresh fcb264a79f7a"
    View test on GitHub ↗
  396. ✓test_cache_hit_is_zero_cost_and_suffixedspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "What is the capital of France? Answer in one word. 92757c17319b"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  397. ✓test_cache_hit_reduces_tpm_by_non_cached_only TestTpmExcludesCachedTokensratelimit/test_tpm_excludes_cached_tokens_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • prompt_caching
    1. Add a deployment named e2e-tpm-cache-8fb96751a521 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-tpm-cache-8fb96751a521 and tpm limit: 100000
    3. POST /chat/completions
    4. Send a request to /chat/completions
    View test on GitHub ↗
  398. ✓test_cache_write_tokens_billed_at_cache_creation_rate TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.6-luna
    • prompt_caching
    1. List the deployments from /model/info
    2. Generate a virtual key with user id: e2e-test-user
    3. Add a deployment with custom rates that calls openai/gpt-5.6-luna
    4. Send a /chat/completions request to cache-write-priced-fdf0597f6e23
    5. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  399. ✓test_capture_rate_reports_or_names_the_missing_billing_keyspend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/capture_rate
    View test on GitHub ↗
  400. ✓test_chat_completion_writes_nonzero_spend_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. List the deployments from /model/info
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to gemini-2.5-flash with the prompt "reply with one word c5de039db019"
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  401. ✓test_cold_counter_reseed_keeps_counter_equal_to_db_spendbudgets/test_spend_counter_reseed_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and max budget: 1.0
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed e8b25114fe33"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 7ce4ec543362"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed e0140f0a2e76"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 04573d0cf7fa"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 043ce7814511"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 9d6f32bc3c44"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 0b81e37daffb"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed cf3571b5623c"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed d3d3b2e82d40"
    11. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed c48873e840e0"
    12. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 11d99e192419"
    13. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed ddec9ec811f2"
    14. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed c0b364f0302c"
    15. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed f0adcd81ec3a"
    16. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 0b9c51b62e10"
    17. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 2cc30a3ca7c7"
    18. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 43281ded46db"
    19. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed d1722deb23e7"
    20. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed d34f95b83852"
    21. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 975a8d809de3"
    22. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 4a861dc930af"
    23. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed c48a164b894e"
    24. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed f0e17862ec0d"
    25. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed de40953c6909"
    26. Read the key's settings back from /key/info
    27. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed cc11e1b6ab77"
    28. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed d658c216636c"
    29. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed db9a080b4608"
    30. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 1d36f9444a3d"
    31. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed b65f617e8451"
    32. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 6d437467cd6b"
    View test on GitHub ↗
  402. ✓test_component_cost_headers_sum_to_total TestCostHeadersspend_tracking/test_cost_headers_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a request to /chat/completions
    View test on GitHub ↗
  403. ✓test_cost_breakdown_reports_component_costs TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.5
    • prompt_caching
    • reasoning
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a /chat/completions request to breakdown-priced-d8d7fb70f7e3
    4. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  404. ✓test_each_model_on_a_shared_key_gets_its_own_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • gemini
    • claude-haiku-4-5
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "one word 5ce1e5d2ada1"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "one word e6ff6d61fcc0"
    4. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  405. ✓test_edge_wired_chat_writes_nonzero_spend_rowspend_tracking/test_provider_edge_spend_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-edge-openai-bcf7b3377d86 that calls openai/gpt-5.5
    3. Send a /chat/completions request to e2e-edge-openai-bcf7b3377d86 with the prompt "reply with one word daf3487d26f9"
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  406. ✓test_embedding_writes_nonzero_spend_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • embeddings
    • nonstream
    • openai
    • openai-text-embedding-3-small
    1. Generate a virtual key with user id: e2e-test-user
    2. Send an /embeddings request to openai-text-embedding-3-small for "vectorize this sentence d0a6e5bc97e4"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  407. ✓test_end_user_budget_blocks_attributed_calls TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the end user e2e-budget-cust-5ca689954634
    2. Generate a virtual key with models: claude-haiku-4-5
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 353140b1d626"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 3d472f4f50d8"
    View test on GitHub ↗
  408. ✓test_end_user_header_attributes_responses_row[x-litellm-customer-id]spend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai-responses-codex
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to openai-responses-codex with custom headers and the prompt "one word 1aeda4b8e45d"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    4. Wait for the end user's spend in /customer/info to go above 0.0
    View test on GitHub ↗
  409. ✓test_end_user_header_attributes_responses_row[x-litellm-end-user-id]spend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai-responses-codex
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to openai-responses-codex with custom headers and the prompt "one word 1465fedeac61"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    4. Wait for the end user's spend in /customer/info to go above 0.0
    View test on GitHub ↗
  410. ✓test_end_user_spend_attributed_on_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "hi"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  411. ✓test_every_streamed_chunk_carries_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a streaming /chat/completions request to tier-echo-stream-03451d6a1b5d
    View test on GitHub ↗
  412. ✓test_every_write_path_row_joins_the_key TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • spend_reporting
    • anthropic
    • gemini
    • openai
    • claude-haiku-4-5
    • gemini-2.5-flash
    • openai-gpt-4o-mini
    • openai-responses-codex
    • openai-text-embedding-3-small
    1. List the deployments from /model/info
    2. Create an internal user with the role proxy_admin
    3. Generate a virtual key with user id: key-attribution-7c2e9f6de952 and key alias: key-attribution-7c2e9f6de952
    4. Send a /chat/completions request to gemini-2.5-flash with the prompt "Reply with the word ok. 0848ab9253aa"
    5. Send a /queue/chat/completions request to gemini-2.5-flash with the prompt "Reply with the word ok. 0848ab9253aa"
    6. Send a /v1/messages request to claude-haiku-4-5 with the prompt "Reply with the word ok. 0848ab9253aa"
    7. Send a /v1/responses request to openai-responses-codex with the prompt "Reply with the word ok. 0848ab9253aa"
    8. Send an /embeddings request to openai-text-embedding-3-small for "Reply with the word ok. 0848ab9253aa"
    9. Send a Gemini generateContent request to gemini-2.5-flash through /gemini with the prompt "Reply with the word ok. 0848ab9253aa"
    10. Upload a batch input file for openai-gpt-4o-mini to /v1/files
    11. Create a batch for openai-gpt-4o-mini on /v1/batches
    12. Post a callback log for gemini-2.5-flash to /v1/rust_control_plane/logs
    13. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  413. ✓test_failure_call_writes_failure_status_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-spend-failure-745ea5c49315 that calls openai/gpt-5.5
    3. Send a /chat/completions request to e2e-spend-failure-745ea5c49315 with the prompt "trigger failure da2047c5bb7c"
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  414. ✓test_failure_rows_share_normalized_error_across_provider_wordingspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • openai
    • anthropic/claude-haiku-4-5
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-norm-openai-b31102d26af5 that calls openai/gpt-5.5
    3. Send a /chat/completions request to e2e-norm-openai-b31102d26af5 with the prompt "normalize failure b31102d26af5"
    4. Add a deployment named e2e-norm-anthropic-b31102d26af5 that calls anthropic/claude-haiku-4-5
    5. Send a /chat/completions request to e2e-norm-anthropic-b31102d26af5 with the prompt "normalize failure b31102d26af5"
    6. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  415. ✓test_generous_mode_lets_priority_borrow_past_reservation TestDynamicRateLimitPriorityratelimit/test_dynamic_rate_limit_priority_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-dynpri-generous-9fc36bf3995a that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-dynpri-generous-9fc36bf3995a, key alias: e2e-dynpri-generous-dev-c88d82f6661e and metadata: priority: dev
    3. Generate a virtual key with models: e2e-dynpri-generous-9fc36bf3995a, key alias: e2e-dynpri-generous-prod-0c167a2ab91a and metadata: priority: prod
    4. Read the key's settings back from /key/info
    5. Send a /chat/completions request to e2e-dynpri-generous-9fc36bf3995a with the prompt "reply with one word fe462e170fbf"
    6. Send a /chat/completions request to e2e-dynpri-generous-9fc36bf3995a with the prompt "reply with one word b1daf90bee64"
    7. Send a /chat/completions request to e2e-dynpri-generous-9fc36bf3995a with the prompt "reply with one word 34a4c8f32e0b"
    8. Send a /chat/completions request to e2e-dynpri-generous-9fc36bf3995a with the prompt "reply with one word e6d86dbccd83"
    View test on GitHub ↗
  416. ✓test_headers_report_limit_and_remaining TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5, tpm limit: 100000 and rpm limit: 5
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word c8b8e232666b"
    View test on GitHub ↗
  417. ✓test_health_check_rows_keep_the_service_account_key TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • health
    • gemini
    • gemini-2.5-flash
    1. Run a health check on gemini-2.5-flash with /health
    2. Read /spend/logs between 2026-10-18 04:06:40.372059+00:00 and 2026-10-18 04:08:41.054547+00:00
    3. Read /spend/logs between 2026-10-18 04:06:40.372059+00:00 and 2026-10-18 04:08:46.124297+00:00
    4. Read /spend/logs between 2026-10-18 04:06:40.372059+00:00 and 2026-10-18 04:08:51.185082+00:00
    View test on GitHub ↗
  418. ✓test_intercepted_search_is_billed_under_the_request_session TestWebSearchInterceptionSessionspend_tracking/test_websearch_interception_session_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • bedrock
    • perplexity
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • web_search
    1. Add a deployment named e2e-websearch-session-a71a83f1569b that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with models: e2e-websearch-session-a71a83f1569b and user id: e2e-test-user
    3. Send a /v1/messages request to e2e-websearch-session-a71a83f1569b
    4. Wait for at least 2 of the session's spend logs in /spend/logs
    View test on GitHub ↗
  419. ✓test_key_budget_reset_at_advances_after_windowbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Read the key's settings back from /key/info
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance e406ef3b16d7"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 8cf43d195ad9"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 75a37b5d3110"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance ea4a0fa9cddd"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 130b59625202"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 5afbb6d4fa40"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance d86bb9ca426f"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 2a5f133f38e5"
    11. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance b092fefe62bd"
    12. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance adc6628217be"
    13. Read the key's settings back from /key/info
    View test on GitHub ↗
  420. ✓test_key_spend_blocks_at_capbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 5dbdfb96a477"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance cd6ae19826db"
    View test on GitHub ↗
  421. ✓test_key_spend_equals_sum_of_logsspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "say hi a15419c9bad5"
    3. Send a /chat/completions request to gemini-2.5-flash with the prompt "say hi aad5ae0c4936"
    4. Wait for at least 2 of the key's spend logs in /spend/logs
    5. Wait for the key's spend in /key/info to reach the expected minimum
    View test on GitHub ↗
  422. ✓test_key_with_budget_duration_schedules_reset_at_creationbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • key_management
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  423. ✓test_long_window_blocks_after_short_window_resetsbudgets/test_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with models: gpt-5.5 and budget limits: budget duration: 30s and max budget: 1e-09, budget duration: 1d and max budget: 5e-07
    2. Send a /chat/completions request to gpt-5.5 with the prompt "window 6f4aed717c7f"
    3. Send a /chat/completions request to gpt-5.5 with the prompt "window 873a393d6eb2"
    4. Read the key's budget windows from /key/info
    5. Send a /chat/completions request to gpt-5.5 with the prompt "window d8cb5e1388b4"
    View test on GitHub ↗
  424. ✓test_member_budget_blocks_without_touching_teammates TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-budget-team-b74f345d4e02 and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Generate a virtual key with user id: b07cb2da-c427-41b2-bfc2-d1436dd3cb3e and team id: 0e5b6087-0449-4c5f-91a0-088cb14e3ce0
    5. Create an internal user with max budget: 100.0
    6. Add the internal user to the team
    7. Generate a virtual key with user id: 65f31de9-54fe-416f-bc61-1e14c5d5151a and team id: 0e5b6087-0449-4c5f-91a0-088cb14e3ce0
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 9f904e532500"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend ba4e37d03900"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend c21b4f34aeab"
    View test on GitHub ↗
  425. ✓test_member_spend_attributed_to_team_and_user TestTeamMemberBudgetbudgets/test_team_member_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-team-member-af07956a0e91 and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Generate a virtual key with user id: 4ab406b7-e076-4b8e-82b7-5b98f95e7aca and team id: f9f2847c-bf33-4180-8cd0-3fb0c20521ba
    5. Send a /chat/completions request to claude-haiku-4-5
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  426. ✓test_member_spend_over_budget_is_blocked TestTeamMemberBudgetbudgets/test_team_member_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend b8b30042ecd5"
    View test on GitHub ↗
  427. ✓test_messages_bridge_keeps_cache_tokens TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-5.3-codex
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.3-codex
    3. Send a /v1/messages request to bridge-cache-priced-59bca2cf039a
    4. Wait for a matching cost breakdown in the key's /spend/logs
    View test on GitHub ↗
  428. ✓test_messages_stream_records_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • spend-budgets
    • messages
    • stream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a streaming /v1/messages request to tier-messages-stream-ad49778f8608
    4. Wait for a matching cost breakdown in the key's /spend/logs
    View test on GitHub ↗
  429. ✓test_missing_end_date_is_rejected TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py›
    • spend-budgets
    • spend_reporting
    1. Probe /team/daily/activity
    View test on GitHub ↗
  430. ✓test_missing_start_date_is_rejected TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py›
    • spend-budgets
    • spend_reporting
    1. Probe /team/daily/activity
    View test on GitHub ↗
  431. ✓test_model_group_shares_rpm_bucket_with_alias TestModelGroupAliasRateLimitratelimit/test_model_group_alias_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • e2e-alias-rl-alias
    • e2e-alias-rl-target
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word 60f4abc3c0d6"
    3. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word 7f7bc434c078"
    4. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word 3f5b9c791f06"
    5. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word cf569d5f76ea"
    View test on GitHub ↗
  432. ✓test_model_max_budget_isolates_per_modelbudgets/test_model_max_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • gemini
    • claude-haiku-4-5
    • gemini-2.5-flash
    1. Generate a virtual key with model max budget: claude-haiku-4-5: budget limit: 1e-06 and time period: 30d and gemini-2.5-flash: budget limit: 1000.0 and time period: 30d
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi 1847b48a475e"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi 72886c734ffc"
    4. Send a /chat/completions request to gemini-2.5-flash with the prompt "hi 83caa7d11426"
    View test on GitHub ↗
  433. ✓test_multi_window_key_resets_each_window_independentlybudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with budget limits: budget duration: 30s and max budget: 3e-06, budget duration: 1m and max budget: 1.0
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 11ded95f38a7"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance a8bba6c3c3d7"
    4. Read the key's settings back from /key/info
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 3f6dc045f7d8"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 2b6885dbc523"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 2ae2adb1aa12"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance d61b8537b051"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance bef9310362eb"
    10. Read the key's settings back from /key/info
    View test on GitHub ↗
  434. ✓test_one_request_lands_the_same_spend_on_every_surface TestSpendSurfaceConsistencyspend_tracking/test_spend_surface_consistency_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.6-luna
    1. List the deployments from /model/info
    2. Add a deployment named e2e-spend-surfaces-f566bbd77910 that calls openai/gpt-5.6-luna
    3. Create a team with team alias: e2e-spend-surfaces-f566bbd77910
    4. Generate a virtual key with models: e2e-spend-surfaces-f566bbd77910, team id: 6455a1cf-c145-46fe-b562-7cd5c1d89cd2 and key alias: e2e-spend-surfaces-f566bbd77910
    5. Send a /chat/completions request to e2e-spend-surfaces-f566bbd77910
    6. Wait for the request's spend log in /spend/logs
    7. Wait for the key's spend in /key/info to reach the expected minimum
    8. Wait for the team's spend in /team/info to reach the expected minimum
    9. Wait for at least 1 of the key's requests in /user/daily/activity/aggregated
    10. Scrape /metrics/ on every proxy replica
    View test on GitHub ↗
  435. ✓test_org_budget_blocks_keys_under_it TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the organization e2e-budget-org-61a1d7181b05 with max budget: 3e-06
    2. Create the team e2e-budget-team-23c5d874647b and wait until /team/info returns it
    3. Generate a virtual key with team id: e5f813d1-fe50-4aab-8de7-1222d7fefc12
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend ca3bee30c9c5"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 3fd4847ab1e6"
    View test on GitHub ↗
  436. ✓test_org_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the organization e2e-org-reset-966276d6ff4e with max budget: 3e-06
    2. Create the team e2e-org-team-7316b872fc61 and wait until /team/info returns it
    3. Generate a virtual key with team id: 6cd23dc7-9555-49ef-b5ca-5330f12add58
    4. Read the organization's budget id from /organization/info
    5. Read the budget from /budget/info
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 6d05a1c299d8"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset c43438a48267"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset e642e7dfe8fa"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset eb596ffb2f9f"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 4ca0a1e8183d"
    11. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 2e1b0940f5c7"
    12. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 9876028fe219"
    13. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 275c46c04bc1"
    View test on GitHub ↗
  437. ✓test_personal_key_blocks_over_its_own_budget TestKeyBudgetBlocksAcrossKeyKindsbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create an internal user with max budget: 100.0
    2. Generate a virtual key with max budget: 3e-06 and user id: a6ccde87-7583-424b-bc07-7a8001f5a425
    3. Generate a virtual key with user id: a6ccde87-7583-424b-bc07-7a8001f5a425
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 7896eb13d5f3"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 8794eded8933"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 7a5d8f19fad5"
    View test on GitHub ↗
  438. ✓test_personal_key_resets_after_window TestKeyBudgetResetAcrossKeyKindsbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create an internal user with max budget: 100.0
    2. Generate a virtual key with max budget: 3e-06, budget duration: 30s and user id: 3e86b1a7-0ab0-47de-b31f-f980d96bdf9b
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 557ca73f5372"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 5ada2ee5c4f8"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset ec7ec03c0433"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset a92beb8f1477"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 899b76496cf6"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset c3000bcdd53f"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset c90ef3c1d932"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 50bcff69ff31"
    View test on GitHub ↗
  439. ✓test_personal_key_user_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create an internal user with max budget: 3e-06
    2. Generate a virtual key with user id: 9cf205e7-3d8f-445b-88c3-cb4bc90dd197
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset c3439b8e9f40"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset a46b43009604"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset fd55b7de882e"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 929c361b8093"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 17734f50a950"
    View test on GitHub ↗
  440. ✓test_pre_call_rejection_row_attributes_provider_and_model_idspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-spend-precall-5c4f279da945 that calls openai/gpt-5.5
    2. Generate a virtual key with models: e2e-spend-precall-5c4f279da945 and rpm limit: 1
    3. Send a /chat/completions request to e2e-spend-precall-5c4f279da945 with the prompt "reply with one word 5de2f48eca45"
    4. Send a /chat/completions request to e2e-spend-precall-5c4f279da945 with the prompt "over the rpm limit 1d678b243140"
    5. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  441. ✓test_priority_tier_bills_priority_rates TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.6-luna
    • reasoning
    1. List the deployments from /model/info
    2. Generate a virtual key with user id: e2e-test-user
    3. Add a deployment with custom rates that calls openai/gpt-5.6-luna
    4. Send a /chat/completions request to tier-priced-8ffef866f940
    5. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  442. ✓test_request_tags_round_tripspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "tagged request"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  443. ✓test_reset_wait_never_yields_non_budget_errorbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 777cc9c4e8c9"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance c3bc9c94e9f3"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 2b7f3f481e57"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 658133c88142"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 45bfe783acb9"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance cf7ea4f60095"
    View test on GitHub ↗
  444. ✓test_responses_stream_records_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • spend-budgets
    • responses
    • stream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a streaming /v1/responses request to tier-responses-stream-f935b1034501
    4. Wait for a matching cost breakdown in the key's /spend/logs
    View test on GitHub ↗
  445. ✓test_rpm_limit_blocks_over_limit TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and rpm limit: 3
    2. Read the key's settings back from /key/info
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word d849b56b3fd4"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word d92692326245"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word e77567a3671d"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 0db636c3a518"
    View test on GitHub ↗
  446. ✓test_rpm_limit_one_blocks_second_call TestRedisBackedRateLimitratelimit/test_redis_backed_ratelimit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-redis-rpm-8899f3db8cc7 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-redis-rpm-8899f3db8cc7, key alias: e2e-redis-rpm-8c94e2b1f143 and rpm limit: 1
    3. Read the key's settings back from /key/info
    4. Send a /chat/completions request to e2e-redis-rpm-8899f3db8cc7 with the prompt "ping 178e3d76d2fd"
    5. Send a /chat/completions request to e2e-redis-rpm-8899f3db8cc7 with the prompt "pong b61b411f3faf"
    View test on GitHub ↗
  447. ✓test_rpm_limit_resets_after_window TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and rpm limit: 1
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word df44bbac4fa5"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 3c6f77da843b"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 3a3a660cb39a"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 58dc563c6ad6"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 89e8f39b7f54"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 7477ad48d1dc"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 23512030e0ef"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word ea90159ebcce"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 85a87835a3b1"
    11. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 3480539e7967"
    12. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 32f9a05fe63a"
    13. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 3b525721031d"
    14. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word a23b443be8c6"
    15. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word d171b0d77e0f"
    16. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 2d43baf11d27"
    View test on GitHub ↗
  448. ✓test_schema_listed_spend_routes_are_responsivespend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Read the OpenAPI schema from /openapi.json
    View test on GitHub ↗
  449. ✓test_short_window_blocks_then_resetsbudgets/test_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with models: gpt-5.5 and budget limits: budget duration: 30s and max budget: 1e-09, budget duration: 1m and max budget: 1.0
    2. Send a /chat/completions request to gpt-5.5 with the prompt "window bdb1a361c8eb"
    3. Send a /chat/completions request to gpt-5.5 with the prompt "window 714f316e76ac"
    4. Send a /chat/completions request to gpt-5.5 with the prompt "window 4880e43c5ac5"
    5. Send a /chat/completions request to gpt-5.5 with the prompt "window feabc6c58f8a"
    View test on GitHub ↗
  450. ✓test_soft_budget_does_not_blockbudgets/test_soft_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 1000.0 and soft budget: 1e-09
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi 6a91adfdf6e7"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi 496c1abff469"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi f6e2018a6a94"
    View test on GitHub ↗
  451. ✓test_spend_calculate_returns_nonzero_costspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • spend_reporting
    • gemini
    • gemini-2.5-flash
    1. Estimate the cost of sending "estimate the cost of this request" to gemini-2.5-flash with /spend/calculate
    View test on GitHub ↗
  452. ✓test_spend_logs_by_key_return_every_row_with_the_alias TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • spend_reporting
    • anthropic
    • gemini
    • openai
    • claude-haiku-4-5
    • gemini-2.5-flash
    • openai-gpt-4o-mini
    • openai-responses-codex
    • openai-text-embedding-3-small
    1. Wait for at least 9 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  453. ✓test_spend_logs_endpoint_returns_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • spend_reporting
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "spend logs 0e3ac3545fee"
    3. GET /spend/logs
    View test on GitHub ↗
  454. ✓test_spend_logs_v2_pagination_caps_pages_and_keeps_totalspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • spend_reporting
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "page fodder da7f59597bcc"
    3. Send a /chat/completions request to gemini-2.5-flash with the prompt "page fodder 801e562dc2fb"
    4. Wait for at least 2 of the key's spend logs in /spend/logs
    5. Read page 1 of /spend/logs/v2 at a page size of 1
    6. Read page 9 of /spend/logs/v2 at a page size of 1
    7. Read page 1 of /spend/logs/v2 at a page size of 1
    View test on GitHub ↗
  455. ✓test_spend_route_responsive[/customer/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /customer/daily/activity
    View test on GitHub ↗
  456. ✓test_spend_route_responsive[/customer/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • customer_management
    1. Call the management route /customer/list
    View test on GitHub ↗
  457. ✓test_spend_route_responsive[/end_user/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /end_user/daily/activity
    View test on GitHub ↗
  458. ✓test_spend_route_responsive[/global/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/activity
    View test on GitHub ↗
  459. ✓test_spend_route_responsive[/global/activity/exceptions]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/activity/exceptions
    View test on GitHub ↗
  460. ✓test_spend_route_responsive[/global/activity/exceptions/deployment]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/activity/exceptions/deployment
    View test on GitHub ↗
  461. ✓test_spend_route_responsive[/global/activity/model]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/activity/model
    View test on GitHub ↗
  462. ✓test_spend_route_responsive[/global/all_end_users]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/all_end_users
    View test on GitHub ↗
  463. ✓test_spend_route_responsive[/global/spend]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend until it answers successfully
    View test on GitHub ↗
  464. ✓test_spend_route_responsive[/global/spend/all_tag_names]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/all_tag_names
    View test on GitHub ↗
  465. ✓test_spend_route_responsive[/global/spend/keys]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/keys until it answers successfully
    View test on GitHub ↗
  466. ✓test_spend_route_responsive[/global/spend/logs]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/logs until it answers successfully
    View test on GitHub ↗
  467. ✓test_spend_route_responsive[/global/spend/models]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/models until it answers successfully
    View test on GitHub ↗
  468. ✓test_spend_route_responsive[/global/spend/provider]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/provider
    View test on GitHub ↗
  469. ✓test_spend_route_responsive[/global/spend/report]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/report
    View test on GitHub ↗
  470. ✓test_spend_route_responsive[/global/spend/tags]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/tags until it answers successfully
    View test on GitHub ↗
  471. ✓test_spend_route_responsive[/global/spend/teams]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/teams
    View test on GitHub ↗
  472. ✓test_spend_route_responsive[/key/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • key_management
    1. Call the management route /key/list
    View test on GitHub ↗
  473. ✓test_spend_route_responsive[/organization/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /organization/daily/activity
    View test on GitHub ↗
  474. ✓test_spend_route_responsive[/organization/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • organization_management
    1. Call the management route /organization/list
    View test on GitHub ↗
  475. ✓test_spend_route_responsive[/spend/keys]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. List the deployments from /model/info
    2. Add a deployment named openai-text-embedding-3-small that calls openai/text-embedding-3-small
    3. Add a deployment named openai-responses-codex that calls openai/gpt-5.3-codex
    4. Add a deployment named openai-gpt-4o-mini that calls openai/gpt-4o-mini
    5. Call the management route /spend/keys
    View test on GitHub ↗
  476. ✓test_spend_route_responsive[/spend/logs]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/logs
    View test on GitHub ↗
  477. ✓test_spend_route_responsive[/spend/logs/session/ui]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/logs/session/ui
    View test on GitHub ↗
  478. ✓test_spend_route_responsive[/spend/logs/ui]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/logs/ui
    View test on GitHub ↗
  479. ✓test_spend_route_responsive[/spend/logs/v2]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/logs/v2
    View test on GitHub ↗
  480. ✓test_spend_route_responsive[/spend/tags]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/tags
    View test on GitHub ↗
  481. ✓test_spend_route_responsive[/spend/users]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/users
    View test on GitHub ↗
  482. ✓test_spend_route_responsive[/tag/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /tag/daily/activity
    View test on GitHub ↗
  483. ✓test_spend_route_responsive[/team/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /team/daily/activity
    View test on GitHub ↗
  484. ✓test_spend_route_responsive[/team/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • team_management
    1. Call the management route /team/list
    View test on GitHub ↗
  485. ✓test_spend_route_responsive[/user/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /user/daily/activity
    View test on GitHub ↗
  486. ✓test_spend_route_responsive[/user/daily/activity/aggregated]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /user/daily/activity/aggregated
    View test on GitHub ↗
  487. ✓test_spend_route_responsive[/user/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • user_management
    1. Call the management route /user/list
    View test on GitHub ↗
  488. ✓test_streamed_call_records_and_bills_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • spend-budgets
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.6-luna
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.6-luna
    3. Send a streaming /chat/completions request to tier-priced-stream-c4006ccb3959
    4. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  489. ✓test_streaming_cache_read_billed_at_cache_read_rate TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • stream
    • openai
    • openai/gpt-5.5
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a /chat/completions request to stream-cache-priced-7cb7f495b44b
    4. Send a streaming /chat/completions request to stream-cache-priced-7cb7f495b44b
    5. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  490. ✓test_streaming_chat_completion_tracks_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • chat_completions
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to gemini-2.5-flash with the prompt "count to three f41ef52ca642"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  491. ✓test_streaming_messages_via_responses_bridge_tracks_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • messages
    • stream
    • openai
    • openai-responses-codex
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to openai-responses-codex with the prompt "reply with exactly one word b32599c6c276"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  492. ✓test_tag_budget_blocks_tagged_requestsbudgets/test_tag_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Create the tag e2e-budget-tag-c5adb99f2541 with max budget: 1e-06
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi dc0b37f97a13"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi 4fd4b7b09f32"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi 15e54a6b5176"
    View test on GitHub ↗
  493. ✓test_tag_spend_matches_sum_of_tagged_logsspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "hi 152a1b6280c7"
    3. Send a /chat/completions request to gemini-2.5-flash with the prompt "hi 56983fd613ba"
    4. Wait for at least 2 of the key's spend logs in /spend/logs
    5. Wait for the tag e2e-tagspend-463aab428045 to reach the expected spend in /spend/tags
    View test on GitHub ↗
  494. ✓test_team_budget_blocks_every_team_key TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-budget-team-7df5d48c25ad and wait until /team/info returns it
    2. Generate a virtual key with team id: 103a4f1c-2c2a-4150-ae4c-e61329ddc377
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 59cef391b07a"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend dffa4acffff2"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend d4b000cebb2e"
    View test on GitHub ↗
  495. ✓test_team_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-team-reset-4e15518192bf and wait until /team/info returns it
    2. Generate a virtual key with team id: cd50c797-d03b-4ac9-b8aa-f6589873d551
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 3ec2b5c46474"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 8eed9a88a2eb"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 581c5f01f3fc"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset a7bf46274db9"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 7813a301cb0e"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 019d52f81221"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 8b71eb43d668"
    View test on GitHub ↗
  496. ✓test_team_key_blocks_over_its_own_budget TestKeyBudgetBlocksAcrossKeyKindsbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-key-cap-team-bfbc73d72cbb and wait until /team/info returns it
    2. Generate a virtual key with max budget: 3e-06 and team id: a7a2428d-720b-4388-a3bc-36e0325aac02
    3. Generate a virtual key with team id: a7a2428d-720b-4388-a3bc-36e0325aac02
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 33058abe6371"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 039fa21c7d06"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend b757883c7fa4"
    View test on GitHub ↗
  497. ✓test_team_key_resets_after_window TestKeyBudgetResetAcrossKeyKindsbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-key-reset-team-f6a746c16abe and wait until /team/info returns it
    2. Generate a virtual key with max budget: 3e-06, budget duration: 30s and team id: 19aba9b8-f25c-4e94-9356-de8c96c963cb
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 74c29bd8ca6c"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 2b7ba13b93aa"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 99bd0c9dda33"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset cc346b4a6ad2"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset f62d81fbfca3"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 716d2f372ac2"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 7b4f9bd85ac8"
    View test on GitHub ↗
  498. ✓test_team_long_window_blocks_after_short_window_resetsbudgets/test_team_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-team-long-window-ec56bc9137bf and wait until /team/info returns it
    2. Generate a virtual key with models: claude-haiku-4-5 and team id: ba30ab0e-3195-4cee-af61-19c66993607f
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 9948a97173ed"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window ee0db94aec67"
    5. Read the team's budget windows from /team/info
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 03baa480940f"
    View test on GitHub ↗
  499. ✓test_team_member_budget_reset_at_advancesbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-member-advance-d4ee303fbd90 and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Update the team member's budget with /team/member_update
    5. Read the team member's budget reset time from /team/info
    6. Generate a virtual key with user id: ca15666d-e828-48f9-8a92-4cb248f662e7 and team id: 4baa40e7-1323-45a4-987c-68de1323938f
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance cd47c66f278a"
    8. Read the team member's budget reset time from /team/info
    View test on GitHub ↗
  500. ✓test_team_member_budget_reset_keeps_advancingbudgets/test_team_member_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-member-reset-cde3f4d12c70 and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Update the team member's budget with /team/member_update
    5. Read the team member's budget reset time from /team/info
    6. Generate a virtual key with user id: aeae3ca5-d061-4613-91d8-e1a523c736fc and team id: ba83def0-7339-49eb-a7d1-cf023d5ac926
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset c293a9220bf6"
    8. Read the team member's budget reset time from /team/info
    View test on GitHub ↗
  501. ✓test_team_member_key_blocks_over_its_own_budget TestKeyBudgetBlocksAcrossKeyKindsbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-key-cap-team-eb3c0f9fb7c8 and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Generate a virtual key with max budget: 3e-06, user id: f461be50-d27e-4fd9-98e5-946c4d8e42f7 and team id: ad74a9d3-36ca-4a30-bf97-7976b93b9ec3
    5. Generate a virtual key with user id: f461be50-d27e-4fd9-98e5-946c4d8e42f7 and team id: ad74a9d3-36ca-4a30-bf97-7976b93b9ec3
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 449e8aa4fad0"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 3f70323ea72a"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend bd420cfdfc66"
    View test on GitHub ↗
  502. ✓test_team_member_key_resets_after_window TestKeyBudgetResetAcrossKeyKindsbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-key-reset-team-de3df90ea03d and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Generate a virtual key with max budget: 3e-06, budget duration: 30s, user id: 5a53e9c7-9855-4aee-9c93-7c936c9fc723 and team id: 6986b41b-b09b-4959-b4c1-49db461dc2d3
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset f49834af6e00"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 5cfd2c4487d6"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 5d8cb93f7325"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset f66363c6bda7"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset fd3236fb8243"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset efe170e810f6"
    11. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset ef26844228ec"
    12. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 04c0ffbd6887"
    View test on GitHub ↗
  503. ✓test_team_short_window_blocks_then_resetsbudgets/test_team_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-team-window-d6950c437225 and wait until /team/info returns it
    2. Generate a virtual key with models: claude-haiku-4-5 and team id: efedbab8-e794-411c-a332-b55d92a8388c
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 016b5a1ab639"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 72fe07ecc1ac"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 5b14049df599"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 081fc9643c15"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 2e01a4e62582"
    View test on GitHub ↗
  504. ✓test_terminal_batch_cost_row_joins_the_retrieving_key TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • openai-gpt-4o-mini
    1. Retrieve the openai batch from /v1/batches
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  505. ✓test_the_budget_read_reports_the_spend_drawn_against_the_pool TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • budget_management
    • openai
    • openai/gpt-5.4-nano
    1. Read the budget and spend of the model access group e2e-mag-budget-0a1b3d48afce
    View test on GitHub ↗
  506. ✓test_the_key_that_drained_the_pool_stays_blocked TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-mag-budget-0a1b3d48afce
    2. Add a deployment named e2e-mag-budgeted-0a1b3d48afce that calls openai/gpt-5.4-nano
    3. Add a deployment named e2e-mag-unbudgeted-0a1b3d48afce that calls openai/gpt-5.4-nano
    4. Set a shared budget on the model access group e2e-mag-budget-0a1b3d48afce
    5. Send a /chat/completions request to e2e-mag-budgeted-0a1b3d48afce with the prompt "hi cbcc8b7e3c79"
    6. Send a /chat/completions request to e2e-mag-budgeted-0a1b3d48afce with the prompt "hi 77040c00e8ed"
    7. Send a /chat/completions request to e2e-mag-budgeted-0a1b3d48afce with the prompt "hi f59978523621"
    View test on GitHub ↗
  507. ✓test_tpm_limit_blocks_over_limit TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and tpm limit: 60
    2. Read the key's settings back from /key/info
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 16f894b6e72d"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 8ab1d51d6270"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 235a4580ed67"
    View test on GitHub ↗
  508. ✓test_user_budget_blocks_a_second_key TestUserBudgetAcrossKeysbudgets/test_user_budget_across_keys_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Create an internal user with max budget: 3e-06
    2. Generate a virtual key with user id: 21d4a092-12b3-49f3-816c-4b2c54746604
    3. Send a /chat/completions request to gpt-5.5 with the prompt "across 0f6c32497961"
    4. Send a /chat/completions request to gpt-5.5 with the prompt "across 076ffa3fce32"
    5. Send a /chat/completions request to gpt-5.5 with the prompt "across e9e8af33f409"
    6. Read the internal user's spend and budget from /user/info
    View test on GitHub ↗
  509. ✓test_user_budget_enforced_across_their_personal_keys TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create an internal user with max budget: 3e-06
    2. Generate a virtual key with user id: 3250825f-aceb-4382-bbff-c0a9460fe8c9
    3. Create the team e2e-budget-team-2ac7b12502d3 and wait until /team/info returns it
    4. Add the internal user to the team
    5. Generate a virtual key with user id: 3250825f-aceb-4382-bbff-c0a9460fe8c9 and team id: 1d8030c8-04c2-419e-80b4-181805265dc1
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend a9a0548357b0"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 6b32abecd60c"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 4c6613f94efe"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 1c9f368ac474"
    View test on GitHub ↗
  510. ✓test_user_daily_activity_reports_alias_and_email TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • spend_reporting
    • anthropic
    • gemini
    • openai
    • claude-haiku-4-5
    • gemini-2.5-flash
    • openai-gpt-4o-mini
    • openai-responses-codex
    • openai-text-embedding-3-small
    1. Wait for at least 9 of the key's requests in /user/daily/activity
    View test on GitHub ↗
  511. ✓test_valid_date_range_returns_results_and_metadata TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py›
    • spend-budgets
    • spend_reporting
    • openai
    • openai/gpt-5.6-luna
    1. Add a priced deployment, then create two teams with one key each and send 7 /chat/completions requests per key, 6 of them at once
    2. Send a /chat/completions request to e2e-reconciliation-5006d7259798
    3. Check that the /spend/logs rows of team 5d4340c1-5f9a-4db3-a156-5db280607621 match each response's tokens and cost
    4. Check that the /spend/logs rows of team 35842378-5090-450c-bc81-cbaf07a6b235 match each response's tokens and cost
    5. Probe /team/daily/activity
    View test on GitHub ↗
  512. −test_end_user_model_max_budget_enforces_per_model_rpmbudgets/test_model_max_budget_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash

    No steps recorded

    stage red: product gap, end-user model_max_budget rpm_limit is stored but never enforced

    View test on GitHub ↗
  513. −test_strict_mode_blocks_saturated_priority_but_serves_the_other TestDynamicRateLimitPriorityratelimit/test_dynamic_rate_limit_priority_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001

    No steps recorded

    LIT-5118: the stage proxy does not run the dynamic_rate_limiter_v3 callbacks + priority_reservation config this module's docstring requires (zero limiter log lines on any pod during the 2026-08-02 run), so strict enforcement can never engage there

    View test on GitHub ↗
  514. ✓test_access_group_info_reports_membership TestModelAccessGroupRoutestest_model_tag_accessgroup_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-agmodel-8342e92bee6b that calls openai/gpt-5.5
    2. POST /access_group/new
    3. GET /access_group/e2e-mgmt-ag-681c9d34108f/info
    View test on GitHub ↗
  515. ✓test_actor_subject_and_database_role[internal_user_viewer] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an actor in the internal_user_viewer role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the internal_user_viewer role
    3. Read the user back from /user/info
    4. GET /user/info
    View test on GitHub ↗
  516. ✓test_actor_subject_and_database_role[internal_user] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an actor in the internal_user role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the internal_user role
    3. Read the user back from /user/info
    4. GET /user/info
    View test on GitHub ↗
  517. ✓test_actor_subject_and_database_role[organization_admin] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the organization_admin role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the organization_admin role
    4. Read the user back from /user/info
    5. GET /user/info
    6. Read the team back from /team/info
    7. Read the organization back from /organization/info
    View test on GitHub ↗
  518. ✓test_actor_subject_and_database_role[proxy_admin_viewer] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an actor in the proxy_admin_viewer role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the proxy_admin_viewer role
    3. Read the user back from /user/info
    4. GET /user/info
    View test on GitHub ↗
  519. ✓test_actor_subject_and_database_role[proxy_admin] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an actor in the proxy_admin role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the proxy_admin role
    3. Read the user back from /user/info
    4. GET /user/info
    View test on GitHub ↗
  520. ✓test_actor_subject_and_database_role[team_admin] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the team_admin role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the team_admin role
    4. Read the user back from /user/info
    5. GET /user/info
    6. Read the team back from /team/info
    7. Read the organization back from /organization/info
    View test on GitHub ↗
  521. ✓test_actor_subject_and_database_role[team_member] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the team_member role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the team_member role
    4. Read the user back from /user/info
    5. GET /user/info
    6. Read the team back from /team/info
    7. Read the organization back from /organization/info
    View test on GitHub ↗
  522. ✓test_actor_subject_and_database_role[unrelated_user] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an actor in the unrelated_user role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the unrelated_user role
    3. Read the user back from /user/info
    4. GET /user/info
    View test on GitHub ↗
  523. ✓test_admin_creates_reads_updates_clears_and_deletes_a_key[direct_jwt] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • key_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the proxy_admin role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the proxy_admin role
    4. Read the user back from /user/info
    5. Generate a virtual key with models: gpt-5.5, team id: ed44f2d4-4386-4458-b269-51e505866fac and key alias: e2e-jwt-key-5ede78cf477a
    6. Read the key's settings back from /key/info
    7. Update the virtual key's settings with /key/update
    8. Read the key's settings back from /key/info
    9. Update the virtual key's settings with /key/update
    10. Read the key's settings back from /key/info
    11. List the keys with the alias e2e-jwt-key-5ede78cf477a-updated from /key/list
    12. Delete the virtual key
    13. List the keys with the alias e2e-jwt-key-5ede78cf477a-updated from /key/list
    View test on GitHub ↗
  524. ✓test_admin_creates_reads_updates_clears_and_deletes_a_key[virtual_key] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • key_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the proxy_admin role, with its identity provider user, internal user and any tenant memberships
    3. Generate a virtual key as the proxy admin
    4. Read the user back from /user/info
    5. Generate a virtual key with models: gpt-5.5, team id: 0b960786-369f-4377-aec0-0271253f540c and key alias: e2e-jwt-key-ce2d4f01c203
    6. Read the key's settings back from /key/info
    7. Update the virtual key's settings with /key/update
    8. Read the key's settings back from /key/info
    9. Update the virtual key's settings with /key/update
    10. Read the key's settings back from /key/info
    11. List the keys with the alias e2e-jwt-key-ce2d4f01c203-updated from /key/list
    12. Delete the virtual key
    13. List the keys with the alias e2e-jwt-key-ce2d4f01c203-updated from /key/list
    View test on GitHub ↗
  525. ✓test_admin_viewer_reads_but_cannot_update TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • key_management
    1. Create an actor in the proxy_admin_viewer role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the proxy_admin_viewer role
    3. Generate a virtual key as the proxy admin
    4. Update the virtual key's settings with /key/update
    5. Read the key's settings back from /key/info
    6. Update the virtual key's settings with /key/update
    7. Read the key's settings back from /key/info
    View test on GitHub ↗
  526. ✓test_approve_activates_submission_and_persists TestMcpServerSubmissiontest_config_misc_endpoints_e2e.py›
    • mcp
    • mcp
    1. Create a team with team alias: e2e-mcp-team-e7add0ab9af2
    2. Generate a virtual key with team id: 313143ba-2a2d-4dae-924d-0e5779b67f42
    3. POST /v1/mcp/server/register
    4. PUT /v1/mcp/server/ba6d00db-e844-4432-8269-32d4d374d353/approve
    5. GET /v1/mcp/server/ba6d00db-e844-4432-8269-32d4d374d353
    View test on GitHub ↗
  527. ✓test_bedrock_mantle_responses_connection_succeeds TestModelTestConnectiontest_model_test_connection_e2e.py›
    • management
    • health
    • nonstream
    • bedrock_mantle
    • bedrock_mantle/openai.gpt-5.6-luna
    1. Run Test Connection on bedrock_mantle/openai.gpt-5.6-luna in responses mode with /health/test_connection
    View test on GitHub ↗
  528. ✓test_block_persists_to_key_info TestKeyRoutestest_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with models: gemini-2.5-flash
    2. Read the key's settings back from /key/info
    3. Block the virtual key
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  529. ✓test_block_then_unblock_persists_to_model_info TestModelRoutestest_model_tag_accessgroup_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-model-block-94a40dbc449a that calls openai/gpt-5.5
    2. GET /model/info
    3. POST /model/block
    4. GET /model/info
    5. POST /model/unblock
    6. GET /model/info
    View test on GitHub ↗
  530. ✓test_block_then_unblock_persists_to_team_info TestTeamManagementRoutestest_team_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-team-block-3ebefea2583c and models: gemini-2.5-flash
    2. GET /team/info
    3. POST /team/block
    4. GET /team/info
    5. POST /team/unblock
    6. GET /team/info
    View test on GitHub ↗
  531. ✓test_bulk_update_applies_max_budget_to_target_key TestKeyManagementRoutestest_key_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with models: gpt-5.5 and max budget: 5.0
    2. Read the key's settings back from /key/info
    3. POST /key/bulk_update
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  532. ✓test_callbacks_list_reports_active_logging_callbacks TestInventoryRoutestest_config_misc_endpoints_e2e.py›
    • observability
    1. GET /callbacks/list
    View test on GitHub ↗
  533. ✓test_clearing_the_description_with_null_reads_back_null TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_5b20ef60d91e
    2. Update the MCP server's settings with PUT /v1/mcp/server
    3. Read /v1/mcp/server/119e189f-f194-482a-b41e-3bcfa4348dc0 on every proxy replica until it settles
    View test on GitHub ↗
  534. ✓test_clearing_the_description_with_null_reads_back_null TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_2f880bac9814
    2. Create an MCP toolset with the tools server id: f4558986-00c7-489c-ba2d-b1fd5d733acc and tool name: search_datadog_logs, server id: f4558986-00c7-489c-ba2d-b1fd5d733acc and tool name: get_datadog_metr
    3. Update an MCP toolset with toolset id: ff1418e1-dd88-4d7e-8449-7aaba0bce46c
    4. Read /v1/mcp/toolset/ff1418e1-dd88-4d7e-8449-7aaba0bce46c on every proxy replica until it settles
    View test on GitHub ↗
  535. ✓test_config_update_persists_router_setting_to_get TestRouterSettingstest_config_misc_endpoints_e2e.py›
    • routing
    • proxy_config
    1. GET /router/settings
    2. POST /config/update
    3. GET /router/settings
    4. POST /config/update
    5. GET /router/settings
    View test on GitHub ↗
  536. ✓test_create_echoes_every_field_written TestKeyLifecycletest_key_lifecycle_e2e.py›
    • management
    • key_management
    1. Add a deployment named e2e-key-lifecycle-9d02cbc562be that calls gpt-4o-mini
    2. Create a team with team alias: e2e-key-lifecycle-team-17df92149710
    3. Generate a virtual key with models: e2e-key-lifecycle-9d02cbc562be, max budget: 25.0, budget duration: 30d, team id: 3c145414-afe0-48ee-87aa-6fdcf6de1494, key alias: e2e-key-lifecycle-17df92149710, tp
    View test on GitHub ↗
  537. ✓test_create_persists_and_is_read_back TestFallbackManagementtest_config_misc_endpoints_e2e.py›
    • routing
    1. Add a deployment named e2e-fallback-primary-9632754bbc2b that calls openai/gpt-5.5
    2. Add a deployment named e2e-fallback-secondary-577bddb5627f that calls openai/gpt-5.5
    3. POST /fallback
    4. GET /fallback/e2e-fallback-primary-9632754bbc2b
    View test on GitHub ↗
  538. ✓test_create_persists_both_tools_under_the_exact_names_written TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_4b47bf3aba6b
    2. Create an MCP toolset with the tools server id: d890908d-4f2b-421d-a6d8-816036b27d29 and tool name: search_datadog_logs, server id: d890908d-4f2b-421d-a6d8-816036b27d29 and tool name: get_datadog_metr
    3. Read /v1/mcp/toolset/d5a6e69c-c761-4493-b695-ac76c3cea85a on every proxy replica until it settles
    4. Read /v1/mcp/toolset on every proxy replica until it settles
    View test on GitHub ↗
  539. ✓test_create_persists_every_field_on_every_replica TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_a1396a08f18e
    2. Read /v1/mcp/server/a2020bd6-3089-4087-9029-fd12c3d68a3d on every proxy replica until it settles
    View test on GitHub ↗
  540. ✓test_created_budget_appears_in_budget_list TestBudgetManagementtest_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. POST /budget/new
    2. GET /budget/list
    View test on GitHub ↗
  541. ✓test_created_key_appears_in_key_list_inventory TestKeyRoutestest_management_e2e.py›
    • management
    • key_management
    1. Count the keys with the alias e2e-mgmt-keylist-022e0d1e065e in /key/list
    2. Generate a virtual key with key alias: e2e-mgmt-keylist-022e0d1e065e
    3. Count the keys with the alias e2e-mgmt-keylist-022e0d1e065e in /key/list
    View test on GitHub ↗
  542. ✓test_created_team_appears_in_team_list TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-mgmt-team-648cb8f06fe0 and models: gemini-2.5-flash
    2. List the teams from /team/list
    View test on GitHub ↗
  543. ✓test_created_users_appear_in_user_list TestUserRoutestest_management_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-mgmt-574157c24c63@example.com and user role: internal_user
    2. Create an internal user with user email: e2e-mgmt-bb8c0e865c22@example.com and user role: internal_user
    3. List the matching users from /user/list
    View test on GitHub ↗
  544. ✓test_creating_a_key_from_the_dashboard_persists_and_works TestDashboardKeyRoutestest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Sign in to the Admin UI with /v2/login
    2. Read the key's settings back from /key/info
    3. Generate a virtual key with models: gemini-2.5-flash, key alias: e2e-mgmt-uicreate-d1340575ca04 and tpm limit: 100
    4. Read the key's settings back from /key/info
    5. List the keys with the alias e2e-mgmt-uicreate-d1340575ca04 from /key/list
    6. Send a /chat/completions request to gemini-2.5-flash
    7. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  545. ✓test_credential_migration_check_reports_residual_scan TestInventoryRoutestest_config_misc_endpoints_e2e.py›
    • deploy-ops
    1. GET /credentials/migrate-encryption/check
    View test on GitHub ↗
  546. ✓test_customer_create_persists_to_info TestCustomertest_management_e2e.py›
    • management
    • customer_management
    1. Create the end user e2e-customer-9352437e9fc6
    2. Read the end user e2e-customer-9352437e9fc6 back from /customer/info
    View test on GitHub ↗
  547. ✓test_delete_forbidden_for_non_admin_key TestKeyManagementRoutestest_key_management_e2e.py›
    • proxy-auth
    • key_management
    1. Generate a virtual key with models: gpt-5.5
    2. Send a request to /key/delete
    3. Read the key's settings back from /key/info
    View test on GitHub ↗
  548. ✓test_delete_persists_and_revokes_team_bound_key TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    • nonstream
    • openai
    • gpt-5.5
    1. Create a team with team alias: e2e-mgmt-team-4572c0955ae9 and models: gpt-5.5
    2. Generate a virtual key with team id: 0479fe1b-f5cd-4277-bb31-24bd0aafc076
    3. Send a /chat/completions request to gpt-5.5
    4. Delete the team
    5. Check whether /team/info finds the team
    6. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  549. ✓test_delete_removes_from_model_info_catalog TestModelRoutestest_management_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-model-2c13c65a4ab1 that calls openai/gpt-5.5
    2. List the deployments from /model/info
    3. Delete the deployment
    4. List the deployments from /model/info
    View test on GitHub ↗
  550. ✓test_delete_removes_from_organization_info TestOrganizationRoutestest_management_e2e.py›
    • management
    • organization_management
    1. Create an organization with organization alias: e2e-mgmt-org-cb84473214db
    2. Check whether /organization/info finds the organization
    3. Delete the organization
    4. Check whether /organization/info finds the organization
    View test on GitHub ↗
  551. ✓test_delete_removes_the_customer TestCustomerManagementtest_budget_customer_user_org_e2e.py›
    • management
    • customer_management
    1. POST /customer/new
    2. GET /customer/info
    3. POST /customer/delete
    4. Probe /customer/info
    View test on GitHub ↗
  552. ✓test_delete_removes_the_server_from_every_replica TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_91d3a7149b1b
    2. Delete the MCP server
    3. Check that /v1/mcp/server/53b43344-a6a1-4f18-8eed-a0e150a391c8 returns 404 on every proxy replica
    4. Read /v1/mcp/server on every proxy replica until it settles
    View test on GitHub ↗
  553. ✓test_delete_removes_the_toolset_from_every_replica TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_82be05935e94
    2. Create an MCP toolset with the tools server id: 76417960-d10e-4cf7-9e2e-41ba2376a955 and tool name: search_datadog_logs, server id: 76417960-d10e-4cf7-9e2e-41ba2376a955 and tool name: get_datadog_metr
    3. Delete the MCP toolset
    4. Check that /v1/mcp/toolset/e4205108-5e3d-4f05-9018-29ffaca314f9 returns 404 on every proxy replica
    5. Read /v1/mcp/toolset on every proxy replica until it settles
    View test on GitHub ↗
  554. ✓test_delete_removes_the_user_from_inventory TestUserRoutestest_management_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-mgmt-bf4146dd0944@example.com and user role: internal_user
    2. Count the matching users in /user/list
    3. Delete the internal user
    4. Count the matching users in /user/list
    View test on GitHub ↗
  555. ✓test_delete_revokes_info_and_chat_on_every_replica TestKeyLifecycletest_key_lifecycle_e2e.py›
    • management
    • key_management
    • nonstream
    • openai
    • gpt-4o-mini
    1. Create a team with team alias: e2e-key-lifecycle-team-2c7bed3f5c3f
    2. Generate a virtual key with models: e2e-key-lifecycle-9d02cbc562be, max budget: 25.0, budget duration: 30d, team id: f69d6d44-5d53-4888-a6c7-174dc30c9369, key alias: e2e-key-lifecycle-2c7bed3f5c3f, tp
    3. Send a request to /chat/completions
    4. Delete the virtual key
    5. Read /key/info on every proxy replica until they all agree
    6. Send a request to /chat/completions
    View test on GitHub ↗
  556. ✓test_delete_revokes_the_key_on_chat TestKeyRoutestest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with models: gemini-2.5-flash
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Delete the virtual key
    4. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  557. ✓test_editing_a_key_from_the_dashboard_persists_and_is_enforced TestDashboardKeyRoutestest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • gemini
    • openai
    • gemini-2.5-flash
    • gpt-5.5
    1. Generate a virtual key with models: gemini-2.5-flash, key alias: e2e-mgmt-uiedit-fab2a53382e2, tpm limit: 100 and rpm limit: 200
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Send a /chat/completions request to gpt-5.5
    4. Sign in to the Admin UI with /v2/login
    5. Update the virtual key's settings with /key/update
    6. Read the key's settings back from /key/info
    7. Send a /chat/completions request to gpt-5.5
    8. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  558. ✓test_end_user_new_persists_to_end_user_info TestCustomerManagementtest_budget_customer_user_org_e2e.py›
    • management
    • customer_management
    1. POST /end_user/new
    2. GET /end_user/info
    View test on GitHub ↗
  559. ✓test_estimate_computes_cost_from_token_counts TestCostEstimatetest_config_misc_endpoints_e2e.py›
    • cost-map
    1. POST /cost/estimate
    View test on GitHub ↗
  560. ✓test_explicit_null_clears_the_budget_and_its_reset_time TestKeyLifecycletest_key_lifecycle_e2e.py›
    • management
    • key_management
    1. Create a team with team alias: e2e-key-lifecycle-team-d29420fefdd5
    2. Generate a virtual key with models: e2e-key-lifecycle-9d02cbc562be, max budget: 25.0, budget duration: 30d, team id: 33e8bcf8-f063-4c43-91b9-151ae19ef4e4, key alias: e2e-key-lifecycle-d29420fefdd5, tp
    3. Read /key/info on every proxy replica until they all agree
    4. Update the virtual key's settings with /key/update
    5. Read /key/info on every proxy replica until they all agree
    View test on GitHub ↗
  561. ✓test_gdpr_check_derives_verdict_from_the_request TestComplianceRoutestest_config_misc_endpoints_e2e.py›
    • management
    1. POST /compliance/gdpr
    View test on GitHub ↗
  562. ✓test_generate_forbidden_for_non_admin_key TestKeyManagementRoutestest_key_management_e2e.py›
    • proxy-auth
    • key_management
    1. Generate a virtual key with models: gpt-5.5
    2. Send a request to /key/generate
    View test on GitHub ↗
  563. ✓test_generate_persists_to_key_info_and_scopes_chat TestKeyRoutestest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with models: gemini-2.5-flash, key alias: e2e-mgmt-key-73594514bfb6, tpm limit: 424242 and rpm limit: 424243
    2. Read the key's settings back from /key/info
    3. Send a /chat/completions request to gemini-2.5-flash
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  564. ✓test_health_reports_the_calling_key_healthy TestKeyManagementRoutestest_key_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with models: gpt-5.5
    2. POST /key/health
    View test on GitHub ↗
  565. ✓test_info_reflects_the_fields_the_key_was_created_with TestKeyManagementRoutestest_key_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with models: gpt-5.5, gemini-2.5-flash, key alias: e2e-mgmt-keyinfo-2d71246a70e5, tpm limit: 131313 and rpm limit: 141414
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  566. ✓test_info_returns_created_team_fields TestTeamManagementRoutestest_team_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-team-info-a235de04d5ed and models: gemini-2.5-flash
    2. GET /team/info
    View test on GitHub ↗
  567. ✓test_is_null_finds_the_budget_left_uncapped TestBudgetListV1test_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. POST /budget/new
    2. GET /management/v1/budgets
    View test on GitHub ↗
  568. ✓test_is_refused_for_a_non_admin_key TestBudgetListV1test_budget_customer_user_org_e2e.py›
    • proxy-auth
    • budget_management
    1. Generate a virtual key with default settings
    2. GET /management/v1/budgets
    View test on GitHub ↗
  569. ✓test_key_delete_by_alias_writes_audit_row TestKeyDeletionAuditLogtest_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with key alias: e2e-audit-a528479245d9
    2. Delete the virtual key with the alias e2e-audit-a528479245d9
    3. Read the key's settings back from /key/info
    4. Read the key's deletion entries from the /audit log
    View test on GitHub ↗
  570. ✓test_key_delete_by_key_writes_audit_row TestKeyDeletionAuditLogtest_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with key alias: e2e-audit-44a0d993f78a
    2. Delete the virtual key
    3. Read the key's settings back from /key/info
    4. Read the key's deletion entries from the /audit log
    View test on GitHub ↗
  571. ✓test_key_serves_its_model_and_is_denied_others TestKeyLifecycletest_key_lifecycle_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-4o-mini
    1. Create a team with team alias: e2e-key-lifecycle-team-c9fc1db4f221
    2. Generate a virtual key with models: e2e-key-lifecycle-9d02cbc562be, max budget: 25.0, budget duration: 30d, team id: f8101425-5f2b-4439-95c1-00f4141b79af, key alias: e2e-key-lifecycle-c9fc1db4f221, tp
    3. Send a request to /chat/completions
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  572. ✓test_llm_only_key_forbidden_from_management_writes TestManagementRoutePermissionstest_management_e2e.py›
    • proxy-auth
    1. Generate a virtual key limited to the LLM API routes
    2. Try to generate a virtual key with key alias: e2e-mgmt-forbidden-key-09dbee923f2c, calling with a virtual key
    3. Try to create a team with team alias: e2e-mgmt-forbidden-team-09dbee923f2c and team id: e2e-mgmt-forbidden-team-09dbee923f2c, calling with a virtual key
    4. Try to create an internal user with user email: e2e-mgmt-forbidden-user-09dbee923f2c@example.com, user role: internal_user and user id: e2e-mgmt-forbidden-user-09dbee923f2c, calling with a virtual key
    5. Count the keys with the alias e2e-mgmt-forbidden-key-09dbee923f2c in /key/list
    6. Check whether /team/info finds the team
    7. Count the matching users in /user/list
    View test on GitHub ↗
  573. ✓test_member_add_and_delete_persist_to_team_info TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    1. Create an internal user with user email: e2e-mgmt-c37075aa7ce9@example.com and user role: internal_user
    2. Create a team with team alias: e2e-mgmt-team-17792ede5ee6 and models: gemini-2.5-flash
    3. Add a user to the team with /team/member_add
    4. Read the team back from /team/info
    5. Remove a user from the team with /team/member_delete
    6. Read the team back from /team/info
    View test on GitHub ↗
  574. ✓test_member_add_forbidden_to_plain_member TestTeamManagementRoutestest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-fb-e832bb40d219@example.com and user role: internal_user
    2. Create an internal user with user email: e2e-team-fb-4d8d3015d267@example.com and user role: internal_user
    3. Create a team with team alias: e2e-team-fb-4b41de3ef0d8 and models: gemini-2.5-flash
    4. Add a user to the team with /team/member_add
    5. Generate a virtual key with user id: 9d8dbc03-f41b-48ee-b1c0-12ee81c0cd99 and team id: bc9d6a56-6b93-40da-9f32-2996a909a3a1
    6. Send a request to /team/member_add
    View test on GitHub ↗
  575. ✓test_member_add_records_membership TestOrganizationMembershiptest_budget_customer_user_org_e2e.py›
    • management
    • organization_management
    1. Create an organization with organization alias: e2e-mgmt-org-45d9a609ea8d
    2. Create an internal user with user email: e2e-mgmt-155b0e8a1eac@example.com and user role: internal_user
    3. POST /organization/member_add
    4. GET /organization/info
    View test on GitHub ↗
  576. ✓test_member_cannot_write_and_another_team_cannot_read_the_key TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • key_management
    1. Create the Keycloak group e2e-jwt-team-51cff6b1de37 and the user e2e-jwt-user-51cff6b1de37 in it
    2. POST /user/new
    3. Create a team with team alias: e2e-jwt-51cff6b1de37 and team id: e2e-jwt-team-51cff6b1de37
    4. Get a Keycloak access token for e2e-jwt-user-51cff6b1de37 from the client litellm-e2e-admin
    5. Get a Keycloak access token for e2e-jwt-user-51cff6b1de37 from the client litellm-e2e-tests
    6. Generate a virtual key with team id: e2e-jwt-team-51cff6b1de37 and key alias: e2e-jwt-owned-dfe8741647a5
    7. Add a user to the team with /team/member_add
    8. Read the key's settings back from /key/info
    9. Update the virtual key's settings with /key/update
    10. Read the key's settings back from /key/info
    11. Create the Keycloak group e2e-jwt-team-2a711b1be4bb and the user e2e-jwt-user-2a711b1be4bb in it
    12. Create an internal user with user email: e2e-jwt-user-2a711b1be4bb@example.com, user role: internal_user and user id: 80abe067-d81c-4778-beba-ee9d9076e870
    13. Create a team with team alias: 2a711b1be4bb and team id: e2e-jwt-team-2a711b1be4bb
    14. Add a user to the team with /team/member_add
    15. Get a Keycloak access token for e2e-jwt-user-2a711b1be4bb from the client litellm-e2e-tests
    16. Read the key's settings back from /key/info
    View test on GitHub ↗
  577. ✓test_member_delete_forbidden_to_plain_member TestTeamManagementRoutestest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-fb-e278f7d847d5@example.com and user role: internal_user
    2. Create an internal user with user email: e2e-team-fb-f7337e79fdd8@example.com and user role: internal_user
    3. Create a team with team alias: e2e-team-fb-7d643dc431d6 and models: gemini-2.5-flash
    4. Add a user to the team with /team/member_add
    5. Generate a virtual key with user id: 780b4b82-3384-4662-b9b9-8f237bb476c6 and team id: 102f60ea-d784-4a81-940e-3fe2ba1450d0
    6. Send a request to /team/member_delete
    View test on GitHub ↗
  578. ✓test_member_delete_persists_to_team_info TestTeamManagementRoutestest_team_management_e2e.py›
    • management
    • team_management
    1. Create an internal user with user email: e2e-team-md-627c5943ec1e@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-md-6439b83f8e38 and models: gemini-2.5-flash
    3. Add a user to the team with /team/member_add
    4. GET /team/info
    5. Remove a user from the team with /team/member_delete
    6. GET /team/info
    View test on GitHub ↗
  579. ✓test_member_update_persists_role_and_budget TestTeamManagementRoutestest_team_management_e2e.py›
    • management
    • team_management
    1. Create an internal user with user email: e2e-team-mu-d9fa28d53d49@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-mu-11c3d6bde7f3 and models: gemini-2.5-flash
    3. Add a user to the team with /team/member_add
    4. GET /team/info
    5. POST /team/member_update
    6. GET /team/info
    View test on GitHub ↗
  580. ✓test_multi_group_actor_keeps_exact_memberships TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the team_member role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the team_member role
    4. Read the team back from /team/info
    View test on GitHub ↗
  581. ✓test_new_access_group_tags_the_deployment TestModelAccessGroupRoutestest_model_tag_accessgroup_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-agmodel-c61df01ada30 that calls openai/gpt-5.5
    2. POST /access_group/new
    3. GET /access_group/e2e-mgmt-ag-5d00b2d36eea/info
    View test on GitHub ↗
  582. ✓test_new_is_denied_to_non_admin_keys TestTeamManagementRoutestest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Generate a virtual key with default settings
    2. Create an internal user with user email: e2e-team-adm-e5ddbb5594ae@example.com and user role: internal_user
    3. Generate a virtual key with user id: 8d06c2a9-b6d8-4886-87e3-c00ddcdfcef0
    4. Try to create a team with team alias: e2e-team-adm-d125e7c3425d, calling with a virtual key
    5. Try to create a team with team alias: e2e-team-adm-1420f39b18a2, calling with a virtual key
    View test on GitHub ↗
  583. ✓test_new_is_refused_for_a_non_admin_key TestBudgetManagementtest_budget_customer_user_org_e2e.py›
    • proxy-auth
    • budget_management
    1. Generate a virtual key with default settings
    2. Send a request to /budget/new
    View test on GitHub ↗
  584. ✓test_new_persists_mapping_and_is_read_back TestJwtKeyMappingtest_config_misc_endpoints_e2e.py›
    • proxy-auth
    1. Generate a virtual key with default settings
    2. POST /jwt/key/mapping/new
    3. GET /jwt/key/mapping/info
    View test on GitHub ↗
  585. ✓test_new_persists_to_customer_info TestCustomerManagementtest_budget_customer_user_org_e2e.py›
    • management
    • customer_management
    1. POST /customer/new
    2. GET /customer/info
    View test on GitHub ↗
  586. ✓test_new_persists_to_model_info_catalog TestModelRoutestest_management_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-model-57fe2b72a5a3 that calls openai/gpt-5.5
    2. List the deployments from /model/info
    View test on GitHub ↗
  587. ✓test_new_persists_to_organization_info TestOrganizationRoutestest_management_e2e.py›
    • management
    • organization_management
    1. Create an organization with organization alias: e2e-mgmt-org-db970413ab45 and models: gemini-2.5-flash
    2. Read the organization back from /organization/info
    View test on GitHub ↗
  588. ✓test_new_persists_to_tag_list TestTagRoutestest_management_e2e.py›
    • management
    • tag_management
    1. List the tags from /tag/list
    2. Create a tag with name: e2e-mgmt-tag-8c831c86b055 and description: Tag for spend categorization
    3. List the tags from /tag/list
    View test on GitHub ↗
  589. ✓test_new_persists_to_team_info_and_binds_keys TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-mgmt-team-b5806505b9e0 and models: gemini-2.5-flash
    2. Read the team back from /team/info
    3. Generate a virtual key with team id: 3aded2a9-5c14-4d7e-9f04-964d6baae909
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  590. ✓test_new_persists_to_user_info TestUserRoutestest_management_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-mgmt-2465818d8d0e@example.com and user role: internal_user
    2. Read the user back from /user/info
    View test on GitHub ↗
  591. ✓test_new_user_is_readable_via_user_info TestUserManagementtest_budget_customer_user_org_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-mgmt-20c860b4d9ab@example.com and user role: internal_user
    2. Read the user back from /user/info
    View test on GitHub ↗
  592. ✓test_non_admin_key_cannot_add_global_model TestModelRoutestest_model_tag_accessgroup_e2e.py›
    • proxy-auth
    • model_management
    1. Generate a virtual key with default settings
    2. Send a request to /model/new
    3. List the deployments from /model/info
    View test on GitHub ↗
  593. ✓test_oidc_browser_profile_identity_mapping TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    1. Create an actor in the internal_user role, with its identity provider user, internal user and any tenant memberships
    2. Read Keycloak's OpenID Connect discovery document
    3. Register a browser SSO client in Keycloak
    4. Get a Keycloak access token for e2e-jwt-user-ffce62ae70bf through the browser SSO client
    5. Read the signed-in user's profile from Keycloak's userinfo endpoint
    View test on GitHub ↗
  594. ✓test_partial_setup_removes_previously_created_identities[group] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    1. Create the Keycloak group e2e-partial-08e539b10e0d
    2. Delete every resource the test created, newest first
    3. Check that Keycloak's admin API returns 404 for the deleted entry under /groups
    View test on GitHub ↗
  595. ✓test_partial_setup_removes_previously_created_identities[user] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    1. Create the Keycloak group e2e-partial-3b0d98365d5e
    2. Create the Keycloak user e2e-jwt-user-3b0d98365d5e with a password of its own
    3. Create the Keycloak user e2e-jwt-user-3b0d98365d5e
    4. Delete every resource the test created, newest first
    5. Check that Keycloak's admin API returns 404 for the deleted entry under /groups
    6. Check that Keycloak's admin API returns 404 for the deleted entry under /users
    View test on GitHub ↗
  596. ✓test_partial_update_changes_only_the_named_field TestKeyLifecycletest_key_lifecycle_e2e.py›
    • management
    • key_management
    1. Create a team with team alias: e2e-key-lifecycle-team-0eac471bfa39
    2. Generate a virtual key with models: e2e-key-lifecycle-9d02cbc562be, max budget: 25.0, budget duration: 30d, team id: 5ce0aee7-eafe-4a5f-a481-f535e8155889, key alias: e2e-key-lifecycle-0eac471bfa39, tp
    3. Read /key/info on every proxy replica until they all agree
    4. Update the virtual key's settings with /key/update
    5. Read /key/info on every proxy replica until they all agree
    View test on GitHub ↗
  597. ✓test_project_detachment_preserves_key_scope_and_refreshes_auth TestKeyManagementRoutestest_key_management_e2e.py›
    • management
    • key_management
    • nonstream
    • openai
    • openai/synthetic-detachment
    1. Add a deployment named e2e-detach-d014b0d4ed2e that calls openai/synthetic-detachment
    2. Create an organization with organization alias: e2e-detach-d014b0d4ed2e and models: e2e-detach-d014b0d4ed2e
    3. Create a team with team alias: e2e-detach-d014b0d4ed2e, models: e2e-detach-d014b0d4ed2e and organization id: 43bad0db-994d-4d63-8320-4f12e5cb7538
    4. POST /project/new
    5. Generate a virtual key with models: e2e-detach-d014b0d4ed2e, max budget: 5.0, team id: ad05cc68-473a-46a6-90a3-f861e0d6dd89, project id: e9c77f61-d4f1-4e98-b094-fa6465a96730, organization id: 43bad0db
    6. Send a /chat/completions request to e2e-detach-d014b0d4ed2e
    7. Update the virtual key's settings with /key/update
    8. Read the key's settings back from /key/info
    9. Update the virtual key's settings with /key/update
    10. Send a request to /key/update
    11. Read the key's settings back from /key/info
    12. POST /project/update
    13. Send a /chat/completions request to e2e-detach-d014b0d4ed2e
    14. POST /key/update
    15. Read the key's settings back from /key/info
    16. Send a /chat/completions request to e2e-detach-d014b0d4ed2e
    17. Update the virtual key's settings with /key/update
    18. Read the key's settings back from /key/info
    19. Send a /chat/completions request to e2e-detach-d014b0d4ed2e-outside
    View test on GitHub ↗
  598. ✓test_read_reflects_the_create_on_every_replica TestKeyLifecycletest_key_lifecycle_e2e.py›
    • management
    • key_management
    1. Create a team with team alias: e2e-key-lifecycle-team-0ea93b6c1596
    2. Generate a virtual key with models: e2e-key-lifecycle-9d02cbc562be, max budget: 25.0, budget duration: 30d, team id: c053309b-b7f1-484f-a1d8-4acb57549007, key alias: e2e-key-lifecycle-0ea93b6c1596, tp
    3. Read /key/info on every proxy replica until they all agree
    View test on GitHub ↗
  599. ✓test_refuses_a_sort_field_and_a_parameter_it_does_not_support TestBudgetListV1test_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. GET /management/v1/budgets
    View test on GitHub ↗
  600. ✓test_regenerate_rotates_to_a_working_new_key TestKeyRegenerationtest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with models: gpt-5.5
    2. Regenerate the virtual key with /key/regenerate
    3. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  601. ✓test_regenerate_with_grace_period_keeps_old_key_until_revoked TestKeyRegenerationtest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with models: gpt-5.5
    2. Regenerate the virtual key with /key/regenerate
    3. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  602. ✓test_register_submits_pending_server TestMcpServerSubmissiontest_config_misc_endpoints_e2e.py›
    • mcp
    • mcp
    1. Create a team with team alias: e2e-mcp-team-f344b4b4f88f
    2. Generate a virtual key with team id: 61bfdd55-58a7-4895-a6e5-b945b779516b
    3. POST /v1/mcp/server/register
    View test on GitHub ↗
  603. ✓test_reset_spend_zeroes_recorded_spend_and_lifts_the_budget_block TestKeyManagementRoutestest_key_management_e2e.py›
    • spend-budgets
    • key_management
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and max budget: 3e-06
    2. Send a /chat/completions request to claude-haiku-4-5
    3. Read the key's settings back from /key/info
    4. Reset the virtual key's spend to 0.0
    5. Read the key's settings back from /key/info
    6. Send a /chat/completions request to claude-haiku-4-5
    View test on GitHub ↗
  604. ✓test_sorts_pages_and_filters_the_budgets_it_created TestBudgetListV1test_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. POST /budget/new
    2. GET /management/v1/budgets
    View test on GitHub ↗
  605. ✓test_successful_actor_cleanup_removes_owned_state TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the team_member role, with its identity provider user, internal user and any tenant memberships
    3. Generate a virtual key as the proxy admin
    4. Read the key's settings back from /key/info
    5. Delete every resource the test created, newest first
    6. Count the matching users in /user/list
    7. Count the keys with the alias e2e-actor-key-29f0e787a404 in /key/list
    8. Check whether /team/info finds the team
    9. Check whether /organization/info finds the organization
    10. Check that Keycloak's admin API returns 404 for the deleted entry under /users
    11. Check that Keycloak's admin API returns 404 for the deleted entry under /groups
    View test on GitHub ↗
  606. ✓test_tag_delete_removes_from_list TestTagRoutestest_model_tag_accessgroup_e2e.py›
    • management
    • tag_management
    1. POST /tag/new
    2. GET /tag/list
    3. POST /tag/delete
    4. GET /tag/list
    View test on GitHub ↗
  607. ✓test_tag_list_reports_created_tag TestTagRoutestest_model_tag_accessgroup_e2e.py›
    • management
    • tag_management
    1. GET /tag/list
    2. POST /tag/new
    3. GET /tag/list
    View test on GitHub ↗
  608. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[budget_duration] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-00135d2a83d0@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-a3517747c6f3, tpm limit: 1000 and members with roles: role: admin and user id: 8ccfd82b-922e-4ee0-831f-fc76a848ce39
    3. Generate a virtual key with user id: 8ccfd82b-922e-4ee0-831f-fc76a848ce39
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  609. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[max_budget] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-a95c15e84e4a@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-c16bd4c18f0b, tpm limit: 1000 and members with roles: role: admin and user id: 3be4fc57-382c-4589-83ad-a4ba8929e5c5
    3. Generate a virtual key with user id: 3be4fc57-382c-4589-83ad-a4ba8929e5c5
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  610. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[metadata] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-2a7a30e7f1de@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-e3787be74565, tpm limit: 1000 and members with roles: role: admin and user id: e0893266-bf30-478c-bbac-f59a29357734
    3. Generate a virtual key with user id: e0893266-bf30-478c-bbac-f59a29357734
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  611. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[models] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-109f0aba3131@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-2b6a8142b463, tpm limit: 1000 and members with roles: role: admin and user id: 3e2402a1-f7fd-439e-af93-b84a10b73510
    3. Generate a virtual key with user id: 3e2402a1-f7fd-439e-af93-b84a10b73510
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  612. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[rpm_limit] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-c8cdf27b6a9e@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-036a7f7f615c, tpm limit: 1000 and members with roles: role: admin and user id: 01448621-3501-4532-abb5-46248e63305b
    3. Generate a virtual key with user id: 01448621-3501-4532-abb5-46248e63305b
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  613. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[team_alias] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-e51d112255fd@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-01527b0ea185, tpm limit: 1000 and members with roles: role: admin and user id: e70eaebf-7f83-4444-b5b6-3e21dea0a2e2
    3. Generate a virtual key with user id: e70eaebf-7f83-4444-b5b6-3e21dea0a2e2
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  614. ✓test_team_admin_cannot_change_any_team_setting TestTeamAdminWithNoEditableFieldstest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. GET /get/ui_settings
    2. PATCH /update/ui_settings
    3. Wait for the last control-plane write to reach every proxy replica
    4. Create an internal user with user email: e2e-team-admin-36b4d82f0836@example.com and user role: internal_user
    5. Create a team with team alias: e2e-team-admin-d6eae8bb32e6, tpm limit: 1000 and members with roles: role: admin and user id: 00459ae7-23b0-4456-baff-37dce4f7897d
    6. Generate a virtual key with user id: 00459ae7-23b0-4456-baff-37dce4f7897d
    7. GET /team/info
    8. Send a request to /team/update
    9. GET /team/info
    View test on GitHub ↗
  615. ✓test_team_admin_cannot_raise_an_org_team_budget_under_the_org_cap TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an organization with organization alias: e2e-team-admin-org-dd0da2034d09 and max budget: 100.0
    2. Create an internal user with user email: e2e-team-admin-62886fedbe2c@example.com and user role: internal_user
    3. Create a team with team alias: e2e-team-admin-cd1afa7b3574, organization id: 5c45384d-96a5-4f06-9ed8-9b3e3d6142e6, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: dc
    4. Generate a virtual key with user id: dc33770a-f0c6-416b-b5de-93610df9a7dd
    5. GET /team/info
    6. Send a request to /team/update
    7. GET /team/info
    View test on GitHub ↗
  616. ✓test_team_admin_cannot_raise_or_remove_the_budget[raise] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-bc805a5b7959@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-03756d73d81a, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: 47cb4906-4006-4888-86d0-a996066d6e6c
    3. Generate a virtual key with user id: 47cb4906-4006-4888-86d0-a996066d6e6c
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  617. ✓test_team_admin_cannot_raise_or_remove_the_budget[remove] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-d77f82573476@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-ce5d7f5b0fb0, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: 58f52e6d-4fa5-4779-b50f-79d45253fd50
    3. Generate a virtual key with user id: 58f52e6d-4fa5-4779-b50f-79d45253fd50
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  618. ✓test_team_admin_resending_the_budget_settings_keeps_the_next_budget_reset TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • spend-budgets
    • team_management
    1. Create an internal user with user email: e2e-team-admin-abd076d7e5b3@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-fa6f3b847dd7, tpm limit: 1000 and members with roles: role: admin and user id: df8089cf-06aa-4d9d-9224-d9f33857eb17
    3. Generate a virtual key with user id: df8089cf-06aa-4d9d-9224-d9f33857eb17
    4. POST /team/update
    5. GET /team/info
    6. Send a request to /team/update
    7. GET /team/info
    View test on GitHub ↗
  619. ✓test_team_admin_saves_a_new_rpm_limit_and_a_tighter_budget[first-budget] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-bea252928ebf@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-5566041a42e0, tpm limit: 1000 and members with roles: role: admin and user id: 53311fb6-fe24-4bdd-addc-6e201d56a896
    3. Generate a virtual key with user id: 53311fb6-fe24-4bdd-addc-6e201d56a896
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  620. ✓test_team_admin_saves_a_new_rpm_limit_and_a_tighter_budget[lower] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. GET /get/ui_settings
    2. PATCH /update/ui_settings
    3. Wait for the last control-plane write to reach every proxy replica
    4. Create an internal user with user email: e2e-team-admin-39ad8cdbb455@example.com and user role: internal_user
    5. Create a team with team alias: e2e-team-admin-1fa70a70c0af, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: cbe801b1-1dd3-41a6-8608-8c3dd48b950f
    6. Generate a virtual key with user id: cbe801b1-1dd3-41a6-8608-8c3dd48b950f
    7. GET /team/info
    8. Send a request to /team/update
    9. GET /team/info
    View test on GitHub ↗
  621. ✓test_team_admin_saves_the_settings_form_with_a_new_tpm_limit TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. GET /get/ui_settings
    2. PATCH /update/ui_settings
    3. Wait for the last control-plane write to reach every proxy replica
    4. Create an internal user with user email: e2e-team-admin-263c270d765e@example.com and user role: internal_user
    5. Create a team with team alias: e2e-team-admin-f8ba1362e911, tpm limit: 1000 and members with roles: role: admin and user id: a7a776ff-cec7-401e-9779-1ff324439dc2
    6. Generate a virtual key with user id: a7a776ff-cec7-401e-9779-1ff324439dc2
    7. GET /team/info
    8. Send a request to /team/update
    9. GET /team/info
    View test on GitHub ↗
  622. ✓test_team_delete_succeeds_for_team_larger_than_db_pool TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    1. (205 earlier steps not recorded)
    2. Create an internal user with user email: e2e-mgmt-bulk-204-476e404f0647@example.com and user role: internal_user
    3. Create an internal user with user email: e2e-mgmt-bulk-205-40dbdada3fb8@example.com and user role: internal_user
    4. Create an internal user with user email: e2e-mgmt-bulk-206-7f6c8984e4c2@example.com and user role: internal_user
    5. Create an internal user with user email: e2e-mgmt-bulk-207-92306eb18734@example.com and user role: internal_user
    6. Create an internal user with user email: e2e-mgmt-bulk-208-2ad1f429824e@example.com and user role: internal_user
    7. Create an internal user with user email: e2e-mgmt-bulk-209-32731d17420a@example.com and user role: internal_user
    8. Create an internal user with user email: e2e-mgmt-bulk-210-19a28effa2a7@example.com and user role: internal_user
    9. Create an internal user with user email: e2e-mgmt-bulk-211-ccd3ca4a1849@example.com and user role: internal_user
    10. Create an internal user with user email: e2e-mgmt-bulk-212-06dc0edc208d@example.com and user role: internal_user
    11. Create an internal user with user email: e2e-mgmt-bulk-213-2248388ec9c9@example.com and user role: internal_user
    12. Create an internal user with user email: e2e-mgmt-bulk-214-3863d05ae547@example.com and user role: internal_user
    13. Create an internal user with user email: e2e-mgmt-bulk-215-b81f349dd40a@example.com and user role: internal_user
    14. Create an internal user with user email: e2e-mgmt-bulk-216-b60ea6d9a046@example.com and user role: internal_user
    15. Create an internal user with user email: e2e-mgmt-bulk-217-507dfe31fd16@example.com and user role: internal_user
    16. Create an internal user with user email: e2e-mgmt-bulk-218-035387889e6a@example.com and user role: internal_user
    17. Create an internal user with user email: e2e-mgmt-bulk-219-e553678f82f7@example.com and user role: internal_user
    18. Create an internal user with user email: e2e-mgmt-bulk-220-00d6e5fdc023@example.com and user role: internal_user
    19. Create an internal user with user email: e2e-mgmt-bulk-221-144cb14dba81@example.com and user role: internal_user
    20. Create an internal user with user email: e2e-mgmt-bulk-222-f31affbe8747@example.com and user role: internal_user
    21. Create an internal user with user email: e2e-mgmt-bulk-223-a7bcbb1e4f90@example.com and user role: internal_user
    22. Create an internal user with user email: e2e-mgmt-bulk-224-90b81b5b50e7@example.com and user role: internal_user
    23. Create an internal user with user email: e2e-mgmt-bulk-225-b91d7b1cc660@example.com and user role: internal_user
    24. Create an internal user with user email: e2e-mgmt-bulk-226-49bbe68fe3d0@example.com and user role: internal_user
    25. Create an internal user with user email: e2e-mgmt-bulk-227-1fe7d3a0b6ee@example.com and user role: internal_user
    26. Create an internal user with user email: e2e-mgmt-bulk-228-e7761c642f15@example.com and user role: internal_user
    27. Create an internal user with user email: e2e-mgmt-bulk-229-5d9b7247cd06@example.com and user role: internal_user
    28. Create an internal user with user email: e2e-mgmt-bulk-230-bf90b6b64e18@example.com and user role: internal_user
    29. Create an internal user with user email: e2e-mgmt-bulk-231-0073c5ec3fa5@example.com and user role: internal_user
    30. Create an internal user with user email: e2e-mgmt-bulk-232-21b4b2bd789f@example.com and user role: internal_user
    31. Create an internal user with user email: e2e-mgmt-bulk-233-119c4742038a@example.com and user role: internal_user
    32. Create an internal user with user email: e2e-mgmt-bulk-234-80846267ecdb@example.com and user role: internal_user
    33. Create an internal user with user email: e2e-mgmt-bulk-235-00cdc6dc2d19@example.com and user role: internal_user
    34. Create an internal user with user email: e2e-mgmt-bulk-236-2e5aa2e0a357@example.com and user role: internal_user
    35. Create an internal user with user email: e2e-mgmt-bulk-237-4b587e9d7489@example.com and user role: internal_user
    36. Create an internal user with user email: e2e-mgmt-bulk-238-b8882fd2cb1c@example.com and user role: internal_user
    37. Create an internal user with user email: e2e-mgmt-bulk-239-10cfbb07d7a5@example.com and user role: internal_user
    38. Create an internal user with user email: e2e-mgmt-bulk-240-ed0054352b2a@example.com and user role: internal_user
    39. Create an internal user with user email: e2e-mgmt-bulk-241-86b118918cfa@example.com and user role: internal_user
    40. Create an internal user with user email: e2e-mgmt-bulk-242-8e98b7651b82@example.com and user role: internal_user
    41. Create an internal user with user email: e2e-mgmt-bulk-243-7bbbedb69e89@example.com and user role: internal_user
    42. Create an internal user with user email: e2e-mgmt-bulk-244-b1b923398988@example.com and user role: internal_user
    43. Create an internal user with user email: e2e-mgmt-bulk-245-b14c875ac023@example.com and user role: internal_user
    44. Create an internal user with user email: e2e-mgmt-bulk-246-6ab648fb00cf@example.com and user role: internal_user
    45. Create an internal user with user email: e2e-mgmt-bulk-247-cf3b89df31ad@example.com and user role: internal_user
    46. Create an internal user with user email: e2e-mgmt-bulk-248-68b5d4ce7134@example.com and user role: internal_user
    47. Create an internal user with user email: e2e-mgmt-bulk-249-8753058afde2@example.com and user role: internal_user
    48. Add a roster of members to the team with /team/member_add
    49. Read the team back from /team/info
    50. Try to delete the team with /team/delete
    51. Check whether /team/info finds the team
    View test on GitHub ↗
  623. ✓test_team_delete_writes_audit_row_for_team_keys TestKeyDeletionAuditLogtest_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-audit-team-2d6e30b86a5b
    2. Generate a virtual key with team id: 38c34d17-d9de-444c-8bb6-aaa4a09f6e67
    3. Delete the team
    4. Read the key's settings back from /key/info
    5. Read the key's deletion entries from the /audit log
    View test on GitHub ↗
  624. ✓test_team_member_delete_writes_audit_row_for_member_keys TestKeyDeletionAuditLogtest_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-audit-team-f4eaff1b345f
    2. Create an internal user with user email: e2e-audit-a6be77a422e9@example.com and user role: internal_user
    3. Add a user to the team with /team/member_add
    4. Generate a virtual key with user id: 278304c0-d18b-4867-b70e-d82ea87690f4 and team id: fe46216c-6a2d-4bed-8057-7717b7105eb8
    5. Remove a user from the team with /team/member_delete
    6. Read the key's settings back from /key/info
    7. Read the key's deletion entries from the /audit log
    View test on GitHub ↗
  625. ✓test_tool_list_returns_catalog_with_consistent_total TestInventoryRoutestest_config_misc_endpoints_e2e.py›
    • management
    1. GET /v1/tool/list
    View test on GitHub ↗
  626. ✓test_two_actor_sets_keep_tenants_and_keys_isolated TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • key_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the team_member role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the team_member role
    4. Generate a virtual key as the proxy admin
    5. Read the key's settings back from /key/info
    View test on GitHub ↗
  627. ✓test_unblock_flips_key_info_blocked_back TestKeyManagementRoutestest_key_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with models: gpt-5.5
    2. POST /key/block
    3. Read the key's settings back from /key/info
    4. POST /key/unblock
    5. Read the key's settings back from /key/info
    View test on GitHub ↗
  628. ✓test_update_accepts_per_model_budgets_including_punctuated_names TestBudgetManagementtest_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. POST /budget/new
    2. POST /budget/update
    3. POST /budget/info
    4. POST /budget/new
    5. POST /budget/update
    6. POST /budget/info
    View test on GitHub ↗
  629. ✓test_update_alias_persists_to_organization_info TestOrganizationRoutestest_management_e2e.py›
    • management
    • organization_management
    1. Create an organization with organization alias: e2e-mgmt-org-619c11a01746
    2. Update an organization with organization id: 0540d67f-553e-492f-80d2-b42d013d51e5 and organization alias: e2e-mgmt-org-78fc14a2c591
    3. Read the organization back from /organization/info
    View test on GitHub ↗
  630. ✓test_update_forbidden_for_non_admin_key TestKeyManagementRoutestest_key_management_e2e.py›
    • proxy-auth
    • key_management
    1. Generate a virtual key with models: gpt-5.5
    2. Send a request to /key/update
    3. Read the key's settings back from /key/info
    View test on GitHub ↗
  631. ✓test_update_max_budget_persists_to_budget_info TestBudgetManagementtest_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. POST /budget/new
    2. POST /budget/info
    3. POST /budget/update
    4. POST /budget/info
    View test on GitHub ↗
  632. ✓test_update_models_persists_and_flips_enforcement TestKeyRoutestest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • gemini
    • openai
    • gemini-2.5-flash
    • gpt-5.5
    1. Generate a virtual key with models: gemini-2.5-flash
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Send a /chat/completions request to gpt-5.5
    4. Set the virtual key's models to [gpt-5.5]
    5. Read the key's settings back from /key/info
    6. Send a /chat/completions request to gpt-5.5
    7. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  633. ✓test_update_persists_input_cost_to_model_info TestModelRoutestest_management_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-model-ec13418a632e that calls gpt-4o-mini
    2. List the deployments from /model/info
    3. Update a deployment's settings to model: gpt-4o-mini and input cost per token: 2.22e-06
    4. List the deployments from /model/info
    View test on GitHub ↗
  634. ✓test_update_persists_to_team_info TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-mgmt-team-236ced83613c and models: gemini-2.5-flash
    2. Update a team with team id: ddd4e1f1-29fb-457c-8e8b-1fd32f082dbb and team alias: e2e-mgmt-team-updated-dab37b73ed44
    3. Read the team back from /team/info
    View test on GitHub ↗
  635. ✓test_update_persists_to_user_info TestUserRoutestest_management_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-mgmt-b9672475fbd2@example.com and user role: internal_user
    2. Read the user back from /user/info
    3. Update an internal user with user id: aee2121a-8f4c-4f35-a4f5-953e6ea79dd8 and user role: internal_user_viewer
    4. Read the user back from /user/info
    View test on GitHub ↗
  636. ✓test_updating_only_the_alias_keeps_every_other_field_on_every_replica TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_cbdb2bb97244
    2. Update the MCP server's settings with PUT /v1/mcp/server
    3. Read /v1/mcp/server/195dc914-b8ad-412c-b3bf-8b121e3585ec on every proxy replica until it settles
    View test on GitHub ↗
  637. ✓test_updating_only_the_description_keeps_the_tools_and_name TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_c7a57632b261
    2. Create an MCP toolset with the tools server id: 2edca686-0684-4c98-83f9-8d9838af99a7 and tool name: search_datadog_logs, server id: 2edca686-0684-4c98-83f9-8d9838af99a7 and tool name: get_datadog_metr
    3. Update an MCP toolset with toolset id: adc8db7a-f574-4357-b774-3fc35956edc5 and description: edited
    4. Read /v1/mcp/toolset/adc8db7a-f574-4357-b774-3fc35956edc5 on every proxy replica until it settles
    View test on GitHub ↗
  638. ✓test_updating_the_tools_to_one_entry_reads_back_exactly_that_entry TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_7679033f2f8a
    2. Create an MCP toolset with the tools server id: 5922509a-0e2e-4f28-a5e5-06eab4063da3 and tool name: search_datadog_logs, server id: 5922509a-0e2e-4f28-a5e5-06eab4063da3 and tool name: get_datadog_metr
    3. Update an MCP toolset with toolset id: f7c73cd4-0d0f-4490-be74-a0a867fc1bfd and tools: server id: 5922509a-0e2e-4f28-a5e5-06eab4063da3 and tool name: search_datadog_logs
    4. Read /v1/mcp/toolset/f7c73cd4-0d0f-4490-be74-a0a867fc1bfd on every proxy replica until it settles
    View test on GitHub ↗
  639. ✓test_user_delete_writes_audit_row_for_user_keys TestKeyDeletionAuditLogtest_management_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-audit-7ab5cc4723f8@example.com and user role: internal_user
    2. Generate a virtual key with user id: 8f4e3034-6088-4b3e-b597-7832d5e1d4ce
    3. Delete the internal user
    4. Read the key's settings back from /key/info
    5. Read the key's deletion entries from the /audit log
    View test on GitHub ↗
  640. ✓test_workflow_runs_list_returns_consistent_count TestInventoryRoutestest_config_misc_endpoints_e2e.py›
    • management
    1. GET /v1/workflows/runs
    View test on GitHub ↗
  641. −test_created_server_is_listed_with_every_field TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp

    No steps recorded

    product gap: GET /v1/mcp/server builds each row from the in-memory registry, whose _build_mcp_server_table sets description from mcp_info['description'], so the list reports the mcp_info description while GET /v1/mcp/server/{server_id} reports the stored description column. A server created with both set to different text reads back with two different descriptions depending on the route

    View test on GitHub ↗
  642. ✓test_basic_messaging_non_streaming_anthropicbasic_messaging_non_streaming/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 through the proxy and check every model replies
    26. Run Claude Code headless against claude-haiku-4-5 through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    28. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  643. ✓test_basic_messaging_non_streaming_azurebasic_messaging_non_streaming/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure through the proxy and check every model replies
    26. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    28. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  644. ✓test_basic_messaging_non_streaming_azure_openaibasic_messaging_non_streaming/test_azure_openai.py›
    • llm-translation
    • messages
    • nonstream
    • azure
    • gpt-5-6-luna-azure-openai
    • gpt-5-6-sol-azure-openai
    • gpt-5-6-terra-azure-openai
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against gpt-5-6-sol-azure-openai, gpt-5-6-terra-azure-openai, gpt-5-6-luna-azure-openai through the proxy and check every model replies
    26. Run Claude Code headless against gpt-5-6-sol-azure-openai through the proxy
    27. Run Claude Code headless against gpt-5-6-terra-azure-openai through the proxy
    28. Run Claude Code headless against gpt-5-6-luna-azure-openai through the proxy
    View test on GitHub ↗
  645. ✓test_basic_messaging_non_streaming_bedrock_conversebasic_messaging_non_streaming/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse through the proxy and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  646. ✓test_basic_messaging_non_streaming_bedrock_invokebasic_messaging_non_streaming/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke through the proxy and check every model replies
    26. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    28. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  647. ✓test_basic_messaging_non_streaming_vertex_aibasic_messaging_non_streaming/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex through the proxy and check every model replies
    26. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    28. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  648. ✓test_basic_messaging_non_streaming_vertex_ai_gptbasic_messaging_non_streaming/test_vertex_ai_gpt.py›
    • llm-translation
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    View test on GitHub ↗
  649. ✓test_basic_messaging_streaming_anthropicbasic_messaging_streaming/test_anthropic.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 through the proxy and check every model replies
    26. Run Claude Code headless against claude-haiku-4-5 through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    28. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  650. ✓test_basic_messaging_streaming_azurebasic_messaging_streaming/test_azure.py›
    • llm-translation
    • messages
    • stream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure through the proxy and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  651. ✓test_basic_messaging_streaming_azure_openaibasic_messaging_streaming/test_azure_openai.py›
    • llm-translation
    • messages
    • stream
    • azure
    • gpt-5-6-luna-azure-openai
    • gpt-5-6-sol-azure-openai
    • gpt-5-6-terra-azure-openai
    1. Run Claude Code headless against gpt-5-6-sol-azure-openai, gpt-5-6-terra-azure-openai, gpt-5-6-luna-azure-openai through the proxy and check every model replies
    2. Run Claude Code headless against gpt-5-6-sol-azure-openai through the proxy
    3. Run Claude Code headless against gpt-5-6-terra-azure-openai through the proxy
    4. Run Claude Code headless against gpt-5-6-luna-azure-openai through the proxy
    View test on GitHub ↗
  652. ✓test_basic_messaging_streaming_bedrock_conversebasic_messaging_streaming/test_bedrock_converse.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse through the proxy and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  653. ✓test_basic_messaging_streaming_bedrock_invokebasic_messaging_streaming/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke through the proxy and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  654. ✓test_basic_messaging_streaming_vertex_aibasic_messaging_streaming/test_vertex_ai.py›
    • llm-translation
    • messages
    • stream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex through the proxy and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  655. ✓test_basic_messaging_streaming_vertex_ai_gptbasic_messaging_streaming/test_vertex_ai_gpt.py›
    • llm-translation

    No steps recorded

    View test on GitHub ↗
  656. ✓test_count_tokens_anthropiccount_tokens/test_anthropic.py›
    • llm-translation
    • count_tokens
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5 on the message "hello world"
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5 on the message "hello world"
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7 on the message "hello world"
    View test on GitHub ↗
  657. ✓test_count_tokens_azurecount_tokens/test_azure.py›
    • llm-translation
    • count_tokens
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5-azure on the message "hello world"
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5-azure on the message "hello world"
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7-azure on the message "hello world"
    View test on GitHub ↗
  658. ✓test_count_tokens_bedrock_conversecount_tokens/test_bedrock_converse.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5-bedrock-converse on the message "hello world"
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5-bedrock-converse on the message "hello world"
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7-bedrock-converse on the message "hello world"
    View test on GitHub ↗
  659. ✓test_count_tokens_bedrock_invokecount_tokens/test_bedrock_invoke.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5-bedrock-invoke on the message "hello world"
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5-bedrock-invoke on the message "hello world"
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7-bedrock-invoke on the message "hello world"
    View test on GitHub ↗
  660. ✓test_passthrough_anthropicpassthrough/test_anthropic.py›
    • passthrough
    • passthrough
    • stream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 through the proxy's native provider passthrough route and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  661. ✓test_passthrough_bedrock_conversepassthrough/test_bedrock_converse.py›
    • passthrough

    No steps recorded

    View test on GitHub ↗
  662. ✓test_passthrough_bedrock_invokepassthrough/test_bedrock_invoke.py›
    • passthrough
    • passthrough
    • stream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke through the proxy's native provider passthrough route and check every
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  663. ✓test_passthrough_vertex_aipassthrough/test_vertex_ai.py›
    • passthrough
    • passthrough
    • stream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex through the proxy's native provider passthrough route and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  664. ✓test_pdf_input_anthropicpdf_input/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • pdf_input
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  665. ✓test_pdf_input_azurepdf_input/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • pdf_input
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    26. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    28. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  666. ✓test_pdf_input_bedrock_conversepdf_input/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • pdf_input
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  667. ✓test_pdf_input_bedrock_invokepdf_input/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • pdf_input
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  668. ✓test_pdf_input_vertex_aipdf_input/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • pdf_input
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  669. ✓test_prompt_caching_1h_anthropicprompt_caching_1h/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  670. ✓test_prompt_caching_1h_azureprompt_caching_1h/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  671. ✓test_prompt_caching_1h_bedrock_converseprompt_caching_1h/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  672. ✓test_prompt_caching_1h_bedrock_invokeprompt_caching_1h/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  673. ✓test_prompt_caching_1h_vertex_aiprompt_caching_1h/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  674. ✓test_prompt_caching_5m_anthropicprompt_caching_5m/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  675. ✓test_prompt_caching_5m_azureprompt_caching_5m/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  676. ✓test_prompt_caching_5m_bedrock_converseprompt_caching_5m/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • prompt_caching
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    26. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    28. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  677. ✓test_prompt_caching_5m_bedrock_invokeprompt_caching_5m/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  678. ✓test_prompt_caching_5m_vertex_aiprompt_caching_5m/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  679. ✓test_structured_outputs_anthropicstructured_outputs/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • response_schema
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  680. ✓test_structured_outputs_azurestructured_outputs/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • response_schema
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  681. ✓test_structured_outputs_bedrock_conversestructured_outputs/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • response_schema
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  682. ✓test_structured_outputs_bedrock_invokestructured_outputs/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • response_schema
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  683. ✓test_structured_outputs_vertex_aistructured_outputs/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • response_schema
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  684. ✓test_thinking_anthropicthinking/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  685. ✓test_thinking_azurethinking/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  686. ✓test_thinking_bedrock_conversethinking/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  687. ✓test_thinking_bedrock_invokethinking/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  688. ✓test_thinking_vertex_aithinking/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  689. ✓test_thinking_with_tool_use_anthropicthinking_with_tool_use/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • function_calling
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  690. ✓test_thinking_with_tool_use_azurethinking_with_tool_use/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • function_calling
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  691. ✓test_thinking_with_tool_use_bedrock_conversethinking_with_tool_use/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • function_calling
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  692. ✓test_thinking_with_tool_use_bedrock_invokethinking_with_tool_use/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • function_calling
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  693. ✓test_thinking_with_tool_use_vertex_aithinking_with_tool_use/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • function_calling
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  694. ✓test_tool_search_anthropictool_search/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • tool_search
    1. Send a /v1/messages request to claude-haiku-4-5 with the tool_search tool declared
    2. Send a /v1/messages request to claude-sonnet-4-5 with the tool_search tool declared
    3. Send a /v1/messages request to claude-opus-4-7 with the tool_search tool declared
    View test on GitHub ↗
  695. ✓test_tool_search_bedrock_conversetool_search/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • tool_search
    1. Send a /v1/messages request to claude-haiku-4-5-bedrock-converse with the tool_search tool declared
    2. Send a /v1/messages request to claude-sonnet-4-5-bedrock-converse with the tool_search tool declared
    3. Send a /v1/messages request to claude-opus-4-7-bedrock-converse with the tool_search tool declared
    View test on GitHub ↗
  696. ✓test_tool_search_bedrock_invoketool_search/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • tool_search
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Send a /v1/messages request to claude-haiku-4-5-bedrock-invoke with the tool_search tool declared
    26. Send a /v1/messages request to claude-sonnet-4-5-bedrock-invoke with the tool_search tool declared
    27. Send a /v1/messages request to claude-opus-4-7-bedrock-invoke with the tool_search tool declared
    View test on GitHub ↗
  697. ✓test_tool_search_history_bedrock_invoketool_search/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • tool_search
    1. Send a /v1/messages request to claude-haiku-4-5-bedrock-invoke with the tool_search tool declared, then send its answer back as history in a second request
    2. Send a /v1/messages request to claude-sonnet-4-5-bedrock-invoke with the tool_search tool declared, then send its answer back as history in a second request
    3. Send a /v1/messages request to claude-opus-4-7-bedrock-invoke with the tool_search tool declared, then send its answer back as history in a second request
    View test on GitHub ↗
  698. ✓test_tool_use_anthropictool_use/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  699. ✓test_tool_use_azuretool_use/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  700. ✓test_tool_use_azure_openaitool_use/test_azure_openai.py›
    • llm-translation
    • messages
    • nonstream
    • azure
    • gpt-5-6-luna-azure-openai
    • gpt-5-6-sol-azure-openai
    • gpt-5-6-terra-azure-openai
    • function_calling
    1. Run Claude Code headless against gpt-5-6-sol-azure-openai, gpt-5-6-terra-azure-openai, gpt-5-6-luna-azure-openai in parallel through the proxy
    2. Run Claude Code headless against gpt-5-6-sol-azure-openai through the proxy
    3. Run Claude Code headless against gpt-5-6-terra-azure-openai through the proxy
    4. Run Claude Code headless against gpt-5-6-luna-azure-openai through the proxy
    View test on GitHub ↗
  701. ✓test_tool_use_bedrock_conversetool_use/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  702. ✓test_tool_use_bedrock_invoketool_use/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  703. ✓test_tool_use_streaming_anthropictool_use_streaming/test_anthropic.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  704. ✓test_tool_use_streaming_azuretool_use_streaming/test_azure.py›
    • llm-translation
    • messages
    • stream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  705. ✓test_tool_use_streaming_azure_openaitool_use_streaming/test_azure_openai.py›
    • llm-translation
    • messages
    • stream
    • azure
    • gpt-5-6-luna-azure-openai
    • gpt-5-6-sol-azure-openai
    • gpt-5-6-terra-azure-openai
    • function_calling
    1. Run Claude Code headless against gpt-5-6-sol-azure-openai, gpt-5-6-terra-azure-openai, gpt-5-6-luna-azure-openai in parallel through the proxy
    2. Run Claude Code headless against gpt-5-6-sol-azure-openai through the proxy
    3. Run Claude Code headless against gpt-5-6-terra-azure-openai through the proxy
    4. Run Claude Code headless against gpt-5-6-luna-azure-openai through the proxy
    View test on GitHub ↗
  706. ✓test_tool_use_streaming_bedrock_conversetool_use_streaming/test_bedrock_converse.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  707. ✓test_tool_use_streaming_bedrock_invoketool_use_streaming/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  708. ✓test_tool_use_streaming_vertex_aitool_use_streaming/test_vertex_ai.py›
    • llm-translation
    • messages
    • stream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  709. ✓test_tool_use_streaming_vertex_ai_gpttool_use_streaming/test_vertex_ai_gpt.py›
    • llm-translation

    No steps recorded

    View test on GitHub ↗
  710. ✓test_tool_use_vertex_aitool_use/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  711. ✓test_tool_use_vertex_ai_gpttool_use/test_vertex_ai_gpt.py›
    • llm-translation

    No steps recorded

    View test on GitHub ↗
  712. ✓test_vision_anthropicvision/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • vision
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  713. ✓test_vision_azurevision/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • vision
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  714. ✓test_vision_bedrock_conversevision/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • vision
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  715. ✓test_vision_bedrock_invokevision/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • vision
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  716. ✓test_vision_vertex_aivision/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • vision
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  717. ✓test_web_search_anthropicweb_search/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • web_search
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  718. ✓test_web_search_azureweb_search/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • web_search
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  719. ✓test_web_search_bedrock_converseweb_search/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • web_search
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  720. ✓test_web_search_bedrock_invokeweb_search/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • web_search
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  721. ✓test_web_search_vertex_aiweb_search/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • web_search
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  722. −test_basic_messaging_non_streaming_bedrock_mantlebasic_messaging_non_streaming/test_bedrock_mantle.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock_mantle
    • gpt-5-6-luna-bedrock-mantle
    • gpt-5-6-sol-bedrock-mantle
    • gpt-5-6-terra-bedrock-mantle

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  723. −test_basic_messaging_non_streaming_openaibasic_messaging_non_streaming/test_openai.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • gpt-5-6-luna-openai
    • gpt-5-6-sol-openai
    • gpt-5-6-terra-openai

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  724. −test_basic_messaging_streaming_bedrock_mantlebasic_messaging_streaming/test_bedrock_mantle.py›
    • llm-translation
    • messages
    • stream
    • bedrock_mantle
    • gpt-5-6-luna-bedrock-mantle
    • gpt-5-6-sol-bedrock-mantle
    • gpt-5-6-terra-bedrock-mantle

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  725. −test_basic_messaging_streaming_openaibasic_messaging_streaming/test_openai.py›
    • llm-translation
    • messages
    • stream
    • openai
    • gpt-5-6-luna-openai
    • gpt-5-6-sol-openai
    • gpt-5-6-terra-openai

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  726. −test_count_tokens_vertex_aicount_tokens/test_vertex_ai.py›
    • llm-translation
    • count_tokens
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex

    No steps recorded

    stage red: Vertex returns not supported for token counting for Claude aliases

    View test on GitHub ↗
  727. −test_long_context_1m_anthropiclong_context_1m/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-opus-4-7
    • claude-sonnet-4-5

    No steps recorded

    stage red: 1M long_context not green on stage Anthropic path yet (200k sonnet / model alias)

    View test on GitHub ↗
  728. −test_long_context_1m_azurelong_context_1m/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure

    No steps recorded

    stage red: 1M long_context not green on stage Azure Foundry deployments yet

    View test on GitHub ↗
  729. −test_long_context_1m_bedrock_converselong_context_1m/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse

    No steps recorded

    stage red: 1M long_context not green on stage Bedrock Converse deployments yet

    View test on GitHub ↗
  730. −test_long_context_1m_bedrock_invokelong_context_1m/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke

    No steps recorded

    stage red: 1M long_context not green on stage Bedrock Invoke deployments yet

    View test on GitHub ↗
  731. −test_long_context_1m_vertex_ailong_context_1m/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex

    No steps recorded

    stage red: 1M long_context not green on stage Vertex deployments yet

    View test on GitHub ↗
  732. −test_passthrough_azurepassthrough/test_azure.py›
    • passthrough
    • passthrough
    • stream
    • azure
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5

    No steps recorded

    stage red: /azure passthrough drops client headers (e.g. anthropic-version); product gap

    View test on GitHub ↗
  733. −test_tool_search_azuretool_search/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • tool_search

    No steps recorded

    stage red: Azure Foundry tool_search_server not supported in workspace for probed models

    View test on GitHub ↗
  734. −test_tool_search_vertex_aitool_search/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • tool_search

    No steps recorded

    stage red: Vertex rejects tool_search when deployment extra_headers inject context-1m beta; product/config

    View test on GitHub ↗
  735. −test_tool_use_bedrock_mantletool_use/test_bedrock_mantle.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock_mantle
    • gpt-5-6-luna-bedrock-mantle
    • gpt-5-6-sol-bedrock-mantle
    • gpt-5-6-terra-bedrock-mantle
    • function_calling

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  736. −test_tool_use_openaitool_use/test_openai.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • gpt-5-6-luna-openai
    • gpt-5-6-sol-openai
    • gpt-5-6-terra-openai
    • function_calling

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  737. −test_tool_use_streaming_bedrock_mantletool_use_streaming/test_bedrock_mantle.py›
    • llm-translation
    • messages
    • stream
    • bedrock_mantle
    • gpt-5-6-luna-bedrock-mantle
    • gpt-5-6-sol-bedrock-mantle
    • gpt-5-6-terra-bedrock-mantle
    • function_calling

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  738. −test_tool_use_streaming_openaitool_use_streaming/test_openai.py›
    • llm-translation
    • messages
    • stream
    • openai
    • gpt-5-6-luna-openai
    • gpt-5-6-sol-openai
    • gpt-5-6-terra-openai
    • function_calling

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  739. ✓test_batch_create_over_rpm_returns_mapped_429 TestBatchRateLimitErrorMappingtest_batches_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-batch-rl-map-66dca1068f48, tpm limit: 1000000 and rpm limit: 2
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    View test on GitHub ↗
  740. ✓test_batch_key_model_access_deniedtest_batches_e2e.py›
    • proxy-auth
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with models: openai-batch-34f8fec52438 and user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    View test on GitHub ↗
  741. ✓test_batch_lifecycle[azure-encoded]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  742. ✓test_batch_lifecycle[azure-model_param]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  743. ✓test_batch_lifecycle[azure-provider_fallback]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  744. ✓test_batch_lifecycle[azure-unified]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  745. ✓test_batch_lifecycle[bedrock-unified]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  746. ✓test_batch_lifecycle[openai-encoded]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Call the management route /health/liveliness
    2. Add a batch deployment named openai-batch-34f8fec52438 that calls openai/gpt-4o-mini
    3. Add a batch deployment named azure-batch-34f8fec52438 that calls azure/gpt-5.4-mini-batch
    4. Add a batch deployment named vertex-batch-34f8fec52438 that calls vertex_ai/gemini-2.5-flash
    5. Add a batch deployment named bedrock-batch-34f8fec52438 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    6. Generate a virtual key with user id: e2e-test-user
    7. Upload a batch input file to /v1/files
    8. Create a batch of /v1/chat/completions requests from the uploaded file
    9. Retrieve the batch
    10. Cancel the batch
    11. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  747. ✓test_batch_lifecycle[openai-model_param]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  748. ✓test_batch_lifecycle[openai-provider_fallback]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  749. ✓test_batch_lifecycle[openai-unified]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  750. ✓test_batch_lifecycle[vertex_ai-encoded]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  751. ✓test_batch_lifecycle[vertex_ai-model_param]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  752. ✓test_batch_lifecycle[vertex_ai-provider_fallback]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  753. ✓test_batch_lifecycle[vertex_ai-unified]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  754. ✓test_completed_batch_downloads_output_and_books_cost TestBatchTerminalStatetest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  755. ✓test_endpoint_mismatch_fails_batch_and_cancel_conflicts TestBatchFailurePathstest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Wait for the request's spend log in /spend/logs
    6. Cancel the batch
    View test on GitHub ↗
  756. ✓test_enqueued_allowance_accepts_batch_over_key_rpm TestBatchEnqueuedTokenLimittest_batches_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-batch-enq-rpm-16bd257e1d32, rpm limit: 2 and metadata: batch enqueued token limit: 100000
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    View test on GitHub ↗
  757. ✓test_exhausted_allowance_blocks_until_cancel_refunds TestBatchEnqueuedTokenLimittest_batches_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-batch-enq-size-91e87f1c90f9 and metadata: batch enqueued token limit: 1
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Generate a virtual key with user id: e2e-batch-enq-refund-36503787d8ab and metadata: batch enqueued token limit: 36
    5. Upload a batch input file to /v1/files
    6. Create a batch of /v1/chat/completions requests from the uploaded file
    7. Cancel the batch
    8. Create a batch of /v1/chat/completions requests from the uploaded file
    View test on GitHub ↗
  758. ✓test_file_content_matches_upload TestBatchFileContenttest_batches_e2e.py›
    • llm-translation
    • files
    • openai
    • gpt-4o-mini
    1. Start a provider edge server
    2. Add a batch deployment named e2e-file-content-0e4e6297713e that calls openai/gpt-4o-mini
    3. Generate a virtual key with user id: e2e-test-user
    4. Upload a batch input file to /v1/files
    5. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLGVmMDY2MGVjLTRlNGUtNDQwOC04NWQ2LTZhNDFhNGQ3NTAyZDt0YXJnZXRfbW9kZWxfbmFtZXMsZTJlLWZpbGUtY29udGVudC0wZTRlNjI5NzcxM2U7bGxtX291d
    View test on GitHub ↗
  759. ✓test_file_lifecycle_signs_s3_with_s3_credentials TestBedrockBatchSplitS3Credentialstest_batches_e2e.py›
    • llm-translation
    • files
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a batch deployment named bedrock-split-s3-batch-34f8fec52438 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload a batch input file to /v1/files
    4. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLDI3MjhiZGM4LWYwZjEtNDQ2Zi1iZDg5LThhOGMzOTk0NWVmMzt0YXJnZXRfbW9kZWxfbmFtZXMsYmVkcm9jay1zcGxpdC1zMy1iYXRjaC0zNGY4ZmVjNTI0Mzg7b
    5. Delete the uploaded file
    View test on GitHub ↗
  760. ✓test_file_upload_and_delete_outputstest_batches_e2e.py›
    • llm-translation
    • files
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Delete the uploaded file
    View test on GitHub ↗
  761. ✓test_foreign_encoded_file_id_routes_by_file_model TestBatchFailurePathstest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    View test on GitHub ↗
  762. ✓test_gemini_file_upload TestGeminiFilestest_batches_e2e.py›
    • llm-translation
    • files
    • gemini
    • gemini-2.5-flash
    1. Add a batch deployment named gemini-files-34f8fec52438 that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload a batch input file to /v1/files
    View test on GitHub ↗
  763. ✓test_list_page_cursors_address_only_the_callers_own_files TestOpenAIFilestest_batches_e2e.py›
    • llm-translation
    • files
    • openai
    1. Generate a virtual key with user id: e2e-file-list-5ed3fa474bb0
    2. List the files the key can see from /v1/files
    View test on GitHub ↗
  764. ✓test_malformed_jsonl_upload_rejected TestBatchFailurePathstest_batches_e2e.py›
    • llm-translation
    • files
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    View test on GitHub ↗
  765. ✓test_native_jsonl_round_trips_untouched_and_starts_a_batch TestVertexNativePassthroughtest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLDNkZTYzOWRmLTMyN2ItNGJmOS1iYjQwLTk5ZDA1ZDZlNjNiMzt0YXJnZXRfbW9kZWxfbmFtZXMsdmVydGV4LWJhdGNoLTM0ZjhmZWM1MjQzODtsbG1fb3V0cHV0X
    4. Create a batch of /v1/chat/completions requests from the uploaded file
    View test on GitHub ↗
  766. ✓test_passthrough_upload_is_rejected_outside_a_native_vertex_batch[no-target-model] TestVertexNativePassthroughtest_batches_e2e.py›
    • llm-translation
    • files
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    View test on GitHub ↗
  767. ✓test_passthrough_upload_is_rejected_outside_a_native_vertex_batch[non-vertex-target-model] TestVertexNativePassthroughtest_batches_e2e.py›
    • llm-translation
    • files
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    View test on GitHub ↗
  768. ✓test_passthrough_upload_is_rejected_outside_a_native_vertex_batch[openai-shaped-rows] TestVertexNativePassthroughtest_batches_e2e.py›
    • llm-translation
    • files
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    View test on GitHub ↗
  769. ✓test_retrieve_round_trips_metadata TestOpenAIFilestest_batches_e2e.py›
    • llm-translation
    • files
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Retrieve the uploaded file
    View test on GitHub ↗
  770. ✓test_unified_batch_create_with_assume_role TestBedrockBatchAssumeRoletest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a batch deployment named bedrock-sts-batch-34f8fec52438 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload a batch input file to /v1/files
    4. Create a batch of /v1/chat/completions requests from the uploaded file
    5. Retrieve the batch
    View test on GitHub ↗
  771. ✓test_unified_create_and_retrieve_via_chained_gateway TestBatchSecondHoptest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • litellm_proxy
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a batch deployment named openai-batch-hop-34f8fec52438 that calls litellm_proxy/openai-batch-34f8fec52438
    3. Upload a batch input file to /v1/files
    4. Create a batch of /v1/chat/completions requests from the uploaded file
    5. Retrieve the batch
    View test on GitHub ↗
  772. ✓test_unified_file_content_downloads[azure] TestBatchFileContenttest_batches_e2e.py›
    • llm-translation
    • files
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLDU5MzYyOTYxLWRkZGEtNGVjOS05YzAzLTRlNmY4ZjRlMmRkNTt0YXJnZXRfbW9kZWxfbmFtZXMsYXp1cmUtYmF0Y2gtMzRmOGZlYzUyNDM4O2xsbV9vdXRwdXRfZ
    View test on GitHub ↗
  773. ✓test_unified_file_content_downloads[bedrock] TestBatchFileContenttest_batches_e2e.py›
    • llm-translation
    • files
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLDU0OGFkNmE5LWFkNzYtNDY0OC1hZjA5LWExYzdjMTM4YmNhMjt0YXJnZXRfbW9kZWxfbmFtZXMsYmVkcm9jay1iYXRjaC0zNGY4ZmVjNTI0Mzg7bGxtX291dHB1d
    View test on GitHub ↗
  774. ✓test_unified_file_content_downloads[vertex_ai] TestBatchFileContenttest_batches_e2e.py›
    • llm-translation
    • files
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLGEzMzU3ZjZiLWIxYjYtNDIwMS05ZmE3LTAwN2U3MzhjN2E5Yzt0YXJnZXRfbW9kZWxfbmFtZXMsdmVydGV4LWJhdGNoLTM0ZjhmZWM1MjQzODtsbG1fb3V0cHV0X
    View test on GitHub ↗
  775. ✓test_unified_file_upload_and_batch_create_in_govcloud TestBedrockBatchGovCloudtest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • bedrock
    • bedrock/amazon.nova-lite-v1:0
    1. Add a batch deployment named bedrock-govcloud-batch-34f8fec52438 that calls bedrock/amazon.nova-lite-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload a batch input file to /v1/files
    4. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLDVmYTYwYTZjLWMzYmYtNDYwZi04MmIyLWRlZGVmNGMwN2JjNTt0YXJnZXRfbW9kZWxfbmFtZXMsYmVkcm9jay1nb3ZjbG91ZC1iYXRjaC0zNGY4ZmVjNTI0Mzg7b
    5. Create a batch of /v1/chat/completions requests from the uploaded file
    6. Retrieve the batch
    View test on GitHub ↗
  776. −test_batch_runs_to_completion_with_a_downloadable_output[model_query] TestHostedVllmBatchtest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • hosted_vllm
    • Qwen/Qwen2.5-0.5B-Instruct

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  777. −test_batch_runs_to_completion_with_a_downloadable_output[target_model_names] TestHostedVllmBatchtest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • hosted_vllm
    • Qwen/Qwen2.5-0.5B-Instruct

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  778. −test_failing_line_lands_in_the_error_file_not_the_batch_status TestHostedVllmBatchtest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • hosted_vllm
    • Qwen/Qwen2.5-0.5B-Instruct

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  779. −test_rate_limited_batch_create_leaves_no_unattributed_spend_rowtest_batches_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • gpt-4o-mini

    No steps recorded

    LIT-5027: the path under test hangs. The batch rate limiter reads the input file to count tokens by awaiting litellm.afile_content with no timeout, so a slow Files API holds POST /v1/batches open past any client deadline (63.6s observed on stage against a 60s read timeout). The unattributed-spend-row contract below is never reached, so the test reports a timeout rather than the behavior it guards. Unskip once the fetch is bounded.

    View test on GitHub ↗
  780. −test_uploaded_file_appears_in_list TestOpenAIFilestest_batches_e2e.py›
    • llm-translation
    • files
    • openai
    • gpt-4o-mini

    No steps recorded

    LIT-4820 (https://linear.app/litellm-ai/issue/LIT-4820): GET /v1/files omits newly uploaded files. The upload succeeds and GET /v1/files/{id} returns the file, but it never appears in the listing; the returned set is stable with its newest entry ~10h old, on both the managed (/v1/files?model=) and provider-scoped (/openai/v1/files) routes. Skipped rather than weakened because the assertion below is the correct contract. Remove this marker when LIT-4820 is fixed; do not relax the assertion to make it pass.

    View test on GitHub ↗
  781. ✓test_429_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    • routing
    • nonstream
    • openai
    • gpt-5.5
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Generate a virtual key with models: gpt-5.5, user id: e2e-test-user and rpm limit: 1
    3. Add a deployment named reliability-retry-429-153003d9f6df that fronts gpt-5.5 on this proxy with a key out of rpm, so it always gets a 429 and is benched on the first one
    4. Add a zero-weight backup deployment named reliability-retry-429-153003d9f6df that calls openai/gpt-5.5
    5. Use up the rpm-limited key's one allowed request with a /chat/completions call to gpt-5.5
    6. Send a /chat/completions request to reliability-retry-429-153003d9f6df with stream set to False
    View test on GitHub ↗
  782. ✓test_429_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    • routing
    • nonstream
    • openai
    • gpt-5.5
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Generate a virtual key with models: gpt-5.5, user id: e2e-test-user and rpm limit: 1
    3. Add a deployment named reliability-cooldown-429-72bc42e2af84 that fronts gpt-5.5 on this proxy with a key out of rpm, so it always gets a 429 and is benched on the first one
    4. Add a zero-weight backup deployment named reliability-cooldown-429-72bc42e2af84 that calls openai/gpt-5.5
    5. Use up the rpm-limited key's one allowed request with a /chat/completions call to gpt-5.5
    6. Send a /chat/completions request to reliability-cooldown-429-72bc42e2af84 with stream set to False
    View test on GitHub ↗
  783. ✓test_5xx_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-5xx-upstream-cf86a794ccb4 that calls openai/gpt-5.5 at an unreachable address
    3. Add a deployment named reliability-retry-5xx-6be601f0dc6d that fronts reliability-5xx-upstream-cf86a794ccb4 on this proxy, so it always gets a 500 and is benched on the first one
    4. Add a zero-weight backup deployment named reliability-retry-5xx-6be601f0dc6d that calls openai/gpt-5.5
    5. Send a /chat/completions request to reliability-retry-5xx-6be601f0dc6d with stream set to False
    View test on GitHub ↗
  784. ✓test_5xx_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    • routing
    • nonstream
    • openai
    • gpt-5.5
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-fail-fd326b1e9547 that calls openai/gpt-5.5 at an unreachable address
    3. Send a /chat/completions request to reliability-fail-fd326b1e9547 with stream set to False
    View test on GitHub ↗
  785. ✓test_5xx_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cooldown-5xx-upstream-cff8e9adfff5 that calls openai/gpt-5.5 at an unreachable address
    3. Add a deployment named reliability-cooldown-5xx-844165ee9f22 that fronts reliability-cooldown-5xx-upstream-cff8e9adfff5 on this proxy, so it always gets a 500 and is benched on the first one
    4. Add a zero-weight backup deployment named reliability-cooldown-5xx-844165ee9f22 that calls openai/gpt-5.5
    5. Send a /chat/completions request to reliability-cooldown-5xx-844165ee9f22 with stream set to False
    View test on GitHub ↗
  786. ✓test_alias_api_key_never_overrides_the_tier_credential TestAliasParamForwardingtest_auto_router_regressions_e2e.py›
    • routing
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-cred-tier-d855eb1d8481 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named e2e-cred-alias-d855eb1d8481 that calls auto_router/complexity_router
    3. Generate a virtual key with models: e2e-cred-alias-d855eb1d8481, e2e-cred-tier-d855eb1d8481 and user id: e2e-auto-router-regressions
    4. Send a /chat/completions request to e2e-cred-alias-d855eb1d8481
    5. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  787. ✓test_auth_failure_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-retry-auth-e324e1a78143 that calls openai/gpt-5.5 with an invalid key and is benched on its first 401
    3. Add a zero-weight backup deployment named reliability-retry-auth-e324e1a78143 that calls openai/gpt-5.5
    4. Send a /chat/completions request to reliability-retry-auth-e324e1a78143 with stream set to False
    View test on GitHub ↗
  788. ✓test_auth_failure_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cooldown-auth-e52fb8eaafd6 that calls openai/gpt-5.5 with an invalid key and is benched on its first 401
    3. Add a zero-weight backup deployment named reliability-cooldown-auth-e52fb8eaafd6 that calls openai/gpt-5.5
    4. Send a /chat/completions request to reliability-cooldown-auth-e52fb8eaafd6 with stream set to False
    View test on GitHub ↗
  789. ✓test_body_tagged_chat_reaches_the_untagged_tier_after_marker_rewrite TestUntaggedTierDeploymentstest_auto_router_regressions_e2e.py›
    • routing
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with models: e2e-autoroute-386412ea40bb, e2e-tier-386412ea40bb, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /chat/completions request to e2e-autoroute-386412ea40bb
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  790. ✓test_body_tagged_chat_routes_through_the_marker_to_its_tier TestTagSplitRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-autoroute-310707a0beb8 that calls anthropic/claude-sonnet-5
    2. Add a deployment named e2e-tier-310707a0beb8 that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-autoroute-310707a0beb8 that calls auto_router/complexity_router
    4. Generate a virtual key with models: e2e-autoroute-310707a0beb8, e2e-tier-310707a0beb8, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    5. Send a /chat/completions request to e2e-autoroute-310707a0beb8
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  791. ✓test_body_tagged_responses_with_list_input_routes_to_the_tier TestResponsesApiTagRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with models: e2e-autoroute-1167180c9239, e2e-tier-1167180c9239, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. POST /v1/responses
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  792. ✓test_cached_conversation_stays_on_deployment_holding_its_cache TestReliabilityPromptCachingAffinitytest_reliability_prompt_caching_e2e.py›
    • routing
    • nonstream
    • anthropic
    • openai
    • anthropic/claude-haiku-4-5
    • openai/gpt-5.5
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Read the router settings from /router/settings
    3. Add a deployment named reliability-cache-7dbd00d02682 that calls anthropic/claude-haiku-4-5 with prompt caching
    4. Send a /chat/completions request to reliability-cache-7dbd00d02682 with a full message history and stream set to False
    5. Add a deployment named reliability-cache-7dbd00d02682 that calls openai/gpt-5.5
    6. Send a /chat/completions request to reliability-cache-7dbd00d02682 with a full message history and stream set to False
    View test on GitHub ↗
  793. ✓test_client_hanging_up_never_benches_the_deployment TestReliabilityCancelOnDisconnecttest_reliability_cancel_on_disconnect_e2e.py›
    • routing
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Generate a virtual key with user id: e2e-test-user
    2. Check whether the general setting cancel_on_disconnect is on
    3. Add a deployment named reliability-cooldown-disconnect-14879a0b320a that calls azure/gpt-5.4-nano and is benched for 300.0s on its first failure
    4. Add a zero-weight backup deployment named reliability-cooldown-disconnect-14879a0b320a that calls openai/gpt-5.5
    5. Send a /chat/completions request to reliability-cooldown-disconnect-14879a0b320a with stream set to False
    6. Abandon the request to /chat/completions after 5.0s
    7. Send a /chat/completions request to reliability-cooldown-disconnect-14879a0b320a with stream set to False
    View test on GitHub ↗
  794. ✓test_content_policy_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    • routing
    • nonstream
    • azure
    • openai
    • azure/gpt-5.4-nano
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-policyfail-27afabea2ce2 that calls azure/gpt-5.4-nano behind Azure's content filter
    3. Send a /chat/completions request to reliability-policyfail-27afabea2ce2 with stream set to False
    View test on GitHub ↗
  795. ✓test_context_window_refusal_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-3.5-turbo
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-retry-context-2525a3b4dcd1 that calls openai/gpt-3.5-turbo and takes all of its group's traffic
    3. Add a zero-weight backup deployment named reliability-retry-context-2525a3b4dcd1 that calls openai/gpt-5.5
    4. Send a /chat/completions request to reliability-retry-context-2525a3b4dcd1 with stream set to False
    View test on GitHub ↗
  796. ✓test_context_window_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    • routing
    • nonstream
    • openai
    • gpt-5.5
    • openai/gpt-3.5-turbo
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-ctxfail-d634e224a0be that calls the small-context model openai/gpt-3.5-turbo
    3. Send a /chat/completions request to reliability-ctxfail-d634e224a0be with stream set to False
    View test on GitHub ↗
  797. ✓test_cost_based_picks_cheapest_deployment TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cost-3a89636313ba that calls openai/gpt-5.5
    3. Send a /chat/completions request to reliability-cost-3a89636313ba with stream set to False
    View test on GitHub ↗
  798. ✓test_exact_cache_returns_cached TestReliabilityCachetest_reliability_cache_e2e.py›
    • caching
    • nonstream
    • openai
    • openai/gpt-5.6
    1. Generate a virtual key with user id: e2e-test-user
    2. Run an observed provider edge server
    3. Add a deployment named e2e-cache-60f4fa075cd4 that calls openai/gpt-5.6
    4. Send a request to /chat/completions
    View test on GitHub ↗
  799. ✓test_failing_requests_do_not_grow_rss_or_stored_request TestReliabilityMemorytest_reliability_memory_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-memory-556c1fb64314 that calls openai/gpt-5.5 at an unreachable address and is never benched
    3. Add a deployment named reliability-memory-fb-556c1fb64314 that calls openai/gpt-5.5 at an unreachable address and is never benched
    4. Send a /chat/completions request to reliability-memory-556c1fb64314 with stream set to False
    5. Wait for the request's spend log in /spend/logs
    6. Send a /chat/completions request to reliability-memory-556c1fb64314 with stream set to False
    7. Read memory usage from /debug/memory/summary on every proxy replica
    8. Send a /chat/completions request to reliability-memory-556c1fb64314 with stream set to False
    9. Read memory usage from /debug/memory/summary on every proxy replica
    View test on GitHub ↗
  800. ✓test_header_tagged_messages_routes_through_the_marker_to_an_untagged_tier TestUntaggedTierDeploymentstest_auto_router_regressions_e2e.py›
    • routing
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-autoroute-386412ea40bb that calls auto_router/complexity_router
    2. Add a deployment named e2e-tier-386412ea40bb that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-autoroute-386412ea40bb that calls anthropic/claude-sonnet-5
    4. Generate a virtual key with models: e2e-autoroute-386412ea40bb, e2e-tier-386412ea40bb, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    5. POST /v1/messages
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  801. ✓test_header_tagged_responses_with_string_input_routes_to_the_tier TestResponsesApiTagRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-autoroute-1167180c9239 that calls anthropic/claude-sonnet-5
    2. Add a deployment named e2e-tier-1167180c9239 that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-autoroute-1167180c9239 that calls auto_router/complexity_router
    4. Generate a virtual key with models: e2e-autoroute-1167180c9239, e2e-tier-1167180c9239, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    5. POST /v1/responses
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  802. ✓test_request_timeout_exceeds_deadline TestReliabilityTimeoutstest_reliability_timeouts_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-timeout-1c150a348f98 that calls openai/gpt-5.5 with a 1ms timeout
    3. Send a /chat/completions request to reliability-timeout-1c150a348f98 with stream set to False
    View test on GitHub ↗
  803. ✓test_responses_input_reaches_the_semantic_auto_router TestSemanticAutoRouterResponsestest_auto_router_regressions_e2e.py›
    • routing
    • responses
    • nonstream
    • anthropic
    • openai
    • anthropic/claude-haiku-4-5
    • openai/text-embedding-3-small
    1. Add a deployment named e2e-semantic-embedding-30fb43d986b1 that calls openai/text-embedding-3-small
    2. Add a deployment named e2e-semantic-target-30fb43d986b1 that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-semantic-fallback-30fb43d986b1 that calls anthropic/claude-sonnet-5
    4. Add a deployment named e2e-semantic-router-30fb43d986b1 that calls auto_router/e2e-semantic-router-30fb43d986b1
    5. Generate a virtual key with models: e2e-semantic-router-30fb43d986b1, e2e-semantic-target-30fb43d986b1, e2e-semantic-fallback-30fb43d986b1 and user id: e2e-auto-router-regressions
    6. POST /v1/responses
    7. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  804. ✓test_sibling_replica_serves_backup_within_redis_read_interval TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cooldown-sibling-upstream-4346e1342a2d that calls openai/gpt-5.5 at an unreachable address
    3. Add a deployment named reliability-cooldown-sibling-a6ec5223ef16 that fronts reliability-cooldown-sibling-upstream-4346e1342a2d on this proxy, so it always gets a 500 and is benched on the first one
    4. Add a zero-weight backup deployment named reliability-cooldown-sibling-a6ec5223ef16 that calls openai/gpt-5.5
    5. Add a deployment named reliability-cooldown-sibling-warm-473ae5ec9c20 for openai/gpt-5.5 that answers with a canned reply
    6. Send a /chat/completions request to reliability-cooldown-sibling-warm-473ae5ec9c20 with stream set to False
    7. Send a /chat/completions request to reliability-cooldown-sibling-a6ec5223ef16 with stream set to False
    View test on GitHub ↗
  805. ✓test_simple_shuffle_honors_weights TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-shuffle-422f1466a818 that calls openai/gpt-5.5
    3. Send a /chat/completions request to reliability-shuffle-422f1466a818 with stream set to False
    View test on GitHub ↗
  806. ✓test_stream_timeout_exceeds_deadline TestReliabilityTimeoutstest_reliability_timeouts_e2e.py›
    • routing
    • stream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-stream-timeout-9be96a014f73 that calls openai/gpt-5.5 with a 1ms timeout
    3. Send a /chat/completions request to reliability-stream-timeout-9be96a014f73 with stream set to True
    View test on GitHub ↗
  807. ✓test_tagged_call_straight_at_an_untagged_deployment_stays_denied TestUntaggedTierDeploymentstest_auto_router_regressions_e2e.py›
    • routing
    • chat_completions
    1. Generate a virtual key with models: e2e-tier-386412ea40bb, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /chat/completions request to e2e-tier-386412ea40bb
    View test on GitHub ↗
  808. ✓test_timeout_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-retry-a293c1d4b958 that calls openai/gpt-5.5 with a 1ms timeout and is benched on its first timeout
    3. Add a zero-weight backup deployment named reliability-retry-a293c1d4b958 that calls openai/gpt-5.5
    4. Send a /chat/completions request to reliability-retry-a293c1d4b958 with stream set to False
    View test on GitHub ↗
  809. ✓test_timeout_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    • routing
    • nonstream
    • openai
    • gpt-5.5
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-tofail-4cd896bb72da that calls openai/gpt-5.5 with a 1ms timeout
    3. Send a /chat/completions request to reliability-tofail-4cd896bb72da with stream set to False
    View test on GitHub ↗
  810. ✓test_timeout_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cooldown-timeout-b92d037414fc that calls openai/gpt-5.5 with a 1ms timeout and is benched on its first timeout
    3. Add a zero-weight backup deployment named reliability-cooldown-timeout-b92d037414fc that calls openai/gpt-5.5
    4. Send a /chat/completions request to reliability-cooldown-timeout-b92d037414fc with stream set to False
    View test on GitHub ↗
  811. ✓test_trivial_ask_behind_keyword_heavy_system_prompt_stays_on_the_cheap_tier TestComplexityHeuristicScopetest_auto_router_regressions_e2e.py›
    • routing
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-heuristic-cheap-92bf7eb3c4e9 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named e2e-heuristic-strong-92bf7eb3c4e9 that calls openai/gpt-5.6
    3. Add a deployment named e2e-heuristic-router-92bf7eb3c4e9 that calls auto_router/complexity_router
    4. Generate a virtual key with models: e2e-heuristic-router-92bf7eb3c4e9, e2e-heuristic-cheap-92bf7eb3c4e9, e2e-heuristic-strong-92bf7eb3c4e9 and user id: e2e-auto-router-regressions
    5. Send a /chat/completions request to e2e-heuristic-router-92bf7eb3c4e9
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  812. ✓test_untagged_chat_is_always_served_by_the_plain_deployment TestTagSplitRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5
    1. Generate a virtual key with models: e2e-autoroute-310707a0beb8, e2e-tier-310707a0beb8, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /chat/completions request to e2e-autoroute-310707a0beb8
    3. Wait for at least 5 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  813. ✓test_untagged_messages_is_served_by_the_plain_deployment TestTagSplitRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5
    1. Generate a virtual key with models: e2e-autoroute-310707a0beb8, e2e-tier-310707a0beb8, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /v1/messages request to e2e-autoroute-310707a0beb8
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  814. ✓test_untagged_responses_is_served_by_the_plain_deployment TestResponsesApiTagRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5
    1. Generate a virtual key with models: e2e-autoroute-1167180c9239, e2e-tier-1167180c9239, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. POST /v1/responses
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  815. ✓test_usage_based_picks_deployment_with_tpm_headroom TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-usage-6e359bbf1278 that calls openai/gpt-5.5
    3. Send a /chat/completions request to reliability-usage-6e359bbf1278 with stream set to False
    View test on GitHub ↗
  816. ✓test_workers_idle_under_rss_budget_before_traffic TestReliabilityMemorytest_reliability_memory_e2e.py›
    • deploy-ops

    No steps recorded

    View test on GitHub ↗
  817. ✓test_zero_priced_alias_still_logs_spend_at_the_tier_rate TestStrategyAliasPricingtest_auto_router_regressions_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-priced-tier-73d5a8b446dd that calls anthropic/claude-haiku-4-5
    2. Add a deployment named e2e-priced-alias-73d5a8b446dd that calls auto_router/complexity_router
    3. Generate a virtual key with models: e2e-priced-alias-73d5a8b446dd, e2e-priced-tier-73d5a8b446dd and user id: e2e-auto-router-regressions
    4. Send a /chat/completions request to e2e-priced-alias-73d5a8b446dd
    5. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  818. −test_latency_based_routes_around_deployment_that_times_out TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5

    No steps recorded

    LIT-7682: since #40229 the per-request routing_strategy override runs without the latency sampler, so latency-based has no signal to route on

    View test on GitHub ↗
  819. −test_least_busy_avoids_deployment_with_request_in_flight TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    • routing
    • openai
    • openai/gpt-5.5

    No steps recorded

    LIT-7682: since #40229 the per-request routing_strategy override runs without the in-flight counter, so least-busy has no signal to route on

    View test on GitHub ↗
  820. −test_llm_classifier_runs_and_routes_by_semantic_tier TestComplexityRouterLlmClassifiertest_complexity_router_e2e.py›
    • routing
    • nonstream
    • anthropic
    • openai
    • anthropic/claude-haiku-4-5
    • openai/gpt-5.5

    No steps recorded

    product bug LIT-4521: LLM classifier returns SIMPLE for short hard prompts (e.g. Is P equal to NP?); re-enable when classifier tier quality is fixed

    View test on GitHub ↗
  821. ✓test_apply_guardrail_blocks_banned_and_allows_clean TestApplyGuardrailEndpointtest_apply_guardrail_e2e.py›
    • guardrails
    • guardrails
    1. Register the content filter guardrail e2e-apply-db95b1c735b2 that blocks prompts containing e2e-banned-3609a7ba9441
    2. Apply the guardrail e2e-apply-db95b1c735b2 to a piece of text with /guardrails/apply_guardrail
    View test on GitHub ↗
  822. ✓test_bedrock_post_call_blocks_denied_model_output TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the bedrock guardrail e2e-bedrock-post-95ff99afb9aa with mode post_call
    3. Retry the call until a guardrail blocks it
    View test on GitHub ↗
  823. ✓test_bedrock_post_call_blocks_denied_streamed_output_and_passes_clean_streams TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the bedrock guardrail e2e-bedrock-post-stream-ee41deea65e0 with mode post_call
    3. Retry the streamed call until a guardrail blocks it
    4. Send a streaming /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  824. ✓test_bedrock_pre_call_blocks_harmful_prompt TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the Bedrock guardrail e2e-bedrock-guard-f0ff24964663
    3. Retry the call until a guardrail blocks it
    View test on GitHub ↗
  825. ✓test_bedrock_pre_call_blocks_on_messages TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • messages
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the Bedrock guardrail e2e-bedrock-messages-83e4dc92c674
    3. Retry the streamed call until a guardrail blocks it
    View test on GitHub ↗
  826. ✓test_bedrock_pre_call_blocks_on_responses TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • responses
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the Bedrock guardrail e2e-bedrock-responses-cef37d727fc4
    3. Retry the streamed call until a guardrail blocks it
    View test on GitHub ↗
  827. ✓test_blocks_execution_request_but_allows_explanation TestBlockCodeExecutionGuardrailtest_block_code_execution_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-blockcode-backend-23bde407a5a3 that calls gemini/gemini-2.5-flash
    3. Register the block_code_execution guardrail e2e-block-code-0271b8583a5d with mode pre_call
    4. Send a /chat/completions request to e2e-blockcode-backend-23bde407a5a3
    View test on GitHub ↗
  828. ✓test_chat_blocks_category TestOpenAIModerationCategoryMatrixtest_openai_moderation_category_matrix_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-mod-cat-chat-a1998115003d that calls gemini/gemini-2.5-flash
    3. Register the openai_moderation guardrail e2e-mod-cat-chat-55485aa96929 with mode pre_call
    4. Send a /chat/completions request to e2e-mod-cat-chat-a1998115003d
    View test on GitHub ↗
  829. ✓test_child_condition_match_applies_child_and_inherited_parent_guardrails TestPolicyInheritedGuardrailtest_policy_inherited_guardrail_e2e.py›
    • guardrails
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the content filter guardrail e2e-parent-guard-8e3436b78b9b that blocks prompts containing 8e3436b78b9b
    3. Register the content filter guardrail e2e-child-guard-31e5c5247b1a that blocks prompts containing 31e5c5247b1a
    4. Create the guardrail policy e2e-parent-policy-8e3436b78b9b that adds the guardrails e2e-parent-guard-8e3436b78b9b
    5. Create the guardrail policy e2e-child-policy-31e5c5247b1a that adds the guardrails e2e-child-guard-31e5c5247b1a
    6. Attach the guardrail policy e2e-child-policy-31e5c5247b1a to requests tagged e2e-policy-tag-91430015358c
    7. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  830. ✓test_child_condition_miss_still_applies_inherited_parent_guardrail TestPolicyInheritedGuardrailtest_policy_inherited_guardrail_e2e.py›
    • guardrails
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the content filter guardrail e2e-parent-guard-037270e710bd that blocks prompts containing 037270e710bd
    3. Register the content filter guardrail e2e-child-guard-cca5c134f5b8 that blocks prompts containing cca5c134f5b8
    4. Create the guardrail policy e2e-parent-policy-037270e710bd that adds the guardrails e2e-parent-guard-037270e710bd
    5. Create the guardrail policy e2e-child-policy-cca5c134f5b8 that adds the guardrails e2e-child-guard-cca5c134f5b8
    6. Attach the guardrail policy e2e-child-policy-cca5c134f5b8 to requests tagged e2e-policy-tag-aaa91acb613f
    7. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  831. ✓test_during_call_blocks_stream_before_first_chunk TestBedrockDuringCallStreamingtest_streaming_guardrail_e2e.py›
    • guardrails
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the bedrock guardrail e2e-bedrock-during-3d8a52168917 with mode during_call
    3. Retry the streamed call until a guardrail blocks it
    View test on GitHub ↗
  832. ✓test_flag_returns_guardrail_information_for_the_guardrail_that_ran TestGuardrailInformationResponsetest_guardrail_information_response_e2e.py›
    • guardrails
    • nonstream
    • openai
    • openai/gpt-4.1-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the litellm_content_filter guardrail e2e-guardrail-information-06434397f57a with mode pre_call
    3. Add a deployment named e2e-guardrail-info-backend-53ccc9c0cd44 that calls openai/gpt-4.1-mini
    4. Send a /chat/completions request to e2e-guardrail-info-backend-53ccc9c0cd44
    View test on GitHub ↗
  833. ✓test_global_guardrail_blocks_key_without_team_opt_out TestTeamDisableGlobalGuardrailtest_team_disable_global_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the content filter guardrail e2e-content-filter-d9b4403ebb12 that blocks prompts containing d9b4403ebb12
    3. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  834. ✓test_key_attached_content_filter_blocks_banned_image_edit_prompt TestKeyAttachedGuardrailOnImageEditstest_key_guardrail_image_edit_e2e.py›
    • guardrails
    • images
    • nonstream
    • gemini
    • openai
    • gemini-2.5-flash
    • openai/gpt-image-2.5-flare
    1. Register the content filter guardrail e2e-image-edit-filter-415538e11cef that blocks prompts containing 415538e11cef
    2. Generate a virtual key with the guardrails e2e-image-edit-filter-415538e11cef
    3. Add a deployment named e2e-guard-image-edit-b65264397c08 that calls openai/gpt-image-2.5-flare
    4. Retry the call until a guardrail blocks it
    5. Send a /v1/images/edits request to e2e-guard-image-edit-b65264397c08
    View test on GitHub ↗
  835. ✓test_key_attached_content_filter_blocks_banned_video_prompt TestKeyAttachedGuardrailOnVideostest_key_guardrail_video_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • vertex_ai
    • gemini-2.5-flash
    • vertex_ai/veo-3.1-fast-generate-001
    1. Register the content filter guardrail e2e-video-filter-b49d09c63bad that blocks prompts containing b49d09c63bad
    2. Generate a virtual key with the guardrails e2e-video-filter-b49d09c63bad
    3. Add a deployment named e2e-guard-video-017fc7dcd2b7 that calls vertex_ai/veo-3.1-fast-generate-001
    4. Retry the call until a guardrail blocks it
    5. Send a /v1/videos request to e2e-guard-video-017fc7dcd2b7
    View test on GitHub ↗
  836. ✓test_masking_run_is_recorded_on_the_spend_log TestPresidioSpendLogRecordtest_presidio_masking_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-25b43ed496ec with mode pre_call
    3. Send a /chat/completions request to gemini-2.5-flash
    4. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  837. ✓test_messages_blocks_category TestOpenAIModerationCategoryMatrixtest_openai_moderation_category_matrix_e2e.py›
    • guardrails
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-mod-cat-msg-f80285000f5d that calls anthropic/claude-haiku-4-5
    3. Register the openai_moderation guardrail e2e-mod-cat-msg-f5dda4235fc0 with mode pre_call
    4. Send a /v1/messages request to e2e-mod-cat-msg-f80285000f5d
    View test on GitHub ↗
  838. ✓test_moderation_blocks_flagged_input TestOpenAIModerationGuardrailtest_openai_moderation_guardrail_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-moderation-backend-a739efaa242b that calls gemini/gemini-2.5-flash
    3. Register the openai_moderation guardrail e2e-openai-moderation-8eb25113ad6b with mode pre_call
    4. Retry the call until a guardrail blocks it
    5. Send a /chat/completions request to e2e-moderation-backend-a739efaa242b
    View test on GitHub ↗
  839. ✓test_moderation_blocks_flagged_input_on_messages TestOpenAIModerationGuardrailtest_openai_moderation_guardrail_e2e.py›
    • guardrails
    • messages
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-moderation-msg-backend-4c9636d60bbe that calls gemini/gemini-2.5-flash
    3. Register the openai_moderation guardrail e2e-openai-moderation-msg-b503b23c732e with mode pre_call
    4. Retry the call until a guardrail blocks it
    5. Send a /v1/messages request to e2e-moderation-msg-backend-4c9636d60bbe
    View test on GitHub ↗
  840. ✓test_non_stream_block_is_schema_valid_json TestResponsesPreCallBlocktest_responses_pre_call_block_stream_e2e.py›
    • guardrails
    • responses
    • nonstream
    • openai
    • openai/gpt-4.1-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the custom_code guardrail e2e-custom-code-responses-block-e0db6699289d with mode pre_call
    3. Add a deployment named e2e-responses-block-74f6d1f08a05 that calls openai/gpt-4.1-mini
    4. Send a /v1/responses request to e2e-responses-block-74f6d1f08a05
    View test on GitHub ↗
  841. ✓test_post_call_masks_pii_in_model_output TestPresidioPostCallMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-post-chat-8a037d0861d2 with mode post_call
    3. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  842. ✓test_pre_call_allows_permitted_tool TestToolPermissionPreCalltest_tool_permission_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the tool_permission guardrail e2e-toolperm-allow-8309b2cc21ec with mode pre_call
    3. Retry the call until the guardrail e2e-toolperm-allow-8309b2cc21ec is applied
    View test on GitHub ↗
  843. ✓test_pre_call_blocks_tool_outside_the_allow_list TestToolPermissionPreCalltest_tool_permission_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the tool_permission guardrail e2e-toolperm-block-b411d419d75f with mode pre_call
    3. Retry the call until a guardrail blocks it
    View test on GitHub ↗
  844. ✓test_pre_call_masks_pii_on_chat_completions TestPresidioPreCallMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-pre-chat-b54e64e3f47d with mode pre_call
    3. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  845. ✓test_pre_call_masks_pii_on_messages TestPresidioPreCallMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • messages
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-pre-msg-225f667c2fb5 with mode pre_call
    3. Send a /v1/messages request to gemini-2.5-flash
    View test on GitHub ↗
  846. ✓test_responses_blocks_category TestOpenAIModerationCategoryMatrixtest_openai_moderation_category_matrix_e2e.py›
    • guardrails
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-mod-cat-resp-8b4d6ae59a55 that calls openai/gpt-4o-mini
    3. Register the openai_moderation guardrail e2e-mod-cat-resp-4e835bfe5c52 with mode pre_call
    4. Send a /v1/responses request to e2e-mod-cat-resp-8b4d6ae59a55
    View test on GitHub ↗
  847. ✓test_spend_log_stores_masked_output_on_anthropic_messages TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    • guardrails
    • messages
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-card-messages-f22f3169aef7 with mode post_call
    3. Send a /v1/messages request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  848. ✓test_spend_log_stores_masked_output_on_chat_completions TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-card-chat-ee07297df64c with mode post_call
    3. Send a /chat/completions request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  849. ✓test_spend_log_stores_masked_output_on_responses TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    • guardrails
    • responses
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-card-responses-facff37bfe44 with mode post_call
    3. Send a /v1/responses request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  850. ✓test_spend_log_stores_masked_output_on_streaming_anthropic_messages TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    • guardrails
    • messages
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-card-messages-stream-7d693fda3a24 with mode post_call
    3. Send a streaming /v1/messages request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  851. ✓test_spend_log_stores_masked_output_on_streaming_chat_completions TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    • guardrails
    • chat_completions
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-card-chat-stream-11fd97078f42 with mode post_call
    3. Send a streaming /chat/completions request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  852. ✓test_stream_block_is_sse_with_completed_assistant_message TestResponsesPreCallBlocktest_responses_pre_call_block_stream_e2e.py›
    • guardrails
    • responses
    • stream
    • openai
    • openai/gpt-4.1-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the custom_code guardrail e2e-custom-code-responses-block-020850756312 with mode pre_call
    3. Add a deployment named e2e-responses-block-9f071a170662 that calls openai/gpt-4.1-mini
    4. Send a streaming /v1/responses request to e2e-responses-block-9f071a170662
    View test on GitHub ↗
  853. ✓test_team_with_disable_flag_bypasses_global_guardrail TestTeamDisableGlobalGuardrailtest_team_disable_global_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Register the content filter guardrail e2e-content-filter-ff7b4826c29c that blocks prompts containing ff7b4826c29c
    2. Create the team e2e-guardrail-optout-ff7b4826c29c opted out of global guardrails and wait until /team/info returns it
    3. Generate a virtual key in the team
    4. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  854. ✓test_ui_default_scope_masks_a_card_number_the_model_generates_on_chat_completions TestPresidioCreditCardOutputMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-card-chat-07fedddba6c1 with mode pre_call, post_call
    3. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  855. ✓test_ui_default_scope_masks_a_card_number_the_model_generates_on_streaming_anthropic_messages TestPresidioCreditCardOutputMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • messages
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-card-messages-stream-30b7148fc57b with mode pre_call, post_call
    3. Send a streaming /v1/messages request to gemini-2.5-flash
    View test on GitHub ↗
  856. ✓test_ui_default_scope_masks_a_card_number_the_model_generates_on_streaming_chat_completions TestPresidioCreditCardOutputMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • chat_completions
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-card-stream-2c0ac917cfd3 with mode pre_call, post_call
    3. Send a streaming /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  857. ✓test_without_flag_response_has_no_guardrail_information TestGuardrailInformationResponsetest_guardrail_information_response_e2e.py›
    • guardrails
    • nonstream
    • openai
    • openai/gpt-4.1-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the litellm_content_filter guardrail e2e-guardrail-information-default-e8a18c7fc3e4 with mode pre_call
    3. Add a deployment named e2e-guardrail-info-backend-35eb75bb2fa9 that calls openai/gpt-4.1-mini
    4. Send a /chat/completions request to e2e-guardrail-info-backend-35eb75bb2fa9
    View test on GitHub ↗
  858. −test_request_naming_an_unknown_guardrail_fails_closedtest_guardrail_dispatch_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash

    No steps recorded

    stage red: product gap, a request naming a guardrail the proxy does not serve is silently served unguarded (200) instead of failing closed

    View test on GitHub ↗
  859. ✓test_chat_completions_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • chat_completions
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-chat-6e3af92e12ed with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for DataDog to ingest logs carrying the marker 6d0f3eb9a934, then watch for duplicates
    View test on GitHub ↗
  860. ✓test_chat_completions_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • chat_completions
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-trace-chat-6bd2f253c843 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call 803c22a1-b5ce-4e33-9f6b-114590eca9df
    View test on GitHub ↗
  861. ✓test_chat_completions_failure_writes_one_object TestS3LogDeliverytest_s3_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Add a deployment named s3-err-0e475430ac92 that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key named s3-err-key-7338241d2079 with models: s3-err-0e475430ac92
    4. Send a /chat/completions request to s3-err-0e475430ac92 with stream=False and the prompt "trigger an upstream auth failure"
    5. Wait for the request's log object to land in the S3 bucket under s3-err-key-7338241d2079/, then watch for duplicates
    View test on GitHub ↗
  862. ✓test_chat_completions_object_key_follows_the_partition_granularity TestS3LogDeliverytest_s3_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named s3-layout-163055d344c7 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the request's log object to land in the S3 bucket under s3-layout-163055d344c7/, then watch for duplicates
    5. List the log objects in the S3 bucket under s3-layout-163055d344c7/
    View test on GitHub ↗
  863. ✓test_chat_completions_stream_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • chat_completions
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-stream-chat-a2104c01483d with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the key's spend log in /spend/logs
    5. Wait for DataDog to ingest logs carrying the marker 43587934de41, then watch for duplicates
    View test on GitHub ↗
  864. ✓test_chat_completions_stream_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • chat_completions
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-stream-chat-54609e5eac99 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call 8e9564dc-43e3-408a-b874-881dfea445e8
    View test on GitHub ↗
  865. ✓test_chat_completions_stream_records_real_ttft TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • chat_completions
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-ttft-chat-c8523fefeb56 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call 365cdb6f-f57a-45ec-9930-40c84320a362
    View test on GitHub ↗
  866. ✓test_chat_completions_writes_one_success_object TestS3LogDeliverytest_s3_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named s3-chat-06534b7ad08f with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the request's log object to land in the S3 bucket under s3-chat-06534b7ad08f/, then watch for duplicates
    View test on GitHub ↗
  867. ✓test_chat_completions_writes_one_success_record TestGcsLogDeliverytest_gcs_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named gcs-chat-bf37727125c2 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the log of the response chatcmpl-e70ad191-6afc-4154-bae3-6de00f8a8b64 to land in the GCS bucket, then watch for duplicates
    View test on GitHub ↗
  868. ✓test_distinct_key_aliases_produce_distinct_series TestPrometheusPerKeyCardinalitytest_prometheus_cardinality_e2e.py›
    • observability
    • metrics
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key named e2e-prom-f009967215c6 with models: gemini-2.5-flash
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "reply with one word e2e-prom-f009967215c6"
    3. Generate a virtual key named e2e-prom-2425ea602910 with models: gemini-2.5-flash
    4. Send a /chat/completions request to gemini-2.5-flash with the prompt "reply with one word e2e-prom-2425ea602910"
    5. Generate a virtual key named e2e-prom-f24944b87345 with models: gemini-2.5-flash
    6. Send a /chat/completions request to gemini-2.5-flash with the prompt "reply with one word e2e-prom-f24944b87345"
    7. Scrape the Prometheus metrics from /metrics
    View test on GitHub ↗
  869. ✓test_failed_chat_completions_emits_one_error_event TestDataDogFailureDeliverytest_datadog_log_e2e.py›
    • observability
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Add a deployment named dd-err-98a9345da957 that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key named dd-err-key-e73b9826bff8 with models: dd-err-98a9345da957
    4. Send a /chat/completions request to dd-err-98a9345da957 with stream=False and the prompt "trigger an upstream auth failure"
    5. Wait for DataDog to ingest logs matching @model_group:dd-err-98a9345da957, then watch for duplicates
    View test on GitHub ↗
  870. ✓test_failed_chat_completions_error_span_attributes TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Add a deployment named otel-err-9b7434fa7eac that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key named otel-err-ba8d97207462 with models: otel-err-9b7434fa7eac
    4. Send a /chat/completions request to otel-err-9b7434fa7eac with stream=False and the prompt "trigger an upstream auth failure"
    5. Poll Jaeger for the traces of call dfce0f37-9dab-410f-94ad-f89ad0e3ce6e
    View test on GitHub ↗
  871. ✓test_failed_chat_completions_stream_emits_one_error_event TestDataDogFailureDeliverytest_datadog_log_e2e.py›
    • observability
    • chat_completions
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Add a deployment named dd-err-stream-9717e1adab9e that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key named dd-err-stream-key-24fe200201c3 with models: dd-err-stream-9717e1adab9e
    4. Stream a POST to /chat/completions
    5. Wait for DataDog to ingest logs matching @model_group:dd-err-stream-9717e1adab9e, then watch for duplicates
    View test on GitHub ↗
  872. ✓test_failed_messages_error_span_attributes TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Add a deployment named otel-err-1d2b42174e41 that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key named otel-err-3f2d07a6d61c with models: otel-err-1d2b42174e41
    4. Send a /v1/messages request to otel-err-1d2b42174e41 with stream=False and the prompt "trigger an upstream auth failure"
    5. Poll Jaeger for the traces of call ed213836-a094-4c1a-a64d-1a82daa7ef6e
    View test on GitHub ↗
  873. ✓test_messages_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-messages-2f28c2e4f889 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for DataDog to ingest logs carrying the marker b5557de1e560, then watch for duplicates
    View test on GitHub ↗
  874. ✓test_messages_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-trace-messages-ba181fe5de33 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call d44b5e29-1501-4aa1-aac1-419fc9d91e6e
    View test on GitHub ↗
  875. ✓test_messages_stream_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • messages
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-stream-messages-b6de5c642884 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the key's spend log in /spend/logs
    5. Wait for DataDog to ingest logs carrying the marker 6c182cf1452e, then watch for duplicates
    View test on GitHub ↗
  876. ✓test_messages_stream_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • messages
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-stream-messages-69b19c851b40 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call d8a0bcb0-3e5c-4a04-822e-79bbc8075540
    View test on GitHub ↗
  877. ✓test_messages_stream_records_real_ttft TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • messages
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-ttft-messages-faf6f785bfba with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call 1c40e02d-52b0-4703-8d54-211cb5ec7afc
    View test on GitHub ↗
  878. ✓test_non_json_native_metadata_reaches_langsmith TestLangsmithBatchSerializationtest_langsmith_batch_serialization_e2e.py›
    • observability
    • nonstream
    • openai
    • gpt-5.5
    1. GET the external URL https://api.smith.langchain.com/runs/8c4cf6ce-1675-447c-866d-49a849eb6cda
    View test on GitHub ↗
  879. ✓test_queue_time_histogram_records_an_observation TestPrometheusRequestQueueTimetest_prometheus_queue_time_e2e.py›
    • observability
    • metrics
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key named e2e-queue-time-8013ab231c21 with models: gemini-2.5-flash
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "reply with one word e2e-queue-time-8013ab231c21"
    3. Scrape the Prometheus metrics from /metrics
    View test on GitHub ↗
  880. ✓test_responses_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • responses
    • nonstream
    • openai
    • gpt-5.5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-responses-5bf01b9ebed0 with models: gpt-5.5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for DataDog to ingest logs carrying the marker 328b7080be5e, then watch for duplicates
    View test on GitHub ↗
  881. ✓test_responses_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • responses
    • nonstream
    • openai
    • gpt-5.5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-trace-responses-d2fc9652564c with models: gpt-5.5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call 392c6140-6c56-466d-a046-6be06a6cb059
    View test on GitHub ↗
  882. ✓test_responses_stream_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • responses
    • stream
    • openai
    • gpt-5.5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-stream-responses-b2bcb60f47ea with models: gpt-5.5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the key's spend log in /spend/logs
    5. Wait for DataDog to ingest logs carrying the marker 05182efeb760, then watch for duplicates
    View test on GitHub ↗
  883. ✓test_responses_stream_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • responses
    • stream
    • openai
    • gpt-5.5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-stream-responses-7ad0faca9162 with models: gpt-5.5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call dbcfb62f-8cf7-45bf-b984-cd974565c8d3
    5. Wait for the key's spend log in /spend/logs
    View test on GitHub ↗
  884. ✓test_responses_stream_records_real_ttft TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • responses
    • stream
    • openai
    • gpt-5.5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-ttft-responses-c35465c1b22c with models: gpt-5.5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call 80b05447-2df9-433d-909d-0f21881e2291
    View test on GitHub ↗
  885. ✓test_team_callback_delivers_and_isolates TestTeamLangfuseCallbacktest_team_langfuse_callback_e2e.py›
    • observability
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team lf-team-1d83a23d5160 with models: claude-haiku-4-5
    2. Add a Langfuse OTel logging callback for success_and_failure events to the team
    3. Generate a virtual key named lf-team-key-f9f899682bd9 with models: claude-haiku-4-5
    4. Generate a virtual key named lf-solo-key-84a0438bfd40 with models: claude-haiku-4-5
    5. Retry the call until the fresh key stops answering 401
    6. Wait for the Langfuse generation for the key lf-team-key-f9f899682bd9
    7. Retry the call until the fresh key stops answering 401
    8. Look up the Langfuse generation for the key lf-solo-key-84a0438bfd40
    View test on GitHub ↗
  886. −test_chat_completions_delivers_one_call_with_spend TestWeaveLogDeliverytest_weave_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • claude-haiku-4-5

    No steps recorded

    stage red: product gap, key-scoped weave_otel spans are not delivered when the OTEL v2 callback is active

    View test on GitHub ↗
  887. −test_failed_chat_completions_delivers_one_error_call TestWeaveLogDeliverytest_weave_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5

    No steps recorded

    stage red: product gap, key-scoped weave_otel spans are not delivered when the OTEL v2 callback is active

    View test on GitHub ↗
  888. ✓test_encrypted_stored_value_is_not_a_bearer_token TestSessionTokentest_session_token_e2e.py›
    • proxy-auth
    1. Generate a virtual key with key alias: e2e-session-37ad8af4600b and metadata: logging: callback name: langfuse and callback vars: default settings
    2. Read the key's settings back from /key/info
    3. List users from /user/list with the given key
    View test on GitHub ↗
  889. ✓test_expired_session_token_is_denied TestSessionTokentest_session_token_e2e.py›
    • proxy-auth
    1. List users from /user/list with the given key
    View test on GitHub ↗
  890. ✓test_expired_token_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-dc2340e14f7f and the user e2e-jwt-user-dc2340e14f7f in it
    2. Create a team with team alias: e2e-jwt-dc2340e14f7f and team id: e2e-jwt-team-dc2340e14f7f
    3. Get a Keycloak access token for e2e-jwt-user-dc2340e14f7f from the client litellm-e2e-shortlived
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  891. ✓test_liveness_reports_alive_without_auth TestHealthLifecycletest_health_lifecycle_e2e.py›
    • deploy-ops
    • health
    1. Call /health/liveliness without credentials
    View test on GitHub ↗
  892. ✓test_master_key_authenticates_and_grants_admin_route TestMasterKeyAuthtest_master_key_auth_e2e.py›
    • proxy-auth
    1. List users from /user/list with the given key
    View test on GitHub ↗
  893. ✓test_non_matching_master_key_is_denied TestMasterKeyAuthtest_master_key_auth_e2e.py›
    • proxy-auth
    1. List users from /user/list with the given key
    View test on GitHub ↗
  894. ✓test_plain_virtual_key_still_works_with_jwt_auth_enabled TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  895. ✓test_readiness_details_require_auth_and_expose_diagnostics TestHealthLifecycletest_health_lifecycle_e2e.py›
    • deploy-ops
    • health
    1. Call /health/readiness/details without credentials
    2. Call /health/readiness/details with the given key
    View test on GitHub ↗
  896. ✓test_readiness_is_reachable_without_credentials TestHealthLifecycletest_health_lifecycle_e2e.py›
    • deploy-ops
    • health
    1. Call /health/readiness without credentials
    View test on GitHub ↗
  897. ✓test_readiness_reports_connected_db TestHealthLifecycletest_health_lifecycle_e2e.py›
    • deploy-ops
    • health
    1. Call /health/readiness without credentials
    View test on GitHub ↗
  898. ✓test_signed_token_for_another_application_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-63d05c3f494e and the user e2e-jwt-user-63d05c3f494e in it
    2. Create a team with team alias: e2e-jwt-63d05c3f494e and team id: e2e-jwt-team-63d05c3f494e
    3. Get a Keycloak access token for e2e-jwt-user-63d05c3f494e from the client litellm-e2e-other-app
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  899. ✓test_signed_token_from_the_wrong_issuer_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-7b7116fcd9c2 and the user e2e-jwt-user-7b7116fcd9c2 in it
    2. Create a team with team alias: e2e-jwt-7b7116fcd9c2 and team id: e2e-jwt-team-7b7116fcd9c2
    3. Get a Keycloak access token for e2e-jwt-user-7b7116fcd9c2 from the client litellm-e2e-tests
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  900. ✓test_tampered_signature_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-6f829df8d6b4 and the user e2e-jwt-user-6f829df8d6b4 in it
    2. Create a team with team alias: e2e-jwt-6f829df8d6b4 and team id: e2e-jwt-team-6f829df8d6b4
    3. Get a Keycloak access token for e2e-jwt-user-6f829df8d6b4 from the client litellm-e2e-tests
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  901. ✓test_team_header_with_the_alias_of_a_team_the_caller_is_not_in_is_rejected_like_an_unknown_value TestJwtTeamHeadertest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-707645200d47 and the user e2e-jwt-user-707645200d47 in it
    2. Create a team with team alias: e2e-jwt-alias-707645200d47 and team id: e2e-jwt-team-707645200d47
    3. Get a Keycloak access token for e2e-jwt-user-707645200d47 from the client litellm-e2e-tests
    4. Create a team with team alias: e2e-jwt-alias-841a524550f9 and team id: e2e-jwt-other-841a524550f9
    5. Send a /chat/completions request to gpt-5.5 as team e2e-jwt-alias-841a524550f9 with the given token
    6. Send a /chat/completions request to gpt-5.5 as team e2e-jwt-unknown-946fe99f0de6 with the given token
    View test on GitHub ↗
  902. ✓test_team_header_with_the_team_alias_binds_the_same_team_as_the_team_id TestJwtTeamHeadertest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-f0950379a0a8 and the user e2e-jwt-user-f0950379a0a8 in it
    2. Create a team with team alias: e2e-jwt-alias-f0950379a0a8 and team id: e2e-jwt-team-f0950379a0a8
    3. Get a Keycloak access token for e2e-jwt-user-f0950379a0a8 from the client litellm-e2e-tests
    4. Send a /chat/completions request to gpt-5.5 as team e2e-jwt-team-f0950379a0a8 with the given token
    5. Wait for the request's spend log in /spend/logs
    6. Send a /chat/completions request to gpt-5.5 as team e2e-jwt-alias-f0950379a0a8 with the given token
    7. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  903. ✓test_team_model_alias_is_listed_by_v1_models_under_the_same_token_that_routes_it[anthropic_shape] TestJwtTeamHeadertest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-9aa1010908e4 and the user e2e-jwt-user-9aa1010908e4 in it
    2. Create a team with team alias: e2e-jwt-aliased-9aa1010908e4, models: gpt-5.5, team id: e2e-jwt-team-9aa1010908e4 and model aliases: e2e-jwt-model-alias-9aa1010908e4: gpt-5.5
    3. Get a Keycloak access token for e2e-jwt-user-9aa1010908e4 from the client litellm-e2e-tests
    4. Send a /chat/completions request to e2e-jwt-model-alias-9aa1010908e4
    5. List the models from /v1/models with the given token, in the Anthropic shape: True
    View test on GitHub ↗
  904. ✓test_team_model_alias_is_listed_by_v1_models_under_the_same_token_that_routes_it[openai_shape] TestJwtTeamHeadertest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-287f591461ad and the user e2e-jwt-user-287f591461ad in it
    2. Create a team with team alias: e2e-jwt-aliased-287f591461ad, models: gpt-5.5, team id: e2e-jwt-team-287f591461ad and model aliases: e2e-jwt-model-alias-287f591461ad: gpt-5.5
    3. Get a Keycloak access token for e2e-jwt-user-287f591461ad from the client litellm-e2e-tests
    4. Send a /chat/completions request to e2e-jwt-model-alias-287f591461ad
    5. List the models from /v1/models with the given token, in the Anthropic shape: False
    View test on GitHub ↗
  905. ✓test_token_naming_a_team_that_does_not_exist_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-deff0200e548 and the user e2e-jwt-user-deff0200e548 in it
    2. Get a Keycloak access token for e2e-jwt-user-deff0200e548 from the client litellm-e2e-tests
    3. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  906. ✓test_unexpired_session_token_reaches_admin_route TestSessionTokentest_session_token_e2e.py›
    • proxy-auth
    1. List users from /user/list with the given key
    View test on GitHub ↗
  907. ✓test_valid_token_for_an_existing_team_is_accepted_and_attributed TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-05cf51066cea and the user e2e-jwt-user-05cf51066cea in it
    2. Create a team with team alias: e2e-jwt-05cf51066cea and team id: e2e-jwt-team-05cf51066cea
    3. Get a Keycloak access token for e2e-jwt-user-05cf51066cea from the client litellm-e2e-tests
    4. Send a /chat/completions request to gpt-5.5
    5. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  908. ✓test_allowed_model_is_permitted TestAccessControltest_access_control_e2e.py›
    • proxy-auth
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with models: gemini-2.5-flash and user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "capital of France? c72d3d2fa1fd"
    View test on GitHub ↗
  909. ✓test_bearer_invalid_token_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›
    • proxy-auth
    • chat_completions
    1. Send a request to /chat/completions
    View test on GitHub ↗
  910. ✓test_disallowed_model_is_denied_403 TestAccessControltest_access_control_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Generate a virtual key with models: gemini-2.5-flash and user id: e2e-test-user
    2. Send a /chat/completions request to gpt-5.5 with the prompt "capital of France? 854218f9484c"
    View test on GitHub ↗
  911. ✓test_empty_bearer_token_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›
    • proxy-auth
    • chat_completions
    1. Send a request to /chat/completions
    View test on GitHub ↗
  912. ✓test_group_grants_every_deployment_in_it[bare name the group's wildcard covers-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.4-nano
    • openai/gpt-5.4-nano
    1. Add a deployment named openai/gpt-5.4* that calls openai/gpt-5.4*
    2. Add a deployment named e2e-ag-member-dd4037f2299c that calls openai/gpt-5.4-nano
    3. Add a deployment named e2e-ag-outsider-dd4037f2299c that calls openai/gpt-5.4-nano
    4. List the deployments in the model access group e2e-ag-dd4037f2299c
    5. Generate a virtual key with models: e2e-ag-dd4037f2299c and user id: e2e-test-user
    6. Send a /chat/completions request to gpt-5.4-nano with the prompt "Reply with exactly: OK 11b938b1a5d8"
    View test on GitHub ↗
  913. ✓test_group_grants_every_deployment_in_it[exactly-named deployment in the group-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.4-nano
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-ag-dd4037f2299c and user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ag-member-dd4037f2299c with the prompt "Reply with exactly: OK 03adf24452b6"
    View test on GitHub ↗
  914. ✓test_group_grants_every_deployment_in_it[provider-prefixed name the group's wildcard covers-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.4-nano
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-ag-dd4037f2299c and user id: e2e-test-user
    2. Send a /chat/completions request to openai/gpt-5.4-nano with the prompt "Reply with exactly: OK b36216d1b0da"
    View test on GitHub ↗
  915. ✓test_group_grants_nothing_outside_it[deployment outside the group-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.2
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-ag-dd4037f2299c and user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ag-outsider-dd4037f2299c with the prompt "Reply with exactly: OK 59778f4b3387"
    View test on GitHub ↗
  916. ✓test_group_grants_nothing_outside_it[name no provider claims-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.2
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-ag-dd4037f2299c and user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ag-unknown-de6fb5ff483c with the prompt "Reply with exactly: OK 0eca31bde9b6"
    View test on GitHub ↗
  917. ✓test_group_grants_nothing_outside_it[provider model outside the group's wildcard-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.2
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-ag-dd4037f2299c and user id: e2e-test-user
    2. Send a /chat/completions request to gpt-5.2 with the prompt "Reply with exactly: OK 1f8d7dd45ccd"
    View test on GitHub ↗
  918. ✓test_group_grants_the_team_nothing_outside_it TestTeamScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    1. Send a /chat/completions request to e2e-ag-unknown-52d063830142 with the prompt "Reply with exactly: OK 2891a806167e"
    View test on GitHub ↗
  919. ✓test_group_grants_the_teams_own_wildcard TestTeamScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.6-luna
    1. Create the team e2e-ag-team-5e43c8140664 with models: e2e-agt-5e43c8140664
    2. Generate a virtual key with team id: d997096a-7490-47d6-a4fa-f2e3dc184729
    3. Add a deployment named openai/gpt-5.6* that calls openai/gpt-5.6*
    4. Set the team e2e-ag-team-5e43c8140664's models to e2e-agt-5e43c8140664 through /team/update
    5. Read /team/info?team_id=d997096a-7490-47d6-a4fa-f2e3dc184729 on every proxy replica until it settles
    6. Wait for the last control-plane write to reach every proxy replica
    7. Send a /chat/completions request to gpt-5.6-luna with the prompt "Reply with exactly: OK 91b8d40c30ea"
    View test on GitHub ↗
  920. ✓test_llm_api_routes_group_grants_every_llm_endpoint TestAccessControltest_access_control_e2e.py›
    • proxy-auth
    • gemini
    • openai
    • gemini-2.5-flash
    • openai-text-embedding-3-small
    1. Generate a virtual key that can only call LLM API routes
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "capital of France? 8ebfbaf78b8c"
    3. Send an /embeddings request to openai-text-embedding-3-small for "route group a49d1f5eb603"
    4. Add a deployment named e2e-route-group-9ea0cf84c9f0 that calls openai/gpt-4o-mini with the given key
    View test on GitHub ↗
  921. ✓test_llm_only_key_forbidden_from_management_route_403 TestAccessControltest_access_control_e2e.py›
    • proxy-auth
    • model_management
    1. Generate a virtual key that can only call LLM API routes
    2. Add a deployment named e2e-forbidden-0da686857af6 that calls openai/gpt-4o-mini with the given key
    View test on GitHub ↗
  922. ✓test_missing_authorization_header_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›
    • proxy-auth
    • chat_completions
    1. Send a request to /chat/completions
    View test on GitHub ↗
  923. ✓test_not_bearer_scheme_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›
    • proxy-auth
    • chat_completions
    1. Send a request to /chat/completions
    View test on GitHub ↗
  924. ✓test_token_without_bearer_prefix_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›
    • proxy-auth
    • chat_completions
    1. Send a request to /chat/completions
    View test on GitHub ↗
  925. ✓test_unknown_model_returns_400 TestAccessControltest_access_control_e2e.py›
    • proxy-auth
    • nonstream
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to nonexistent-model-7bea577582f0 with the prompt "hi this is a test"
    View test on GitHub ↗
  926. ✓test_valid_key_allows_and_invalid_key_denied TestVirtualKeyAuthtest_access_control_e2e.py›
    • proxy-auth
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-auth-chat-b258da8ab54e that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-auth-chat-b258da8ab54e
    View test on GitHub ↗
  927. ✓test_access_group_scopes_tool_selection TestMcpAccessGroupToolSelectiontest_mcp_access_group_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-ag-granted-263031a74fc1
    4. Generate a virtual key for the user e2e-mcp-ag-other-ee1d66eb1113
    5. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    6. List the MCP tools the key can see from /mcp-rest/tools/list
    View test on GitHub ↗
  928. ✓test_alias_grant_persists_verbatim_and_lists_tools TestMcpKeyGrantByAliastest_mcp_key_access_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-allowed-83a5bd7385e0
    4. Read the key's settings back from /key/info
    5. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    View test on GitHub ↗
  929. ✓test_call_tool_denied_without_permission TestMcpKeyWithoutAccessIsDeniedtest_mcp_key_access_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-allowed-1fa04c40c187
    4. Generate a virtual key for the user e2e-mcp-denied-c8e88145e262
    5. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    6. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call
    7. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call and wait for a 403
    View test on GitHub ↗
  930. ✓test_content_filter_blocks_banned_keyword_in_tool_args TestMcpToolCallGuardrailtest_mcp_guardrail_e2e.py›
    • guardrails
    • mcp
    1. Create the guardrail e2e-mcp-cf-aa9d3c30703f that blocks MCP tool calls containing "e2eblockedaa9d3c30703f"
    2. Register the Datadog remote MCP server with its credentials from the environment
    3. Generate a virtual key for the user e2e-mcp-guard-aa9d3c30703f
    4. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    5. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: tell me about e2eblockedaa9d3c30703f, from: now-30m, to: now and max tokens: 500
    6. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: still about e2eblockedaa9d3c30703f #1, from: now-30m, to: now and max tokens: 500
    7. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: still about e2eblockedaa9d3c30703f #2, from: now-30m, to: now and max tokens: 500
    8. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: still about e2eblockedaa9d3c30703f #3, from: now-30m, to: now and max tokens: 500
    9. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: still about e2eblockedaa9d3c30703f #4, from: now-30m, to: now and max tokens: 500
    10. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: e2e-clean-aa9d3c30703f, from: now-30m, to: now and max tokens: 500
    View test on GitHub ↗
  931. ✓test_key_granted_a_toolset_lists_exactly_its_tools TestMcpToolsetEnforcementtest_mcp_toolset_enforcement_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-catalog-c14b5088a93b
    4. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    5. List the MCP tools the key can see from /mcp-rest/tools/list
    6. Create an MCP toolset with the tools server id: c48afae5-36a9-4991-992f-bc5b225d94d7 and tool name: _dd_gff, server id: c48afae5-36a9-4991-992f-bc5b225d94d7 and tool name: _dd_guc
    7. Generate a virtual key for the user e2e-mcp-toolset-a77e84724762
    8. Wait for /mcp-rest/tools/list to show the key exactly the expected tools on the MCP server
    View test on GitHub ↗
  932. ✓test_list_tools_denied_without_permission TestMcpKeyWithoutAccessIsDeniedtest_mcp_key_access_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-allowed-1dd4a0bd32c4
    4. Generate a virtual key for the user e2e-mcp-denied-1d17ad27b1fb
    5. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    6. List the MCP tools the key can see from /mcp-rest/tools/list
    View test on GitHub ↗
  933. ✓test_route_restricted_health_matches_server_grants TestMcpHealthVisibilitytest_mcp_key_access_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-allowed-7892a094b0bd
    4. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    5. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call
    6. Generate a virtual key with user id: e2e-mcp-health-cb22f46b9a61, allowed routes: /v1/mcp/server, /v1/mcp/server/health and object permission: mcp servers: b771ae74-faed-43d0-84f3-123a43895996
    7. List the MCP servers the key can see from /v1/mcp/server
    8. Check the health of the MCP servers the key can see from /v1/mcp/server/health
    9. Generate a virtual key with user id: e2e-mcp-health-4c5c0561de3b, allowed routes: /v1/mcp/server, /v1/mcp/server/health and object permission: mcp servers: 1fe25c93-f0f1-47e8-bde8-374f526f0928
    10. List the MCP servers the key can see from /v1/mcp/server
    11. Check the health of the MCP servers the key can see from /v1/mcp/server/health
    12. Generate a virtual key with user id: e2e-mcp-health-29301e8c9da8, allowed routes: /v1/mcp/server, /v1/mcp/server/health and object permission: default settings
    13. List the MCP servers the key can see from /v1/mcp/server
    14. Check the health of the MCP servers the key can see from /v1/mcp/server/health
    View test on GitHub ↗
  934. ✓test_search_logs_finds_seeded_completion TestDatadogMcpRoundTriptest_mcp_datadog_e2e.py›
    • mcp
    • mcp
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Call the management route /health/readiness/details
    2. Register the Datadog remote MCP server with its credentials from the environment
    3. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    4. Generate a virtual key for the user e2e-dd-mcp-00029a8575e0
    5. Send a /chat/completions request to claude-haiku-4-5
    6. Wait for DataDog to ingest logs carrying the marker e2e-datadog-mcp-e63f132f2394, then watch for duplicates
    7. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    8. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call
    View test on GitHub ↗
  935. −test_chat_completion_uses_linear_with_authorization_bearer_header TestMcpChatCompletionOauthtest_mcp_chat_completion_oauth_e2e.py›
    • mcp
    • chat_completions
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • function_calling

    No steps recorded

    set E2E_LINEAR_STORAGE_STATE to a Linear session captured via mcp/linear_session_capture.py

    View test on GitHub ↗
  936. −test_chat_completion_uses_linear_with_x_litellm_api_key_header TestMcpChatCompletionOauthtest_mcp_chat_completion_oauth_e2e.py›
    • mcp
    • chat_completions
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • function_calling

    No steps recorded

    set E2E_LINEAR_STORAGE_STATE to a Linear session captured via mcp/linear_session_capture.py

    View test on GitHub ↗
  937. ✓test_discovery_card_is_proxy_fronted TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the A2A agent e2e-a2a-563dfa895c50 through /v1/agents
    3. Read the A2A agent's card from /a2a/{agent_id}/.well-known/agent-card.json with the given key
    View test on GitHub ↗
  938. ✓test_malformed_protocol_version_rejected TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    1. Register the A2A agent e2e-a2a-bad-28696ed65a07 through /v1/agents
    View test on GitHub ↗
  939. ✓test_message_send_runs_completion_bridge TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the A2A agent e2e-a2a-64031cc67ade through /v1/agents
    3. Send an A2A message to /a2a/{agent_id} with text: Reply with exactly the word PONG and nothing else
    4. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  940. ✓test_pinned_v0_3_serves_flat_message_shape TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the A2A agent e2e-a2a-362e50e2b603 through /v1/agents
    3. Send an A2A message to /a2a/{agent_id} with text: Say hi in one word
    View test on GitHub ↗
  941. ✓test_pinned_v1_0_serves_nested_message_shape TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the A2A agent e2e-a2a-f68b18dcc5d5 through /v1/agents
    3. Send an A2A message to /a2a/{agent_id} with text: Say hi in one word
    View test on GitHub ↗
  942. ✓test_real_world_agent_replies_to_property_query TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    1. Generate a virtual key with user id: e2e-test-user
    2. Fetch a published A2A agent card from its /.well-known endpoint
    3. Register the A2A agent e2e-a2a-real-773129160c2a through /v1/agents
    4. Send an A2A message to /a2a/{agent_id} with data: skill: search_properties and params: un locode: GBLON, service type: long_term and limit: 3
    View test on GitHub ↗
  943. ✓test_register_persists TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    1. Register the A2A agent e2e-a2a-ab665698198e through /v1/agents
    2. Read the A2A agent back from /v1/agents/{agent_id}
    View test on GitHub ↗
  944. ✓test_semver_protocol_version_registers_and_serves TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the A2A agent e2e-a2a-83d9343d2923 through /v1/agents
    3. Read the A2A agent's card from /a2a/{agent_id}/.well-known/agent-card.json with the given key
    4. Send an A2A message to /a2a/{agent_id} with text: Say hi in one word
    View test on GitHub ↗
  945. ✓test_unsupported_protocol_version_rejected TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    1. Register the A2A agent e2e-a2a-bad-77f93a24ef11 through /v1/agents
    View test on GitHub ↗
  946. −llm_translation/realtime/test_realtime_pipecat_audio_e2e.pyllm_translation/realtime/test_realtime_pipecat_audio_e2e.py›

    No steps recorded

    collection skipped