litellm-e2e · build #400

889 passed, 0 failed, 57 skipped

1.106.0rc1rc

Oct 11, 2026, 7:22 AM UTC ·5a53186 ↗· build #400 ↗· 1.106.0rc1 ↗· run history

All 946 tests

  1. ✓test_anthropic_chat_returns_thinking_content TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • reasoning
    1. Add a deployment named e2e-anthropic-thinking-2f12462651ce that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-thinking-2f12462651ce
    View test on GitHub ↗
  2. ✓test_anthropic_chat_returns_tool_call TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • function_calling
    1. Add a deployment named e2e-anthropic-tool-4eafa87d57c0 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-tool-4eafa87d57c0
    View test on GitHub ↗
  3. ✓test_anthropic_chat_streams_real_content TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-anthropic-stream-8d43a0335391 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-anthropic-stream-8d43a0335391
    View test on GitHub ↗
  4. ✓test_anthropic_chat_streams_tool_call TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • function_calling
    1. Add a deployment named e2e-anthropic-tool-stream-33b6d96588fc that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-anthropic-tool-stream-33b6d96588fc
    View test on GitHub ↗
  5. ✓test_anthropic_chat_structured_output_conforms_to_schema TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • response_schema
    1. Add a deployment named e2e-anthropic-schema-da0b76e79e91 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-schema-da0b76e79e91
    View test on GitHub ↗
  6. ✓test_anthropic_chat_vision_describes_image TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • vision
    1. Add a deployment named e2e-anthropic-vision-28fa9769366d that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-vision-28fa9769366d
    View test on GitHub ↗
  7. ✓test_anthropic_passthrough_nonstreaming_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to claude-haiku-4-5 through /anthropic with streaming set to False
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  8. ✓test_anthropic_passthrough_streaming_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to claude-haiku-4-5 through /anthropic with streaming set to True
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  9. ✓test_anthropic_passthrough_tool_call_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to claude-haiku-4-5 through /anthropic with streaming set to False
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  10. ✓test_anthropic_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • prompt_caching
    1. Add a deployment named e2e-anthropic-cache-46704207fafa that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /chat/completions
    View test on GitHub ↗
  11. ✓test_anthropic_with_extended_thinking TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    • reasoning
    1. Add a deployment named e2e-chat-tool-loop-2cecb8a49e8f that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-2cecb8a49e8f
    View test on GitHub ↗
  12. ✓test_array_input_returns_vectors TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • openai
    • openai/text-embedding-3-small
    1. Add a deployment named e2e-embeddings-array-5c814e72fd2c that calls openai/text-embedding-3-small
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  13. ✓test_audio_speech_returns_audio TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-tts
    1. Add a deployment named e2e-speech-aa5f6049600a that calls openai/gpt-4o-mini-tts
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  14. ✓test_audio_speech_streams_audio_chunks TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • stream
    • openai
    • openai/gpt-4o-mini-tts
    1. Add a deployment named e2e-speech-ce473f41edea that calls openai/gpt-4o-mini-tts
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  15. ✓test_audio_transcriptions_returns_text TestAudioTranscriptionstest_audio_transcriptions_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-transcribe
    1. Add a deployment named e2e-transcribe-99736505b2c8 that calls openai/gpt-4o-mini-transcribe
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  16. ✓test_azure_foundry_chat_returns_content TestAzureFoundryChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    1. Add a deployment named e2e-azure-foundry-chat-3d3bd68fcc2d that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-azure-foundry-chat-3d3bd68fcc2d
    View test on GitHub ↗
  17. ✓test_azure_openai_chat_returns_content TestAzureOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-azure-openai-chat-1a1a3e5e2c47 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-azure-openai-chat-1a1a3e5e2c47
    View test on GitHub ↗
  18. ✓test_azure_openai_chat_returns_tool_call TestAzureOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    • function_calling
    1. Add a deployment named e2e-azure-openai-tool-0cfe73ebf643 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-azure-openai-tool-0cfe73ebf643
    View test on GitHub ↗
  19. ✓test_basic_nonstream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    1. Add a deployment named e2e-azure-foundry-messages-ea928d74b621 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-ea928d74b621 and user id: e2e-test-user
    View test on GitHub ↗
  20. ✓test_basic_stream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    • llm-translation
    • messages
    • stream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    1. Add a deployment named e2e-azure-foundry-messages-5bfa3e42f9d1 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-5bfa3e42f9d1 and user id: e2e-test-user
    View test on GitHub ↗
  21. ✓test_batch_s3_keys_do_not_break_chat TestBedrockBatchDeploymentServesChattest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-batch-chat-830005356a2d that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  22. ✓test_bedrock_converse TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/converse/us.anthropic.claude-sonnet-5-5
    • function_calling
    1. Add a deployment named e2e-chat-tool-loop-678e8e1b8a55 that calls bedrock/converse/us.anthropic.claude-sonnet-5-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-678e8e1b8a55
    View test on GitHub ↗
  23. ✓test_bedrock_converse_chat_returns_content TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-chat-076249e557d9 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-chat-076249e557d9
    View test on GitHub ↗
  24. ✓test_bedrock_converse_chat_returns_reasoning TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • reasoning
    1. Add a deployment named e2e-bedrock-thinking-c0e7513b6d29 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-thinking-c0e7513b6d29
    View test on GitHub ↗
  25. ✓test_bedrock_converse_chat_returns_tool_call TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • function_calling
    1. Add a deployment named e2e-bedrock-tool-d522f672f48a that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-tool-d522f672f48a
    View test on GitHub ↗
  26. ✓test_bedrock_converse_chat_streams_real_content TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-stream-b13489bfad53 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bedrock-stream-b13489bfad53
    View test on GitHub ↗
  27. ✓test_bedrock_converse_chat_vision_describes_image TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • vision
    1. Add a deployment named e2e-bedrock-vision-95a7e9e49fd3 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-vision-95a7e9e49fd3
    View test on GitHub ↗
  28. ✓test_bedrock_converse_reads_a_pdf_sent_by_url TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.amazon.nova-2-lite-v1:0
    • pdf_input
    1. Add a deployment named e2e-bedrock-document-d5333463bbcb that calls bedrock/us.amazon.nova-2-lite-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-document-d5333463bbcb
    View test on GitHub ↗
  29. ✓test_bedrock_converse_with_extended_thinking TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/converse/us.anthropic.claude-sonnet-4-6
    • function_calling
    • reasoning
    1. Add a deployment named e2e-chat-tool-loop-b152ad07f03d that calls bedrock/converse/us.anthropic.claude-sonnet-4-6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-b152ad07f03d
    View test on GitHub ↗
  30. ✓test_bedrock_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • bedrock
    • bedrock/amazon.titan-embed-text-v2:0
    1. Add a deployment named e2e-embeddings-bedrock-b85f917284b5 that calls bedrock/amazon.titan-embed-text-v2:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  31. ✓test_bedrock_image_generation_returns_image TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images
    • nonstream
    • bedrock
    • bedrock/amazon.nova-canvas-v1:0
    1. Add a deployment named e2e-bedrock-image-3286b711c6ab that calls bedrock/amazon.nova-canvas-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  32. ✓test_bedrock_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • prompt_caching
    1. Add a deployment named e2e-bedrock-cache-7509bd36bf24 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /chat/completions
    View test on GitHub ↗
  33. ✓test_bedrock_request_id_header_surfaces TestBedrockResponseHeaderstest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-headers-d6b070602dd3 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  34. ✓test_bedrock_request_id_header_surfaces_on_stream TestBedrockResponseHeaderstest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-headers-stream-1ded6f765148 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bedrock-headers-stream-1ded6f765148
    View test on GitHub ↗
  35. ✓test_bedrock_rerank_scores_top_n TestReranktest_rerank_e2e.py›
    • llm-translation
    • rerank
    • nonstream
    • bedrock
    • bedrock/arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0
    1. Add a deployment named e2e-bedrock-rerank-774dbff5b2a2 that calls bedrock/arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/rerank request to e2e-bedrock-rerank-774dbff5b2a2 for "What is the capital of the United States?"
    View test on GitHub ↗
  36. ✓test_bedrock_titan_embeds_token_array_input_as_its_decoded_text TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • bedrock
    • bedrock/amazon.titan-embed-text-v2:0
    1. Add a deployment named e2e-embeddings-titan-tokens-c7f7286ab631 that calls bedrock/amazon.titan-embed-text-v2:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  37. ✓test_bridged_stream_delivers_content_finish_reason_and_done TestResponsesBridgeChatCompletionsStreamingtest_responses_bridge_streaming_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.3-codex
    1. Add a deployment named e2e-bridge-stream-56b49867f497 that calls openai/gpt-5.3-codex
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bridge-stream-56b49867f497
    View test on GitHub ↗
  38. ✓test_bridged_stream_reassembles_tool_call TestResponsesBridgeChatCompletionsStreamingtest_responses_bridge_streaming_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.3-codex
    • function_calling
    1. Add a deployment named e2e-bridge-stream-61b83fd276e1 that calls openai/gpt-5.3-codex
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bridge-stream-61b83fd276e1
    View test on GitHub ↗
  39. ✓test_bridged_stream_shares_one_chunk_id TestResponsesBridgeChatCompletionsStreamingtest_responses_bridge_streaming_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.3-codex
    1. Add a deployment named e2e-bridge-stream-28cd964b6d0d that calls openai/gpt-5.3-codex
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bridge-stream-28cd964b6d0d
    View test on GitHub ↗
  40. ✓test_cache_read_tokens_bill_at_the_cache_read_rate TestTogetherChatCompletionstest_together_ai_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • together_ai
    • prompt_caching
    1. Add a deployment named e2e-together-e67dffbbbc5b that calls together_ai/deepseek-ai/DeepSeek-V4-Flash-0731
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-e67dffbbbc5b
    4. Send a request to /chat/completions
    5. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  41. ✓test_caller_completion_window_bills_its_rates TestSailResponsestest_sail_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-0f18be3df0dd that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  42. ✓test_calls_without_auth_is_denied TestRealtimeHttptest_realtime_http_e2e.py›
    • proxy-auth
    • realtime
    1. Send a request to /v1/realtime/calls
    View test on GitHub ↗
  43. ✓test_cancel_background_response[azure] TestBackgroundResponseCanceltest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-resp-cancel-0f0c4d341ec2 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  44. ✓test_cancel_background_response[openai] TestBackgroundResponseCanceltest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-resp-cancel-3e0ff7c9dab2 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  45. ✓test_cancel_background_streaming_response_by_streamed_id TestBackgroundResponseCanceltest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-resp-cancel-stream-1e715deb9d76 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  46. ✓test_chat_returns_real_completion[claude-haiku-4-5-anthropic] TestChatCompletionsRegressiontest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to claude-haiku-4-5
    View test on GitHub ↗
  47. ✓test_chat_returns_real_completion[gemini-2.5-flash-gemini] TestChatCompletionsRegressiontest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  48. ✓test_chat_returns_real_completion[gpt-5.5-openai] TestChatCompletionsRegressiontest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  49. ✓test_chat_stream_is_sse_and_ends_with_done TestChatStreamContracttest_chat_stream_contract_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.6
    1. Add a deployment named e2e-chat-stream-de8354149c65 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-chat-stream-de8354149c65
    View test on GitHub ↗
  50. ✓test_chat_template_kwargs_reach_together TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • together_ai/Qwen/Qwen3.5-9B
    • reasoning
    1. Add a deployment named e2e-together-6cd3224e26c1 that calls together_ai/Qwen/Qwen3.5-9B
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-6cd3224e26c1
    View test on GitHub ↗
  51. ✓test_client_secret_missing_auth_is_denied TestRealtimeHttptest_realtime_http_e2e.py›
    • proxy-auth
    • realtime
    1. Add a deployment named e2e-realtime-http-3f7ffd3d8f8e that calls openai/gpt-realtime
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/realtime/client_secrets
    View test on GitHub ↗
  52. ✓test_client_tool_call_marks_work_without_injection_points[azure-foundry] TestCacheControlInjectionToolCallstest_cache_control_injection_tool_calls_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    • function_calling
    • prompt_caching
    1. Add a deployment named e2e-cache-control-tool-calls-azure_foundry-41e6706a11e7 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  53. ✓test_client_tool_call_marks_work_without_injection_points[vertex] TestCacheControlInjectionToolCallstest_cache_control_injection_tool_calls_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/claude-sonnet-5
    • function_calling
    • prompt_caching
    1. Add a deployment named e2e-cache-control-tool-calls-vertex-8e5c0658730a that calls vertex_ai/claude-sonnet-5
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  54. ✓test_cohere_chat_returns_content TestCohereChattest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • cohere
    • cohere/command-r-08-2024
    1. Add a deployment named e2e-cohere-chat-3429b010dfd4 that calls cohere/command-r-08-2024
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-cohere-chat-3429b010dfd4
    View test on GitHub ↗
  55. ✓test_cohere_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • cohere
    • cohere/embed-v4.0
    1. Add a deployment named e2e-embeddings-cohere-3dd160e879c2 that calls cohere/embed-v4.0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  56. ✓test_container_file_lifecycle_through_the_gateway TestOpenAIContainerFilestest_containers_e2e.py›
    • llm-translation
    • containers
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  57. ✓test_contents_are_counted_by_the_provider_endpoint[gemini-2.5-flash-vertex] TestGeminiContentsTokenCountingtest_token_counter_gemini_contents_e2e.py›
    • llm-translation
    • count_tokens
    • vertex_ai
    • gemini-2.5-flash-vertex
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /utils/token_counter
    View test on GitHub ↗
  58. ✓test_contents_are_counted_by_the_provider_endpoint[gemini-2.5-flash] TestGeminiContentsTokenCountingtest_token_counter_gemini_contents_e2e.py›
    • llm-translation
    • count_tokens
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /utils/token_counter
    View test on GitHub ↗
  59. ✓test_converse_empty_messages_returns_client_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-2a3e9fa819e5 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-2a3e9fa819e5/converse
    View test on GitHub ↗
  60. ✓test_converse_invalid_model_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    1. Add a deployment named e2e-bedrock-native-c770308eae76 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/does-not-exist/converse
    View test on GitHub ↗
  61. ✓test_converse_missing_messages_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-f8adc6ef6fa6 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-f8adc6ef6fa6/converse
    View test on GitHub ↗
  62. ✓test_converse_output_format_returns_schema_json_text TestBedrockMessagestest_messages_bedrock_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • response_schema
    1. Add a deployment named e2e-messages-bedrock-1d9786263181 that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  63. ✓test_converse_returns_assistant TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-aa639090b2fe that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-aa639090b2fe/converse
    View test on GitHub ↗
  64. ✓test_converse_stream_returns_chunks TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • stream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-90a462a153ae that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-90a462a153ae/converse-stream
    View test on GitHub ↗
  65. ✓test_cost_header_and_spend_row_match_the_registry_price TestTogetherChatCompletionstest_together_ai_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • together_ai
    1. Add a deployment named e2e-together-b3b60cbd29f9 that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  66. ✓test_cost_header_matches_the_spend_log[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 with the prompt "Say hello. 1db00dd230b5" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  67. ✓test_cost_header_matches_the_spend_log[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f with the prompt "Say hello. 363c33ffc8dc" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  68. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 with the prompt "Say hello. ede4f0044f37" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  69. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 with the prompt "Say hello. 6c2a3896647d" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  70. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc with the prompt "Say hello. f8d2dfc09684" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  71. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 with the prompt "Say hello. 355cff60f53e" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  72. ✓test_cost_header_matches_the_spend_log[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 with the prompt "Say hello. 92ae41f7fead" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  73. ✓test_cost_header_matches_the_spend_log[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f with the prompt "Say hello. 413324310582" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  74. ✓test_cost_header_matches_the_spend_log[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 with the prompt "Say hello. 5b1a9f878318" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  75. ✓test_cost_header_matches_the_spend_log[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 with the prompt "Say hello. eae580439be1" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  76. ✓test_cost_header_matches_the_spend_log[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc with the prompt "Say hello. 0150527a1808" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  77. ✓test_cost_header_matches_the_spend_log[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 with the prompt "Say hello. 3ea970d23a5b" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  78. ✓test_cost_header_matches_the_spend_log[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 with the prompt "Say hello. 26a2ceb618d7" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  79. ✓test_cost_header_matches_the_spend_log[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f with the prompt "Say hello. 22ae53fca7fc" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  80. ✓test_cost_header_matches_the_spend_log[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 with the prompt "Say hello. fd07ad07c273" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  81. ✓test_cost_header_matches_the_spend_log[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 with the prompt "Say hello. 62ecb067ff37" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  82. ✓test_cost_header_matches_the_spend_log[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc with the prompt "Say hello. a6945fa83dce" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  83. ✓test_cost_header_matches_the_spend_log[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 with the prompt "Say hello. 32e0cd619e1d" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  84. ✓test_count_matches_billed_input_tokens_for_a_model_bedrock_runtime_cannot_count TestBedrockMessagesCountTokenstest_messages_count_tokens_bedrock_e2e.py›
    • llm-translation
    • count_tokens
    • bedrock
    • bedrock/global.anthropic.claude-opus-4-8
    1. Add a deployment named e2e-count-tokens-bedrock-b9412c4ad504 that calls bedrock/global.anthropic.claude-opus-4-8
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  85. ✓test_create_batch_missing_input_file_id_returns_error TestFilesBatchesContracttest_files_batches_contract_e2e.py›
    • llm-translation
    • batches
    • batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /v1/batches
    View test on GitHub ↗
  86. ✓test_create_client_secret TestRealtimeHttptest_realtime_http_e2e.py›
    • llm-translation
    • realtime
    • openai
    • openai/gpt-realtime
    1. Add a deployment named e2e-realtime-http-4454d0e76890 that calls openai/gpt-realtime
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /v1/realtime/client_secrets
    View test on GitHub ↗
  87. ✓test_create_list_retrieve_delete_lifecycle TestVectorStorestest_vector_stores_e2e.py›
    • llm-translation
    • vector_stores
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. POST /v1/vector_stores
    3. GET /v1/vector_stores/vs_6acb3ddd5ccc81919efe56c43aecfada
    4. GET /v1/vector_stores
    5. DELETE /v1/vector_stores/vs_6acb3ddd5ccc81919efe56c43aecfada
    View test on GitHub ↗
  88. ✓test_credential_backed_messages TestCredentialBackedMessagestest_credential_messages_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Save the provider credential e2e-cred-790fab51af4d
    2. Add a deployment named e2e-cred-messages-790fab51af4d that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  89. ✓test_custom_pricing_is_billed_at_configured_rate TestCustomPricingtest_custom_pricing_e2e.py›
    • cost-map
    • spend_reporting
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named custom-priced-flash-21e50060560b that calls gemini/gemini-2.5-flash
    3. Send a /chat/completions request to custom-priced-flash-21e50060560b
    4. GET /spend/logs
    View test on GitHub ↗
  90. ✓test_custom_pricing_is_isolated_from_sibling_deployment TestCustomPricingtest_custom_pricing_e2e.py›
    • cost-map
    • model_management
    1. Add a deployment named custom-priced-flash-b26c3452d404 that calls gemini/gemini-2.5-flash
    2. Add a deployment named base-flash-e61020f5124f that calls gemini/gemini-2.5-flash
    3. List the deployments from /model/info
    View test on GitHub ↗
  91. ✓test_deleted_response_is_no_longer_retrievable[azure] TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-resp-delete-460f35070ca5 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  92. ✓test_deleted_response_is_no_longer_retrievable[openai] TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-resp-delete-cdc188219e4f that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  93. ✓test_embeddings_call_logs_its_cost TestOpenAIPassthroughSpendtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • openai
    • text-embedding-3-small
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/embeddings request to text-embedding-3-small through /openai_passthrough for "cost this sentence 37a6eb1097d9"
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  94. ✓test_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • openai
    • openai/text-embedding-3-small
    1. Add a deployment named e2e-embeddings-17fb21747244 that calls openai/text-embedding-3-small
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  95. ✓test_empty_image_returns_error TestImageEdittest_image_edits_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1
    1. Add a deployment named e2e-image-edit-4cc321fe47d9 that calls openai/gpt-image-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  96. ✓test_empty_input_returns_client_error TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-responses-val-65a08e7c908a that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/responses
    View test on GitHub ↗
  97. ✓test_empty_messages_returns_client_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-a54b5ccde269 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  98. ✓test_empty_prompt_returns_client_error TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1-mini
    1. Add a deployment named e2e-image-6d281142267d that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/images/generations
    View test on GitHub ↗
  99. ✓test_empty_prompt_returns_error TestImageEdittest_image_edits_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1
    1. Add a deployment named e2e-image-edit-b32bc5b6f53b that calls openai/gpt-image-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  100. ✓test_file_attach_poll_and_search TestVectorStorestest_vector_stores_e2e.py›
    • llm-translation
    • vector_stores
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload vs_doc.txt to /v1/files
    3. POST /v1/vector_stores
    4. POST /v1/vector_stores/vs_6acb3de91238819195f12c276f1ec9d6/files
    5. GET /v1/vector_stores/vs_6acb3de91238819195f12c276f1ec9d6/files/file-4J1ZHAHNt5EC41sYXSnTCa
    6. POST /v1/vector_stores/vs_6acb3de91238819195f12c276f1ec9d6/search
    7. DELETE /v1/vector_stores/vs_6acb3de91238819195f12c276f1ec9d6/files/file-4J1ZHAHNt5EC41sYXSnTCa
    View test on GitHub ↗
  101. ✓test_flagged_model_keeps_prompt_cache_across_system_reminder TestAnthropicChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-opus-4-8
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-chat-midsys-890a8ba2e1af that calls anthropic/claude-opus-4-8
    2. Generate a virtual key with models: e2e-chat-midsys-890a8ba2e1af and user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  102. ✓test_flagged_model_keeps_prompt_cache_across_system_reminder TestBedrockInvokeChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-sonnet-5
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-chat-midsys-523309278c9e that calls bedrock/invoke/us.anthropic.claude-sonnet-5
    2. Generate a virtual key with models: e2e-chat-midsys-523309278c9e and user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  103. ✓test_gemini TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • gemini
    • gemini/gemini-3.5-flash-lite
    • function_calling
    1. Add a deployment named e2e-chat-tool-loop-e2e91397cacc that calls gemini/gemini-3.5-flash-lite
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-e2e91397cacc
    View test on GitHub ↗
  104. ✓test_gemini_chat_returns_content_and_logs_cost TestGeminiChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Add a deployment named e2e-gemini-chat-af2663d239db that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-gemini-chat-af2663d239db
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  105. ✓test_gemini_passthrough_nonstreaming_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a Gemini generateContent request to gemini-2.5-flash through /gemini
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  106. ✓test_gemini_passthrough_streaming_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a Gemini streamGenerateContent request to gemini-2.5-flash through /gemini
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  107. ✓test_gemini_passthrough_tool_call_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • gemini
    • gemini-2.5-flash
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a Gemini generateContent request to gemini-2.5-flash through /gemini
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  108. ✓test_generate_content_returns_response_cost_header TestGoogleNativeGenerateContenttest_google_native_e2e.py›
    • llm-translation
    • google_genai
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-google-native-c93a194c9865 that calls gemini/gemini-2.5-flash
    3. Send a request to /v1beta/models/e2e-google-native-c93a194c9865:generateContent
    View test on GitHub ↗
  109. ✓test_image_edit_returns_image TestImageEdittest_image_edits_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1
    1. Add a deployment named e2e-image-edit-c40442e7c57f that calls openai/gpt-image-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  110. ✓test_image_generation_returns_image TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1-mini
    1. Add a deployment named e2e-image-5cd71f1fd087 that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  111. ✓test_injection_points_respect_cap_with_tool_call_marks[azure-foundry] TestCacheControlInjectionToolCallstest_cache_control_injection_tool_calls_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    • function_calling
    • prompt_caching
    1. Add a deployment named e2e-cache-control-tool-calls-azure_foundry-4c597f21141f that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  112. ✓test_injection_points_respect_cap_with_tool_call_marks[vertex] TestCacheControlInjectionToolCallstest_cache_control_injection_tool_calls_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/claude-sonnet-5
    • function_calling
    • prompt_caching
    1. Add a deployment named e2e-cache-control-tool-calls-vertex-5400aa2c088d that calls vertex_ai/claude-sonnet-5
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  113. ✓test_input_items_list_the_stored_prompt TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-resp-items-c00d89e5c35f that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  114. ✓test_invalid_chunking_returns_error TestVectorStorestest_vector_stores_e2e.py›
    • llm-translation
    • vector_stores
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /v1/vector_stores
    View test on GitHub ↗
  115. ✓test_invalid_max_completion_tokens_return_client_errors TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-84d114ade68e that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  116. ✓test_invalid_n_returns_client_error TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1-mini
    1. Add a deployment named e2e-image-12c18978d965 that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/images/generations
    View test on GitHub ↗
  117. ✓test_invalid_role_returns_client_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-8a291676c7f4 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  118. ✓test_invalid_size_returns_client_error TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1-mini
    1. Add a deployment named e2e-image-26cc55392e41 that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/images/generations
    View test on GitHub ↗
  119. ✓test_invalid_temperatures_return_client_errors TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-b116e54a4bc6 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  120. ✓test_invoke_invalid_temperature_returns_client_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-327a11f02cbb that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-327a11f02cbb/invoke
    View test on GitHub ↗
  121. ✓test_invoke_missing_max_tokens_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-aef3ff5f1e14 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-aef3ff5f1e14/invoke
    View test on GitHub ↗
  122. ✓test_invoke_missing_messages_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-06371232a6c4 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-06371232a6c4/invoke
    View test on GitHub ↗
  123. ✓test_invoke_regional_id_completes TestBedrockInvokeRegionalModelIdstest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-invoke-0c3bec19a3b5 that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-invoke-0c3bec19a3b5
    View test on GitHub ↗
  124. ✓test_invoke_regional_id_streams TestBedrockInvokeRegionalModelIdstest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-invoke-stream-e5475ae1f43b that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bedrock-invoke-stream-e5475ae1f43b
    View test on GitHub ↗
  125. ✓test_invoke_returns_message TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-4dec4791bdc5 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-4dec4791bdc5/invoke
    View test on GitHub ↗
  126. ✓test_invoke_stream_returns_chunks TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • stream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-d94c224c51d4 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-d94c224c51d4/invoke-with-response-stream
    View test on GitHub ↗
  127. ✓test_messages_logs_cost_matching_the_response_header TestAnthropicMessagestest_messages_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-messages-cost-6ffcd1fc3f96 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  128. ✓test_messages_output_format_returns_schema_json TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • response_schema
    1. Add a deployment named e2e-messages-ff0dbfb48938 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  129. ✓test_messages_returns_completion TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-messages-723dd76621dc that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  130. ✓test_messages_streams_completion TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-messages-aafe8690b48d that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  131. ✓test_messages_tool_use TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-messages-629717440513 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  132. ✓test_missing_file_returns_error TestAudioTranscriptionstest_audio_transcriptions_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-transcribe
    1. Add a deployment named e2e-transcribe-a3f3453eb94e that calls openai/gpt-4o-mini-transcribe
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload empty.wav to /v1/audio/transcriptions
    View test on GitHub ↗
  133. ✓test_missing_input_returns_error TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    1. Add a deployment named e2e-embeddings-missin-c8b0e84bff62 that calls openai/text-embedding-3-small
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /embeddings
    View test on GitHub ↗
  134. ✓test_missing_messages_returns_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-2e70662078e5 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  135. ✓test_missing_model_returns_client_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    1. Add a deployment named e2e-chat-sec-515274f09a5a that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  136. ✓test_missing_model_returns_client_error TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /embeddings
    View test on GitHub ↗
  137. ✓test_missing_model_returns_client_error TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /v1/responses
    View test on GitHub ↗
  138. ✓test_missing_model_returns_error TestAudioTranscriptionstest_audio_transcriptions_e2e.py›
    • llm-translation
    • audio
    1. Add a deployment named e2e-transcribe-ef47ea81c767 that calls openai/gpt-4o-mini-transcribe
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload weather_question_24k.wav to /v1/audio/transcriptions
    View test on GitHub ↗
  139. ✓test_missing_model_returns_error TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages
    1. Add a deployment named e2e-messages-e60beb09bb40 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/messages
    View test on GitHub ↗
  140. ✓test_mistral TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • mistral
    • mistral/mistral-medium-3.5
    • function_calling
    1. Add a deployment named e2e-chat-tool-loop-f0eaa7e98196 that calls mistral/mistral-medium-3.5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-f0eaa7e98196
    View test on GitHub ↗
  141. ✓test_mistral_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • mistral
    • mistral/mistral-embed
    1. Add a deployment named e2e-embeddings-mistral-e2f96be3226e that calls mistral/mistral-embed
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  142. ✓test_model_info_reports_custom_pricing TestCustomPricingtest_custom_pricing_e2e.py›
    • cost-map
    • model_management
    1. Add a deployment named custom-priced-flash-b10c8253a6ab that calls gemini/gemini-2.5-flash
    2. List the deployments from /model/info
    View test on GitHub ↗
  143. ✓test_moderations_flags_violent_content TestModerationstest_moderations_e2e.py›
    • llm-translation
    • moderations
    • nonstream
    • openai
    • openai/omni-moderation-latest
    1. Add a deployment named e2e-moderation-e350ee1e3746 that calls openai/omni-moderation-latest
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  144. ✓test_moderations_passes_benign_content TestModerationstest_moderations_e2e.py›
    • llm-translation
    • moderations
    • nonstream
    • openai
    • openai/omni-moderation-latest
    1. Add a deployment named e2e-moderation-c83a17831075 that calls openai/omni-moderation-latest
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  145. ✓test_multi_turn_history_is_honored TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-8e9728517dcd that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-sec-8e9728517dcd
    View test on GitHub ↗
  146. ✓test_nova_sonic_response_create_completes TestNovaSonicRealtimerealtime/test_realtime_bedrock_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • bedrock
    • bedrock/amazon.nova-2-sonic-v1:0
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-nova-sonic-12198cfdae05 that calls bedrock/amazon.nova-2-sonic-v1:0
    3. Open a /v1/realtime websocket session to e2e-nova-sonic-12198cfdae05
    4. Wait for a session.created event on the realtime websocket
    5. Send the realtime event session.update over the websocket
    6. Wait for a session.updated event on the realtime websocket
    7. Send the realtime event conversation.item.create over the websocket
    8. Send the realtime event response.create over the websocket
    9. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  147. ✓test_nova_sonic_voice_turn_shows_each_sentence_once TestNovaSonicAssistantTextrealtime/test_realtime_bedrock_duplicate_text_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • bedrock
    • bedrock/amazon.nova-2-sonic-v1:0
    • audio_input
    • audio_output
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-nova-sonic-dupe-25ef003cc025 that calls bedrock/amazon.nova-2-sonic-v1:0
    3. Open a /v1/realtime websocket session to e2e-nova-sonic-dupe-25ef003cc025
    4. Wait for a session.created event on the realtime websocket
    5. Send the realtime event session.update over the websocket
    6. Wait for a session.updated event on the realtime websocket
    7. Stream the spoken question as mic chunks, then silence until the turn goes idle
    View test on GitHub ↗
  148. ✓test_nova_stream_relays_text_usage_and_stop TestBedrockMessagestest_messages_bedrock_e2e.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • bedrock/us.amazon.nova-2-lite-v1:0
    1. Add a deployment named e2e-messages-bedrock-37d9e6d0584e that calls bedrock/us.amazon.nova-2-lite-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  149. ✓test_ocr_reads_the_document_and_bills_its_pages[azure-ai-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • azure_ai
    • azure_ai/mistral-document-ai-2512
    1. Add a deployment named ocr-content-azure-ai-image-787cdd97597d that calls azure_ai/mistral-document-ai-2512
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/ocr
    View test on GitHub ↗
  150. ✓test_ocr_reads_the_document_and_bills_its_pages[cohere-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • cohere
    • cohere/parse-v5.0
    1. Add a deployment named ocr-content-cohere-image-a17641f5d54a that calls cohere/parse-v5.0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/ocr
    View test on GitHub ↗
  151. ✓test_ocr_reads_the_document_and_bills_its_pages[mistral-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • mistral
    • mistral/mistral-ocr-latest
    1. Add a deployment named ocr-content-mistral-image-28010c2efc68 that calls mistral/mistral-ocr-latest
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/ocr
    View test on GitHub ↗
  152. ✓test_ocr_reads_the_document_and_bills_its_pages[mistral-pdf] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • mistral
    • mistral/mistral-ocr-latest
    1. Add a deployment named ocr-content-mistral-pdf-776630fac894 that calls mistral/mistral-ocr-latest
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/ocr
    View test on GitHub ↗
  153. ✓test_ocr_reads_the_document_and_bills_its_pages[vertex-mistral-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • vertex_ai
    • vertex_ai/mistral-ocr-2505
    1. Add a deployment named ocr-content-vertex-mistral-image-6f4962df3406 that calls vertex_ai/mistral-ocr-2505
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/ocr
    View test on GitHub ↗
  154. ✓test_openai_chat_logs_cost TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    1. Add a deployment named e2e-openai-cost-9b366debcc2a that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-cost-9b366debcc2a
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  155. ✓test_openai_chat_reasoning_reports_reasoning_tokens TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • reasoning
    1. Add a deployment named e2e-openai-reasoning-ec042e62aa46 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-reasoning-ec042e62aa46
    View test on GitHub ↗
  156. ✓test_openai_chat_returns_tool_call TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • function_calling
    1. Add a deployment named e2e-openai-tool-7d8036e36a29 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-tool-7d8036e36a29
    View test on GitHub ↗
  157. ✓test_openai_chat_streams_real_content TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.6
    1. Add a deployment named e2e-openai-chat-55b58c542adf that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-openai-chat-55b58c542adf
    View test on GitHub ↗
  158. ✓test_openai_chat_streams_tool_call TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.6
    • function_calling
    1. Add a deployment named e2e-openai-tool-stream-6e9bfc3ab82b that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-openai-tool-stream-6e9bfc3ab82b
    View test on GitHub ↗
  159. ✓test_openai_chat_structured_output_conforms_to_schema TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • response_schema
    1. Add a deployment named e2e-openai-schema-14b46b092599 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-schema-14b46b092599
    View test on GitHub ↗
  160. ✓test_openai_chat_vision_describes_image TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o
    • vision
    1. Add a deployment named e2e-openai-vision-4a998ed5dab6 that calls openai/gpt-4o
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-vision-4a998ed5dab6
    View test on GitHub ↗
  161. ✓test_openai_family_model_id_completes_with_max_tokens TestBedrockOpenAIFamilyDefaultRoutetest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/global.openai.gpt-6-sol
    1. Add a deployment named e2e-bedrock-openai-family-d1526d1b7336 that calls bedrock/global.openai.gpt-6-sol
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-openai-family-d1526d1b7336
    View test on GitHub ↗
  162. ✓test_openai_prefix_chat_returns_completion_and_logs_its_cost TestOpenAIProviderPrefixChattest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/chat/completions request to gpt-5.5 through /openai
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  163. ✓test_openai_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • prompt_caching
    1. Add a deployment named e2e-openai-cache-7309148bedc5 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-cache-7309148bedc5
    View test on GitHub ↗
  164. ✓test_openai_service_tier_is_echoed TestServiceTiertest_provider_features_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-service-tier-53cc87fa0dd3 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-service-tier-53cc87fa0dd3
    View test on GitHub ↗
  165. ✓test_output_format_returns_schema_json TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    • response_schema
    1. Add a deployment named e2e-azure-foundry-messages-32409f59449b that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-32409f59449b and user id: e2e-test-user
    View test on GitHub ↗
  166. ✓test_passthrough_denies_model_outside_key_allowlist TestPassthroughModelAllowlisttest_passthrough_e2e.py›
    • proxy-auth
    • passthrough
    1. Generate a virtual key with models: gemini-2.5-flash
    2. Send a /v1/messages request to claude-haiku-4-5 through /anthropic with streaming set to False
    View test on GitHub ↗
  167. ✓test_passthrough_prefix_lists_batches_from_openai TestOpenAIPassthroughPrefixtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. List batches from /openai_passthrough/v1/batches
    View test on GitHub ↗
  168. ✓test_passthrough_prefix_uploads_a_file_to_openai TestOpenAIPassthroughPrefixtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload e2e-passthrough-batch.jsonl to /openai_passthrough/v1/files
    View test on GitHub ↗
  169. ✓test_pipecat_tool_smoke[azure]realtime/test_realtime_pipecat_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • azure
    • azure/gpt-realtime
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  170. ✓test_pipecat_tool_smoke[gemini]realtime/test_realtime_pipecat_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • gemini
    • gemini/gemini-3.1-flash-live-preview
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  171. ✓test_pipecat_tool_smoke[openai]realtime/test_realtime_pipecat_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • openai
    • openai/gpt-realtime-2
    • function_calling
    1. Add a realtime deployment that calls openai/gpt-realtime-2
    2. Add a realtime deployment that calls azure/gpt-realtime
    3. Add a realtime deployment that calls gemini/gemini-3.1-flash-live-preview
    4. Add a realtime deployment that calls vertex_ai/gemini-live-2.5-flash-native-audio
    5. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  172. ✓test_plain_call_returns_a_message TestSailMessagestest_sail_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-87dc043ca4fe that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  173. ✓test_polly_generative_voice_returns_mp3 TestAwsPollySpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • aws_polly
    • aws_polly/generative
    1. Add a deployment named e2e-speech-polly-4c3a3ac9696d that calls aws_polly/generative
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  174. ✓test_realtime_upgrade_reaches_openai_through_the_passthrough_prefix TestOpenAIPassthroughWebsockettest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • websocket
    • openai
    • gpt-realtime-2
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a websocket to /openai_passthrough/v1/realtime and wait for its first event
    View test on GitHub ↗
  175. ✓test_reasoner_returns_reasoning_by_default TestDeepSeekReasoningDisabletest_deepseek_reasoning_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • deepseek
    • deepseek/deepseek-v4-pro
    • reasoning
    1. Add a deployment named e2e-deepseek-reasoner-07027df448e0 that calls deepseek/deepseek-v4-pro
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-deepseek-reasoner-07027df448e0
    View test on GitHub ↗
  176. ✓test_reasoning_effort_none_disables_reasoning TestDeepSeekReasoningDisabletest_deepseek_reasoning_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • deepseek
    • deepseek/deepseek-v4-pro
    • reasoning
    1. Add a deployment named e2e-deepseek-reasoner-75f026798c91 that calls deepseek/deepseek-v4-pro
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-deepseek-reasoner-75f026798c91
    View test on GitHub ↗
  177. ✓test_reasoning_effort_none_reaches_together TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • together_ai/Qwen/Qwen3.5-9B
    • reasoning
    1. Add a deployment named e2e-together-06979c8a0c06 that calls together_ai/Qwen/Qwen3.5-9B
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-06979c8a0c06
    View test on GitHub ↗
  178. ✓test_reasoning_items_replay_into_structured_output_after_tool_call TestResponsesOpenAIHostedFeaturestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    • reasoning
    • response_schema
    1. Add a deployment named e2e-responses-reasoning-ab0fb2c50218 that calls openai/gpt-5.4-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  179. ✓test_reasoning_streams_as_reasoning_content_deltas TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • together_ai
    • reasoning
    1. Add a deployment named e2e-together-808caa3865b5 that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-together-808caa3865b5
    View test on GitHub ↗
  180. ✓test_reasoning_surfaces_as_reasoning_content TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • reasoning
    1. Read the model cost map
    2. Add a deployment named e2e-together-7aa33c6db1e0 that calls together_ai/zai-org/GLM-5.3-Flash
    3. Generate a virtual key with user id: e2e-test-user
    4. Send a /chat/completions request to e2e-together-7aa33c6db1e0
    View test on GitHub ↗
  181. ✓test_replayed_reasoning_content_reaches_together TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • together_ai/MiniMaxAI/MiniMax-M3
    • reasoning
    1. Add a deployment named e2e-together-6718474ba851 that calls together_ai/MiniMaxAI/MiniMax-M3
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-6718474ba851
    View test on GitHub ↗
  182. ✓test_reply_carries_assistant_text_and_usage[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  183. ✓test_reply_carries_assistant_text_and_usage[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  184. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Start a provider edge server
    2. Add a deployment named e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 that calls openai/gpt-4o-mini
    3. Save the provider credential e2e-matrix-gpt-4o-mini-1006c1974cf2
    4. Add a deployment named e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 that calls openai/gpt-4o-mini
    5. Add a deployment named e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc that calls openai/gpt-5.4-mini
    6. Save the provider credential e2e-matrix-gpt-5.4-mini-7f34bcf39c53
    7. Add a deployment named e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 that calls openai/gpt-5.4-mini
    8. Add a deployment named e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 that calls anthropic/claude-haiku-4-5
    9. Save the provider credential e2e-matrix-claude-haiku-4-5-d0759446514f
    10. Add a deployment named e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f that calls anthropic/claude-haiku-4-5
    11. Generate a virtual key with user id: e2e-test-user
    12. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  185. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  186. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  187. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  188. ✓test_reply_carries_assistant_text_and_usage[chat_completions-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Add a deployment named e2e-ollama-ollama_chat-cfc984df5765 that calls ollama_chat/gemma4:31b
    2. Add a deployment named e2e-ollama-ollama-74e436802975 that calls ollama/gemma4:31b
    3. Generate a virtual key with user id: e2e-test-user
    4. Send a /chat/completions request to e2e-ollama-ollama_chat-cfc984df5765 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  189. ✓test_reply_carries_assistant_text_and_usage[chat_completions-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ollama-ollama-74e436802975 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  190. ✓test_reply_carries_assistant_text_and_usage[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  191. ✓test_reply_carries_assistant_text_and_usage[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  192. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  193. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  194. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  195. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  196. ✓test_reply_carries_assistant_text_and_usage[messages-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama_chat-cfc984df5765 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  197. ✓test_reply_carries_assistant_text_and_usage[messages-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama-74e436802975 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  198. ✓test_reply_carries_assistant_text_and_usage[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  199. ✓test_reply_carries_assistant_text_and_usage[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  200. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  201. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  202. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  203. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  204. ✓test_reply_carries_assistant_text_and_usage[responses-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama_chat-cfc984df5765 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  205. ✓test_reply_carries_assistant_text_and_usage[responses-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama-74e436802975 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  206. ✓test_required_tool_arguments_and_correlated_result[nonstream] TestOpenAIMessagesToolContinuationtest_messages_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.6
    • function_calling
    1. Add a deployment named e2e-bridge-tool-826233342259 that calls openai/gpt-5.6
    2. Generate a virtual key with models: e2e-bridge-tool-826233342259 and user id: e2e-test-user
    View test on GitHub ↗
  207. ✓test_required_tool_arguments_and_correlated_result[stream] TestOpenAIMessagesToolContinuationtest_messages_e2e.py›
    • llm-translation
    • messages
    • stream
    • openai
    • openai/gpt-5.6
    • function_calling
    1. Add a deployment named e2e-bridge-tool-d38363d90ea7 that calls openai/gpt-5.6
    2. Generate a virtual key with models: e2e-bridge-tool-d38363d90ea7 and user id: e2e-test-user
    View test on GitHub ↗
  208. ✓test_rerank_scores_top_n TestReranktest_rerank_e2e.py›
    • llm-translation
    • rerank
    • nonstream
    • cohere
    • cohere/rerank-v3.5
    1. Add a deployment named e2e-rerank-51465ea202c7 that calls cohere/rerank-v3.5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/rerank request to e2e-rerank-51465ea202c7 for "What is the capital of the United States?"
    View test on GitHub ↗
  209. ✓test_response_format_json_schema_shapes_the_reply TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • together_ai/Qwen/Qwen3.5-9B
    • response_schema
    1. Add a deployment named e2e-together-312136e8a6f8 that calls together_ai/Qwen/Qwen3.5-9B
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-312136e8a6f8
    View test on GitHub ↗
  210. ✓test_responses_anthropic_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-responses-4efaf28f473a that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  211. ✓test_responses_anthropic_returns_function_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-responses-eba219d1ebf5 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  212. ✓test_responses_anthropic_strict_array_schema_tool_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-responses-eab4e75bf65a that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  213. ✓test_responses_anthropic_tool_output_continues_with_previous_response_id TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-responses-fc33880b53f7 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  214. ✓test_responses_azure_openai_preview_api_version_accepts_truncation TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-responses-azure-preview-d012f3a754aa that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  215. ✓test_responses_azure_openai_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-responses-azure-openai-5cea7cf1c03d that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  216. ✓test_responses_azure_openai_returns_function_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    • function_calling
    1. Add a deployment named e2e-responses-azure-openai-tool-ceeb6544fbf0 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  217. ✓test_responses_azure_openai_streaming_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • stream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-responses-azure-stream-02600b5446c1 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  218. ✓test_responses_bedrock_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-responses-2622afe20219 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  219. ✓test_responses_bedrock_returns_function_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • function_calling
    1. Add a deployment named e2e-responses-a10ad6a5c5a7 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  220. ✓test_responses_compact_returns_compacted_conversation TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-responses-compact-4de142dd7cbd that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  221. ✓test_responses_context_management_compacts_server_side TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-responses-context-compaction-dd92576b2439 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  222. ✓test_responses_gemini_replays_legacy_function_call_output TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    • function_calling
    1. Add a deployment named e2e-responses-gemini-tool-da28609b0de8 that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  223. ✓test_responses_gemini_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Add a deployment named e2e-responses-gemini-4d3c1a3c3cc4 that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  224. ✓test_responses_gemini_streaming_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • stream
    • gemini
    • gemini/gemini-2.5-flash
    1. Add a deployment named e2e-responses-gemini-a5d40ee09d60 that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  225. ✓test_responses_logs_cost TestResponsestest_responses_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-responses-025bc352cadc that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  226. ✓test_responses_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-responses-cc4468ae77c5 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  227. ✓test_responses_returns_function_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Add a deployment named e2e-responses-1e53ccd09d76 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  228. ✓test_responses_streaming_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-responses-44b53b9e573a that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  229. ✓test_responses_upgrade_is_accepted_on_the_openai_prefix TestOpenAIPassthroughWebsockettest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • websocket
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a websocket to /openai/v1/responses and wait for its first event
    View test on GitHub ↗
  230. ✓test_responses_vertex_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    1. Add a deployment named e2e-responses-vertex-c56a9d58c5cb that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  231. ✓test_responses_vertex_returns_function_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    • function_calling
    1. Add a deployment named e2e-responses-vertex-tool-62a80c80d069 that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  232. ✓test_responses_vision_describes_image TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o
    • vision
    1. Add a deployment named e2e-responses-dbbc6e78686c that calls openai/gpt-4o
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  233. ✓test_retrieve_invalid_batch_id_returns_error TestFilesBatchesContracttest_files_batches_contract_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. GET /v1/batches/invalid-batch-id
    View test on GitHub ↗
  234. ✓test_rust_ocr_response[azure-ai] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • azure_ai
    • azure_ai/mistral-document-ai-2512
    1. Add a deployment named rust-ocr-azure-ai-fe4951929529 that calls azure_ai/mistral-document-ai-2512
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-azure-ai-fe4951929529
    View test on GitHub ↗
  235. ✓test_rust_ocr_response[azure-document-intelligence] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • azure_ai
    • azure_ai/doc-intelligence/prebuilt-layout
    1. Add a deployment named rust-ocr-azure-document-intelligence-f728c805f729 that calls azure_ai/doc-intelligence/prebuilt-layout
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-azure-document-intelligence-f728c805f729
    View test on GitHub ↗
  236. ✓test_rust_ocr_response[mistral] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • mistral
    • mistral/mistral-ocr-latest
    1. Add a deployment named rust-ocr-mistral-fec5b25124ae that calls mistral/mistral-ocr-latest
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-mistral-fec5b25124ae
    View test on GitHub ↗
  237. ✓test_rust_ocr_response[vertex-mistral] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • vertex_ai
    • vertex_ai/mistral-ocr-2505
    1. Add a deployment named rust-ocr-vertex-mistral-98a1835b3664 that calls vertex_ai/mistral-ocr-2505
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-vertex-mistral-98a1835b3664
    View test on GitHub ↗
  238. ✓test_service_account_key_reads_container_file_by_native_id TestAzureContainerFilestest_containers_e2e.py›
    • llm-translation
    • containers
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-containers-decoy-a54729287562 that calls azure/gpt-5.4-nano
    2. Add a deployment named e2e-containers-a54729287562 that calls azure/gpt-5.4-nano
    3. Create a team with team alias: e2e-containers-a54729287562 and models: e2e-containers-a54729287562
    4. Create an internal user with user email: e2e-containers-a54729287562@example.com and user role: internal_user
    5. Add a user to the team with /team/member_add
    6. POST /key/service-account/generate
    View test on GitHub ↗
  239. ✓test_service_account_key_reads_container_file_created_by_a_streamed_response TestAzureContainerFilestest_containers_e2e.py›
    • llm-translation
    • containers
    • stream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-containers-decoy-cbf584c8e397 that calls azure/gpt-5.4-nano
    2. Add a deployment named e2e-containers-cbf584c8e397 that calls azure/gpt-5.4-nano
    3. Create a team with team alias: e2e-containers-cbf584c8e397 and models: e2e-containers-cbf584c8e397
    4. Create an internal user with user email: e2e-containers-cbf584c8e397@example.com and user role: internal_user
    5. Add a user to the team with /team/member_add
    6. POST /key/service-account/generate
    View test on GitHub ↗
  240. ✓test_service_tier_bills_the_matching_completion_window[auto-base] TestSailChatCompletionstest_sail_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-cf3e1b10ff7c that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  241. ✓test_service_tier_bills_the_matching_completion_window[balanced-balanced] TestSailChatCompletionstest_sail_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-33585bddfa8b that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  242. ✓test_shell_tool_stream_surfaces_shell_call_and_its_output TestResponsesOpenAIHostedFeaturestest_responses_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-responses-shell-3dc989e567f5 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  243. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 with the prompt "Say hello."
    View test on GitHub ↗
  244. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f with the prompt "Say hello."
    View test on GitHub ↗
  245. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 with the prompt "Say hello."
    View test on GitHub ↗
  246. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 with the prompt "Say hello."
    View test on GitHub ↗
  247. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc with the prompt "Say hello."
    View test on GitHub ↗
  248. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 with the prompt "Say hello."
    View test on GitHub ↗
  249. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-ollama-ollama_chat-cfc984df5765 with the prompt "Say hello."
    View test on GitHub ↗
  250. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-ollama-ollama-74e436802975 with the prompt "Say hello."
    View test on GitHub ↗
  251. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 with the prompt "Say hello."
    View test on GitHub ↗
  252. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f with the prompt "Say hello."
    View test on GitHub ↗
  253. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 with the prompt "Say hello."
    View test on GitHub ↗
  254. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 with the prompt "Say hello."
    View test on GitHub ↗
  255. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc with the prompt "Say hello."
    View test on GitHub ↗
  256. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 with the prompt "Say hello."
    View test on GitHub ↗
  257. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-ollama-ollama_chat-cfc984df5765 with the prompt "Say hello."
    View test on GitHub ↗
  258. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-ollama-ollama-74e436802975 with the prompt "Say hello."
    View test on GitHub ↗
  259. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 with the prompt "Say hello."
    View test on GitHub ↗
  260. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f with the prompt "Say hello."
    View test on GitHub ↗
  261. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 with the prompt "Say hello."
    View test on GitHub ↗
  262. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 with the prompt "Say hello."
    View test on GitHub ↗
  263. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc with the prompt "Say hello."
    View test on GitHub ↗
  264. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 with the prompt "Say hello."
    View test on GitHub ↗
  265. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-ollama-ollama_chat-cfc984df5765 with the prompt "Say hello."
    View test on GitHub ↗
  266. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-ollama-ollama-74e436802975 with the prompt "Say hello."
    View test on GitHub ↗
  267. ✓test_stream_generate_content_frames_sse_the_way_google_sdks_expect TestGoogleNativeGenerateContenttest_google_native_e2e.py›
    • llm-translation
    • google_genai
    • stream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-google-native-eec04ac34a50 that calls gemini/gemini-2.5-flash
    3. Send a request to /v1beta/models/e2e-google-native-eec04ac34a50:streamGenerateContent
    View test on GitHub ↗
  268. ✓test_streamed_response_can_be_deleted[azure] TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • stream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-resp-delete-stream-64c02f16efd4 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  269. ✓test_streamed_response_can_be_deleted[openai] TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-resp-delete-stream-7d2e3086bd74 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  270. ✓test_streamed_responses_call_logs_its_cost TestOpenAIPassthroughSpendtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • stream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to gpt-5.5 through /openai_passthrough with streaming set to True
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  271. ✓test_streams_text_deltas TestTogetherMessagestest_together_ai_e2e.py›
    • llm-translation
    • messages
    • stream
    • together_ai
    1. Add a deployment named e2e-together-54004a6b28b4 that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /v1/messages request to e2e-together-54004a6b28b4
    View test on GitHub ↗
  272. ✓test_success_response_matches_chat_completion_contract TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-479826e5c7a8 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  273. ✓test_temperature_boundaries_succeed TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-4d824a62d69c that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  274. ✓test_text_completion_returns_text TestCompletionsEndpointtest_completions_endpoint_e2e.py›
    • llm-translation
    • completions
    • nonstream
    • openai
    • openai/gpt-5.4-nano
    1. Add a deployment named e2e-completions-cd32f4536719 that calls openai/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  275. ✓test_text_conversation[azure]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • azure
    • azure/gpt-realtime
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to azure-realtime-8ded514b55ac
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  276. ✓test_text_conversation[gemini]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • gemini
    • gemini/gemini-3.1-flash-live-preview
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to gemini-realtime-e759ffc26d6d
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  277. ✓test_text_conversation[openai]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • openai
    • openai/gpt-realtime-2
    1. Add a realtime deployment that calls openai/gpt-realtime-2
    2. Add a realtime deployment that calls azure/gpt-realtime
    3. Add a realtime deployment that calls gemini/gemini-3.1-flash-live-preview
    4. Add a realtime deployment that calls vertex_ai/gemini-live-2.5-flash-native-audio
    5. Generate a virtual key with user id: e2e-test-user
    6. Open a /v1/realtime websocket session to openai-realtime-b7ac866d7e7e
    7. Wait for a session.created event on the realtime websocket
    8. Send the realtime event session.update over the websocket
    9. Wait for a session.updated event on the realtime websocket
    10. Send the realtime event conversation.item.create over the websocket
    11. Send the realtime event response.create over the websocket
    12. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  278. ✓test_text_conversation[vertex_ai]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • vertex_ai
    • vertex_ai/gemini-live-2.5-flash-native-audio
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to vertex-realtime-5d5b3994fc7a
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  279. ✓test_thinking_disabled_disables_reasoning TestDeepSeekReasoningDisabletest_deepseek_reasoning_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • deepseek
    • deepseek/deepseek-v4-pro
    • reasoning
    1. Add a deployment named e2e-deepseek-reasoner-d5810741a2b9 that calls deepseek/deepseek-v4-pro
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-deepseek-reasoner-d5810741a2b9
    View test on GitHub ↗
  280. ✓test_tool_call_is_returned TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • function_calling
    1. Add a deployment named e2e-together-ff28abd34492 that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-ff28abd34492
    View test on GitHub ↗
  281. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set
    View test on GitHub ↗
  282. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather to
    View test on GitHub ↗
  283. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to T
    View test on GitHub ↗
  284. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool us
    View test on GitHub ↗
  285. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to
    View test on GitHub ↗
  286. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool u
    View test on GitHub ↗
  287. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ollama-ollama_chat-cfc984df5765 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  288. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ollama-ollama-74e436802975 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  289. ✓test_tool_call_is_returned_named_and_addressable[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to T
    View test on GitHub ↗
  290. ✓test_tool_call_is_returned_named_and_addressable[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool us
    View test on GitHub ↗
  291. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  292. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set
    View test on GitHub ↗
  293. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  294. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use se
    View test on GitHub ↗
  295. ✓test_tool_call_is_returned_named_and_addressable[messages-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama_chat-cfc984df5765 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  296. ✓test_tool_call_is_returned_named_and_addressable[messages-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama-74e436802975 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  297. ✓test_tool_call_is_returned_named_and_addressable[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to
    View test on GitHub ↗
  298. ✓test_tool_call_is_returned_named_and_addressable[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool u
    View test on GitHub ↗
  299. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  300. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use se
    View test on GitHub ↗
  301. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  302. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use s
    View test on GitHub ↗
  303. ✓test_tool_call_is_returned_named_and_addressable[responses-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama_chat-cfc984df5765 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  304. ✓test_tool_call_is_returned_named_and_addressable[responses-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama-74e436802975 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  305. ✓test_tool_call_is_streamed TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • together_ai
    • function_calling
    1. Add a deployment named e2e-together-346347adbb69 that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-together-346347adbb69
    View test on GitHub ↗
  306. ✓test_tool_call_round_trip[azure]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • azure
    • azure/gpt-realtime
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to azure-realtime-8ded514b55ac
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    9. Send the realtime event conversation.item.create over the websocket
    10. Send the realtime event response.create over the websocket
    11. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  307. ✓test_tool_call_round_trip[gemini]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • gemini
    • gemini/gemini-3.1-flash-live-preview
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to gemini-realtime-e759ffc26d6d
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    9. Send the realtime event conversation.item.create over the websocket
    10. Send the realtime event response.create over the websocket
    11. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  308. ✓test_tool_call_round_trip[openai]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • openai
    • openai/gpt-realtime-2
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to openai-realtime-b7ac866d7e7e
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    9. Send the realtime event conversation.item.create over the websocket
    10. Send the realtime event response.create over the websocket
    11. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  309. ✓test_tool_call_round_trip[vertex_ai]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • vertex_ai
    • vertex_ai/gemini-live-2.5-flash-native-audio
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to vertex-realtime-5d5b3994fc7a
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    9. Send the realtime event conversation.item.create over the websocket
    10. Send the realtime event response.create over the websocket
    11. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  310. ✓test_tool_call_streams_as_tool_call_deltas[ollama_chat] TestOllamaStreamedToolCalltest_ollama_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  311. ✓test_tool_call_streams_as_tool_call_deltas[ollama] TestOllamaStreamedToolCalltest_ollama_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  312. ✓test_tool_result_round_trip TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • function_calling
    1. Add a deployment named e2e-together-e6bb8ab61a26 that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-e6bb8ab61a26
    View test on GitHub ↗
  313. ✓test_tool_result_round_trip TestTogetherMessagestest_together_ai_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • together_ai
    • function_calling
    1. Add a deployment named e2e-together-0eb3dfa3bf92 that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/messages request to e2e-together-0eb3dfa3bf92
    View test on GitHub ↗
  314. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 over /chat/completions
    View test on GitHub ↗
  315. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather to
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f over /chat/completions
    View test on GitHub ↗
  316. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to T
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 over /chat/completions
    View test on GitHub ↗
  317. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool us
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 over /chat/completions
    View test on GitHub ↗
  318. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc over /chat/completions
    View test on GitHub ↗
  319. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool u
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 over /chat/completions
    View test on GitHub ↗
  320. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ollama-ollama_chat-cfc984df5765 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama_chat-cfc984df5765 over /chat/completions
    View test on GitHub ↗
  321. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ollama-ollama-74e436802975 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama-74e436802975 over /chat/completions
    View test on GitHub ↗
  322. ✓test_tool_result_round_trip_reaches_the_model[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to T
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 over /v1/messages
    View test on GitHub ↗
  323. ✓test_tool_result_round_trip_reaches_the_model[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool us
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f over /v1/messages
    View test on GitHub ↗
  324. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 over /v1/messages
    View test on GitHub ↗
  325. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 over /v1/messages
    View test on GitHub ↗
  326. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc over /v1/messages
    View test on GitHub ↗
  327. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use se
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 over /v1/messages
    View test on GitHub ↗
  328. ✓test_tool_result_round_trip_reaches_the_model[messages-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama_chat-cfc984df5765 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama_chat-cfc984df5765 over /v1/messages
    View test on GitHub ↗
  329. ✓test_tool_result_round_trip_reaches_the_model[messages-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama-74e436802975 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama-74e436802975 over /v1/messages
    View test on GitHub ↗
  330. ✓test_tool_result_round_trip_reaches_the_model[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-env_ref-ac848f8690c0 over /v1/responses
    View test on GitHub ↗
  331. ✓test_tool_result_round_trip_reaches_the_model[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool u
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-stored_credential-d0759446514f over /v1/responses
    View test on GitHub ↗
  332. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-env_ref-fa4a91186a15 over /v1/responses
    View test on GitHub ↗
  333. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use se
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-stored_credential-1006c1974cf2 over /v1/responses
    View test on GitHub ↗
  334. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-env_ref-b8e07aca14cc over /v1/responses
    View test on GitHub ↗
  335. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use s
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-stored_credential-7f34bcf39c53 over /v1/responses
    View test on GitHub ↗
  336. ✓test_tool_result_round_trip_reaches_the_model[responses-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama_chat-cfc984df5765 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama_chat-cfc984df5765 over /v1/responses
    View test on GitHub ↗
  337. ✓test_tool_result_round_trip_reaches_the_model[responses-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama-74e436802975 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama-74e436802975 over /v1/responses
    View test on GitHub ↗
  338. ✓test_tool_use_block_is_returned TestTogetherMessagestest_together_ai_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • together_ai
    • function_calling
    1. Add a deployment named e2e-together-443bff5c6c08 that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/messages request to e2e-together-443bff5c6c08
    View test on GitHub ↗
  339. ✓test_tool_use_nonstream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-azure-foundry-messages-681c30cae5f6 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-681c30cae5f6 and user id: e2e-test-user
    View test on GitHub ↗
  340. ✓test_tool_use_stream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    • llm-translation
    • messages
    • stream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-azure-foundry-messages-23b0841d6c5f that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-23b0841d6c5f and user id: e2e-test-user
    View test on GitHub ↗
  341. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestAnthropicChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-chat-midsys-048fc4d95099 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-chat-midsys-048fc4d95099 and user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  342. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestBedrockInvokeChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-chat-midsys-cd7070e029ff that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with models: e2e-chat-midsys-cd7070e029ff and user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  343. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestBedrockInvokeMidConversationSystemtest_messages_mid_conversation_system_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-midsys-a8c1791b11e9 that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with models: e2e-midsys-a8c1791b11e9 and user id: e2e-test-user
    View test on GitHub ↗
  344. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestAzureFoundryMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • azure_ai/claude-opus-4-7
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-midsys-7d0cc0465951 that calls azure_ai/claude-opus-4-7
    2. Generate a virtual key with models: e2e-midsys-7d0cc0465951 and user id: e2e-test-user
    View test on GitHub ↗
  345. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestVertexMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • vertex_ai/claude-sonnet-4-6
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-midsys-dd09ed70b960 that calls vertex_ai/claude-sonnet-4-6
    2. Generate a virtual key with models: e2e-midsys-dd09ed70b960 and user id: e2e-test-user
    View test on GitHub ↗
  346. ✓test_unknown_service_tier_is_dropped_and_billed_asap[5] TestSailChatCompletionstest_sail_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-16e0fca88996 that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  347. ✓test_unknown_service_tier_is_dropped_and_billed_asap[bogus] TestSailChatCompletionstest_sail_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-469a205d12be that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  348. ✓test_upload_without_purpose_returns_error TestFilesBatchesContracttest_files_batches_contract_e2e.py›
    • llm-translation
    • files
    • batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload batch_input.jsonl to /v1/files
    View test on GitHub ↗
  349. ✓test_upstream_handshake_refusal_is_an_error_event_and_policy_close[azure-bad-key]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • azure
    • azure/gpt-realtime
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a realtime deployment that calls azure/gpt-realtime
    3. Open a /v1/realtime websocket session to azure-realtime-refused-ad0f886c9854
    View test on GitHub ↗
  350. ✓test_verbose_json_returns_word_timestamps TestWhisperTranscriptionFormatstest_audio_transcriptions_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/whisper-1
    1. Add a deployment named e2e-whisper-verbose-336745131944 that calls openai/whisper-1
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload weather_question_24k.wav to /v1/audio/transcriptions
    View test on GitHub ↗
  351. ✓test_vertex_chat_returns_content TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    1. Add a deployment named e2e-vertex-chat-22e8a99d309e that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-vertex-chat-22e8a99d309e
    View test on GitHub ↗
  352. ✓test_vertex_chat_returns_tool_call TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    • function_calling
    1. Add a deployment named e2e-vertex-tool-2f2c5e832173 that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-vertex-tool-2f2c5e832173
    View test on GitHub ↗
  353. ✓test_vertex_chat_streams_real_content TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    1. Add a deployment named e2e-vertex-stream-b487f744adcf that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-vertex-stream-b487f744adcf
    View test on GitHub ↗
  354. ✓test_vertex_chat_vision_describes_image TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    • vision
    1. Add a deployment named e2e-vertex-vision-da01af0d45ec that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-vertex-vision-da01af0d45ec
    View test on GitHub ↗
  355. ✓test_vertex_embeddings_honor_requested_dimensions TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • vertex_ai
    • vertex_ai/text-embedding-005
    1. Add a deployment named e2e-embeddings-vertex-dims-2b09801dc2d7 that calls vertex_ai/text-embedding-005
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  356. ✓test_vertex_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • vertex_ai
    • vertex_ai/text-embedding-005
    1. Add a deployment named e2e-embeddings-vertex-abaabfcdb44e that calls vertex_ai/text-embedding-005
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  357. ✓test_vertex_multimodal_embeddings_honor_dimensions_and_are_costed TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • vertex_ai
    • vertex_ai/multimodalembedding@001
    1. Add a deployment named e2e-embeddings-vertex-mm-b01bfebf297a that calls vertex_ai/multimodalembedding@001
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  358. ✓test_vertex_passthrough_via_managed_model_logs_cost TestVertexPassthroughSpendTrackingtest_vertex_passthrough_e2e.py›
    • spend-budgets
    • passthrough
    • nonstream
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. POST /model/new
    3. Wait for the last control-plane write to reach every proxy replica
    4. Send a Vertex AI generateContent request to gemini-2.5-flash in us-central1 through /vertex_ai
    5. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  359. ✓test_vertex_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    • prompt_caching
    1. Add a deployment named e2e-vertex-cache-17667fcde15e that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /chat/completions
    4. Wait for the request's spend log in /spend/logs
    5. List the deployments from /model/info
    View test on GitHub ↗
  360. ✓test_vtt_format_returns_webvtt_transcript TestWhisperTranscriptionFormatstest_audio_transcriptions_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/whisper-1
    1. Add a deployment named e2e-whisper-vtt-598e82831320 that calls openai/whisper-1
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload weather_question_24k.wav to /v1/audio/transcriptions
    View test on GitHub ↗
  361. −test_empty_input_returns_error TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-tts

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on empty input instead of surfacing the provider 4xx

    View test on GitHub ↗
  362. −test_flagged_model_keeps_prompt_cache_across_system_reminder TestBedrockInvokeMidConversationSystemtest_messages_mid_conversation_system_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-sonnet-5
    • mid_conversation_system
    • prompt_caching

    No steps recorded

    LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

    View test on GitHub ↗
  363. −test_flagged_model_keeps_prompt_cache_across_system_reminder TestAzureFoundryMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • azure_ai/claude-opus-4-8
    • mid_conversation_system
    • prompt_caching

    No steps recorded

    LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

    View test on GitHub ↗
  364. −test_flagged_model_keeps_prompt_cache_across_system_reminder TestVertexMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • vertex_ai/claude-opus-4-8
    • mid_conversation_system
    • prompt_caching

    No steps recorded

    LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

    View test on GitHub ↗
  365. −test_gemini_passthrough_returns_the_same_header_contract_as_the_managed_routetest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • gemini
    • gemini-2.5-flash

    No steps recorded

    stage red: product gap, native passthrough returns no x-litellm-response-cost or x-ratelimit-* headers

    View test on GitHub ↗
  366. −test_hosted_vllm_chat_returns_content TestHostedVllmChattest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • hosted_vllm

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  367. −test_invalid_response_id_returns_error TestResponsesRetrievetest_responses_retrieve_e2e.py›
    • llm-translation
    • responses

    No steps recorded

    stage red: product gap (LIT-5447), retrieving an unknown response id returns 400 (model=None) instead of 404

    View test on GitHub ↗
  368. −test_invalid_voice_returns_error TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-tts

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on invalid voice instead of surfacing the provider 4xx

    View test on GitHub ↗
  369. −test_missing_document_returns_error TestRustOcrGatewaytest_ocr_rust_e2e.py›
    • llm-translation
    • ocr

    No steps recorded

    stage red: product gap, /v1/ocr 500s (aocr TypeError) on missing document instead of 400

    View test on GitHub ↗
  370. −test_missing_input_returns_error TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-tts

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on missing input instead of 400

    View test on GitHub ↗
  371. −test_missing_input_returns_error TestModerationstest_moderations_e2e.py›
    • llm-translation
    • moderations

    No steps recorded

    stage red: product gap, /v1/moderations 500s (KeyError 'input') on missing input instead of 400

    View test on GitHub ↗
  372. −test_missing_input_returns_error TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • openai
    • openai/gpt-4o-mini

    No steps recorded

    stage red: product gap, /v1/responses 500s (aresponses TypeError) on missing input instead of 400

    View test on GitHub ↗
  373. −test_missing_max_tokens_returns_error TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages

    No steps recorded

    stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing max_tokens instead of 400

    View test on GitHub ↗
  374. −test_missing_messages_returns_error TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages

    No steps recorded

    stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing messages instead of 400

    View test on GitHub ↗
  375. −test_missing_model_returns_error TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on missing model instead of 400

    View test on GitHub ↗
  376. −test_missing_prompt_returns_error TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images

    No steps recorded

    stage red: product gap, /v1/images/generations 500s (aimage_generation TypeError) on missing prompt instead of 400

    View test on GitHub ↗
  377. −test_retrieve_invalid_id_returns_error TestVectorStorestest_vector_stores_e2e.py›
    • llm-translation
    • vector_stores
    • openai

    No steps recorded

    stage red: product gap, retrieving a nonexistent vector store returns 2xx with an error envelope in the body instead of 404

    View test on GitHub ↗
  378. −test_search_missing_query_returns_error TestVectorStorestest_vector_stores_e2e.py›
    • llm-translation
    • vector_stores
    • openai

    No steps recorded

    stage red: product gap, vector store search 500s (asearch TypeError) on missing query instead of 400

    View test on GitHub ↗
  379. −test_static_and_x_pass_headers_reach_upstream TestPassthroughHeaderstest_passthrough_headers_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • anthropic
    • claude-haiku-4-5-20251001

    No steps recorded

    stage red: ALB sends custom passthrough paths to backend pods, which lack ANTHROPIC_API_KEY, so the os.environ header forwards unresolved and Anthropic 401s; routing user-defined paths to the gateway needs a product/infra decision (fixed path namespace or dynamic ingress)

    View test on GitHub ↗
  380. −test_store_and_retrieve_by_id TestResponsesRetrievetest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini

    No steps recorded

    stage red: product gap (LIT-5446), retrieve returns a different id than the stored response (non-idempotent response-id re-encryption)

    View test on GitHub ↗
  381. −test_vertex_partner_model_streams_openai_shaped_chunks[vertex_ai/mistral-small-2503] TestVertexPartnerChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • vertex_ai
    • vertex_ai/mistral-small-2503

    No steps recorded

    the e2e Vertex project has no access to mistral-small-2503 (404 publisher model not found)

    View test on GitHub ↗
  382. −test_vertex_partner_model_streams_openai_shaped_chunks[vertex_ai/openai/gpt-oss-120b-maas] TestVertexPartnerChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • vertex_ai
    • vertex_ai/openai/gpt-oss-120b-maas

    No steps recorded

    never served by the e2e Vertex project (60s read timeout, no headers)

    View test on GitHub ↗
  383. −test_web_search_server_tool_is_served TestBedrockWebSearchServerTooltest_bedrock_web_search_server_tool_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • web_search

    No steps recorded

    stage red: environment gap, the e2e stack neither enables the websearch_interception callback nor declares a search backend, so the request reaches bedrock's transformation and takes its by-design 400. Unskip once the ephemeral stack ships the config in this module's docstring.

    View test on GitHub ↗
  384. ✓test_a_drained_group_does_not_block_a_different_group TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-mag-free-9e16b94e76be and user id: e2e-test-user
    2. Send a /chat/completions request to e2e-mag-unbudgeted-9e16b94e76be with the prompt "hi b97b6af90991"
    View test on GitHub ↗
  385. ✓test_a_key_that_spent_nothing_is_blocked_by_the_shared_pool TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-mag-budget-9e16b94e76be and user id: e2e-test-user
    2. Send a /chat/completions request to e2e-mag-budgeted-9e16b94e76be with the prompt "hi 0732615319d3"
    View test on GitHub ↗
  386. ✓test_alias_shares_rpm_bucket_with_model_group TestModelGroupAliasRateLimitratelimit/test_model_group_alias_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • e2e-alias-rl-alias
    • e2e-alias-rl-target
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word bada8e0ae8fb"
    3. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word 8c3e5395ba03"
    4. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word b49db2763def"
    5. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word aba7f52f5c14"
    View test on GitHub ↗
  387. ✓test_bare_key_blocks_over_its_own_budget TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 95e589fd2d2f"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend d476b05d5687"
    View test on GitHub ↗
  388. ✓test_bare_key_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset ae3a3ef48581"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 78e91792b9a7"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 642aa8ac7f42"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 0af82c197be9"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 5f0c5cd7586b"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 46bbe609bcc8"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 9ae52eed9000"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 4f9281742ad4"
    View test on GitHub ↗
  389. ✓test_blocked_member_does_not_block_peer TestTeamMemberBudgetIsolationbudgets/test_team_member_budget_isolation_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Create the team e2e-member-iso-b0f42f6d366a and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Generate a virtual key with user id: a98f9c08-fab0-427d-a1fc-76e72495bd2f and team id: 4598a71a-3234-4813-807a-753e6beef391
    5. Generate a virtual key with user id: eb39bd8e-3f9a-44d8-a9b8-2b64f8f43b1a and team id: 4598a71a-3234-4813-807a-753e6beef391
    6. Send a /chat/completions request to gpt-5.5 with the prompt "tight 1738a3ff752e"
    7. Send a /chat/completions request to gpt-5.5 with the prompt "tight c6cfb0388da0"
    8. Send a /chat/completions request to gpt-5.5
    9. Send a /chat/completions request to gpt-5.5 with the prompt "tight ba198ddcedcf"
    10. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  390. ✓test_budget_crud_roundtripbudgets/test_budget_crud_e2e.py›
    • spend-budgets
    • budget_management
    1. Create a budget with /budget/new
    2. Read the budget from /budget/info
    3. Generate a virtual key with budget id: eb696f8e-5088-4d43-9e66-1de305c46069
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  391. ✓test_budget_delete_removes_itbudgets/test_budget_crud_e2e.py›
    • spend-budgets
    • budget_management
    1. Create a budget with /budget/new
    2. Delete the budget
    3. Read the budget from /budget/info
    View test on GitHub ↗
  392. ✓test_budget_duration_schedules_reset_on_keybudgets/test_budget_crud_e2e.py›
    • spend-budgets
    • key_management
    1. Generate a virtual key with max budget: 10.0 and budget duration: 30d
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  393. ✓test_budget_fallback_reroutes_anthropic_messages_to_openaibudgets/test_budget_fallback_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • anthropic
    • openai
    • claude-haiku-4-5
    • gpt-5.5
    1. Generate a virtual key with model max budget: claude-haiku-4-5: budget limit: 1e-06 and time period: 30d and budget fallbacks: claude-haiku-4-5: gpt-5.5
    2. Send a /v1/messages request to claude-haiku-4-5 with the prompt "hi 905d32e5449e"
    3. Send a /v1/messages request to claude-haiku-4-5 with the prompt "hi 9a26dfe4ed1b"
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  394. ✓test_burst_of_concurrent_calls_loses_no_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.6-luna
    1. Add a priced deployment, then create two teams with one key each and send 7 /chat/completions requests per key, 6 of them at once
    2. Send a /chat/completions request to e2e-reconciliation-2ecadf6426ab
    3. Check that the /spend/logs rows of team a2073a3b-248c-4dbb-936f-5164e0f20a28 match each response's tokens and cost
    4. Wait for the key's spend in /key/info to reach the expected minimum
    5. Check that the /spend/logs rows of team 5e199874-7dd8-4c0d-b81d-1b15a8af7be0 match each response's tokens and cost
    6. Wait for the key's spend in /key/info to reach the expected minimum
    View test on GitHub ↗
  395. ✓test_burst_rate_limit_does_not_freeze_fresh_key TestRedisCircuitBreakerPathratelimit/test_redis_circuit_breaker_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-cb-model-8eff17807ba2 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-cb-model-8eff17807ba2, key alias: e2e-cb-hot-68f938d04cfd and rpm limit: 1
    3. Generate a virtual key with models: e2e-cb-model-8eff17807ba2 and key alias: e2e-cb-cool-1505791dc99f
    4. Send a /chat/completions request to e2e-cb-model-8eff17807ba2 with the prompt "burst eb167de396e8"
    5. Send a /chat/completions request to e2e-cb-model-8eff17807ba2 with the prompt "burst 002970ed752e"
    6. Send a /chat/completions request to e2e-cb-model-8eff17807ba2 with the prompt "burst 4b4840769641"
    7. Send a /chat/completions request to e2e-cb-model-8eff17807ba2 with the prompt "burst e24bcf9a6aec"
    8. Send a /chat/completions request to e2e-cb-model-8eff17807ba2 with the prompt "burst 933083c68f63"
    9. Send a /chat/completions request to e2e-cb-model-8eff17807ba2 with the prompt "burst 6770c1ed873d"
    10. Send a /chat/completions request to e2e-cb-model-8eff17807ba2 with the prompt "burst 13e7b42aa994"
    11. Send a /chat/completions request to e2e-cb-model-8eff17807ba2 with the prompt "burst b017f50ba825"
    12. Send a /chat/completions request to e2e-cb-model-8eff17807ba2 with the prompt "burst a28f2f0d9cd1"
    13. Send a /chat/completions request to e2e-cb-model-8eff17807ba2 with the prompt "burst ebb4fcc3e354"
    14. Send a /chat/completions request to e2e-cb-model-8eff17807ba2 with the prompt "burst a0af9f62dfc6"
    15. Send a /chat/completions request to e2e-cb-model-8eff17807ba2 with the prompt "burst 1d4980058ff3"
    16. Send a /chat/completions request to e2e-cb-model-8eff17807ba2 with the prompt "fresh 43bb84be2de0"
    View test on GitHub ↗
  396. ✓test_cache_hit_is_zero_cost_and_suffixedspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "What is the capital of France? Answer in one word. c5c3a31dc3f7"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  397. ✓test_cache_hit_reduces_tpm_by_non_cached_only TestTpmExcludesCachedTokensratelimit/test_tpm_excludes_cached_tokens_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • prompt_caching
    1. Add a deployment named e2e-tpm-cache-3f5c3be792b2 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-tpm-cache-3f5c3be792b2 and tpm limit: 100000
    3. POST /chat/completions
    4. Send a request to /chat/completions
    View test on GitHub ↗
  398. ✓test_cache_write_tokens_billed_at_cache_creation_rate TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.6-luna
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.6-luna
    3. Send a /chat/completions request to cache-write-priced-85c5d1d259c2
    4. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  399. ✓test_capture_rate_reports_or_names_the_missing_billing_keyspend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/capture_rate
    View test on GitHub ↗
  400. ✓test_chat_completion_writes_nonzero_spend_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. List the deployments from /model/info
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to gemini-2.5-flash with the prompt "reply with one word 28ad0d2f0000"
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  401. ✓test_cold_counter_reseed_keeps_counter_equal_to_db_spendbudgets/test_spend_counter_reseed_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and max budget: 1.0
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 23981c1b2b2c"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 4c898a8b6375"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 12e4e51f313e"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed ec05a253e3bc"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed a36bc2a27f6c"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 2f1a81a89094"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 4e15f74a993f"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 96c40c619813"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 7ec673eb0324"
    11. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed a05d25a54b14"
    12. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 704a2fe428bf"
    13. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed aeeb78353210"
    14. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 2ed13910df64"
    15. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 7e2a0f89b759"
    16. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed c4fd2d39a460"
    17. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed e3df4677c479"
    18. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 96d9e45166dd"
    19. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 4150c1da3c33"
    20. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 1549e184caa2"
    21. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 8eff29433e24"
    22. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 3bcb27543174"
    23. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 3085cfad2626"
    24. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed ea731a15456f"
    25. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 2090abaebfcb"
    26. Read the key's settings back from /key/info
    27. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 18450e02b460"
    28. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 0bf21d8fdecb"
    29. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed f68bb553138c"
    30. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 03a56b7dbd96"
    31. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed bc57d5eef7e9"
    32. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 355f0c8111ee"
    View test on GitHub ↗
  402. ✓test_component_cost_headers_sum_to_total TestCostHeadersspend_tracking/test_cost_headers_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a request to /chat/completions
    View test on GitHub ↗
  403. ✓test_cost_breakdown_reports_component_costs TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.5
    • prompt_caching
    • reasoning
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a /chat/completions request to breakdown-priced-1da7870acf41
    4. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  404. ✓test_each_model_on_a_shared_key_gets_its_own_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • gemini
    • claude-haiku-4-5
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "one word f1289baf99c3"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "one word 8251260cf68d"
    4. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  405. ✓test_edge_wired_chat_writes_nonzero_spend_rowspend_tracking/test_provider_edge_spend_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-edge-openai-71b7cfdb98f8 that calls openai/gpt-5.5
    3. Send a /chat/completions request to e2e-edge-openai-71b7cfdb98f8 with the prompt "reply with one word 5d73d8275c43"
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  406. ✓test_embedding_writes_nonzero_spend_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • embeddings
    • nonstream
    • openai
    • openai-text-embedding-3-small
    1. Generate a virtual key with user id: e2e-test-user
    2. Send an /embeddings request to openai-text-embedding-3-small for "vectorize this sentence 0fec703a59ba"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  407. ✓test_end_user_budget_blocks_attributed_calls TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the end user e2e-budget-cust-85480256d2f8
    2. Generate a virtual key with models: claude-haiku-4-5
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 715eb8d025d0"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend ecf7ccc34992"
    View test on GitHub ↗
  408. ✓test_end_user_header_attributes_responses_row[x-litellm-customer-id]spend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai-responses-codex
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to openai-responses-codex with custom headers and the prompt "one word 61ca164901ad"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    4. Wait for the end user's spend in /customer/info to go above 0.0
    View test on GitHub ↗
  409. ✓test_end_user_header_attributes_responses_row[x-litellm-end-user-id]spend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai-responses-codex
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to openai-responses-codex with custom headers and the prompt "one word 38b66e25d822"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    4. Wait for the end user's spend in /customer/info to go above 0.0
    View test on GitHub ↗
  410. ✓test_end_user_spend_attributed_on_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "hi"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  411. ✓test_every_streamed_chunk_carries_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a streaming /chat/completions request to tier-echo-stream-4b88714f3e24
    View test on GitHub ↗
  412. ✓test_every_write_path_row_joins_the_key TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • spend_reporting
    • anthropic
    • gemini
    • openai
    • claude-haiku-4-5
    • gemini-2.5-flash
    • openai-gpt-4o-mini
    • openai-responses-codex
    • openai-text-embedding-3-small
    1. List the deployments from /model/info
    2. Create an internal user with the role proxy_admin
    3. Generate a virtual key with user id: key-attribution-1bf611b770f2 and key alias: key-attribution-1bf611b770f2
    4. Send a /chat/completions request to gemini-2.5-flash with the prompt "Reply with the word ok. a80f376e7087"
    5. Send a /queue/chat/completions request to gemini-2.5-flash with the prompt "Reply with the word ok. a80f376e7087"
    6. Send a /v1/messages request to claude-haiku-4-5 with the prompt "Reply with the word ok. a80f376e7087"
    7. Send a /v1/responses request to openai-responses-codex with the prompt "Reply with the word ok. a80f376e7087"
    8. Send an /embeddings request to openai-text-embedding-3-small for "Reply with the word ok. a80f376e7087"
    9. Send a Gemini generateContent request to gemini-2.5-flash through /gemini with the prompt "Reply with the word ok. a80f376e7087"
    10. Upload a batch input file for openai-gpt-4o-mini to /v1/files
    11. Create a batch for openai-gpt-4o-mini on /v1/batches
    12. Post a callback log for gemini-2.5-flash to /v1/rust_control_plane/logs
    13. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  413. ✓test_failure_call_writes_failure_status_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-spend-failure-0cde769c0ca2 that calls openai/gpt-5.5
    3. Send a /chat/completions request to e2e-spend-failure-0cde769c0ca2 with the prompt "trigger failure 6ed9db542a2a"
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  414. ✓test_failure_rows_share_normalized_error_across_provider_wordingspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • openai
    • anthropic/claude-haiku-4-5
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-norm-openai-4d58b9edb371 that calls openai/gpt-5.5
    3. Send a /chat/completions request to e2e-norm-openai-4d58b9edb371 with the prompt "normalize failure 4d58b9edb371"
    4. Add a deployment named e2e-norm-anthropic-4d58b9edb371 that calls anthropic/claude-haiku-4-5
    5. Send a /chat/completions request to e2e-norm-anthropic-4d58b9edb371 with the prompt "normalize failure 4d58b9edb371"
    6. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  415. ✓test_generous_mode_lets_priority_borrow_past_reservation TestDynamicRateLimitPriorityratelimit/test_dynamic_rate_limit_priority_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-dynpri-generous-fbf3d7097fe5 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-dynpri-generous-fbf3d7097fe5, key alias: e2e-dynpri-generous-dev-016b564f3a83 and metadata: priority: dev
    3. Generate a virtual key with models: e2e-dynpri-generous-fbf3d7097fe5, key alias: e2e-dynpri-generous-prod-c2477c30ac57 and metadata: priority: prod
    4. Read the key's settings back from /key/info
    5. Send a /chat/completions request to e2e-dynpri-generous-fbf3d7097fe5 with the prompt "reply with one word b55fa989391d"
    6. Send a /chat/completions request to e2e-dynpri-generous-fbf3d7097fe5 with the prompt "reply with one word b2ab625b99d1"
    7. Send a /chat/completions request to e2e-dynpri-generous-fbf3d7097fe5 with the prompt "reply with one word fc026263c053"
    8. Send a /chat/completions request to e2e-dynpri-generous-fbf3d7097fe5 with the prompt "reply with one word 5fe6baa7ab6f"
    View test on GitHub ↗
  416. ✓test_headers_report_limit_and_remaining TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5, tpm limit: 100000 and rpm limit: 5
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word a5a80934341d"
    View test on GitHub ↗
  417. ✓test_health_check_rows_keep_the_service_account_key TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • health
    • gemini
    • gemini-2.5-flash
    1. Run a health check on gemini-2.5-flash with /health
    2. Read /spend/logs between 2026-10-11 07:41:50.651390+00:00 and 2026-10-11 07:43:51.212424+00:00
    3. Read /spend/logs between 2026-10-11 07:41:50.651390+00:00 and 2026-10-11 07:43:56.304834+00:00
    4. Read /spend/logs between 2026-10-11 07:41:50.651390+00:00 and 2026-10-11 07:44:01.372282+00:00
    View test on GitHub ↗
  418. ✓test_intercepted_search_is_billed_under_the_request_session TestWebSearchInterceptionSessionspend_tracking/test_websearch_interception_session_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • bedrock
    • perplexity
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • web_search
    1. Add a deployment named e2e-websearch-session-c171825368df that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with models: e2e-websearch-session-c171825368df and user id: e2e-test-user
    3. Send a /v1/messages request to e2e-websearch-session-c171825368df
    4. Wait for at least 2 of the session's spend logs in /spend/logs
    View test on GitHub ↗
  419. ✓test_key_budget_reset_at_advances_after_windowbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Read the key's settings back from /key/info
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 30edddb412c1"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 08a22d6afb22"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 91234884d44b"
    6. Read the key's settings back from /key/info
    View test on GitHub ↗
  420. ✓test_key_spend_blocks_at_capbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 3af0237b9217"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 5324dda5209f"
    View test on GitHub ↗
  421. ✓test_key_spend_equals_sum_of_logsspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "say hi dd48981674cb"
    3. Send a /chat/completions request to gemini-2.5-flash with the prompt "say hi 5e8b580eaa53"
    4. Wait for at least 2 of the key's spend logs in /spend/logs
    5. Wait for the key's spend in /key/info to reach the expected minimum
    View test on GitHub ↗
  422. ✓test_key_with_budget_duration_schedules_reset_at_creationbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • key_management
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  423. ✓test_long_window_blocks_after_short_window_resetsbudgets/test_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with models: gpt-5.5 and budget limits: budget duration: 30s and max budget: 1e-09, budget duration: 1d and max budget: 5e-07
    2. Send a /chat/completions request to gpt-5.5 with the prompt "window 15cd362360a3"
    3. Send a /chat/completions request to gpt-5.5 with the prompt "window b798a76fe2b2"
    4. Read the key's budget windows from /key/info
    5. Send a /chat/completions request to gpt-5.5 with the prompt "window 04ff1f3cf72a"
    View test on GitHub ↗
  424. ✓test_member_budget_blocks_without_touching_teammates TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-budget-team-37dc96914504 and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Generate a virtual key with user id: f5f6d91c-d6b7-46b4-ac5d-9fcafaf634b6 and team id: 7a32bfcb-e559-456a-b50f-e9729d8f25d9
    5. Create an internal user with max budget: 100.0
    6. Add the internal user to the team
    7. Generate a virtual key with user id: 3da4aec8-4bab-4d2e-a12c-a24f677174d8 and team id: 7a32bfcb-e559-456a-b50f-e9729d8f25d9
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 9009e2648931"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend e1ce77f89df8"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 59926eebb179"
    View test on GitHub ↗
  425. ✓test_member_spend_attributed_to_team_and_user TestTeamMemberBudgetbudgets/test_team_member_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-team-member-38fb0f4c15f1 and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Generate a virtual key with user id: 765e39b1-da54-42aa-8284-8885b9d806dd and team id: 14a37170-0f07-4ed0-8f36-75f3a8289dac
    5. Send a /chat/completions request to claude-haiku-4-5
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  426. ✓test_member_spend_over_budget_is_blocked TestTeamMemberBudgetbudgets/test_team_member_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 06518eb37744"
    View test on GitHub ↗
  427. ✓test_messages_bridge_keeps_cache_tokens TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-5.3-codex
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.3-codex
    3. Send a /v1/messages request to bridge-cache-priced-b98bf5d30683
    4. Wait for a matching cost breakdown in the key's /spend/logs
    View test on GitHub ↗
  428. ✓test_messages_stream_records_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • spend-budgets
    • messages
    • stream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a streaming /v1/messages request to tier-messages-stream-2c2d28c55062
    4. Wait for a matching cost breakdown in the key's /spend/logs
    View test on GitHub ↗
  429. ✓test_missing_end_date_is_rejected TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py›
    • spend-budgets
    • spend_reporting
    1. Probe /team/daily/activity
    View test on GitHub ↗
  430. ✓test_missing_start_date_is_rejected TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py›
    • spend-budgets
    • spend_reporting
    1. Probe /team/daily/activity
    View test on GitHub ↗
  431. ✓test_model_group_shares_rpm_bucket_with_alias TestModelGroupAliasRateLimitratelimit/test_model_group_alias_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • e2e-alias-rl-alias
    • e2e-alias-rl-target
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word ce3505192030"
    3. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word 5523fd4b7c73"
    4. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word 49955e8847f6"
    5. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word cf4991c513d4"
    View test on GitHub ↗
  432. ✓test_model_max_budget_isolates_per_modelbudgets/test_model_max_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • gemini
    • claude-haiku-4-5
    • gemini-2.5-flash
    1. Generate a virtual key with model max budget: claude-haiku-4-5: budget limit: 1e-06 and time period: 30d and gemini-2.5-flash: budget limit: 1000.0 and time period: 30d
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi bcb60127f5f8"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi fdecc4727816"
    4. Send a /chat/completions request to gemini-2.5-flash with the prompt "hi 8adfc5f459a4"
    View test on GitHub ↗
  433. ✓test_multi_window_key_resets_each_window_independentlybudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with budget limits: budget duration: 30s and max budget: 3e-06, budget duration: 1m and max budget: 1.0
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 84199cdf50c1"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance a280ab4656c1"
    4. Read the key's settings back from /key/info
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance db067fb4fc6e"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 74e834b61402"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance fbaebb6b8b39"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance dcdc574f6216"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 7ea56a9b4d2a"
    10. Read the key's settings back from /key/info
    View test on GitHub ↗
  434. ✓test_one_request_lands_the_same_spend_on_every_surface TestSpendSurfaceConsistencyspend_tracking/test_spend_surface_consistency_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.6-luna
    1. List the deployments from /model/info
    2. Add a deployment named e2e-spend-surfaces-9ff6d21dec69 that calls openai/gpt-5.6-luna
    3. Create a team with team alias: e2e-spend-surfaces-9ff6d21dec69
    4. Generate a virtual key with models: e2e-spend-surfaces-9ff6d21dec69, team id: 43a6ffbd-1eff-4825-a465-f9022ccf51e8 and key alias: e2e-spend-surfaces-9ff6d21dec69
    5. Send a /chat/completions request to e2e-spend-surfaces-9ff6d21dec69
    6. Wait for the request's spend log in /spend/logs
    7. Wait for the key's spend in /key/info to reach the expected minimum
    8. Wait for the team's spend in /team/info to reach the expected minimum
    9. Wait for at least 1 of the key's requests in /user/daily/activity/aggregated
    10. Scrape /metrics/ on every proxy replica
    View test on GitHub ↗
  435. ✓test_org_budget_blocks_keys_under_it TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the organization e2e-budget-org-4177d06e0939 with max budget: 3e-06
    2. Create the team e2e-budget-team-744a09f2a99b and wait until /team/info returns it
    3. Generate a virtual key with team id: 8c6d5345-9274-47d3-8b10-603ed88162c5
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 5b923159f95d"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 3e36f8ff1596"
    View test on GitHub ↗
  436. ✓test_org_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the organization e2e-org-reset-c14b85c05a16 with max budget: 3e-06
    2. Create the team e2e-org-team-14cb940f2565 and wait until /team/info returns it
    3. Generate a virtual key with team id: 82e52cc2-8ceb-4e47-86df-8673023bf671
    4. Read the organization's budget id from /organization/info
    5. Read the budget from /budget/info
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset caedd9b176b0"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 433d2b6779e3"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset a785d04ac84d"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset f21374ea8799"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 97c7083b1821"
    11. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset e51af60503d9"
    12. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 4444c2c38250"
    13. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset d013b7064d09"
    View test on GitHub ↗
  437. ✓test_personal_key_blocks_over_its_own_budget TestKeyBudgetBlocksAcrossKeyKindsbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create an internal user with max budget: 100.0
    2. Generate a virtual key with max budget: 3e-06 and user id: ca991df0-b5a5-43ec-8280-e53d098a869f
    3. Generate a virtual key with user id: ca991df0-b5a5-43ec-8280-e53d098a869f
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 901a05ed4512"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend ea68da0cadd3"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 3995a37474c2"
    View test on GitHub ↗
  438. ✓test_personal_key_resets_after_window TestKeyBudgetResetAcrossKeyKindsbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create an internal user with max budget: 100.0
    2. Generate a virtual key with max budget: 3e-06, budget duration: 30s and user id: 188d4d0c-f8ed-46a4-a8b9-af7e859f9c0b
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 754cf7f2e3d5"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 1f0f7f8d656a"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset ca2a9754fc35"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 384041764bd0"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset fd702e1efc14"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 824a6bbb596a"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset b1af71435fc7"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset e60fedaabbbb"
    11. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset e622ab8bde8e"
    View test on GitHub ↗
  439. ✓test_personal_key_user_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create an internal user with max budget: 3e-06
    2. Generate a virtual key with user id: 2b2327dd-4968-4dd7-b144-3b1265cbf028
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset b0668ca637d9"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 1e8d34f51a1b"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset c25e9b6354f7"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 92a581a67aa8"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset e3078c238dac"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 8fae76d342fa"
    View test on GitHub ↗
  440. ✓test_pre_call_rejection_row_attributes_provider_and_model_idspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-spend-precall-5e1e329afb8b that calls openai/gpt-5.5
    2. Generate a virtual key with models: e2e-spend-precall-5e1e329afb8b and rpm limit: 1
    3. Send a /chat/completions request to e2e-spend-precall-5e1e329afb8b with the prompt "reply with one word ca1ff7cebbe8"
    4. Send a /chat/completions request to e2e-spend-precall-5e1e329afb8b with the prompt "over the rpm limit db5ced912641"
    5. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  441. ✓test_priority_tier_bills_priority_rates TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.6-luna
    • reasoning
    1. List the deployments from /model/info
    2. Generate a virtual key with user id: e2e-test-user
    3. Add a deployment with custom rates that calls openai/gpt-5.6-luna
    4. Send a /chat/completions request to tier-priced-894bd7ce0d66
    5. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  442. ✓test_request_tags_round_tripspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "tagged request"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  443. ✓test_reset_wait_never_yields_non_budget_errorbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 0116b5fd6e72"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance b132a573841d"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 7e8c1e730a64"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 5b21bd4d13df"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 7ad0e678b4ef"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance f83783b65fec"
    View test on GitHub ↗
  444. ✓test_responses_stream_records_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • spend-budgets
    • responses
    • stream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a streaming /v1/responses request to tier-responses-stream-6d260f52dd8f
    4. Wait for a matching cost breakdown in the key's /spend/logs
    View test on GitHub ↗
  445. ✓test_rpm_limit_blocks_over_limit TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and rpm limit: 3
    2. Read the key's settings back from /key/info
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 3ddb57cfd703"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 8e8ab725e7be"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 29e2d9bbeebe"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 5994872e921a"
    View test on GitHub ↗
  446. ✓test_rpm_limit_one_blocks_second_call TestRedisBackedRateLimitratelimit/test_redis_backed_ratelimit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-redis-rpm-48b1346226aa that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-redis-rpm-48b1346226aa, key alias: e2e-redis-rpm-0ce788626a09 and rpm limit: 1
    3. Read the key's settings back from /key/info
    4. Send a /chat/completions request to e2e-redis-rpm-48b1346226aa with the prompt "ping 52ee14038ed3"
    5. Send a /chat/completions request to e2e-redis-rpm-48b1346226aa with the prompt "pong 57e8fa384686"
    View test on GitHub ↗
  447. ✓test_rpm_limit_resets_after_window TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and rpm limit: 1
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 2e2d1325d89a"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word b0be01cef9aa"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 45708f7703ef"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word f3eae28492e1"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 5beb1a2e36bb"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word aa0af2a38740"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word cbda4b646fbb"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word be1c8ea2a97f"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 4d3c07042da3"
    11. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 08a429cd231b"
    12. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 109eaa2375b5"
    13. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 1436a1af3a69"
    14. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 2687ae25c30e"
    15. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 4f9e25ef8949"
    16. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 0c950e30632e"
    View test on GitHub ↗
  448. ✓test_schema_listed_spend_routes_are_responsivespend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Read the OpenAPI schema from /openapi.json
    View test on GitHub ↗
  449. ✓test_short_window_blocks_then_resetsbudgets/test_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with models: gpt-5.5 and budget limits: budget duration: 30s and max budget: 1e-09, budget duration: 1m and max budget: 1.0
    2. Send a /chat/completions request to gpt-5.5 with the prompt "window bad3b4588ce7"
    3. Send a /chat/completions request to gpt-5.5 with the prompt "window ea2a6b71ce35"
    4. Send a /chat/completions request to gpt-5.5 with the prompt "window 1a6f6da3471b"
    5. Send a /chat/completions request to gpt-5.5 with the prompt "window c23802683420"
    6. Send a /chat/completions request to gpt-5.5 with the prompt "window d5de7a8105de"
    7. Send a /chat/completions request to gpt-5.5 with the prompt "window 9c0d576a9281"
    8. Send a /chat/completions request to gpt-5.5 with the prompt "window a4c0f135b624"
    9. Send a /chat/completions request to gpt-5.5 with the prompt "window 4070d1876894"
    View test on GitHub ↗
  450. ✓test_soft_budget_does_not_blockbudgets/test_soft_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 1000.0 and soft budget: 1e-09
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi ab737dc6e00f"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi f7ca2171f593"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi 2f720583c575"
    View test on GitHub ↗
  451. ✓test_spend_calculate_returns_nonzero_costspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • spend_reporting
    • gemini
    • gemini-2.5-flash
    1. Estimate the cost of sending "estimate the cost of this request" to gemini-2.5-flash with /spend/calculate
    View test on GitHub ↗
  452. ✓test_spend_logs_by_key_return_every_row_with_the_alias TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • spend_reporting
    • anthropic
    • gemini
    • openai
    • claude-haiku-4-5
    • gemini-2.5-flash
    • openai-gpt-4o-mini
    • openai-responses-codex
    • openai-text-embedding-3-small
    1. Wait for at least 9 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  453. ✓test_spend_logs_endpoint_returns_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • spend_reporting
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "spend logs 7e9ba02a8acd"
    3. GET /spend/logs
    View test on GitHub ↗
  454. ✓test_spend_logs_v2_pagination_caps_pages_and_keeps_totalspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • spend_reporting
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "page fodder c4f1365d837b"
    3. Send a /chat/completions request to gemini-2.5-flash with the prompt "page fodder 345c6b53e95d"
    4. Wait for at least 2 of the key's spend logs in /spend/logs
    5. Read page 1 of /spend/logs/v2 at a page size of 1
    6. Read page 9 of /spend/logs/v2 at a page size of 1
    7. Read page 1 of /spend/logs/v2 at a page size of 1
    View test on GitHub ↗
  455. ✓test_spend_route_responsive[/customer/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /customer/daily/activity
    View test on GitHub ↗
  456. ✓test_spend_route_responsive[/customer/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • customer_management
    1. Call the management route /customer/list
    View test on GitHub ↗
  457. ✓test_spend_route_responsive[/end_user/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /end_user/daily/activity
    View test on GitHub ↗
  458. ✓test_spend_route_responsive[/global/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/activity
    View test on GitHub ↗
  459. ✓test_spend_route_responsive[/global/activity/exceptions]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/activity/exceptions
    View test on GitHub ↗
  460. ✓test_spend_route_responsive[/global/activity/exceptions/deployment]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/activity/exceptions/deployment
    View test on GitHub ↗
  461. ✓test_spend_route_responsive[/global/activity/model]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/activity/model
    View test on GitHub ↗
  462. ✓test_spend_route_responsive[/global/all_end_users]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/all_end_users
    View test on GitHub ↗
  463. ✓test_spend_route_responsive[/global/spend]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend until it answers successfully
    View test on GitHub ↗
  464. ✓test_spend_route_responsive[/global/spend/all_tag_names]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/all_tag_names
    View test on GitHub ↗
  465. ✓test_spend_route_responsive[/global/spend/keys]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/keys until it answers successfully
    View test on GitHub ↗
  466. ✓test_spend_route_responsive[/global/spend/logs]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/logs until it answers successfully
    View test on GitHub ↗
  467. ✓test_spend_route_responsive[/global/spend/models]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/models until it answers successfully
    View test on GitHub ↗
  468. ✓test_spend_route_responsive[/global/spend/provider]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/provider
    View test on GitHub ↗
  469. ✓test_spend_route_responsive[/global/spend/report]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/report
    View test on GitHub ↗
  470. ✓test_spend_route_responsive[/global/spend/tags]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/tags until it answers successfully
    View test on GitHub ↗
  471. ✓test_spend_route_responsive[/global/spend/teams]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/teams
    View test on GitHub ↗
  472. ✓test_spend_route_responsive[/key/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • key_management
    1. Call the management route /key/list
    View test on GitHub ↗
  473. ✓test_spend_route_responsive[/organization/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /organization/daily/activity
    View test on GitHub ↗
  474. ✓test_spend_route_responsive[/organization/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • organization_management
    1. Call the management route /organization/list
    View test on GitHub ↗
  475. ✓test_spend_route_responsive[/spend/keys]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. List the deployments from /model/info
    2. Add a deployment named openai-text-embedding-3-small that calls openai/text-embedding-3-small
    3. Add a deployment named openai-responses-codex that calls openai/gpt-5.3-codex
    4. Add a deployment named openai-gpt-4o-mini that calls openai/gpt-4o-mini
    5. Call the management route /spend/keys
    View test on GitHub ↗
  476. ✓test_spend_route_responsive[/spend/logs]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/logs
    View test on GitHub ↗
  477. ✓test_spend_route_responsive[/spend/logs/session/ui]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/logs/session/ui
    View test on GitHub ↗
  478. ✓test_spend_route_responsive[/spend/logs/ui]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/logs/ui
    View test on GitHub ↗
  479. ✓test_spend_route_responsive[/spend/logs/v2]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/logs/v2
    View test on GitHub ↗
  480. ✓test_spend_route_responsive[/spend/tags]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/tags
    View test on GitHub ↗
  481. ✓test_spend_route_responsive[/spend/users]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/users
    View test on GitHub ↗
  482. ✓test_spend_route_responsive[/tag/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /tag/daily/activity
    View test on GitHub ↗
  483. ✓test_spend_route_responsive[/team/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /team/daily/activity
    View test on GitHub ↗
  484. ✓test_spend_route_responsive[/team/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • team_management
    1. Call the management route /team/list
    View test on GitHub ↗
  485. ✓test_spend_route_responsive[/user/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /user/daily/activity
    View test on GitHub ↗
  486. ✓test_spend_route_responsive[/user/daily/activity/aggregated]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /user/daily/activity/aggregated
    View test on GitHub ↗
  487. ✓test_spend_route_responsive[/user/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • user_management
    1. Call the management route /user/list
    View test on GitHub ↗
  488. ✓test_streamed_call_records_and_bills_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • spend-budgets
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.6-luna
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.6-luna
    3. Send a streaming /chat/completions request to tier-priced-stream-59359732e47e
    4. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  489. ✓test_streaming_cache_read_billed_at_cache_read_rate TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • stream
    • openai
    • openai/gpt-5.5
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a /chat/completions request to stream-cache-priced-259d10238ebc
    4. Send a streaming /chat/completions request to stream-cache-priced-259d10238ebc
    5. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  490. ✓test_streaming_chat_completion_tracks_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • chat_completions
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to gemini-2.5-flash with the prompt "count to three 408333dfc910"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  491. ✓test_streaming_messages_via_responses_bridge_tracks_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • messages
    • stream
    • openai
    • openai-responses-codex
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to openai-responses-codex with the prompt "reply with exactly one word 366c975b1a51"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  492. ✓test_tag_budget_blocks_tagged_requestsbudgets/test_tag_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Create the tag e2e-budget-tag-39fd6f116ce7 with max budget: 1e-06
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi 74dffd730eb7"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi fba156fe3bae"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi 9244ba971084"
    View test on GitHub ↗
  493. ✓test_tag_spend_matches_sum_of_tagged_logsspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "hi 385e228087d0"
    3. Send a /chat/completions request to gemini-2.5-flash with the prompt "hi 002a5a955388"
    4. Wait for at least 2 of the key's spend logs in /spend/logs
    5. Wait for the tag e2e-tagspend-cf46454c631e to reach the expected spend in /spend/tags
    View test on GitHub ↗
  494. ✓test_team_budget_blocks_every_team_key TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-budget-team-d8a8aecfbae4 and wait until /team/info returns it
    2. Generate a virtual key with team id: f29a1f92-2b57-4d3f-9e1f-f8263257b992
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 13f64d277d83"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 6c757fb56aae"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend c45f1afa75f9"
    View test on GitHub ↗
  495. ✓test_team_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-team-reset-55f56dd84c64 and wait until /team/info returns it
    2. Generate a virtual key with team id: 9835da9f-0fc0-4498-ad95-572aa12aa85b
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset e61c838dc5a2"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset fbeb44027700"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 8d669a64867e"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 78028701da5e"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 8e09a61cf39e"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 5eaaef70f364"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 5372e080ae7c"
    View test on GitHub ↗
  496. ✓test_team_key_blocks_over_its_own_budget TestKeyBudgetBlocksAcrossKeyKindsbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-key-cap-team-14a79e17d13a and wait until /team/info returns it
    2. Generate a virtual key with max budget: 3e-06 and team id: 545a4ac9-9c1d-4b63-9ef7-81fb55b380df
    3. Generate a virtual key with team id: 545a4ac9-9c1d-4b63-9ef7-81fb55b380df
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 0909ce9dc41e"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 3a4bac71ad72"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend a411ad0a7d5b"
    View test on GitHub ↗
  497. ✓test_team_key_resets_after_window TestKeyBudgetResetAcrossKeyKindsbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-key-reset-team-cb9564055020 and wait until /team/info returns it
    2. Generate a virtual key with max budget: 3e-06, budget duration: 30s and team id: 749f9ac2-8c24-40ea-a8db-0e9c05ba452e
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 6afa848d2ccf"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 4c737bab03be"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 209f71e98f14"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset f986cde7f0e7"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 412b4489b175"
    View test on GitHub ↗
  498. ✓test_team_long_window_blocks_after_short_window_resetsbudgets/test_team_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-team-long-window-742be2c1941c and wait until /team/info returns it
    2. Generate a virtual key with models: claude-haiku-4-5 and team id: 1579ecd9-7086-4b1f-b950-a91da6ab0ffc
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 8185c5c4dcbf"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 1cc859223ef1"
    5. Read the team's budget windows from /team/info
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 96d7403b8e73"
    View test on GitHub ↗
  499. ✓test_team_member_budget_reset_at_advancesbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-member-advance-9089abb89b35 and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Update the team member's budget with /team/member_update
    5. Read the team member's budget reset time from /team/info
    6. Generate a virtual key with user id: a56f6800-cdf7-4b92-b73f-9c3091b12ebd and team id: dcb092d2-a6d3-4014-8f4b-101ca24ee290
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance f2044c2f92e5"
    8. Read the team member's budget reset time from /team/info
    View test on GitHub ↗
  500. ✓test_team_member_budget_reset_keeps_advancingbudgets/test_team_member_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-member-reset-0f1fde594baa and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Update the team member's budget with /team/member_update
    5. Read the team member's budget reset time from /team/info
    6. Generate a virtual key with user id: d6e699c4-adac-4b58-84ea-faf8df0e0b33 and team id: 23e72e87-b7c0-4b6e-a163-aacbd9fecfba
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 45cd7f84b14d"
    8. Read the team member's budget reset time from /team/info
    View test on GitHub ↗
  501. ✓test_team_member_key_blocks_over_its_own_budget TestKeyBudgetBlocksAcrossKeyKindsbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-key-cap-team-e172cb3ef783 and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Generate a virtual key with max budget: 3e-06, user id: 385f407b-1731-4d4e-b03c-1ccdb8dac720 and team id: 4340475f-249a-4bf6-ac3e-f9e9e627804b
    5. Generate a virtual key with user id: 385f407b-1731-4d4e-b03c-1ccdb8dac720 and team id: 4340475f-249a-4bf6-ac3e-f9e9e627804b
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 2984d62a982f"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 55bec73fa8b7"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 2456624a3e30"
    View test on GitHub ↗
  502. ✓test_team_member_key_resets_after_window TestKeyBudgetResetAcrossKeyKindsbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-key-reset-team-1639cd96ce73 and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Generate a virtual key with max budget: 3e-06, budget duration: 30s, user id: 7fd08806-8447-4054-8dac-f79e94af8ff5 and team id: fb02ba85-3b26-4d20-867f-07f7b9c82061
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 9d0cfe943929"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset c98a105dde25"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 015fcfdad552"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset d6e1f413ccab"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 2a2833ae51ab"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 7447dce798db"
    11. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset ee8517d38d14"
    12. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 4bf0834d2393"
    View test on GitHub ↗
  503. ✓test_team_short_window_blocks_then_resetsbudgets/test_team_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-team-window-5f39dc4bc560 and wait until /team/info returns it
    2. Generate a virtual key with models: claude-haiku-4-5 and team id: 0c1e597d-a11c-491e-b35b-814586a96886
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 37301e113ac8"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 92a885afb730"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 684da1894e2d"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window bf06de4b69b8"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 768d1334a77f"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window c4bb889417dc"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 91479460a14d"
    View test on GitHub ↗
  504. ✓test_terminal_batch_cost_row_joins_the_retrieving_key TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • openai-gpt-4o-mini
    1. Retrieve the openai batch from /v1/batches
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  505. ✓test_the_budget_read_reports_the_spend_drawn_against_the_pool TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • budget_management
    • openai
    • openai/gpt-5.4-nano
    1. Read the budget and spend of the model access group e2e-mag-budget-9e16b94e76be
    View test on GitHub ↗
  506. ✓test_the_key_that_drained_the_pool_stays_blocked TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-mag-budget-9e16b94e76be
    2. Add a deployment named e2e-mag-budgeted-9e16b94e76be that calls openai/gpt-5.4-nano
    3. Add a deployment named e2e-mag-unbudgeted-9e16b94e76be that calls openai/gpt-5.4-nano
    4. Set a shared budget on the model access group e2e-mag-budget-9e16b94e76be
    5. Send a /chat/completions request to e2e-mag-budgeted-9e16b94e76be with the prompt "hi f1389ccd3544"
    6. Send a /chat/completions request to e2e-mag-budgeted-9e16b94e76be with the prompt "hi 34c08d0c4754"
    7. Send a /chat/completions request to e2e-mag-budgeted-9e16b94e76be with the prompt "hi c80039dcb686"
    View test on GitHub ↗
  507. ✓test_tpm_limit_blocks_over_limit TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and tpm limit: 60
    2. Read the key's settings back from /key/info
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word fb28f765a235"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word afa9afe14db0"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word d3751ba9c98a"
    View test on GitHub ↗
  508. ✓test_user_budget_blocks_a_second_key TestUserBudgetAcrossKeysbudgets/test_user_budget_across_keys_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Create an internal user with max budget: 3e-06
    2. Generate a virtual key with user id: a12e40ec-3854-4529-a0c6-725f83f49c81
    3. Send a /chat/completions request to gpt-5.5 with the prompt "across 8897bc88f07d"
    4. Send a /chat/completions request to gpt-5.5 with the prompt "across 89f20952d234"
    5. Send a /chat/completions request to gpt-5.5 with the prompt "across 000d5d32fede"
    6. Read the internal user's spend and budget from /user/info
    View test on GitHub ↗
  509. ✓test_user_budget_enforced_across_their_personal_keys TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create an internal user with max budget: 3e-06
    2. Generate a virtual key with user id: 1b88a696-f230-4b47-b185-ebb2017176cf
    3. Create the team e2e-budget-team-27e8ac55c252 and wait until /team/info returns it
    4. Add the internal user to the team
    5. Generate a virtual key with user id: 1b88a696-f230-4b47-b185-ebb2017176cf and team id: 1d378949-575d-4c04-b456-784636301a06
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend dcdb48d98fd1"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 933eb16f7749"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 68de8d1a6db9"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 14db35080efa"
    View test on GitHub ↗
  510. ✓test_user_daily_activity_reports_alias_and_email TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • spend_reporting
    • anthropic
    • gemini
    • openai
    • claude-haiku-4-5
    • gemini-2.5-flash
    • openai-gpt-4o-mini
    • openai-responses-codex
    • openai-text-embedding-3-small
    1. Wait for at least 9 of the key's requests in /user/daily/activity
    View test on GitHub ↗
  511. ✓test_valid_date_range_returns_results_and_metadata TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py›
    • spend-budgets
    • spend_reporting
    • openai
    • openai/gpt-5.6-luna
    1. Add a priced deployment, then create two teams with one key each and send 7 /chat/completions requests per key, 6 of them at once
    2. Send a /chat/completions request to e2e-reconciliation-523a1ffefad7
    3. Check that the /spend/logs rows of team 2ab2b1bd-0e7b-471e-ae08-4bda07e6931e match each response's tokens and cost
    4. Check that the /spend/logs rows of team 3942f0b1-a230-4b0f-a9e4-57ef3d682e4f match each response's tokens and cost
    5. Probe /team/daily/activity
    View test on GitHub ↗
  512. −test_end_user_model_max_budget_enforces_per_model_rpmbudgets/test_model_max_budget_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash

    No steps recorded

    stage red: product gap, end-user model_max_budget rpm_limit is stored but never enforced

    View test on GitHub ↗
  513. −test_strict_mode_blocks_saturated_priority_but_serves_the_other TestDynamicRateLimitPriorityratelimit/test_dynamic_rate_limit_priority_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001

    No steps recorded

    LIT-5118: the stage proxy does not run the dynamic_rate_limiter_v3 callbacks + priority_reservation config this module's docstring requires (zero limiter log lines on any pod during the 2026-08-02 run), so strict enforcement can never engage there

    View test on GitHub ↗
  514. ✓test_access_group_info_reports_membership TestModelAccessGroupRoutestest_model_tag_accessgroup_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-agmodel-80e5ec10703a that calls openai/gpt-5.5
    2. POST /access_group/new
    3. GET /access_group/e2e-mgmt-ag-aad25bd7c4ea/info
    View test on GitHub ↗
  515. ✓test_actor_subject_and_database_role[internal_user_viewer] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an actor in the internal_user_viewer role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the internal_user_viewer role
    3. Read the user back from /user/info
    4. GET /user/info
    View test on GitHub ↗
  516. ✓test_actor_subject_and_database_role[internal_user] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an actor in the internal_user role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the internal_user role
    3. Read the user back from /user/info
    4. GET /user/info
    View test on GitHub ↗
  517. ✓test_actor_subject_and_database_role[organization_admin] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the organization_admin role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the organization_admin role
    4. Read the user back from /user/info
    5. GET /user/info
    6. Read the team back from /team/info
    7. Read the organization back from /organization/info
    View test on GitHub ↗
  518. ✓test_actor_subject_and_database_role[proxy_admin_viewer] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an actor in the proxy_admin_viewer role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the proxy_admin_viewer role
    3. Read the user back from /user/info
    4. GET /user/info
    View test on GitHub ↗
  519. ✓test_actor_subject_and_database_role[proxy_admin] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an actor in the proxy_admin role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the proxy_admin role
    3. Read the user back from /user/info
    4. GET /user/info
    View test on GitHub ↗
  520. ✓test_actor_subject_and_database_role[team_admin] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the team_admin role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the team_admin role
    4. Read the user back from /user/info
    5. GET /user/info
    6. Read the team back from /team/info
    7. Read the organization back from /organization/info
    View test on GitHub ↗
  521. ✓test_actor_subject_and_database_role[team_member] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the team_member role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the team_member role
    4. Read the user back from /user/info
    5. GET /user/info
    6. Read the team back from /team/info
    7. Read the organization back from /organization/info
    View test on GitHub ↗
  522. ✓test_actor_subject_and_database_role[unrelated_user] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an actor in the unrelated_user role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the unrelated_user role
    3. Read the user back from /user/info
    4. GET /user/info
    View test on GitHub ↗
  523. ✓test_admin_creates_reads_updates_clears_and_deletes_a_key[direct_jwt] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • key_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the proxy_admin role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the proxy_admin role
    4. Read the user back from /user/info
    5. Generate a virtual key with models: gpt-5.5, team id: 2b03a1fe-9fc7-47b2-82c6-b9165719e854 and key alias: e2e-jwt-key-863295644454
    6. Read the key's settings back from /key/info
    7. Update the virtual key's settings with /key/update
    8. Read the key's settings back from /key/info
    9. Update the virtual key's settings with /key/update
    10. Read the key's settings back from /key/info
    11. List the keys with the alias e2e-jwt-key-863295644454-updated from /key/list
    12. Delete the virtual key
    13. List the keys with the alias e2e-jwt-key-863295644454-updated from /key/list
    View test on GitHub ↗
  524. ✓test_admin_creates_reads_updates_clears_and_deletes_a_key[virtual_key] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • key_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the proxy_admin role, with its identity provider user, internal user and any tenant memberships
    3. Generate a virtual key as the proxy admin
    4. Read the user back from /user/info
    5. Generate a virtual key with models: gpt-5.5, team id: 564fc4da-4d2a-4196-add2-7bdebdcfe468 and key alias: e2e-jwt-key-00bd0b2fdc37
    6. Read the key's settings back from /key/info
    7. Update the virtual key's settings with /key/update
    8. Read the key's settings back from /key/info
    9. Update the virtual key's settings with /key/update
    10. Read the key's settings back from /key/info
    11. List the keys with the alias e2e-jwt-key-00bd0b2fdc37-updated from /key/list
    12. Delete the virtual key
    13. List the keys with the alias e2e-jwt-key-00bd0b2fdc37-updated from /key/list
    View test on GitHub ↗
  525. ✓test_admin_viewer_reads_but_cannot_update TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • key_management
    1. Create an actor in the proxy_admin_viewer role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the proxy_admin_viewer role
    3. Generate a virtual key as the proxy admin
    4. Update the virtual key's settings with /key/update
    5. Read the key's settings back from /key/info
    6. Update the virtual key's settings with /key/update
    7. Read the key's settings back from /key/info
    View test on GitHub ↗
  526. ✓test_approve_activates_submission_and_persists TestMcpServerSubmissiontest_config_misc_endpoints_e2e.py›
    • mcp
    • mcp
    1. Create a team with team alias: e2e-mcp-team-cca8f395f975
    2. Generate a virtual key with team id: c7df666b-4cbf-462d-99ca-556e7dcfbcbe
    3. POST /v1/mcp/server/register
    4. PUT /v1/mcp/server/f9b9f14c-51de-482c-9c23-081e83b67809/approve
    5. GET /v1/mcp/server/f9b9f14c-51de-482c-9c23-081e83b67809
    View test on GitHub ↗
  527. ✓test_bedrock_mantle_responses_connection_succeeds TestModelTestConnectiontest_model_test_connection_e2e.py›
    • management
    • health
    • nonstream
    • bedrock_mantle
    • bedrock_mantle/openai.gpt-5.6-luna
    1. Run Test Connection on bedrock_mantle/openai.gpt-5.6-luna in responses mode with /health/test_connection
    View test on GitHub ↗
  528. ✓test_block_persists_to_key_info TestKeyRoutestest_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with models: gemini-2.5-flash
    2. Read the key's settings back from /key/info
    3. Block the virtual key
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  529. ✓test_block_then_unblock_persists_to_model_info TestModelRoutestest_model_tag_accessgroup_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-model-block-6cc53dd69ce3 that calls openai/gpt-5.5
    2. GET /model/info
    3. POST /model/block
    4. GET /model/info
    5. POST /model/unblock
    6. GET /model/info
    View test on GitHub ↗
  530. ✓test_block_then_unblock_persists_to_team_info TestTeamManagementRoutestest_team_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-team-block-b7de1ba2b634 and models: gemini-2.5-flash
    2. GET /team/info
    3. POST /team/block
    4. GET /team/info
    5. POST /team/unblock
    6. GET /team/info
    View test on GitHub ↗
  531. ✓test_bulk_update_applies_max_budget_to_target_key TestKeyManagementRoutestest_key_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with models: gpt-5.5 and max budget: 5.0
    2. Read the key's settings back from /key/info
    3. POST /key/bulk_update
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  532. ✓test_callbacks_list_reports_active_logging_callbacks TestInventoryRoutestest_config_misc_endpoints_e2e.py›
    • observability
    1. GET /callbacks/list
    View test on GitHub ↗
  533. ✓test_clearing_the_description_with_null_reads_back_null TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_46a6cac900dd
    2. Update the MCP server's settings with PUT /v1/mcp/server
    3. Read /v1/mcp/server/18728cc1-c71b-4df0-a08a-d09ce2b23563 on every proxy replica until it settles
    View test on GitHub ↗
  534. ✓test_clearing_the_description_with_null_reads_back_null TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_345e9decc736
    2. Create an MCP toolset with the tools server id: d902a1f7-7e7a-4e76-994c-7fb8a6b1f43c and tool name: search_datadog_logs, server id: d902a1f7-7e7a-4e76-994c-7fb8a6b1f43c and tool name: get_datadog_metr
    3. Update an MCP toolset with toolset id: 8b218b14-1e5e-4471-895b-962be046ea41
    4. Read /v1/mcp/toolset/8b218b14-1e5e-4471-895b-962be046ea41 on every proxy replica until it settles
    View test on GitHub ↗
  535. ✓test_config_update_persists_router_setting_to_get TestRouterSettingstest_config_misc_endpoints_e2e.py›
    • routing
    • proxy_config
    1. GET /router/settings
    2. POST /config/update
    3. GET /router/settings
    4. POST /config/update
    5. GET /router/settings
    View test on GitHub ↗
  536. ✓test_create_echoes_every_field_written TestKeyLifecycletest_key_lifecycle_e2e.py›
    • management
    • key_management
    1. Add a deployment named e2e-key-lifecycle-b2b2ed21ef33 that calls gpt-4o-mini
    2. Create a team with team alias: e2e-key-lifecycle-team-6e426aed49ce
    3. Generate a virtual key with models: e2e-key-lifecycle-b2b2ed21ef33, max budget: 25.0, budget duration: 30d, team id: b89bc7a4-1652-4fb2-b4cc-f5dd0bd09cc4, key alias: e2e-key-lifecycle-6e426aed49ce, tp
    View test on GitHub ↗
  537. ✓test_create_persists_and_is_read_back TestFallbackManagementtest_config_misc_endpoints_e2e.py›
    • routing
    1. Add a deployment named e2e-fallback-primary-c214b872a442 that calls openai/gpt-5.5
    2. Add a deployment named e2e-fallback-secondary-610d9bcc3228 that calls openai/gpt-5.5
    3. POST /fallback
    4. GET /fallback/e2e-fallback-primary-c214b872a442
    View test on GitHub ↗
  538. ✓test_create_persists_both_tools_under_the_exact_names_written TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_b90a78b2d4d7
    2. Create an MCP toolset with the tools server id: f6fae562-4aaa-4282-8302-bfb2841bcfe1 and tool name: search_datadog_logs, server id: f6fae562-4aaa-4282-8302-bfb2841bcfe1 and tool name: get_datadog_metr
    3. Read /v1/mcp/toolset/a6abc5cb-942e-4431-85f6-9489e1c83281 on every proxy replica until it settles
    4. Read /v1/mcp/toolset on every proxy replica until it settles
    View test on GitHub ↗
  539. ✓test_create_persists_every_field_on_every_replica TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_c461ba639ad2
    2. Read /v1/mcp/server/97208dd2-79d0-40bf-8197-5ff86a455e11 on every proxy replica until it settles
    View test on GitHub ↗
  540. ✓test_created_budget_appears_in_budget_list TestBudgetManagementtest_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. POST /budget/new
    2. GET /budget/list
    View test on GitHub ↗
  541. ✓test_created_key_appears_in_key_list_inventory TestKeyRoutestest_management_e2e.py›
    • management
    • key_management
    1. Count the keys with the alias e2e-mgmt-keylist-37837213e384 in /key/list
    2. Generate a virtual key with key alias: e2e-mgmt-keylist-37837213e384
    3. Count the keys with the alias e2e-mgmt-keylist-37837213e384 in /key/list
    View test on GitHub ↗
  542. ✓test_created_team_appears_in_team_list TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-mgmt-team-68693f6a8d56 and models: gemini-2.5-flash
    2. List the teams from /team/list
    View test on GitHub ↗
  543. ✓test_created_users_appear_in_user_list TestUserRoutestest_management_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-mgmt-22afc4de1da5@example.com and user role: internal_user
    2. Create an internal user with user email: e2e-mgmt-4921f66ed8ab@example.com and user role: internal_user
    3. List the matching users from /user/list
    View test on GitHub ↗
  544. ✓test_creating_a_key_from_the_dashboard_persists_and_works TestDashboardKeyRoutestest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Sign in to the Admin UI with /v2/login
    2. Read the key's settings back from /key/info
    3. Generate a virtual key with models: gemini-2.5-flash, key alias: e2e-mgmt-uicreate-b02dd3d36b32 and tpm limit: 100
    4. Read the key's settings back from /key/info
    5. List the keys with the alias e2e-mgmt-uicreate-b02dd3d36b32 from /key/list
    6. Send a /chat/completions request to gemini-2.5-flash
    7. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  545. ✓test_credential_migration_check_reports_residual_scan TestInventoryRoutestest_config_misc_endpoints_e2e.py›
    • deploy-ops
    1. GET /credentials/migrate-encryption/check
    View test on GitHub ↗
  546. ✓test_customer_create_persists_to_info TestCustomertest_management_e2e.py›
    • management
    • customer_management
    1. Create the end user e2e-customer-7e19dcc338d5
    2. Read the end user e2e-customer-7e19dcc338d5 back from /customer/info
    View test on GitHub ↗
  547. ✓test_delete_forbidden_for_non_admin_key TestKeyManagementRoutestest_key_management_e2e.py›
    • proxy-auth
    • key_management
    1. Generate a virtual key with models: gpt-5.5
    2. Send a request to /key/delete
    3. Read the key's settings back from /key/info
    View test on GitHub ↗
  548. ✓test_delete_persists_and_revokes_team_bound_key TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    • nonstream
    • openai
    • gpt-5.5
    1. Create a team with team alias: e2e-mgmt-team-e6b013802e11 and models: gpt-5.5
    2. Generate a virtual key with team id: 3375b75a-f7dd-4d17-8f87-a7d6f8e9007c
    3. Send a /chat/completions request to gpt-5.5
    4. Delete the team
    5. Check whether /team/info finds the team
    6. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  549. ✓test_delete_removes_from_model_info_catalog TestModelRoutestest_management_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-model-cbbecded8d3a that calls openai/gpt-5.5
    2. List the deployments from /model/info
    3. Delete the deployment
    4. List the deployments from /model/info
    View test on GitHub ↗
  550. ✓test_delete_removes_from_organization_info TestOrganizationRoutestest_management_e2e.py›
    • management
    • organization_management
    1. Create an organization with organization alias: e2e-mgmt-org-22ed92bbd550
    2. Check whether /organization/info finds the organization
    3. Delete the organization
    4. Check whether /organization/info finds the organization
    View test on GitHub ↗
  551. ✓test_delete_removes_the_customer TestCustomerManagementtest_budget_customer_user_org_e2e.py›
    • management
    • customer_management
    1. POST /customer/new
    2. GET /customer/info
    3. POST /customer/delete
    4. Probe /customer/info
    View test on GitHub ↗
  552. ✓test_delete_removes_the_server_from_every_replica TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_a782d62ef584
    2. Delete the MCP server
    3. Check that /v1/mcp/server/acc0f9fa-e0d9-475b-bfd9-ea9e386a08ec returns 404 on every proxy replica
    4. Read /v1/mcp/server on every proxy replica until it settles
    View test on GitHub ↗
  553. ✓test_delete_removes_the_toolset_from_every_replica TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_b1ea1c5802b1
    2. Create an MCP toolset with the tools server id: b9a305da-af29-48f7-be7f-2315839435a1 and tool name: search_datadog_logs, server id: b9a305da-af29-48f7-be7f-2315839435a1 and tool name: get_datadog_metr
    3. Delete the MCP toolset
    4. Check that /v1/mcp/toolset/2166e233-1ddc-427a-ae6d-b0e82a14b5e3 returns 404 on every proxy replica
    5. Read /v1/mcp/toolset on every proxy replica until it settles
    View test on GitHub ↗
  554. ✓test_delete_removes_the_user_from_inventory TestUserRoutestest_management_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-mgmt-824455b88c4b@example.com and user role: internal_user
    2. Count the matching users in /user/list
    3. Delete the internal user
    4. Count the matching users in /user/list
    View test on GitHub ↗
  555. ✓test_delete_revokes_info_and_chat_on_every_replica TestKeyLifecycletest_key_lifecycle_e2e.py›
    • management
    • key_management
    • nonstream
    • openai
    • gpt-4o-mini
    1. Create a team with team alias: e2e-key-lifecycle-team-4a57dd4acb4a
    2. Generate a virtual key with models: e2e-key-lifecycle-b2b2ed21ef33, max budget: 25.0, budget duration: 30d, team id: 6d970191-752c-443d-9589-9c4020d8dd70, key alias: e2e-key-lifecycle-4a57dd4acb4a, tp
    3. Send a request to /chat/completions
    4. Delete the virtual key
    5. Read /key/info on every proxy replica until they all agree
    6. Send a request to /chat/completions
    View test on GitHub ↗
  556. ✓test_delete_revokes_the_key_on_chat TestKeyRoutestest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with models: gemini-2.5-flash
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Delete the virtual key
    4. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  557. ✓test_editing_a_key_from_the_dashboard_persists_and_is_enforced TestDashboardKeyRoutestest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • gemini
    • openai
    • gemini-2.5-flash
    • gpt-5.5
    1. Generate a virtual key with models: gemini-2.5-flash, key alias: e2e-mgmt-uiedit-c74221b80df7, tpm limit: 100 and rpm limit: 200
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Send a /chat/completions request to gpt-5.5
    4. Sign in to the Admin UI with /v2/login
    5. Update the virtual key's settings with /key/update
    6. Read the key's settings back from /key/info
    7. Send a /chat/completions request to gpt-5.5
    8. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  558. ✓test_end_user_new_persists_to_end_user_info TestCustomerManagementtest_budget_customer_user_org_e2e.py›
    • management
    • customer_management
    1. POST /end_user/new
    2. GET /end_user/info
    View test on GitHub ↗
  559. ✓test_estimate_computes_cost_from_token_counts TestCostEstimatetest_config_misc_endpoints_e2e.py›
    • cost-map
    1. POST /cost/estimate
    View test on GitHub ↗
  560. ✓test_explicit_null_clears_the_budget_and_its_reset_time TestKeyLifecycletest_key_lifecycle_e2e.py›
    • management
    • key_management
    1. Create a team with team alias: e2e-key-lifecycle-team-12655f2ba6af
    2. Generate a virtual key with models: e2e-key-lifecycle-b2b2ed21ef33, max budget: 25.0, budget duration: 30d, team id: 5394cf72-f24f-4bf4-a070-f1be71311f29, key alias: e2e-key-lifecycle-12655f2ba6af, tp
    3. Read /key/info on every proxy replica until they all agree
    4. Update the virtual key's settings with /key/update
    5. Read /key/info on every proxy replica until they all agree
    View test on GitHub ↗
  561. ✓test_gdpr_check_derives_verdict_from_the_request TestComplianceRoutestest_config_misc_endpoints_e2e.py›
    • management
    1. POST /compliance/gdpr
    View test on GitHub ↗
  562. ✓test_generate_forbidden_for_non_admin_key TestKeyManagementRoutestest_key_management_e2e.py›
    • proxy-auth
    • key_management
    1. Generate a virtual key with models: gpt-5.5
    2. Send a request to /key/generate
    View test on GitHub ↗
  563. ✓test_generate_persists_to_key_info_and_scopes_chat TestKeyRoutestest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with models: gemini-2.5-flash, key alias: e2e-mgmt-key-61a37a672b83, tpm limit: 424242 and rpm limit: 424243
    2. Read the key's settings back from /key/info
    3. Send a /chat/completions request to gemini-2.5-flash
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  564. ✓test_health_reports_the_calling_key_healthy TestKeyManagementRoutestest_key_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with models: gpt-5.5
    2. POST /key/health
    View test on GitHub ↗
  565. ✓test_info_reflects_the_fields_the_key_was_created_with TestKeyManagementRoutestest_key_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with models: gpt-5.5, gemini-2.5-flash, key alias: e2e-mgmt-keyinfo-bee1c34d1878, tpm limit: 131313 and rpm limit: 141414
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  566. ✓test_info_returns_created_team_fields TestTeamManagementRoutestest_team_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-team-info-23c9193064d9 and models: gemini-2.5-flash
    2. GET /team/info
    View test on GitHub ↗
  567. ✓test_is_null_finds_the_budget_left_uncapped TestBudgetListV1test_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. POST /budget/new
    2. GET /management/v1/budgets
    View test on GitHub ↗
  568. ✓test_is_refused_for_a_non_admin_key TestBudgetListV1test_budget_customer_user_org_e2e.py›
    • proxy-auth
    • budget_management
    1. Generate a virtual key with default settings
    2. GET /management/v1/budgets
    View test on GitHub ↗
  569. ✓test_key_delete_by_alias_writes_audit_row TestKeyDeletionAuditLogtest_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with key alias: e2e-audit-ac58580e21dc
    2. Delete the virtual key with the alias e2e-audit-ac58580e21dc
    3. Read the key's settings back from /key/info
    4. Read the key's deletion entries from the /audit log
    View test on GitHub ↗
  570. ✓test_key_delete_by_key_writes_audit_row TestKeyDeletionAuditLogtest_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with key alias: e2e-audit-0abe913db76f
    2. Delete the virtual key
    3. Read the key's settings back from /key/info
    4. Read the key's deletion entries from the /audit log
    View test on GitHub ↗
  571. ✓test_key_serves_its_model_and_is_denied_others TestKeyLifecycletest_key_lifecycle_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-4o-mini
    1. Create a team with team alias: e2e-key-lifecycle-team-26b683f7daa0
    2. Generate a virtual key with models: e2e-key-lifecycle-b2b2ed21ef33, max budget: 25.0, budget duration: 30d, team id: c7741864-90d3-480c-8885-e841704fc9ba, key alias: e2e-key-lifecycle-26b683f7daa0, tp
    3. Send a request to /chat/completions
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  572. ✓test_llm_only_key_forbidden_from_management_writes TestManagementRoutePermissionstest_management_e2e.py›
    • proxy-auth
    1. Generate a virtual key limited to the LLM API routes
    2. Try to generate a virtual key with key alias: e2e-mgmt-forbidden-key-94d360053924, calling with a virtual key
    3. Try to create a team with team alias: e2e-mgmt-forbidden-team-94d360053924 and team id: e2e-mgmt-forbidden-team-94d360053924, calling with a virtual key
    4. Try to create an internal user with user email: e2e-mgmt-forbidden-user-94d360053924@example.com, user role: internal_user and user id: e2e-mgmt-forbidden-user-94d360053924, calling with a virtual key
    5. Count the keys with the alias e2e-mgmt-forbidden-key-94d360053924 in /key/list
    6. Check whether /team/info finds the team
    7. Count the matching users in /user/list
    View test on GitHub ↗
  573. ✓test_member_add_and_delete_persist_to_team_info TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    1. Create an internal user with user email: e2e-mgmt-5c66644c32c5@example.com and user role: internal_user
    2. Create a team with team alias: e2e-mgmt-team-0b6f5050adf1 and models: gemini-2.5-flash
    3. Add a user to the team with /team/member_add
    4. Read the team back from /team/info
    5. Remove a user from the team with /team/member_delete
    6. Read the team back from /team/info
    View test on GitHub ↗
  574. ✓test_member_add_forbidden_to_plain_member TestTeamManagementRoutestest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-fb-00f4326cf428@example.com and user role: internal_user
    2. Create an internal user with user email: e2e-team-fb-08ce31b06118@example.com and user role: internal_user
    3. Create a team with team alias: e2e-team-fb-0a6fe2fc1b12 and models: gemini-2.5-flash
    4. Add a user to the team with /team/member_add
    5. Generate a virtual key with user id: 06f42d0b-19c3-411f-bab6-85ce1619b970 and team id: c373f94d-0abb-47a7-a13e-8c2e40aa11f9
    6. Send a request to /team/member_add
    View test on GitHub ↗
  575. ✓test_member_add_records_membership TestOrganizationMembershiptest_budget_customer_user_org_e2e.py›
    • management
    • organization_management
    1. Create an organization with organization alias: e2e-mgmt-org-63e213530989
    2. Create an internal user with user email: e2e-mgmt-cafb1f19878b@example.com and user role: internal_user
    3. POST /organization/member_add
    4. GET /organization/info
    View test on GitHub ↗
  576. ✓test_member_cannot_write_and_another_team_cannot_read_the_key TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • key_management
    1. Create the Keycloak group e2e-jwt-team-1ecb4919b4c3 and the user e2e-jwt-user-1ecb4919b4c3 in it
    2. POST /user/new
    3. Create a team with team alias: e2e-jwt-1ecb4919b4c3 and team id: e2e-jwt-team-1ecb4919b4c3
    4. Get a Keycloak access token for e2e-jwt-user-1ecb4919b4c3 from the client litellm-e2e-admin
    5. Get a Keycloak access token for e2e-jwt-user-1ecb4919b4c3 from the client litellm-e2e-tests
    6. Generate a virtual key with team id: e2e-jwt-team-1ecb4919b4c3 and key alias: e2e-jwt-owned-268116093ebd
    7. Add a user to the team with /team/member_add
    8. Read the key's settings back from /key/info
    9. Update the virtual key's settings with /key/update
    10. Read the key's settings back from /key/info
    11. Create the Keycloak group e2e-jwt-team-72c1b13b2e4a and the user e2e-jwt-user-72c1b13b2e4a in it
    12. Create an internal user with user email: e2e-jwt-user-72c1b13b2e4a@example.com, user role: internal_user and user id: e3dfd3f1-e21e-461b-b949-528db182dfb8
    13. Create a team with team alias: 72c1b13b2e4a and team id: e2e-jwt-team-72c1b13b2e4a
    14. Add a user to the team with /team/member_add
    15. Get a Keycloak access token for e2e-jwt-user-72c1b13b2e4a from the client litellm-e2e-tests
    16. Read the key's settings back from /key/info
    View test on GitHub ↗
  577. ✓test_member_delete_forbidden_to_plain_member TestTeamManagementRoutestest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-fb-2dc555b43fd6@example.com and user role: internal_user
    2. Create an internal user with user email: e2e-team-fb-5e5700b4469a@example.com and user role: internal_user
    3. Create a team with team alias: e2e-team-fb-9366a1bfc902 and models: gemini-2.5-flash
    4. Add a user to the team with /team/member_add
    5. Generate a virtual key with user id: 59ca44ce-bb27-44d9-ab92-31b99a2cbe4d and team id: 4e881776-df52-433b-9304-ee7b90fc3387
    6. Send a request to /team/member_delete
    View test on GitHub ↗
  578. ✓test_member_delete_persists_to_team_info TestTeamManagementRoutestest_team_management_e2e.py›
    • management
    • team_management
    1. Create an internal user with user email: e2e-team-md-83764be345ba@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-md-24eab554996f and models: gemini-2.5-flash
    3. Add a user to the team with /team/member_add
    4. GET /team/info
    5. Remove a user from the team with /team/member_delete
    6. GET /team/info
    View test on GitHub ↗
  579. ✓test_member_update_persists_role_and_budget TestTeamManagementRoutestest_team_management_e2e.py›
    • management
    • team_management
    1. Create an internal user with user email: e2e-team-mu-45b3c912cb21@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-mu-00e879662650 and models: gemini-2.5-flash
    3. Add a user to the team with /team/member_add
    4. GET /team/info
    5. POST /team/member_update
    6. GET /team/info
    View test on GitHub ↗
  580. ✓test_multi_group_actor_keeps_exact_memberships TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the team_member role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the team_member role
    4. Read the team back from /team/info
    View test on GitHub ↗
  581. ✓test_new_access_group_tags_the_deployment TestModelAccessGroupRoutestest_model_tag_accessgroup_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-agmodel-881df3a61f2f that calls openai/gpt-5.5
    2. POST /access_group/new
    3. GET /access_group/e2e-mgmt-ag-6d9e90422417/info
    View test on GitHub ↗
  582. ✓test_new_is_denied_to_non_admin_keys TestTeamManagementRoutestest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Generate a virtual key with default settings
    2. Create an internal user with user email: e2e-team-adm-7876913df1ca@example.com and user role: internal_user
    3. Generate a virtual key with user id: be1feb6c-215c-4f53-919d-8d9b8b3f0f11
    4. Try to create a team with team alias: e2e-team-adm-a95364d6421c, calling with a virtual key
    5. Try to create a team with team alias: e2e-team-adm-64ea56b167ba, calling with a virtual key
    View test on GitHub ↗
  583. ✓test_new_is_refused_for_a_non_admin_key TestBudgetManagementtest_budget_customer_user_org_e2e.py›
    • proxy-auth
    • budget_management
    1. Generate a virtual key with default settings
    2. Send a request to /budget/new
    View test on GitHub ↗
  584. ✓test_new_persists_mapping_and_is_read_back TestJwtKeyMappingtest_config_misc_endpoints_e2e.py›
    • proxy-auth
    1. Generate a virtual key with default settings
    2. POST /jwt/key/mapping/new
    3. GET /jwt/key/mapping/info
    View test on GitHub ↗
  585. ✓test_new_persists_to_customer_info TestCustomerManagementtest_budget_customer_user_org_e2e.py›
    • management
    • customer_management
    1. POST /customer/new
    2. GET /customer/info
    View test on GitHub ↗
  586. ✓test_new_persists_to_model_info_catalog TestModelRoutestest_management_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-model-edb62707b199 that calls openai/gpt-5.5
    2. List the deployments from /model/info
    View test on GitHub ↗
  587. ✓test_new_persists_to_organization_info TestOrganizationRoutestest_management_e2e.py›
    • management
    • organization_management
    1. Create an organization with organization alias: e2e-mgmt-org-2cde176bf385 and models: gemini-2.5-flash
    2. Read the organization back from /organization/info
    View test on GitHub ↗
  588. ✓test_new_persists_to_tag_list TestTagRoutestest_management_e2e.py›
    • management
    • tag_management
    1. List the tags from /tag/list
    2. Create a tag with name: e2e-mgmt-tag-c34b15365363 and description: Tag for spend categorization
    3. List the tags from /tag/list
    View test on GitHub ↗
  589. ✓test_new_persists_to_team_info_and_binds_keys TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-mgmt-team-afbbdbd4fc0e and models: gemini-2.5-flash
    2. Read the team back from /team/info
    3. Generate a virtual key with team id: e3c763c9-e1e5-4fee-be5d-bb11043835ef
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  590. ✓test_new_persists_to_user_info TestUserRoutestest_management_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-mgmt-71aad8f4123c@example.com and user role: internal_user
    2. Read the user back from /user/info
    View test on GitHub ↗
  591. ✓test_new_user_is_readable_via_user_info TestUserManagementtest_budget_customer_user_org_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-mgmt-e475911e5910@example.com and user role: internal_user
    2. Read the user back from /user/info
    View test on GitHub ↗
  592. ✓test_non_admin_key_cannot_add_global_model TestModelRoutestest_model_tag_accessgroup_e2e.py›
    • proxy-auth
    • model_management
    1. Generate a virtual key with default settings
    2. Send a request to /model/new
    3. List the deployments from /model/info
    View test on GitHub ↗
  593. ✓test_oidc_browser_profile_identity_mapping TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    1. Create an actor in the internal_user role, with its identity provider user, internal user and any tenant memberships
    2. Read Keycloak's OpenID Connect discovery document
    3. Register a browser SSO client in Keycloak
    4. Get a Keycloak access token for e2e-jwt-user-d61bbca659e7 through the browser SSO client
    5. Read the signed-in user's profile from Keycloak's userinfo endpoint
    View test on GitHub ↗
  594. ✓test_partial_setup_removes_previously_created_identities[group] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    1. Create the Keycloak group e2e-partial-a0aac0104574
    2. Delete every resource the test created, newest first
    3. Check that Keycloak's admin API returns 404 for the deleted entry under /groups
    View test on GitHub ↗
  595. ✓test_partial_setup_removes_previously_created_identities[user] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    1. Create the Keycloak group e2e-partial-d4a6f66a1284
    2. Create the Keycloak user e2e-jwt-user-d4a6f66a1284 with a password of its own
    3. Create the Keycloak user e2e-jwt-user-d4a6f66a1284
    4. Delete every resource the test created, newest first
    5. Check that Keycloak's admin API returns 404 for the deleted entry under /groups
    6. Check that Keycloak's admin API returns 404 for the deleted entry under /users
    View test on GitHub ↗
  596. ✓test_partial_update_changes_only_the_named_field TestKeyLifecycletest_key_lifecycle_e2e.py›
    • management
    • key_management
    1. Create a team with team alias: e2e-key-lifecycle-team-d10f44d6b319
    2. Generate a virtual key with models: e2e-key-lifecycle-b2b2ed21ef33, max budget: 25.0, budget duration: 30d, team id: 76601cd6-b514-458a-98de-a6918258ebd4, key alias: e2e-key-lifecycle-d10f44d6b319, tp
    3. Read /key/info on every proxy replica until they all agree
    4. Update the virtual key's settings with /key/update
    5. Read /key/info on every proxy replica until they all agree
    View test on GitHub ↗
  597. ✓test_project_detachment_preserves_key_scope_and_refreshes_auth TestKeyManagementRoutestest_key_management_e2e.py›
    • management
    • key_management
    • nonstream
    • openai
    • openai/synthetic-detachment
    1. Add a deployment named e2e-detach-6f6211c65b0d that calls openai/synthetic-detachment
    2. Create an organization with organization alias: e2e-detach-6f6211c65b0d and models: e2e-detach-6f6211c65b0d
    3. Create a team with team alias: e2e-detach-6f6211c65b0d, models: e2e-detach-6f6211c65b0d and organization id: 46645121-ea3f-4a74-b384-144128014263
    4. POST /project/new
    5. Generate a virtual key with models: e2e-detach-6f6211c65b0d, max budget: 5.0, team id: e68ebac0-1433-4bad-94f2-5013dbe0d5a7, project id: a7992f3d-2bc9-4d32-96d3-0064ce871699, organization id: 46645121
    6. Send a /chat/completions request to e2e-detach-6f6211c65b0d
    7. Update the virtual key's settings with /key/update
    8. Read the key's settings back from /key/info
    9. Update the virtual key's settings with /key/update
    10. Send a request to /key/update
    11. Read the key's settings back from /key/info
    12. POST /project/update
    13. Send a /chat/completions request to e2e-detach-6f6211c65b0d
    14. POST /key/update
    15. Read the key's settings back from /key/info
    16. Send a /chat/completions request to e2e-detach-6f6211c65b0d
    17. Update the virtual key's settings with /key/update
    18. Read the key's settings back from /key/info
    19. Send a /chat/completions request to e2e-detach-6f6211c65b0d-outside
    View test on GitHub ↗
  598. ✓test_read_reflects_the_create_on_every_replica TestKeyLifecycletest_key_lifecycle_e2e.py›
    • management
    • key_management
    1. Create a team with team alias: e2e-key-lifecycle-team-59cb643e5d9c
    2. Generate a virtual key with models: e2e-key-lifecycle-b2b2ed21ef33, max budget: 25.0, budget duration: 30d, team id: f4935b2f-04f5-461f-ac53-f57810307ee8, key alias: e2e-key-lifecycle-59cb643e5d9c, tp
    3. Read /key/info on every proxy replica until they all agree
    View test on GitHub ↗
  599. ✓test_refuses_a_sort_field_and_a_parameter_it_does_not_support TestBudgetListV1test_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. GET /management/v1/budgets
    View test on GitHub ↗
  600. ✓test_regenerate_rotates_to_a_working_new_key TestKeyRegenerationtest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with models: gpt-5.5
    2. Regenerate the virtual key with /key/regenerate
    3. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  601. ✓test_regenerate_with_grace_period_keeps_old_key_until_revoked TestKeyRegenerationtest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with models: gpt-5.5
    2. Regenerate the virtual key with /key/regenerate
    3. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  602. ✓test_register_submits_pending_server TestMcpServerSubmissiontest_config_misc_endpoints_e2e.py›
    • mcp
    • mcp
    1. Create a team with team alias: e2e-mcp-team-4c66938ff6ac
    2. Generate a virtual key with team id: 8a4e8396-fdab-416f-b68c-d2aed967d3f8
    3. POST /v1/mcp/server/register
    View test on GitHub ↗
  603. ✓test_reset_spend_zeroes_recorded_spend_and_lifts_the_budget_block TestKeyManagementRoutestest_key_management_e2e.py›
    • spend-budgets
    • key_management
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and max budget: 3e-06
    2. Send a /chat/completions request to claude-haiku-4-5
    3. Read the key's settings back from /key/info
    4. Reset the virtual key's spend to 0.0
    5. Read the key's settings back from /key/info
    6. Send a /chat/completions request to claude-haiku-4-5
    View test on GitHub ↗
  604. ✓test_sorts_pages_and_filters_the_budgets_it_created TestBudgetListV1test_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. POST /budget/new
    2. GET /management/v1/budgets
    View test on GitHub ↗
  605. ✓test_successful_actor_cleanup_removes_owned_state TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the team_member role, with its identity provider user, internal user and any tenant memberships
    3. Generate a virtual key as the proxy admin
    4. Read the key's settings back from /key/info
    5. Delete every resource the test created, newest first
    6. Count the matching users in /user/list
    7. Count the keys with the alias e2e-actor-key-53a03588ce97 in /key/list
    8. Check whether /team/info finds the team
    9. Check whether /organization/info finds the organization
    10. Check that Keycloak's admin API returns 404 for the deleted entry under /users
    11. Check that Keycloak's admin API returns 404 for the deleted entry under /groups
    View test on GitHub ↗
  606. ✓test_tag_delete_removes_from_list TestTagRoutestest_model_tag_accessgroup_e2e.py›
    • management
    • tag_management
    1. POST /tag/new
    2. GET /tag/list
    3. POST /tag/delete
    4. GET /tag/list
    View test on GitHub ↗
  607. ✓test_tag_list_reports_created_tag TestTagRoutestest_model_tag_accessgroup_e2e.py›
    • management
    • tag_management
    1. GET /tag/list
    2. POST /tag/new
    3. GET /tag/list
    View test on GitHub ↗
  608. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[budget_duration] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-4bb704bf7bc4@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-3eb295835918, tpm limit: 1000 and members with roles: role: admin and user id: 77125248-6442-4f45-b304-e33b1cc90c8b
    3. Generate a virtual key with user id: 77125248-6442-4f45-b304-e33b1cc90c8b
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  609. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[max_budget] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-68ec64934b40@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-3522abe0b5c5, tpm limit: 1000 and members with roles: role: admin and user id: 4f7b5f10-abf2-4369-afe1-02f345c742f0
    3. Generate a virtual key with user id: 4f7b5f10-abf2-4369-afe1-02f345c742f0
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  610. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[metadata] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-23738228a3f1@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-d9a1da3fd5ba, tpm limit: 1000 and members with roles: role: admin and user id: 9bdd3764-0b93-4e1f-ae46-d7c3ffd19455
    3. Generate a virtual key with user id: 9bdd3764-0b93-4e1f-ae46-d7c3ffd19455
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  611. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[models] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-576e0a8ce5bd@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-15e07719ae0f, tpm limit: 1000 and members with roles: role: admin and user id: 740fbdb9-56fb-4b09-b415-a1245b1f6030
    3. Generate a virtual key with user id: 740fbdb9-56fb-4b09-b415-a1245b1f6030
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  612. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[rpm_limit] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-94ea66aa284b@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-d027460cd2f8, tpm limit: 1000 and members with roles: role: admin and user id: c8bc335c-cb58-4473-97d7-e3f3a2d0f5a0
    3. Generate a virtual key with user id: c8bc335c-cb58-4473-97d7-e3f3a2d0f5a0
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  613. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[team_alias] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-b2e2fdbddad9@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-2bd007978d67, tpm limit: 1000 and members with roles: role: admin and user id: 73b2a580-9290-420e-9e69-47786cd19169
    3. Generate a virtual key with user id: 73b2a580-9290-420e-9e69-47786cd19169
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  614. ✓test_team_admin_cannot_change_any_team_setting TestTeamAdminWithNoEditableFieldstest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. GET /get/ui_settings
    2. PATCH /update/ui_settings
    3. Wait for the last control-plane write to reach every proxy replica
    4. Create an internal user with user email: e2e-team-admin-20c37a221c02@example.com and user role: internal_user
    5. Create a team with team alias: e2e-team-admin-63a0a5a19760, tpm limit: 1000 and members with roles: role: admin and user id: e43d5611-7501-407e-b742-0d26854566aa
    6. Generate a virtual key with user id: e43d5611-7501-407e-b742-0d26854566aa
    7. GET /team/info
    8. Send a request to /team/update
    9. GET /team/info
    View test on GitHub ↗
  615. ✓test_team_admin_cannot_raise_an_org_team_budget_under_the_org_cap TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an organization with organization alias: e2e-team-admin-org-d92ff624e304 and max budget: 100.0
    2. Create an internal user with user email: e2e-team-admin-9d574f0e9113@example.com and user role: internal_user
    3. Create a team with team alias: e2e-team-admin-32dbb4645f46, organization id: b704f7a0-9778-4538-aa1e-452f631a02b9, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: 49
    4. Generate a virtual key with user id: 49654b5c-826b-4124-a61e-5a98bf26364a
    5. GET /team/info
    6. Send a request to /team/update
    7. GET /team/info
    View test on GitHub ↗
  616. ✓test_team_admin_cannot_raise_or_remove_the_budget[raise] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-b959e5db060b@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-27856abe5ee5, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: 374e9754-b480-4de9-a108-2af9192f8353
    3. Generate a virtual key with user id: 374e9754-b480-4de9-a108-2af9192f8353
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  617. ✓test_team_admin_cannot_raise_or_remove_the_budget[remove] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-ae6e164edc4a@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-b0b14d96dd1f, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: 6fd6e2a0-3bb7-497d-8acc-c634d9c52a49
    3. Generate a virtual key with user id: 6fd6e2a0-3bb7-497d-8acc-c634d9c52a49
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  618. ✓test_team_admin_resending_the_budget_settings_keeps_the_next_budget_reset TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • spend-budgets
    • team_management
    1. Create an internal user with user email: e2e-team-admin-3519bbb83b5c@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-85df20f9ef33, tpm limit: 1000 and members with roles: role: admin and user id: ee71e363-923a-4898-be10-a2db99eb29b6
    3. Generate a virtual key with user id: ee71e363-923a-4898-be10-a2db99eb29b6
    4. POST /team/update
    5. GET /team/info
    6. Send a request to /team/update
    7. GET /team/info
    View test on GitHub ↗
  619. ✓test_team_admin_saves_a_new_rpm_limit_and_a_tighter_budget[first-budget] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-f4b4072ae2fa@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-04e8ec2b2b8b, tpm limit: 1000 and members with roles: role: admin and user id: 249cc8be-7ebf-4e8e-9fe2-c06dee0bb273
    3. Generate a virtual key with user id: 249cc8be-7ebf-4e8e-9fe2-c06dee0bb273
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  620. ✓test_team_admin_saves_a_new_rpm_limit_and_a_tighter_budget[lower] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. GET /get/ui_settings
    2. PATCH /update/ui_settings
    3. Wait for the last control-plane write to reach every proxy replica
    4. Create an internal user with user email: e2e-team-admin-864d6459fe84@example.com and user role: internal_user
    5. Create a team with team alias: e2e-team-admin-10b3069debcd, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: 1de97fd9-62e4-48cf-bc39-58f5283ed250
    6. Generate a virtual key with user id: 1de97fd9-62e4-48cf-bc39-58f5283ed250
    7. GET /team/info
    8. Send a request to /team/update
    9. GET /team/info
    View test on GitHub ↗
  621. ✓test_team_admin_saves_the_settings_form_with_a_new_tpm_limit TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. GET /get/ui_settings
    2. PATCH /update/ui_settings
    3. Wait for the last control-plane write to reach every proxy replica
    4. Create an internal user with user email: e2e-team-admin-7eda0e645be2@example.com and user role: internal_user
    5. Create a team with team alias: e2e-team-admin-42061761b579, tpm limit: 1000 and members with roles: role: admin and user id: ce9f56c3-1b04-4945-a5f6-bc7480f93461
    6. Generate a virtual key with user id: ce9f56c3-1b04-4945-a5f6-bc7480f93461
    7. GET /team/info
    8. Send a request to /team/update
    9. GET /team/info
    View test on GitHub ↗
  622. ✓test_team_delete_succeeds_for_team_larger_than_db_pool TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    1. (205 earlier steps not recorded)
    2. Create an internal user with user email: e2e-mgmt-bulk-204-6ffd42a0062b@example.com and user role: internal_user
    3. Create an internal user with user email: e2e-mgmt-bulk-205-f2322f95807f@example.com and user role: internal_user
    4. Create an internal user with user email: e2e-mgmt-bulk-206-609194ee242a@example.com and user role: internal_user
    5. Create an internal user with user email: e2e-mgmt-bulk-207-fbfbadcfb196@example.com and user role: internal_user
    6. Create an internal user with user email: e2e-mgmt-bulk-208-2ceece9728b1@example.com and user role: internal_user
    7. Create an internal user with user email: e2e-mgmt-bulk-209-f40ac3c26074@example.com and user role: internal_user
    8. Create an internal user with user email: e2e-mgmt-bulk-210-3df148e0cbc6@example.com and user role: internal_user
    9. Create an internal user with user email: e2e-mgmt-bulk-211-990912299369@example.com and user role: internal_user
    10. Create an internal user with user email: e2e-mgmt-bulk-212-ca1aaad6d1eb@example.com and user role: internal_user
    11. Create an internal user with user email: e2e-mgmt-bulk-213-3b8755d725b2@example.com and user role: internal_user
    12. Create an internal user with user email: e2e-mgmt-bulk-214-cb0d792bac7d@example.com and user role: internal_user
    13. Create an internal user with user email: e2e-mgmt-bulk-215-90106c16c252@example.com and user role: internal_user
    14. Create an internal user with user email: e2e-mgmt-bulk-216-a4e964d27091@example.com and user role: internal_user
    15. Create an internal user with user email: e2e-mgmt-bulk-217-534c04bf131b@example.com and user role: internal_user
    16. Create an internal user with user email: e2e-mgmt-bulk-218-45d64e5f51ac@example.com and user role: internal_user
    17. Create an internal user with user email: e2e-mgmt-bulk-219-88c2f0b3e95a@example.com and user role: internal_user
    18. Create an internal user with user email: e2e-mgmt-bulk-220-739e10fb7633@example.com and user role: internal_user
    19. Create an internal user with user email: e2e-mgmt-bulk-221-3de83a659f36@example.com and user role: internal_user
    20. Create an internal user with user email: e2e-mgmt-bulk-222-7dc119bbba08@example.com and user role: internal_user
    21. Create an internal user with user email: e2e-mgmt-bulk-223-a45223be249a@example.com and user role: internal_user
    22. Create an internal user with user email: e2e-mgmt-bulk-224-098e8ca86bb6@example.com and user role: internal_user
    23. Create an internal user with user email: e2e-mgmt-bulk-225-167cc827493b@example.com and user role: internal_user
    24. Create an internal user with user email: e2e-mgmt-bulk-226-c86fb07899d9@example.com and user role: internal_user
    25. Create an internal user with user email: e2e-mgmt-bulk-227-4c1e849cd259@example.com and user role: internal_user
    26. Create an internal user with user email: e2e-mgmt-bulk-228-615d6e7b2778@example.com and user role: internal_user
    27. Create an internal user with user email: e2e-mgmt-bulk-229-ee6085fbbd84@example.com and user role: internal_user
    28. Create an internal user with user email: e2e-mgmt-bulk-230-d695425a1e08@example.com and user role: internal_user
    29. Create an internal user with user email: e2e-mgmt-bulk-231-e176edcd6d3a@example.com and user role: internal_user
    30. Create an internal user with user email: e2e-mgmt-bulk-232-16d03381b7a9@example.com and user role: internal_user
    31. Create an internal user with user email: e2e-mgmt-bulk-233-58f35f9b79bb@example.com and user role: internal_user
    32. Create an internal user with user email: e2e-mgmt-bulk-234-89addcfec694@example.com and user role: internal_user
    33. Create an internal user with user email: e2e-mgmt-bulk-235-e2989e6b12cb@example.com and user role: internal_user
    34. Create an internal user with user email: e2e-mgmt-bulk-236-2475438c7104@example.com and user role: internal_user
    35. Create an internal user with user email: e2e-mgmt-bulk-237-31888951f08e@example.com and user role: internal_user
    36. Create an internal user with user email: e2e-mgmt-bulk-238-e3c00113468c@example.com and user role: internal_user
    37. Create an internal user with user email: e2e-mgmt-bulk-239-e9d189b06996@example.com and user role: internal_user
    38. Create an internal user with user email: e2e-mgmt-bulk-240-0f38cddd4fd7@example.com and user role: internal_user
    39. Create an internal user with user email: e2e-mgmt-bulk-241-374983ebe9d8@example.com and user role: internal_user
    40. Create an internal user with user email: e2e-mgmt-bulk-242-c2fbbb1d8815@example.com and user role: internal_user
    41. Create an internal user with user email: e2e-mgmt-bulk-243-2a6972a044a9@example.com and user role: internal_user
    42. Create an internal user with user email: e2e-mgmt-bulk-244-2a41a53bdc88@example.com and user role: internal_user
    43. Create an internal user with user email: e2e-mgmt-bulk-245-b5a50b4bf5e4@example.com and user role: internal_user
    44. Create an internal user with user email: e2e-mgmt-bulk-246-c27f8a5e1107@example.com and user role: internal_user
    45. Create an internal user with user email: e2e-mgmt-bulk-247-d45134dbff92@example.com and user role: internal_user
    46. Create an internal user with user email: e2e-mgmt-bulk-248-216bf37652eb@example.com and user role: internal_user
    47. Create an internal user with user email: e2e-mgmt-bulk-249-bff9dada9ae7@example.com and user role: internal_user
    48. Add a roster of members to the team with /team/member_add
    49. Read the team back from /team/info
    50. Try to delete the team with /team/delete
    51. Check whether /team/info finds the team
    View test on GitHub ↗
  623. ✓test_team_delete_writes_audit_row_for_team_keys TestKeyDeletionAuditLogtest_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-audit-team-9a0cd44259b4
    2. Generate a virtual key with team id: 67056460-6348-4ff3-9f2a-c0fddade19e0
    3. Delete the team
    4. Read the key's settings back from /key/info
    5. Read the key's deletion entries from the /audit log
    View test on GitHub ↗
  624. ✓test_team_member_delete_writes_audit_row_for_member_keys TestKeyDeletionAuditLogtest_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-audit-team-094b3637a837
    2. Create an internal user with user email: e2e-audit-f30307f823b9@example.com and user role: internal_user
    3. Add a user to the team with /team/member_add
    4. Generate a virtual key with user id: b4f8def1-0c6f-4540-be2d-fc951f1eed3c and team id: 2a448cc5-b2cc-4e8a-a694-88e05b24dc97
    5. Remove a user from the team with /team/member_delete
    6. Read the key's settings back from /key/info
    7. Read the key's deletion entries from the /audit log
    View test on GitHub ↗
  625. ✓test_tool_list_returns_catalog_with_consistent_total TestInventoryRoutestest_config_misc_endpoints_e2e.py›
    • management
    1. GET /v1/tool/list
    View test on GitHub ↗
  626. ✓test_two_actor_sets_keep_tenants_and_keys_isolated TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • key_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the team_member role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the team_member role
    4. Generate a virtual key as the proxy admin
    5. Read the key's settings back from /key/info
    View test on GitHub ↗
  627. ✓test_unblock_flips_key_info_blocked_back TestKeyManagementRoutestest_key_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with models: gpt-5.5
    2. POST /key/block
    3. Read the key's settings back from /key/info
    4. POST /key/unblock
    5. Read the key's settings back from /key/info
    View test on GitHub ↗
  628. ✓test_update_accepts_per_model_budgets_including_punctuated_names TestBudgetManagementtest_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. POST /budget/new
    2. POST /budget/update
    3. POST /budget/info
    4. POST /budget/new
    5. POST /budget/update
    6. POST /budget/info
    View test on GitHub ↗
  629. ✓test_update_alias_persists_to_organization_info TestOrganizationRoutestest_management_e2e.py›
    • management
    • organization_management
    1. Create an organization with organization alias: e2e-mgmt-org-dcc60fbb2bc8
    2. Update an organization with organization id: de2068a6-2c86-4a9b-9890-20dcc55815ab and organization alias: e2e-mgmt-org-a60f4d5fe4f8
    3. Read the organization back from /organization/info
    View test on GitHub ↗
  630. ✓test_update_forbidden_for_non_admin_key TestKeyManagementRoutestest_key_management_e2e.py›
    • proxy-auth
    • key_management
    1. Generate a virtual key with models: gpt-5.5
    2. Send a request to /key/update
    3. Read the key's settings back from /key/info
    View test on GitHub ↗
  631. ✓test_update_max_budget_persists_to_budget_info TestBudgetManagementtest_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. POST /budget/new
    2. POST /budget/info
    3. POST /budget/update
    4. POST /budget/info
    View test on GitHub ↗
  632. ✓test_update_models_persists_and_flips_enforcement TestKeyRoutestest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • gemini
    • openai
    • gemini-2.5-flash
    • gpt-5.5
    1. Generate a virtual key with models: gemini-2.5-flash
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Send a /chat/completions request to gpt-5.5
    4. Set the virtual key's models to [gpt-5.5]
    5. Read the key's settings back from /key/info
    6. Send a /chat/completions request to gpt-5.5
    7. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  633. ✓test_update_persists_input_cost_to_model_info TestModelRoutestest_management_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-model-82a8a4fa3d9f that calls gpt-4o-mini
    2. List the deployments from /model/info
    3. Update a deployment's settings to model: gpt-4o-mini and input cost per token: 2.22e-06
    4. List the deployments from /model/info
    View test on GitHub ↗
  634. ✓test_update_persists_to_team_info TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-mgmt-team-8f340af72ef1 and models: gemini-2.5-flash
    2. Update a team with team id: 1f0f0e1c-5fd2-473b-9c77-0255fb2ddcea and team alias: e2e-mgmt-team-updated-fe349ad1fec3
    3. Read the team back from /team/info
    View test on GitHub ↗
  635. ✓test_update_persists_to_user_info TestUserRoutestest_management_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-mgmt-09ae456fbcf0@example.com and user role: internal_user
    2. Read the user back from /user/info
    3. Update an internal user with user id: a155312f-d4f2-45f6-9d40-602c9b33fa76 and user role: internal_user_viewer
    4. Read the user back from /user/info
    View test on GitHub ↗
  636. ✓test_updating_only_the_alias_keeps_every_other_field_on_every_replica TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_c24d538c7307
    2. Update the MCP server's settings with PUT /v1/mcp/server
    3. Read /v1/mcp/server/7e347d57-65d6-49fa-affe-d98de54fb45d on every proxy replica until it settles
    View test on GitHub ↗
  637. ✓test_updating_only_the_description_keeps_the_tools_and_name TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_5d47b693fd20
    2. Create an MCP toolset with the tools server id: eed4a691-422d-42f7-9a28-a945fbb1c52d and tool name: search_datadog_logs, server id: eed4a691-422d-42f7-9a28-a945fbb1c52d and tool name: get_datadog_metr
    3. Update an MCP toolset with toolset id: d085b90f-ea60-4beb-aeb8-336b76a0de75 and description: edited
    4. Read /v1/mcp/toolset/d085b90f-ea60-4beb-aeb8-336b76a0de75 on every proxy replica until it settles
    View test on GitHub ↗
  638. ✓test_updating_the_tools_to_one_entry_reads_back_exactly_that_entry TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_335e33cd6f4b
    2. Create an MCP toolset with the tools server id: 7e9b6741-b57f-4327-b83e-b933e464fb84 and tool name: search_datadog_logs, server id: 7e9b6741-b57f-4327-b83e-b933e464fb84 and tool name: get_datadog_metr
    3. Update an MCP toolset with toolset id: da2ca8f3-0f71-45ac-a201-ba3be44dee42 and tools: server id: 7e9b6741-b57f-4327-b83e-b933e464fb84 and tool name: search_datadog_logs
    4. Read /v1/mcp/toolset/da2ca8f3-0f71-45ac-a201-ba3be44dee42 on every proxy replica until it settles
    View test on GitHub ↗
  639. ✓test_user_delete_writes_audit_row_for_user_keys TestKeyDeletionAuditLogtest_management_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-audit-5d743c8566c8@example.com and user role: internal_user
    2. Generate a virtual key with user id: ad7c4716-d7c8-4e5b-9a49-37c489030910
    3. Delete the internal user
    4. Read the key's settings back from /key/info
    5. Read the key's deletion entries from the /audit log
    View test on GitHub ↗
  640. ✓test_workflow_runs_list_returns_consistent_count TestInventoryRoutestest_config_misc_endpoints_e2e.py›
    • management
    1. GET /v1/workflows/runs
    View test on GitHub ↗
  641. −test_created_server_is_listed_with_every_field TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp

    No steps recorded

    product gap: GET /v1/mcp/server builds each row from the in-memory registry, whose _build_mcp_server_table sets description from mcp_info['description'], so the list reports the mcp_info description while GET /v1/mcp/server/{server_id} reports the stored description column. A server created with both set to different text reads back with two different descriptions depending on the route

    View test on GitHub ↗
  642. ✓test_basic_messaging_non_streaming_anthropicbasic_messaging_non_streaming/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 through the proxy and check every model replies
    26. Run Claude Code headless against claude-haiku-4-5 through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    28. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  643. ✓test_basic_messaging_non_streaming_azurebasic_messaging_non_streaming/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure through the proxy and check every model replies
    26. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    28. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  644. ✓test_basic_messaging_non_streaming_azure_openaibasic_messaging_non_streaming/test_azure_openai.py›
    • llm-translation
    • messages
    • nonstream
    • azure
    • gpt-5-6-luna-azure-openai
    • gpt-5-6-sol-azure-openai
    • gpt-5-6-terra-azure-openai
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against gpt-5-6-sol-azure-openai, gpt-5-6-terra-azure-openai, gpt-5-6-luna-azure-openai through the proxy and check every model replies
    26. Run Claude Code headless against gpt-5-6-sol-azure-openai through the proxy
    27. Run Claude Code headless against gpt-5-6-terra-azure-openai through the proxy
    28. Run Claude Code headless against gpt-5-6-luna-azure-openai through the proxy
    View test on GitHub ↗
  645. ✓test_basic_messaging_non_streaming_bedrock_conversebasic_messaging_non_streaming/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse through the proxy and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  646. ✓test_basic_messaging_non_streaming_bedrock_invokebasic_messaging_non_streaming/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke through the proxy and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  647. ✓test_basic_messaging_non_streaming_vertex_aibasic_messaging_non_streaming/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex through the proxy and check every model replies
    26. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    28. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  648. ✓test_basic_messaging_non_streaming_vertex_ai_gptbasic_messaging_non_streaming/test_vertex_ai_gpt.py›
    • llm-translation
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    View test on GitHub ↗
  649. ✓test_basic_messaging_streaming_anthropicbasic_messaging_streaming/test_anthropic.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 through the proxy and check every model replies
    26. Run Claude Code headless against claude-haiku-4-5 through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    28. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  650. ✓test_basic_messaging_streaming_azurebasic_messaging_streaming/test_azure.py›
    • llm-translation
    • messages
    • stream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure through the proxy and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  651. ✓test_basic_messaging_streaming_azure_openaibasic_messaging_streaming/test_azure_openai.py›
    • llm-translation
    • messages
    • stream
    • azure
    • gpt-5-6-luna-azure-openai
    • gpt-5-6-sol-azure-openai
    • gpt-5-6-terra-azure-openai
    1. Run Claude Code headless against gpt-5-6-sol-azure-openai, gpt-5-6-terra-azure-openai, gpt-5-6-luna-azure-openai through the proxy and check every model replies
    2. Run Claude Code headless against gpt-5-6-sol-azure-openai through the proxy
    3. Run Claude Code headless against gpt-5-6-terra-azure-openai through the proxy
    4. Run Claude Code headless against gpt-5-6-luna-azure-openai through the proxy
    View test on GitHub ↗
  652. ✓test_basic_messaging_streaming_bedrock_conversebasic_messaging_streaming/test_bedrock_converse.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse through the proxy and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  653. ✓test_basic_messaging_streaming_bedrock_invokebasic_messaging_streaming/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke through the proxy and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  654. ✓test_basic_messaging_streaming_vertex_aibasic_messaging_streaming/test_vertex_ai.py›
    • llm-translation
    • messages
    • stream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex through the proxy and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  655. ✓test_basic_messaging_streaming_vertex_ai_gptbasic_messaging_streaming/test_vertex_ai_gpt.py›
    • llm-translation

    No steps recorded

    View test on GitHub ↗
  656. ✓test_count_tokens_anthropiccount_tokens/test_anthropic.py›
    • llm-translation
    • count_tokens
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5 on the message "hello world"
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5 on the message "hello world"
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7 on the message "hello world"
    View test on GitHub ↗
  657. ✓test_count_tokens_azurecount_tokens/test_azure.py›
    • llm-translation
    • count_tokens
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5-azure on the message "hello world"
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5-azure on the message "hello world"
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7-azure on the message "hello world"
    View test on GitHub ↗
  658. ✓test_count_tokens_bedrock_conversecount_tokens/test_bedrock_converse.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5-bedrock-converse on the message "hello world"
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5-bedrock-converse on the message "hello world"
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7-bedrock-converse on the message "hello world"
    View test on GitHub ↗
  659. ✓test_count_tokens_bedrock_invokecount_tokens/test_bedrock_invoke.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5-bedrock-invoke on the message "hello world"
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5-bedrock-invoke on the message "hello world"
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7-bedrock-invoke on the message "hello world"
    View test on GitHub ↗
  660. ✓test_passthrough_anthropicpassthrough/test_anthropic.py›
    • passthrough
    • passthrough
    • stream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 through the proxy's native provider passthrough route and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  661. ✓test_passthrough_bedrock_conversepassthrough/test_bedrock_converse.py›
    • passthrough

    No steps recorded

    View test on GitHub ↗
  662. ✓test_passthrough_bedrock_invokepassthrough/test_bedrock_invoke.py›
    • passthrough
    • passthrough
    • stream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke through the proxy's native provider passthrough route and check every
    26. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    28. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  663. ✓test_passthrough_vertex_aipassthrough/test_vertex_ai.py›
    • passthrough
    • passthrough
    • stream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex through the proxy's native provider passthrough route and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  664. ✓test_pdf_input_anthropicpdf_input/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • pdf_input
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  665. ✓test_pdf_input_azurepdf_input/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • pdf_input
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  666. ✓test_pdf_input_bedrock_conversepdf_input/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • pdf_input
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  667. ✓test_pdf_input_bedrock_invokepdf_input/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • pdf_input
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  668. ✓test_pdf_input_vertex_aipdf_input/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • pdf_input
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  669. ✓test_prompt_caching_1h_anthropicprompt_caching_1h/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  670. ✓test_prompt_caching_1h_azureprompt_caching_1h/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  671. ✓test_prompt_caching_1h_bedrock_converseprompt_caching_1h/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  672. ✓test_prompt_caching_1h_bedrock_invokeprompt_caching_1h/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  673. ✓test_prompt_caching_1h_vertex_aiprompt_caching_1h/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  674. ✓test_prompt_caching_5m_anthropicprompt_caching_5m/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  675. ✓test_prompt_caching_5m_azureprompt_caching_5m/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  676. ✓test_prompt_caching_5m_bedrock_converseprompt_caching_5m/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  677. ✓test_prompt_caching_5m_bedrock_invokeprompt_caching_5m/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  678. ✓test_prompt_caching_5m_vertex_aiprompt_caching_5m/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  679. ✓test_structured_outputs_anthropicstructured_outputs/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • response_schema
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  680. ✓test_structured_outputs_azurestructured_outputs/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • response_schema
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  681. ✓test_structured_outputs_bedrock_conversestructured_outputs/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • response_schema
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  682. ✓test_structured_outputs_bedrock_invokestructured_outputs/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • response_schema
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  683. ✓test_structured_outputs_vertex_aistructured_outputs/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • response_schema
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  684. ✓test_thinking_anthropicthinking/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  685. ✓test_thinking_azurethinking/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  686. ✓test_thinking_bedrock_conversethinking/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  687. ✓test_thinking_bedrock_invokethinking/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  688. ✓test_thinking_vertex_aithinking/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  689. ✓test_thinking_with_tool_use_anthropicthinking_with_tool_use/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • function_calling
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  690. ✓test_thinking_with_tool_use_azurethinking_with_tool_use/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • function_calling
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  691. ✓test_thinking_with_tool_use_bedrock_conversethinking_with_tool_use/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • function_calling
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  692. ✓test_thinking_with_tool_use_bedrock_invokethinking_with_tool_use/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • function_calling
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  693. ✓test_thinking_with_tool_use_vertex_aithinking_with_tool_use/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • function_calling
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  694. ✓test_tool_search_anthropictool_search/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • tool_search
    1. Send a /v1/messages request to claude-haiku-4-5 with the tool_search tool declared
    2. Send a /v1/messages request to claude-sonnet-4-5 with the tool_search tool declared
    3. Send a /v1/messages request to claude-opus-4-7 with the tool_search tool declared
    View test on GitHub ↗
  695. ✓test_tool_search_bedrock_conversetool_search/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • tool_search
    1. Send a /v1/messages request to claude-haiku-4-5-bedrock-converse with the tool_search tool declared
    2. Send a /v1/messages request to claude-sonnet-4-5-bedrock-converse with the tool_search tool declared
    3. Send a /v1/messages request to claude-opus-4-7-bedrock-converse with the tool_search tool declared
    View test on GitHub ↗
  696. ✓test_tool_search_bedrock_invoketool_search/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • tool_search
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Send a /v1/messages request to claude-haiku-4-5-bedrock-invoke with the tool_search tool declared
    26. Send a /v1/messages request to claude-sonnet-4-5-bedrock-invoke with the tool_search tool declared
    27. Send a /v1/messages request to claude-opus-4-7-bedrock-invoke with the tool_search tool declared
    View test on GitHub ↗
  697. ✓test_tool_search_history_bedrock_invoketool_search/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • tool_search
    1. Send a /v1/messages request to claude-haiku-4-5-bedrock-invoke with the tool_search tool declared, then send its answer back as history in a second request
    2. Send a /v1/messages request to claude-sonnet-4-5-bedrock-invoke with the tool_search tool declared, then send its answer back as history in a second request
    3. Send a /v1/messages request to claude-opus-4-7-bedrock-invoke with the tool_search tool declared, then send its answer back as history in a second request
    View test on GitHub ↗
  698. ✓test_tool_use_anthropictool_use/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  699. ✓test_tool_use_azuretool_use/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  700. ✓test_tool_use_azure_openaitool_use/test_azure_openai.py›
    • llm-translation
    • messages
    • nonstream
    • azure
    • gpt-5-6-luna-azure-openai
    • gpt-5-6-sol-azure-openai
    • gpt-5-6-terra-azure-openai
    • function_calling
    1. Run Claude Code headless against gpt-5-6-sol-azure-openai, gpt-5-6-terra-azure-openai, gpt-5-6-luna-azure-openai in parallel through the proxy
    2. Run Claude Code headless against gpt-5-6-sol-azure-openai through the proxy
    3. Run Claude Code headless against gpt-5-6-terra-azure-openai through the proxy
    4. Run Claude Code headless against gpt-5-6-luna-azure-openai through the proxy
    View test on GitHub ↗
  701. ✓test_tool_use_bedrock_conversetool_use/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  702. ✓test_tool_use_bedrock_invoketool_use/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  703. ✓test_tool_use_streaming_anthropictool_use_streaming/test_anthropic.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  704. ✓test_tool_use_streaming_azuretool_use_streaming/test_azure.py›
    • llm-translation
    • messages
    • stream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  705. ✓test_tool_use_streaming_azure_openaitool_use_streaming/test_azure_openai.py›
    • llm-translation
    • messages
    • stream
    • azure
    • gpt-5-6-luna-azure-openai
    • gpt-5-6-sol-azure-openai
    • gpt-5-6-terra-azure-openai
    • function_calling
    1. Run Claude Code headless against gpt-5-6-sol-azure-openai, gpt-5-6-terra-azure-openai, gpt-5-6-luna-azure-openai in parallel through the proxy
    2. Run Claude Code headless against gpt-5-6-sol-azure-openai through the proxy
    3. Run Claude Code headless against gpt-5-6-terra-azure-openai through the proxy
    4. Run Claude Code headless against gpt-5-6-luna-azure-openai through the proxy
    View test on GitHub ↗
  706. ✓test_tool_use_streaming_bedrock_conversetool_use_streaming/test_bedrock_converse.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  707. ✓test_tool_use_streaming_bedrock_invoketool_use_streaming/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  708. ✓test_tool_use_streaming_vertex_aitool_use_streaming/test_vertex_ai.py›
    • llm-translation
    • messages
    • stream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  709. ✓test_tool_use_streaming_vertex_ai_gpttool_use_streaming/test_vertex_ai_gpt.py›
    • llm-translation

    No steps recorded

    View test on GitHub ↗
  710. ✓test_tool_use_vertex_aitool_use/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  711. ✓test_tool_use_vertex_ai_gpttool_use/test_vertex_ai_gpt.py›
    • llm-translation

    No steps recorded

    View test on GitHub ↗
  712. ✓test_vision_anthropicvision/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • vision
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  713. ✓test_vision_azurevision/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • vision
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  714. ✓test_vision_bedrock_conversevision/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • vision
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  715. ✓test_vision_bedrock_invokevision/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • vision
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  716. ✓test_vision_vertex_aivision/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • vision
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  717. ✓test_web_search_anthropicweb_search/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • web_search
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  718. ✓test_web_search_azureweb_search/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • web_search
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  719. ✓test_web_search_bedrock_converseweb_search/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • web_search
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  720. ✓test_web_search_bedrock_invokeweb_search/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • web_search
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  721. ✓test_web_search_vertex_aiweb_search/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • web_search
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  722. −test_basic_messaging_non_streaming_bedrock_mantlebasic_messaging_non_streaming/test_bedrock_mantle.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock_mantle
    • gpt-5-6-luna-bedrock-mantle
    • gpt-5-6-sol-bedrock-mantle
    • gpt-5-6-terra-bedrock-mantle

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  723. −test_basic_messaging_non_streaming_openaibasic_messaging_non_streaming/test_openai.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • gpt-5-6-luna-openai
    • gpt-5-6-sol-openai
    • gpt-5-6-terra-openai

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  724. −test_basic_messaging_streaming_bedrock_mantlebasic_messaging_streaming/test_bedrock_mantle.py›
    • llm-translation
    • messages
    • stream
    • bedrock_mantle
    • gpt-5-6-luna-bedrock-mantle
    • gpt-5-6-sol-bedrock-mantle
    • gpt-5-6-terra-bedrock-mantle

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  725. −test_basic_messaging_streaming_openaibasic_messaging_streaming/test_openai.py›
    • llm-translation
    • messages
    • stream
    • openai
    • gpt-5-6-luna-openai
    • gpt-5-6-sol-openai
    • gpt-5-6-terra-openai

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  726. −test_count_tokens_vertex_aicount_tokens/test_vertex_ai.py›
    • llm-translation
    • count_tokens
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex

    No steps recorded

    stage red: Vertex returns not supported for token counting for Claude aliases

    View test on GitHub ↗
  727. −test_long_context_1m_anthropiclong_context_1m/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-opus-4-7
    • claude-sonnet-4-5

    No steps recorded

    stage red: 1M long_context not green on stage Anthropic path yet (200k sonnet / model alias)

    View test on GitHub ↗
  728. −test_long_context_1m_azurelong_context_1m/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure

    No steps recorded

    stage red: 1M long_context not green on stage Azure Foundry deployments yet

    View test on GitHub ↗
  729. −test_long_context_1m_bedrock_converselong_context_1m/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse

    No steps recorded

    stage red: 1M long_context not green on stage Bedrock Converse deployments yet

    View test on GitHub ↗
  730. −test_long_context_1m_bedrock_invokelong_context_1m/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke

    No steps recorded

    stage red: 1M long_context not green on stage Bedrock Invoke deployments yet

    View test on GitHub ↗
  731. −test_long_context_1m_vertex_ailong_context_1m/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex

    No steps recorded

    stage red: 1M long_context not green on stage Vertex deployments yet

    View test on GitHub ↗
  732. −test_passthrough_azurepassthrough/test_azure.py›
    • passthrough
    • passthrough
    • stream
    • azure
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5

    No steps recorded

    stage red: /azure passthrough drops client headers (e.g. anthropic-version); product gap

    View test on GitHub ↗
  733. −test_tool_search_azuretool_search/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • tool_search

    No steps recorded

    stage red: Azure Foundry tool_search_server not supported in workspace for probed models

    View test on GitHub ↗
  734. −test_tool_search_vertex_aitool_search/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • tool_search

    No steps recorded

    stage red: Vertex rejects tool_search when deployment extra_headers inject context-1m beta; product/config

    View test on GitHub ↗
  735. −test_tool_use_bedrock_mantletool_use/test_bedrock_mantle.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock_mantle
    • gpt-5-6-luna-bedrock-mantle
    • gpt-5-6-sol-bedrock-mantle
    • gpt-5-6-terra-bedrock-mantle
    • function_calling

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  736. −test_tool_use_openaitool_use/test_openai.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • gpt-5-6-luna-openai
    • gpt-5-6-sol-openai
    • gpt-5-6-terra-openai
    • function_calling

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  737. −test_tool_use_streaming_bedrock_mantletool_use_streaming/test_bedrock_mantle.py›
    • llm-translation
    • messages
    • stream
    • bedrock_mantle
    • gpt-5-6-luna-bedrock-mantle
    • gpt-5-6-sol-bedrock-mantle
    • gpt-5-6-terra-bedrock-mantle
    • function_calling

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  738. −test_tool_use_streaming_openaitool_use_streaming/test_openai.py›
    • llm-translation
    • messages
    • stream
    • openai
    • gpt-5-6-luna-openai
    • gpt-5-6-sol-openai
    • gpt-5-6-terra-openai
    • function_calling

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  739. ✓test_batch_create_over_rpm_returns_mapped_429 TestBatchRateLimitErrorMappingtest_batches_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-batch-rl-map-8c7cba90b8c2, tpm limit: 1000000 and rpm limit: 2
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    View test on GitHub ↗
  740. ✓test_batch_key_model_access_deniedtest_batches_e2e.py›
    • proxy-auth
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with models: openai-batch-72b300251ef2 and user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    View test on GitHub ↗
  741. ✓test_batch_lifecycle[azure-encoded]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  742. ✓test_batch_lifecycle[azure-model_param]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  743. ✓test_batch_lifecycle[azure-provider_fallback]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  744. ✓test_batch_lifecycle[azure-unified]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  745. ✓test_batch_lifecycle[bedrock-unified]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  746. ✓test_batch_lifecycle[openai-encoded]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Call the management route /health/liveliness
    2. Add a batch deployment named openai-batch-72b300251ef2 that calls openai/gpt-4o-mini
    3. Add a batch deployment named azure-batch-72b300251ef2 that calls azure/gpt-5.4-mini-batch
    4. Add a batch deployment named vertex-batch-72b300251ef2 that calls vertex_ai/gemini-2.5-flash
    5. Add a batch deployment named bedrock-batch-72b300251ef2 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    6. Generate a virtual key with user id: e2e-test-user
    7. Upload a batch input file to /v1/files
    8. Create a batch of /v1/chat/completions requests from the uploaded file
    9. Retrieve the batch
    10. Cancel the batch
    11. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  747. ✓test_batch_lifecycle[openai-model_param]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  748. ✓test_batch_lifecycle[openai-provider_fallback]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  749. ✓test_batch_lifecycle[openai-unified]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  750. ✓test_batch_lifecycle[vertex_ai-encoded]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  751. ✓test_batch_lifecycle[vertex_ai-model_param]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  752. ✓test_batch_lifecycle[vertex_ai-provider_fallback]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  753. ✓test_batch_lifecycle[vertex_ai-unified]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  754. ✓test_completed_batch_downloads_output_and_books_cost TestBatchTerminalStatetest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  755. ✓test_endpoint_mismatch_fails_batch_and_cancel_conflicts TestBatchFailurePathstest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Wait for the request's spend log in /spend/logs
    6. Cancel the batch
    View test on GitHub ↗
  756. ✓test_enqueued_allowance_accepts_batch_over_key_rpm TestBatchEnqueuedTokenLimittest_batches_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-batch-enq-rpm-bebe132ee500, rpm limit: 2 and metadata: batch enqueued token limit: 100000
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    View test on GitHub ↗
  757. ✓test_exhausted_allowance_blocks_until_cancel_refunds TestBatchEnqueuedTokenLimittest_batches_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-batch-enq-size-a132501854f9 and metadata: batch enqueued token limit: 1
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Generate a virtual key with user id: e2e-batch-enq-refund-886524650a3f and metadata: batch enqueued token limit: 36
    5. Upload a batch input file to /v1/files
    6. Create a batch of /v1/chat/completions requests from the uploaded file
    7. Cancel the batch
    8. Create a batch of /v1/chat/completions requests from the uploaded file
    View test on GitHub ↗
  758. ✓test_file_content_matches_upload TestBatchFileContenttest_batches_e2e.py›
    • llm-translation
    • files
    • openai
    • gpt-4o-mini
    1. Start a provider edge server
    2. Add a batch deployment named e2e-file-content-e603c6c5b7ac that calls openai/gpt-4o-mini
    3. Generate a virtual key with user id: e2e-test-user
    4. Upload a batch input file to /v1/files
    5. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLDYwNzcyM2RhLWY1ZjMtNDE3Mi05NzczLTYwNDcwNGUwOWE4ODt0YXJnZXRfbW9kZWxfbmFtZXMsZTJlLWZpbGUtY29udGVudC1lNjAzYzZjNWI3YWM7bGxtX291d
    View test on GitHub ↗
  759. ✓test_file_lifecycle_signs_s3_with_s3_credentials TestBedrockBatchSplitS3Credentialstest_batches_e2e.py›
    • llm-translation
    • files
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a batch deployment named bedrock-split-s3-batch-72b300251ef2 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload a batch input file to /v1/files
    4. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLGVkNGU4NzU0LTA2N2EtNDhjYS1hYTc4LWM5MGQzZGM2NmQ2Njt0YXJnZXRfbW9kZWxfbmFtZXMsYmVkcm9jay1zcGxpdC1zMy1iYXRjaC03MmIzMDAyNTFlZjI7b
    5. Delete the uploaded file
    View test on GitHub ↗
  760. ✓test_file_upload_and_delete_outputstest_batches_e2e.py›
    • llm-translation
    • files
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Delete the uploaded file
    View test on GitHub ↗
  761. ✓test_foreign_encoded_file_id_routes_by_file_model TestBatchFailurePathstest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    View test on GitHub ↗
  762. ✓test_gemini_file_upload TestGeminiFilestest_batches_e2e.py›
    • llm-translation
    • files
    • gemini
    • gemini-2.5-flash
    1. Add a batch deployment named gemini-files-72b300251ef2 that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload a batch input file to /v1/files
    View test on GitHub ↗
  763. ✓test_list_page_cursors_address_only_the_callers_own_files TestOpenAIFilestest_batches_e2e.py›
    • llm-translation
    • files
    • openai
    1. Generate a virtual key with user id: e2e-file-list-12e0422ffdb9
    2. List the files the key can see from /v1/files
    View test on GitHub ↗
  764. ✓test_malformed_jsonl_upload_rejected TestBatchFailurePathstest_batches_e2e.py›
    • llm-translation
    • files
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    View test on GitHub ↗
  765. ✓test_native_jsonl_round_trips_untouched_and_starts_a_batch TestVertexNativePassthroughtest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLDMwMWMwMzc4LTRlMTUtNDg1MC04NGJlLTJlOGQyYzc2NTQwMjt0YXJnZXRfbW9kZWxfbmFtZXMsdmVydGV4LWJhdGNoLTcyYjMwMDI1MWVmMjtsbG1fb3V0cHV0X
    4. Create a batch of /v1/chat/completions requests from the uploaded file
    View test on GitHub ↗
  766. ✓test_passthrough_upload_is_rejected_outside_a_native_vertex_batch[no-target-model] TestVertexNativePassthroughtest_batches_e2e.py›
    • llm-translation
    • files
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    View test on GitHub ↗
  767. ✓test_passthrough_upload_is_rejected_outside_a_native_vertex_batch[non-vertex-target-model] TestVertexNativePassthroughtest_batches_e2e.py›
    • llm-translation
    • files
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    View test on GitHub ↗
  768. ✓test_passthrough_upload_is_rejected_outside_a_native_vertex_batch[openai-shaped-rows] TestVertexNativePassthroughtest_batches_e2e.py›
    • llm-translation
    • files
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    View test on GitHub ↗
  769. ✓test_retrieve_round_trips_metadata TestOpenAIFilestest_batches_e2e.py›
    • llm-translation
    • files
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Retrieve the uploaded file
    View test on GitHub ↗
  770. ✓test_unified_batch_create_with_assume_role TestBedrockBatchAssumeRoletest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a batch deployment named bedrock-sts-batch-72b300251ef2 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload a batch input file to /v1/files
    4. Create a batch of /v1/chat/completions requests from the uploaded file
    5. Retrieve the batch
    View test on GitHub ↗
  771. ✓test_unified_create_and_retrieve_via_chained_gateway TestBatchSecondHoptest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • litellm_proxy
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a batch deployment named openai-batch-hop-72b300251ef2 that calls litellm_proxy/openai-batch-72b300251ef2
    3. Upload a batch input file to /v1/files
    4. Create a batch of /v1/chat/completions requests from the uploaded file
    5. Retrieve the batch
    View test on GitHub ↗
  772. ✓test_unified_file_content_downloads[azure] TestBatchFileContenttest_batches_e2e.py›
    • llm-translation
    • files
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLGRjZGE1NzkzLWU0MzQtNDM5Ni1hMGNmLTcwNWVkNDhiNWUwODt0YXJnZXRfbW9kZWxfbmFtZXMsYXp1cmUtYmF0Y2gtNzJiMzAwMjUxZWYyO2xsbV9vdXRwdXRfZ
    View test on GitHub ↗
  773. ✓test_unified_file_content_downloads[bedrock] TestBatchFileContenttest_batches_e2e.py›
    • llm-translation
    • files
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLGNkYTJlMjE0LWFhM2YtNGJiZi1hNTJiLTg0NGFhMjEwODYyZTt0YXJnZXRfbW9kZWxfbmFtZXMsYmVkcm9jay1iYXRjaC03MmIzMDAyNTFlZjI7bGxtX291dHB1d
    View test on GitHub ↗
  774. ✓test_unified_file_content_downloads[vertex_ai] TestBatchFileContenttest_batches_e2e.py›
    • llm-translation
    • files
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLDg0ODA5MTM1LTUxZGItNGJlYS1iZTJhLTMzYzEwYzNkMjhlNTt0YXJnZXRfbW9kZWxfbmFtZXMsdmVydGV4LWJhdGNoLTcyYjMwMDI1MWVmMjtsbG1fb3V0cHV0X
    View test on GitHub ↗
  775. ✓test_unified_file_upload_and_batch_create_in_govcloud TestBedrockBatchGovCloudtest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • bedrock
    • bedrock/amazon.nova-lite-v1:0
    1. Add a batch deployment named bedrock-govcloud-batch-72b300251ef2 that calls bedrock/amazon.nova-lite-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload a batch input file to /v1/files
    4. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLDBhMzc0NTI1LTIzMmQtNGMyMy1hMWEwLTQ2OTUzOWQ0MDVjMjt0YXJnZXRfbW9kZWxfbmFtZXMsYmVkcm9jay1nb3ZjbG91ZC1iYXRjaC03MmIzMDAyNTFlZjI7b
    5. Create a batch of /v1/chat/completions requests from the uploaded file
    6. Retrieve the batch
    View test on GitHub ↗
  776. −test_batch_runs_to_completion_with_a_downloadable_output[model_query] TestHostedVllmBatchtest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • hosted_vllm
    • Qwen/Qwen2.5-0.5B-Instruct

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  777. −test_batch_runs_to_completion_with_a_downloadable_output[target_model_names] TestHostedVllmBatchtest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • hosted_vllm
    • Qwen/Qwen2.5-0.5B-Instruct

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  778. −test_failing_line_lands_in_the_error_file_not_the_batch_status TestHostedVllmBatchtest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • hosted_vllm
    • Qwen/Qwen2.5-0.5B-Instruct

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  779. −test_rate_limited_batch_create_leaves_no_unattributed_spend_rowtest_batches_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • gpt-4o-mini

    No steps recorded

    LIT-5027: the path under test hangs. The batch rate limiter reads the input file to count tokens by awaiting litellm.afile_content with no timeout, so a slow Files API holds POST /v1/batches open past any client deadline (63.6s observed on stage against a 60s read timeout). The unattributed-spend-row contract below is never reached, so the test reports a timeout rather than the behavior it guards. Unskip once the fetch is bounded.

    View test on GitHub ↗
  780. −test_uploaded_file_appears_in_list TestOpenAIFilestest_batches_e2e.py›
    • llm-translation
    • files
    • openai
    • gpt-4o-mini

    No steps recorded

    LIT-4820 (https://linear.app/litellm-ai/issue/LIT-4820): GET /v1/files omits newly uploaded files. The upload succeeds and GET /v1/files/{id} returns the file, but it never appears in the listing; the returned set is stable with its newest entry ~10h old, on both the managed (/v1/files?model=) and provider-scoped (/openai/v1/files) routes. Skipped rather than weakened because the assertion below is the correct contract. Remove this marker when LIT-4820 is fixed; do not relax the assertion to make it pass.

    View test on GitHub ↗
  781. ✓test_429_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    • routing
    • nonstream
    • openai
    • gpt-5.5
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Generate a virtual key with models: gpt-5.5, user id: e2e-test-user and rpm limit: 1
    3. Add a deployment named reliability-retry-429-45d90e78107e that fronts gpt-5.5 on this proxy with a key out of rpm, so it always gets a 429 and is benched on the first one
    4. Add a zero-weight backup deployment named reliability-retry-429-45d90e78107e that calls openai/gpt-5.5
    5. Use up the rpm-limited key's one allowed request with a /chat/completions call to gpt-5.5
    6. Send a /chat/completions request to reliability-retry-429-45d90e78107e with stream set to False
    View test on GitHub ↗
  782. ✓test_429_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    • routing
    • nonstream
    • openai
    • gpt-5.5
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Generate a virtual key with models: gpt-5.5, user id: e2e-test-user and rpm limit: 1
    3. Add a deployment named reliability-cooldown-429-ea63878d5b45 that fronts gpt-5.5 on this proxy with a key out of rpm, so it always gets a 429 and is benched on the first one
    4. Add a zero-weight backup deployment named reliability-cooldown-429-ea63878d5b45 that calls openai/gpt-5.5
    5. Use up the rpm-limited key's one allowed request with a /chat/completions call to gpt-5.5
    6. Send a /chat/completions request to reliability-cooldown-429-ea63878d5b45 with stream set to False
    View test on GitHub ↗
  783. ✓test_5xx_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-5xx-upstream-21a3c48ce677 that calls openai/gpt-5.5 at an unreachable address
    3. Add a deployment named reliability-retry-5xx-6dc3d35c5b31 that fronts reliability-5xx-upstream-21a3c48ce677 on this proxy, so it always gets a 500 and is benched on the first one
    4. Add a zero-weight backup deployment named reliability-retry-5xx-6dc3d35c5b31 that calls openai/gpt-5.5
    5. Send a /chat/completions request to reliability-retry-5xx-6dc3d35c5b31 with stream set to False
    View test on GitHub ↗
  784. ✓test_5xx_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    • routing
    • nonstream
    • openai
    • gpt-5.5
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-fail-ad7deceb0c13 that calls openai/gpt-5.5 at an unreachable address
    3. Send a /chat/completions request to reliability-fail-ad7deceb0c13 with stream set to False
    View test on GitHub ↗
  785. ✓test_5xx_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cooldown-5xx-upstream-398450c74522 that calls openai/gpt-5.5 at an unreachable address
    3. Add a deployment named reliability-cooldown-5xx-d11d837f69b9 that fronts reliability-cooldown-5xx-upstream-398450c74522 on this proxy, so it always gets a 500 and is benched on the first one
    4. Add a zero-weight backup deployment named reliability-cooldown-5xx-d11d837f69b9 that calls openai/gpt-5.5
    5. Send a /chat/completions request to reliability-cooldown-5xx-d11d837f69b9 with stream set to False
    View test on GitHub ↗
  786. ✓test_alias_api_key_never_overrides_the_tier_credential TestAliasParamForwardingtest_auto_router_regressions_e2e.py›
    • routing
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-cred-tier-470a78abe078 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named e2e-cred-alias-470a78abe078 that calls auto_router/complexity_router
    3. Generate a virtual key with models: e2e-cred-alias-470a78abe078, e2e-cred-tier-470a78abe078 and user id: e2e-auto-router-regressions
    4. Send a /chat/completions request to e2e-cred-alias-470a78abe078
    5. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  787. ✓test_auth_failure_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-retry-auth-00ef9ccfde16 that calls openai/gpt-5.5 with an invalid key and is benched on its first 401
    3. Add a zero-weight backup deployment named reliability-retry-auth-00ef9ccfde16 that calls openai/gpt-5.5
    4. Send a /chat/completions request to reliability-retry-auth-00ef9ccfde16 with stream set to False
    View test on GitHub ↗
  788. ✓test_auth_failure_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cooldown-auth-f699c18e1624 that calls openai/gpt-5.5 with an invalid key and is benched on its first 401
    3. Add a zero-weight backup deployment named reliability-cooldown-auth-f699c18e1624 that calls openai/gpt-5.5
    4. Send a /chat/completions request to reliability-cooldown-auth-f699c18e1624 with stream set to False
    View test on GitHub ↗
  789. ✓test_body_tagged_chat_reaches_the_untagged_tier_after_marker_rewrite TestUntaggedTierDeploymentstest_auto_router_regressions_e2e.py›
    • routing
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with models: e2e-autoroute-94f1d02859e2, e2e-tier-94f1d02859e2, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /chat/completions request to e2e-autoroute-94f1d02859e2
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  790. ✓test_body_tagged_chat_routes_through_the_marker_to_its_tier TestTagSplitRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-autoroute-27a5e2bdf08e that calls anthropic/claude-sonnet-5
    2. Add a deployment named e2e-tier-27a5e2bdf08e that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-autoroute-27a5e2bdf08e that calls auto_router/complexity_router
    4. Generate a virtual key with models: e2e-autoroute-27a5e2bdf08e, e2e-tier-27a5e2bdf08e, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    5. Send a /chat/completions request to e2e-autoroute-27a5e2bdf08e
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  791. ✓test_body_tagged_responses_with_list_input_routes_to_the_tier TestResponsesApiTagRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with models: e2e-autoroute-f731359a7305, e2e-tier-f731359a7305, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. POST /v1/responses
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  792. ✓test_cached_conversation_stays_on_deployment_holding_its_cache TestReliabilityPromptCachingAffinitytest_reliability_prompt_caching_e2e.py›
    • routing
    • nonstream
    • anthropic
    • openai
    • anthropic/claude-haiku-4-5
    • openai/gpt-5.5
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Read the router settings from /router/settings
    3. Add a deployment named reliability-cache-133a54b14f30 that calls anthropic/claude-haiku-4-5 with prompt caching
    4. Send a /chat/completions request to reliability-cache-133a54b14f30 with a full message history and stream set to False
    5. Add a deployment named reliability-cache-133a54b14f30 that calls openai/gpt-5.5
    6. Send a /chat/completions request to reliability-cache-133a54b14f30 with a full message history and stream set to False
    View test on GitHub ↗
  793. ✓test_client_hanging_up_never_benches_the_deployment TestReliabilityCancelOnDisconnecttest_reliability_cancel_on_disconnect_e2e.py›
    • routing
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Generate a virtual key with user id: e2e-test-user
    2. Check whether the general setting cancel_on_disconnect is on
    3. Add a deployment named reliability-cooldown-disconnect-f200939212dc that calls azure/gpt-5.4-nano and is benched for 300.0s on its first failure
    4. Add a zero-weight backup deployment named reliability-cooldown-disconnect-f200939212dc that calls openai/gpt-5.5
    5. Send a /chat/completions request to reliability-cooldown-disconnect-f200939212dc with stream set to False
    6. Abandon the request to /chat/completions after 5.0s
    7. Send a /chat/completions request to reliability-cooldown-disconnect-f200939212dc with stream set to False
    View test on GitHub ↗
  794. ✓test_content_policy_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    • routing
    • nonstream
    • azure
    • openai
    • azure/gpt-5.4-nano
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-policyfail-f7501d654aab that calls azure/gpt-5.4-nano behind Azure's content filter
    3. Send a /chat/completions request to reliability-policyfail-f7501d654aab with stream set to False
    View test on GitHub ↗
  795. ✓test_context_window_refusal_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-3.5-turbo
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-retry-context-f64969ce1eeb that calls openai/gpt-3.5-turbo and takes all of its group's traffic
    3. Add a zero-weight backup deployment named reliability-retry-context-f64969ce1eeb that calls openai/gpt-5.5
    4. Send a /chat/completions request to reliability-retry-context-f64969ce1eeb with stream set to False
    View test on GitHub ↗
  796. ✓test_context_window_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    • routing
    • nonstream
    • openai
    • gpt-5.5
    • openai/gpt-3.5-turbo
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-ctxfail-09e618917a4b that calls the small-context model openai/gpt-3.5-turbo
    3. Send a /chat/completions request to reliability-ctxfail-09e618917a4b with stream set to False
    View test on GitHub ↗
  797. ✓test_cost_based_picks_cheapest_deployment TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cost-de1ca084d391 that calls openai/gpt-5.5
    3. Send a /chat/completions request to reliability-cost-de1ca084d391 with stream set to False
    View test on GitHub ↗
  798. ✓test_exact_cache_returns_cached TestReliabilityCachetest_reliability_cache_e2e.py›
    • caching
    • nonstream
    • openai
    • openai/gpt-5.6
    1. Generate a virtual key with user id: e2e-test-user
    2. Run an observed provider edge server
    3. Add a deployment named e2e-cache-f8b08075f045 that calls openai/gpt-5.6
    4. Send a request to /chat/completions
    View test on GitHub ↗
  799. ✓test_failing_requests_do_not_grow_rss_or_stored_request TestReliabilityMemorytest_reliability_memory_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-memory-b3104a8fb7b2 that calls openai/gpt-5.5 at an unreachable address and is never benched
    3. Add a deployment named reliability-memory-fb-b3104a8fb7b2 that calls openai/gpt-5.5 at an unreachable address and is never benched
    4. Send a /chat/completions request to reliability-memory-b3104a8fb7b2 with stream set to False
    5. Wait for the request's spend log in /spend/logs
    6. Send a /chat/completions request to reliability-memory-b3104a8fb7b2 with stream set to False
    7. Read memory usage from /debug/memory/summary on every proxy replica
    8. Send a /chat/completions request to reliability-memory-b3104a8fb7b2 with stream set to False
    9. Read memory usage from /debug/memory/summary on every proxy replica
    View test on GitHub ↗
  800. ✓test_header_tagged_messages_routes_through_the_marker_to_an_untagged_tier TestUntaggedTierDeploymentstest_auto_router_regressions_e2e.py›
    • routing
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-autoroute-94f1d02859e2 that calls auto_router/complexity_router
    2. Add a deployment named e2e-tier-94f1d02859e2 that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-autoroute-94f1d02859e2 that calls anthropic/claude-sonnet-5
    4. Generate a virtual key with models: e2e-autoroute-94f1d02859e2, e2e-tier-94f1d02859e2, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    5. POST /v1/messages
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  801. ✓test_header_tagged_responses_with_string_input_routes_to_the_tier TestResponsesApiTagRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-autoroute-f731359a7305 that calls anthropic/claude-sonnet-5
    2. Add a deployment named e2e-tier-f731359a7305 that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-autoroute-f731359a7305 that calls auto_router/complexity_router
    4. Generate a virtual key with models: e2e-autoroute-f731359a7305, e2e-tier-f731359a7305, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    5. POST /v1/responses
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  802. ✓test_request_timeout_exceeds_deadline TestReliabilityTimeoutstest_reliability_timeouts_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-timeout-c21da3fb608d that calls openai/gpt-5.5 with a 1ms timeout
    3. Send a /chat/completions request to reliability-timeout-c21da3fb608d with stream set to False
    View test on GitHub ↗
  803. ✓test_responses_input_reaches_the_semantic_auto_router TestSemanticAutoRouterResponsestest_auto_router_regressions_e2e.py›
    • routing
    • responses
    • nonstream
    • anthropic
    • openai
    • anthropic/claude-haiku-4-5
    • openai/text-embedding-3-small
    1. Add a deployment named e2e-semantic-embedding-f3cd3955b4f2 that calls openai/text-embedding-3-small
    2. Add a deployment named e2e-semantic-target-f3cd3955b4f2 that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-semantic-fallback-f3cd3955b4f2 that calls anthropic/claude-sonnet-5
    4. Add a deployment named e2e-semantic-router-f3cd3955b4f2 that calls auto_router/e2e-semantic-router-f3cd3955b4f2
    5. Generate a virtual key with models: e2e-semantic-router-f3cd3955b4f2, e2e-semantic-target-f3cd3955b4f2, e2e-semantic-fallback-f3cd3955b4f2 and user id: e2e-auto-router-regressions
    6. POST /v1/responses
    7. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  804. ✓test_sibling_replica_serves_backup_within_redis_read_interval TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cooldown-sibling-upstream-8dafe770a99a that calls openai/gpt-5.5 at an unreachable address
    3. Add a deployment named reliability-cooldown-sibling-5a6a6109683a that fronts reliability-cooldown-sibling-upstream-8dafe770a99a on this proxy, so it always gets a 500 and is benched on the first one
    4. Add a zero-weight backup deployment named reliability-cooldown-sibling-5a6a6109683a that calls openai/gpt-5.5
    5. Add a deployment named reliability-cooldown-sibling-warm-1638474c64b3 for openai/gpt-5.5 that answers with a canned reply
    6. Send a /chat/completions request to reliability-cooldown-sibling-warm-1638474c64b3 with stream set to False
    7. Send a /chat/completions request to reliability-cooldown-sibling-5a6a6109683a with stream set to False
    View test on GitHub ↗
  805. ✓test_simple_shuffle_honors_weights TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-shuffle-e8c4919d549b that calls openai/gpt-5.5
    3. Send a /chat/completions request to reliability-shuffle-e8c4919d549b with stream set to False
    View test on GitHub ↗
  806. ✓test_stream_timeout_exceeds_deadline TestReliabilityTimeoutstest_reliability_timeouts_e2e.py›
    • routing
    • stream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-stream-timeout-506b2b76a5cf that calls openai/gpt-5.5 with a 1ms timeout
    3. Send a /chat/completions request to reliability-stream-timeout-506b2b76a5cf with stream set to True
    View test on GitHub ↗
  807. ✓test_tagged_call_straight_at_an_untagged_deployment_stays_denied TestUntaggedTierDeploymentstest_auto_router_regressions_e2e.py›
    • routing
    • chat_completions
    1. Generate a virtual key with models: e2e-tier-94f1d02859e2, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /chat/completions request to e2e-tier-94f1d02859e2
    View test on GitHub ↗
  808. ✓test_timeout_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-retry-56ff5b63d112 that calls openai/gpt-5.5 with a 1ms timeout and is benched on its first timeout
    3. Add a zero-weight backup deployment named reliability-retry-56ff5b63d112 that calls openai/gpt-5.5
    4. Send a /chat/completions request to reliability-retry-56ff5b63d112 with stream set to False
    View test on GitHub ↗
  809. ✓test_timeout_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    • routing
    • nonstream
    • openai
    • gpt-5.5
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-tofail-af4717db3fc4 that calls openai/gpt-5.5 with a 1ms timeout
    3. Send a /chat/completions request to reliability-tofail-af4717db3fc4 with stream set to False
    View test on GitHub ↗
  810. ✓test_timeout_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cooldown-timeout-c4296d2451ea that calls openai/gpt-5.5 with a 1ms timeout and is benched on its first timeout
    3. Add a zero-weight backup deployment named reliability-cooldown-timeout-c4296d2451ea that calls openai/gpt-5.5
    4. Send a /chat/completions request to reliability-cooldown-timeout-c4296d2451ea with stream set to False
    View test on GitHub ↗
  811. ✓test_trivial_ask_behind_keyword_heavy_system_prompt_stays_on_the_cheap_tier TestComplexityHeuristicScopetest_auto_router_regressions_e2e.py›
    • routing
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-heuristic-cheap-75da4d010d8f that calls anthropic/claude-haiku-4-5
    2. Add a deployment named e2e-heuristic-strong-75da4d010d8f that calls openai/gpt-5.6
    3. Add a deployment named e2e-heuristic-router-75da4d010d8f that calls auto_router/complexity_router
    4. Generate a virtual key with models: e2e-heuristic-router-75da4d010d8f, e2e-heuristic-cheap-75da4d010d8f, e2e-heuristic-strong-75da4d010d8f and user id: e2e-auto-router-regressions
    5. Send a /chat/completions request to e2e-heuristic-router-75da4d010d8f
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  812. ✓test_untagged_chat_is_always_served_by_the_plain_deployment TestTagSplitRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5
    1. Generate a virtual key with models: e2e-autoroute-27a5e2bdf08e, e2e-tier-27a5e2bdf08e, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /chat/completions request to e2e-autoroute-27a5e2bdf08e
    3. Wait for at least 5 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  813. ✓test_untagged_messages_is_served_by_the_plain_deployment TestTagSplitRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5
    1. Generate a virtual key with models: e2e-autoroute-27a5e2bdf08e, e2e-tier-27a5e2bdf08e, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /v1/messages request to e2e-autoroute-27a5e2bdf08e
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  814. ✓test_untagged_responses_is_served_by_the_plain_deployment TestResponsesApiTagRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5
    1. Generate a virtual key with models: e2e-autoroute-f731359a7305, e2e-tier-f731359a7305, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. POST /v1/responses
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  815. ✓test_usage_based_picks_deployment_with_tpm_headroom TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-usage-f02114dc3f97 that calls openai/gpt-5.5
    3. Send a /chat/completions request to reliability-usage-f02114dc3f97 with stream set to False
    View test on GitHub ↗
  816. ✓test_workers_idle_under_rss_budget_before_traffic TestReliabilityMemorytest_reliability_memory_e2e.py›
    • deploy-ops

    No steps recorded

    View test on GitHub ↗
  817. ✓test_zero_priced_alias_still_logs_spend_at_the_tier_rate TestStrategyAliasPricingtest_auto_router_regressions_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-priced-tier-8d03bf3af78e that calls anthropic/claude-haiku-4-5
    2. Add a deployment named e2e-priced-alias-8d03bf3af78e that calls auto_router/complexity_router
    3. Generate a virtual key with models: e2e-priced-alias-8d03bf3af78e, e2e-priced-tier-8d03bf3af78e and user id: e2e-auto-router-regressions
    4. Send a /chat/completions request to e2e-priced-alias-8d03bf3af78e
    5. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  818. −test_latency_based_routes_around_deployment_that_times_out TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5

    No steps recorded

    LIT-7682: since #40229 the per-request routing_strategy override runs without the latency sampler, so latency-based has no signal to route on

    View test on GitHub ↗
  819. −test_least_busy_avoids_deployment_with_request_in_flight TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    • routing
    • openai
    • openai/gpt-5.5

    No steps recorded

    LIT-7682: since #40229 the per-request routing_strategy override runs without the in-flight counter, so least-busy has no signal to route on

    View test on GitHub ↗
  820. −test_llm_classifier_runs_and_routes_by_semantic_tier TestComplexityRouterLlmClassifiertest_complexity_router_e2e.py›
    • routing
    • nonstream
    • anthropic
    • openai
    • anthropic/claude-haiku-4-5
    • openai/gpt-5.5

    No steps recorded

    product bug LIT-4521: LLM classifier returns SIMPLE for short hard prompts (e.g. Is P equal to NP?); re-enable when classifier tier quality is fixed

    View test on GitHub ↗
  821. ✓test_apply_guardrail_blocks_banned_and_allows_clean TestApplyGuardrailEndpointtest_apply_guardrail_e2e.py›
    • guardrails
    • guardrails
    1. Register the content filter guardrail e2e-apply-38bfe89c1149 that blocks prompts containing e2e-banned-d2bb250c664c
    2. Apply the guardrail e2e-apply-38bfe89c1149 to a piece of text with /guardrails/apply_guardrail
    View test on GitHub ↗
  822. ✓test_bedrock_post_call_blocks_denied_model_output TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the bedrock guardrail e2e-bedrock-post-1a5fb1661901 with mode post_call
    3. Retry the call until a guardrail blocks it
    View test on GitHub ↗
  823. ✓test_bedrock_post_call_blocks_denied_streamed_output_and_passes_clean_streams TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the bedrock guardrail e2e-bedrock-post-stream-791623cdfdfd with mode post_call
    3. Retry the streamed call until a guardrail blocks it
    4. Send a streaming /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  824. ✓test_bedrock_pre_call_blocks_harmful_prompt TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the Bedrock guardrail e2e-bedrock-guard-2eda81212d7c
    3. Retry the call until a guardrail blocks it
    View test on GitHub ↗
  825. ✓test_bedrock_pre_call_blocks_on_messages TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • messages
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the Bedrock guardrail e2e-bedrock-messages-44b273310142
    3. Retry the streamed call until a guardrail blocks it
    View test on GitHub ↗
  826. ✓test_bedrock_pre_call_blocks_on_responses TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • responses
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the Bedrock guardrail e2e-bedrock-responses-fbcd9aa85061
    3. Retry the streamed call until a guardrail blocks it
    View test on GitHub ↗
  827. ✓test_blocks_execution_request_but_allows_explanation TestBlockCodeExecutionGuardrailtest_block_code_execution_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-blockcode-backend-6a18367c7a94 that calls gemini/gemini-2.5-flash
    3. Register the block_code_execution guardrail e2e-block-code-22f9b8da3417 with mode pre_call
    4. Send a /chat/completions request to e2e-blockcode-backend-6a18367c7a94
    View test on GitHub ↗
  828. ✓test_chat_blocks_category TestOpenAIModerationCategoryMatrixtest_openai_moderation_category_matrix_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-mod-cat-chat-d24bca285183 that calls gemini/gemini-2.5-flash
    3. Register the openai_moderation guardrail e2e-mod-cat-chat-8b7c9fb3fe47 with mode pre_call
    4. Send a /chat/completions request to e2e-mod-cat-chat-d24bca285183
    View test on GitHub ↗
  829. ✓test_child_condition_match_applies_child_and_inherited_parent_guardrails TestPolicyInheritedGuardrailtest_policy_inherited_guardrail_e2e.py›
    • guardrails
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the content filter guardrail e2e-parent-guard-8f8b0e91120d that blocks prompts containing 8f8b0e91120d
    3. Register the content filter guardrail e2e-child-guard-8cccb285d287 that blocks prompts containing 8cccb285d287
    4. Create the guardrail policy e2e-parent-policy-8f8b0e91120d that adds the guardrails e2e-parent-guard-8f8b0e91120d
    5. Create the guardrail policy e2e-child-policy-8cccb285d287 that adds the guardrails e2e-child-guard-8cccb285d287
    6. Attach the guardrail policy e2e-child-policy-8cccb285d287 to requests tagged e2e-policy-tag-b7d31d247898
    7. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  830. ✓test_child_condition_miss_still_applies_inherited_parent_guardrail TestPolicyInheritedGuardrailtest_policy_inherited_guardrail_e2e.py›
    • guardrails
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the content filter guardrail e2e-parent-guard-af97d9fea7a2 that blocks prompts containing af97d9fea7a2
    3. Register the content filter guardrail e2e-child-guard-c6ce95a4f30b that blocks prompts containing c6ce95a4f30b
    4. Create the guardrail policy e2e-parent-policy-af97d9fea7a2 that adds the guardrails e2e-parent-guard-af97d9fea7a2
    5. Create the guardrail policy e2e-child-policy-c6ce95a4f30b that adds the guardrails e2e-child-guard-c6ce95a4f30b
    6. Attach the guardrail policy e2e-child-policy-c6ce95a4f30b to requests tagged e2e-policy-tag-b234d405f2be
    7. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  831. ✓test_during_call_blocks_stream_before_first_chunk TestBedrockDuringCallStreamingtest_streaming_guardrail_e2e.py›
    • guardrails
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the bedrock guardrail e2e-bedrock-during-18971e6e6985 with mode during_call
    3. Retry the streamed call until a guardrail blocks it
    View test on GitHub ↗
  832. ✓test_flag_returns_guardrail_information_for_the_guardrail_that_ran TestGuardrailInformationResponsetest_guardrail_information_response_e2e.py›
    • guardrails
    • nonstream
    • openai
    • openai/gpt-4.1-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the litellm_content_filter guardrail e2e-guardrail-information-0304d9aa1c32 with mode pre_call
    3. Add a deployment named e2e-guardrail-info-backend-635df5c523e8 that calls openai/gpt-4.1-mini
    4. Send a /chat/completions request to e2e-guardrail-info-backend-635df5c523e8
    View test on GitHub ↗
  833. ✓test_global_guardrail_blocks_key_without_team_opt_out TestTeamDisableGlobalGuardrailtest_team_disable_global_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the content filter guardrail e2e-content-filter-b9aa5678c5cd that blocks prompts containing b9aa5678c5cd
    3. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  834. ✓test_key_attached_content_filter_blocks_banned_image_edit_prompt TestKeyAttachedGuardrailOnImageEditstest_key_guardrail_image_edit_e2e.py›
    • guardrails
    • images
    • nonstream
    • gemini
    • openai
    • gemini-2.5-flash
    • openai/gpt-image-2.5-flare
    1. Register the content filter guardrail e2e-image-edit-filter-bdf94ddaf69e that blocks prompts containing bdf94ddaf69e
    2. Generate a virtual key with the guardrails e2e-image-edit-filter-bdf94ddaf69e
    3. Add a deployment named e2e-guard-image-edit-03d8c32c390c that calls openai/gpt-image-2.5-flare
    4. Retry the call until a guardrail blocks it
    5. Send a /v1/images/edits request to e2e-guard-image-edit-03d8c32c390c
    View test on GitHub ↗
  835. ✓test_key_attached_content_filter_blocks_banned_video_prompt TestKeyAttachedGuardrailOnVideostest_key_guardrail_video_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • vertex_ai
    • gemini-2.5-flash
    • vertex_ai/veo-3.1-fast-generate-001
    1. Register the content filter guardrail e2e-video-filter-1dd025142da8 that blocks prompts containing 1dd025142da8
    2. Generate a virtual key with the guardrails e2e-video-filter-1dd025142da8
    3. Add a deployment named e2e-guard-video-587051eff417 that calls vertex_ai/veo-3.1-fast-generate-001
    4. Retry the call until a guardrail blocks it
    5. Send a /v1/videos request to e2e-guard-video-587051eff417
    View test on GitHub ↗
  836. ✓test_masking_run_is_recorded_on_the_spend_log TestPresidioSpendLogRecordtest_presidio_masking_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-2575f1a99df9 with mode pre_call
    3. Send a /chat/completions request to gemini-2.5-flash
    4. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  837. ✓test_messages_blocks_category TestOpenAIModerationCategoryMatrixtest_openai_moderation_category_matrix_e2e.py›
    • guardrails
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-mod-cat-msg-5721c3932bdc that calls anthropic/claude-haiku-4-5
    3. Register the openai_moderation guardrail e2e-mod-cat-msg-fe10e057f7c6 with mode pre_call
    4. Send a /v1/messages request to e2e-mod-cat-msg-5721c3932bdc
    View test on GitHub ↗
  838. ✓test_moderation_blocks_flagged_input TestOpenAIModerationGuardrailtest_openai_moderation_guardrail_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-moderation-backend-d65672c31826 that calls gemini/gemini-2.5-flash
    3. Register the openai_moderation guardrail e2e-openai-moderation-b4c659304adb with mode pre_call
    4. Retry the call until a guardrail blocks it
    5. Send a /chat/completions request to e2e-moderation-backend-d65672c31826
    View test on GitHub ↗
  839. ✓test_moderation_blocks_flagged_input_on_messages TestOpenAIModerationGuardrailtest_openai_moderation_guardrail_e2e.py›
    • guardrails
    • messages
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-moderation-msg-backend-6945437d5358 that calls gemini/gemini-2.5-flash
    3. Register the openai_moderation guardrail e2e-openai-moderation-msg-1d2eda48bd45 with mode pre_call
    4. Retry the call until a guardrail blocks it
    5. Send a /v1/messages request to e2e-moderation-msg-backend-6945437d5358
    View test on GitHub ↗
  840. ✓test_non_stream_block_is_schema_valid_json TestResponsesPreCallBlocktest_responses_pre_call_block_stream_e2e.py›
    • guardrails
    • responses
    • nonstream
    • openai
    • openai/gpt-4.1-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the custom_code guardrail e2e-custom-code-responses-block-eaeb58637946 with mode pre_call
    3. Add a deployment named e2e-responses-block-1c30f0366720 that calls openai/gpt-4.1-mini
    4. Send a /v1/responses request to e2e-responses-block-1c30f0366720
    View test on GitHub ↗
  841. ✓test_post_call_masks_pii_in_model_output TestPresidioPostCallMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-post-chat-6059d1dbac64 with mode post_call
    3. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  842. ✓test_pre_call_allows_permitted_tool TestToolPermissionPreCalltest_tool_permission_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the tool_permission guardrail e2e-toolperm-allow-ac7208a69793 with mode pre_call
    3. Retry the call until the guardrail e2e-toolperm-allow-ac7208a69793 is applied
    View test on GitHub ↗
  843. ✓test_pre_call_blocks_tool_outside_the_allow_list TestToolPermissionPreCalltest_tool_permission_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the tool_permission guardrail e2e-toolperm-block-2cf20610fde1 with mode pre_call
    3. Retry the call until a guardrail blocks it
    View test on GitHub ↗
  844. ✓test_pre_call_masks_pii_on_chat_completions TestPresidioPreCallMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-pre-chat-6c4f12c309aa with mode pre_call
    3. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  845. ✓test_pre_call_masks_pii_on_messages TestPresidioPreCallMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • messages
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-pre-msg-847fa55a0933 with mode pre_call
    3. Send a /v1/messages request to gemini-2.5-flash
    View test on GitHub ↗
  846. ✓test_responses_blocks_category TestOpenAIModerationCategoryMatrixtest_openai_moderation_category_matrix_e2e.py›
    • guardrails
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-mod-cat-resp-5542126b8914 that calls openai/gpt-4o-mini
    3. Register the openai_moderation guardrail e2e-mod-cat-resp-d06a6fd932f9 with mode pre_call
    4. Send a /v1/responses request to e2e-mod-cat-resp-5542126b8914
    View test on GitHub ↗
  847. ✓test_spend_log_stores_masked_output_on_anthropic_messages TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    • guardrails
    • messages
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-card-messages-9f3442867208 with mode post_call
    3. Send a /v1/messages request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  848. ✓test_spend_log_stores_masked_output_on_chat_completions TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-card-chat-44e0ee5e30cf with mode post_call
    3. Send a /chat/completions request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  849. ✓test_spend_log_stores_masked_output_on_responses TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    • guardrails
    • responses
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-card-responses-b71092be6dab with mode post_call
    3. Send a /v1/responses request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  850. ✓test_spend_log_stores_masked_output_on_streaming_anthropic_messages TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    • guardrails
    • messages
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-card-messages-stream-3b702f30fcab with mode post_call
    3. Send a streaming /v1/messages request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  851. ✓test_spend_log_stores_masked_output_on_streaming_chat_completions TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    • guardrails
    • chat_completions
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-card-chat-stream-286f1b652bee with mode post_call
    3. Send a streaming /chat/completions request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  852. ✓test_stream_block_is_sse_with_completed_assistant_message TestResponsesPreCallBlocktest_responses_pre_call_block_stream_e2e.py›
    • guardrails
    • responses
    • stream
    • openai
    • openai/gpt-4.1-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the custom_code guardrail e2e-custom-code-responses-block-64fd50679d8c with mode pre_call
    3. Add a deployment named e2e-responses-block-c11c15bfe395 that calls openai/gpt-4.1-mini
    4. Send a streaming /v1/responses request to e2e-responses-block-c11c15bfe395
    View test on GitHub ↗
  853. ✓test_team_with_disable_flag_bypasses_global_guardrail TestTeamDisableGlobalGuardrailtest_team_disable_global_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Register the content filter guardrail e2e-content-filter-988f1612cb27 that blocks prompts containing 988f1612cb27
    2. Create the team e2e-guardrail-optout-988f1612cb27 opted out of global guardrails and wait until /team/info returns it
    3. Generate a virtual key in the team
    4. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  854. ✓test_ui_default_scope_masks_a_card_number_the_model_generates_on_chat_completions TestPresidioCreditCardOutputMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-card-chat-d461c92a845d with mode pre_call, post_call
    3. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  855. ✓test_ui_default_scope_masks_a_card_number_the_model_generates_on_streaming_anthropic_messages TestPresidioCreditCardOutputMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • messages
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-card-messages-stream-d0ceed33f446 with mode pre_call, post_call
    3. Send a streaming /v1/messages request to gemini-2.5-flash
    View test on GitHub ↗
  856. ✓test_ui_default_scope_masks_a_card_number_the_model_generates_on_streaming_chat_completions TestPresidioCreditCardOutputMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • chat_completions
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-card-stream-9ba3b08c9681 with mode pre_call, post_call
    3. Send a streaming /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  857. ✓test_without_flag_response_has_no_guardrail_information TestGuardrailInformationResponsetest_guardrail_information_response_e2e.py›
    • guardrails
    • nonstream
    • openai
    • openai/gpt-4.1-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the litellm_content_filter guardrail e2e-guardrail-information-default-b91bffa79c4e with mode pre_call
    3. Add a deployment named e2e-guardrail-info-backend-da86b2afe356 that calls openai/gpt-4.1-mini
    4. Send a /chat/completions request to e2e-guardrail-info-backend-da86b2afe356
    View test on GitHub ↗
  858. −test_request_naming_an_unknown_guardrail_fails_closedtest_guardrail_dispatch_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash

    No steps recorded

    stage red: product gap, a request naming a guardrail the proxy does not serve is silently served unguarded (200) instead of failing closed

    View test on GitHub ↗
  859. ✓test_chat_completions_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • chat_completions
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-chat-947f66f51f26 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for DataDog to ingest logs carrying the marker 9b3b696d80e6, then watch for duplicates
    View test on GitHub ↗
  860. ✓test_chat_completions_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • chat_completions
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-trace-chat-a1cec9efecb8 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call 23272ae0-d179-4147-a4bc-dc276f883e54
    View test on GitHub ↗
  861. ✓test_chat_completions_failure_writes_one_object TestS3LogDeliverytest_s3_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Add a deployment named s3-err-79b24b01ba6b that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key named s3-err-key-14b69bd12562 with models: s3-err-79b24b01ba6b
    4. Send a /chat/completions request to s3-err-79b24b01ba6b with stream=False and the prompt "trigger an upstream auth failure"
    5. Wait for the request's log object to land in the S3 bucket under s3-err-key-14b69bd12562/, then watch for duplicates
    View test on GitHub ↗
  862. ✓test_chat_completions_object_key_follows_the_partition_granularity TestS3LogDeliverytest_s3_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named s3-layout-17eee9671a3c with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the request's log object to land in the S3 bucket under s3-layout-17eee9671a3c/, then watch for duplicates
    5. List the log objects in the S3 bucket under s3-layout-17eee9671a3c/
    View test on GitHub ↗
  863. ✓test_chat_completions_stream_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • chat_completions
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-stream-chat-73512fd702be with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the key's spend log in /spend/logs
    5. Wait for DataDog to ingest logs carrying the marker f13b9bbd0cc1, then watch for duplicates
    View test on GitHub ↗
  864. ✓test_chat_completions_stream_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • chat_completions
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-stream-chat-02bdda27ce80 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call 57f9ab58-b99a-49e1-a3b8-5dc68e078600
    View test on GitHub ↗
  865. ✓test_chat_completions_stream_records_real_ttft TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • chat_completions
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-ttft-chat-fe2a7183e6f0 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call 63e39e42-eb1d-4cb4-b124-111b4432ce2e
    View test on GitHub ↗
  866. ✓test_chat_completions_writes_one_success_object TestS3LogDeliverytest_s3_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named s3-chat-4e48a30c1eee with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the request's log object to land in the S3 bucket under s3-chat-4e48a30c1eee/, then watch for duplicates
    View test on GitHub ↗
  867. ✓test_chat_completions_writes_one_success_record TestGcsLogDeliverytest_gcs_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named gcs-chat-4db4f4de4504 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the log of the response chatcmpl-53a3c466-04dd-4b02-95ce-c4dbfd3a7f46 to land in the GCS bucket, then watch for duplicates
    View test on GitHub ↗
  868. ✓test_distinct_key_aliases_produce_distinct_series TestPrometheusPerKeyCardinalitytest_prometheus_cardinality_e2e.py›
    • observability
    • metrics
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key named e2e-prom-02cf16955e2c with models: gemini-2.5-flash
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "reply with one word e2e-prom-02cf16955e2c"
    3. Generate a virtual key named e2e-prom-9fa43bdd0cfa with models: gemini-2.5-flash
    4. Send a /chat/completions request to gemini-2.5-flash with the prompt "reply with one word e2e-prom-9fa43bdd0cfa"
    5. Generate a virtual key named e2e-prom-e47b400e416e with models: gemini-2.5-flash
    6. Send a /chat/completions request to gemini-2.5-flash with the prompt "reply with one word e2e-prom-e47b400e416e"
    7. Scrape the Prometheus metrics from /metrics
    View test on GitHub ↗
  869. ✓test_failed_chat_completions_emits_one_error_event TestDataDogFailureDeliverytest_datadog_log_e2e.py›
    • observability
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Add a deployment named dd-err-36749eb42583 that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key named dd-err-key-bc8436444523 with models: dd-err-36749eb42583
    4. Send a /chat/completions request to dd-err-36749eb42583 with stream=False and the prompt "trigger an upstream auth failure"
    5. Wait for DataDog to ingest logs matching @model_group:dd-err-36749eb42583, then watch for duplicates
    View test on GitHub ↗
  870. ✓test_failed_chat_completions_error_span_attributes TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Add a deployment named otel-err-875e6d7f726d that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key named otel-err-c44161db5abb with models: otel-err-875e6d7f726d
    4. Send a /chat/completions request to otel-err-875e6d7f726d with stream=False and the prompt "trigger an upstream auth failure"
    5. Poll Jaeger for the traces of call 9e7c34e8-1df9-4b97-b3be-dca06efead63
    View test on GitHub ↗
  871. ✓test_failed_chat_completions_stream_emits_one_error_event TestDataDogFailureDeliverytest_datadog_log_e2e.py›
    • observability
    • chat_completions
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Add a deployment named dd-err-stream-d9fbce68224d that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key named dd-err-stream-key-5a31f06a6c18 with models: dd-err-stream-d9fbce68224d
    4. Stream a POST to /chat/completions
    5. Wait for DataDog to ingest logs matching @model_group:dd-err-stream-d9fbce68224d, then watch for duplicates
    View test on GitHub ↗
  872. ✓test_failed_messages_error_span_attributes TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Add a deployment named otel-err-f439e50e787a that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key named otel-err-c2ed56f7e7a7 with models: otel-err-f439e50e787a
    4. Send a /v1/messages request to otel-err-f439e50e787a with stream=False and the prompt "trigger an upstream auth failure"
    5. Poll Jaeger for the traces of call c6aa0b69-365c-42ef-ae4b-3f788a79fe27
    View test on GitHub ↗
  873. ✓test_messages_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-messages-55cc1757e951 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for DataDog to ingest logs carrying the marker 0f48b6ee5295, then watch for duplicates
    View test on GitHub ↗
  874. ✓test_messages_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-trace-messages-fcfcb271fa1e with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call fcc5cd9d-b51c-426e-86f0-4a5fc9055c2d
    View test on GitHub ↗
  875. ✓test_messages_stream_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • messages
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-stream-messages-9646a4412629 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the key's spend log in /spend/logs
    5. Wait for DataDog to ingest logs carrying the marker f9e435682919, then watch for duplicates
    View test on GitHub ↗
  876. ✓test_messages_stream_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • messages
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-stream-messages-19bbe6149c35 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call 09792ac0-a015-4f7c-a7a7-c387b6194f93
    View test on GitHub ↗
  877. ✓test_messages_stream_records_real_ttft TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • messages
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-ttft-messages-d3299e783d7a with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call bfa2ad3a-e46c-4acf-8f4f-34e6271cb94f
    View test on GitHub ↗
  878. ✓test_non_json_native_metadata_reaches_langsmith TestLangsmithBatchSerializationtest_langsmith_batch_serialization_e2e.py›
    • observability
    • nonstream
    • openai
    • gpt-5.5
    1. GET the external URL https://api.smith.langchain.com/runs/fda44f86-99f1-4949-b776-5c2bc6b14d1c
    View test on GitHub ↗
  879. ✓test_queue_time_histogram_records_an_observation TestPrometheusRequestQueueTimetest_prometheus_queue_time_e2e.py›
    • observability
    • metrics
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key named e2e-queue-time-4223651d1b83 with models: gemini-2.5-flash
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "reply with one word e2e-queue-time-4223651d1b83"
    3. Scrape the Prometheus metrics from /metrics
    View test on GitHub ↗
  880. ✓test_responses_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • responses
    • nonstream
    • openai
    • gpt-5.5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-responses-b2ed1375ba57 with models: gpt-5.5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for DataDog to ingest logs carrying the marker 63ea98b04908, then watch for duplicates
    View test on GitHub ↗
  881. ✓test_responses_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • responses
    • nonstream
    • openai
    • gpt-5.5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-trace-responses-1ebfe0c0a486 with models: gpt-5.5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call f472acf2-ba43-4d7f-a2c9-f5c7b3443989
    View test on GitHub ↗
  882. ✓test_responses_stream_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • responses
    • stream
    • openai
    • gpt-5.5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-stream-responses-527f7f5d9943 with models: gpt-5.5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the key's spend log in /spend/logs
    5. Wait for DataDog to ingest logs carrying the marker ea6d99673a91, then watch for duplicates
    View test on GitHub ↗
  883. ✓test_responses_stream_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • responses
    • stream
    • openai
    • gpt-5.5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-stream-responses-5cbe6d88c553 with models: gpt-5.5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call 5d6c7cb7-c6a1-4035-8330-7c23c0fbb0bd
    5. Wait for the key's spend log in /spend/logs
    View test on GitHub ↗
  884. ✓test_responses_stream_records_real_ttft TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • responses
    • stream
    • openai
    • gpt-5.5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-ttft-responses-4961c2bf5e2b with models: gpt-5.5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call bbd7e227-b438-478d-8e71-5e69968be1a8
    View test on GitHub ↗
  885. ✓test_team_callback_delivers_and_isolates TestTeamLangfuseCallbacktest_team_langfuse_callback_e2e.py›
    • observability
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team lf-team-5862050b2e60 with models: claude-haiku-4-5
    2. Add a Langfuse OTel logging callback for success_and_failure events to the team
    3. Generate a virtual key named lf-team-key-90e2843d28eb with models: claude-haiku-4-5
    4. Generate a virtual key named lf-solo-key-ad91f186f16b with models: claude-haiku-4-5
    5. Retry the call until the fresh key stops answering 401
    6. Wait for the Langfuse generation for the key lf-team-key-90e2843d28eb
    7. Retry the call until the fresh key stops answering 401
    8. Look up the Langfuse generation for the key lf-solo-key-ad91f186f16b
    View test on GitHub ↗
  886. −test_chat_completions_delivers_one_call_with_spend TestWeaveLogDeliverytest_weave_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • claude-haiku-4-5

    No steps recorded

    stage red: product gap, key-scoped weave_otel spans are not delivered when the OTEL v2 callback is active

    View test on GitHub ↗
  887. −test_failed_chat_completions_delivers_one_error_call TestWeaveLogDeliverytest_weave_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5

    No steps recorded

    stage red: product gap, key-scoped weave_otel spans are not delivered when the OTEL v2 callback is active

    View test on GitHub ↗
  888. ✓test_encrypted_stored_value_is_not_a_bearer_token TestSessionTokentest_session_token_e2e.py›
    • proxy-auth
    1. Generate a virtual key with key alias: e2e-session-4b7379209a56 and metadata: logging: callback name: langfuse and callback vars: default settings
    2. Read the key's settings back from /key/info
    3. List users from /user/list with the given key
    View test on GitHub ↗
  889. ✓test_expired_session_token_is_denied TestSessionTokentest_session_token_e2e.py›
    • proxy-auth
    1. List users from /user/list with the given key
    View test on GitHub ↗
  890. ✓test_expired_token_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-51a3e134ebc1 and the user e2e-jwt-user-51a3e134ebc1 in it
    2. Create a team with team alias: e2e-jwt-51a3e134ebc1 and team id: e2e-jwt-team-51a3e134ebc1
    3. Get a Keycloak access token for e2e-jwt-user-51a3e134ebc1 from the client litellm-e2e-shortlived
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  891. ✓test_liveness_reports_alive_without_auth TestHealthLifecycletest_health_lifecycle_e2e.py›
    • deploy-ops
    • health
    1. Call /health/liveliness without credentials
    View test on GitHub ↗
  892. ✓test_master_key_authenticates_and_grants_admin_route TestMasterKeyAuthtest_master_key_auth_e2e.py›
    • proxy-auth
    1. List users from /user/list with the given key
    View test on GitHub ↗
  893. ✓test_non_matching_master_key_is_denied TestMasterKeyAuthtest_master_key_auth_e2e.py›
    • proxy-auth
    1. List users from /user/list with the given key
    View test on GitHub ↗
  894. ✓test_plain_virtual_key_still_works_with_jwt_auth_enabled TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  895. ✓test_readiness_details_require_auth_and_expose_diagnostics TestHealthLifecycletest_health_lifecycle_e2e.py›
    • deploy-ops
    • health
    1. Call /health/readiness/details without credentials
    2. Call /health/readiness/details with the given key
    View test on GitHub ↗
  896. ✓test_readiness_is_reachable_without_credentials TestHealthLifecycletest_health_lifecycle_e2e.py›
    • deploy-ops
    • health
    1. Call /health/readiness without credentials
    View test on GitHub ↗
  897. ✓test_readiness_reports_connected_db TestHealthLifecycletest_health_lifecycle_e2e.py›
    • deploy-ops
    • health
    1. Call /health/readiness without credentials
    View test on GitHub ↗
  898. ✓test_signed_token_for_another_application_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-113126ed5886 and the user e2e-jwt-user-113126ed5886 in it
    2. Create a team with team alias: e2e-jwt-113126ed5886 and team id: e2e-jwt-team-113126ed5886
    3. Get a Keycloak access token for e2e-jwt-user-113126ed5886 from the client litellm-e2e-other-app
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  899. ✓test_signed_token_from_the_wrong_issuer_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-53221901d900 and the user e2e-jwt-user-53221901d900 in it
    2. Create a team with team alias: e2e-jwt-53221901d900 and team id: e2e-jwt-team-53221901d900
    3. Get a Keycloak access token for e2e-jwt-user-53221901d900 from the client litellm-e2e-tests
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  900. ✓test_tampered_signature_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-fd4a99d26d31 and the user e2e-jwt-user-fd4a99d26d31 in it
    2. Create a team with team alias: e2e-jwt-fd4a99d26d31 and team id: e2e-jwt-team-fd4a99d26d31
    3. Get a Keycloak access token for e2e-jwt-user-fd4a99d26d31 from the client litellm-e2e-tests
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  901. ✓test_team_header_with_the_alias_of_a_team_the_caller_is_not_in_is_rejected_like_an_unknown_value TestJwtTeamHeadertest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-1a5ecb235ad0 and the user e2e-jwt-user-1a5ecb235ad0 in it
    2. Create a team with team alias: e2e-jwt-alias-1a5ecb235ad0 and team id: e2e-jwt-team-1a5ecb235ad0
    3. Get a Keycloak access token for e2e-jwt-user-1a5ecb235ad0 from the client litellm-e2e-tests
    4. Create a team with team alias: e2e-jwt-alias-6c2470e83064 and team id: e2e-jwt-other-6c2470e83064
    5. Send a /chat/completions request to gpt-5.5 as team e2e-jwt-alias-6c2470e83064 with the given token
    6. Send a /chat/completions request to gpt-5.5 as team e2e-jwt-unknown-8c1f878edf56 with the given token
    View test on GitHub ↗
  902. ✓test_team_header_with_the_team_alias_binds_the_same_team_as_the_team_id TestJwtTeamHeadertest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-b4f864cb814f and the user e2e-jwt-user-b4f864cb814f in it
    2. Create a team with team alias: e2e-jwt-alias-b4f864cb814f and team id: e2e-jwt-team-b4f864cb814f
    3. Get a Keycloak access token for e2e-jwt-user-b4f864cb814f from the client litellm-e2e-tests
    4. Send a /chat/completions request to gpt-5.5 as team e2e-jwt-team-b4f864cb814f with the given token
    5. Wait for the request's spend log in /spend/logs
    6. Send a /chat/completions request to gpt-5.5 as team e2e-jwt-alias-b4f864cb814f with the given token
    7. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  903. ✓test_team_model_alias_is_listed_by_v1_models_under_the_same_token_that_routes_it[anthropic_shape] TestJwtTeamHeadertest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-feebd7affe6d and the user e2e-jwt-user-feebd7affe6d in it
    2. Create a team with team alias: e2e-jwt-aliased-feebd7affe6d, models: gpt-5.5, team id: e2e-jwt-team-feebd7affe6d and model aliases: e2e-jwt-model-alias-feebd7affe6d: gpt-5.5
    3. Get a Keycloak access token for e2e-jwt-user-feebd7affe6d from the client litellm-e2e-tests
    4. Send a /chat/completions request to e2e-jwt-model-alias-feebd7affe6d
    5. List the models from /v1/models with the given token, in the Anthropic shape: True
    View test on GitHub ↗
  904. ✓test_team_model_alias_is_listed_by_v1_models_under_the_same_token_that_routes_it[openai_shape] TestJwtTeamHeadertest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-cde5f511a5bd and the user e2e-jwt-user-cde5f511a5bd in it
    2. Create a team with team alias: e2e-jwt-aliased-cde5f511a5bd, models: gpt-5.5, team id: e2e-jwt-team-cde5f511a5bd and model aliases: e2e-jwt-model-alias-cde5f511a5bd: gpt-5.5
    3. Get a Keycloak access token for e2e-jwt-user-cde5f511a5bd from the client litellm-e2e-tests
    4. Send a /chat/completions request to e2e-jwt-model-alias-cde5f511a5bd
    5. List the models from /v1/models with the given token, in the Anthropic shape: False
    View test on GitHub ↗
  905. ✓test_token_naming_a_team_that_does_not_exist_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-6d29a683ff54 and the user e2e-jwt-user-6d29a683ff54 in it
    2. Get a Keycloak access token for e2e-jwt-user-6d29a683ff54 from the client litellm-e2e-tests
    3. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  906. ✓test_unexpired_session_token_reaches_admin_route TestSessionTokentest_session_token_e2e.py›
    • proxy-auth
    1. List users from /user/list with the given key
    View test on GitHub ↗
  907. ✓test_valid_token_for_an_existing_team_is_accepted_and_attributed TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-afbb9ba9e45a and the user e2e-jwt-user-afbb9ba9e45a in it
    2. Create a team with team alias: e2e-jwt-afbb9ba9e45a and team id: e2e-jwt-team-afbb9ba9e45a
    3. Get a Keycloak access token for e2e-jwt-user-afbb9ba9e45a from the client litellm-e2e-tests
    4. Send a /chat/completions request to gpt-5.5
    5. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  908. ✓test_allowed_model_is_permitted TestAccessControltest_access_control_e2e.py›
    • proxy-auth
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with models: gemini-2.5-flash and user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "capital of France? 99fcb8cb7554"
    View test on GitHub ↗
  909. ✓test_bearer_invalid_token_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›
    • proxy-auth
    • chat_completions
    1. Send a request to /chat/completions
    View test on GitHub ↗
  910. ✓test_disallowed_model_is_denied_403 TestAccessControltest_access_control_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Generate a virtual key with models: gemini-2.5-flash and user id: e2e-test-user
    2. Send a /chat/completions request to gpt-5.5 with the prompt "capital of France? cb31299b34ea"
    View test on GitHub ↗
  911. ✓test_empty_bearer_token_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›
    • proxy-auth
    • chat_completions
    1. Send a request to /chat/completions
    View test on GitHub ↗
  912. ✓test_group_grants_every_deployment_in_it[bare name the group's wildcard covers-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.4-nano
    • openai/gpt-5.4-nano
    1. Add a deployment named openai/gpt-5.4* that calls openai/gpt-5.4*
    2. Add a deployment named e2e-ag-member-1da0bc4463de that calls openai/gpt-5.4-nano
    3. Add a deployment named e2e-ag-outsider-1da0bc4463de that calls openai/gpt-5.4-nano
    4. List the deployments in the model access group e2e-ag-1da0bc4463de
    5. Generate a virtual key with models: e2e-ag-1da0bc4463de and user id: e2e-test-user
    6. Send a /chat/completions request to gpt-5.4-nano with the prompt "Reply with exactly: OK 84e08e356530"
    View test on GitHub ↗
  913. ✓test_group_grants_every_deployment_in_it[exactly-named deployment in the group-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.4-nano
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-ag-1da0bc4463de and user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ag-member-1da0bc4463de with the prompt "Reply with exactly: OK c8419b14a4c6"
    View test on GitHub ↗
  914. ✓test_group_grants_every_deployment_in_it[provider-prefixed name the group's wildcard covers-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.4-nano
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-ag-1da0bc4463de and user id: e2e-test-user
    2. Send a /chat/completions request to openai/gpt-5.4-nano with the prompt "Reply with exactly: OK 117e10b50989"
    View test on GitHub ↗
  915. ✓test_group_grants_nothing_outside_it[deployment outside the group-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.2
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-ag-1da0bc4463de and user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ag-outsider-1da0bc4463de with the prompt "Reply with exactly: OK 688b39e42bb5"
    View test on GitHub ↗
  916. ✓test_group_grants_nothing_outside_it[name no provider claims-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.2
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-ag-1da0bc4463de and user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ag-unknown-d4fdc99f072d with the prompt "Reply with exactly: OK 092aef8efb58"
    View test on GitHub ↗
  917. ✓test_group_grants_nothing_outside_it[provider model outside the group's wildcard-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.2
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-ag-1da0bc4463de and user id: e2e-test-user
    2. Send a /chat/completions request to gpt-5.2 with the prompt "Reply with exactly: OK 347c71a3b77c"
    View test on GitHub ↗
  918. ✓test_group_grants_the_team_nothing_outside_it TestTeamScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    1. Send a /chat/completions request to e2e-ag-unknown-3fe346da10a4 with the prompt "Reply with exactly: OK 2d739c28ee98"
    View test on GitHub ↗
  919. ✓test_group_grants_the_teams_own_wildcard TestTeamScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.6-luna
    1. Create the team e2e-ag-team-bb505a5fc767 with models: e2e-agt-bb505a5fc767
    2. Generate a virtual key with team id: b7b1640e-0647-4dd3-acde-e954e998f8de
    3. Add a deployment named openai/gpt-5.6* that calls openai/gpt-5.6*
    4. Set the team e2e-ag-team-bb505a5fc767's models to e2e-agt-bb505a5fc767 through /team/update
    5. Read /team/info?team_id=b7b1640e-0647-4dd3-acde-e954e998f8de on every proxy replica until it settles
    6. Wait for the last control-plane write to reach every proxy replica
    7. Send a /chat/completions request to gpt-5.6-luna with the prompt "Reply with exactly: OK 214b31c9ba27"
    View test on GitHub ↗
  920. ✓test_llm_api_routes_group_grants_every_llm_endpoint TestAccessControltest_access_control_e2e.py›
    • proxy-auth
    • gemini
    • openai
    • gemini-2.5-flash
    • openai-text-embedding-3-small
    1. Generate a virtual key that can only call LLM API routes
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "capital of France? 27e268730673"
    3. Send an /embeddings request to openai-text-embedding-3-small for "route group 2d46f92182c7"
    4. Add a deployment named e2e-route-group-40aaab142984 that calls openai/gpt-4o-mini with the given key
    View test on GitHub ↗
  921. ✓test_llm_only_key_forbidden_from_management_route_403 TestAccessControltest_access_control_e2e.py›
    • proxy-auth
    • model_management
    1. Generate a virtual key that can only call LLM API routes
    2. Add a deployment named e2e-forbidden-721e02ea4f92 that calls openai/gpt-4o-mini with the given key
    View test on GitHub ↗
  922. ✓test_missing_authorization_header_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›
    • proxy-auth
    • chat_completions
    1. Send a request to /chat/completions
    View test on GitHub ↗
  923. ✓test_not_bearer_scheme_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›
    • proxy-auth
    • chat_completions
    1. Send a request to /chat/completions
    View test on GitHub ↗
  924. ✓test_token_without_bearer_prefix_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›
    • proxy-auth
    • chat_completions
    1. Send a request to /chat/completions
    View test on GitHub ↗
  925. ✓test_unknown_model_returns_400 TestAccessControltest_access_control_e2e.py›
    • proxy-auth
    • nonstream
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to nonexistent-model-576e96f794e9 with the prompt "hi this is a test"
    View test on GitHub ↗
  926. ✓test_valid_key_allows_and_invalid_key_denied TestVirtualKeyAuthtest_access_control_e2e.py›
    • proxy-auth
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-auth-chat-24fe6e07ced8 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-auth-chat-24fe6e07ced8
    View test on GitHub ↗
  927. ✓test_access_group_scopes_tool_selection TestMcpAccessGroupToolSelectiontest_mcp_access_group_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-ag-granted-e5add7f98d2d
    4. Generate a virtual key for the user e2e-mcp-ag-other-763e83f5cca8
    5. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    6. List the MCP tools the key can see from /mcp-rest/tools/list
    View test on GitHub ↗
  928. ✓test_alias_grant_persists_verbatim_and_lists_tools TestMcpKeyGrantByAliastest_mcp_key_access_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-allowed-156f2c25757e
    4. Read the key's settings back from /key/info
    5. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    View test on GitHub ↗
  929. ✓test_call_tool_denied_without_permission TestMcpKeyWithoutAccessIsDeniedtest_mcp_key_access_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-allowed-0a8a68ed1795
    4. Generate a virtual key for the user e2e-mcp-denied-94fcea4493bd
    5. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    6. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call
    7. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call and wait for a 403
    View test on GitHub ↗
  930. ✓test_content_filter_blocks_banned_keyword_in_tool_args TestMcpToolCallGuardrailtest_mcp_guardrail_e2e.py›
    • guardrails
    • mcp
    1. Create the guardrail e2e-mcp-cf-c774fe10786c that blocks MCP tool calls containing "e2eblockedc774fe10786c"
    2. Register the Datadog remote MCP server with its credentials from the environment
    3. Generate a virtual key for the user e2e-mcp-guard-c774fe10786c
    4. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    5. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: tell me about e2eblockedc774fe10786c, from: now-30m, to: now and max tokens: 500
    6. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: still about e2eblockedc774fe10786c #1, from: now-30m, to: now and max tokens: 500
    7. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: still about e2eblockedc774fe10786c #2, from: now-30m, to: now and max tokens: 500
    8. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: still about e2eblockedc774fe10786c #3, from: now-30m, to: now and max tokens: 500
    9. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: still about e2eblockedc774fe10786c #4, from: now-30m, to: now and max tokens: 500
    10. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: e2e-clean-c774fe10786c, from: now-30m, to: now and max tokens: 500
    View test on GitHub ↗
  931. ✓test_key_granted_a_toolset_lists_exactly_its_tools TestMcpToolsetEnforcementtest_mcp_toolset_enforcement_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-catalog-04ddf4440fcf
    4. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    5. List the MCP tools the key can see from /mcp-rest/tools/list
    6. Create an MCP toolset with the tools server id: d7f30f64-89e3-4b47-b896-f945ae0279cf and tool name: _dd_gff, server id: d7f30f64-89e3-4b47-b896-f945ae0279cf and tool name: _dd_guc
    7. Generate a virtual key for the user e2e-mcp-toolset-eea6ebbb4fd4
    8. Wait for /mcp-rest/tools/list to show the key exactly the expected tools on the MCP server
    View test on GitHub ↗
  932. ✓test_list_tools_denied_without_permission TestMcpKeyWithoutAccessIsDeniedtest_mcp_key_access_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-allowed-86058d642195
    4. Generate a virtual key for the user e2e-mcp-denied-755a59bec40e
    5. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    6. List the MCP tools the key can see from /mcp-rest/tools/list
    View test on GitHub ↗
  933. ✓test_route_restricted_health_matches_server_grants TestMcpHealthVisibilitytest_mcp_key_access_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-allowed-4dcd42599577
    4. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    5. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call
    6. Generate a virtual key with user id: e2e-mcp-health-052903196c1b, allowed routes: /v1/mcp/server, /v1/mcp/server/health and object permission: mcp servers: 0869cd58-cb2e-437e-b149-db78945ed747
    7. List the MCP servers the key can see from /v1/mcp/server
    8. Check the health of the MCP servers the key can see from /v1/mcp/server/health
    9. Generate a virtual key with user id: e2e-mcp-health-b6e2dd671e2c, allowed routes: /v1/mcp/server, /v1/mcp/server/health and object permission: mcp servers: 9b30b9b9-b9da-40ab-8c11-a8aa6826dbdd
    10. List the MCP servers the key can see from /v1/mcp/server
    11. Check the health of the MCP servers the key can see from /v1/mcp/server/health
    12. Generate a virtual key with user id: e2e-mcp-health-fc829d928463, allowed routes: /v1/mcp/server, /v1/mcp/server/health and object permission: default settings
    13. List the MCP servers the key can see from /v1/mcp/server
    14. Check the health of the MCP servers the key can see from /v1/mcp/server/health
    View test on GitHub ↗
  934. ✓test_search_logs_finds_seeded_completion TestDatadogMcpRoundTriptest_mcp_datadog_e2e.py›
    • mcp
    • mcp
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Call the management route /health/readiness/details
    2. Register the Datadog remote MCP server with its credentials from the environment
    3. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    4. Generate a virtual key for the user e2e-dd-mcp-d21433a9d75b
    5. Send a /chat/completions request to claude-haiku-4-5
    6. Wait for DataDog to ingest logs carrying the marker e2e-datadog-mcp-fab38aeac2d3, then watch for duplicates
    7. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    8. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call
    View test on GitHub ↗
  935. −test_chat_completion_uses_linear_with_authorization_bearer_header TestMcpChatCompletionOauthtest_mcp_chat_completion_oauth_e2e.py›
    • mcp
    • chat_completions
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • function_calling

    No steps recorded

    set E2E_LINEAR_STORAGE_STATE to a Linear session captured via mcp/linear_session_capture.py

    View test on GitHub ↗
  936. −test_chat_completion_uses_linear_with_x_litellm_api_key_header TestMcpChatCompletionOauthtest_mcp_chat_completion_oauth_e2e.py›
    • mcp
    • chat_completions
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • function_calling

    No steps recorded

    set E2E_LINEAR_STORAGE_STATE to a Linear session captured via mcp/linear_session_capture.py

    View test on GitHub ↗
  937. ✓test_discovery_card_is_proxy_fronted TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the A2A agent e2e-a2a-fe54306a1955 through /v1/agents
    3. Read the A2A agent's card from /a2a/{agent_id}/.well-known/agent-card.json with the given key
    View test on GitHub ↗
  938. ✓test_malformed_protocol_version_rejected TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    1. Register the A2A agent e2e-a2a-bad-8de4603ae63f through /v1/agents
    View test on GitHub ↗
  939. ✓test_message_send_runs_completion_bridge TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the A2A agent e2e-a2a-4f96eb4d4d73 through /v1/agents
    3. Send an A2A message to /a2a/{agent_id} with text: Reply with exactly the word PONG and nothing else
    4. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  940. ✓test_pinned_v0_3_serves_flat_message_shape TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the A2A agent e2e-a2a-cd16591f759e through /v1/agents
    3. Send an A2A message to /a2a/{agent_id} with text: Say hi in one word
    View test on GitHub ↗
  941. ✓test_pinned_v1_0_serves_nested_message_shape TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the A2A agent e2e-a2a-943485f74140 through /v1/agents
    3. Send an A2A message to /a2a/{agent_id} with text: Say hi in one word
    View test on GitHub ↗
  942. ✓test_real_world_agent_replies_to_property_query TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    1. Generate a virtual key with user id: e2e-test-user
    2. Fetch a published A2A agent card from its /.well-known endpoint
    3. Register the A2A agent e2e-a2a-real-109798b260d7 through /v1/agents
    4. Send an A2A message to /a2a/{agent_id} with data: skill: search_properties and params: un locode: GBLON, service type: long_term and limit: 3
    View test on GitHub ↗
  943. ✓test_register_persists TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    1. Register the A2A agent e2e-a2a-f7f1005f3593 through /v1/agents
    2. Read the A2A agent back from /v1/agents/{agent_id}
    View test on GitHub ↗
  944. ✓test_semver_protocol_version_registers_and_serves TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the A2A agent e2e-a2a-0423bb8c632d through /v1/agents
    3. Read the A2A agent's card from /a2a/{agent_id}/.well-known/agent-card.json with the given key
    4. Send an A2A message to /a2a/{agent_id} with text: Say hi in one word
    View test on GitHub ↗
  945. ✓test_unsupported_protocol_version_rejected TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    1. Register the A2A agent e2e-a2a-bad-d8537c97954a through /v1/agents
    View test on GitHub ↗
  946. −llm_translation/realtime/test_realtime_pipecat_audio_e2e.pyllm_translation/realtime/test_realtime_pipecat_audio_e2e.py›

    No steps recorded

    collection skipped