litellm-e2e · build #721

975 passed, 0 failed, 65 skipped

1.107.0rc2rc1.107.0rc1rc

Oct 18, 2026, 5:50 AM UTC ·c4f545c ↗· build #721 ↗· 1.107.0rc1 ↗· run history

All 1040 tests

  1. ✓test_anthropic_chat_returns_thinking_content TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • reasoning
    1. Add a deployment named e2e-anthropic-thinking-f71b84c94dc9 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-thinking-f71b84c94dc9
    View test on GitHub ↗
  2. ✓test_anthropic_chat_returns_tool_call TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • function_calling
    1. Add a deployment named e2e-anthropic-tool-edf95cf8f00a that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-tool-edf95cf8f00a
    View test on GitHub ↗
  3. ✓test_anthropic_chat_streams_real_content TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-anthropic-stream-c6a331fc8e63 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-anthropic-stream-c6a331fc8e63
    View test on GitHub ↗
  4. ✓test_anthropic_chat_streams_tool_call TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • function_calling
    1. Add a deployment named e2e-anthropic-tool-stream-1855ef0541d9 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-anthropic-tool-stream-1855ef0541d9
    View test on GitHub ↗
  5. ✓test_anthropic_chat_structured_output_conforms_to_schema TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • response_schema
    1. Add a deployment named e2e-anthropic-schema-b10e6cdd39de that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-schema-b10e6cdd39de
    View test on GitHub ↗
  6. ✓test_anthropic_chat_vision_describes_image TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • vision
    1. Add a deployment named e2e-anthropic-vision-f752a9c3dc5b that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-vision-f752a9c3dc5b
    View test on GitHub ↗
  7. ✓test_anthropic_passthrough_nonstreaming_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to claude-haiku-4-5 through /anthropic with streaming set to False
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  8. ✓test_anthropic_passthrough_streaming_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to claude-haiku-4-5 through /anthropic with streaming set to True
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  9. ✓test_anthropic_passthrough_tool_call_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to claude-haiku-4-5 through /anthropic with streaming set to False
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  10. ✓test_anthropic_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • prompt_caching
    1. Add a deployment named e2e-anthropic-cache-83c7864abe6b that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /chat/completions
    View test on GitHub ↗
  11. ✓test_anthropic_prompt_caching_reads_cache_with_version_header TestCacheControltest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • prompt_caching
    1. Add a deployment named e2e-anthropic-cache-version-84e36b3146e7 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /chat/completions
    View test on GitHub ↗
  12. ✓test_anthropic_prompt_caching_reports_creation_tokens TestCacheControltest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • prompt_caching
    1. Add a deployment named e2e-anthropic-cache-create-9f88017236fa that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /chat/completions
    View test on GitHub ↗
  13. ✓test_anthropic_tool_search_returns_a_message[anthropic] TestAnthropicToolSearchtest_messages_prompt_caching_and_tool_search_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • bedrock
    • anthropic/claude-sonnet-4-5-20250929
    • us.anthropic.claude-sonnet-4-5-20250929-v1:0
    1. Add a deployment named e2e-tool-search-7f5a687c4781 that calls anthropic/claude-sonnet-4-5-20250929
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  14. ✓test_anthropic_tool_search_returns_a_message[bedrock-converse] TestAnthropicToolSearchtest_messages_prompt_caching_and_tool_search_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • bedrock
    • anthropic/claude-sonnet-4-5-20250929
    • us.anthropic.claude-sonnet-4-5-20250929-v1:0
    1. Add a deployment named e2e-tool-search-89dd13abb5dd that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  15. ✓test_anthropic_tool_search_returns_a_message[bedrock-invoke] TestAnthropicToolSearchtest_messages_prompt_caching_and_tool_search_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • bedrock
    • anthropic/claude-sonnet-4-5-20250929
    • us.anthropic.claude-sonnet-4-5-20250929-v1:0
    1. Add a deployment named e2e-tool-search-d0c64fbe2e48 that calls bedrock/invoke/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  16. ✓test_anthropic_tool_search_streaming_returns_a_message[anthropic] TestAnthropicToolSearchtest_messages_prompt_caching_and_tool_search_e2e.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • bedrock
    • anthropic/claude-sonnet-4-5-20250929
    • us.anthropic.claude-sonnet-4-5-20250929-v1:0
    1. Add a deployment named e2e-tool-search-stream-ef9d8e1ff9db that calls anthropic/claude-sonnet-4-5-20250929
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  17. ✓test_anthropic_tool_search_streaming_returns_a_message[bedrock-converse] TestAnthropicToolSearchtest_messages_prompt_caching_and_tool_search_e2e.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • bedrock
    • anthropic/claude-sonnet-4-5-20250929
    • us.anthropic.claude-sonnet-4-5-20250929-v1:0
    1. Add a deployment named e2e-tool-search-stream-48ceee9542e9 that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  18. ✓test_anthropic_tool_search_streaming_returns_a_message[bedrock-invoke] TestAnthropicToolSearchtest_messages_prompt_caching_and_tool_search_e2e.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • bedrock
    • anthropic/claude-sonnet-4-5-20250929
    • us.anthropic.claude-sonnet-4-5-20250929-v1:0
    1. Add a deployment named e2e-tool-search-stream-a36e53eaeeb9 that calls bedrock/invoke/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  19. ✓test_anthropic_with_extended_thinking TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    • reasoning
    1. Add a deployment named e2e-chat-tool-loop-e32e88a4cb9c that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-e32e88a4cb9c
    View test on GitHub ↗
  20. ✓test_array_input_returns_vectors TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • openai
    • openai/text-embedding-3-small
    1. Add a deployment named e2e-embeddings-array-79ff0112dd60 that calls openai/text-embedding-3-small
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  21. ✓test_audio_speech_returns_audio TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-tts
    1. Add a deployment named e2e-speech-5aabe752ba81 that calls openai/gpt-4o-mini-tts
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  22. ✓test_audio_speech_streams_audio_chunks TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • stream
    • openai
    • openai/gpt-4o-mini-tts
    1. Add a deployment named e2e-speech-ea1da9bd1cb2 that calls openai/gpt-4o-mini-tts
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  23. ✓test_audio_transcriptions_returns_text TestAudioTranscriptionstest_audio_transcriptions_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-transcribe
    1. Add a deployment named e2e-transcribe-60560b848ba7 that calls openai/gpt-4o-mini-transcribe
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  24. ✓test_azure_foundry_chat_returns_content TestAzureFoundryChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    1. Add a deployment named e2e-azure-foundry-chat-e910efb90e52 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-azure-foundry-chat-e910efb90e52
    View test on GitHub ↗
  25. ✓test_azure_openai_chat_returns_content TestAzureOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-azure-openai-chat-cf2ae22fe043 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-azure-openai-chat-cf2ae22fe043
    View test on GitHub ↗
  26. ✓test_azure_openai_chat_returns_tool_call TestAzureOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    • function_calling
    1. Add a deployment named e2e-azure-openai-tool-1b976e9f6b51 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-azure-openai-tool-1b976e9f6b51
    View test on GitHub ↗
  27. ✓test_basic_nonstream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    1. Add a deployment named e2e-azure-foundry-messages-4998720f5928 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-4998720f5928 and user id: e2e-test-user
    View test on GitHub ↗
  28. ✓test_basic_stream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    • llm-translation
    • messages
    • stream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    1. Add a deployment named e2e-azure-foundry-messages-322bad8ea0e1 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-322bad8ea0e1 and user id: e2e-test-user
    View test on GitHub ↗
  29. ✓test_batch_s3_keys_do_not_break_chat TestBedrockBatchDeploymentServesChattest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-batch-chat-8e7e8a5f49eb that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  30. ✓test_bedrock_chat_completion_pydantic_response TestBedrockNativetest_bedrock_native_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • response_schema
    1. Add a deployment named e2e-bedrock-native-4314d3aff442 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  31. ✓test_bedrock_converse TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/converse/us.anthropic.claude-sonnet-5-5
    • function_calling
    1. Add a deployment named e2e-chat-tool-loop-463aef668d05 that calls bedrock/converse/us.anthropic.claude-sonnet-5-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-463aef668d05
    View test on GitHub ↗
  32. ✓test_bedrock_converse_chat_returns_content TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-chat-2298489c96bc that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-chat-2298489c96bc
    View test on GitHub ↗
  33. ✓test_bedrock_converse_chat_returns_reasoning TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • reasoning
    1. Add a deployment named e2e-bedrock-thinking-cee7d467ad9e that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-thinking-cee7d467ad9e
    View test on GitHub ↗
  34. ✓test_bedrock_converse_chat_returns_tool_call TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • function_calling
    1. Add a deployment named e2e-bedrock-tool-ec4d6d50106b that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-tool-ec4d6d50106b
    View test on GitHub ↗
  35. ✓test_bedrock_converse_chat_streams_real_content TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-stream-83a306e9c27c that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bedrock-stream-83a306e9c27c
    View test on GitHub ↗
  36. ✓test_bedrock_converse_chat_vision_describes_image TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • vision
    1. Add a deployment named e2e-bedrock-vision-329c2c13b614 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-vision-329c2c13b614
    View test on GitHub ↗
  37. ✓test_bedrock_converse_reads_a_pdf_sent_by_url TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.amazon.nova-2-lite-v1:0
    • pdf_input
    1. Add a deployment named e2e-bedrock-document-a05add73b6ae that calls bedrock/us.amazon.nova-2-lite-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-document-a05add73b6ae
    View test on GitHub ↗
  38. ✓test_bedrock_converse_with_extended_thinking TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/converse/us.anthropic.claude-sonnet-4-6
    • function_calling
    • reasoning
    1. Add a deployment named e2e-chat-tool-loop-45038722b71d that calls bedrock/converse/us.anthropic.claude-sonnet-4-6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-45038722b71d
    View test on GitHub ↗
  39. ✓test_bedrock_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • bedrock
    • bedrock/amazon.titan-embed-text-v2:0
    1. Add a deployment named e2e-embeddings-bedrock-2daaf36714dd that calls bedrock/amazon.titan-embed-text-v2:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  40. ✓test_bedrock_image_generation_returns_image TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images
    • nonstream
    • bedrock
    • bedrock/stability.stable-image-core-v1:1
    1. Add a deployment named e2e-bedrock-image-a4dd97fe2c59 that calls bedrock/stability.stable-image-core-v1:1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  41. ✓test_bedrock_invoke_messages_accepts_forwarded_headers TestBedrockInvokeForwardedHeaderstest_messages_prompt_caching_and_tool_search_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • us.anthropic.claude-sonnet-4-5-20250929-v1:0
    1. Add a deployment named e2e-bedrock-forwarded-headers-c3e752b62fe9 that calls bedrock/invoke/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  42. ✓test_bedrock_messages_prompt_caching_creates_cache[bedrock-converse] TestBedrockMessagesPromptCachingtest_messages_prompt_caching_and_tool_search_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • us.anthropic.claude-sonnet-4-5-20250929-v1:0
    1. Add a deployment named e2e-prompt-cache-create-b30c5e185494 that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  43. ✓test_bedrock_messages_prompt_caching_creates_cache[bedrock-invoke] TestBedrockMessagesPromptCachingtest_messages_prompt_caching_and_tool_search_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • us.anthropic.claude-sonnet-4-5-20250929-v1:0
    1. Add a deployment named e2e-prompt-cache-create-3cb67cc61fb2 that calls bedrock/invoke/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  44. ✓test_bedrock_messages_prompt_caching_reads_cache_on_second_call[bedrock-converse] TestBedrockMessagesPromptCachingtest_messages_prompt_caching_and_tool_search_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • us.anthropic.claude-sonnet-4-5-20250929-v1:0
    1. Add a deployment named e2e-prompt-cache-read-446c322b87cb that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  45. ✓test_bedrock_messages_prompt_caching_reads_cache_on_second_call[bedrock-invoke] TestBedrockMessagesPromptCachingtest_messages_prompt_caching_and_tool_search_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • us.anthropic.claude-sonnet-4-5-20250929-v1:0
    1. Add a deployment named e2e-prompt-cache-read-9cdf760fc301 that calls bedrock/invoke/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  46. ✓test_bedrock_messages_streaming_prompt_caching_reads_cache[bedrock-converse] TestBedrockMessagesPromptCachingtest_messages_prompt_caching_and_tool_search_e2e.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • us.anthropic.claude-sonnet-4-5-20250929-v1:0
    1. Add a deployment named e2e-prompt-cache-stream-4afe77df2967 that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  47. ✓test_bedrock_messages_streaming_prompt_caching_reads_cache[bedrock-invoke] TestBedrockMessagesPromptCachingtest_messages_prompt_caching_and_tool_search_e2e.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • us.anthropic.claude-sonnet-4-5-20250929-v1:0
    1. Add a deployment named e2e-prompt-cache-stream-77790dad5316 that calls bedrock/invoke/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  48. ✓test_bedrock_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • prompt_caching
    1. Add a deployment named e2e-bedrock-cache-b033f2a57abd that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /chat/completions
    View test on GitHub ↗
  49. ✓test_bedrock_request_id_header_surfaces TestBedrockResponseHeaderstest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-headers-518983f92b09 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  50. ✓test_bedrock_request_id_header_surfaces_on_stream TestBedrockResponseHeaderstest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-headers-stream-c86cf4c2b50b that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bedrock-headers-stream-c86cf4c2b50b
    View test on GitHub ↗
  51. ✓test_bedrock_rerank_scores_top_n TestReranktest_rerank_e2e.py›
    • llm-translation
    • rerank
    • nonstream
    • bedrock
    • bedrock/arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0
    1. Add a deployment named e2e-bedrock-rerank-146619ba8ecc that calls bedrock/arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/rerank request to e2e-bedrock-rerank-146619ba8ecc for "What is the capital of the United States?"
    View test on GitHub ↗
  52. ✓test_bedrock_titan_embeds_token_array_input_as_its_decoded_text TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • bedrock
    • bedrock/amazon.titan-embed-text-v2:0
    1. Add a deployment named e2e-embeddings-titan-tokens-b184b4b7914d that calls bedrock/amazon.titan-embed-text-v2:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  53. ✓test_bridged_stream_delivers_content_finish_reason_and_done TestResponsesBridgeChatCompletionsStreamingtest_responses_bridge_streaming_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.3-codex
    1. Add a deployment named e2e-bridge-stream-2f6ddcb5f2da that calls openai/gpt-5.3-codex
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bridge-stream-2f6ddcb5f2da
    View test on GitHub ↗
  54. ✓test_bridged_stream_reassembles_tool_call TestResponsesBridgeChatCompletionsStreamingtest_responses_bridge_streaming_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.3-codex
    • function_calling
    1. Add a deployment named e2e-bridge-stream-c6cdc70f0bdf that calls openai/gpt-5.3-codex
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bridge-stream-c6cdc70f0bdf
    View test on GitHub ↗
  55. ✓test_bridged_stream_shares_one_chunk_id TestResponsesBridgeChatCompletionsStreamingtest_responses_bridge_streaming_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.3-codex
    1. Add a deployment named e2e-bridge-stream-86ce281093cf that calls openai/gpt-5.3-codex
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bridge-stream-86ce281093cf
    View test on GitHub ↗
  56. ✓test_cache_control_on_a_string_system_message TestCacheControlOnOpenAIExplicitCachingtest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-openai-explicit-cache-a46c7a409387 that calls openai/gpt-5.6
    3. Send a /chat/completions request to e2e-openai-explicit-cache-a46c7a409387
    View test on GitHub ↗
  57. ✓test_cache_control_on_a_system_text_part TestCacheControlOnOpenAIExplicitCachingtest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-openai-explicit-cache-cea0271421e1 that calls openai/gpt-5.6
    3. Send a /chat/completions request to e2e-openai-explicit-cache-cea0271421e1
    View test on GitHub ↗
  58. ✓test_cache_control_on_a_tool_result TestCacheControlOnOpenAIExplicitCachingtest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-openai-explicit-cache-6eec51556c1a that calls openai/gpt-5.6
    3. Send a /chat/completions request to e2e-openai-explicit-cache-6eec51556c1a
    View test on GitHub ↗
  59. ✓test_cache_control_on_a_user_text_part TestCacheControlOnOpenAIExplicitCachingtest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-openai-explicit-cache-a5774db11b73 that calls openai/gpt-5.6
    3. Send a /chat/completions request to e2e-openai-explicit-cache-a5774db11b73
    View test on GitHub ↗
  60. ✓test_cache_read_tokens_bill_at_the_cache_read_rate TestTogetherChatCompletionstest_together_ai_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • together_ai
    • prompt_caching
    1. Add a deployment named e2e-together-99a6ed3ba9b1 that calls together_ai/deepseek-ai/DeepSeek-V4-Flash-0731
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-99a6ed3ba9b1
    4. Send a request to /chat/completions
    5. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  61. ✓test_caller_completion_window_bills_its_rates TestSailResponsestest_sail_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-b4b7d008d69e that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  62. ✓test_calls_without_auth_is_denied TestRealtimeHttptest_realtime_http_e2e.py›
    • proxy-auth
    • realtime
    1. Send a request to /v1/realtime/calls
    View test on GitHub ↗
  63. ✓test_cancel_background_response[azure] TestBackgroundResponseCanceltest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-resp-cancel-7632b738a7e7 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  64. ✓test_cancel_background_response[openai] TestBackgroundResponseCanceltest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-resp-cancel-2cbf31bc7871 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  65. ✓test_cancel_background_streaming_response_by_streamed_id TestBackgroundResponseCanceltest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-resp-cancel-stream-85403c90a645 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  66. ✓test_chat_completion_returns_prediction_usage TestOpenAIPredictiontest_chat_completion_prediction_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4.1
    1. Add a deployment named e2e-openai-prediction-e29e79694a59 that calls openai/gpt-4.1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  67. ✓test_chat_completion_returns_reasoning[anthropic-direct] TestReasoningEfforttest_reasoning_effort_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • azure_ai
    • bedrock
    • vertex_ai
    • anthropic/claude-fable-5-1
    • azure_ai/claude-fable-5-1
    • bedrock/converse/us.anthropic.claude-fable-5-1
    • bedrock/invoke/us.anthropic.claude-opus-4-6-v1
    • vertex_ai/claude-fable-5-1
    • reasoning
    1. Add a deployment named e2e-reasoning-effort-anthropic_direct-cb50f59231f0 that calls anthropic/claude-fable-5-1
    2. Generate a virtual key with models: e2e-reasoning-effort-anthropic_direct-cb50f59231f0 and user id: e2e-test-user
    View test on GitHub ↗
  68. ✓test_chat_returns_real_completion[claude-haiku-4-5-anthropic] TestChatCompletionsRegressiontest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to claude-haiku-4-5
    View test on GitHub ↗
  69. ✓test_chat_returns_real_completion[gemini-2.5-flash-gemini] TestChatCompletionsRegressiontest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  70. ✓test_chat_returns_real_completion[gpt-5.5-openai] TestChatCompletionsRegressiontest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  71. ✓test_chat_stream_is_sse_and_ends_with_done TestChatStreamContracttest_chat_stream_contract_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.6
    1. Add a deployment named e2e-chat-stream-a61b03638cae that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-chat-stream-a61b03638cae
    View test on GitHub ↗
  72. ✓test_chat_template_kwargs_reach_together TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • together_ai/Qwen/Qwen3.5-9B
    • reasoning
    1. Add a deployment named e2e-together-abea2964fd15 that calls together_ai/Qwen/Qwen3.5-9B
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-abea2964fd15
    View test on GitHub ↗
  73. ✓test_client_max_completion_tokens_returns_completion TestBedrockMantleChatOutputLimittest_bedrock_mantle_output_limit_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock_mantle
    • bedrock_mantle/openai.gpt-5.6-terra
    1. Add a deployment named e2e-mantle-chat-output-limit-660e1b7015a1 that calls bedrock_mantle/openai.gpt-5.6-terra
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  74. ✓test_client_max_tokens_returns_completion TestBedrockMantleChatOutputLimittest_bedrock_mantle_output_limit_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock_mantle
    • bedrock_mantle/openai.gpt-5.6-terra
    1. Add a deployment named e2e-mantle-chat-output-limit-4aa9255f7217 that calls bedrock_mantle/openai.gpt-5.6-terra
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  75. ✓test_client_secret_missing_auth_is_denied TestRealtimeHttptest_realtime_http_e2e.py›
    • proxy-auth
    • realtime
    1. Add a deployment named e2e-realtime-http-2ec232cc80ee that calls openai/gpt-realtime
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/realtime/client_secrets
    View test on GitHub ↗
  76. ✓test_client_tool_call_marks_work_without_injection_points[azure-foundry] TestCacheControlInjectionToolCallstest_cache_control_injection_tool_calls_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    • function_calling
    • prompt_caching
    1. Add a deployment named e2e-cache-control-tool-calls-azure_foundry-caf9098578c3 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  77. ✓test_client_tool_call_marks_work_without_injection_points[vertex] TestCacheControlInjectionToolCallstest_cache_control_injection_tool_calls_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/claude-sonnet-5
    • function_calling
    • prompt_caching
    1. Add a deployment named e2e-cache-control-tool-calls-vertex-ccbf5b38b2f2 that calls vertex_ai/claude-sonnet-5
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  78. ✓test_cohere_chat_returns_content TestCohereChattest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • cohere
    • cohere/command-r-08-2024
    1. Add a deployment named e2e-cohere-chat-6f4d17b89450 that calls cohere/command-r-08-2024
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-cohere-chat-6f4d17b89450
    View test on GitHub ↗
  79. ✓test_cohere_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • cohere
    • cohere/embed-v4.0
    1. Add a deployment named e2e-embeddings-cohere-a90d4b2df96f that calls cohere/embed-v4.0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  80. ✓test_cohere_image_embeddings_report_image_tokens[image] TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • cohere
    • cohere/embed-english-v3.0
    • vision
    1. Add a deployment named e2e-cohere-image-fe838f1f95c1 that calls cohere/embed-english-v3.0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  81. ✓test_cohere_image_embeddings_report_image_tokens[text] TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • cohere
    • cohere/embed-english-v3.0
    • vision
    1. Add a deployment named e2e-cohere-image-3a5011c63c1b that calls cohere/embed-english-v3.0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  82. ✓test_computer_tool_returns_computer_tool_call TestAnthropicCompletiontest_anthropic_completion_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-4-5-20250929
    • function_calling
    1. Add a deployment named e2e-anthropic-computer-acd8ba0539fc that calls anthropic/claude-sonnet-4-5-20250929
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  83. ✓test_container_file_lifecycle_through_the_gateway TestOpenAIContainerFilestest_containers_e2e.py›
    • llm-translation
    • containers
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  84. ✓test_contents_are_counted_by_the_provider_endpoint[gemini-2.5-flash-vertex] TestGeminiContentsTokenCountingtest_token_counter_gemini_contents_e2e.py›
    • llm-translation
    • count_tokens
    • vertex_ai
    • gemini-2.5-flash-vertex
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /utils/token_counter
    View test on GitHub ↗
  85. ✓test_contents_are_counted_by_the_provider_endpoint[gemini-2.5-flash] TestGeminiContentsTokenCountingtest_token_counter_gemini_contents_e2e.py›
    • llm-translation
    • count_tokens
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /utils/token_counter
    View test on GitHub ↗
  86. ✓test_converse_empty_messages_returns_client_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-241201f46225 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-241201f46225/converse
    View test on GitHub ↗
  87. ✓test_converse_invalid_model_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    1. Add a deployment named e2e-bedrock-native-18ef096c31f9 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/does-not-exist/converse
    View test on GitHub ↗
  88. ✓test_converse_missing_messages_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-e1dad096dfe5 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-e1dad096dfe5/converse
    View test on GitHub ↗
  89. ✓test_converse_output_format_returns_schema_json_text TestBedrockMessagestest_messages_bedrock_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • response_schema
    1. Add a deployment named e2e-messages-bedrock-7a390d41da2e that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  90. ✓test_converse_returns_assistant TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-d0be8909126c that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-d0be8909126c/converse
    View test on GitHub ↗
  91. ✓test_converse_stream_returns_chunks TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • stream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-60741cc751b9 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-60741cc751b9/converse-stream
    View test on GitHub ↗
  92. ✓test_converse_tool_result_of_only_tool_references_is_accepted TestBedrockMessagestest_messages_bedrock_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • bedrock/global.openai.gpt-6.1-sol
    • tool_search
    1. Add a deployment named e2e-messages-bedrock-bbdb1c04169c that calls bedrock/global.openai.gpt-6.1-sol
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  93. ✓test_cost_header_and_spend_row_match_the_registry_price TestTogetherChatCompletionstest_together_ai_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • together_ai
    1. Add a deployment named e2e-together-c886d465b5b5 that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  94. ✓test_cost_header_matches_the_spend_log[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf with the prompt "Say hello. b5b03df2b6f4" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  95. ✓test_cost_header_matches_the_spend_log[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 with the prompt "Say hello. 74318ae6051a" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  96. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 with the prompt "Say hello. 4ec31ce2ca27" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  97. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 with the prompt "Say hello. ee7bf4a62373" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  98. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 with the prompt "Say hello. 901b0a45034a" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  99. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 with the prompt "Say hello. 0fa7201a0f4a" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  100. ✓test_cost_header_matches_the_spend_log[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf with the prompt "Say hello. 3cb8fec578d0" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  101. ✓test_cost_header_matches_the_spend_log[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 with the prompt "Say hello. 3a14239bb452" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  102. ✓test_cost_header_matches_the_spend_log[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 with the prompt "Say hello. dd64ee8be356" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  103. ✓test_cost_header_matches_the_spend_log[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 with the prompt "Say hello. 37bd3781d6be" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  104. ✓test_cost_header_matches_the_spend_log[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 with the prompt "Say hello. 07f24631e866" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  105. ✓test_cost_header_matches_the_spend_log[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 with the prompt "Say hello. b94e982bd193" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  106. ✓test_cost_header_matches_the_spend_log[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf with the prompt "Say hello. 7d559a6be42d" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  107. ✓test_cost_header_matches_the_spend_log[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 with the prompt "Say hello. 2a37ccf56ef7" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  108. ✓test_cost_header_matches_the_spend_log[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 with the prompt "Say hello. 1ce15a4825b3" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  109. ✓test_cost_header_matches_the_spend_log[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 with the prompt "Say hello. a7e0be227020" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  110. ✓test_cost_header_matches_the_spend_log[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 with the prompt "Say hello. e3b1f41e459b" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  111. ✓test_cost_header_matches_the_spend_log[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 with the prompt "Say hello. 757c1e0d029a" and forced weather tool use set to False
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  112. ✓test_count_comes_from_the_provider_and_matches_billed_input[baseline] TestAnthropicClaudeCountsWithProvidertest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5-5
    • function_calling
    • pdf_input
    1. Add a deployment named e2e-count-source-402d3a07cf82 that calls anthropic/claude-sonnet-5-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  113. ✓test_count_comes_from_the_provider_and_matches_billed_input[large_system] TestAnthropicClaudeCountsWithProvidertest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5-5
    • function_calling
    • pdf_input

    No steps recorded

    View test on GitHub ↗
  114. ✓test_count_comes_from_the_provider_and_matches_billed_input[large_tool] TestAnthropicClaudeCountsWithProvidertest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5-5
    • function_calling
    • pdf_input

    No steps recorded

    View test on GitHub ↗
  115. ✓test_count_comes_from_the_provider_and_matches_billed_input[pdf_1_page] TestAnthropicClaudeCountsWithProvidertest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5-5
    • function_calling
    • pdf_input

    No steps recorded

    View test on GitHub ↗
  116. ✓test_count_comes_from_the_provider_and_matches_billed_input[pdf_12_pages] TestAnthropicClaudeCountsWithProvidertest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5-5
    • function_calling
    • pdf_input

    No steps recorded

    View test on GitHub ↗
  117. ✓test_count_comes_from_the_provider_and_matches_billed_input[system_and_tool] TestAnthropicClaudeCountsWithProvidertest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5-5
    • function_calling
    • pdf_input

    No steps recorded

    View test on GitHub ↗
  118. ✓test_count_keeps_the_provider_400_instead_of_estimating TestProviderCountErrorWithoutCountModeltest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • bedrock
    • bedrock/us.amazon.nova-pro-v1:0
    1. Add a deployment named e2e-count-source-4ff8f4eec514 that calls bedrock/us.amazon.nova-pro-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  119. ✓test_count_matches_billed_input_tokens_for_a_model_bedrock_runtime_cannot_count TestBedrockMessagesCountTokenstest_messages_count_tokens_bedrock_e2e.py›
    • llm-translation
    • count_tokens
    • bedrock
    • bedrock/global.anthropic.claude-opus-4-8
    1. Add a deployment named e2e-count-tokens-bedrock-a9a4542b339b that calls bedrock/global.anthropic.claude-opus-4-8
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  120. ✓test_count_names_the_model_info_count_model TestConfiguredCountModelOverridesCostMaptest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • bedrock
    • bedrock/us.anthropic.claude-sonnet-5-5
    • us.anthropic.claude-sonnet-5
    1. Add a deployment named e2e-count-source-6c4b61eb0c0b that calls bedrock/us.anthropic.claude-sonnet-5-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  121. ✓test_create_batch_missing_input_file_id_returns_error TestFilesBatchesContracttest_files_batches_contract_e2e.py›
    • llm-translation
    • batches
    • batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /v1/batches
    View test on GitHub ↗
  122. ✓test_create_client_secret TestRealtimeHttptest_realtime_http_e2e.py›
    • llm-translation
    • realtime
    • openai
    • openai/gpt-realtime
    1. Add a deployment named e2e-realtime-http-7e1f95ee66eb that calls openai/gpt-realtime
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /v1/realtime/client_secrets
    View test on GitHub ↗
  123. ✓test_create_list_retrieve_delete_lifecycle TestVectorStorestest_vector_stores_e2e.py›
    • llm-translation
    • vector_stores
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. POST /v1/vector_stores
    3. GET /v1/vector_stores/vs_6ad463392b308191812848b4e80bc7a1
    4. GET /v1/vector_stores
    5. DELETE /v1/vector_stores/vs_6ad463392b308191812848b4e80bc7a1
    View test on GitHub ↗
  124. ✓test_credential_backed_messages TestCredentialBackedMessagestest_credential_messages_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Save the provider credential e2e-cred-44bce3acd645
    2. Add a deployment named e2e-cred-messages-44bce3acd645 that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  125. ✓test_custom_pricing_is_billed_at_configured_rate TestCustomPricingtest_custom_pricing_e2e.py›
    • cost-map
    • spend_reporting
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named custom-priced-flash-3043eabbc069 that calls gemini/gemini-2.5-flash
    3. Send a /chat/completions request to custom-priced-flash-3043eabbc069
    4. GET /spend/logs
    View test on GitHub ↗
  126. ✓test_custom_pricing_is_isolated_from_sibling_deployment TestCustomPricingtest_custom_pricing_e2e.py›
    • cost-map
    • model_management
    1. Add a deployment named custom-priced-flash-7edccfab5292 that calls gemini/gemini-2.5-flash
    2. Add a deployment named base-flash-6ab32652deff that calls gemini/gemini-2.5-flash
    3. List the deployments from /model/info
    View test on GitHub ↗
  127. ✓test_deleted_response_is_no_longer_retrievable[azure] TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-resp-delete-f0589f029fa7 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  128. ✓test_deleted_response_is_no_longer_retrievable[openai] TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-resp-delete-f6a9b371d15c that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  129. ✓test_deployment_max_tokens_streams_tool_call_completion TestBedrockMantleChatOutputLimittest_bedrock_mantle_output_limit_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • bedrock_mantle
    • bedrock_mantle/openai.gpt-5.6-terra
    • function_calling
    • tool_choice
    1. Add a deployment named e2e-mantle-chat-output-limit-f1bc9cdc5679 that calls bedrock_mantle/openai.gpt-5.6-terra
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  130. ✓test_embeddings_call_logs_its_cost TestOpenAIPassthroughSpendtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • openai
    • text-embedding-3-small
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/embeddings request to text-embedding-3-small through /openai_passthrough for "cost this sentence 19f9ff3cc5b9"
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  131. ✓test_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • openai
    • openai/text-embedding-3-small
    1. Add a deployment named e2e-embeddings-2c155d785811 that calls openai/text-embedding-3-small
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  132. ✓test_empty_image_returns_error TestImageEdittest_image_edits_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1
    1. Add a deployment named e2e-image-edit-332409009e4f that calls openai/gpt-image-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  133. ✓test_empty_input_returns_client_error TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-responses-val-1f5731bca227 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/responses
    View test on GitHub ↗
  134. ✓test_empty_messages_returns_client_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-55f0a272bf83 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  135. ✓test_empty_prompt_returns_client_error TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1-mini
    1. Add a deployment named e2e-image-0fca69c305d3 that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/images/generations
    View test on GitHub ↗
  136. ✓test_empty_prompt_returns_error TestImageEdittest_image_edits_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1
    1. Add a deployment named e2e-image-edit-26153d9b318b that calls openai/gpt-image-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  137. ✓test_every_mapped_beta_header_is_accepted_by_the_provider[anthropic] TestAnthropicBetaHeaderstest_messages_beta_headers_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • bedrock
    • anthropic/claude-sonnet-5-5
    • us.anthropic.claude-fable-5-1
    1. Add a deployment named e2e-messages-beta-d88a524c1376 that calls anthropic/claude-sonnet-5-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  138. ✓test_every_mapped_beta_header_is_accepted_by_the_provider[bedrock-converse] TestAnthropicBetaHeaderstest_messages_beta_headers_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • bedrock
    • anthropic/claude-sonnet-5-5
    • us.anthropic.claude-fable-5-1
    1. Add a deployment named e2e-messages-beta-74f27f0d3bcd that calls bedrock/converse/us.anthropic.claude-fable-5-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  139. ✓test_every_mapped_beta_header_is_accepted_by_the_provider[bedrock-invoke] TestAnthropicBetaHeaderstest_messages_beta_headers_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • bedrock
    • anthropic/claude-sonnet-5-5
    • us.anthropic.claude-fable-5-1
    1. Add a deployment named e2e-messages-beta-4416b9ee12d8 that calls bedrock/invoke/us.anthropic.claude-fable-5-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  140. ✓test_file_attach_poll_and_search TestVectorStorestest_vector_stores_e2e.py›
    • llm-translation
    • vector_stores
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload vs_doc.txt to /v1/files
    3. POST /v1/vector_stores
    4. POST /v1/vector_stores/vs_6ad46347a49081919d41516527d01eca/files
    5. GET /v1/vector_stores/vs_6ad46347a49081919d41516527d01eca/files/file-HawdijtAktAw9KCLDTnNKS
    6. POST /v1/vector_stores/vs_6ad46347a49081919d41516527d01eca/search
    7. DELETE /v1/vector_stores/vs_6ad46347a49081919d41516527d01eca/files/file-HawdijtAktAw9KCLDTnNKS
    View test on GitHub ↗
  141. ✓test_flagged_model_keeps_prompt_cache_across_system_reminder TestAnthropicChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-opus-4-8
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-chat-midsys-73ee9ee7216e that calls anthropic/claude-opus-4-8
    2. Generate a virtual key with models: e2e-chat-midsys-73ee9ee7216e and user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  142. ✓test_flagged_model_keeps_prompt_cache_across_system_reminder TestBedrockInvokeChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-sonnet-5
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-chat-midsys-ab3ee23214f0 that calls bedrock/invoke/us.anthropic.claude-sonnet-5
    2. Generate a virtual key with models: e2e-chat-midsys-ab3ee23214f0 and user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  143. ✓test_gemini TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • gemini
    • gemini/gemini-3.5-flash-lite
    • function_calling
    1. Add a deployment named e2e-chat-tool-loop-64601cf668fa that calls gemini/gemini-3.5-flash-lite
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-64601cf668fa
    View test on GitHub ↗
  144. ✓test_gemini_chat_returns_content_and_logs_cost TestGeminiChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Add a deployment named e2e-gemini-chat-5f11f4707497 that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-gemini-chat-5f11f4707497
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  145. ✓test_gemini_passthrough_nonstreaming_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a Gemini generateContent request to gemini-2.5-flash through /gemini
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  146. ✓test_gemini_passthrough_streaming_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a Gemini streamGenerateContent request to gemini-2.5-flash through /gemini
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  147. ✓test_gemini_passthrough_tool_call_logs_costtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • gemini
    • gemini-2.5-flash
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a Gemini generateContent request to gemini-2.5-flash through /gemini
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  148. ✓test_generate_content_returns_response_cost_header TestGoogleNativeGenerateContenttest_google_native_e2e.py›
    • llm-translation
    • google_genai
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-google-native-14c4e8ffa458 that calls gemini/gemini-2.5-flash
    3. Send a request to /v1beta/models/e2e-google-native-14c4e8ffa458:generateContent
    View test on GitHub ↗
  149. ✓test_google_maps_tool_accepts_widget_and_location TestVertexCapabilitiestest_vertex_capabilities_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-3-flash-preview
    1. Add a deployment named e2e-vertex-maps-edd3b28b37f7 that calls vertex_ai/gemini-3-flash-preview
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  150. ✓test_google_search_grounding_metadata_is_returned TestVertexCapabilitiestest_vertex_capabilities_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • vertex_ai
    • vertex_ai/gemini-3-flash-preview
    • web_search
    1. Add a deployment named e2e-vertex-grounding-5e0fc3a3c767 that calls vertex_ai/gemini-3-flash-preview
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  151. ✓test_guardrail_blocks_and_returns_trace TestBedrockGuardrailtest_bedrock_guardrail_config_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-opus-5-5
    1. Add a deployment named bedrock-guardrail-46ccc21a87de that calls bedrock/us.anthropic.claude-opus-5-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  152. ✓test_haiku_count_names_the_count_model_and_matches_billed_input[baseline] TestBedrockClaudeCountsWithCostMapCountModeltest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-5-5
    • us.anthropic.claude-opus-5-5
    • function_calling
    • pdf_input
    1. Add a deployment named e2e-count-source-1347beb3e551 that calls bedrock/us.anthropic.claude-haiku-5-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  153. ✓test_haiku_count_names_the_count_model_and_matches_billed_input[large_system] TestBedrockClaudeCountsWithCostMapCountModeltest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-5-5
    • us.anthropic.claude-opus-5-5
    • function_calling
    • pdf_input

    No steps recorded

    View test on GitHub ↗
  154. ✓test_haiku_count_names_the_count_model_and_matches_billed_input[large_tool] TestBedrockClaudeCountsWithCostMapCountModeltest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-5-5
    • us.anthropic.claude-opus-5-5
    • function_calling
    • pdf_input

    No steps recorded

    View test on GitHub ↗
  155. ✓test_haiku_count_names_the_count_model_and_matches_billed_input[pdf_1_page] TestBedrockClaudeCountsWithCostMapCountModeltest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-5-5
    • us.anthropic.claude-opus-5-5
    • function_calling
    • pdf_input

    No steps recorded

    View test on GitHub ↗
  156. ✓test_haiku_count_names_the_count_model_and_matches_billed_input[pdf_12_pages] TestBedrockClaudeCountsWithCostMapCountModeltest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-5-5
    • us.anthropic.claude-opus-5-5
    • function_calling
    • pdf_input

    No steps recorded

    View test on GitHub ↗
  157. ✓test_haiku_count_names_the_count_model_and_matches_billed_input[system_and_tool] TestBedrockClaudeCountsWithCostMapCountModeltest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-5-5
    • us.anthropic.claude-opus-5-5
    • function_calling
    • pdf_input

    No steps recorded

    View test on GitHub ↗
  158. ✓test_image_edit_returns_image TestImageEdittest_image_edits_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1
    1. Add a deployment named e2e-image-edit-c3be66399559 that calls openai/gpt-image-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  159. ✓test_image_generation_returns_image TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1-mini
    1. Add a deployment named e2e-image-7126bae05709 that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  160. ✓test_injection_points_respect_cap_with_tool_call_marks[azure-foundry] TestCacheControlInjectionToolCallstest_cache_control_injection_tool_calls_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    • function_calling
    • prompt_caching
    1. Add a deployment named e2e-cache-control-tool-calls-azure_foundry-9ca90d2b05ef that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  161. ✓test_injection_points_respect_cap_with_tool_call_marks[vertex] TestCacheControlInjectionToolCallstest_cache_control_injection_tool_calls_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/claude-sonnet-5
    • function_calling
    • prompt_caching
    1. Add a deployment named e2e-cache-control-tool-calls-vertex-58e19447011d that calls vertex_ai/claude-sonnet-5
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  162. ✓test_input_items_list_the_stored_prompt TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-resp-items-f3de2aa7fecc that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  163. ✓test_interaction_accepts_content_list TestGoogleInteractionstest_google_interactions_e2e.py›
    • llm-translation
    • google_genai
    • nonstream
    • gemini
    • gemini/gemini-3.8-flash
    1. Add a deployment named e2e-gemini-interactions-e7c79a1f484b that calls gemini/gemini-3.8-flash
    2. Generate a virtual key with models: e2e-gemini-interactions-e7c79a1f484b and user id: e2e-test-user
    3. POST /v1beta/interactions
    View test on GitHub ↗
  164. ✓test_interaction_applies_system_instruction TestGoogleInteractionstest_google_interactions_e2e.py›
    • llm-translation
    • google_genai
    • nonstream
    • gemini
    • gemini/gemini-3.8-flash
    1. Add a deployment named e2e-gemini-interactions-eb9c78f83451 that calls gemini/gemini-3.8-flash
    2. Generate a virtual key with models: e2e-gemini-interactions-eb9c78f83451 and user id: e2e-test-user
    3. POST /v1beta/interactions
    View test on GitHub ↗
  165. ✓test_interaction_calls_declared_tool TestGoogleInteractionstest_google_interactions_e2e.py›
    • llm-translation
    • google_genai
    • nonstream
    • gemini
    • gemini/gemini-3.8-flash
    1. Add a deployment named e2e-gemini-interactions-0a45ff647650 that calls gemini/gemini-3.8-flash
    2. Generate a virtual key with models: e2e-gemini-interactions-0a45ff647650 and user id: e2e-test-user
    3. POST /v1beta/interactions
    View test on GitHub ↗
  166. ✓test_invalid_chunking_returns_error TestVectorStorestest_vector_stores_e2e.py›
    • llm-translation
    • vector_stores
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /v1/vector_stores
    View test on GitHub ↗
  167. ✓test_invalid_max_completion_tokens_return_client_errors TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-a248d54517e6 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  168. ✓test_invalid_n_returns_client_error TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1-mini
    1. Add a deployment named e2e-image-51a347b268cb that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/images/generations
    View test on GitHub ↗
  169. ✓test_invalid_role_returns_client_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-7584db129fb7 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  170. ✓test_invalid_size_returns_client_error TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images
    • nonstream
    • openai
    • openai/gpt-image-1-mini
    1. Add a deployment named e2e-image-9a27d2e03f06 that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/images/generations
    View test on GitHub ↗
  171. ✓test_invalid_temperatures_return_client_errors TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-9b1de0828a1f that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  172. ✓test_invoke_invalid_temperature_returns_client_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-21e2d210bb18 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-21e2d210bb18/invoke
    View test on GitHub ↗
  173. ✓test_invoke_missing_max_tokens_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-a38909d1097a that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-a38909d1097a/invoke
    View test on GitHub ↗
  174. ✓test_invoke_missing_messages_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-486feffd9aba that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-486feffd9aba/invoke
    View test on GitHub ↗
  175. ✓test_invoke_regional_id_completes TestBedrockInvokeRegionalModelIdstest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-invoke-1406b4f09724 that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-invoke-1406b4f09724
    View test on GitHub ↗
  176. ✓test_invoke_regional_id_streams TestBedrockInvokeRegionalModelIdstest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-invoke-stream-e731ae2a5f28 that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bedrock-invoke-stream-e731ae2a5f28
    View test on GitHub ↗
  177. ✓test_invoke_returns_message TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-9967685b2d97 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-9967685b2d97/invoke
    View test on GitHub ↗
  178. ✓test_invoke_stream_returns_chunks TestBedrockNativetest_bedrock_native_e2e.py›
    • passthrough
    • passthrough
    • stream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-bedrock-native-6f13d7fe6769 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /bedrock/model/e2e-bedrock-native-6f13d7fe6769/invoke-with-response-stream
    View test on GitHub ↗
  179. ✓test_messages_logs_cost_matching_the_response_header TestAnthropicMessagestest_messages_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-messages-cost-2bb82577087d that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  180. ✓test_messages_output_format_returns_schema_json TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • response_schema
    1. Add a deployment named e2e-messages-88d7db22ba04 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  181. ✓test_messages_returns_completion TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-messages-644de0764d70 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  182. ✓test_messages_streams_completion TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-messages-b53092e1c64b that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  183. ✓test_messages_tool_use TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-messages-2186c2dc8333 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  184. ✓test_missing_file_returns_error TestAudioTranscriptionstest_audio_transcriptions_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-transcribe
    1. Add a deployment named e2e-transcribe-82589d7cd184 that calls openai/gpt-4o-mini-transcribe
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload empty.wav to /v1/audio/transcriptions
    View test on GitHub ↗
  185. ✓test_missing_input_returns_error TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    1. Add a deployment named e2e-embeddings-missin-717a7e1d93e4 that calls openai/text-embedding-3-small
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /embeddings
    View test on GitHub ↗
  186. ✓test_missing_messages_returns_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-4caf1969dccd that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  187. ✓test_missing_model_returns_client_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    1. Add a deployment named e2e-chat-sec-c7dd245aedfe that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  188. ✓test_missing_model_returns_client_error TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /embeddings
    View test on GitHub ↗
  189. ✓test_missing_model_returns_client_error TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a request to /v1/responses
    View test on GitHub ↗
  190. ✓test_missing_model_returns_error TestAudioTranscriptionstest_audio_transcriptions_e2e.py›
    • llm-translation
    • audio
    1. Add a deployment named e2e-transcribe-08bb7d822279 that calls openai/gpt-4o-mini-transcribe
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload weather_question_24k.wav to /v1/audio/transcriptions
    View test on GitHub ↗
  191. ✓test_missing_model_returns_error TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages
    1. Add a deployment named e2e-messages-336e948731e2 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/messages
    View test on GitHub ↗
  192. ✓test_mistral TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • mistral
    • mistral/mistral-medium-3.5
    • function_calling
    1. Add a deployment named e2e-chat-tool-loop-2d06e315905e that calls mistral/mistral-medium-3.5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-2d06e315905e
    View test on GitHub ↗
  193. ✓test_mistral_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • mistral
    • mistral/mistral-embed
    1. Add a deployment named e2e-embeddings-mistral-80ff9c690781 that calls mistral/mistral-embed
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  194. ✓test_model_info_reports_custom_pricing TestCustomPricingtest_custom_pricing_e2e.py›
    • cost-map
    • model_management
    1. Add a deployment named custom-priced-flash-9f9a027e0acf that calls gemini/gemini-2.5-flash
    2. List the deployments from /model/info
    View test on GitHub ↗
  195. ✓test_moderations_flags_violent_content TestModerationstest_moderations_e2e.py›
    • llm-translation
    • moderations
    • nonstream
    • openai
    • openai/omni-moderation-latest
    1. Add a deployment named e2e-moderation-409a8885406a that calls openai/omni-moderation-latest
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  196. ✓test_moderations_passes_benign_content TestModerationstest_moderations_e2e.py›
    • llm-translation
    • moderations
    • nonstream
    • openai
    • openai/omni-moderation-latest
    1. Add a deployment named e2e-moderation-830772426c01 that calls openai/omni-moderation-latest
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  197. ✓test_multi_turn_history_is_honored TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-0b69fc28beaf that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-sec-0b69fc28beaf
    View test on GitHub ↗
  198. ✓test_nova_sonic_response_create_completes TestNovaSonicRealtimerealtime/test_realtime_bedrock_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • bedrock
    • bedrock/amazon.nova-2-sonic-v1:0
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-nova-sonic-18c65fa2de78 that calls bedrock/amazon.nova-2-sonic-v1:0
    3. Open a /v1/realtime websocket session to e2e-nova-sonic-18c65fa2de78
    4. Wait for a session.created event on the realtime websocket
    5. Send the realtime event session.update over the websocket
    6. Wait for a session.updated event on the realtime websocket
    7. Send the realtime event conversation.item.create over the websocket
    8. Send the realtime event response.create over the websocket
    9. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  199. ✓test_nova_sonic_voice_turn_shows_each_sentence_once TestNovaSonicAssistantTextrealtime/test_realtime_bedrock_duplicate_text_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • bedrock
    • bedrock/amazon.nova-2-sonic-v1:0
    • audio_input
    • audio_output
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-nova-sonic-dupe-b197ad6bccdb that calls bedrock/amazon.nova-2-sonic-v1:0
    3. Open a /v1/realtime websocket session to e2e-nova-sonic-dupe-b197ad6bccdb
    4. Wait for a session.created event on the realtime websocket
    5. Send the realtime event session.update over the websocket
    6. Wait for a session.updated event on the realtime websocket
    7. Stream the spoken question as mic chunks, then silence until the turn goes idle
    View test on GitHub ↗
  200. ✓test_nova_stream_relays_text_usage_and_stop TestBedrockMessagestest_messages_bedrock_e2e.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • bedrock/us.amazon.nova-2-lite-v1:0
    1. Add a deployment named e2e-messages-bedrock-5a1f1db1e4df that calls bedrock/us.amazon.nova-2-lite-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  201. ✓test_ocr_reads_the_document_and_bills_its_pages[azure-ai-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • azure_ai
    • azure_ai/mistral-document-ai-2512
    1. Add a deployment named ocr-content-azure-ai-image-e807937ea0ce that calls azure_ai/mistral-document-ai-2512
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/ocr
    View test on GitHub ↗
  202. ✓test_ocr_reads_the_document_and_bills_its_pages[cohere-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • cohere
    • cohere/parse-v5.0
    1. Add a deployment named ocr-content-cohere-image-1b9a5140817c that calls cohere/parse-v5.0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/ocr
    View test on GitHub ↗
  203. ✓test_ocr_reads_the_document_and_bills_its_pages[mistral-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • mistral
    • mistral/mistral-ocr-latest
    1. Add a deployment named ocr-content-mistral-image-99264cba2d9b that calls mistral/mistral-ocr-latest
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/ocr
    View test on GitHub ↗
  204. ✓test_ocr_reads_the_document_and_bills_its_pages[mistral-pdf] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • mistral
    • mistral/mistral-ocr-latest
    1. Add a deployment named ocr-content-mistral-pdf-ca714e1e8c4f that calls mistral/mistral-ocr-latest
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/ocr
    View test on GitHub ↗
  205. ✓test_ocr_reads_the_document_and_bills_its_pages[vertex-mistral-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • vertex_ai
    • vertex_ai/mistral-ocr-2505
    1. Add a deployment named ocr-content-vertex-mistral-image-8d0975243403 that calls vertex_ai/mistral-ocr-2505
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /v1/ocr
    View test on GitHub ↗
  206. ✓test_openai_chat_logs_cost TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    1. Add a deployment named e2e-openai-cost-41c8bf769248 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-cost-41c8bf769248
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  207. ✓test_openai_chat_reasoning_reports_reasoning_tokens TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • reasoning
    1. Add a deployment named e2e-openai-reasoning-2a7511b55f38 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-reasoning-2a7511b55f38
    View test on GitHub ↗
  208. ✓test_openai_chat_returns_tool_call TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • function_calling
    1. Add a deployment named e2e-openai-tool-c8b14f6d5c62 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-tool-c8b14f6d5c62
    View test on GitHub ↗
  209. ✓test_openai_chat_streams_real_content TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.6
    1. Add a deployment named e2e-openai-chat-a1953323a75a that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-openai-chat-a1953323a75a
    View test on GitHub ↗
  210. ✓test_openai_chat_streams_tool_call TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.6
    • function_calling
    1. Add a deployment named e2e-openai-tool-stream-d6a103b29899 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-openai-tool-stream-d6a103b29899
    View test on GitHub ↗
  211. ✓test_openai_chat_structured_output_conforms_to_schema TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • response_schema
    1. Add a deployment named e2e-openai-schema-2ef48d5223f7 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-schema-2ef48d5223f7
    View test on GitHub ↗
  212. ✓test_openai_chat_vision_describes_image TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o
    • vision
    1. Add a deployment named e2e-openai-vision-815ee31dbff3 that calls openai/gpt-4o
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-vision-815ee31dbff3
    View test on GitHub ↗
  213. ✓test_openai_family_model_id_completes_with_max_tokens TestBedrockOpenAIFamilyDefaultRoutetest_bedrock_provider_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/global.openai.gpt-6-sol
    1. Add a deployment named e2e-bedrock-openai-family-5b0228cf8d9f that calls bedrock/global.openai.gpt-6-sol
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-openai-family-5b0228cf8d9f
    View test on GitHub ↗
  214. ✓test_openai_prefix_chat_returns_completion_and_logs_its_cost TestOpenAIProviderPrefixChattest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/chat/completions request to gpt-5.5 through /openai
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  215. ✓test_openai_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.6
    • prompt_caching
    1. Add a deployment named e2e-openai-cache-9a5d7b25f25c that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-cache-9a5d7b25f25c
    View test on GitHub ↗
  216. ✓test_openai_service_tier_is_echoed TestServiceTiertest_provider_features_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-service-tier-03df28f92866 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-service-tier-03df28f92866
    View test on GitHub ↗
  217. ✓test_output_format_returns_schema_json TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    • response_schema
    1. Add a deployment named e2e-azure-foundry-messages-d26467fa6042 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-d26467fa6042 and user id: e2e-test-user
    View test on GitHub ↗
  218. ✓test_passthrough_denies_model_outside_key_allowlist TestPassthroughModelAllowlisttest_passthrough_e2e.py›
    • proxy-auth
    • passthrough
    1. Generate a virtual key with models: gemini-2.5-flash
    2. Send a /v1/messages request to claude-haiku-4-5 through /anthropic with streaming set to False
    View test on GitHub ↗
  219. ✓test_passthrough_prefix_lists_batches_from_openai TestOpenAIPassthroughPrefixtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. List batches from /openai_passthrough/v1/batches
    View test on GitHub ↗
  220. ✓test_passthrough_prefix_uploads_a_file_to_openai TestOpenAIPassthroughPrefixtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload e2e-passthrough-batch.jsonl to /openai_passthrough/v1/files
    View test on GitHub ↗
  221. ✓test_pipecat_tool_smoke[azure]realtime/test_realtime_pipecat_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • azure
    • azure/gpt-realtime
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  222. ✓test_pipecat_tool_smoke[gemini]realtime/test_realtime_pipecat_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • gemini
    • gemini/gemini-3.1-flash-live-preview
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  223. ✓test_pipecat_tool_smoke[openai]realtime/test_realtime_pipecat_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • openai
    • openai/gpt-realtime-2
    • function_calling
    1. Add a realtime deployment that calls openai/gpt-realtime-2
    2. Add a realtime deployment that calls azure/gpt-realtime
    3. Add a realtime deployment that calls gemini/gemini-3.1-flash-live-preview
    4. Add a realtime deployment that calls vertex_ai/gemini-live-2.5-flash-native-audio
    5. Add a realtime deployment that calls bedrock/amazon.nova-2-sonic-v1:0
    6. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  224. ✓test_plain_call_returns_a_message TestSailMessagestest_sail_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-a691c094efdf that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  225. ✓test_polly_generative_voice_returns_mp3 TestAwsPollySpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • aws_polly
    • aws_polly/generative
    1. Add a deployment named e2e-speech-polly-cf9f634c49f7 that calls aws_polly/generative
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  226. ✓test_realtime_upgrade_reaches_openai_through_the_passthrough_prefix TestOpenAIPassthroughWebsockettest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • websocket
    • openai
    • gpt-realtime-2
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a websocket to /openai_passthrough/v1/realtime and wait for its first event
    View test on GitHub ↗
  227. ✓test_reasoner_returns_reasoning_by_default TestDeepSeekReasoningDisabletest_deepseek_reasoning_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • deepseek
    • deepseek/deepseek-v4-pro
    • reasoning
    1. Add a deployment named e2e-deepseek-reasoner-8fbe8342d3c1 that calls deepseek/deepseek-v4-pro
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-deepseek-reasoner-8fbe8342d3c1
    View test on GitHub ↗
  228. ✓test_reasoning_effort_none_disables_reasoning TestDeepSeekReasoningDisabletest_deepseek_reasoning_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • deepseek
    • deepseek/deepseek-v4-pro
    • reasoning
    1. Add a deployment named e2e-deepseek-reasoner-35f672f6bef7 that calls deepseek/deepseek-v4-pro
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-deepseek-reasoner-35f672f6bef7
    View test on GitHub ↗
  229. ✓test_reasoning_effort_none_reaches_together TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • together_ai/Qwen/Qwen3.5-9B
    • reasoning
    1. Add a deployment named e2e-together-8d519c023d31 that calls together_ai/Qwen/Qwen3.5-9B
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-8d519c023d31
    View test on GitHub ↗
  230. ✓test_reasoning_items_replay_into_structured_output_after_tool_call TestResponsesOpenAIHostedFeaturestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    • reasoning
    • response_schema
    1. Add a deployment named e2e-responses-reasoning-57652c88cd30 that calls openai/gpt-5.4-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  231. ✓test_reasoning_streams_as_reasoning_content_deltas TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • together_ai
    • reasoning
    1. Add a deployment named e2e-together-04305c2d10b0 that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-together-04305c2d10b0
    View test on GitHub ↗
  232. ✓test_reasoning_surfaces_as_reasoning_content TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • reasoning
    1. Read the model cost map
    2. Add a deployment named e2e-together-7b3301efa650 that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    3. Generate a virtual key with user id: e2e-test-user
    4. Send a /chat/completions request to e2e-together-7b3301efa650
    View test on GitHub ↗
  233. ✓test_replayed_reasoning_content_reaches_together TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • together_ai/MiniMaxAI/MiniMax-M3
    • reasoning
    1. Add a deployment named e2e-together-e8a5e23e9a87 that calls together_ai/MiniMaxAI/MiniMax-M3
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-e8a5e23e9a87
    View test on GitHub ↗
  234. ✓test_reply_carries_assistant_text_and_usage[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  235. ✓test_reply_carries_assistant_text_and_usage[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  236. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Start a provider edge server
    2. Add a deployment named e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 that calls openai/gpt-4o-mini
    3. Save the provider credential e2e-matrix-gpt-4o-mini-bd009d9a6d49
    4. Add a deployment named e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 that calls openai/gpt-4o-mini
    5. Add a deployment named e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 that calls openai/gpt-5.4-mini
    6. Save the provider credential e2e-matrix-gpt-5.4-mini-b86055fd4108
    7. Add a deployment named e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 that calls openai/gpt-5.4-mini
    8. Add a deployment named e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf that calls anthropic/claude-haiku-4-5
    9. Save the provider credential e2e-matrix-claude-haiku-4-5-f39cf5e532b5
    10. Add a deployment named e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 that calls anthropic/claude-haiku-4-5
    11. Generate a virtual key with user id: e2e-test-user
    12. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  237. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  238. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  239. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  240. ✓test_reply_carries_assistant_text_and_usage[chat_completions-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Add a deployment named e2e-ollama-ollama_chat-44ad05adf838 that calls ollama_chat/gemma4:31b
    2. Add a deployment named e2e-ollama-ollama-8167e9114b0b that calls ollama/gemma4:31b
    3. Generate a virtual key with user id: e2e-test-user
    4. Send a /chat/completions request to e2e-ollama-ollama_chat-44ad05adf838 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  241. ✓test_reply_carries_assistant_text_and_usage[chat_completions-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ollama-ollama-8167e9114b0b with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  242. ✓test_reply_carries_assistant_text_and_usage[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  243. ✓test_reply_carries_assistant_text_and_usage[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  244. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  245. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  246. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  247. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  248. ✓test_reply_carries_assistant_text_and_usage[messages-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama_chat-44ad05adf838 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  249. ✓test_reply_carries_assistant_text_and_usage[messages-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama-8167e9114b0b with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  250. ✓test_reply_carries_assistant_text_and_usage[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  251. ✓test_reply_carries_assistant_text_and_usage[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  252. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  253. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  254. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  255. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  256. ✓test_reply_carries_assistant_text_and_usage[responses-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama_chat-44ad05adf838 with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  257. ✓test_reply_carries_assistant_text_and_usage[responses-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama-8167e9114b0b with the prompt "Say hello." and forced weather tool use set to False
    View test on GitHub ↗
  258. ✓test_required_tool_arguments_and_correlated_result[nonstream] TestOpenAIMessagesToolContinuationtest_messages_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.6
    • function_calling
    1. Add a deployment named e2e-bridge-tool-bdee7e348502 that calls openai/gpt-5.6
    2. Generate a virtual key with models: e2e-bridge-tool-bdee7e348502 and user id: e2e-test-user
    View test on GitHub ↗
  259. ✓test_required_tool_arguments_and_correlated_result[stream] TestOpenAIMessagesToolContinuationtest_messages_e2e.py›
    • llm-translation
    • messages
    • stream
    • openai
    • openai/gpt-5.6
    • function_calling
    1. Add a deployment named e2e-bridge-tool-2c17809ff94d that calls openai/gpt-5.6
    2. Generate a virtual key with models: e2e-bridge-tool-2c17809ff94d and user id: e2e-test-user
    View test on GitHub ↗
  260. ✓test_rerank_scores_top_n TestReranktest_rerank_e2e.py›
    • llm-translation
    • rerank
    • nonstream
    • cohere
    • cohere/rerank-v3.5
    1. Add a deployment named e2e-rerank-6a68bcd653d6 that calls cohere/rerank-v3.5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/rerank request to e2e-rerank-6a68bcd653d6 for "What is the capital of the United States?"
    View test on GitHub ↗
  261. ✓test_response_format_json_schema_shapes_the_reply TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • together_ai/Qwen/Qwen3.5-9B
    • response_schema
    1. Add a deployment named e2e-together-8f6389b330a9 that calls together_ai/Qwen/Qwen3.5-9B
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-8f6389b330a9
    View test on GitHub ↗
  262. ✓test_responses_anthropic_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-responses-82d2130a6c90 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  263. ✓test_responses_anthropic_returns_function_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-responses-c89b422919c8 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  264. ✓test_responses_anthropic_strict_array_schema_tool_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-responses-0d0de4f4eea0 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  265. ✓test_responses_anthropic_tool_output_continues_with_previous_response_id TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-responses-c7c42cf457db that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  266. ✓test_responses_azure_openai_preview_api_version_accepts_truncation TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-responses-azure-preview-7b012126ed37 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  267. ✓test_responses_azure_openai_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-responses-azure-openai-d2eabe82b87b that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  268. ✓test_responses_azure_openai_returns_function_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    • function_calling
    1. Add a deployment named e2e-responses-azure-openai-tool-7c56fa40a5c6 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  269. ✓test_responses_azure_openai_streaming_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • stream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-responses-azure-stream-676d4371c19c that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  270. ✓test_responses_bedrock_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a deployment named e2e-responses-4d541cd85722 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  271. ✓test_responses_bedrock_returns_function_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • function_calling
    1. Add a deployment named e2e-responses-5e945f387fec that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  272. ✓test_responses_compact_returns_compacted_conversation TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-responses-compact-770d4cc6f522 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  273. ✓test_responses_context_management_compacts_server_side TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-responses-context-compaction-11c99da16e30 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  274. ✓test_responses_gemini_replays_legacy_function_call_output TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    • function_calling
    1. Add a deployment named e2e-responses-gemini-tool-0e3c0e6c9da2 that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  275. ✓test_responses_gemini_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Add a deployment named e2e-responses-gemini-94674e67ce50 that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  276. ✓test_responses_gemini_streaming_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • stream
    • gemini
    • gemini/gemini-2.5-flash
    1. Add a deployment named e2e-responses-gemini-cbacbdac6d48 that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  277. ✓test_responses_input_tokens_says_the_provider_counted TestCountSourceOnOtherCountEndpointstest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • openai
    • openai/gpt-5.6
    1. Add a deployment named e2e-count-source-89079c52a2a2 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  278. ✓test_responses_logs_cost TestResponsestest_responses_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-responses-28e2d5689a90 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  279. ✓test_responses_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-responses-cd101e321174 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  280. ✓test_responses_returns_function_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Add a deployment named e2e-responses-e550a3cb40f4 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  281. ✓test_responses_streaming_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-responses-259c1ce0ec48 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  282. ✓test_responses_upgrade_is_accepted_on_the_openai_prefix TestOpenAIPassthroughWebsockettest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • websocket
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a websocket to /openai/v1/responses and wait for its first event
    View test on GitHub ↗
  283. ✓test_responses_vertex_returns_completion TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    1. Add a deployment named e2e-responses-vertex-55c962d4f54b that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  284. ✓test_responses_vertex_returns_function_call TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    • function_calling
    1. Add a deployment named e2e-responses-vertex-tool-781fcadcbade that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  285. ✓test_responses_vision_describes_image TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o
    • vision
    1. Add a deployment named e2e-responses-94d1320515b9 that calls openai/gpt-4o
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  286. ✓test_retrieve_invalid_batch_id_returns_error TestFilesBatchesContracttest_files_batches_contract_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    1. Generate a virtual key with user id: e2e-test-user
    2. GET /v1/batches/invalid-batch-id
    View test on GitHub ↗
  287. ✓test_rust_ocr_response[azure-ai] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • azure_ai
    • azure_ai/mistral-document-ai-2512
    1. Add a deployment named rust-ocr-azure-ai-c5df5724d2d9 that calls azure_ai/mistral-document-ai-2512
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-azure-ai-c5df5724d2d9
    View test on GitHub ↗
  288. ✓test_rust_ocr_response[azure-document-intelligence] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • azure_ai
    • azure_ai/doc-intelligence/prebuilt-layout
    1. Add a deployment named rust-ocr-azure-document-intelligence-cbb754f743c1 that calls azure_ai/doc-intelligence/prebuilt-layout
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-azure-document-intelligence-cbb754f743c1
    View test on GitHub ↗
  289. ✓test_rust_ocr_response[mistral] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • mistral
    • mistral/mistral-ocr-latest
    1. Add a deployment named rust-ocr-mistral-c3e0ab49caa0 that calls mistral/mistral-ocr-latest
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-mistral-c3e0ab49caa0
    View test on GitHub ↗
  290. ✓test_rust_ocr_response[vertex-mistral] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    • llm-translation
    • ocr
    • nonstream
    • vertex_ai
    • vertex_ai/mistral-ocr-2505
    1. Add a deployment named rust-ocr-vertex-mistral-776de8f1c8c7 that calls vertex_ai/mistral-ocr-2505
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-vertex-mistral-776de8f1c8c7
    View test on GitHub ↗
  291. ✓test_service_account_key_reads_container_file_by_native_id TestAzureContainerFilestest_containers_e2e.py›
    • llm-translation
    • containers
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-containers-decoy-c2a4cd2d8344 that calls azure/gpt-5.4-nano
    2. Add a deployment named e2e-containers-c2a4cd2d8344 that calls azure/gpt-5.4-nano
    3. Create a team with team alias: e2e-containers-c2a4cd2d8344 and models: e2e-containers-c2a4cd2d8344
    4. Create an internal user with user email: e2e-containers-c2a4cd2d8344@example.com and user role: internal_user
    5. Add a user to the team with /team/member_add
    6. POST /key/service-account/generate
    View test on GitHub ↗
  292. ✓test_service_account_key_reads_container_file_created_by_a_streamed_response TestAzureContainerFilestest_containers_e2e.py›
    • llm-translation
    • containers
    • stream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-containers-decoy-76576639f765 that calls azure/gpt-5.4-nano
    2. Add a deployment named e2e-containers-76576639f765 that calls azure/gpt-5.4-nano
    3. Create a team with team alias: e2e-containers-76576639f765 and models: e2e-containers-76576639f765
    4. Create an internal user with user email: e2e-containers-76576639f765@example.com and user role: internal_user
    5. Add a user to the team with /team/member_add
    6. POST /key/service-account/generate
    View test on GitHub ↗
  293. ✓test_service_tier_bills_the_matching_completion_window[auto-base] TestSailChatCompletionstest_sail_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-5a16b4f75da9 that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  294. ✓test_service_tier_bills_the_matching_completion_window[balanced-balanced] TestSailChatCompletionstest_sail_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-430376b9968a that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  295. ✓test_shell_tool_stream_surfaces_shell_call_and_its_output TestResponsesOpenAIHostedFeaturestest_responses_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-responses-shell-86083caa2fca that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  296. ✓test_sonnet_count_names_the_count_model_and_matches_billed_input[baseline] TestBedrockClaudeCountsWithCostMapCountModeltest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-sonnet-5-5
    • us.anthropic.claude-opus-5-5
    • function_calling
    • pdf_input
    1. Add a deployment named e2e-count-source-0163d1b3adb2 that calls bedrock/us.anthropic.claude-sonnet-5-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  297. ✓test_sonnet_count_names_the_count_model_and_matches_billed_input[large_system] TestBedrockClaudeCountsWithCostMapCountModeltest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-sonnet-5-5
    • us.anthropic.claude-opus-5-5
    • function_calling
    • pdf_input

    No steps recorded

    View test on GitHub ↗
  298. ✓test_sonnet_count_names_the_count_model_and_matches_billed_input[large_tool] TestBedrockClaudeCountsWithCostMapCountModeltest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-sonnet-5-5
    • us.anthropic.claude-opus-5-5
    • function_calling
    • pdf_input

    No steps recorded

    View test on GitHub ↗
  299. ✓test_sonnet_count_names_the_count_model_and_matches_billed_input[pdf_1_page] TestBedrockClaudeCountsWithCostMapCountModeltest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-sonnet-5-5
    • us.anthropic.claude-opus-5-5
    • function_calling
    • pdf_input

    No steps recorded

    View test on GitHub ↗
  300. ✓test_sonnet_count_names_the_count_model_and_matches_billed_input[pdf_12_pages] TestBedrockClaudeCountsWithCostMapCountModeltest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-sonnet-5-5
    • us.anthropic.claude-opus-5-5
    • function_calling
    • pdf_input

    No steps recorded

    View test on GitHub ↗
  301. ✓test_sonnet_count_names_the_count_model_and_matches_billed_input[system_and_tool] TestBedrockClaudeCountsWithCostMapCountModeltest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • bedrock/us.anthropic.claude-sonnet-5-5
    • us.anthropic.claude-opus-5-5
    • function_calling
    • pdf_input

    No steps recorded

    View test on GitHub ↗
  302. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf with the prompt "Say hello."
    View test on GitHub ↗
  303. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 with the prompt "Say hello."
    View test on GitHub ↗
  304. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 with the prompt "Say hello."
    View test on GitHub ↗
  305. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 with the prompt "Say hello."
    View test on GitHub ↗
  306. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 with the prompt "Say hello."
    View test on GitHub ↗
  307. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 with the prompt "Say hello."
    View test on GitHub ↗
  308. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-ollama-ollama_chat-44ad05adf838 with the prompt "Say hello."
    View test on GitHub ↗
  309. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to e2e-ollama-ollama-8167e9114b0b with the prompt "Say hello."
    View test on GitHub ↗
  310. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf with the prompt "Say hello."
    View test on GitHub ↗
  311. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 with the prompt "Say hello."
    View test on GitHub ↗
  312. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 with the prompt "Say hello."
    View test on GitHub ↗
  313. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 with the prompt "Say hello."
    View test on GitHub ↗
  314. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 with the prompt "Say hello."
    View test on GitHub ↗
  315. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 with the prompt "Say hello."
    View test on GitHub ↗
  316. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-ollama-ollama_chat-44ad05adf838 with the prompt "Say hello."
    View test on GitHub ↗
  317. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to e2e-ollama-ollama-8167e9114b0b with the prompt "Say hello."
    View test on GitHub ↗
  318. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf with the prompt "Say hello."
    View test on GitHub ↗
  319. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 with the prompt "Say hello."
    View test on GitHub ↗
  320. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 with the prompt "Say hello."
    View test on GitHub ↗
  321. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 with the prompt "Say hello."
    View test on GitHub ↗
  322. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 with the prompt "Say hello."
    View test on GitHub ↗
  323. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-5.4-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 with the prompt "Say hello."
    View test on GitHub ↗
  324. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-ollama-ollama_chat-44ad05adf838 with the prompt "Say hello."
    View test on GitHub ↗
  325. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • stream
    • ollama
    • gemma4:31b
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/responses request to e2e-ollama-ollama-8167e9114b0b with the prompt "Say hello."
    View test on GitHub ↗
  326. ✓test_stream_generate_content_frames_sse_the_way_google_sdks_expect TestGoogleNativeGenerateContenttest_google_native_e2e.py›
    • llm-translation
    • google_genai
    • stream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-google-native-b80657d04cfa that calls gemini/gemini-2.5-flash
    3. Send a request to /v1beta/models/e2e-google-native-b80657d04cfa:streamGenerateContent
    View test on GitHub ↗
  327. ✓test_streamed_response_can_be_deleted[azure] TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • stream
    • azure
    • azure/gpt-5.4-nano
    1. Add a deployment named e2e-resp-delete-stream-13cc9a7d9691 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  328. ✓test_streamed_response_can_be_deleted[openai] TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • stream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-resp-delete-stream-e08b3dbfeced that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  329. ✓test_streamed_responses_call_logs_its_cost TestOpenAIPassthroughSpendtest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • stream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to gpt-5.5 through /openai_passthrough with streaming set to True
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  330. ✓test_streams_text_deltas TestTogetherMessagestest_together_ai_e2e.py›
    • llm-translation
    • messages
    • stream
    • together_ai
    1. Add a deployment named e2e-together-8e42b5f2b3a6 that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /v1/messages request to e2e-together-8e42b5f2b3a6
    View test on GitHub ↗
  331. ✓test_success_response_matches_chat_completion_contract TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-82df717be574 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  332. ✓test_temperature_boundaries_succeed TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Add a deployment named e2e-chat-sec-64f2b4e2e56a that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a request to /chat/completions
    View test on GitHub ↗
  333. ✓test_text_completion_returns_text TestCompletionsEndpointtest_completions_endpoint_e2e.py›
    • llm-translation
    • completions
    • nonstream
    • openai
    • openai/gpt-5.4-nano
    1. Add a deployment named e2e-completions-66b81395d573 that calls openai/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  334. ✓test_text_conversation[azure]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • azure
    • azure/gpt-realtime
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to azure-realtime-a9e71532da4c
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  335. ✓test_text_conversation[bedrock]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • bedrock
    • bedrock/amazon.nova-2-sonic-v1:0
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to bedrock-realtime-7e3cc855d912
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  336. ✓test_text_conversation[gemini]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • gemini
    • gemini/gemini-3.1-flash-live-preview
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to gemini-realtime-57d5e25f60a9
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  337. ✓test_text_conversation[openai]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • openai
    • openai/gpt-realtime-2
    1. Add a realtime deployment that calls openai/gpt-realtime-2
    2. Add a realtime deployment that calls azure/gpt-realtime
    3. Add a realtime deployment that calls gemini/gemini-3.1-flash-live-preview
    4. Add a realtime deployment that calls vertex_ai/gemini-live-2.5-flash-native-audio
    5. Add a realtime deployment that calls bedrock/amazon.nova-2-sonic-v1:0
    6. Generate a virtual key with user id: e2e-test-user
    7. Open a /v1/realtime websocket session to openai-realtime-9f43828dd53e
    8. Wait for a session.created event on the realtime websocket
    9. Send the realtime event session.update over the websocket
    10. Wait for a session.updated event on the realtime websocket
    11. Send the realtime event conversation.item.create over the websocket
    12. Send the realtime event response.create over the websocket
    13. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  338. ✓test_text_conversation[vertex_ai]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • vertex_ai
    • vertex_ai/gemini-live-2.5-flash-native-audio
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to vertex-realtime-b8440228631a
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  339. ✓test_text_editor_returns_tool_call TestAnthropicCompletiontest_anthropic_completion_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5-5
    • function_calling
    1. Add a deployment named e2e-anthropic-text-editor-ad408b22b074 that calls anthropic/claude-sonnet-5-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  340. ✓test_text_estimate_matches_billed_input[baseline] TestBedrockKimiCountsWithFramedLocalEstimatetest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • bedrock/us.moonshotai.kimi-k3
    1. Add a deployment named e2e-count-source-d5bae56f2a19 that calls bedrock/us.moonshotai.kimi-k3
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  341. ✓test_text_estimate_matches_billed_input[large_system] TestBedrockKimiCountsWithFramedLocalEstimatetest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • bedrock/us.moonshotai.kimi-k3

    No steps recorded

    View test on GitHub ↗
  342. ✓test_thinking_disabled_disables_reasoning TestDeepSeekReasoningDisabletest_deepseek_reasoning_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • deepseek
    • deepseek/deepseek-v4-pro
    • reasoning
    1. Add a deployment named e2e-deepseek-reasoner-6b6c2716efb1 that calls deepseek/deepseek-v4-pro
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-deepseek-reasoner-6b6c2716efb1
    View test on GitHub ↗
  343. ✓test_token_counter_says_kimi_was_estimated_locally TestCountSourceOnOtherCountEndpointstest_count_tokens_source_e2e.py›
    • llm-translation
    • count_tokens
    • bedrock
    • bedrock/us.moonshotai.kimi-k3
    1. Count tokens with /utils/token_counter for e2e-count-source-d5bae56f2a19
    View test on GitHub ↗
  344. ✓test_tool_call_is_returned TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • function_calling
    1. Add a deployment named e2e-together-d7f61d6a9d42 that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-d7f61d6a9d42
    View test on GitHub ↗
  345. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set
    View test on GitHub ↗
  346. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather to
    View test on GitHub ↗
  347. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to T
    View test on GitHub ↗
  348. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool us
    View test on GitHub ↗
  349. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to
    View test on GitHub ↗
  350. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool u
    View test on GitHub ↗
  351. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ollama-ollama_chat-44ad05adf838 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  352. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ollama-ollama-8167e9114b0b with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  353. ✓test_tool_call_is_returned_named_and_addressable[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to T
    View test on GitHub ↗
  354. ✓test_tool_call_is_returned_named_and_addressable[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool us
    View test on GitHub ↗
  355. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  356. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set
    View test on GitHub ↗
  357. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  358. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use se
    View test on GitHub ↗
  359. ✓test_tool_call_is_returned_named_and_addressable[messages-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama_chat-44ad05adf838 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  360. ✓test_tool_call_is_returned_named_and_addressable[messages-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama-8167e9114b0b with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  361. ✓test_tool_call_is_returned_named_and_addressable[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to
    View test on GitHub ↗
  362. ✓test_tool_call_is_returned_named_and_addressable[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool u
    View test on GitHub ↗
  363. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  364. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use se
    View test on GitHub ↗
  365. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  366. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use s
    View test on GitHub ↗
  367. ✓test_tool_call_is_returned_named_and_addressable[responses-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama_chat-44ad05adf838 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  368. ✓test_tool_call_is_returned_named_and_addressable[responses-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama-8167e9114b0b with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    View test on GitHub ↗
  369. ✓test_tool_call_is_streamed TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • together_ai
    • function_calling
    1. Add a deployment named e2e-together-f30b5f1ab972 that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-together-f30b5f1ab972
    View test on GitHub ↗
  370. ✓test_tool_call_round_trip[azure]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • azure
    • azure/gpt-realtime
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to azure-realtime-a9e71532da4c
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    9. Send the realtime event conversation.item.create over the websocket
    10. Send the realtime event response.create over the websocket
    11. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  371. ✓test_tool_call_round_trip[bedrock]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • bedrock
    • bedrock/amazon.nova-2-sonic-v1:0
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to bedrock-realtime-7e3cc855d912
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    9. Send the realtime event conversation.item.create over the websocket
    10. Send the realtime event response.create over the websocket
    11. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  372. ✓test_tool_call_round_trip[gemini]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • gemini
    • gemini/gemini-3.1-flash-live-preview
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to gemini-realtime-57d5e25f60a9
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    9. Send the realtime event conversation.item.create over the websocket
    10. Send the realtime event response.create over the websocket
    11. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  373. ✓test_tool_call_round_trip[openai]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • openai
    • openai/gpt-realtime-2
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to openai-realtime-9f43828dd53e
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    9. Send the realtime event conversation.item.create over the websocket
    10. Send the realtime event response.create over the websocket
    11. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  374. ✓test_tool_call_round_trip[vertex_ai]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • vertex_ai
    • vertex_ai/gemini-live-2.5-flash-native-audio
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Open a /v1/realtime websocket session to vertex-realtime-b8440228631a
    3. Wait for a session.created event on the realtime websocket
    4. Send the realtime event session.update over the websocket
    5. Wait for a session.updated event on the realtime websocket
    6. Send the realtime event conversation.item.create over the websocket
    7. Send the realtime event response.create over the websocket
    8. Wait for a response.done event on the realtime websocket
    9. Send the realtime event conversation.item.create over the websocket
    10. Send the realtime event response.create over the websocket
    11. Wait for a response.done event on the realtime websocket
    View test on GitHub ↗
  375. ✓test_tool_call_streams_as_tool_call_deltas[ollama_chat] TestOllamaStreamedToolCalltest_ollama_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  376. ✓test_tool_call_streams_as_tool_call_deltas[ollama] TestOllamaStreamedToolCalltest_ollama_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  377. ✓test_tool_result_round_trip TestTogetherChatCompletionstest_together_ai_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • together_ai
    • function_calling
    1. Add a deployment named e2e-together-faff3a2d9a83 that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-faff3a2d9a83
    View test on GitHub ↗
  378. ✓test_tool_result_round_trip TestTogetherMessagestest_together_ai_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • together_ai
    • function_calling
    1. Add a deployment named e2e-together-b8cb7e0022ec that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/messages request to e2e-together-b8cb7e0022ec
    View test on GitHub ↗
  379. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf over /chat/completions
    View test on GitHub ↗
  380. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather to
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 over /chat/completions
    View test on GitHub ↗
  381. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to T
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 over /chat/completions
    View test on GitHub ↗
  382. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool us
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 over /chat/completions
    View test on GitHub ↗
  383. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 over /chat/completions
    View test on GitHub ↗
  384. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool u
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 over /chat/completions
    View test on GitHub ↗
  385. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ollama-ollama_chat-44ad05adf838 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama_chat-44ad05adf838 over /chat/completions
    View test on GitHub ↗
  386. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ollama-ollama-8167e9114b0b with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama-8167e9114b0b over /chat/completions
    View test on GitHub ↗
  387. ✓test_tool_result_round_trip_reaches_the_model[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to T
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf over /v1/messages
    View test on GitHub ↗
  388. ✓test_tool_result_round_trip_reaches_the_model[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool us
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 over /v1/messages
    View test on GitHub ↗
  389. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 over /v1/messages
    View test on GitHub ↗
  390. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 over /v1/messages
    View test on GitHub ↗
  391. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 over /v1/messages
    View test on GitHub ↗
  392. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use se
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 over /v1/messages
    View test on GitHub ↗
  393. ✓test_tool_result_round_trip_reaches_the_model[messages-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama_chat-44ad05adf838 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama_chat-44ad05adf838 over /v1/messages
    View test on GitHub ↗
  394. ✓test_tool_result_round_trip_reaches_the_model[messages-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to e2e-ollama-ollama-8167e9114b0b with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama-8167e9114b0b over /v1/messages
    View test on GitHub ↗
  395. ✓test_tool_result_round_trip_reaches_the_model[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-env_ref-c556bf9689cf over /v1/responses
    View test on GitHub ↗
  396. ✓test_tool_result_round_trip_reaches_the_model[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool u
    3. Send the get_weather tool result back to e2e-matrix-claude-haiku-4-5-stored_credential-f39cf5e532b5 over /v1/responses
    View test on GitHub ↗
  397. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-env_ref-cf02233538b0 over /v1/responses
    View test on GitHub ↗
  398. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use se
    3. Send the get_weather tool result back to e2e-matrix-gpt-4o-mini-stored_credential-bd009d9a6d49 over /v1/responses
    View test on GitHub ↗
  399. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-env_ref-4ab8274ff7e0 over /v1/responses
    View test on GitHub ↗
  400. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-5.4-mini
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use s
    3. Send the get_weather tool result back to e2e-matrix-gpt-5.4-mini-stored_credential-b86055fd4108 over /v1/responses
    View test on GitHub ↗
  401. ✓test_tool_result_round_trip_reaches_the_model[responses-ollama_chat] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama_chat-44ad05adf838 with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama_chat-44ad05adf838 over /v1/responses
    View test on GitHub ↗
  402. ✓test_tool_result_round_trip_reaches_the_model[responses-ollama] TestOllamaConversationtest_ollama_e2e.py›
    • llm-translation
    • nonstream
    • ollama
    • gemma4:31b
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to e2e-ollama-ollama-8167e9114b0b with the prompt "What is the weather in Paris right now? Use the get_weather tool." and forced weather tool use set to True
    3. Send the get_weather tool result back to e2e-ollama-ollama-8167e9114b0b over /v1/responses
    View test on GitHub ↗
  403. ✓test_tool_use_block_is_returned TestTogetherMessagestest_together_ai_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • together_ai
    • function_calling
    1. Add a deployment named e2e-together-9b5444ced62a that calls together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/messages request to e2e-together-9b5444ced62a
    View test on GitHub ↗
  404. ✓test_tool_use_nonstream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-azure-foundry-messages-7efd6e7297f4 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-7efd6e7297f4 and user id: e2e-test-user
    View test on GitHub ↗
  405. ✓test_tool_use_stream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    • llm-translation
    • messages
    • stream
    • azure_ai
    • azure_ai/claude-haiku-4-5
    • function_calling
    1. Add a deployment named e2e-azure-foundry-messages-17dfedb1c065 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-17dfedb1c065 and user id: e2e-test-user
    View test on GitHub ↗
  406. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestAnthropicChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-chat-midsys-e9c75f18fe1d that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-chat-midsys-e9c75f18fe1d and user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  407. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestBedrockInvokeChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-chat-midsys-3bd7f75bc109 that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with models: e2e-chat-midsys-3bd7f75bc109 and user id: e2e-test-user
    3. POST /v1/chat/completions
    View test on GitHub ↗
  408. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestBedrockInvokeMidConversationSystemtest_messages_mid_conversation_system_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-midsys-1979acbafb33 that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with models: e2e-midsys-1979acbafb33 and user id: e2e-test-user
    View test on GitHub ↗
  409. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestAzureFoundryMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • azure_ai/claude-opus-4-7
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-midsys-12e2ba41d759 that calls azure_ai/claude-opus-4-7
    2. Generate a virtual key with models: e2e-midsys-12e2ba41d759 and user id: e2e-test-user
    View test on GitHub ↗
  410. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestVertexMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • vertex_ai/claude-sonnet-4-6
    • mid_conversation_system
    • prompt_caching
    1. Add a deployment named e2e-midsys-9374cbd9245e that calls vertex_ai/claude-sonnet-4-6
    2. Generate a virtual key with models: e2e-midsys-9374cbd9245e and user id: e2e-test-user
    View test on GitHub ↗
  411. ✓test_unknown_service_tier_is_dropped_and_billed_asap[5] TestSailChatCompletionstest_sail_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-7a62522bf707 that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  412. ✓test_unknown_service_tier_is_dropped_and_billed_asap[bogus] TestSailChatCompletionstest_sail_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • sail
    • sail/zai-org/GLM-5.3
    1. Add a deployment named e2e-sail-93502a29ffe5 that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  413. ✓test_upload_without_purpose_returns_error TestFilesBatchesContracttest_files_batches_contract_e2e.py›
    • llm-translation
    • files
    • batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload batch_input.jsonl to /v1/files
    View test on GitHub ↗
  414. ✓test_upstream_handshake_refusal_is_an_error_event_and_policy_close[azure-bad-key]realtime/test_realtime_e2e.py›
    • llm-translation
    • realtime
    • websocket
    • azure
    • azure/gpt-realtime
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a realtime deployment that calls azure/gpt-realtime
    3. Open a /v1/realtime websocket session to azure-realtime-refused-e0a7b29722eb
    View test on GitHub ↗
  415. ✓test_url_context_metadata_is_returned TestGeminiUrlContexttest_gemini_url_context_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • gemini
    • gemini/gemini-3.8-flash
    1. Add a deployment named e2e-gemini-url-context-61a1f1263a5a that calls gemini/gemini-3.8-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  416. ✓test_verbose_json_returns_word_timestamps TestWhisperTranscriptionFormatstest_audio_transcriptions_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/whisper-1
    1. Add a deployment named e2e-whisper-verbose-1ccfbc1ee8bd that calls openai/whisper-1
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload weather_question_24k.wav to /v1/audio/transcriptions
    View test on GitHub ↗
  417. ✓test_vertex_chat_returns_content TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    1. Add a deployment named e2e-vertex-chat-d0f4409ca59f that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-vertex-chat-d0f4409ca59f
    View test on GitHub ↗
  418. ✓test_vertex_chat_returns_tool_call TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    • function_calling
    1. Add a deployment named e2e-vertex-tool-cc3ae3711716 that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-vertex-tool-cc3ae3711716
    View test on GitHub ↗
  419. ✓test_vertex_chat_streams_real_content TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    1. Add a deployment named e2e-vertex-stream-ffa6c93a21f1 that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-vertex-stream-ffa6c93a21f1
    View test on GitHub ↗
  420. ✓test_vertex_chat_vision_describes_image TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    • vision
    1. Add a deployment named e2e-vertex-vision-478a17aeef91 that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-vertex-vision-478a17aeef91
    View test on GitHub ↗
  421. ✓test_vertex_embeddings_honor_requested_dimensions TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • vertex_ai
    • vertex_ai/text-embedding-005
    1. Add a deployment named e2e-embeddings-vertex-dims-bb40d58d36d6 that calls vertex_ai/text-embedding-005
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  422. ✓test_vertex_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • vertex_ai
    • vertex_ai/text-embedding-005
    1. Add a deployment named e2e-embeddings-vertex-1f68737783e9 that calls vertex_ai/text-embedding-005
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  423. ✓test_vertex_multimodal_embeddings_honor_dimensions_and_are_costed TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    • llm-translation
    • embeddings
    • nonstream
    • vertex_ai
    • vertex_ai/multimodalembedding@001
    1. Add a deployment named e2e-embeddings-vertex-mm-e3136a5cc2c8 that calls vertex_ai/multimodalembedding@001
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  424. ✓test_vertex_passthrough_via_managed_model_logs_cost TestVertexPassthroughSpendTrackingtest_vertex_passthrough_e2e.py›
    • spend-budgets
    • passthrough
    • nonstream
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. POST /model/new
    3. Wait for the last control-plane write to reach every proxy replica
    4. Send a Vertex AI generateContent request to gemini-2.5-flash in us-central1 through /vertex_ai
    5. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  425. ✓test_vertex_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    • llm-translation
    • chat_completions
    • nonstream
    • vertex_ai
    • vertex_ai/gemini-2.5-flash
    • prompt_caching
    1. Add a deployment named e2e-vertex-cache-f14e9f236fee that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. POST /chat/completions
    4. Wait for the request's spend log in /spend/logs
    5. List the deployments from /model/info
    6. POST /chat/completions
    View test on GitHub ↗
  426. ✓test_vtt_format_returns_webvtt_transcript TestWhisperTranscriptionFormatstest_audio_transcriptions_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/whisper-1
    1. Add a deployment named e2e-whisper-vtt-ddb245b3db99 that calls openai/whisper-1
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload weather_question_24k.wav to /v1/audio/transcriptions
    View test on GitHub ↗
  427. ✓test_web_fetch_tool_result_is_surfaced TestAnthropicCompletiontest_anthropic_completion_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5-5
    • function_calling
    1. Add a deployment named e2e-anthropic-web-fetch-b19a2913f16f that calls anthropic/claude-sonnet-5-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  428. ✓test_web_search_reports_server_tool_use TestAnthropicCompletiontest_anthropic_completion_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5-5
    • function_calling
    1. Add a deployment named e2e-anthropic-web-search-c0b7b06e409f that calls anthropic/claude-sonnet-5-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  429. −test_bedrock_invoke_messages_returns_reasoning TestReasoningEfforttest_reasoning_effort_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-opus-4-6-v1
    • reasoning

    No steps recorded

    bedrock_invoke_messages requires provider credentials: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY

    View test on GitHub ↗
  430. −test_chat_completion_returns_reasoning[azure-ai] TestReasoningEfforttest_reasoning_effort_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • azure_ai
    • bedrock
    • vertex_ai
    • anthropic/claude-fable-5-1
    • azure_ai/claude-fable-5-1
    • bedrock/converse/us.anthropic.claude-fable-5-1
    • bedrock/invoke/us.anthropic.claude-opus-4-6-v1
    • vertex_ai/claude-fable-5-1
    • reasoning

    No steps recorded

    azure_ai requires provider credentials: AZURE_AI_API_BASE, AZURE_AI_API_KEY

    View test on GitHub ↗
  431. −test_chat_completion_returns_reasoning[bedrock-converse] TestReasoningEfforttest_reasoning_effort_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • azure_ai
    • bedrock
    • vertex_ai
    • anthropic/claude-fable-5-1
    • azure_ai/claude-fable-5-1
    • bedrock/converse/us.anthropic.claude-fable-5-1
    • bedrock/invoke/us.anthropic.claude-opus-4-6-v1
    • vertex_ai/claude-fable-5-1
    • reasoning

    No steps recorded

    bedrock_converse requires provider credentials: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY

    View test on GitHub ↗
  432. −test_chat_completion_returns_reasoning[bedrock-invoke-chat] TestReasoningEfforttest_reasoning_effort_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • azure_ai
    • bedrock
    • vertex_ai
    • anthropic/claude-fable-5-1
    • azure_ai/claude-fable-5-1
    • bedrock/converse/us.anthropic.claude-fable-5-1
    • bedrock/invoke/us.anthropic.claude-opus-4-6-v1
    • vertex_ai/claude-fable-5-1
    • reasoning

    No steps recorded

    bedrock_invoke_chat requires provider credentials: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY

    View test on GitHub ↗
  433. −test_chat_completion_returns_reasoning[vertex-ai] TestReasoningEfforttest_reasoning_effort_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • azure_ai
    • bedrock
    • vertex_ai
    • anthropic/claude-fable-5-1
    • azure_ai/claude-fable-5-1
    • bedrock/converse/us.anthropic.claude-fable-5-1
    • bedrock/invoke/us.anthropic.claude-opus-4-6-v1
    • vertex_ai/claude-fable-5-1
    • reasoning

    No steps recorded

    vertex_ai requires provider credentials: VERTEXAI_LOCATION

    View test on GitHub ↗
  434. −test_empty_input_returns_error TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-tts

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on empty input instead of surfacing the provider 4xx

    View test on GitHub ↗
  435. −test_flagged_model_keeps_prompt_cache_across_system_reminder TestBedrockInvokeMidConversationSystemtest_messages_mid_conversation_system_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-sonnet-5
    • mid_conversation_system
    • prompt_caching

    No steps recorded

    LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

    View test on GitHub ↗
  436. −test_flagged_model_keeps_prompt_cache_across_system_reminder TestAzureFoundryMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • azure_ai/claude-opus-4-8
    • mid_conversation_system
    • prompt_caching

    No steps recorded

    LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

    View test on GitHub ↗
  437. −test_flagged_model_keeps_prompt_cache_across_system_reminder TestVertexMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • vertex_ai/claude-opus-4-8
    • mid_conversation_system
    • prompt_caching

    No steps recorded

    LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

    View test on GitHub ↗
  438. −test_gemini_passthrough_returns_the_same_header_contract_as_the_managed_routetest_passthrough_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • gemini
    • gemini-2.5-flash

    No steps recorded

    stage red: product gap, native passthrough returns no x-litellm-response-cost or x-ratelimit-* headers

    View test on GitHub ↗
  439. −test_hosted_vllm_chat_returns_content TestHostedVllmChattest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • hosted_vllm

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  440. −test_invalid_response_id_returns_error TestResponsesRetrievetest_responses_retrieve_e2e.py›
    • llm-translation
    • responses

    No steps recorded

    stage red: product gap (LIT-5447), retrieving an unknown response id returns 400 (model=None) instead of 404

    View test on GitHub ↗
  441. −test_invalid_voice_returns_error TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-tts

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on invalid voice instead of surfacing the provider 4xx

    View test on GitHub ↗
  442. −test_mcp_tool_use_by_spec[anthropic] TestAnthropicCompletiontest_anthropic_completion_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5-5
    • function_calling

    No steps recorded

    ZAPIER_CI_CD_MCP_TOKEN is not available; Shared vault has no Zapier MCP credential

    View test on GitHub ↗
  443. −test_mcp_tool_use_by_spec[openai] TestAnthropicCompletiontest_anthropic_completion_e2e.py›
    • llm-translation
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5-5
    • function_calling

    No steps recorded

    ZAPIER_CI_CD_MCP_TOKEN is not available; Shared vault has no Zapier MCP credential

    View test on GitHub ↗
  444. −test_mcp_tool_use_via_responses TestAnthropicCompletiontest_anthropic_completion_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5-5
    • function_calling

    No steps recorded

    ZAPIER_CI_CD_MCP_TOKEN is not available; Shared vault has no Zapier MCP credential

    View test on GitHub ↗
  445. −test_missing_document_returns_error TestRustOcrGatewaytest_ocr_rust_e2e.py›
    • llm-translation
    • ocr

    No steps recorded

    stage red: product gap, /v1/ocr 500s (aocr TypeError) on missing document instead of 400

    View test on GitHub ↗
  446. −test_missing_input_returns_error TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio
    • nonstream
    • openai
    • openai/gpt-4o-mini-tts

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on missing input instead of 400

    View test on GitHub ↗
  447. −test_missing_input_returns_error TestModerationstest_moderations_e2e.py›
    • llm-translation
    • moderations

    No steps recorded

    stage red: product gap, /v1/moderations 500s (KeyError 'input') on missing input instead of 400

    View test on GitHub ↗
  448. −test_missing_input_returns_error TestResponsestest_responses_e2e.py›
    • llm-translation
    • responses
    • openai
    • openai/gpt-4o-mini

    No steps recorded

    stage red: product gap, /v1/responses 500s (aresponses TypeError) on missing input instead of 400

    View test on GitHub ↗
  449. −test_missing_max_tokens_returns_error TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages

    No steps recorded

    stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing max_tokens instead of 400

    View test on GitHub ↗
  450. −test_missing_messages_returns_error TestAnthropicMessagestest_messages_e2e.py›
    • llm-translation
    • messages

    No steps recorded

    stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing messages instead of 400

    View test on GitHub ↗
  451. −test_missing_model_returns_error TestAudioSpeechtest_audio_speech_e2e.py›
    • llm-translation
    • audio

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on missing model instead of 400

    View test on GitHub ↗
  452. −test_missing_prompt_returns_error TestImageGenerationtest_image_generation_e2e.py›
    • llm-translation
    • images

    No steps recorded

    stage red: product gap, /v1/images/generations 500s (aimage_generation TypeError) on missing prompt instead of 400

    View test on GitHub ↗
  453. −test_retrieve_invalid_id_returns_error TestVectorStorestest_vector_stores_e2e.py›
    • llm-translation
    • vector_stores
    • openai

    No steps recorded

    stage red: product gap, retrieving a nonexistent vector store returns 2xx with an error envelope in the body instead of 404

    View test on GitHub ↗
  454. −test_search_missing_query_returns_error TestVectorStorestest_vector_stores_e2e.py›
    • llm-translation
    • vector_stores
    • openai

    No steps recorded

    stage red: product gap, vector store search 500s (asearch TypeError) on missing query instead of 400

    View test on GitHub ↗
  455. −test_static_and_x_pass_headers_reach_upstream TestPassthroughHeaderstest_passthrough_headers_e2e.py›
    • passthrough
    • passthrough
    • nonstream
    • anthropic
    • claude-haiku-4-5-20251001

    No steps recorded

    stage red: ALB sends custom passthrough paths to backend pods, which lack ANTHROPIC_API_KEY, so the os.environ header forwards unresolved and Anthropic 401s; routing user-defined paths to the gateway needs a product/infra decision (fixed path namespace or dynamic ingress)

    View test on GitHub ↗
  456. −test_store_and_retrieve_by_id TestResponsesRetrievetest_responses_retrieve_e2e.py›
    • llm-translation
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini

    No steps recorded

    stage red: product gap (LIT-5446), retrieve returns a different id than the stored response (non-idempotent response-id re-encryption)

    View test on GitHub ↗
  457. −test_vertex_partner_model_streams_openai_shaped_chunks[vertex_ai/mistral-small-2503] TestVertexPartnerChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • vertex_ai
    • vertex_ai/mistral-small-2503

    No steps recorded

    the e2e Vertex project has no access to mistral-small-2503 (404 publisher model not found)

    View test on GitHub ↗
  458. −test_vertex_partner_model_streams_openai_shaped_chunks[vertex_ai/openai/gpt-oss-120b-maas] TestVertexPartnerChatCompletionstest_chat_completions_regression_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • vertex_ai
    • vertex_ai/openai/gpt-oss-120b-maas

    No steps recorded

    never served by the e2e Vertex project (60s read timeout, no headers)

    View test on GitHub ↗
  459. −test_web_search_server_tool_is_served TestBedrockWebSearchServerTooltest_bedrock_web_search_server_tool_e2e.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • web_search

    No steps recorded

    stage red: environment gap, the e2e stack neither enables the websearch_interception callback nor declares a search backend, so the request reaches bedrock's transformation and takes its by-design 400. Unskip once the ephemeral stack ships the config in this module's docstring.

    View test on GitHub ↗
  460. ✓test_access_group_info_reports_membership TestModelAccessGroupRoutestest_model_tag_accessgroup_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-agmodel-da31fdb7875b that calls openai/gpt-5.5
    2. POST /access_group/new
    3. GET /access_group/e2e-mgmt-ag-93f7d29165a4/info
    View test on GitHub ↗
  461. ✓test_actor_subject_and_database_role[internal_user_viewer] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an actor in the internal_user_viewer role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the internal_user_viewer role
    3. Read the user back from /user/info
    4. GET /user/info
    View test on GitHub ↗
  462. ✓test_actor_subject_and_database_role[internal_user] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an actor in the internal_user role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the internal_user role
    3. Read the user back from /user/info
    4. GET /user/info
    View test on GitHub ↗
  463. ✓test_actor_subject_and_database_role[organization_admin] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the organization_admin role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the organization_admin role
    4. Read the user back from /user/info
    5. GET /user/info
    6. Read the team back from /team/info
    7. Read the organization back from /organization/info
    View test on GitHub ↗
  464. ✓test_actor_subject_and_database_role[proxy_admin_viewer] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an actor in the proxy_admin_viewer role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the proxy_admin_viewer role
    3. Read the user back from /user/info
    4. GET /user/info
    View test on GitHub ↗
  465. ✓test_actor_subject_and_database_role[proxy_admin] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an actor in the proxy_admin role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the proxy_admin role
    3. Read the user back from /user/info
    4. GET /user/info
    View test on GitHub ↗
  466. ✓test_actor_subject_and_database_role[team_admin] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the team_admin role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the team_admin role
    4. Read the user back from /user/info
    5. GET /user/info
    6. Read the team back from /team/info
    7. Read the organization back from /organization/info
    View test on GitHub ↗
  467. ✓test_actor_subject_and_database_role[team_member] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the team_member role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the team_member role
    4. Read the user back from /user/info
    5. GET /user/info
    6. Read the team back from /team/info
    7. Read the organization back from /organization/info
    View test on GitHub ↗
  468. ✓test_actor_subject_and_database_role[unrelated_user] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • user_management
    1. Create an actor in the unrelated_user role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the unrelated_user role
    3. Read the user back from /user/info
    4. GET /user/info
    View test on GitHub ↗
  469. ✓test_admin_creates_reads_updates_clears_and_deletes_a_key[direct_jwt] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • key_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the proxy_admin role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the proxy_admin role
    4. Read the user back from /user/info
    5. Generate a virtual key with models: gpt-5.5, team id: 42c79851-91ac-455e-9be5-b6bb88921679 and key alias: e2e-jwt-key-5bc4357402f7
    6. Read the key's settings back from /key/info
    7. Update the virtual key's settings with /key/update
    8. Read the key's settings back from /key/info
    9. Update the virtual key's settings with /key/update
    10. Read the key's settings back from /key/info
    11. List the keys with the alias e2e-jwt-key-5bc4357402f7-updated from /key/list
    12. Delete the virtual key
    13. List the keys with the alias e2e-jwt-key-5bc4357402f7-updated from /key/list
    View test on GitHub ↗
  470. ✓test_admin_creates_reads_updates_clears_and_deletes_a_key[virtual_key] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • key_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the proxy_admin role, with its identity provider user, internal user and any tenant memberships
    3. Generate a virtual key as the proxy admin
    4. Read the user back from /user/info
    5. Generate a virtual key with models: gpt-5.5, team id: 714ee8ac-0950-4845-b67c-7eb013ce533b and key alias: e2e-jwt-key-6d6c7d821500
    6. Read the key's settings back from /key/info
    7. Update the virtual key's settings with /key/update
    8. Read the key's settings back from /key/info
    9. Update the virtual key's settings with /key/update
    10. Read the key's settings back from /key/info
    11. List the keys with the alias e2e-jwt-key-6d6c7d821500-updated from /key/list
    12. Delete the virtual key
    13. List the keys with the alias e2e-jwt-key-6d6c7d821500-updated from /key/list
    View test on GitHub ↗
  471. ✓test_admin_viewer_reads_but_cannot_update TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • key_management
    1. Create an actor in the proxy_admin_viewer role, with its identity provider user, internal user and any tenant memberships
    2. Get a JWT from the identity provider for the actor in the proxy_admin_viewer role
    3. Generate a virtual key as the proxy admin
    4. Update the virtual key's settings with /key/update
    5. Read the key's settings back from /key/info
    6. Update the virtual key's settings with /key/update
    7. Read the key's settings back from /key/info
    View test on GitHub ↗
  472. ✓test_approve_activates_submission_and_persists TestMcpServerSubmissiontest_config_misc_endpoints_e2e.py›
    • mcp
    • mcp
    1. Create a team with team alias: e2e-mcp-team-bbd523b4e72f
    2. Generate a virtual key with team id: 845403a6-566a-45f0-b57e-e735ad7cc3f3
    3. POST /v1/mcp/server/register
    4. PUT /v1/mcp/server/9d87710a-d0e6-4abe-b708-a8fe8137723e/approve
    5. GET /v1/mcp/server/9d87710a-d0e6-4abe-b708-a8fe8137723e
    View test on GitHub ↗
  473. ✓test_bedrock_mantle_responses_connection_succeeds TestModelTestConnectiontest_model_test_connection_e2e.py›
    • management
    • health
    • nonstream
    • bedrock_mantle
    • bedrock_mantle/openai.gpt-5.6-luna
    1. Run Test Connection on bedrock_mantle/openai.gpt-5.6-luna in responses mode with /health/test_connection
    View test on GitHub ↗
  474. ✓test_block_persists_to_key_info TestKeyRoutestest_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with models: gemini-2.5-flash
    2. Read the key's settings back from /key/info
    3. Block the virtual key
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  475. ✓test_block_then_unblock_persists_to_model_info TestModelRoutestest_model_tag_accessgroup_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-model-block-c23f58a8b2b2 that calls openai/gpt-5.5
    2. GET /model/info
    3. POST /model/block
    4. GET /model/info
    5. POST /model/unblock
    6. GET /model/info
    View test on GitHub ↗
  476. ✓test_block_then_unblock_persists_to_team_info TestTeamManagementRoutestest_team_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-team-block-c1436803ff45 and models: gemini-2.5-flash
    2. GET /team/info
    3. POST /team/block
    4. GET /team/info
    5. POST /team/unblock
    6. GET /team/info
    View test on GitHub ↗
  477. ✓test_bulk_update_applies_max_budget_to_target_key TestKeyManagementRoutestest_key_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with models: gpt-5.5 and max budget: 5.0
    2. Read the key's settings back from /key/info
    3. POST /key/bulk_update
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  478. ✓test_callbacks_list_reports_active_logging_callbacks TestInventoryRoutestest_config_misc_endpoints_e2e.py›
    • observability
    1. GET /callbacks/list
    View test on GitHub ↗
  479. ✓test_clearing_the_description_with_null_reads_back_null TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_900db0224d11
    2. Update the MCP server's settings with PUT /v1/mcp/server
    3. Read /v1/mcp/server/36e2369c-762c-4b26-a465-356832bb2df7 on every proxy replica until it settles
    View test on GitHub ↗
  480. ✓test_clearing_the_description_with_null_reads_back_null TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_1f2f6990ed62
    2. Create an MCP toolset with the tools server id: 1cbcd980-dba2-413f-ad09-c0a6f3205bb9 and tool name: search_datadog_logs, server id: 1cbcd980-dba2-413f-ad09-c0a6f3205bb9 and tool name: get_datadog_metr
    3. Update an MCP toolset with toolset id: 6ee093df-49cd-4df5-8ed4-f832c4866bb3
    4. Read /v1/mcp/toolset/6ee093df-49cd-4df5-8ed4-f832c4866bb3 on every proxy replica until it settles
    View test on GitHub ↗
  481. ✓test_config_update_persists_router_setting_to_get TestRouterSettingstest_config_misc_endpoints_e2e.py›
    • routing
    • proxy_config
    1. GET /router/settings
    2. POST /config/update
    3. GET /router/settings
    4. POST /config/update
    5. GET /router/settings
    View test on GitHub ↗
  482. ✓test_create_echoes_every_field_written TestKeyLifecycletest_key_lifecycle_e2e.py›
    • management
    • key_management
    1. Add a deployment named e2e-key-lifecycle-f3982975b816 that calls gpt-4o-mini
    2. Create a team with team alias: e2e-key-lifecycle-team-d03e756b0b00
    3. Generate a virtual key with models: e2e-key-lifecycle-f3982975b816, max budget: 25.0, budget duration: 30d, team id: 86a8fcaf-0fea-48bf-ad3a-02bb2c7e71c0, key alias: e2e-key-lifecycle-d03e756b0b00, tp
    View test on GitHub ↗
  483. ✓test_create_persists_and_is_read_back TestFallbackManagementtest_config_misc_endpoints_e2e.py›
    • routing
    1. GET /router/settings
    2. Add a deployment named e2e-fallback-primary-a40112c2ad6b that calls openai/gpt-5.5
    3. Add a deployment named e2e-fallback-secondary-6081f92f3b14 that calls openai/gpt-5.5
    4. POST /fallback
    5. GET /fallback/e2e-fallback-primary-a40112c2ad6b
    6. DELETE /fallback/e2e-fallback-primary-a40112c2ad6b
    7. GET /fallback/e2e-fallback-primary-a40112c2ad6b
    View test on GitHub ↗
  484. ✓test_create_persists_both_tools_under_the_exact_names_written TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_ea90b24155bd
    2. Create an MCP toolset with the tools server id: a2c6443c-65f4-4bc0-a48c-986d9ebbaf15 and tool name: search_datadog_logs, server id: a2c6443c-65f4-4bc0-a48c-986d9ebbaf15 and tool name: get_datadog_metr
    3. Read /v1/mcp/toolset/86c3cd29-ee79-44d4-b51d-b1902f63dfcf on every proxy replica until it settles
    4. Read /v1/mcp/toolset on every proxy replica until it settles
    View test on GitHub ↗
  485. ✓test_create_persists_every_field_on_every_replica TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_58626003e026
    2. Read /v1/mcp/server/afc96406-fef7-4266-a686-00d38c11ced1 on every proxy replica until it settles
    View test on GitHub ↗
  486. ✓test_created_budget_appears_in_budget_list TestBudgetManagementtest_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. POST /budget/new
    2. GET /budget/list
    View test on GitHub ↗
  487. ✓test_created_key_appears_in_key_list_inventory TestKeyRoutestest_management_e2e.py›
    • management
    • key_management
    1. Count the keys with the alias e2e-mgmt-keylist-62ce9ab4850a in /key/list
    2. Generate a virtual key with key alias: e2e-mgmt-keylist-62ce9ab4850a
    3. Count the keys with the alias e2e-mgmt-keylist-62ce9ab4850a in /key/list
    View test on GitHub ↗
  488. ✓test_created_team_appears_in_team_list TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-mgmt-team-9475f0a8b448 and models: gemini-2.5-flash
    2. List the teams from /team/list
    View test on GitHub ↗
  489. ✓test_created_users_appear_in_user_list TestUserRoutestest_management_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-mgmt-8e6ccd6ee33d@example.com and user role: internal_user
    2. Create an internal user with user email: e2e-mgmt-7956b7b9ca84@example.com and user role: internal_user
    3. List the matching users from /user/list
    View test on GitHub ↗
  490. ✓test_creating_a_key_from_the_dashboard_persists_and_works TestDashboardKeyRoutestest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Sign in to the Admin UI with /v2/login
    2. Read the key's settings back from /key/info
    3. Generate a virtual key with models: gemini-2.5-flash, key alias: e2e-mgmt-uicreate-82ed5e503fa5 and tpm limit: 100
    4. Read the key's settings back from /key/info
    5. List the keys with the alias e2e-mgmt-uicreate-82ed5e503fa5 from /key/list
    6. Send a /chat/completions request to gemini-2.5-flash
    7. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  491. ✓test_credential_migration_check_reports_residual_scan TestInventoryRoutestest_config_misc_endpoints_e2e.py›
    • deploy-ops
    1. GET /credentials/migrate-encryption/check
    View test on GitHub ↗
  492. ✓test_customer_create_persists_to_info TestCustomertest_management_e2e.py›
    • management
    • customer_management
    1. Create the end user e2e-customer-a56e7d0ed4c0
    2. Read the end user e2e-customer-a56e7d0ed4c0 back from /customer/info
    View test on GitHub ↗
  493. ✓test_delete_forbidden_for_non_admin_key TestKeyManagementRoutestest_key_management_e2e.py›
    • proxy-auth
    • key_management
    1. Generate a virtual key with models: gpt-5.5
    2. Send a request to /key/delete
    3. Read the key's settings back from /key/info
    View test on GitHub ↗
  494. ✓test_delete_persists_and_revokes_team_bound_key TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    • nonstream
    • openai
    • gpt-5.5
    1. Create a team with team alias: e2e-mgmt-team-3477caa62220 and models: gpt-5.5
    2. Generate a virtual key with team id: 62759056-0b5e-45dc-ae7c-26d017738e89
    3. Send a /chat/completions request to gpt-5.5
    4. Delete the team
    5. Check whether /team/info finds the team
    6. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  495. ✓test_delete_removes_from_model_info_catalog TestModelRoutestest_management_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-model-558af7a47c33 that calls openai/gpt-5.5
    2. List the deployments from /model/info
    3. Delete the deployment
    4. List the deployments from /model/info
    View test on GitHub ↗
  496. ✓test_delete_removes_from_organization_info TestOrganizationRoutestest_management_e2e.py›
    • management
    • organization_management
    1. Create an organization with organization alias: e2e-mgmt-org-add42cda5f67
    2. Check whether /organization/info finds the organization
    3. Delete the organization
    4. Check whether /organization/info finds the organization
    View test on GitHub ↗
  497. ✓test_delete_removes_the_customer TestCustomerManagementtest_budget_customer_user_org_e2e.py›
    • management
    • customer_management
    1. POST /customer/new
    2. GET /customer/info
    3. POST /customer/delete
    4. Probe /customer/info
    View test on GitHub ↗
  498. ✓test_delete_removes_the_server_from_every_replica TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_a3e4df0821c3
    2. Delete the MCP server
    3. Check that /v1/mcp/server/8544c4b8-800d-4457-9b12-520c3965ab17 returns 404 on every proxy replica
    4. Read /v1/mcp/server on every proxy replica until it settles
    View test on GitHub ↗
  499. ✓test_delete_removes_the_toolset_from_every_replica TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_cd3d2539e389
    2. Create an MCP toolset with the tools server id: d8616b35-c580-4146-8da1-b974de2079b4 and tool name: search_datadog_logs, server id: d8616b35-c580-4146-8da1-b974de2079b4 and tool name: get_datadog_metr
    3. Delete the MCP toolset
    4. Check that /v1/mcp/toolset/167fcfe2-6d45-4ff6-b31d-9eb32feda4cd returns 404 on every proxy replica
    5. Read /v1/mcp/toolset on every proxy replica until it settles
    View test on GitHub ↗
  500. ✓test_delete_removes_the_user_from_inventory TestUserRoutestest_management_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-mgmt-85ca442baf08@example.com and user role: internal_user
    2. Count the matching users in /user/list
    3. Delete the internal user
    4. Count the matching users in /user/list
    View test on GitHub ↗
  501. ✓test_delete_revokes_info_and_chat_on_every_replica TestKeyLifecycletest_key_lifecycle_e2e.py›
    • management
    • key_management
    • nonstream
    • openai
    • gpt-4o-mini
    1. Create a team with team alias: e2e-key-lifecycle-team-b77fdbe64b6f
    2. Generate a virtual key with models: e2e-key-lifecycle-f3982975b816, max budget: 25.0, budget duration: 30d, team id: 3369bc98-eff9-457b-a6e5-6c7e32fbc831, key alias: e2e-key-lifecycle-b77fdbe64b6f, tp
    3. Send a request to /chat/completions
    4. Delete the virtual key
    5. Read /key/info on every proxy replica until they all agree
    6. Send a request to /chat/completions
    View test on GitHub ↗
  502. ✓test_delete_revokes_the_key_on_chat TestKeyRoutestest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with models: gemini-2.5-flash
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Delete the virtual key
    4. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  503. ✓test_editing_a_key_from_the_dashboard_persists_and_is_enforced TestDashboardKeyRoutestest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • gemini
    • openai
    • gemini-2.5-flash
    • gpt-5.5
    1. Generate a virtual key with models: gemini-2.5-flash, key alias: e2e-mgmt-uiedit-aa85805dfefe, tpm limit: 100 and rpm limit: 200
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Send a /chat/completions request to gpt-5.5
    4. Sign in to the Admin UI with /v2/login
    5. Update the virtual key's settings with /key/update
    6. Read the key's settings back from /key/info
    7. Send a /chat/completions request to gpt-5.5
    8. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  504. ✓test_end_user_new_persists_to_end_user_info TestCustomerManagementtest_budget_customer_user_org_e2e.py›
    • management
    • customer_management
    1. POST /end_user/new
    2. GET /end_user/info
    View test on GitHub ↗
  505. ✓test_estimate_computes_cost_from_token_counts TestCostEstimatetest_config_misc_endpoints_e2e.py›
    • cost-map
    1. POST /cost/estimate
    View test on GitHub ↗
  506. ✓test_explicit_null_clears_the_budget_and_its_reset_time TestKeyLifecycletest_key_lifecycle_e2e.py›
    • management
    • key_management
    1. Create a team with team alias: e2e-key-lifecycle-team-fbda1043f8a7
    2. Generate a virtual key with models: e2e-key-lifecycle-f3982975b816, max budget: 25.0, budget duration: 30d, team id: cbe363cb-6734-4cae-99c4-f58601244bbf, key alias: e2e-key-lifecycle-fbda1043f8a7, tp
    3. Read /key/info on every proxy replica until they all agree
    4. Update the virtual key's settings with /key/update
    5. Read /key/info on every proxy replica until they all agree
    View test on GitHub ↗
  507. ✓test_gdpr_check_derives_verdict_from_the_request TestComplianceRoutestest_config_misc_endpoints_e2e.py›
    • management
    1. POST /compliance/gdpr
    View test on GitHub ↗
  508. ✓test_generate_forbidden_for_non_admin_key TestKeyManagementRoutestest_key_management_e2e.py›
    • proxy-auth
    • key_management
    1. Generate a virtual key with models: gpt-5.5
    2. Send a request to /key/generate
    View test on GitHub ↗
  509. ✓test_generate_persists_to_key_info_and_scopes_chat TestKeyRoutestest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with models: gemini-2.5-flash, key alias: e2e-mgmt-key-253822075076, tpm limit: 424242 and rpm limit: 424243
    2. Read the key's settings back from /key/info
    3. Send a /chat/completions request to gemini-2.5-flash
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  510. ✓test_health_reports_the_calling_key_healthy TestKeyManagementRoutestest_key_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with models: gpt-5.5
    2. POST /key/health
    View test on GitHub ↗
  511. ✓test_info_reflects_the_fields_the_key_was_created_with TestKeyManagementRoutestest_key_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with models: gpt-5.5, gemini-2.5-flash, key alias: e2e-mgmt-keyinfo-a1d20b6484f2, tpm limit: 131313 and rpm limit: 141414
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  512. ✓test_info_returns_created_team_fields TestTeamManagementRoutestest_team_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-team-info-1ef4da155a78 and models: gemini-2.5-flash
    2. GET /team/info
    View test on GitHub ↗
  513. ✓test_is_null_finds_the_budget_left_uncapped TestBudgetListV1test_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. POST /budget/new
    2. GET /management/v1/budgets
    View test on GitHub ↗
  514. ✓test_is_refused_for_a_non_admin_key TestBudgetListV1test_budget_customer_user_org_e2e.py›
    • proxy-auth
    • budget_management
    1. Generate a virtual key with default settings
    2. GET /management/v1/budgets
    View test on GitHub ↗
  515. ✓test_key_delete_by_alias_writes_audit_row TestKeyDeletionAuditLogtest_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with key alias: e2e-audit-657940d5665a
    2. Delete the virtual key with the alias e2e-audit-657940d5665a
    3. Read the key's settings back from /key/info
    4. Read the key's deletion entries from the /audit log
    View test on GitHub ↗
  516. ✓test_key_delete_by_key_writes_audit_row TestKeyDeletionAuditLogtest_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with key alias: e2e-audit-7c2eb42bcdd7
    2. Delete the virtual key
    3. Read the key's settings back from /key/info
    4. Read the key's deletion entries from the /audit log
    View test on GitHub ↗
  517. ✓test_key_serves_its_model_and_is_denied_others TestKeyLifecycletest_key_lifecycle_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-4o-mini
    1. Create a team with team alias: e2e-key-lifecycle-team-4038715c8b23
    2. Generate a virtual key with models: e2e-key-lifecycle-f3982975b816, max budget: 25.0, budget duration: 30d, team id: 7f340b06-6134-437a-903a-992d3d338ec7, key alias: e2e-key-lifecycle-4038715c8b23, tp
    3. Send a request to /chat/completions
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  518. ✓test_llm_only_key_forbidden_from_management_writes TestManagementRoutePermissionstest_management_e2e.py›
    • proxy-auth
    1. Generate a virtual key limited to the LLM API routes
    2. Try to generate a virtual key with key alias: e2e-mgmt-forbidden-key-1bab6a4c2a02, calling with a virtual key
    3. Try to create a team with team alias: e2e-mgmt-forbidden-team-1bab6a4c2a02 and team id: e2e-mgmt-forbidden-team-1bab6a4c2a02, calling with a virtual key
    4. Try to create an internal user with user email: e2e-mgmt-forbidden-user-1bab6a4c2a02@example.com, user role: internal_user and user id: e2e-mgmt-forbidden-user-1bab6a4c2a02, calling with a virtual key
    5. Count the keys with the alias e2e-mgmt-forbidden-key-1bab6a4c2a02 in /key/list
    6. Check whether /team/info finds the team
    7. Count the matching users in /user/list
    View test on GitHub ↗
  519. ✓test_member_add_and_delete_persist_to_team_info TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    1. Create an internal user with user email: e2e-mgmt-7fa5a6cf9a7f@example.com and user role: internal_user
    2. Create a team with team alias: e2e-mgmt-team-74ee903c71f9 and models: gemini-2.5-flash
    3. Add a user to the team with /team/member_add
    4. Read the team back from /team/info
    5. Remove a user from the team with /team/member_delete
    6. Read the team back from /team/info
    View test on GitHub ↗
  520. ✓test_member_add_forbidden_to_plain_member TestTeamManagementRoutestest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-fb-d52ffea43749@example.com and user role: internal_user
    2. Create an internal user with user email: e2e-team-fb-6c364e309a19@example.com and user role: internal_user
    3. Create a team with team alias: e2e-team-fb-583295525a23 and models: gemini-2.5-flash
    4. Add a user to the team with /team/member_add
    5. Generate a virtual key with user id: cdb6f989-67dd-407d-ae14-f44343362da5 and team id: c1f26775-d340-4650-9ae7-1bba7ef5f6eb
    6. Send a request to /team/member_add
    View test on GitHub ↗
  521. ✓test_member_add_records_membership TestOrganizationMembershiptest_budget_customer_user_org_e2e.py›
    • management
    • organization_management
    1. Create an organization with organization alias: e2e-mgmt-org-bd6f29759e00
    2. Create an internal user with user email: e2e-mgmt-c88dedbb3c8d@example.com and user role: internal_user
    3. POST /organization/member_add
    4. GET /organization/info
    View test on GitHub ↗
  522. ✓test_member_cannot_write_and_another_team_cannot_read_the_key TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • key_management
    1. Create the Keycloak group e2e-jwt-team-8db69f8cfcf9 and the user e2e-jwt-user-8db69f8cfcf9 in it
    2. POST /user/new
    3. Create a team with team alias: e2e-jwt-8db69f8cfcf9 and team id: e2e-jwt-team-8db69f8cfcf9
    4. Get a Keycloak access token for e2e-jwt-user-8db69f8cfcf9 from the client litellm-e2e-admin
    5. Get a Keycloak access token for e2e-jwt-user-8db69f8cfcf9 from the client litellm-e2e-tests
    6. Generate a virtual key with team id: e2e-jwt-team-8db69f8cfcf9 and key alias: e2e-jwt-owned-6316ad28f7cc
    7. Add a user to the team with /team/member_add
    8. Read the key's settings back from /key/info
    9. Update the virtual key's settings with /key/update
    10. Read the key's settings back from /key/info
    11. Create the Keycloak group e2e-jwt-team-780c50157ebd and the user e2e-jwt-user-780c50157ebd in it
    12. Create an internal user with user email: e2e-jwt-user-780c50157ebd@example.com, user role: internal_user and user id: dc16f043-3a47-41d7-ad63-3632a0501986
    13. Create a team with team alias: 780c50157ebd and team id: e2e-jwt-team-780c50157ebd
    14. Add a user to the team with /team/member_add
    15. Get a Keycloak access token for e2e-jwt-user-780c50157ebd from the client litellm-e2e-tests
    16. Read the key's settings back from /key/info
    View test on GitHub ↗
  523. ✓test_member_delete_forbidden_to_plain_member TestTeamManagementRoutestest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-fb-c14fa9a61799@example.com and user role: internal_user
    2. Create an internal user with user email: e2e-team-fb-6b8872b9468d@example.com and user role: internal_user
    3. Create a team with team alias: e2e-team-fb-ed2840c0ee39 and models: gemini-2.5-flash
    4. Add a user to the team with /team/member_add
    5. Generate a virtual key with user id: 06cfe9ba-2ad8-4d69-b409-d7391a439ead and team id: faee0b2c-479e-4bf8-a3c6-dc2e1a089be8
    6. Send a request to /team/member_delete
    View test on GitHub ↗
  524. ✓test_member_delete_persists_to_team_info TestTeamManagementRoutestest_team_management_e2e.py›
    • management
    • team_management
    1. Create an internal user with user email: e2e-team-md-cb79007bd2ba@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-md-e83d848ff3a8 and models: gemini-2.5-flash
    3. Add a user to the team with /team/member_add
    4. GET /team/info
    5. Remove a user from the team with /team/member_delete
    6. GET /team/info
    View test on GitHub ↗
  525. ✓test_member_update_persists_role_and_budget TestTeamManagementRoutestest_team_management_e2e.py›
    • management
    • team_management
    1. Create an internal user with user email: e2e-team-mu-ec34206e5ddc@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-mu-63539b157501 and models: gemini-2.5-flash
    3. Add a user to the team with /team/member_add
    4. GET /team/info
    5. POST /team/member_update
    6. GET /team/info
    View test on GitHub ↗
  526. ✓test_multi_group_actor_keeps_exact_memberships TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the team_member role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the team_member role
    4. Read the team back from /team/info
    View test on GitHub ↗
  527. ✓test_new_access_group_tags_the_deployment TestModelAccessGroupRoutestest_model_tag_accessgroup_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-agmodel-f44a181d5a00 that calls openai/gpt-5.5
    2. POST /access_group/new
    3. GET /access_group/e2e-mgmt-ag-a0e886bcb469/info
    View test on GitHub ↗
  528. ✓test_new_is_denied_to_non_admin_keys TestTeamManagementRoutestest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Generate a virtual key with default settings
    2. Create an internal user with user email: e2e-team-adm-634b3794cf69@example.com and user role: internal_user
    3. Generate a virtual key with user id: 64b403e0-ce1d-46e7-bd57-d7a58bf56bc6
    4. Try to create a team with team alias: e2e-team-adm-7439e8cfed17, calling with a virtual key
    5. Try to create a team with team alias: e2e-team-adm-5273d61ad429, calling with a virtual key
    View test on GitHub ↗
  529. ✓test_new_is_refused_for_a_non_admin_key TestBudgetManagementtest_budget_customer_user_org_e2e.py›
    • proxy-auth
    • budget_management
    1. Generate a virtual key with default settings
    2. Send a request to /budget/new
    View test on GitHub ↗
  530. ✓test_new_persists_mapping_and_is_read_back TestJwtKeyMappingtest_config_misc_endpoints_e2e.py›
    • proxy-auth
    1. Generate a virtual key with default settings
    2. POST /jwt/key/mapping/new
    3. GET /jwt/key/mapping/info
    View test on GitHub ↗
  531. ✓test_new_persists_to_customer_info TestCustomerManagementtest_budget_customer_user_org_e2e.py›
    • management
    • customer_management
    1. POST /customer/new
    2. GET /customer/info
    View test on GitHub ↗
  532. ✓test_new_persists_to_model_info_catalog TestModelRoutestest_management_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-model-cf52481b84fc that calls openai/gpt-5.5
    2. List the deployments from /model/info
    View test on GitHub ↗
  533. ✓test_new_persists_to_organization_info TestOrganizationRoutestest_management_e2e.py›
    • management
    • organization_management
    1. Create an organization with organization alias: e2e-mgmt-org-2818109fade6 and models: gemini-2.5-flash
    2. Read the organization back from /organization/info
    View test on GitHub ↗
  534. ✓test_new_persists_to_tag_list TestTagRoutestest_management_e2e.py›
    • management
    • tag_management
    1. List the tags from /tag/list
    2. Create a tag with name: e2e-mgmt-tag-f67b46bee2ec and description: Tag for spend categorization
    3. List the tags from /tag/list
    View test on GitHub ↗
  535. ✓test_new_persists_to_team_info_and_binds_keys TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-mgmt-team-a3511d40d2a4 and models: gemini-2.5-flash
    2. Read the team back from /team/info
    3. Generate a virtual key with team id: 68510a9a-16cb-4469-8ec5-8191ece0fe8e
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  536. ✓test_new_persists_to_user_info TestUserRoutestest_management_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-mgmt-3e576af969c4@example.com and user role: internal_user
    2. Read the user back from /user/info
    View test on GitHub ↗
  537. ✓test_new_user_is_readable_via_user_info TestUserManagementtest_budget_customer_user_org_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-mgmt-3c91810fe64c@example.com and user role: internal_user
    2. Read the user back from /user/info
    View test on GitHub ↗
  538. ✓test_non_admin_key_cannot_add_global_model TestModelRoutestest_model_tag_accessgroup_e2e.py›
    • proxy-auth
    • model_management
    1. Generate a virtual key with default settings
    2. Send a request to /model/new
    3. List the deployments from /model/info
    View test on GitHub ↗
  539. ✓test_oidc_browser_profile_identity_mapping TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    1. Create an actor in the internal_user role, with its identity provider user, internal user and any tenant memberships
    2. Read Keycloak's OpenID Connect discovery document
    3. Register a browser SSO client in Keycloak
    4. Get a Keycloak access token for e2e-jwt-user-f8efd4a1041a through the browser SSO client
    5. Read the signed-in user's profile from Keycloak's userinfo endpoint
    View test on GitHub ↗
  540. ✓test_partial_setup_removes_previously_created_identities[group] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    1. Create the Keycloak group e2e-partial-72cc1b5a46e9
    2. Delete every resource the test created, newest first
    3. Check that Keycloak's admin API returns 404 for the deleted entry under /groups
    View test on GitHub ↗
  541. ✓test_partial_setup_removes_previously_created_identities[user] TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    1. Create the Keycloak group e2e-partial-bede33573650
    2. Create the Keycloak user e2e-jwt-user-bede33573650 with a password of its own
    3. Create the Keycloak user e2e-jwt-user-bede33573650
    4. Delete every resource the test created, newest first
    5. Check that Keycloak's admin API returns 404 for the deleted entry under /groups
    6. Check that Keycloak's admin API returns 404 for the deleted entry under /users
    View test on GitHub ↗
  542. ✓test_partial_update_changes_only_the_named_field TestKeyLifecycletest_key_lifecycle_e2e.py›
    • management
    • key_management
    1. Create a team with team alias: e2e-key-lifecycle-team-ff35c902648d
    2. Generate a virtual key with models: e2e-key-lifecycle-f3982975b816, max budget: 25.0, budget duration: 30d, team id: f426ed9d-3fe3-4876-9940-fa729d5fa65c, key alias: e2e-key-lifecycle-ff35c902648d, tp
    3. Read /key/info on every proxy replica until they all agree
    4. Update the virtual key's settings with /key/update
    5. Read /key/info on every proxy replica until they all agree
    View test on GitHub ↗
  543. ✓test_project_detachment_preserves_key_scope_and_refreshes_auth TestKeyManagementRoutestest_key_management_e2e.py›
    • management
    • key_management
    • nonstream
    • openai
    • openai/synthetic-detachment
    1. Add a deployment named e2e-detach-fde6d9af83f0 that calls openai/synthetic-detachment
    2. Create an organization with organization alias: e2e-detach-fde6d9af83f0 and models: e2e-detach-fde6d9af83f0
    3. Create a team with team alias: e2e-detach-fde6d9af83f0, models: e2e-detach-fde6d9af83f0 and organization id: 621da401-b585-4e4c-baab-7e3aea6fccd2
    4. POST /project/new
    5. Generate a virtual key with models: e2e-detach-fde6d9af83f0, max budget: 5.0, team id: a2283f37-615f-4fb2-98c8-221a0f2f1df1, project id: ab94f913-7fe7-4c6f-9ead-0e3dc219ad3f, organization id: 621da401
    6. Send a /chat/completions request to e2e-detach-fde6d9af83f0
    7. Update the virtual key's settings with /key/update
    8. Read the key's settings back from /key/info
    9. Update the virtual key's settings with /key/update
    10. Send a request to /key/update
    11. Read the key's settings back from /key/info
    12. POST /project/update
    13. Send a /chat/completions request to e2e-detach-fde6d9af83f0
    14. POST /key/update
    15. Read the key's settings back from /key/info
    16. Send a /chat/completions request to e2e-detach-fde6d9af83f0
    17. Update the virtual key's settings with /key/update
    18. Read the key's settings back from /key/info
    19. Send a /chat/completions request to e2e-detach-fde6d9af83f0-outside
    View test on GitHub ↗
  544. ✓test_read_reflects_the_create_on_every_replica TestKeyLifecycletest_key_lifecycle_e2e.py›
    • management
    • key_management
    1. Create a team with team alias: e2e-key-lifecycle-team-4a916e1a7d65
    2. Generate a virtual key with models: e2e-key-lifecycle-f3982975b816, max budget: 25.0, budget duration: 30d, team id: 5a7ac60e-a36f-4601-b413-e3359bfadbd9, key alias: e2e-key-lifecycle-4a916e1a7d65, tp
    3. Read /key/info on every proxy replica until they all agree
    View test on GitHub ↗
  545. ✓test_refuses_a_sort_field_and_a_parameter_it_does_not_support TestBudgetListV1test_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. GET /management/v1/budgets
    View test on GitHub ↗
  546. ✓test_regenerate_rotates_to_a_working_new_key TestKeyRegenerationtest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with models: gpt-5.5
    2. Regenerate the virtual key with /key/regenerate
    3. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  547. ✓test_regenerate_with_grace_period_keeps_old_key_until_revoked TestKeyRegenerationtest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with models: gpt-5.5
    2. Regenerate the virtual key with /key/regenerate
    3. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  548. ✓test_register_submits_pending_server TestMcpServerSubmissiontest_config_misc_endpoints_e2e.py›
    • mcp
    • mcp
    1. Create a team with team alias: e2e-mcp-team-74a15c390cc9
    2. Generate a virtual key with team id: 33cb9e74-06df-4397-9d90-a7ca3874dd92
    3. POST /v1/mcp/server/register
    View test on GitHub ↗
  549. ✓test_reset_spend_zeroes_recorded_spend_and_lifts_the_budget_block TestKeyManagementRoutestest_key_management_e2e.py›
    • spend-budgets
    • key_management
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and max budget: 3e-06
    2. Send a /chat/completions request to claude-haiku-4-5
    3. Read the key's settings back from /key/info
    4. Reset the virtual key's spend to 0.0
    5. Read the key's settings back from /key/info
    6. Send a /chat/completions request to claude-haiku-4-5
    View test on GitHub ↗
  550. ✓test_sorts_pages_and_filters_the_budgets_it_created TestBudgetListV1test_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. POST /budget/new
    2. GET /management/v1/budgets
    View test on GitHub ↗
  551. ✓test_successful_actor_cleanup_removes_owned_state TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the team_member role, with its identity provider user, internal user and any tenant memberships
    3. Generate a virtual key as the proxy admin
    4. Read the key's settings back from /key/info
    5. Delete every resource the test created, newest first
    6. Count the matching users in /user/list
    7. Count the keys with the alias e2e-actor-key-3873819fd49f in /key/list
    8. Check whether /team/info finds the team
    9. Check whether /organization/info finds the organization
    10. Check that Keycloak's admin API returns 404 for the deleted entry under /users
    11. Check that Keycloak's admin API returns 404 for the deleted entry under /groups
    View test on GitHub ↗
  552. ✓test_tag_delete_removes_from_list TestTagRoutestest_model_tag_accessgroup_e2e.py›
    • management
    • tag_management
    1. POST /tag/new
    2. GET /tag/list
    3. POST /tag/delete
    4. GET /tag/list
    View test on GitHub ↗
  553. ✓test_tag_list_reports_created_tag TestTagRoutestest_model_tag_accessgroup_e2e.py›
    • management
    • tag_management
    1. GET /tag/list
    2. POST /tag/new
    3. GET /tag/list
    View test on GitHub ↗
  554. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[budget_duration] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-38ce86473422@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-2f1675d2e2c5, tpm limit: 1000 and members with roles: role: admin and user id: c5402486-3ddd-43d4-8404-e9cf126ddbe6
    3. Generate a virtual key with user id: c5402486-3ddd-43d4-8404-e9cf126ddbe6
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  555. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[max_budget] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-6b3bf3f77399@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-9e3bc44562c5, tpm limit: 1000 and members with roles: role: admin and user id: cf056b17-f75a-40be-9428-2392eb2a4502
    3. Generate a virtual key with user id: cf056b17-f75a-40be-9428-2392eb2a4502
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  556. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[metadata] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-be2f4a58bdf1@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-f62c697c114d, tpm limit: 1000 and members with roles: role: admin and user id: e5fa5e3f-1d9a-43cd-b5a8-036f105591cb
    3. Generate a virtual key with user id: e5fa5e3f-1d9a-43cd-b5a8-036f105591cb
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  557. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[models] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-3092e395c762@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-2cc6f17bdb09, tpm limit: 1000 and members with roles: role: admin and user id: 285bc42f-fbf3-457a-82bf-097847b513ea
    3. Generate a virtual key with user id: 285bc42f-fbf3-457a-82bf-097847b513ea
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  558. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[rpm_limit] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-826aa4e79548@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-125894c74ebd, tpm limit: 1000 and members with roles: role: admin and user id: 4dda27a2-9ab5-4a08-92c8-a880abb51020
    3. Generate a virtual key with user id: 4dda27a2-9ab5-4a08-92c8-a880abb51020
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  559. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[team_alias] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-8c37b4fad0c4@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-fb7c3b91cd71, tpm limit: 1000 and members with roles: role: admin and user id: 9b6d0d20-57c0-4f64-86be-90d70095c41f
    3. Generate a virtual key with user id: 9b6d0d20-57c0-4f64-86be-90d70095c41f
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  560. ✓test_team_admin_cannot_change_any_team_setting TestTeamAdminWithNoEditableFieldstest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. GET /get/ui_settings
    2. PATCH /update/ui_settings
    3. Wait for the last control-plane write to reach every proxy replica
    4. Create an internal user with user email: e2e-team-admin-9f706f751428@example.com and user role: internal_user
    5. Create a team with team alias: e2e-team-admin-fd2e2f1aba14, tpm limit: 1000 and members with roles: role: admin and user id: 99ee3dbe-97cb-42dd-84e0-c63418ce2ec1
    6. Generate a virtual key with user id: 99ee3dbe-97cb-42dd-84e0-c63418ce2ec1
    7. GET /team/info
    8. Send a request to /team/update
    9. GET /team/info
    View test on GitHub ↗
  561. ✓test_team_admin_cannot_raise_an_org_team_budget_under_the_org_cap TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an organization with organization alias: e2e-team-admin-org-3e56edbb6c7a and max budget: 100.0
    2. Create an internal user with user email: e2e-team-admin-7463e6c6e224@example.com and user role: internal_user
    3. Create a team with team alias: e2e-team-admin-201d98851156, organization id: 69a99169-1e89-4958-86c5-e1c329cd1969, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: 50
    4. Generate a virtual key with user id: 50146d96-88bf-474d-8c2c-f42dc247fef7
    5. GET /team/info
    6. Send a request to /team/update
    7. GET /team/info
    View test on GitHub ↗
  562. ✓test_team_admin_cannot_raise_or_remove_the_budget[raise] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-4c96a695f310@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-6d46ca1f4bb3, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: cd3c644d-2eb6-4ddc-9416-b1381f439daa
    3. Generate a virtual key with user id: cd3c644d-2eb6-4ddc-9416-b1381f439daa
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  563. ✓test_team_admin_cannot_raise_or_remove_the_budget[remove] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-f3422a4f889e@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-2985c927ff80, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: d56779d7-5038-4d1f-932c-b9d4b9c5b3cf
    3. Generate a virtual key with user id: d56779d7-5038-4d1f-932c-b9d4b9c5b3cf
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  564. ✓test_team_admin_raises_a_standalone_team_budget TestTeamAdminWithRaiseMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. GET /get/ui_settings
    2. PATCH /update/ui_settings
    3. Wait for the last control-plane write to reach every proxy replica
    4. Create an internal user with user email: e2e-team-admin-9d4e5576a242@example.com and user role: internal_user
    5. Create a team with team alias: e2e-team-admin-749ed14f1cb1, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: 37afefaf-c670-4a81-b97b-cb1b58a9e4c6
    6. Generate a virtual key with user id: 37afefaf-c670-4a81-b97b-cb1b58a9e4c6
    7. GET /team/info
    8. Send a request to /team/update
    9. GET /team/info
    View test on GitHub ↗
  565. ✓test_team_admin_raises_an_org_team_budget_only_up_to_the_org_cap[above-it] TestTeamAdminWithRaiseMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an organization with organization alias: e2e-team-admin-org-91737e784775 and max budget: 100.0
    2. Create an internal user with user email: e2e-team-admin-ecfcaae33a43@example.com and user role: internal_user
    3. Create a team with team alias: e2e-team-admin-fbb7989d3201, organization id: 4392c19a-a7d8-40c3-9527-3d83dc9cd1cb, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: 49
    4. Generate a virtual key with user id: 4944805d-1a8a-4ba9-9001-59aeba35e77f
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  566. ✓test_team_admin_raises_an_org_team_budget_only_up_to_the_org_cap[up-to-the-org-cap] TestTeamAdminWithRaiseMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an organization with organization alias: e2e-team-admin-org-492687bfe0b6 and max budget: 100.0
    2. Create an internal user with user email: e2e-team-admin-720106e6eb88@example.com and user role: internal_user
    3. Create a team with team alias: e2e-team-admin-6ffd5c78390c, organization id: 8b6904e7-a51c-4333-8f18-6349cd7b0c49, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: 56
    4. Generate a virtual key with user id: 56214939-80c2-401b-9071-bf42b3089db0
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  567. ✓test_team_admin_resending_the_budget_settings_keeps_the_next_budget_reset TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • spend-budgets
    • team_management
    1. Create an internal user with user email: e2e-team-admin-c3eec99580a6@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-2ac4b8e4419b, tpm limit: 1000 and members with roles: role: admin and user id: ee1d8e43-3767-4132-91bb-e810065441ff
    3. Generate a virtual key with user id: ee1d8e43-3767-4132-91bb-e810065441ff
    4. POST /team/update
    5. GET /team/info
    6. Send a request to /team/update
    7. GET /team/info
    View test on GitHub ↗
  568. ✓test_team_admin_saves_a_new_rpm_limit_and_a_tighter_budget[first-budget] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-a077ab850f42@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-0d2aa16a8423, tpm limit: 1000 and members with roles: role: admin and user id: 08f67d4b-84a2-4993-bbe7-462e1b37e416
    3. Generate a virtual key with user id: 08f67d4b-84a2-4993-bbe7-462e1b37e416
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  569. ✓test_team_admin_saves_a_new_rpm_limit_and_a_tighter_budget[lower] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. GET /get/ui_settings
    2. PATCH /update/ui_settings
    3. Wait for the last control-plane write to reach every proxy replica
    4. Create an internal user with user email: e2e-team-admin-abbc2bddf9fa@example.com and user role: internal_user
    5. Create a team with team alias: e2e-team-admin-c208da570ba7, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: fa10c53f-2eb2-4f29-82ad-a68dbde22ad8
    6. Generate a virtual key with user id: fa10c53f-2eb2-4f29-82ad-a68dbde22ad8
    7. GET /team/info
    8. Send a request to /team/update
    9. GET /team/info
    View test on GitHub ↗
  570. ✓test_team_admin_saves_the_settings_form_with_a_new_tpm_limit TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. GET /get/ui_settings
    2. PATCH /update/ui_settings
    3. Wait for the last control-plane write to reach every proxy replica
    4. Create an internal user with user email: e2e-team-admin-9ae25f04e4ac@example.com and user role: internal_user
    5. Create a team with team alias: e2e-team-admin-18ad7bf90e81, tpm limit: 1000 and members with roles: role: admin and user id: aadddb41-8a1c-4bea-a7b7-fb8cff56ebba
    6. Generate a virtual key with user id: aadddb41-8a1c-4bea-a7b7-fb8cff56ebba
    7. GET /team/info
    8. Send a request to /team/update
    9. GET /team/info
    View test on GitHub ↗
  571. ✓test_team_admin_still_cannot_remove_the_budget TestTeamAdminWithRaiseMaxBudgetEnabledtest_team_management_e2e.py›
    • proxy-auth
    • team_management
    1. Create an internal user with user email: e2e-team-admin-f4007806a0ff@example.com and user role: internal_user
    2. Create a team with team alias: e2e-team-admin-3b5ff4f25e80, tpm limit: 1000, max budget: 10.0 and members with roles: role: admin and user id: 1ec512fb-6528-4b63-acb6-bf44fd54f5bb
    3. Generate a virtual key with user id: 1ec512fb-6528-4b63-acb6-bf44fd54f5bb
    4. GET /team/info
    5. Send a request to /team/update
    6. GET /team/info
    View test on GitHub ↗
  572. ✓test_team_delete_succeeds_for_team_larger_than_db_pool TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    1. (205 earlier steps not recorded)
    2. Create an internal user with user email: e2e-mgmt-bulk-204-3163c8d5ec02@example.com and user role: internal_user
    3. Create an internal user with user email: e2e-mgmt-bulk-205-def67cfdb1be@example.com and user role: internal_user
    4. Create an internal user with user email: e2e-mgmt-bulk-206-c4d8170f3262@example.com and user role: internal_user
    5. Create an internal user with user email: e2e-mgmt-bulk-207-2fbe2340716e@example.com and user role: internal_user
    6. Create an internal user with user email: e2e-mgmt-bulk-208-7b3f2e31d597@example.com and user role: internal_user
    7. Create an internal user with user email: e2e-mgmt-bulk-209-559b2638cb72@example.com and user role: internal_user
    8. Create an internal user with user email: e2e-mgmt-bulk-210-16e557089d43@example.com and user role: internal_user
    9. Create an internal user with user email: e2e-mgmt-bulk-211-47120fe1c85a@example.com and user role: internal_user
    10. Create an internal user with user email: e2e-mgmt-bulk-212-ac9224978e8f@example.com and user role: internal_user
    11. Create an internal user with user email: e2e-mgmt-bulk-213-0ea3c0b8232a@example.com and user role: internal_user
    12. Create an internal user with user email: e2e-mgmt-bulk-214-4e2dcbba4ed4@example.com and user role: internal_user
    13. Create an internal user with user email: e2e-mgmt-bulk-215-510c677aa75c@example.com and user role: internal_user
    14. Create an internal user with user email: e2e-mgmt-bulk-216-c665cf2478fa@example.com and user role: internal_user
    15. Create an internal user with user email: e2e-mgmt-bulk-217-a2ad4b330cd6@example.com and user role: internal_user
    16. Create an internal user with user email: e2e-mgmt-bulk-218-ad18f0a545d3@example.com and user role: internal_user
    17. Create an internal user with user email: e2e-mgmt-bulk-219-dae112311a34@example.com and user role: internal_user
    18. Create an internal user with user email: e2e-mgmt-bulk-220-0ec8aa762a62@example.com and user role: internal_user
    19. Create an internal user with user email: e2e-mgmt-bulk-221-ee512c5d533c@example.com and user role: internal_user
    20. Create an internal user with user email: e2e-mgmt-bulk-222-16e119fbea15@example.com and user role: internal_user
    21. Create an internal user with user email: e2e-mgmt-bulk-223-6be4e12209e7@example.com and user role: internal_user
    22. Create an internal user with user email: e2e-mgmt-bulk-224-df21b8a11ed9@example.com and user role: internal_user
    23. Create an internal user with user email: e2e-mgmt-bulk-225-55ad6a45bdde@example.com and user role: internal_user
    24. Create an internal user with user email: e2e-mgmt-bulk-226-9b1e2a60aac7@example.com and user role: internal_user
    25. Create an internal user with user email: e2e-mgmt-bulk-227-43c9f0bb68e7@example.com and user role: internal_user
    26. Create an internal user with user email: e2e-mgmt-bulk-228-44e8315a29e7@example.com and user role: internal_user
    27. Create an internal user with user email: e2e-mgmt-bulk-229-627f0682cdd8@example.com and user role: internal_user
    28. Create an internal user with user email: e2e-mgmt-bulk-230-13c434df3c3f@example.com and user role: internal_user
    29. Create an internal user with user email: e2e-mgmt-bulk-231-de0f74372af3@example.com and user role: internal_user
    30. Create an internal user with user email: e2e-mgmt-bulk-232-6620bf889cd0@example.com and user role: internal_user
    31. Create an internal user with user email: e2e-mgmt-bulk-233-dafb891f801e@example.com and user role: internal_user
    32. Create an internal user with user email: e2e-mgmt-bulk-234-2ab939e90c59@example.com and user role: internal_user
    33. Create an internal user with user email: e2e-mgmt-bulk-235-9587a15e0e93@example.com and user role: internal_user
    34. Create an internal user with user email: e2e-mgmt-bulk-236-a4edaa525b07@example.com and user role: internal_user
    35. Create an internal user with user email: e2e-mgmt-bulk-237-4235bc8529c1@example.com and user role: internal_user
    36. Create an internal user with user email: e2e-mgmt-bulk-238-0d3e191e98ad@example.com and user role: internal_user
    37. Create an internal user with user email: e2e-mgmt-bulk-239-91562bc8d34b@example.com and user role: internal_user
    38. Create an internal user with user email: e2e-mgmt-bulk-240-7c509d01407a@example.com and user role: internal_user
    39. Create an internal user with user email: e2e-mgmt-bulk-241-121ba702efc9@example.com and user role: internal_user
    40. Create an internal user with user email: e2e-mgmt-bulk-242-089d632b1bb8@example.com and user role: internal_user
    41. Create an internal user with user email: e2e-mgmt-bulk-243-a51b5dcf247a@example.com and user role: internal_user
    42. Create an internal user with user email: e2e-mgmt-bulk-244-d951c39e2a3b@example.com and user role: internal_user
    43. Create an internal user with user email: e2e-mgmt-bulk-245-1c59657f684d@example.com and user role: internal_user
    44. Create an internal user with user email: e2e-mgmt-bulk-246-ffd646a59018@example.com and user role: internal_user
    45. Create an internal user with user email: e2e-mgmt-bulk-247-e25e8c76cb43@example.com and user role: internal_user
    46. Create an internal user with user email: e2e-mgmt-bulk-248-b3b0f900449a@example.com and user role: internal_user
    47. Create an internal user with user email: e2e-mgmt-bulk-249-6b71b5a056b6@example.com and user role: internal_user
    48. Add a roster of members to the team with /team/member_add
    49. Read the team back from /team/info
    50. Try to delete the team with /team/delete
    51. Check whether /team/info finds the team
    View test on GitHub ↗
  573. ✓test_team_delete_writes_audit_row_for_team_keys TestKeyDeletionAuditLogtest_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-audit-team-71ccbd1f2b56
    2. Generate a virtual key with team id: 5808b2ea-60df-49ec-a4ed-8c61cdab4bd3
    3. Delete the team
    4. Read the key's settings back from /key/info
    5. Read the key's deletion entries from the /audit log
    View test on GitHub ↗
  574. ✓test_team_member_delete_writes_audit_row_for_member_keys TestKeyDeletionAuditLogtest_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-audit-team-2e04e6332d5e
    2. Create an internal user with user email: e2e-audit-afde9f291e59@example.com and user role: internal_user
    3. Add a user to the team with /team/member_add
    4. Generate a virtual key with user id: 20f4f406-37e8-411d-8fe4-ba2162df9a0c and team id: cef4e877-00ac-4447-8452-54b0df21bab1
    5. Remove a user from the team with /team/member_delete
    6. Read the key's settings back from /key/info
    7. Read the key's deletion entries from the /audit log
    View test on GitHub ↗
  575. ✓test_tool_list_returns_catalog_with_consistent_total TestInventoryRoutestest_config_misc_endpoints_e2e.py›
    • management
    1. GET /v1/tool/list
    View test on GitHub ↗
  576. ✓test_two_actor_sets_keep_tenants_and_keys_isolated TestJwtManagementtest_jwt_management_e2e.py›
    • proxy-auth
    • key_management
    1. Create an organization, a team in it and a matching identity provider group
    2. Create an actor in the team_member role, with its identity provider user, internal user and any tenant memberships
    3. Get a JWT from the identity provider for the actor in the team_member role
    4. Generate a virtual key as the proxy admin
    5. Read the key's settings back from /key/info
    View test on GitHub ↗
  577. ✓test_unblock_flips_key_info_blocked_back TestKeyManagementRoutestest_key_management_e2e.py›
    • management
    • key_management
    1. Generate a virtual key with models: gpt-5.5
    2. POST /key/block
    3. Read the key's settings back from /key/info
    4. POST /key/unblock
    5. Read the key's settings back from /key/info
    View test on GitHub ↗
  578. ✓test_update_accepts_per_model_budgets_including_punctuated_names TestBudgetManagementtest_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. POST /budget/new
    2. POST /budget/update
    3. POST /budget/info
    4. POST /budget/new
    5. POST /budget/update
    6. POST /budget/info
    View test on GitHub ↗
  579. ✓test_update_alias_persists_to_organization_info TestOrganizationRoutestest_management_e2e.py›
    • management
    • organization_management
    1. Create an organization with organization alias: e2e-mgmt-org-9411fc2b144b
    2. Update an organization with organization id: 33e206dc-053c-4e76-851e-6a24652cb594 and organization alias: e2e-mgmt-org-048107053451
    3. Read the organization back from /organization/info
    View test on GitHub ↗
  580. ✓test_update_forbidden_for_non_admin_key TestKeyManagementRoutestest_key_management_e2e.py›
    • proxy-auth
    • key_management
    1. Generate a virtual key with models: gpt-5.5
    2. Send a request to /key/update
    3. Read the key's settings back from /key/info
    View test on GitHub ↗
  581. ✓test_update_max_budget_persists_to_budget_info TestBudgetManagementtest_budget_customer_user_org_e2e.py›
    • spend-budgets
    • budget_management
    1. POST /budget/new
    2. POST /budget/info
    3. POST /budget/update
    4. POST /budget/info
    View test on GitHub ↗
  582. ✓test_update_models_persists_and_flips_enforcement TestKeyRoutestest_management_e2e.py›
    • management
    • key_management
    • nonstream
    • gemini
    • openai
    • gemini-2.5-flash
    • gpt-5.5
    1. Generate a virtual key with models: gemini-2.5-flash
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Send a /chat/completions request to gpt-5.5
    4. Set the virtual key's models to [gpt-5.5]
    5. Read the key's settings back from /key/info
    6. Send a /chat/completions request to gpt-5.5
    7. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  583. ✓test_update_persists_input_cost_to_model_info TestModelRoutestest_management_e2e.py›
    • management
    • model_management
    1. Add a deployment named e2e-mgmt-model-67be9d9f341b that calls gpt-4o-mini
    2. List the deployments from /model/info
    3. Update a deployment's settings to model: gpt-4o-mini and input cost per token: 2.22e-06
    4. List the deployments from /model/info
    View test on GitHub ↗
  584. ✓test_update_persists_to_team_info TestTeamRoutestest_management_e2e.py›
    • management
    • team_management
    1. Create a team with team alias: e2e-mgmt-team-aba65daa4e3d and models: gemini-2.5-flash
    2. Update a team with team id: fff23e31-93a6-4d49-97e3-98b058649968 and team alias: e2e-mgmt-team-updated-19653b501662
    3. Read the team back from /team/info
    View test on GitHub ↗
  585. ✓test_update_persists_to_user_info TestUserRoutestest_management_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-mgmt-2b6fb3bbf480@example.com and user role: internal_user
    2. Read the user back from /user/info
    3. Update an internal user with user id: 160ec6c3-0f31-4937-8065-b58ad87acb9d and user role: internal_user_viewer
    4. Read the user back from /user/info
    View test on GitHub ↗
  586. ✓test_updating_only_the_alias_keeps_every_other_field_on_every_replica TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_44a4e0eebd8b
    2. Update the MCP server's settings with PUT /v1/mcp/server
    3. Read /v1/mcp/server/984d7a74-0b96-4fbe-a763-a4d98056342f on every proxy replica until it settles
    View test on GitHub ↗
  587. ✓test_updating_only_the_description_keeps_the_tools_and_name TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_b488b24b9fc4
    2. Create an MCP toolset with the tools server id: 0ca5e307-6e4d-46bc-aed8-3b64f1fbddc5 and tool name: search_datadog_logs, server id: 0ca5e307-6e4d-46bc-aed8-3b64f1fbddc5 and tool name: get_datadog_metr
    3. Update an MCP toolset with toolset id: fdf12ea6-6a55-4d22-bf46-e75e71404d9b and description: edited
    4. Read /v1/mcp/toolset/fdf12ea6-6a55-4d22-bf46-e75e71404d9b on every proxy replica until it settles
    View test on GitHub ↗
  588. ✓test_updating_the_tools_to_one_entry_reads_back_exactly_that_entry TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp
    1. Create an MCP server named e2e_mcp_lifecycle_74117414fd79
    2. Create an MCP toolset with the tools server id: 3409da74-0d6a-49d5-a854-65cc534dc903 and tool name: search_datadog_logs, server id: 3409da74-0d6a-49d5-a854-65cc534dc903 and tool name: get_datadog_metr
    3. Update an MCP toolset with toolset id: a4d54cfd-3291-4166-8dbc-5cc79a93d840 and tools: server id: 3409da74-0d6a-49d5-a854-65cc534dc903 and tool name: search_datadog_logs
    4. Read /v1/mcp/toolset/a4d54cfd-3291-4166-8dbc-5cc79a93d840 on every proxy replica until it settles
    View test on GitHub ↗
  589. ✓test_user_delete_writes_audit_row_for_user_keys TestKeyDeletionAuditLogtest_management_e2e.py›
    • management
    • user_management
    1. Create an internal user with user email: e2e-audit-90595c153726@example.com and user role: internal_user
    2. Generate a virtual key with user id: eeda7a6f-f9e8-4234-b779-9353192ee704
    3. Delete the internal user
    4. Read the key's settings back from /key/info
    5. Read the key's deletion entries from the /audit log
    View test on GitHub ↗
  590. ✓test_workflow_runs_list_returns_consistent_count TestInventoryRoutestest_config_misc_endpoints_e2e.py›
    • management
    1. GET /v1/workflows/runs
    View test on GitHub ↗
  591. ✓test_write_to_config_owned_fallbacks_is_refused TestFallbackManagementtest_config_misc_endpoints_e2e.py›
    • routing
    1. GET /router/settings
    2. Add a deployment named e2e-fallback-primary-740ea3b903b5 that calls openai/gpt-5.5
    3. Add a deployment named e2e-fallback-secondary-cdbc59a14b56 that calls openai/gpt-5.5
    4. POST /fallback
    5. GET /router/settings
    6. GET /fallback/e2e-fallback-primary-740ea3b903b5
    7. GET /fallback/gemini-2.5-flash
    View test on GitHub ↗
  592. −test_created_server_is_listed_with_every_field TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    • mcp
    • mcp

    No steps recorded

    product gap: GET /v1/mcp/server builds each row from the in-memory registry, whose _build_mcp_server_table sets description from mcp_info['description'], so the list reports the mcp_info description while GET /v1/mcp/server/{server_id} reports the stored description column. A server created with both set to different text reads back with two different descriptions depending on the route

    View test on GitHub ↗
  593. ✓test_a_drained_group_does_not_block_a_different_group TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-mag-free-5851981b25e7 and user id: e2e-test-user
    2. Send a /chat/completions request to e2e-mag-unbudgeted-5851981b25e7 with the prompt "hi 32588c906ebc"
    View test on GitHub ↗
  594. ✓test_a_key_that_spent_nothing_is_blocked_by_the_shared_pool TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-mag-budget-5851981b25e7 and user id: e2e-test-user
    2. Send a /chat/completions request to e2e-mag-budgeted-5851981b25e7 with the prompt "hi 059421961efa"
    View test on GitHub ↗
  595. ✓test_alias_shares_rpm_bucket_with_model_group TestModelGroupAliasRateLimitratelimit/test_model_group_alias_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • e2e-alias-rl-alias
    • e2e-alias-rl-target
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word 38e963a0ee6f"
    3. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word 7a395a89c63a"
    4. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word 15c029a5abfe"
    5. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word 2833c4295d0d"
    View test on GitHub ↗
  596. ✓test_bare_key_blocks_over_its_own_budget TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 623aedca217d"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend aea92ad2f7a5"
    View test on GitHub ↗
  597. ✓test_bare_key_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 6d9f77a6bf9f"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 73e437599f2c"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 52fc8bb04ff5"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 7cd124fbfa46"
    View test on GitHub ↗
  598. ✓test_bedrock_guardrail_cost_reaches_persisted_spend_row TestPricingConfigSpendspend_tracking/test_pricing_config_spend_e2e.py›
    • spend-budgets
    • spend_reporting
    • nonstream
    • bedrock
    • openai
    • openai/gpt-6.1-sol
    1. Generate a virtual key with user id: e2e-test-user
    2. GET /config/cost_discount_config
    3. GET /config/cost_margin_config
    4. PATCH /config/cost_discount_config
    5. PATCH /config/cost_margin_config
    6. Wait for the last control-plane write to reach every proxy replica
    7. POST /guardrails
    8. Wait for the last control-plane write to reach every proxy replica
    9. Add a deployment named guardrail-priced-6c00c879fc87 that calls openai/gpt-6.1-sol
    10. Send a request to /chat/completions
    11. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  599. ✓test_blocked_member_does_not_block_peer TestTeamMemberBudgetIsolationbudgets/test_team_member_budget_isolation_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Create the team e2e-member-iso-90e1a0273bb7 and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Generate a virtual key with user id: 86c302b4-588b-4f03-9947-34d6d09f4ed1 and team id: d681ce2e-4d79-4141-871d-1055d1999e36
    5. Generate a virtual key with user id: 8e5f153e-fe0d-41f3-9920-dbb44a340b7e and team id: d681ce2e-4d79-4141-871d-1055d1999e36
    6. Send a /chat/completions request to gpt-5.5 with the prompt "tight 01456845d392"
    7. Send a /chat/completions request to gpt-5.5 with the prompt "tight 406fd405f7bb"
    8. Send a /chat/completions request to gpt-5.5
    9. Send a /chat/completions request to gpt-5.5 with the prompt "tight ad42423cea0b"
    10. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  600. ✓test_budget_crud_roundtripbudgets/test_budget_crud_e2e.py›
    • spend-budgets
    • budget_management
    1. Create a budget with /budget/new
    2. Read the budget from /budget/info
    3. Generate a virtual key with budget id: 96eb4b70-e161-44fd-b844-ecd438ab1107
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  601. ✓test_budget_delete_removes_itbudgets/test_budget_crud_e2e.py›
    • spend-budgets
    • budget_management
    1. Create a budget with /budget/new
    2. Delete the budget
    3. Read the budget from /budget/info
    View test on GitHub ↗
  602. ✓test_budget_duration_schedules_reset_on_keybudgets/test_budget_crud_e2e.py›
    • spend-budgets
    • key_management
    1. Generate a virtual key with max budget: 10.0 and budget duration: 30d
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  603. ✓test_budget_fallback_reroutes_anthropic_messages_to_openaibudgets/test_budget_fallback_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • anthropic
    • openai
    • claude-haiku-4-5
    • gpt-5.5
    1. Generate a virtual key with model max budget: claude-haiku-4-5: budget limit: 1e-06 and time period: 30d and budget fallbacks: claude-haiku-4-5: gpt-5.5
    2. Send a /v1/messages request to claude-haiku-4-5 with the prompt "hi 3d83a333e257"
    3. Send a /v1/messages request to claude-haiku-4-5 with the prompt "hi 48949b5f845b"
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  604. ✓test_burst_of_concurrent_calls_loses_no_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.6-luna
    1. Add a priced deployment, then create two teams with one key each and send 7 /chat/completions requests per key, 6 of them at once
    2. Send a /chat/completions request to e2e-reconciliation-5da6a1bfdf1c
    3. Check that the /spend/logs rows of team d524be9e-3967-457a-a7e1-e51cd798e829 match each response's tokens and cost
    4. Wait for the key's spend in /key/info to reach the expected minimum
    5. Check that the /spend/logs rows of team f3eeb02b-c35f-4773-b62d-a23e81b5d868 match each response's tokens and cost
    6. Wait for the key's spend in /key/info to reach the expected minimum
    View test on GitHub ↗
  605. ✓test_burst_rate_limit_does_not_freeze_fresh_key TestRedisCircuitBreakerPathratelimit/test_redis_circuit_breaker_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-cb-model-a894137cd89c that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-cb-model-a894137cd89c, key alias: e2e-cb-hot-e96a46a80868 and rpm limit: 1
    3. Generate a virtual key with models: e2e-cb-model-a894137cd89c and key alias: e2e-cb-cool-b751c7eb13f1
    4. Send a /chat/completions request to e2e-cb-model-a894137cd89c with the prompt "burst bc12d37f708d"
    5. Send a /chat/completions request to e2e-cb-model-a894137cd89c with the prompt "burst 7b30ac796ae7"
    6. Send a /chat/completions request to e2e-cb-model-a894137cd89c with the prompt "burst 69e42c03d036"
    7. Send a /chat/completions request to e2e-cb-model-a894137cd89c with the prompt "burst d6b838fa1a57"
    8. Send a /chat/completions request to e2e-cb-model-a894137cd89c with the prompt "burst 6cf329c0ce2d"
    9. Send a /chat/completions request to e2e-cb-model-a894137cd89c with the prompt "burst 4aad8fcae4a2"
    10. Send a /chat/completions request to e2e-cb-model-a894137cd89c with the prompt "burst 232be7219ded"
    11. Send a /chat/completions request to e2e-cb-model-a894137cd89c with the prompt "burst 963dd0419ff0"
    12. Send a /chat/completions request to e2e-cb-model-a894137cd89c with the prompt "burst f227d56efe74"
    13. Send a /chat/completions request to e2e-cb-model-a894137cd89c with the prompt "burst 7ec7b1ec7fe7"
    14. Send a /chat/completions request to e2e-cb-model-a894137cd89c with the prompt "burst 1646c7c94eb9"
    15. Send a /chat/completions request to e2e-cb-model-a894137cd89c with the prompt "burst dcb21fa44904"
    16. Send a /chat/completions request to e2e-cb-model-a894137cd89c with the prompt "fresh 81d2288865de"
    View test on GitHub ↗
  606. ✓test_cache_hit_is_zero_cost_and_suffixedspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "What is the capital of France? Answer in one word. 52b4094fe977"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  607. ✓test_cache_hit_reduces_tpm_by_non_cached_only TestTpmExcludesCachedTokensratelimit/test_tpm_excludes_cached_tokens_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • prompt_caching
    1. Add a deployment named e2e-tpm-cache-66e2cd71b2f0 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-tpm-cache-66e2cd71b2f0 and tpm limit: 100000
    3. POST /chat/completions
    4. Send a request to /chat/completions
    View test on GitHub ↗
  608. ✓test_cache_write_tokens_billed_at_cache_creation_rate TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.6-luna
    • prompt_caching
    1. List the deployments from /model/info
    2. Generate a virtual key with user id: e2e-test-user
    3. Add a deployment with custom rates that calls openai/gpt-5.6-luna
    4. Send a /chat/completions request to cache-write-priced-e5750053aad1
    5. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  609. ✓test_capture_rate_reports_or_names_the_missing_billing_keyspend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/capture_rate
    View test on GitHub ↗
  610. ✓test_chat_completion_writes_nonzero_spend_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "reply with one word 03453332f31d"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  611. ✓test_cold_counter_reseed_keeps_counter_equal_to_db_spendbudgets/test_spend_counter_reseed_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and max budget: 1.0
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 15a25d8670ca"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 2a46c0197387"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 2d86d45703ea"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed e24409b91bfb"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 092c49f3f7b2"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 073ae08a450e"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 40892691edfd"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed f839aea9cbf2"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed ee609e056083"
    11. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 3e47ddf857c1"
    12. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 8fed8b33fe6e"
    13. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed e2adada81928"
    14. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed d9ef14d44848"
    15. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 2c19a589d658"
    16. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 28ef8ca83424"
    17. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 0990041517aa"
    18. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed f28b6bf93d1a"
    19. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 35c681694355"
    20. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed be2aca0e1db8"
    21. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 6a62366d9272"
    22. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed d0a26cb64355"
    23. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 6df4f8629bf9"
    24. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 1b55cca3ba43"
    25. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed f6cd2018c27a"
    26. Read the key's settings back from /key/info
    27. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 8df346327256"
    28. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed d960f2d5af18"
    29. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 69d6d39d738e"
    30. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 127f5847651a"
    31. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 7e08bb651fa8"
    32. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reseed 8df77af0245c"
    View test on GitHub ↗
  612. ✓test_component_cost_headers_sum_to_total TestCostHeadersspend_tracking/test_cost_headers_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.5
    1. List the deployments from /model/info
    2. Generate a virtual key with user id: e2e-test-user
    3. Add a deployment with custom rates that calls openai/gpt-5.5
    4. Send a request to /chat/completions
    View test on GitHub ↗
  613. ✓test_configured_discount_reaches_persisted_spend_row TestPricingConfigSpendspend_tracking/test_pricing_config_spend_e2e.py›
    • spend-budgets
    • spend_reporting
    • nonstream
    • openai
    • openai/gpt-6.1-sol
    1. List the deployments from /model/info
    2. Generate a virtual key with user id: e2e-test-user
    3. GET /config/cost_discount_config
    4. GET /config/cost_margin_config
    5. PATCH /config/cost_discount_config
    6. PATCH /config/cost_margin_config
    7. Wait for the last control-plane write to reach every proxy replica
    8. PATCH /config/cost_discount_config
    9. Wait for the last control-plane write to reach every proxy replica
    10. Add a deployment named discount-priced-be9f3ed13e35 that calls openai/gpt-6.1-sol
    11. Send a /chat/completions request to discount-priced-be9f3ed13e35 with the prompt "reply with one word 35596d64e934"
    12. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  614. ✓test_configured_margin_reaches_persisted_spend_row TestPricingConfigSpendspend_tracking/test_pricing_config_spend_e2e.py›
    • spend-budgets
    • spend_reporting
    • nonstream
    • openai
    • openai/gpt-6.1-sol
    1. Generate a virtual key with user id: e2e-test-user
    2. GET /config/cost_discount_config
    3. GET /config/cost_margin_config
    4. PATCH /config/cost_discount_config
    5. PATCH /config/cost_margin_config
    6. Wait for the last control-plane write to reach every proxy replica
    7. PATCH /config/cost_margin_config
    8. Wait for the last control-plane write to reach every proxy replica
    9. Add a deployment named margin-priced-526d6a899a72 that calls openai/gpt-6.1-sol
    10. Send a /chat/completions request to margin-priced-526d6a899a72 with the prompt "reply with one word 945a2c64ffe6"
    11. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  615. ✓test_cost_breakdown_reports_component_costs TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.5
    • prompt_caching
    • reasoning
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a /chat/completions request to breakdown-priced-a47f943a200f
    4. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  616. ✓test_each_model_on_a_shared_key_gets_its_own_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • gemini
    • claude-haiku-4-5
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "one word 10712a818250"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "one word c6486e56171a"
    4. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  617. ✓test_edge_wired_chat_writes_nonzero_spend_rowspend_tracking/test_provider_edge_spend_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-edge-openai-5d3edd03912d that calls openai/gpt-5.5
    3. Send a /chat/completions request to e2e-edge-openai-5d3edd03912d with the prompt "reply with one word 273f65035913"
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  618. ✓test_embedding_writes_nonzero_spend_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • embeddings
    • nonstream
    • openai
    • openai-text-embedding-3-small
    1. Generate a virtual key with user id: e2e-test-user
    2. Send an /embeddings request to openai-text-embedding-3-small for "vectorize this sentence 3243386717b2"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  619. ✓test_end_user_budget_blocks_attributed_calls TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the end user e2e-budget-cust-4f25b155d6d0
    2. Generate a virtual key with models: claude-haiku-4-5
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 2d9316dd41f1"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend af1a4b392865"
    View test on GitHub ↗
  620. ✓test_end_user_header_attributes_responses_row[x-litellm-customer-id]spend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai-responses-codex
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to openai-responses-codex with custom headers and the prompt "one word 903356e73b1a"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    4. Wait for the end user's spend in /customer/info to go above 0.0
    View test on GitHub ↗
  621. ✓test_end_user_header_attributes_responses_row[x-litellm-end-user-id]spend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • responses
    • nonstream
    • openai
    • openai-responses-codex
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/responses request to openai-responses-codex with custom headers and the prompt "one word ce10e4518ab2"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    4. Wait for the end user's spend in /customer/info to go above 0.0
    View test on GitHub ↗
  622. ✓test_end_user_spend_attributed_on_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "hi"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  623. ✓test_every_streamed_chunk_carries_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • llm-translation
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a streaming /chat/completions request to tier-echo-stream-466c33a7fc88
    View test on GitHub ↗
  624. ✓test_every_write_path_row_joins_the_key TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • spend_reporting
    • anthropic
    • gemini
    • openai
    • claude-haiku-4-5
    • gemini-2.5-flash
    • openai-gpt-4o-mini
    • openai-responses-codex
    • openai-text-embedding-3-small
    1. List the deployments from /model/info
    2. Create an internal user with the role proxy_admin
    3. Generate a virtual key with user id: key-attribution-1898b1ec426d and key alias: key-attribution-1898b1ec426d
    4. Send a /chat/completions request to gemini-2.5-flash with the prompt "Reply with the word ok. 7395f59668ef"
    5. Send a /queue/chat/completions request to gemini-2.5-flash with the prompt "Reply with the word ok. 7395f59668ef"
    6. Send a /v1/messages request to claude-haiku-4-5 with the prompt "Reply with the word ok. 7395f59668ef"
    7. Send a /v1/responses request to openai-responses-codex with the prompt "Reply with the word ok. 7395f59668ef"
    8. Send an /embeddings request to openai-text-embedding-3-small for "Reply with the word ok. 7395f59668ef"
    9. Send a Gemini generateContent request to gemini-2.5-flash through /gemini with the prompt "Reply with the word ok. 7395f59668ef"
    10. Upload a batch input file for openai-gpt-4o-mini to /v1/files
    11. Create a batch for openai-gpt-4o-mini on /v1/batches
    12. Post a callback log for gemini-2.5-flash to /v1/rust_control_plane/logs
    13. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  625. ✓test_failure_call_writes_failure_status_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-spend-failure-556c1400facb that calls openai/gpt-5.5
    3. Send a /chat/completions request to e2e-spend-failure-556c1400facb with the prompt "trigger failure 705eafb8f56b"
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  626. ✓test_failure_rows_share_normalized_error_across_provider_wordingspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • openai
    • anthropic/claude-haiku-4-5
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-norm-openai-a810e9493f47 that calls openai/gpt-5.5
    3. Send a /chat/completions request to e2e-norm-openai-a810e9493f47 with the prompt "normalize failure a810e9493f47"
    4. Add a deployment named e2e-norm-anthropic-a810e9493f47 that calls anthropic/claude-haiku-4-5
    5. Send a /chat/completions request to e2e-norm-anthropic-a810e9493f47 with the prompt "normalize failure a810e9493f47"
    6. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  627. ✓test_generous_mode_lets_priority_borrow_past_reservation TestDynamicRateLimitPriorityratelimit/test_dynamic_rate_limit_priority_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-dynpri-generous-7a9f14e9e637 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-dynpri-generous-7a9f14e9e637, key alias: e2e-dynpri-generous-dev-7976cbc88e3e and metadata: priority: dev
    3. Generate a virtual key with models: e2e-dynpri-generous-7a9f14e9e637, key alias: e2e-dynpri-generous-prod-9295fa32acb3 and metadata: priority: prod
    4. Read the key's settings back from /key/info
    5. Send a /chat/completions request to e2e-dynpri-generous-7a9f14e9e637 with the prompt "reply with one word 63f5fd50f5b4"
    6. Send a /chat/completions request to e2e-dynpri-generous-7a9f14e9e637 with the prompt "reply with one word 840ac75a1096"
    7. Send a /chat/completions request to e2e-dynpri-generous-7a9f14e9e637 with the prompt "reply with one word ec5d6e5ba513"
    8. Send a /chat/completions request to e2e-dynpri-generous-7a9f14e9e637 with the prompt "reply with one word f4e40ca645b8"
    9. Send a /chat/completions request to e2e-dynpri-generous-7a9f14e9e637 with the prompt "reply with one word 86c2de7cdc13"
    View test on GitHub ↗
  628. ✓test_headers_report_limit_and_remaining TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5, tpm limit: 100000 and rpm limit: 5
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 286d499d544e"
    View test on GitHub ↗
  629. ✓test_health_check_rows_keep_the_service_account_key TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • health
    • gemini
    • gemini-2.5-flash
    1. Run a health check on gemini-2.5-flash with /health
    2. Read /spend/logs between 2026-10-18 06:12:22.088187+00:00 and 2026-10-18 06:14:24.419308+00:00
    3. Read /spend/logs between 2026-10-18 06:12:22.088187+00:00 and 2026-10-18 06:14:29.574448+00:00
    View test on GitHub ↗
  630. ✓test_intercepted_search_is_billed_under_the_request_session TestWebSearchInterceptionSessionspend_tracking/test_websearch_interception_session_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • bedrock
    • perplexity
    • bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    • web_search
    1. Add a deployment named e2e-websearch-session-400fa57e4590 that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with models: e2e-websearch-session-400fa57e4590 and user id: e2e-test-user
    3. Send a /v1/messages request to e2e-websearch-session-400fa57e4590
    4. Wait for at least 2 of the session's spend logs in /spend/logs
    View test on GitHub ↗
  631. ✓test_key_budget_reset_at_advances_after_windowbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Read the key's settings back from /key/info
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 482c1bcc8b50"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance ab74d4df2002"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance fdd828c60b90"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 33e87d852c3f"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance d8af1a63f61b"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 011c770cb500"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance b8e919b193b2"
    10. Read the key's settings back from /key/info
    View test on GitHub ↗
  632. ✓test_key_spend_blocks_at_capbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 584227a6b930"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 687b1c2f1b1a"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 1c9c82c52607"
    View test on GitHub ↗
  633. ✓test_key_spend_equals_sum_of_logsspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "say hi a34789485599"
    3. Send a /chat/completions request to gemini-2.5-flash with the prompt "say hi 22e4eaca7d5b"
    4. Wait for at least 2 of the key's spend logs in /spend/logs
    5. Wait for the key's spend in /key/info to reach the expected minimum
    View test on GitHub ↗
  634. ✓test_key_with_budget_duration_schedules_reset_at_creationbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • key_management
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  635. ✓test_long_window_blocks_after_short_window_resetsbudgets/test_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with models: gpt-5.5 and budget limits: budget duration: 30s and max budget: 1e-09, budget duration: 1d and max budget: 5e-07
    2. Send a /chat/completions request to gpt-5.5 with the prompt "window c0026f3e696c"
    3. Send a /chat/completions request to gpt-5.5 with the prompt "window 0de7dbd85a74"
    4. Read the key's budget windows from /key/info
    5. Send a /chat/completions request to gpt-5.5 with the prompt "window ec23a91d9765"
    View test on GitHub ↗
  636. ✓test_member_budget_blocks_without_touching_teammates TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-budget-team-b4ada829b3d0 and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Generate a virtual key with user id: 089b9473-adbb-4477-bc87-1523d9144001 and team id: 925fd023-7aab-4ce8-a861-2c5a890b7aae
    5. Create an internal user with max budget: 100.0
    6. Add the internal user to the team
    7. Generate a virtual key with user id: a13b9a40-97a8-42e3-b6c6-bf9230127f9c and team id: 925fd023-7aab-4ce8-a861-2c5a890b7aae
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend f8a480a89d24"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend ca8ccd09739e"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 250357dccfe4"
    View test on GitHub ↗
  637. ✓test_member_spend_attributed_to_team_and_user TestTeamMemberBudgetbudgets/test_team_member_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-team-member-ddd27a921168 and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Generate a virtual key with user id: a38fc341-1bb3-4a99-aa79-fff77551d59b and team id: 11f71e3c-f384-4711-9a55-4f4cbffb1446
    5. Send a /chat/completions request to claude-haiku-4-5
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  638. ✓test_member_spend_over_budget_is_blocked TestTeamMemberBudgetbudgets/test_team_member_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 210a9c89bd0d"
    View test on GitHub ↗
  639. ✓test_messages_bridge_keeps_cache_tokens TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-5.3-codex
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.3-codex
    3. Send a /v1/messages request to bridge-cache-priced-77e008eee55f
    4. Wait for a matching cost breakdown in the key's /spend/logs
    View test on GitHub ↗
  640. ✓test_messages_stream_records_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • spend-budgets
    • messages
    • stream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a streaming /v1/messages request to tier-messages-stream-6f7a4c5d9090
    4. Wait for a matching cost breakdown in the key's /spend/logs
    View test on GitHub ↗
  641. ✓test_missing_end_date_is_rejected TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py›
    • spend-budgets
    • spend_reporting
    1. Probe /team/daily/activity
    View test on GitHub ↗
  642. ✓test_missing_start_date_is_rejected TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py›
    • spend-budgets
    • spend_reporting
    1. Probe /team/daily/activity
    View test on GitHub ↗
  643. ✓test_model_group_shares_rpm_bucket_with_alias TestModelGroupAliasRateLimitratelimit/test_model_group_alias_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • e2e-alias-rl-alias
    • e2e-alias-rl-target
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word 847bbbc88ed6"
    3. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word 73af4f108446"
    4. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word 11830d744726"
    5. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word cac7322a924c"
    View test on GitHub ↗
  644. ✓test_model_max_budget_isolates_per_modelbudgets/test_model_max_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • gemini
    • claude-haiku-4-5
    • gemini-2.5-flash
    1. Generate a virtual key with model max budget: claude-haiku-4-5: budget limit: 1e-06 and time period: 30d and gemini-2.5-flash: budget limit: 1000.0 and time period: 30d
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi 23d9721d6be9"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi 276799eb9e7d"
    4. Send a /chat/completions request to gemini-2.5-flash with the prompt "hi e3a3a5babdb9"
    View test on GitHub ↗
  645. ✓test_multi_window_key_resets_each_window_independentlybudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with budget limits: budget duration: 30s and max budget: 3e-06, budget duration: 1m and max budget: 1.0
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 177ba7be2051"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance edd61af38c93"
    4. Read the key's settings back from /key/info
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 9bfcf48fc087"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 15dd22e64019"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 115333299a72"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 86fe526805cc"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 95140a526c24"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance e89d2cfb7cf1"
    11. Read the key's settings back from /key/info
    View test on GitHub ↗
  646. ✓test_one_request_lands_the_same_spend_on_every_surface TestSpendSurfaceConsistencyspend_tracking/test_spend_surface_consistency_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.6-luna
    1. Add a deployment named e2e-spend-surfaces-6526e665c389 that calls openai/gpt-5.6-luna
    2. Create a team with team alias: e2e-spend-surfaces-6526e665c389
    3. Generate a virtual key with models: e2e-spend-surfaces-6526e665c389, team id: 6a8b5a7a-13c8-41dc-888c-ab0f5ddde2f3 and key alias: e2e-spend-surfaces-6526e665c389
    4. Send a /chat/completions request to e2e-spend-surfaces-6526e665c389
    5. Wait for the request's spend log in /spend/logs
    6. Wait for the key's spend in /key/info to reach the expected minimum
    7. Wait for the team's spend in /team/info to reach the expected minimum
    8. Wait for at least 1 of the key's requests in /user/daily/activity/aggregated
    9. Scrape /metrics/ on every proxy replica
    View test on GitHub ↗
  647. ✓test_org_budget_blocks_keys_under_it TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the organization e2e-budget-org-00ec1ae1d737 with max budget: 3e-06
    2. Create the team e2e-budget-team-bf79ddbf873c and wait until /team/info returns it
    3. Generate a virtual key with team id: 41d21ca5-4af0-4978-9214-dd621f57d8b9
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 305a2a5fb46e"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend a268c485e50f"
    View test on GitHub ↗
  648. ✓test_org_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the organization e2e-org-reset-259af097903c with max budget: 3e-06
    2. Create the team e2e-org-team-2203ccd99daa and wait until /team/info returns it
    3. Generate a virtual key with team id: 52ea641b-66a3-4240-b021-251b93c63bd4
    4. Read the organization's budget id from /organization/info
    5. Read the budget from /budget/info
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 645fee22c44e"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 341efd1f1c72"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 3b3fd8fd253c"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset da70f6060ff6"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset d41d571bc2d2"
    11. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset a62f33b1e4de"
    12. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset d78876b01266"
    View test on GitHub ↗
  649. ✓test_personal_key_blocks_over_its_own_budget TestKeyBudgetBlocksAcrossKeyKindsbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create an internal user with max budget: 100.0
    2. Generate a virtual key with max budget: 3e-06 and user id: ff8149d1-8e9a-4cd0-8a14-0702583a8811
    3. Generate a virtual key with user id: ff8149d1-8e9a-4cd0-8a14-0702583a8811
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend a3ce757b7294"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend cee5b16d146e"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 72b246c320b6"
    View test on GitHub ↗
  650. ✓test_personal_key_resets_after_window TestKeyBudgetResetAcrossKeyKindsbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create an internal user with max budget: 100.0
    2. Generate a virtual key with max budget: 3e-06, budget duration: 30s and user id: 3e72ab19-efe4-4d38-9537-1a2314fd54d0
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 3113d11fb979"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 22da84b094f4"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset d2c71db600f5"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 1af6a1f285a4"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset fefef6d8f34f"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 6b7f3146311b"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 99724dd43509"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 60ec4d350960"
    View test on GitHub ↗
  651. ✓test_personal_key_user_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create an internal user with max budget: 3e-06
    2. Generate a virtual key with user id: d0ee4783-a8d7-4baa-82d5-3910b3360b8f
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 501dea946e8d"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 6cb4fb958276"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 9c8565c77766"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset cb7ca670d6f9"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 0f98b4ca119e"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset f9bd4809016e"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset ee9fc3d1d68d"
    View test on GitHub ↗
  652. ✓test_pre_call_rejection_row_attributes_provider_and_model_idspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Add a deployment named e2e-spend-precall-9f3175e03ce5 that calls openai/gpt-5.5
    2. Generate a virtual key with models: e2e-spend-precall-9f3175e03ce5 and rpm limit: 1
    3. Send a /chat/completions request to e2e-spend-precall-9f3175e03ce5 with the prompt "reply with one word afc8cf5aab71"
    4. Send a /chat/completions request to e2e-spend-precall-9f3175e03ce5 with the prompt "over the rpm limit b855148e78ef"
    5. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  653. ✓test_priority_tier_bills_priority_rates TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.6-luna
    • reasoning
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.6-luna
    3. Send a /chat/completions request to tier-priced-9b17b2601fb1
    4. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  654. ✓test_request_tags_round_tripspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "tagged request"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  655. ✓test_reset_wait_never_yields_non_budget_errorbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance fda19f053bb3"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 75db621824aa"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance fa71eb309c9f"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance a4826bc8d115"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 53abbc6550ca"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance f1c39421cc64"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 2650fbacc8f2"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance c6beee4b6d6b"
    View test on GitHub ↗
  656. ✓test_responses_stream_records_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • spend-budgets
    • responses
    • stream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a streaming /v1/responses request to tier-responses-stream-cfbe4b6bdd9d
    4. Wait for a matching cost breakdown in the key's /spend/logs
    View test on GitHub ↗
  657. ✓test_rpm_limit_blocks_over_limit TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and rpm limit: 3
    2. Read the key's settings back from /key/info
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 20125fff8f75"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 4c2bece91afd"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 351907714789"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 70abbb8f6f49"
    View test on GitHub ↗
  658. ✓test_rpm_limit_one_blocks_second_call TestRedisBackedRateLimitratelimit/test_redis_backed_ratelimit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-redis-rpm-9799390f6065 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-redis-rpm-9799390f6065, key alias: e2e-redis-rpm-9bf5798aee2d and rpm limit: 1
    3. Read the key's settings back from /key/info
    4. Send a /chat/completions request to e2e-redis-rpm-9799390f6065 with the prompt "ping c2156dc6cab7"
    5. Send a /chat/completions request to e2e-redis-rpm-9799390f6065 with the prompt "pong fbf6d226347e"
    View test on GitHub ↗
  659. ✓test_rpm_limit_resets_after_window TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and rpm limit: 1
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word c1c6f6648e9b"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word d89f07abb33b"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word bf3a7679015a"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word bf80dce04ad3"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 744adc713d2f"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 003054d24e27"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 69ae830b0ab7"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 89407c36f704"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 0f92ac3f15d0"
    11. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 47d0dac4abfd"
    12. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word dbd9bb37172b"
    13. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word c794456d5b15"
    14. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 7d5d7eac2ab9"
    15. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 69cfdb08d26b"
    16. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 7983102aff85"
    View test on GitHub ↗
  660. ✓test_schema_listed_spend_routes_are_responsivespend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Read the OpenAPI schema from /openapi.json
    View test on GitHub ↗
  661. ✓test_short_window_blocks_then_resetsbudgets/test_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with models: gpt-5.5 and budget limits: budget duration: 30s and max budget: 1e-09, budget duration: 1m and max budget: 1.0
    2. Send a /chat/completions request to gpt-5.5 with the prompt "window 274b91dfc666"
    3. Send a /chat/completions request to gpt-5.5 with the prompt "window 72def19ef8d8"
    4. Send a /chat/completions request to gpt-5.5 with the prompt "window 707d2c811d39"
    5. Send a /chat/completions request to gpt-5.5 with the prompt "window f3b3580b89be"
    6. Send a /chat/completions request to gpt-5.5 with the prompt "window 1f0786057f5e"
    7. Send a /chat/completions request to gpt-5.5 with the prompt "window c8daa40ea722"
    8. Send a /chat/completions request to gpt-5.5 with the prompt "window 864d2786577f"
    View test on GitHub ↗
  662. ✓test_soft_budget_does_not_blockbudgets/test_soft_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 1000.0 and soft budget: 1e-09
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi d488d8843bab"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi aa641925cdb0"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi a5cb0f0d8e78"
    View test on GitHub ↗
  663. ✓test_spend_calculate_returns_nonzero_costspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • spend_reporting
    • gemini
    • gemini-2.5-flash
    1. Estimate the cost of sending "estimate the cost of this request" to gemini-2.5-flash with /spend/calculate
    View test on GitHub ↗
  664. ✓test_spend_logs_by_key_return_every_row_with_the_alias TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • spend_reporting
    • anthropic
    • gemini
    • openai
    • claude-haiku-4-5
    • gemini-2.5-flash
    • openai-gpt-4o-mini
    • openai-responses-codex
    • openai-text-embedding-3-small
    1. Wait for at least 9 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  665. ✓test_spend_logs_endpoint_returns_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • spend_reporting
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "spend logs 4a599a6acae8"
    3. GET /spend/logs
    View test on GitHub ↗
  666. ✓test_spend_logs_v2_pagination_caps_pages_and_keeps_totalspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • spend_reporting
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "page fodder 2d493dbd6a88"
    3. Send a /chat/completions request to gemini-2.5-flash with the prompt "page fodder ca4ef66f0e69"
    4. Wait for at least 2 of the key's spend logs in /spend/logs
    5. Read page 1 of /spend/logs/v2 at a page size of 1
    6. Read page 9 of /spend/logs/v2 at a page size of 1
    7. Read page 1 of /spend/logs/v2 at a page size of 1
    View test on GitHub ↗
  667. ✓test_spend_route_responsive[/customer/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /customer/daily/activity
    View test on GitHub ↗
  668. ✓test_spend_route_responsive[/customer/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • customer_management
    1. Call the management route /customer/list
    View test on GitHub ↗
  669. ✓test_spend_route_responsive[/end_user/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /end_user/daily/activity
    View test on GitHub ↗
  670. ✓test_spend_route_responsive[/global/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/activity
    View test on GitHub ↗
  671. ✓test_spend_route_responsive[/global/activity/exceptions]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/activity/exceptions
    View test on GitHub ↗
  672. ✓test_spend_route_responsive[/global/activity/exceptions/deployment]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/activity/exceptions/deployment
    View test on GitHub ↗
  673. ✓test_spend_route_responsive[/global/activity/model]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/activity/model
    View test on GitHub ↗
  674. ✓test_spend_route_responsive[/global/all_end_users]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/all_end_users
    View test on GitHub ↗
  675. ✓test_spend_route_responsive[/global/spend]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend until it answers successfully
    View test on GitHub ↗
  676. ✓test_spend_route_responsive[/global/spend/all_tag_names]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/all_tag_names
    View test on GitHub ↗
  677. ✓test_spend_route_responsive[/global/spend/keys]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/keys until it answers successfully
    View test on GitHub ↗
  678. ✓test_spend_route_responsive[/global/spend/logs]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/logs until it answers successfully
    View test on GitHub ↗
  679. ✓test_spend_route_responsive[/global/spend/models]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/models until it answers successfully
    View test on GitHub ↗
  680. ✓test_spend_route_responsive[/global/spend/provider]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/provider
    View test on GitHub ↗
  681. ✓test_spend_route_responsive[/global/spend/report]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/report
    View test on GitHub ↗
  682. ✓test_spend_route_responsive[/global/spend/tags]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/tags until it answers successfully
    View test on GitHub ↗
  683. ✓test_spend_route_responsive[/global/spend/teams]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /global/spend/teams
    View test on GitHub ↗
  684. ✓test_spend_route_responsive[/key/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • key_management
    1. Call the management route /key/list
    View test on GitHub ↗
  685. ✓test_spend_route_responsive[/organization/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /organization/daily/activity
    View test on GitHub ↗
  686. ✓test_spend_route_responsive[/organization/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • organization_management
    1. Call the management route /organization/list
    View test on GitHub ↗
  687. ✓test_spend_route_responsive[/spend/keys]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. List the deployments from /model/info
    2. Add a deployment named openai-text-embedding-3-small that calls openai/text-embedding-3-small
    3. Add a deployment named openai-responses-codex that calls openai/gpt-5.3-codex
    4. Add a deployment named openai-gpt-4o-mini that calls openai/gpt-4o-mini
    5. Call the management route /spend/keys
    View test on GitHub ↗
  688. ✓test_spend_route_responsive[/spend/logs]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/logs
    View test on GitHub ↗
  689. ✓test_spend_route_responsive[/spend/logs/session/ui]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/logs/session/ui
    View test on GitHub ↗
  690. ✓test_spend_route_responsive[/spend/logs/ui]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/logs/ui
    View test on GitHub ↗
  691. ✓test_spend_route_responsive[/spend/logs/v2]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/logs/v2
    View test on GitHub ↗
  692. ✓test_spend_route_responsive[/spend/tags]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/tags
    View test on GitHub ↗
  693. ✓test_spend_route_responsive[/spend/users]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /spend/users
    View test on GitHub ↗
  694. ✓test_spend_route_responsive[/tag/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /tag/daily/activity
    View test on GitHub ↗
  695. ✓test_spend_route_responsive[/team/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /team/daily/activity
    View test on GitHub ↗
  696. ✓test_spend_route_responsive[/team/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • team_management
    1. Call the management route /team/list
    View test on GitHub ↗
  697. ✓test_spend_route_responsive[/user/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /user/daily/activity
    View test on GitHub ↗
  698. ✓test_spend_route_responsive[/user/daily/activity/aggregated]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. Call the management route /user/daily/activity/aggregated
    View test on GitHub ↗
  699. ✓test_spend_route_responsive[/user/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • user_management
    1. Call the management route /user/list
    View test on GitHub ↗
  700. ✓test_streamed_call_records_and_bills_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • spend-budgets
    • chat_completions
    • stream
    • openai
    • openai/gpt-5.6-luna
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.6-luna
    3. Send a streaming /chat/completions request to tier-priced-stream-e5cda5d4eb09
    4. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  701. ✓test_streaming_cache_read_billed_at_cache_read_rate TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • stream
    • openai
    • openai/gpt-5.5
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment with custom rates that calls openai/gpt-5.5
    3. Send a /chat/completions request to stream-cache-priced-332493a6d64b
    4. Send a streaming /chat/completions request to stream-cache-priced-332493a6d64b
    5. Wait for the request's cost breakdown in /spend/logs
    View test on GitHub ↗
  702. ✓test_streaming_chat_completion_tracks_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • chat_completions
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to gemini-2.5-flash with the prompt "count to three 45ca65ae1f02"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  703. ✓test_streaming_messages_via_responses_bridge_tracks_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • messages
    • stream
    • openai
    • openai-responses-codex
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to openai-responses-codex with the prompt "reply with exactly one word 10dff8c8cc7e"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  704. ✓test_tag_budget_blocks_tagged_requestsbudgets/test_tag_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Create the tag e2e-budget-tag-65181da07a2c with max budget: 1e-06
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi 4c55965496e8"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi f6f70a4cd4d1"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "hi 5bc092a5b200"
    View test on GitHub ↗
  705. ✓test_tag_spend_matches_sum_of_tagged_logsspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "hi 9e7da9bf8e79"
    3. Send a /chat/completions request to gemini-2.5-flash with the prompt "hi 8bc27a46a004"
    4. Wait for at least 2 of the key's spend logs in /spend/logs
    5. Wait for the tag e2e-tagspend-5cb60b0008dc to reach the expected spend in /spend/tags
    View test on GitHub ↗
  706. ✓test_team_budget_blocks_every_team_key TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-budget-team-f0ad09717f08 and wait until /team/info returns it
    2. Generate a virtual key with team id: 2c0c445e-1fc7-4300-aaa5-dab9f230fa60
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 7ffa562bca2b"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 483a0552b508"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend b43bfd4b41bd"
    View test on GitHub ↗
  707. ✓test_team_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-team-reset-3eb87516ea0b and wait until /team/info returns it
    2. Generate a virtual key with team id: c54481cf-aaf7-465a-8202-b82bee3f0c47
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset a95e1d1bdb71"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset bd6b07dc2d3b"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 94fc727a8ee6"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 6a6daa4bd186"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 7bb68125566b"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset fa9c83a15b3b"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 0f39671a575d"
    View test on GitHub ↗
  708. ✓test_team_key_blocks_over_its_own_budget TestKeyBudgetBlocksAcrossKeyKindsbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-key-cap-team-d8f5602d7d36 and wait until /team/info returns it
    2. Generate a virtual key with max budget: 3e-06 and team id: adc56a70-ccf8-4c6c-90c3-b9ce455fbbc1
    3. Generate a virtual key with team id: adc56a70-ccf8-4c6c-90c3-b9ce455fbbc1
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 7dd185b4af5f"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 566e6f496a3e"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 7c7acaa670ec"
    View test on GitHub ↗
  709. ✓test_team_key_resets_after_window TestKeyBudgetResetAcrossKeyKindsbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-key-reset-team-bd486b71397b and wait until /team/info returns it
    2. Generate a virtual key with max budget: 3e-06, budget duration: 30s and team id: 73d9d5bc-e82d-4625-875d-11b6c5407b7f
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 36dcc04cd737"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset ec51115b86cf"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 363262a0da72"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset e922a93ae7f5"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 919ab6a261c6"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 8857d1f93747"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset dd10d0baf8f1"
    View test on GitHub ↗
  710. ✓test_team_long_window_blocks_after_short_window_resetsbudgets/test_team_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-team-long-window-e91f5f26cf89 and wait until /team/info returns it
    2. Generate a virtual key with models: claude-haiku-4-5 and team id: aa5855e4-4d89-478d-8f9f-e6c6a9da47f9
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window dc1b5b015c50"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 41f343b90432"
    5. Read the team's budget windows from /team/info
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window a8ba9640ada9"
    View test on GitHub ↗
  711. ✓test_team_member_budget_reset_at_advancesbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-member-advance-7c5e8b57a208 and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Update the team member's budget with /team/member_update
    5. Read the team member's budget reset time from /team/info
    6. Generate a virtual key with user id: 98f956c4-2b29-42c0-8038-cc81e3f1eb03 and team id: c5ef9710-055b-4562-9c37-17051bcceca2
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "advance 94f1b07dbbad"
    8. Read the team member's budget reset time from /team/info
    View test on GitHub ↗
  712. ✓test_team_member_budget_reset_keeps_advancingbudgets/test_team_member_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-member-reset-ee01b5a5fc2e and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Update the team member's budget with /team/member_update
    5. Read the team member's budget reset time from /team/info
    6. Generate a virtual key with user id: 01aee474-53c2-44f9-8b26-42d908a333ff and team id: 872404eb-fb38-4c36-834e-fcd9f67e7d26
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 1db2573cc069"
    8. Read the team member's budget reset time from /team/info
    View test on GitHub ↗
  713. ✓test_team_member_key_blocks_over_its_own_budget TestKeyBudgetBlocksAcrossKeyKindsbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-key-cap-team-fb2d869518ed and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Generate a virtual key with max budget: 3e-06, user id: b8b149af-e600-43ba-b1b9-75e65edbd145 and team id: 2c5e525c-4d5f-4617-9c6e-9d25f3680416
    5. Generate a virtual key with user id: b8b149af-e600-43ba-b1b9-75e65edbd145 and team id: 2c5e525c-4d5f-4617-9c6e-9d25f3680416
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 016b85671fa1"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 5c8b7731c709"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend f06ca3b3c233"
    View test on GitHub ↗
  714. ✓test_team_member_key_resets_after_window TestKeyBudgetResetAcrossKeyKindsbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-key-reset-team-b8fc66190d3d and wait until /team/info returns it
    2. Create an internal user with max budget: 100.0
    3. Add the internal user to the team
    4. Generate a virtual key with max budget: 3e-06, budget duration: 30s, user id: d8b0700d-96d5-415a-b207-087147245d62 and team id: 9b5a2eb2-839e-40f6-90b1-193085f6faf5
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 294955367467"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset c8de9d387029"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 5762de1d435f"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 8bd282c2100e"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset 0efd1c61bbc0"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset e4397e085eef"
    11. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reset d2439e69f20d"
    View test on GitHub ↗
  715. ✓test_team_short_window_blocks_then_resetsbudgets/test_team_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team e2e-team-window-d19dc1eafc01 and wait until /team/info returns it
    2. Generate a virtual key with models: claude-haiku-4-5 and team id: 8fb0677f-ef51-4e0e-b2bd-a3e8db5b9331
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window e82ab6eb943f"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 532d4743eafb"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 0dbf92204074"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window 4b4bb39142b3"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "team-window e5808121493c"
    View test on GitHub ↗
  716. ✓test_terminal_batch_cost_row_joins_the_retrieving_key TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • openai-gpt-4o-mini
    1. Retrieve the openai batch from /v1/batches
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  717. ✓test_the_budget_read_reports_the_spend_drawn_against_the_pool TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • budget_management
    • openai
    • openai/gpt-5.4-nano
    1. Read the budget and spend of the model access group e2e-mag-budget-5851981b25e7
    View test on GitHub ↗
  718. ✓test_the_key_that_drained_the_pool_stays_blocked TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-mag-budget-5851981b25e7
    2. Add a deployment named e2e-mag-budgeted-5851981b25e7 that calls openai/gpt-5.4-nano
    3. Add a deployment named e2e-mag-unbudgeted-5851981b25e7 that calls openai/gpt-5.4-nano
    4. Set a shared budget on the model access group e2e-mag-budget-5851981b25e7
    5. Send a /chat/completions request to e2e-mag-budgeted-5851981b25e7 with the prompt "hi 04b129d9cb00"
    6. Send a /chat/completions request to e2e-mag-budgeted-5851981b25e7 with the prompt "hi d92c299116ca"
    7. Send a /chat/completions request to e2e-mag-budgeted-5851981b25e7 with the prompt "hi 39ac5cebca2d"
    View test on GitHub ↗
  719. ✓test_tpm_limit_blocks_over_limit TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and tpm limit: 60
    2. Read the key's settings back from /key/info
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 1d5af9d6e724"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 207a595d05c8"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 41c64ab3f70c"
    View test on GitHub ↗
  720. ✓test_user_budget_blocks_a_second_key TestUserBudgetAcrossKeysbudgets/test_user_budget_across_keys_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Create an internal user with max budget: 3e-06
    2. Generate a virtual key with user id: efbdca1c-f633-4f0e-b62b-76d8a3330b8e
    3. Send a /chat/completions request to gpt-5.5 with the prompt "across a044ceaf3c80"
    4. Send a /chat/completions request to gpt-5.5 with the prompt "across 01b45aa56d12"
    5. Send a /chat/completions request to gpt-5.5 with the prompt "across 48c45ed75b23"
    6. Read the internal user's spend and budget from /user/info
    View test on GitHub ↗
  721. ✓test_user_budget_enforced_across_their_personal_keys TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create an internal user with max budget: 3e-06
    2. Generate a virtual key with user id: 5b6386d8-a0a5-4346-99ce-104a6d00a9cf
    3. Create the team e2e-budget-team-87b483b5cdd7 and wait until /team/info returns it
    4. Add the internal user to the team
    5. Generate a virtual key with user id: 5b6386d8-a0a5-4346-99ce-104a6d00a9cf and team id: 5fea54cc-fbbf-4227-b4b8-d28e52630e1c
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 47c4d9950194"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 39bd5e62566b"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 0ab25610c780"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "spend 177d8b9d36bb"
    View test on GitHub ↗
  722. ✓test_user_daily_activity_reports_alias_and_email TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • spend_reporting
    • anthropic
    • gemini
    • openai
    • claude-haiku-4-5
    • gemini-2.5-flash
    • openai-gpt-4o-mini
    • openai-responses-codex
    • openai-text-embedding-3-small
    1. Wait for at least 9 of the key's requests in /user/daily/activity
    View test on GitHub ↗
  723. ✓test_valid_date_range_returns_results_and_metadata TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py›
    • spend-budgets
    • spend_reporting
    • openai
    • openai/gpt-5.6-luna
    1. List the deployments from /model/info
    2. Add a priced deployment, then create two teams with one key each and send 7 /chat/completions requests per key, 6 of them at once
    3. Send a /chat/completions request to e2e-reconciliation-d86acc50f642
    4. Check that the /spend/logs rows of team 4573faa0-2f1a-42a9-9d20-7164119e73bb match each response's tokens and cost
    5. Check that the /spend/logs rows of team 767e1574-c36b-4715-a82f-6f15725722d3 match each response's tokens and cost
    6. Probe /team/daily/activity
    View test on GitHub ↗
  724. −test_end_user_model_max_budget_enforces_per_model_rpmbudgets/test_model_max_budget_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash

    No steps recorded

    stage red: product gap, end-user model_max_budget rpm_limit is stored but never enforced

    View test on GitHub ↗
  725. −test_strict_mode_blocks_saturated_priority_but_serves_the_other TestDynamicRateLimitPriorityratelimit/test_dynamic_rate_limit_priority_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001

    No steps recorded

    LIT-5118: the stage proxy does not run the dynamic_rate_limiter_v3 callbacks + priority_reservation config this module's docstring requires (zero limiter log lines on any pod during the 2026-08-02 run), so strict enforcement can never engage there

    View test on GitHub ↗
  726. ✓test_basic_messaging_non_streaming_anthropicbasic_messaging_non_streaming/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 through the proxy and check every model replies
    26. Run Claude Code headless against claude-haiku-4-5 through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    28. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  727. ✓test_basic_messaging_non_streaming_azurebasic_messaging_non_streaming/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure through the proxy and check every model replies
    26. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    28. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  728. ✓test_basic_messaging_non_streaming_azure_openaibasic_messaging_non_streaming/test_azure_openai.py›
    • llm-translation
    • messages
    • nonstream
    • azure
    • gpt-5-6-luna-azure-openai
    • gpt-5-6-sol-azure-openai
    • gpt-5-6-terra-azure-openai
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against gpt-5-6-sol-azure-openai, gpt-5-6-terra-azure-openai, gpt-5-6-luna-azure-openai through the proxy and check every model replies
    26. Run Claude Code headless against gpt-5-6-sol-azure-openai through the proxy
    27. Run Claude Code headless against gpt-5-6-terra-azure-openai through the proxy
    28. Run Claude Code headless against gpt-5-6-luna-azure-openai through the proxy
    View test on GitHub ↗
  729. ✓test_basic_messaging_non_streaming_bedrock_conversebasic_messaging_non_streaming/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse through the proxy and check every model replies
    26. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    28. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  730. ✓test_basic_messaging_non_streaming_bedrock_invokebasic_messaging_non_streaming/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke through the proxy and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  731. ✓test_basic_messaging_non_streaming_vertex_aibasic_messaging_non_streaming/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex through the proxy and check every model replies
    26. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    28. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  732. ✓test_basic_messaging_non_streaming_vertex_ai_gptbasic_messaging_non_streaming/test_vertex_ai_gpt.py›
    • llm-translation

    No steps recorded

    View test on GitHub ↗
  733. ✓test_basic_messaging_streaming_anthropicbasic_messaging_streaming/test_anthropic.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 through the proxy and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  734. ✓test_basic_messaging_streaming_azurebasic_messaging_streaming/test_azure.py›
    • llm-translation
    • messages
    • stream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure through the proxy and check every model replies
    26. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    28. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  735. ✓test_basic_messaging_streaming_azure_openaibasic_messaging_streaming/test_azure_openai.py›
    • llm-translation
    • messages
    • stream
    • azure
    • gpt-5-6-luna-azure-openai
    • gpt-5-6-sol-azure-openai
    • gpt-5-6-terra-azure-openai
    1. Run Claude Code headless against gpt-5-6-sol-azure-openai, gpt-5-6-terra-azure-openai, gpt-5-6-luna-azure-openai through the proxy and check every model replies
    2. Run Claude Code headless against gpt-5-6-sol-azure-openai through the proxy
    3. Run Claude Code headless against gpt-5-6-terra-azure-openai through the proxy
    4. Run Claude Code headless against gpt-5-6-luna-azure-openai through the proxy
    View test on GitHub ↗
  736. ✓test_basic_messaging_streaming_bedrock_conversebasic_messaging_streaming/test_bedrock_converse.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse through the proxy and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  737. ✓test_basic_messaging_streaming_bedrock_invokebasic_messaging_streaming/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke through the proxy and check every model replies
    26. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    27. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    28. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  738. ✓test_basic_messaging_streaming_vertex_aibasic_messaging_streaming/test_vertex_ai.py›
    • llm-translation
    • messages
    • stream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex through the proxy and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  739. ✓test_basic_messaging_streaming_vertex_ai_gptbasic_messaging_streaming/test_vertex_ai_gpt.py›
    • llm-translation

    No steps recorded

    View test on GitHub ↗
  740. ✓test_count_tokens_anthropiccount_tokens/test_anthropic.py›
    • llm-translation
    • count_tokens
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5 on the message "hello world"
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5 on the message "hello world"
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7 on the message "hello world"
    View test on GitHub ↗
  741. ✓test_count_tokens_azurecount_tokens/test_azure.py›
    • llm-translation
    • count_tokens
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5-azure on the message "hello world"
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5-azure on the message "hello world"
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7-azure on the message "hello world"
    View test on GitHub ↗
  742. ✓test_count_tokens_bedrock_conversecount_tokens/test_bedrock_converse.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5-bedrock-converse on the message "hello world"
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5-bedrock-converse on the message "hello world"
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7-bedrock-converse on the message "hello world"
    View test on GitHub ↗
  743. ✓test_count_tokens_bedrock_invokecount_tokens/test_bedrock_invoke.py›
    • llm-translation
    • count_tokens
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5-bedrock-invoke on the message "hello world"
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5-bedrock-invoke on the message "hello world"
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7-bedrock-invoke on the message "hello world"
    View test on GitHub ↗
  744. ✓test_passthrough_anthropicpassthrough/test_anthropic.py›
    • passthrough
    • passthrough
    • stream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 through the proxy's native provider passthrough route and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  745. ✓test_passthrough_bedrock_conversepassthrough/test_bedrock_converse.py›
    • passthrough

    No steps recorded

    View test on GitHub ↗
  746. ✓test_passthrough_bedrock_invokepassthrough/test_bedrock_invoke.py›
    • passthrough
    • passthrough
    • stream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke through the proxy's native provider passthrough route and check every
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  747. ✓test_passthrough_vertex_aipassthrough/test_vertex_ai.py›
    • passthrough
    • passthrough
    • stream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex through the proxy's native provider passthrough route and check every model replies
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  748. ✓test_pdf_input_anthropicpdf_input/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • pdf_input
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  749. ✓test_pdf_input_azurepdf_input/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • pdf_input
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  750. ✓test_pdf_input_bedrock_conversepdf_input/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • pdf_input
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  751. ✓test_pdf_input_bedrock_invokepdf_input/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • pdf_input
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  752. ✓test_pdf_input_vertex_aipdf_input/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • pdf_input
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  753. ✓test_prompt_caching_1h_anthropicprompt_caching_1h/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  754. ✓test_prompt_caching_1h_azureprompt_caching_1h/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  755. ✓test_prompt_caching_1h_bedrock_converseprompt_caching_1h/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  756. ✓test_prompt_caching_1h_bedrock_invokeprompt_caching_1h/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  757. ✓test_prompt_caching_1h_vertex_aiprompt_caching_1h/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  758. ✓test_prompt_caching_5m_anthropicprompt_caching_5m/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  759. ✓test_prompt_caching_5m_azureprompt_caching_5m/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  760. ✓test_prompt_caching_5m_bedrock_converseprompt_caching_5m/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  761. ✓test_prompt_caching_5m_bedrock_invokeprompt_caching_5m/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  762. ✓test_prompt_caching_5m_vertex_aiprompt_caching_5m/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • prompt_caching
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  763. ✓test_structured_outputs_anthropicstructured_outputs/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • response_schema
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  764. ✓test_structured_outputs_azurestructured_outputs/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • response_schema
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  765. ✓test_structured_outputs_bedrock_conversestructured_outputs/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • response_schema
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  766. ✓test_structured_outputs_bedrock_invokestructured_outputs/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • response_schema
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  767. ✓test_structured_outputs_vertex_aistructured_outputs/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • response_schema
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  768. ✓test_thinking_anthropicthinking/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  769. ✓test_thinking_azurethinking/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  770. ✓test_thinking_bedrock_conversethinking/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  771. ✓test_thinking_bedrock_invokethinking/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  772. ✓test_thinking_vertex_aithinking/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  773. ✓test_thinking_with_tool_use_anthropicthinking_with_tool_use/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • function_calling
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  774. ✓test_thinking_with_tool_use_azurethinking_with_tool_use/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • function_calling
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  775. ✓test_thinking_with_tool_use_bedrock_conversethinking_with_tool_use/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • function_calling
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  776. ✓test_thinking_with_tool_use_bedrock_invokethinking_with_tool_use/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • function_calling
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  777. ✓test_thinking_with_tool_use_vertex_aithinking_with_tool_use/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • function_calling
    • reasoning
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  778. ✓test_tool_search_anthropictool_search/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • tool_search
    1. Send a /v1/messages request to claude-haiku-4-5 with the tool_search tool declared
    2. Send a /v1/messages request to claude-sonnet-4-5 with the tool_search tool declared
    3. Send a /v1/messages request to claude-opus-4-7 with the tool_search tool declared
    View test on GitHub ↗
  779. ✓test_tool_search_bedrock_conversetool_search/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • tool_search
    1. Send a /v1/messages request to claude-haiku-4-5-bedrock-converse with the tool_search tool declared
    2. Send a /v1/messages request to claude-sonnet-4-5-bedrock-converse with the tool_search tool declared
    3. Send a /v1/messages request to claude-opus-4-7-bedrock-converse with the tool_search tool declared
    View test on GitHub ↗
  780. ✓test_tool_search_bedrock_invoketool_search/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • tool_search
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    25. Send a /v1/messages request to claude-haiku-4-5-bedrock-invoke with the tool_search tool declared
    26. Send a /v1/messages request to claude-sonnet-4-5-bedrock-invoke with the tool_search tool declared
    27. Send a /v1/messages request to claude-opus-4-7-bedrock-invoke with the tool_search tool declared
    View test on GitHub ↗
  781. ✓test_tool_search_history_bedrock_invoketool_search/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • tool_search
    1. Send a /v1/messages request to claude-haiku-4-5-bedrock-invoke with the tool_search tool declared, then send its answer back as history in a second request
    2. Send a /v1/messages request to claude-sonnet-4-5-bedrock-invoke with the tool_search tool declared, then send its answer back as history in a second request
    3. Send a /v1/messages request to claude-opus-4-7-bedrock-invoke with the tool_search tool declared, then send its answer back as history in a second request
    View test on GitHub ↗
  782. ✓test_tool_use_anthropictool_use/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  783. ✓test_tool_use_azuretool_use/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  784. ✓test_tool_use_azure_openaitool_use/test_azure_openai.py›
    • llm-translation
    • messages
    • nonstream
    • azure
    • gpt-5-6-luna-azure-openai
    • gpt-5-6-sol-azure-openai
    • gpt-5-6-terra-azure-openai
    • function_calling
    1. Run Claude Code headless against gpt-5-6-sol-azure-openai, gpt-5-6-terra-azure-openai, gpt-5-6-luna-azure-openai in parallel through the proxy
    2. Run Claude Code headless against gpt-5-6-sol-azure-openai through the proxy
    3. Run Claude Code headless against gpt-5-6-terra-azure-openai through the proxy
    4. Run Claude Code headless against gpt-5-6-luna-azure-openai through the proxy
    View test on GitHub ↗
  785. ✓test_tool_use_bedrock_conversetool_use/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  786. ✓test_tool_use_bedrock_invoketool_use/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  787. ✓test_tool_use_streaming_anthropictool_use_streaming/test_anthropic.py›
    • llm-translation
    • messages
    • stream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  788. ✓test_tool_use_streaming_azuretool_use_streaming/test_azure.py›
    • llm-translation
    • messages
    • stream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  789. ✓test_tool_use_streaming_azure_openaitool_use_streaming/test_azure_openai.py›
    • llm-translation
    • messages
    • stream
    • azure
    • gpt-5-6-luna-azure-openai
    • gpt-5-6-sol-azure-openai
    • gpt-5-6-terra-azure-openai
    • function_calling
    1. Run Claude Code headless against gpt-5-6-sol-azure-openai, gpt-5-6-terra-azure-openai, gpt-5-6-luna-azure-openai in parallel through the proxy
    2. Run Claude Code headless against gpt-5-6-sol-azure-openai through the proxy
    3. Run Claude Code headless against gpt-5-6-terra-azure-openai through the proxy
    4. Run Claude Code headless against gpt-5-6-luna-azure-openai through the proxy
    View test on GitHub ↗
  790. ✓test_tool_use_streaming_bedrock_conversetool_use_streaming/test_bedrock_converse.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  791. ✓test_tool_use_streaming_bedrock_invoketool_use_streaming/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • stream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  792. ✓test_tool_use_streaming_vertex_aitool_use_streaming/test_vertex_ai.py›
    • llm-translation
    • messages
    • stream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  793. ✓test_tool_use_streaming_vertex_ai_gpttool_use_streaming/test_vertex_ai_gpt.py›
    • llm-translation

    No steps recorded

    View test on GitHub ↗
  794. ✓test_tool_use_vertex_aitool_use/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • function_calling
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  795. ✓test_tool_use_vertex_ai_gpttool_use/test_vertex_ai_gpt.py›
    • llm-translation

    No steps recorded

    View test on GitHub ↗
  796. ✓test_vision_anthropicvision/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • vision
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  797. ✓test_vision_azurevision/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • vision
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  798. ✓test_vision_bedrock_conversevision/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • vision
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  799. ✓test_vision_bedrock_invokevision/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • vision
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  800. ✓test_vision_vertex_aivision/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • vision
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  801. ✓test_web_search_anthropicweb_search/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5
    • web_search
    1. Run Claude Code headless against claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-7 in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5 through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5 through the proxy
    4. Run Claude Code headless against claude-opus-4-7 through the proxy
    View test on GitHub ↗
  802. ✓test_web_search_azureweb_search/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • web_search
    1. Run Claude Code headless against claude-haiku-4-5-azure, claude-sonnet-4-5-azure, claude-opus-4-7-azure in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-azure through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-azure through the proxy
    4. Run Claude Code headless against claude-opus-4-7-azure through the proxy
    View test on GitHub ↗
  803. ✓test_web_search_bedrock_converseweb_search/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-converse
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse
    • web_search
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-converse, claude-sonnet-4-5-bedrock-converse, claude-opus-4-7-bedrock-converse in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-converse through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-converse through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-converse through the proxy
    View test on GitHub ↗
  804. ✓test_web_search_bedrock_invokeweb_search/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-haiku-4-5-bedrock-invoke
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke
    • web_search
    1. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke, claude-sonnet-4-5-bedrock-invoke, claude-opus-4-7-bedrock-invoke in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-bedrock-invoke through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-bedrock-invoke through the proxy
    4. Run Claude Code headless against claude-opus-4-7-bedrock-invoke through the proxy
    View test on GitHub ↗
  805. ✓test_web_search_vertex_aiweb_search/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • web_search
    1. Run Claude Code headless against claude-haiku-4-5-vertex, claude-sonnet-4-5-vertex, claude-opus-4-7-vertex in parallel through the proxy
    2. Run Claude Code headless against claude-haiku-4-5-vertex through the proxy
    3. Run Claude Code headless against claude-sonnet-4-5-vertex through the proxy
    4. Run Claude Code headless against claude-opus-4-7-vertex through the proxy
    View test on GitHub ↗
  806. −test_basic_messaging_non_streaming_bedrock_mantlebasic_messaging_non_streaming/test_bedrock_mantle.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock_mantle
    • gpt-5-6-luna-bedrock-mantle
    • gpt-5-6-sol-bedrock-mantle
    • gpt-5-6-terra-bedrock-mantle

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  807. −test_basic_messaging_non_streaming_openaibasic_messaging_non_streaming/test_openai.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • gpt-5-6-luna-openai
    • gpt-5-6-sol-openai
    • gpt-5-6-terra-openai

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  808. −test_basic_messaging_streaming_bedrock_mantlebasic_messaging_streaming/test_bedrock_mantle.py›
    • llm-translation
    • messages
    • stream
    • bedrock_mantle
    • gpt-5-6-luna-bedrock-mantle
    • gpt-5-6-sol-bedrock-mantle
    • gpt-5-6-terra-bedrock-mantle

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  809. −test_basic_messaging_streaming_openaibasic_messaging_streaming/test_openai.py›
    • llm-translation
    • messages
    • stream
    • openai
    • gpt-5-6-luna-openai
    • gpt-5-6-sol-openai
    • gpt-5-6-terra-openai

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  810. −test_count_tokens_vertex_aicount_tokens/test_vertex_ai.py›
    • llm-translation
    • count_tokens
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex

    No steps recorded

    stage red: Vertex returns not supported for token counting for Claude aliases

    View test on GitHub ↗
  811. −test_long_context_1m_anthropiclong_context_1m/test_anthropic.py›
    • llm-translation
    • messages
    • nonstream
    • anthropic
    • claude-opus-4-7
    • claude-sonnet-4-5

    No steps recorded

    stage red: 1M long_context not green on stage Anthropic path yet (200k sonnet / model alias)

    View test on GitHub ↗
  812. −test_long_context_1m_azurelong_context_1m/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure

    No steps recorded

    stage red: 1M long_context not green on stage Azure Foundry deployments yet

    View test on GitHub ↗
  813. −test_long_context_1m_bedrock_converselong_context_1m/test_bedrock_converse.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-opus-4-7-bedrock-converse
    • claude-sonnet-4-5-bedrock-converse

    No steps recorded

    stage red: 1M long_context not green on stage Bedrock Converse deployments yet

    View test on GitHub ↗
  814. −test_long_context_1m_bedrock_invokelong_context_1m/test_bedrock_invoke.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock
    • claude-opus-4-7-bedrock-invoke
    • claude-sonnet-4-5-bedrock-invoke

    No steps recorded

    stage red: 1M long_context not green on stage Bedrock Invoke deployments yet

    View test on GitHub ↗
  815. −test_long_context_1m_vertex_ailong_context_1m/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex

    No steps recorded

    stage red: 1M long_context not green on stage Vertex deployments yet

    View test on GitHub ↗
  816. −test_passthrough_azurepassthrough/test_azure.py›
    • passthrough
    • passthrough
    • stream
    • azure
    • claude-haiku-4-5
    • claude-opus-4-7
    • claude-sonnet-4-5

    No steps recorded

    stage red: /azure passthrough drops client headers (e.g. anthropic-version); product gap

    View test on GitHub ↗
  817. −test_tool_search_azuretool_search/test_azure.py›
    • llm-translation
    • messages
    • nonstream
    • azure_ai
    • claude-haiku-4-5-azure
    • claude-opus-4-7-azure
    • claude-sonnet-4-5-azure
    • tool_search

    No steps recorded

    stage red: Azure Foundry tool_search_server not supported in workspace for probed models

    View test on GitHub ↗
  818. −test_tool_search_vertex_aitool_search/test_vertex_ai.py›
    • llm-translation
    • messages
    • nonstream
    • vertex_ai
    • claude-haiku-4-5-vertex
    • claude-opus-4-7-vertex
    • claude-sonnet-4-5-vertex
    • tool_search

    No steps recorded

    stage red: Vertex rejects tool_search when deployment extra_headers inject context-1m beta; product/config

    View test on GitHub ↗
  819. −test_tool_use_bedrock_mantletool_use/test_bedrock_mantle.py›
    • llm-translation
    • messages
    • nonstream
    • bedrock_mantle
    • gpt-5-6-luna-bedrock-mantle
    • gpt-5-6-sol-bedrock-mantle
    • gpt-5-6-terra-bedrock-mantle
    • function_calling

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  820. −test_tool_use_openaitool_use/test_openai.py›
    • llm-translation
    • messages
    • nonstream
    • openai
    • gpt-5-6-luna-openai
    • gpt-5-6-sol-openai
    • gpt-5-6-terra-openai
    • function_calling

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  821. −test_tool_use_streaming_bedrock_mantletool_use_streaming/test_bedrock_mantle.py›
    • llm-translation
    • messages
    • stream
    • bedrock_mantle
    • gpt-5-6-luna-bedrock-mantle
    • gpt-5-6-sol-bedrock-mantle
    • gpt-5-6-terra-bedrock-mantle
    • function_calling

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  822. −test_tool_use_streaming_openaitool_use_streaming/test_openai.py›
    • llm-translation
    • messages
    • stream
    • openai
    • gpt-5-6-luna-openai
    • gpt-5-6-sol-openai
    • gpt-5-6-terra-openai
    • function_calling

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  823. ✓test_batch_create_over_rpm_returns_mapped_429 TestBatchRateLimitErrorMappingtest_batches_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-batch-rl-map-33bd1f62394b, tpm limit: 1000000 and rpm limit: 2
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    View test on GitHub ↗
  824. ✓test_batch_key_model_access_deniedtest_batches_e2e.py›
    • proxy-auth
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with models: openai-batch-489058c47ca2 and user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    View test on GitHub ↗
  825. ✓test_batch_lifecycle[azure-encoded]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  826. ✓test_batch_lifecycle[azure-model_param]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  827. ✓test_batch_lifecycle[azure-provider_fallback]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  828. ✓test_batch_lifecycle[azure-unified]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  829. ✓test_batch_lifecycle[bedrock-unified]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  830. ✓test_batch_lifecycle[openai-encoded]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Call the management route /health/liveliness
    2. Add a batch deployment named openai-batch-489058c47ca2 that calls openai/gpt-4o-mini
    3. Add a batch deployment named azure-batch-489058c47ca2 that calls azure/gpt-5.4-mini-batch
    4. Add a batch deployment named vertex-batch-489058c47ca2 that calls vertex_ai/gemini-2.5-flash
    5. Add a batch deployment named bedrock-batch-489058c47ca2 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    6. Generate a virtual key with user id: e2e-test-user
    7. Upload a batch input file to /v1/files
    8. Create a batch of /v1/chat/completions requests from the uploaded file
    9. Retrieve the batch
    10. Cancel the batch
    11. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  831. ✓test_batch_lifecycle[openai-model_param]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  832. ✓test_batch_lifecycle[openai-provider_fallback]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  833. ✓test_batch_lifecycle[openai-unified]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Cancel the batch
    6. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  834. ✓test_batch_lifecycle[vertex_ai-encoded]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  835. ✓test_batch_lifecycle[vertex_ai-model_param]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  836. ✓test_batch_lifecycle[vertex_ai-provider_fallback]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  837. ✓test_batch_lifecycle[vertex_ai-unified]test_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  838. ✓test_completed_batch_downloads_output_and_books_cost TestBatchTerminalStatetest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. List the batches the key can see from /v1/batches
    View test on GitHub ↗
  839. ✓test_endpoint_mismatch_fails_batch_and_cancel_conflicts TestBatchFailurePathstest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    5. Wait for the request's spend log in /spend/logs
    6. Cancel the batch
    View test on GitHub ↗
  840. ✓test_enqueued_allowance_accepts_batch_over_key_rpm TestBatchEnqueuedTokenLimittest_batches_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-batch-enq-rpm-582031f37080, rpm limit: 2 and metadata: batch enqueued token limit: 100000
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    View test on GitHub ↗
  841. ✓test_exhausted_allowance_blocks_until_cancel_refunds TestBatchEnqueuedTokenLimittest_batches_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-batch-enq-size-6ed4d4b30075 and metadata: batch enqueued token limit: 1
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Generate a virtual key with user id: e2e-batch-enq-refund-c254b9e6d0f6 and metadata: batch enqueued token limit: 36
    5. Upload a batch input file to /v1/files
    6. Create a batch of /v1/chat/completions requests from the uploaded file
    7. Cancel the batch
    8. Create a batch of /v1/chat/completions requests from the uploaded file
    View test on GitHub ↗
  842. ✓test_file_content_matches_upload TestBatchFileContenttest_batches_e2e.py›
    • llm-translation
    • files
    • openai
    • gpt-4o-mini
    1. Start a provider edge server
    2. Add a batch deployment named e2e-file-content-ea79194c0fa1 that calls openai/gpt-4o-mini
    3. Generate a virtual key with user id: e2e-test-user
    4. Upload a batch input file to /v1/files
    5. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLDllYmYyM2M1LWNjYmQtNDllMC04NDZhLTllMjE5MDRiYjdiNzt0YXJnZXRfbW9kZWxfbmFtZXMsZTJlLWZpbGUtY29udGVudC1lYTc5MTk0YzBmYTE7bGxtX291d
    View test on GitHub ↗
  843. ✓test_file_lifecycle_signs_s3_with_s3_credentials TestBedrockBatchSplitS3Credentialstest_batches_e2e.py›
    • llm-translation
    • files
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a batch deployment named bedrock-split-s3-batch-489058c47ca2 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload a batch input file to /v1/files
    4. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLDIzYTU0ZTdhLTkxZTctNDA5MS04YjA4LTU5NWNhMzU4ZWY3NDt0YXJnZXRfbW9kZWxfbmFtZXMsYmVkcm9jay1zcGxpdC1zMy1iYXRjaC00ODkwNThjNDdjYTI7b
    5. Delete the uploaded file
    View test on GitHub ↗
  844. ✓test_file_upload_and_delete_outputstest_batches_e2e.py›
    • llm-translation
    • files
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Delete the uploaded file
    View test on GitHub ↗
  845. ✓test_foreign_encoded_file_id_routes_by_file_model TestBatchFailurePathstest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Create a batch of /v1/chat/completions requests from the uploaded file
    4. Retrieve the batch
    View test on GitHub ↗
  846. ✓test_gemini_file_upload TestGeminiFilestest_batches_e2e.py›
    • llm-translation
    • files
    • gemini
    • gemini-2.5-flash
    1. Add a batch deployment named gemini-files-489058c47ca2 that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload a batch input file to /v1/files
    View test on GitHub ↗
  847. ✓test_list_page_cursors_address_only_the_callers_own_files TestOpenAIFilestest_batches_e2e.py›
    • llm-translation
    • files
    • openai
    1. Generate a virtual key with user id: e2e-file-list-2839635ebb31
    2. List the files the key can see from /v1/files
    View test on GitHub ↗
  848. ✓test_malformed_jsonl_upload_rejected TestBatchFailurePathstest_batches_e2e.py›
    • llm-translation
    • files
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    View test on GitHub ↗
  849. ✓test_native_jsonl_round_trips_untouched_and_starts_a_batch TestVertexNativePassthroughtest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLDFiYmFlMTcyLTcyMWQtNGFkMi1hYWU1LTQ0MWMxZTNmOGFlZTt0YXJnZXRfbW9kZWxfbmFtZXMsdmVydGV4LWJhdGNoLTQ4OTA1OGM0N2NhMjtsbG1fb3V0cHV0X
    4. Create a batch of /v1/chat/completions requests from the uploaded file
    View test on GitHub ↗
  850. ✓test_passthrough_upload_is_rejected_outside_a_native_vertex_batch[no-target-model] TestVertexNativePassthroughtest_batches_e2e.py›
    • llm-translation
    • files
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    View test on GitHub ↗
  851. ✓test_passthrough_upload_is_rejected_outside_a_native_vertex_batch[non-vertex-target-model] TestVertexNativePassthroughtest_batches_e2e.py›
    • llm-translation
    • files
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    View test on GitHub ↗
  852. ✓test_passthrough_upload_is_rejected_outside_a_native_vertex_batch[openai-shaped-rows] TestVertexNativePassthroughtest_batches_e2e.py›
    • llm-translation
    • files
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    View test on GitHub ↗
  853. ✓test_retrieve_round_trips_metadata TestOpenAIFilestest_batches_e2e.py›
    • llm-translation
    • files
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Retrieve the uploaded file
    View test on GitHub ↗
  854. ✓test_unified_batch_create_with_assume_role TestBedrockBatchAssumeRoletest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Add a batch deployment named bedrock-sts-batch-489058c47ca2 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload a batch input file to /v1/files
    4. Create a batch of /v1/chat/completions requests from the uploaded file
    5. Retrieve the batch
    View test on GitHub ↗
  855. ✓test_unified_create_and_retrieve_via_chained_gateway TestBatchSecondHoptest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • litellm_proxy
    • openai
    • gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a batch deployment named openai-batch-hop-489058c47ca2 that calls litellm_proxy/openai-batch-489058c47ca2
    3. Upload a batch input file to /v1/files
    4. Create a batch of /v1/chat/completions requests from the uploaded file
    5. Retrieve the batch
    View test on GitHub ↗
  856. ✓test_unified_file_content_downloads[azure] TestBatchFileContenttest_batches_e2e.py›
    • llm-translation
    • files
    • azure
    • gpt-5.4-mini-batch
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLDdiZGFjOTc3LTEzZDYtNGY0OC1hNWRjLWVmOTMwM2YzMDMxNzt0YXJnZXRfbW9kZWxfbmFtZXMsYXp1cmUtYmF0Y2gtNDg5MDU4YzQ3Y2EyO2xsbV9vdXRwdXRfZ
    View test on GitHub ↗
  857. ✓test_unified_file_content_downloads[bedrock] TestBatchFileContenttest_batches_e2e.py›
    • llm-translation
    • files
    • bedrock
    • bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLDBmMzVkYWU2LTA2ZGItNDA1Mi05ODllLTE4OWUxZWVkMjU1Nzt0YXJnZXRfbW9kZWxfbmFtZXMsYmVkcm9jay1iYXRjaC00ODkwNThjNDdjYTI7bGxtX291dHB1d
    View test on GitHub ↗
  858. ✓test_unified_file_content_downloads[vertex_ai] TestBatchFileContenttest_batches_e2e.py›
    • llm-translation
    • files
    • vertex_ai
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Upload a batch input file to /v1/files
    3. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLGQ3NDQ0NTUyLWFjMDctNGYwZC1hNzQ0LTIzMmYxYzNkMWExZDt0YXJnZXRfbW9kZWxfbmFtZXMsdmVydGV4LWJhdGNoLTQ4OTA1OGM0N2NhMjtsbG1fb3V0cHV0X
    View test on GitHub ↗
  859. ✓test_unified_file_upload_and_batch_create_in_govcloud TestBedrockBatchGovCloudtest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • bedrock
    • bedrock/amazon.nova-lite-v1:0
    1. Add a batch deployment named bedrock-govcloud-batch-489058c47ca2 that calls bedrock/amazon.nova-lite-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Upload a batch input file to /v1/files
    4. Download /v1/files/bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9qc29ubDt1bmlmaWVkX2lkLGJjOTM0YTY5LTZmMzItNDE0ZC04NzgyLTQzZDFlNjExMmU5ZTt0YXJnZXRfbW9kZWxfbmFtZXMsYmVkcm9jay1nb3ZjbG91ZC1iYXRjaC00ODkwNThjNDdjYTI7b
    5. Create a batch of /v1/chat/completions requests from the uploaded file
    6. Retrieve the batch
    View test on GitHub ↗
  860. −test_batch_runs_to_completion_with_a_downloadable_output[model_query] TestHostedVllmBatchtest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • hosted_vllm
    • Qwen/Qwen2.5-0.5B-Instruct

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  861. −test_batch_runs_to_completion_with_a_downloadable_output[target_model_names] TestHostedVllmBatchtest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • hosted_vllm
    • Qwen/Qwen2.5-0.5B-Instruct

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  862. −test_failing_line_lands_in_the_error_file_not_the_batch_status TestHostedVllmBatchtest_batches_e2e.py›
    • llm-translation
    • batches
    • batch
    • hosted_vllm
    • Qwen/Qwen2.5-0.5B-Instruct

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  863. −test_rate_limited_batch_create_leaves_no_unattributed_spend_rowtest_batches_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • gpt-4o-mini

    No steps recorded

    LIT-5027: the path under test hangs. The batch rate limiter reads the input file to count tokens by awaiting litellm.afile_content with no timeout, so a slow Files API holds POST /v1/batches open past any client deadline (63.6s observed on stage against a 60s read timeout). The unattributed-spend-row contract below is never reached, so the test reports a timeout rather than the behavior it guards. Unskip once the fetch is bounded.

    View test on GitHub ↗
  864. −test_uploaded_file_appears_in_list TestOpenAIFilestest_batches_e2e.py›
    • llm-translation
    • files
    • openai
    • gpt-4o-mini

    No steps recorded

    LIT-4820 (https://linear.app/litellm-ai/issue/LIT-4820): GET /v1/files omits newly uploaded files. The upload succeeds and GET /v1/files/{id} returns the file, but it never appears in the listing; the returned set is stable with its newest entry ~10h old, on both the managed (/v1/files?model=) and provider-scoped (/openai/v1/files) routes. Skipped rather than weakened because the assertion below is the correct contract. Remove this marker when LIT-4820 is fixed; do not relax the assertion to make it pass.

    View test on GitHub ↗
  865. ✓test_429_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    • routing
    • nonstream
    • openai
    • gpt-5.5
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Generate a virtual key with models: gpt-5.5, user id: e2e-test-user and rpm limit: 1
    3. Add a deployment named reliability-retry-429-5b3bdc4c580a that fronts gpt-5.5 on this proxy with a key out of rpm, so it always gets a 429 and is benched on the first one
    4. Add a zero-weight backup deployment named reliability-retry-429-5b3bdc4c580a that calls openai/gpt-5.5
    5. Use up the rpm-limited key's one allowed request with a /chat/completions call to gpt-5.5
    6. Send a /chat/completions request to reliability-retry-429-5b3bdc4c580a with stream set to False
    View test on GitHub ↗
  866. ✓test_429_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    • routing
    • nonstream
    • openai
    • gpt-5.5
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Generate a virtual key with models: gpt-5.5, user id: e2e-test-user and rpm limit: 1
    3. Add a deployment named reliability-cooldown-429-3b9f06652290 that fronts gpt-5.5 on this proxy with a key out of rpm, so it always gets a 429 and is benched on the first one
    4. Add a zero-weight backup deployment named reliability-cooldown-429-3b9f06652290 that calls openai/gpt-5.5
    5. Use up the rpm-limited key's one allowed request with a /chat/completions call to gpt-5.5
    6. Send a /chat/completions request to reliability-cooldown-429-3b9f06652290 with stream set to False
    View test on GitHub ↗
  867. ✓test_5xx_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-5xx-upstream-757e01d11bbc that calls openai/gpt-5.5 at an unreachable address
    3. Add a deployment named reliability-retry-5xx-93ce51b457eb that fronts reliability-5xx-upstream-757e01d11bbc on this proxy, so it always gets a 500 and is benched on the first one
    4. Add a zero-weight backup deployment named reliability-retry-5xx-93ce51b457eb that calls openai/gpt-5.5
    5. Send a /chat/completions request to reliability-retry-5xx-93ce51b457eb with stream set to False
    View test on GitHub ↗
  868. ✓test_5xx_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    • routing
    • nonstream
    • openai
    • gpt-5.5
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-fail-25eb84cb8d6c that calls openai/gpt-5.5 at an unreachable address
    3. Send a /chat/completions request to reliability-fail-25eb84cb8d6c with stream set to False
    View test on GitHub ↗
  869. ✓test_5xx_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cooldown-5xx-upstream-a9f141b7e41d that calls openai/gpt-5.5 at an unreachable address
    3. Add a deployment named reliability-cooldown-5xx-608d887dd992 that fronts reliability-cooldown-5xx-upstream-a9f141b7e41d on this proxy, so it always gets a 500 and is benched on the first one
    4. Add a zero-weight backup deployment named reliability-cooldown-5xx-608d887dd992 that calls openai/gpt-5.5
    5. Send a /chat/completions request to reliability-cooldown-5xx-608d887dd992 with stream set to False
    View test on GitHub ↗
  870. ✓test_alias_api_key_never_overrides_the_tier_credential TestAliasParamForwardingtest_auto_router_regressions_e2e.py›
    • routing
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-cred-tier-07aa62b98762 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named e2e-cred-alias-07aa62b98762 that calls auto_router/complexity_router
    3. Generate a virtual key with models: e2e-cred-alias-07aa62b98762, e2e-cred-tier-07aa62b98762 and user id: e2e-auto-router-regressions
    4. Send a /chat/completions request to e2e-cred-alias-07aa62b98762
    5. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  871. ✓test_auth_failure_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-retry-auth-dc969663423a that calls openai/gpt-5.5 with an invalid key and is benched on its first 401
    3. Add a zero-weight backup deployment named reliability-retry-auth-dc969663423a that calls openai/gpt-5.5
    4. Send a /chat/completions request to reliability-retry-auth-dc969663423a with stream set to False
    View test on GitHub ↗
  872. ✓test_auth_failure_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cooldown-auth-8fee1b417992 that calls openai/gpt-5.5 with an invalid key and is benched on its first 401
    3. Add a zero-weight backup deployment named reliability-cooldown-auth-8fee1b417992 that calls openai/gpt-5.5
    4. Send a /chat/completions request to reliability-cooldown-auth-8fee1b417992 with stream set to False
    View test on GitHub ↗
  873. ✓test_body_tagged_chat_reaches_the_untagged_tier_after_marker_rewrite TestUntaggedTierDeploymentstest_auto_router_regressions_e2e.py›
    • routing
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with models: e2e-autoroute-7b341ab08c94, e2e-tier-7b341ab08c94, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /chat/completions request to e2e-autoroute-7b341ab08c94
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  874. ✓test_body_tagged_chat_routes_through_the_marker_to_its_tier TestTagSplitRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-autoroute-8584bc433a52 that calls anthropic/claude-sonnet-5
    2. Add a deployment named e2e-tier-8584bc433a52 that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-autoroute-8584bc433a52 that calls auto_router/complexity_router
    4. Generate a virtual key with models: e2e-autoroute-8584bc433a52, e2e-tier-8584bc433a52, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    5. Send a /chat/completions request to e2e-autoroute-8584bc433a52
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  875. ✓test_body_tagged_responses_with_list_input_routes_to_the_tier TestResponsesApiTagRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with models: e2e-autoroute-b7fbe64d18a5, e2e-tier-b7fbe64d18a5, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. POST /v1/responses
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  876. ✓test_cached_conversation_stays_on_deployment_holding_its_cache TestReliabilityPromptCachingAffinitytest_reliability_prompt_caching_e2e.py›
    • routing
    • nonstream
    • anthropic
    • openai
    • anthropic/claude-haiku-4-5
    • openai/gpt-5.5
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Read the router settings from /router/settings
    3. Add a deployment named reliability-cache-9a7bb58c2f6f that calls anthropic/claude-haiku-4-5 with prompt caching
    4. Send a /chat/completions request to reliability-cache-9a7bb58c2f6f with a full message history and stream set to False
    5. Add a deployment named reliability-cache-9a7bb58c2f6f that calls openai/gpt-5.5
    6. Send a /chat/completions request to reliability-cache-9a7bb58c2f6f with a full message history and stream set to False
    View test on GitHub ↗
  877. ✓test_client_hanging_up_never_benches_the_deployment TestReliabilityCancelOnDisconnecttest_reliability_cancel_on_disconnect_e2e.py›
    • routing
    • nonstream
    • azure
    • azure/gpt-5.4-nano
    1. Generate a virtual key with user id: e2e-test-user
    2. Check whether the general setting cancel_on_disconnect is on
    3. Add a deployment named reliability-cooldown-disconnect-db1b12feb50f that calls azure/gpt-5.4-nano and is benched for 300.0s on its first failure
    4. Add a zero-weight backup deployment named reliability-cooldown-disconnect-db1b12feb50f that calls openai/gpt-5.5
    5. Send a /chat/completions request to reliability-cooldown-disconnect-db1b12feb50f with stream set to False
    6. Abandon the request to /chat/completions after 5.0s
    7. Send a /chat/completions request to reliability-cooldown-disconnect-db1b12feb50f with stream set to False
    View test on GitHub ↗
  878. ✓test_content_policy_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    • routing
    • nonstream
    • azure
    • openai
    • azure/gpt-5.4-nano
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-policyfail-27e9158ea6e9 that calls azure/gpt-5.4-nano behind Azure's content filter
    3. Send a /chat/completions request to reliability-policyfail-27e9158ea6e9 with stream set to False
    View test on GitHub ↗
  879. ✓test_context_window_refusal_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-3.5-turbo
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-retry-context-46bddfa1257a that calls openai/gpt-3.5-turbo and takes all of its group's traffic
    3. Add a zero-weight backup deployment named reliability-retry-context-46bddfa1257a that calls openai/gpt-5.5
    4. Send a /chat/completions request to reliability-retry-context-46bddfa1257a with stream set to False
    View test on GitHub ↗
  880. ✓test_context_window_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    • routing
    • nonstream
    • openai
    • gpt-5.5
    • openai/gpt-3.5-turbo
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-ctxfail-1073166c13b8 that calls the small-context model openai/gpt-3.5-turbo
    3. Send a /chat/completions request to reliability-ctxfail-1073166c13b8 with stream set to False
    View test on GitHub ↗
  881. ✓test_cost_based_picks_cheapest_deployment TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cost-a90ece812a47 that calls openai/gpt-5.5
    3. Send a /chat/completions request to reliability-cost-a90ece812a47 with stream set to False
    View test on GitHub ↗
  882. ✓test_exact_cache_returns_cached TestReliabilityCachetest_reliability_cache_e2e.py›
    • caching
    • nonstream
    • openai
    • openai/gpt-5.6
    1. Generate a virtual key with user id: e2e-test-user
    2. Run an observed provider edge server
    3. Add a deployment named e2e-cache-eb490112dea0 that calls openai/gpt-5.6
    4. Send a request to /chat/completions
    View test on GitHub ↗
  883. ✓test_failing_requests_do_not_grow_rss_or_stored_request TestReliabilityMemorytest_reliability_memory_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-memory-f23d7a643a44 that calls openai/gpt-5.5 at an unreachable address and is never benched
    3. Add a deployment named reliability-memory-fb-f23d7a643a44 that calls openai/gpt-5.5 at an unreachable address and is never benched
    4. Send a /chat/completions request to reliability-memory-f23d7a643a44 with stream set to False
    5. Wait for the request's spend log in /spend/logs
    6. Read memory usage from /debug/memory/summary on every proxy replica
    7. Send a /chat/completions request to reliability-memory-f23d7a643a44 with stream set to False
    8. Read memory usage from /debug/memory/summary on every proxy replica
    9. Send a /chat/completions request to reliability-memory-f23d7a643a44 with stream set to False
    10. Read memory usage from /debug/memory/summary on every proxy replica
    11. Send a /chat/completions request to reliability-memory-f23d7a643a44 with stream set to False
    12. Read memory usage from /debug/memory/summary on every proxy replica
    13. Send a /chat/completions request to reliability-memory-f23d7a643a44 with stream set to False
    14. Read memory usage from /debug/memory/summary on every proxy replica
    View test on GitHub ↗
  884. ✓test_header_tagged_messages_routes_through_the_marker_to_an_untagged_tier TestUntaggedTierDeploymentstest_auto_router_regressions_e2e.py›
    • routing
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-autoroute-7b341ab08c94 that calls auto_router/complexity_router
    2. Add a deployment named e2e-tier-7b341ab08c94 that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-autoroute-7b341ab08c94 that calls anthropic/claude-sonnet-5
    4. Generate a virtual key with models: e2e-autoroute-7b341ab08c94, e2e-tier-7b341ab08c94, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    5. POST /v1/messages
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  885. ✓test_header_tagged_responses_with_string_input_routes_to_the_tier TestResponsesApiTagRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-autoroute-b7fbe64d18a5 that calls anthropic/claude-sonnet-5
    2. Add a deployment named e2e-tier-b7fbe64d18a5 that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-autoroute-b7fbe64d18a5 that calls auto_router/complexity_router
    4. Generate a virtual key with models: e2e-autoroute-b7fbe64d18a5, e2e-tier-b7fbe64d18a5, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    5. POST /v1/responses
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  886. ✓test_request_timeout_exceeds_deadline TestReliabilityTimeoutstest_reliability_timeouts_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-timeout-2f14bdc12af0 that calls openai/gpt-5.5 with a 1ms timeout
    3. Send a /chat/completions request to reliability-timeout-2f14bdc12af0 with stream set to False
    View test on GitHub ↗
  887. ✓test_responses_input_reaches_the_semantic_auto_router TestSemanticAutoRouterResponsestest_auto_router_regressions_e2e.py›
    • routing
    • responses
    • nonstream
    • anthropic
    • openai
    • anthropic/claude-haiku-4-5
    • openai/text-embedding-3-small
    1. Add a deployment named e2e-semantic-embedding-6483565a8249 that calls openai/text-embedding-3-small
    2. Add a deployment named e2e-semantic-target-6483565a8249 that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-semantic-fallback-6483565a8249 that calls anthropic/claude-sonnet-5
    4. Add a deployment named e2e-semantic-router-6483565a8249 that calls auto_router/e2e-semantic-router-6483565a8249
    5. Generate a virtual key with models: e2e-semantic-router-6483565a8249, e2e-semantic-target-6483565a8249, e2e-semantic-fallback-6483565a8249 and user id: e2e-auto-router-regressions
    6. POST /v1/responses
    7. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  888. ✓test_sibling_replica_serves_backup_within_redis_read_interval TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cooldown-sibling-upstream-152100d4f3d5 that calls openai/gpt-5.5 at an unreachable address
    3. Add a deployment named reliability-cooldown-sibling-b60c72f84487 that fronts reliability-cooldown-sibling-upstream-152100d4f3d5 on this proxy, so it always gets a 500 and is benched on the first one
    4. Add a zero-weight backup deployment named reliability-cooldown-sibling-b60c72f84487 that calls openai/gpt-5.5
    5. Add a deployment named reliability-cooldown-sibling-warm-f016bb939ae3 for openai/gpt-5.5 that answers with a canned reply
    6. Send a /chat/completions request to reliability-cooldown-sibling-warm-f016bb939ae3 with stream set to False
    7. Send a /chat/completions request to reliability-cooldown-sibling-b60c72f84487 with stream set to False
    View test on GitHub ↗
  889. ✓test_simple_shuffle_honors_weights TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-shuffle-907d2c92228f that calls openai/gpt-5.5
    3. Send a /chat/completions request to reliability-shuffle-907d2c92228f with stream set to False
    View test on GitHub ↗
  890. ✓test_stream_timeout_exceeds_deadline TestReliabilityTimeoutstest_reliability_timeouts_e2e.py›
    • routing
    • stream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-stream-timeout-c6ef5cb943bf that calls openai/gpt-5.5 with a 1ms timeout
    3. Send a /chat/completions request to reliability-stream-timeout-c6ef5cb943bf with stream set to True
    View test on GitHub ↗
  891. ✓test_tagged_call_straight_at_an_untagged_deployment_stays_denied TestUntaggedTierDeploymentstest_auto_router_regressions_e2e.py›
    • routing
    • chat_completions
    1. Generate a virtual key with models: e2e-tier-7b341ab08c94, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /chat/completions request to e2e-tier-7b341ab08c94
    View test on GitHub ↗
  892. ✓test_timeout_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-retry-b7dabc685044 that calls openai/gpt-5.5 with a 1ms timeout and is benched on its first timeout
    3. Add a zero-weight backup deployment named reliability-retry-b7dabc685044 that calls openai/gpt-5.5
    4. Send a /chat/completions request to reliability-retry-b7dabc685044 with stream set to False
    View test on GitHub ↗
  893. ✓test_timeout_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    • routing
    • nonstream
    • openai
    • gpt-5.5
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-tofail-a4380b5b5a48 that calls openai/gpt-5.5 with a 1ms timeout
    3. Send a /chat/completions request to reliability-tofail-a4380b5b5a48 with stream set to False
    View test on GitHub ↗
  894. ✓test_timeout_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cooldown-timeout-0ea4b964539e that calls openai/gpt-5.5 with a 1ms timeout and is benched on its first timeout
    3. Add a zero-weight backup deployment named reliability-cooldown-timeout-0ea4b964539e that calls openai/gpt-5.5
    4. Send a /chat/completions request to reliability-cooldown-timeout-0ea4b964539e with stream set to False
    View test on GitHub ↗
  895. ✓test_trivial_ask_behind_keyword_heavy_system_prompt_stays_on_the_cheap_tier TestComplexityHeuristicScopetest_auto_router_regressions_e2e.py›
    • routing
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-heuristic-cheap-b49fbf1985de that calls anthropic/claude-haiku-4-5
    2. Add a deployment named e2e-heuristic-strong-b49fbf1985de that calls openai/gpt-5.6
    3. Add a deployment named e2e-heuristic-router-b49fbf1985de that calls auto_router/complexity_router
    4. Generate a virtual key with models: e2e-heuristic-router-b49fbf1985de, e2e-heuristic-cheap-b49fbf1985de, e2e-heuristic-strong-b49fbf1985de and user id: e2e-auto-router-regressions
    5. Send a /chat/completions request to e2e-heuristic-router-b49fbf1985de
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  896. ✓test_untagged_chat_is_always_served_by_the_plain_deployment TestTagSplitRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5
    1. Generate a virtual key with models: e2e-autoroute-8584bc433a52, e2e-tier-8584bc433a52, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /chat/completions request to e2e-autoroute-8584bc433a52
    3. Wait for at least 5 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  897. ✓test_untagged_messages_is_served_by_the_plain_deployment TestTagSplitRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5
    1. Generate a virtual key with models: e2e-autoroute-8584bc433a52, e2e-tier-8584bc433a52, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /v1/messages request to e2e-autoroute-8584bc433a52
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  898. ✓test_untagged_responses_is_served_by_the_plain_deployment TestResponsesApiTagRoutingtest_auto_router_regressions_e2e.py›
    • routing
    • responses
    • nonstream
    • anthropic
    • anthropic/claude-sonnet-5
    1. Generate a virtual key with models: e2e-autoroute-b7fbe64d18a5, e2e-tier-b7fbe64d18a5, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. POST /v1/responses
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  899. ✓test_usage_based_picks_deployment_with_tpm_headroom TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-usage-27638a728836 that calls openai/gpt-5.5
    3. Send a /chat/completions request to reliability-usage-27638a728836 with stream set to False
    View test on GitHub ↗
  900. ✓test_workers_idle_under_rss_budget_before_traffic TestReliabilityMemorytest_reliability_memory_e2e.py›
    • deploy-ops

    No steps recorded

    View test on GitHub ↗
  901. ✓test_zero_priced_alias_still_logs_spend_at_the_tier_rate TestStrategyAliasPricingtest_auto_router_regressions_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Add a deployment named e2e-priced-tier-6c5ef5a2ecc3 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named e2e-priced-alias-6c5ef5a2ecc3 that calls auto_router/complexity_router
    3. Generate a virtual key with models: e2e-priced-alias-6c5ef5a2ecc3, e2e-priced-tier-6c5ef5a2ecc3 and user id: e2e-auto-router-regressions
    4. Send a /chat/completions request to e2e-priced-alias-6c5ef5a2ecc3
    5. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  902. −test_latency_based_routes_around_deployment_that_times_out TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    • routing
    • nonstream
    • openai
    • openai/gpt-5.5

    No steps recorded

    LIT-7682: since #40229 the per-request routing_strategy override runs without the latency sampler, so latency-based has no signal to route on

    View test on GitHub ↗
  903. −test_least_busy_avoids_deployment_with_request_in_flight TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    • routing
    • openai
    • openai/gpt-5.5

    No steps recorded

    LIT-7682: since #40229 the per-request routing_strategy override runs without the in-flight counter, so least-busy has no signal to route on

    View test on GitHub ↗
  904. −test_llm_classifier_runs_and_routes_by_semantic_tier TestComplexityRouterLlmClassifiertest_complexity_router_e2e.py›
    • routing
    • nonstream
    • anthropic
    • openai
    • anthropic/claude-haiku-4-5
    • openai/gpt-5.5

    No steps recorded

    product bug LIT-4521: LLM classifier returns SIMPLE for short hard prompts (e.g. Is P equal to NP?); re-enable when classifier tier quality is fixed

    View test on GitHub ↗
  905. ✓test_apply_guardrail_blocks_banned_and_allows_clean TestApplyGuardrailEndpointtest_apply_guardrail_e2e.py›
    • guardrails
    • guardrails
    1. Register the content filter guardrail e2e-apply-566b121d7602 that blocks prompts containing e2e-banned-53261f0aa777
    2. Apply the guardrail e2e-apply-566b121d7602 to a piece of text with /guardrails/apply_guardrail
    View test on GitHub ↗
  906. ✓test_bedrock_guardrail_triggered TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the Bedrock guardrail e2e-bedrock-guard-8f08e0a42c37
    3. Retry the call until a guardrail blocks it
    View test on GitHub ↗
  907. ✓test_bedrock_post_call_blocks_denied_model_output TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the bedrock guardrail e2e-bedrock-post-f8a7f9693bec with mode post_call
    3. Retry the call until a guardrail blocks it
    View test on GitHub ↗
  908. ✓test_bedrock_post_call_blocks_denied_streamed_output_and_passes_clean_streams TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the bedrock guardrail e2e-bedrock-post-stream-e8d8b59402ba with mode post_call
    3. Retry the streamed call until a guardrail blocks it
    4. Send a streaming /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  909. ✓test_bedrock_pre_call_blocks_harmful_prompt TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the Bedrock guardrail e2e-bedrock-guard-54fa21ff323e
    3. Retry the call until a guardrail blocks it
    View test on GitHub ↗
  910. ✓test_bedrock_pre_call_blocks_on_messages TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • messages
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the Bedrock guardrail e2e-bedrock-messages-8107ff037d7d
    3. Retry the streamed call until a guardrail blocks it
    View test on GitHub ↗
  911. ✓test_bedrock_pre_call_blocks_on_responses TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    • guardrails
    • responses
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the Bedrock guardrail e2e-bedrock-responses-d5c6e0bb43a8
    3. Retry the streamed call until a guardrail blocks it
    View test on GitHub ↗
  912. ✓test_blocks_execution_request_but_allows_explanation TestBlockCodeExecutionGuardrailtest_block_code_execution_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-blockcode-backend-86f669a76699 that calls gemini/gemini-2.5-flash
    3. Register the block_code_execution guardrail e2e-block-code-ffc7a443a44f with mode pre_call
    4. Send a /chat/completions request to e2e-blockcode-backend-86f669a76699
    View test on GitHub ↗
  913. ✓test_chat_blocks_category TestOpenAIModerationCategoryMatrixtest_openai_moderation_category_matrix_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-mod-cat-chat-8d3884f07063 that calls gemini/gemini-2.5-flash
    3. Register the openai_moderation guardrail e2e-mod-cat-chat-7c7ad7cf19fb with mode pre_call
    4. Send a /chat/completions request to e2e-mod-cat-chat-8d3884f07063
    View test on GitHub ↗
  914. ✓test_child_condition_match_applies_child_and_inherited_parent_guardrails TestPolicyInheritedGuardrailtest_policy_inherited_guardrail_e2e.py›
    • guardrails
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the content filter guardrail e2e-parent-guard-8379ee5d93c1 that blocks prompts containing 8379ee5d93c1
    3. Register the content filter guardrail e2e-child-guard-b93096bbf1af that blocks prompts containing b93096bbf1af
    4. Create the guardrail policy e2e-parent-policy-8379ee5d93c1 that adds the guardrails e2e-parent-guard-8379ee5d93c1
    5. Create the guardrail policy e2e-child-policy-b93096bbf1af that adds the guardrails e2e-child-guard-b93096bbf1af
    6. Attach the guardrail policy e2e-child-policy-b93096bbf1af to requests tagged e2e-policy-tag-caaaac49d56c
    7. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  915. ✓test_child_condition_miss_still_applies_inherited_parent_guardrail TestPolicyInheritedGuardrailtest_policy_inherited_guardrail_e2e.py›
    • guardrails
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the content filter guardrail e2e-parent-guard-db2404d1ee00 that blocks prompts containing db2404d1ee00
    3. Register the content filter guardrail e2e-child-guard-0b2c16576bca that blocks prompts containing 0b2c16576bca
    4. Create the guardrail policy e2e-parent-policy-db2404d1ee00 that adds the guardrails e2e-parent-guard-db2404d1ee00
    5. Create the guardrail policy e2e-child-policy-0b2c16576bca that adds the guardrails e2e-child-guard-0b2c16576bca
    6. Attach the guardrail policy e2e-child-policy-0b2c16576bca to requests tagged e2e-policy-tag-2745e2808409
    7. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  916. ✓test_during_call_blocks_stream_before_first_chunk TestBedrockDuringCallStreamingtest_streaming_guardrail_e2e.py›
    • guardrails
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the bedrock guardrail e2e-bedrock-during-478609fe54a2 with mode during_call
    3. Retry the streamed call until a guardrail blocks it
    View test on GitHub ↗
  917. ✓test_flag_returns_guardrail_information_for_the_guardrail_that_ran TestGuardrailInformationResponsetest_guardrail_information_response_e2e.py›
    • guardrails
    • nonstream
    • openai
    • openai/gpt-4.1-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the litellm_content_filter guardrail e2e-guardrail-information-19f52f42669e with mode pre_call
    3. Add a deployment named e2e-guardrail-info-backend-479861317241 that calls openai/gpt-4.1-mini
    4. Send a /chat/completions request to e2e-guardrail-info-backend-479861317241
    View test on GitHub ↗
  918. ✓test_global_guardrail_blocks_key_without_team_opt_out TestTeamDisableGlobalGuardrailtest_team_disable_global_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the content filter guardrail e2e-content-filter-5a4f97aeec12 that blocks prompts containing 5a4f97aeec12
    3. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  919. ✓test_key_attached_content_filter_blocks_banned_image_edit_prompt TestKeyAttachedGuardrailOnImageEditstest_key_guardrail_image_edit_e2e.py›
    • guardrails
    • images
    • nonstream
    • gemini
    • openai
    • gemini-2.5-flash
    • openai/gpt-image-2.5-flare
    1. Register the content filter guardrail e2e-image-edit-filter-e47cf6ea4b4e that blocks prompts containing e47cf6ea4b4e
    2. Generate a virtual key with the guardrails e2e-image-edit-filter-e47cf6ea4b4e
    3. Add a deployment named e2e-guard-image-edit-88f5c44b8184 that calls openai/gpt-image-2.5-flare
    4. Retry the call until a guardrail blocks it
    5. Send a /v1/images/edits request to e2e-guard-image-edit-88f5c44b8184
    View test on GitHub ↗
  920. ✓test_key_attached_content_filter_blocks_banned_video_prompt TestKeyAttachedGuardrailOnVideostest_key_guardrail_video_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • vertex_ai
    • gemini-2.5-flash
    • vertex_ai/veo-3.1-fast-generate-001
    1. Register the content filter guardrail e2e-video-filter-804edb3160c6 that blocks prompts containing 804edb3160c6
    2. Generate a virtual key with the guardrails e2e-video-filter-804edb3160c6
    3. Add a deployment named e2e-guard-video-b92e6c27d0be that calls vertex_ai/veo-3.1-fast-generate-001
    4. Retry the call until a guardrail blocks it
    5. Send a /v1/videos request to e2e-guard-video-b92e6c27d0be
    View test on GitHub ↗
  921. ✓test_masking_run_is_recorded_on_the_spend_log TestPresidioSpendLogRecordtest_presidio_masking_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-c5cde1724518 with mode pre_call
    3. Send a /chat/completions request to gemini-2.5-flash
    4. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  922. ✓test_messages_blocks_category TestOpenAIModerationCategoryMatrixtest_openai_moderation_category_matrix_e2e.py›
    • guardrails
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-mod-cat-msg-3ae306996c5f that calls anthropic/claude-haiku-4-5
    3. Register the openai_moderation guardrail e2e-mod-cat-msg-460670d86d32 with mode pre_call
    4. Send a /v1/messages request to e2e-mod-cat-msg-3ae306996c5f
    View test on GitHub ↗
  923. ✓test_moderation_blocks_flagged_input TestOpenAIModerationGuardrailtest_openai_moderation_guardrail_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-moderation-backend-74dd3aa6addf that calls gemini/gemini-2.5-flash
    3. Register the openai_moderation guardrail e2e-openai-moderation-6150b4a4dde7 with mode pre_call
    4. Retry the call until a guardrail blocks it
    5. Send a /chat/completions request to e2e-moderation-backend-74dd3aa6addf
    View test on GitHub ↗
  924. ✓test_moderation_blocks_flagged_input_on_messages TestOpenAIModerationGuardrailtest_openai_moderation_guardrail_e2e.py›
    • guardrails
    • messages
    • nonstream
    • gemini
    • gemini/gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-moderation-msg-backend-10d549f5c278 that calls gemini/gemini-2.5-flash
    3. Register the openai_moderation guardrail e2e-openai-moderation-msg-c5875652de83 with mode pre_call
    4. Retry the call until a guardrail blocks it
    5. Send a /v1/messages request to e2e-moderation-msg-backend-10d549f5c278
    View test on GitHub ↗
  925. ✓test_non_stream_block_is_schema_valid_json TestResponsesPreCallBlocktest_responses_pre_call_block_stream_e2e.py›
    • guardrails
    • responses
    • nonstream
    • openai
    • openai/gpt-4.1-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the custom_code guardrail e2e-custom-code-responses-block-1beccfca31f4 with mode pre_call
    3. Add a deployment named e2e-responses-block-20e3c9c4ccf1 that calls openai/gpt-4.1-mini
    4. Send a /v1/responses request to e2e-responses-block-20e3c9c4ccf1
    View test on GitHub ↗
  926. ✓test_post_call_masks_pii_in_model_output TestPresidioPostCallMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-post-chat-5402979c170a with mode post_call
    3. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  927. ✓test_pre_call_allows_permitted_tool TestToolPermissionPreCalltest_tool_permission_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the tool_permission guardrail e2e-toolperm-allow-3137c82e8f31 with mode pre_call
    3. Retry the call until the guardrail e2e-toolperm-allow-3137c82e8f31 is applied
    View test on GitHub ↗
  928. ✓test_pre_call_blocks_tool_outside_the_allow_list TestToolPermissionPreCalltest_tool_permission_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    • function_calling
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the tool_permission guardrail e2e-toolperm-block-dd0ea41a35b8 with mode pre_call
    3. Retry the call until a guardrail blocks it
    View test on GitHub ↗
  929. ✓test_pre_call_masks_pii_on_chat_completions TestPresidioPreCallMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-pre-chat-2c554300ee89 with mode pre_call
    3. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  930. ✓test_pre_call_masks_pii_on_messages TestPresidioPreCallMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • messages
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-pre-msg-bef4854b7feb with mode pre_call
    3. Send a /v1/messages request to gemini-2.5-flash
    View test on GitHub ↗
  931. ✓test_responses_blocks_category TestOpenAIModerationCategoryMatrixtest_openai_moderation_category_matrix_e2e.py›
    • guardrails
    • responses
    • nonstream
    • openai
    • openai/gpt-4o-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-mod-cat-resp-d413073004bc that calls openai/gpt-4o-mini
    3. Register the openai_moderation guardrail e2e-mod-cat-resp-ebe860874348 with mode pre_call
    4. Send a /v1/responses request to e2e-mod-cat-resp-d413073004bc
    View test on GitHub ↗
  932. ✓test_spend_log_stores_masked_output_on_anthropic_messages TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    • guardrails
    • messages
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-card-messages-53460f64c743 with mode post_call
    3. Send a /v1/messages request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  933. ✓test_spend_log_stores_masked_output_on_chat_completions TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-card-chat-03afc8c61130 with mode post_call
    3. Send a /chat/completions request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  934. ✓test_spend_log_stores_masked_output_on_responses TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    • guardrails
    • responses
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-card-responses-70b97319b29f with mode post_call
    3. Send a /v1/responses request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  935. ✓test_spend_log_stores_masked_output_on_streaming_anthropic_messages TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    • guardrails
    • messages
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-card-messages-stream-6079a2cc38e2 with mode post_call
    3. Send a streaming /v1/messages request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  936. ✓test_spend_log_stores_masked_output_on_streaming_chat_completions TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    • guardrails
    • chat_completions
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-log-card-chat-stream-9e14bdbc5f9d with mode post_call
    3. Send a streaming /chat/completions request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  937. ✓test_stream_block_is_sse_with_completed_assistant_message TestResponsesPreCallBlocktest_responses_pre_call_block_stream_e2e.py›
    • guardrails
    • responses
    • stream
    • openai
    • openai/gpt-4.1-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the custom_code guardrail e2e-custom-code-responses-block-4c26b7acb7b4 with mode pre_call
    3. Add a deployment named e2e-responses-block-307e74c84b0e that calls openai/gpt-4.1-mini
    4. Send a streaming /v1/responses request to e2e-responses-block-307e74c84b0e
    View test on GitHub ↗
  938. ✓test_team_with_disable_flag_bypasses_global_guardrail TestTeamDisableGlobalGuardrailtest_team_disable_global_guardrail_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Register the content filter guardrail e2e-content-filter-e1a6db12cec8 that blocks prompts containing e1a6db12cec8
    2. Create the team e2e-guardrail-optout-e1a6db12cec8 opted out of global guardrails and wait until /team/info returns it
    3. Generate a virtual key in the team
    4. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  939. ✓test_ui_default_scope_masks_a_card_number_the_model_generates_on_chat_completions TestPresidioCreditCardOutputMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-card-chat-44b86798035c with mode pre_call, post_call
    3. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  940. ✓test_ui_default_scope_masks_a_card_number_the_model_generates_on_streaming_anthropic_messages TestPresidioCreditCardOutputMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • messages
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-card-messages-stream-5e82d36e07d7 with mode pre_call, post_call
    3. Send a streaming /v1/messages request to gemini-2.5-flash
    View test on GitHub ↗
  941. ✓test_ui_default_scope_masks_a_card_number_the_model_generates_on_streaming_chat_completions TestPresidioCreditCardOutputMaskingtest_presidio_masking_e2e.py›
    • guardrails
    • chat_completions
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the presidio guardrail e2e-presidio-card-stream-e4f69b815782 with mode pre_call, post_call
    3. Send a streaming /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  942. ✓test_without_flag_response_has_no_guardrail_information TestGuardrailInformationResponsetest_guardrail_information_response_e2e.py›
    • guardrails
    • nonstream
    • openai
    • openai/gpt-4.1-mini
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the litellm_content_filter guardrail e2e-guardrail-information-default-7b9ed1424a55 with mode pre_call
    3. Add a deployment named e2e-guardrail-info-backend-91038bf2bd83 that calls openai/gpt-4.1-mini
    4. Send a /chat/completions request to e2e-guardrail-info-backend-91038bf2bd83
    View test on GitHub ↗
  943. −test_request_naming_an_unknown_guardrail_fails_closedtest_guardrail_dispatch_e2e.py›
    • guardrails
    • nonstream
    • gemini
    • gemini-2.5-flash

    No steps recorded

    stage red: product gap, a request naming a guardrail the proxy does not serve is silently served unguarded (200) instead of failing closed

    View test on GitHub ↗
  944. ✓test_chat_completions_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • chat_completions
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-chat-9f6608a8b6a4 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for DataDog to ingest logs carrying the marker 3c6b61fbd7e3, then watch for duplicates
    View test on GitHub ↗
  945. ✓test_chat_completions_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • chat_completions
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-trace-chat-d2d4319dd45d with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call b89d345a-673b-4095-980c-ae88470fd047
    View test on GitHub ↗
  946. ✓test_chat_completions_failure_writes_one_object TestS3LogDeliverytest_s3_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Add a deployment named s3-err-d728bdc2d1ff that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key named s3-err-key-46f416e9e38a with models: s3-err-d728bdc2d1ff
    4. Send a /chat/completions request to s3-err-d728bdc2d1ff with stream=False and the prompt "trigger an upstream auth failure"
    5. Wait for the request's log object to land in the S3 bucket under s3-err-key-46f416e9e38a/, then watch for duplicates
    View test on GitHub ↗
  947. ✓test_chat_completions_object_key_follows_the_partition_granularity TestS3LogDeliverytest_s3_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named s3-layout-9b67d2ef45be with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the request's log object to land in the S3 bucket under s3-layout-9b67d2ef45be/, then watch for duplicates
    5. List the log objects in the S3 bucket under s3-layout-9b67d2ef45be/
    View test on GitHub ↗
  948. ✓test_chat_completions_stream_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • chat_completions
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-stream-chat-1401792a9c0c with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the key's spend log in /spend/logs
    5. Wait for DataDog to ingest logs carrying the marker ee35c9194f48, then watch for duplicates
    View test on GitHub ↗
  949. ✓test_chat_completions_stream_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • chat_completions
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-stream-chat-dd18bc4a107f with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call 96bcfd1d-97cd-46a1-bf5d-d1b64c01885f
    View test on GitHub ↗
  950. ✓test_chat_completions_stream_records_real_ttft TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • chat_completions
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-ttft-chat-f7412bb825fb with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call 7c3a1ac2-f15c-4db5-9d44-25d032d1259e
    View test on GitHub ↗
  951. ✓test_chat_completions_writes_one_success_object TestS3LogDeliverytest_s3_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named s3-chat-458f4e4cefa3 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the request's log object to land in the S3 bucket under s3-chat-458f4e4cefa3/, then watch for duplicates
    View test on GitHub ↗
  952. ✓test_chat_completions_writes_one_success_record TestGcsLogDeliverytest_gcs_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named gcs-chat-dd772c991cb5 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the log of the response chatcmpl-f521c493-9524-41c7-a95c-793415b3d282 to land in the GCS bucket, then watch for duplicates
    View test on GitHub ↗
  953. ✓test_distinct_key_aliases_produce_distinct_series TestPrometheusPerKeyCardinalitytest_prometheus_cardinality_e2e.py›
    • observability
    • metrics
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key named e2e-prom-78882a374489 with models: gemini-2.5-flash
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "reply with one word e2e-prom-78882a374489"
    3. Generate a virtual key named e2e-prom-c73109d7be99 with models: gemini-2.5-flash
    4. Send a /chat/completions request to gemini-2.5-flash with the prompt "reply with one word e2e-prom-c73109d7be99"
    5. Generate a virtual key named e2e-prom-f8ffa07ee1f3 with models: gemini-2.5-flash
    6. Send a /chat/completions request to gemini-2.5-flash with the prompt "reply with one word e2e-prom-f8ffa07ee1f3"
    7. Scrape the Prometheus metrics from /metrics
    View test on GitHub ↗
  954. ✓test_failed_chat_completions_emits_one_error_event TestDataDogFailureDeliverytest_datadog_log_e2e.py›
    • observability
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Add a deployment named dd-err-e399a52f8e77 that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key named dd-err-key-92292c3de738 with models: dd-err-e399a52f8e77
    4. Send a /chat/completions request to dd-err-e399a52f8e77 with stream=False and the prompt "trigger an upstream auth failure"
    5. Wait for DataDog to ingest logs matching @model_group:dd-err-e399a52f8e77, then watch for duplicates
    View test on GitHub ↗
  955. ✓test_failed_chat_completions_error_span_attributes TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • chat_completions
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Add a deployment named otel-err-fe87f337ef85 that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key named otel-err-fdda6beff091 with models: otel-err-fe87f337ef85
    4. Send a /chat/completions request to otel-err-fe87f337ef85 with stream=False and the prompt "trigger an upstream auth failure"
    5. Poll Jaeger for the traces of call fd18072e-f1c6-44e8-844c-49165635fb9a
    View test on GitHub ↗
  956. ✓test_failed_chat_completions_stream_emits_one_error_event TestDataDogFailureDeliverytest_datadog_log_e2e.py›
    • observability
    • chat_completions
    • stream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Add a deployment named dd-err-stream-fa0603a8f911 that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key named dd-err-stream-key-d4f91d036f7f with models: dd-err-stream-fa0603a8f911
    4. Stream a POST to /chat/completions
    5. Wait for DataDog to ingest logs matching @model_group:dd-err-stream-fa0603a8f911, then watch for duplicates
    View test on GitHub ↗
  957. ✓test_failed_messages_error_span_attributes TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • messages
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Add a deployment named otel-err-caaf93769d1f that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key named otel-err-1270185090b3 with models: otel-err-caaf93769d1f
    4. Send a /v1/messages request to otel-err-caaf93769d1f with stream=False and the prompt "trigger an upstream auth failure"
    5. Poll Jaeger for the traces of call d59c3894-f668-4a79-aced-8e232a7babbf
    View test on GitHub ↗
  958. ✓test_messages_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-messages-47508824e508 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for DataDog to ingest logs carrying the marker a09801851cf4, then watch for duplicates
    View test on GitHub ↗
  959. ✓test_messages_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • messages
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-trace-messages-f0514f04df6d with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call 162398d1-b8c5-45ed-9dd5-5441d822261a
    View test on GitHub ↗
  960. ✓test_messages_stream_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • messages
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-stream-messages-1ba9d857d57e with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the key's spend log in /spend/logs
    5. Wait for DataDog to ingest logs carrying the marker 84db29cc99f0, then watch for duplicates
    View test on GitHub ↗
  961. ✓test_messages_stream_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • messages
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-stream-messages-69a1fda3e4d7 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call d4fdd83e-c1d8-4f45-ad61-21111163cac2
    View test on GitHub ↗
  962. ✓test_messages_stream_records_real_ttft TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • messages
    • stream
    • anthropic
    • claude-haiku-4-5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-ttft-messages-5c97ae3a4a34 with models: claude-haiku-4-5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call 1c9d91ab-c210-4192-91bb-6716b1eb54a6
    View test on GitHub ↗
  963. ✓test_non_json_native_metadata_reaches_langsmith TestLangsmithBatchSerializationtest_langsmith_batch_serialization_e2e.py›
    • observability
    • nonstream
    • openai
    • gpt-5.5
    1. GET the external URL https://api.smith.langchain.com/runs/17d756f9-5bbf-4b62-a25f-d33e6c6575df
    View test on GitHub ↗
  964. ✓test_queue_time_histogram_records_an_observation TestPrometheusRequestQueueTimetest_prometheus_queue_time_e2e.py›
    • observability
    • metrics
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key named e2e-queue-time-6972f5797674 with models: gemini-2.5-flash
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "reply with one word e2e-queue-time-6972f5797674"
    3. Scrape the Prometheus metrics from /metrics
    View test on GitHub ↗
  965. ✓test_responses_call_logs_nonzero_token_usage[non_streaming] TestResponsesLangfuseUsagetest_responses_langfuse_usage_e2e.py›
    • observability
    • responses
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key named lf-resp-key-c3a3111dfedd with models: gpt-5.5
    2. Retry the call until the fresh key stops answering 401
    3. Wait for the Langfuse generation for the key lf-resp-key-c3a3111dfedd
    View test on GitHub ↗
  966. ✓test_responses_call_logs_nonzero_token_usage[streaming] TestResponsesLangfuseUsagetest_responses_langfuse_usage_e2e.py›
    • observability
    • responses
    • stream
    • openai
    • gpt-5.5
    1. Generate a virtual key named lf-resp-key-58d735d084a5 with models: gpt-5.5
    2. Retry the call until the fresh key stops answering 401
    3. Wait for the Langfuse generation for the key lf-resp-key-58d735d084a5
    View test on GitHub ↗
  967. ✓test_responses_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • responses
    • nonstream
    • openai
    • gpt-5.5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-responses-68eb823bf24a with models: gpt-5.5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for DataDog to ingest logs carrying the marker 503bb51859fa, then watch for duplicates
    View test on GitHub ↗
  968. ✓test_responses_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • responses
    • nonstream
    • openai
    • gpt-5.5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-trace-responses-81cdffb7fdaa with models: gpt-5.5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call ca8cfcd4-cb5d-449a-b40b-294975acfc44
    View test on GitHub ↗
  969. ✓test_responses_stream_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    • observability
    • responses
    • stream
    • openai
    • gpt-5.5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named dd-stream-responses-ac073e0700e0 with models: gpt-5.5
    3. Retry the call until the fresh key stops answering 401
    4. Wait for the key's spend log in /spend/logs
    5. Wait for DataDog to ingest logs carrying the marker cfb72b0c7d69, then watch for duplicates
    View test on GitHub ↗
  970. ✓test_responses_stream_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • responses
    • stream
    • openai
    • gpt-5.5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-stream-responses-d970a21bcdcd with models: gpt-5.5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call edf125aa-5412-4705-b282-008db73780c6
    5. Wait for the key's spend log in /spend/logs
    View test on GitHub ↗
  971. ✓test_responses_stream_records_real_ttft TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    • observability
    • responses
    • stream
    • openai
    • gpt-5.5
    1. Read the proxy's callback list from /health/readiness/details
    2. Generate a virtual key named otel-ttft-responses-23bbfa24acdb with models: gpt-5.5
    3. Retry the call until the fresh key stops answering 401
    4. Poll Jaeger for the traces of call ef23ceae-0395-42bc-ac43-933b40c1258e
    View test on GitHub ↗
  972. ✓test_team_callback_delivers_and_isolates TestTeamLangfuseCallbacktest_team_langfuse_callback_e2e.py›
    • observability
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Create the team lf-team-03ab4b8bbfe2 with models: claude-haiku-4-5
    2. Add a Langfuse OTel logging callback for success_and_failure events to the team
    3. Generate a virtual key named lf-team-key-06829b65c99e with models: claude-haiku-4-5
    4. Generate a virtual key named lf-solo-key-c6bcfb9929dc with models: claude-haiku-4-5
    5. Retry the call until the fresh key stops answering 401
    6. Wait for the Langfuse generation for the key lf-team-key-06829b65c99e
    7. Retry the call until the fresh key stops answering 401
    8. Look up the Langfuse generation for the key lf-solo-key-c6bcfb9929dc
    View test on GitHub ↗
  973. −test_chat_completions_delivers_one_call_with_spend TestWeaveLogDeliverytest_weave_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • claude-haiku-4-5

    No steps recorded

    stage red: product gap, key-scoped weave_otel spans are not delivered when the OTEL v2 callback is active

    View test on GitHub ↗
  974. −test_failed_chat_completions_delivers_one_error_call TestWeaveLogDeliverytest_weave_log_e2e.py›
    • observability
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5

    No steps recorded

    stage red: product gap, key-scoped weave_otel spans are not delivered when the OTEL v2 callback is active

    View test on GitHub ↗
  975. ✓test_all_team_models_project_calls_team_allowed_model[team_models0] TestProjectAllTeamModelstest_project_all_team_models_e2e.py›
    • proxy-auth
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Create the team e2e-proj-team-4d11e7bc2ca5 with models:
    2. Create the project e2e-proj-4d11e7bc2ca5 under the team with models: all-team-models
    3. Generate a virtual key scoped to the project with models: all-team-models
    4. Send a /chat/completions request to gemini-2.5-flash with the prompt "capital of France? a5f63f278c4c"
    View test on GitHub ↗
  976. ✓test_all_team_models_project_calls_team_allowed_model[team_models1] TestProjectAllTeamModelstest_project_all_team_models_e2e.py›
    • proxy-auth
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Create the team e2e-proj-team-5ad493b5b2a0 with models: all-proxy-models
    2. Create the project e2e-proj-5ad493b5b2a0 under the team with models: all-team-models
    3. Generate a virtual key scoped to the project with models: all-team-models
    4. Send a /chat/completions request to gemini-2.5-flash with the prompt "capital of France? da98f4278df8"
    View test on GitHub ↗
  977. ✓test_all_team_models_project_denied_outside_team_list TestProjectAllTeamModelstest_project_all_team_models_e2e.py›
    • proxy-auth
    • nonstream
    • gemini
    • openai
    • gemini-2.5-flash
    • gpt-5.5
    1. Create the team e2e-proj-team-95bc083494f4 with models:
    2. Create the project e2e-proj-95bc083494f4 under the team with models: all-team-models
    3. Generate a virtual key scoped to the project with models: all-team-models
    4. Set the team e2e-proj-team-95bc083494f4's models to gemini-2.5-flash through /team/update
    5. Read /team/info?team_id=3a130ad4-ed46-43f1-a262-e937869889ed on every proxy replica until it settles
    6. Wait for the last control-plane write to reach every proxy replica
    7. Send a /chat/completions request to gemini-2.5-flash with the prompt "capital of France? 948d9017aeb7"
    8. Send a /chat/completions request to gpt-5.5 with the prompt "capital of France? 1b082de1da58"
    View test on GitHub ↗
  978. ✓test_allowed_model_is_permitted TestAccessControltest_access_control_e2e.py›
    • proxy-auth
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with models: gemini-2.5-flash and user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "capital of France? 3587db783904"
    View test on GitHub ↗
  979. ✓test_bearer_invalid_token_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›
    • proxy-auth
    • chat_completions
    1. Send a request to /chat/completions
    View test on GitHub ↗
  980. ✓test_disallowed_model_is_denied_403 TestAccessControltest_access_control_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Generate a virtual key with models: gemini-2.5-flash and user id: e2e-test-user
    2. Send a /chat/completions request to gpt-5.5 with the prompt "capital of France? 7e67326abc6c"
    View test on GitHub ↗
  981. ✓test_empty_bearer_token_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›
    • proxy-auth
    • chat_completions
    1. Send a request to /chat/completions
    View test on GitHub ↗
  982. ✓test_group_grants_every_deployment_in_it[bare name the group's wildcard covers-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.4-nano
    • openai/gpt-5.4-nano
    1. Add a deployment named openai/gpt-5.4* that calls openai/gpt-5.4*
    2. Add a deployment named e2e-ag-member-d6f4d62033f9 that calls openai/gpt-5.4-nano
    3. Add a deployment named e2e-ag-outsider-d6f4d62033f9 that calls openai/gpt-5.4-nano
    4. List the deployments in the model access group e2e-ag-d6f4d62033f9
    5. Generate a virtual key with models: e2e-ag-d6f4d62033f9 and user id: e2e-test-user
    6. Send a /chat/completions request to gpt-5.4-nano with the prompt "Reply with exactly: OK c7d60bbcb38d"
    View test on GitHub ↗
  983. ✓test_group_grants_every_deployment_in_it[exactly-named deployment in the group-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.4-nano
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-ag-d6f4d62033f9 and user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ag-member-d6f4d62033f9 with the prompt "Reply with exactly: OK 7070586c7940"
    View test on GitHub ↗
  984. ✓test_group_grants_every_deployment_in_it[provider-prefixed name the group's wildcard covers-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.4-nano
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-ag-d6f4d62033f9 and user id: e2e-test-user
    2. Send a /chat/completions request to openai/gpt-5.4-nano with the prompt "Reply with exactly: OK ee8fdf602176"
    View test on GitHub ↗
  985. ✓test_group_grants_nothing_outside_it[deployment outside the group-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.2
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-ag-d6f4d62033f9 and user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ag-outsider-d6f4d62033f9 with the prompt "Reply with exactly: OK 2e3188d72aec"
    View test on GitHub ↗
  986. ✓test_group_grants_nothing_outside_it[name no provider claims-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.2
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-ag-d6f4d62033f9 and user id: e2e-test-user
    2. Send a /chat/completions request to e2e-ag-unknown-94d7d35335ab with the prompt "Reply with exactly: OK 494d087cfa25"
    View test on GitHub ↗
  987. ✓test_group_grants_nothing_outside_it[provider model outside the group's wildcard-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.2
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-ag-d6f4d62033f9 and user id: e2e-test-user
    2. Send a /chat/completions request to gpt-5.2 with the prompt "Reply with exactly: OK 4b4d7149759d"
    View test on GitHub ↗
  988. ✓test_group_grants_the_team_nothing_outside_it TestTeamScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    1. Send a /chat/completions request to e2e-ag-unknown-2b7416befc42 with the prompt "Reply with exactly: OK 178fdc9bb6e9"
    View test on GitHub ↗
  989. ✓test_group_grants_the_teams_own_wildcard TestTeamScopedToAccessGrouptest_model_access_group_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.6-luna
    1. Create the team e2e-ag-team-2d2fda2fcc29 with models: e2e-agt-2d2fda2fcc29
    2. Generate a virtual key with team id: cfb2ebaf-dffe-48a0-97b6-aa463b54ab2a
    3. Add a deployment named openai/gpt-5.6* that calls openai/gpt-5.6*
    4. Set the team e2e-ag-team-2d2fda2fcc29's models to e2e-agt-2d2fda2fcc29 through /team/update
    5. Read /team/info?team_id=cfb2ebaf-dffe-48a0-97b6-aa463b54ab2a on every proxy replica until it settles
    6. Wait for the last control-plane write to reach every proxy replica
    7. Send a /chat/completions request to gpt-5.6-luna with the prompt "Reply with exactly: OK 68fd49376c8f"
    View test on GitHub ↗
  990. ✓test_llm_api_routes_group_grants_every_llm_endpoint TestAccessControltest_access_control_e2e.py›
    • proxy-auth
    • gemini
    • openai
    • gemini-2.5-flash
    • openai-text-embedding-3-small
    1. Generate a virtual key that can only call LLM API routes
    2. Send a /chat/completions request to gemini-2.5-flash with the prompt "capital of France? 34bf1f392557"
    3. Send an /embeddings request to openai-text-embedding-3-small for "route group 01fc2fb797de"
    4. Add a deployment named e2e-route-group-9106e58810f8 that calls openai/gpt-4o-mini with the given key
    View test on GitHub ↗
  991. ✓test_llm_only_key_forbidden_from_management_route_403 TestAccessControltest_access_control_e2e.py›
    • proxy-auth
    • model_management
    1. Generate a virtual key that can only call LLM API routes
    2. Add a deployment named e2e-forbidden-f5313a8500fe that calls openai/gpt-4o-mini with the given key
    View test on GitHub ↗
  992. ✓test_missing_authorization_header_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›
    • proxy-auth
    • chat_completions
    1. Send a request to /chat/completions
    View test on GitHub ↗
  993. ✓test_not_bearer_scheme_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›
    • proxy-auth
    • chat_completions
    1. Send a request to /chat/completions
    View test on GitHub ↗
  994. ✓test_project_explicit_model_list_calls_model TestProjectAllTeamModelstest_project_all_team_models_e2e.py›
    • proxy-auth
    • nonstream
    • gemini
    • openai
    • gemini-2.5-flash
    • gpt-5.5
    1. Create the team e2e-proj-team-ecd25be8c83e with models:
    2. Create the project e2e-proj-ecd25be8c83e under the team with models: gemini-2.5-flash
    3. Generate a virtual key scoped to the project with models:
    4. Send a /chat/completions request to gemini-2.5-flash with the prompt "capital of France? ef9e4a290e49"
    5. Send a /chat/completions request to gpt-5.5 with the prompt "capital of France? 8b0ddf57fd8c"
    View test on GitHub ↗
  995. ✓test_token_without_bearer_prefix_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›
    • proxy-auth
    • chat_completions
    1. Send a request to /chat/completions
    View test on GitHub ↗
  996. ✓test_unknown_model_returns_400 TestAccessControltest_access_control_e2e.py›
    • proxy-auth
    • nonstream
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to nonexistent-model-4184b5aeb0b2 with the prompt "hi this is a test"
    View test on GitHub ↗
  997. ✓test_valid_key_allows_and_invalid_key_denied TestVirtualKeyAuthtest_access_control_e2e.py›
    • proxy-auth
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-auth-chat-563e4b8faabf that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-auth-chat-563e4b8faabf
    View test on GitHub ↗
  998. ✓test_encrypted_stored_value_is_not_a_bearer_token TestSessionTokentest_session_token_e2e.py›
    • proxy-auth
    1. Generate a virtual key with key alias: e2e-session-2c2a2c755cb8 and metadata: logging: callback name: langfuse and callback vars: default settings
    2. Read the key's settings back from /key/info
    3. List users from /user/list with the given key
    View test on GitHub ↗
  999. ✓test_expired_session_token_is_denied TestSessionTokentest_session_token_e2e.py›
    • proxy-auth
    1. List users from /user/list with the given key
    View test on GitHub ↗
  1000. ✓test_expired_token_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-94ec68e66f61 and the user e2e-jwt-user-94ec68e66f61 in it
    2. Create a team with team alias: e2e-jwt-94ec68e66f61 and team id: e2e-jwt-team-94ec68e66f61
    3. Get a Keycloak access token for e2e-jwt-user-94ec68e66f61 from the client litellm-e2e-shortlived
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  1001. ✓test_liveness_reports_alive_without_auth TestHealthLifecycletest_health_lifecycle_e2e.py›
    • deploy-ops
    • health
    1. Call /health/liveliness without credentials
    View test on GitHub ↗
  1002. ✓test_master_key_authenticates_and_grants_admin_route TestMasterKeyAuthtest_master_key_auth_e2e.py›
    • proxy-auth
    1. List users from /user/list with the given key
    View test on GitHub ↗
  1003. ✓test_non_matching_master_key_is_denied TestMasterKeyAuthtest_master_key_auth_e2e.py›
    • proxy-auth
    1. List users from /user/list with the given key
    View test on GitHub ↗
  1004. ✓test_plain_virtual_key_still_works_with_jwt_auth_enabled TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  1005. ✓test_proxy_admin_token_is_blocked_once_its_personal_budget_is_spent TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak user e2e-jwt-user-4982584e6641 with a password of its own
    2. Create the proxy_admin user e2e-jwt-user-4982584e6641@example.com through /user/new
    3. Get a Keycloak access token for e2e-jwt-user-4982584e6641 from the client litellm-e2e-tests
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  1006. ✓test_readiness_details_require_auth_and_expose_diagnostics TestHealthLifecycletest_health_lifecycle_e2e.py›
    • deploy-ops
    • health
    1. Call /health/readiness/details without credentials
    2. Call /health/readiness/details with the given key
    View test on GitHub ↗
  1007. ✓test_readiness_is_reachable_without_credentials TestHealthLifecycletest_health_lifecycle_e2e.py›
    • deploy-ops
    • health
    1. Call /health/readiness without credentials
    View test on GitHub ↗
  1008. ✓test_readiness_reports_connected_db TestHealthLifecycletest_health_lifecycle_e2e.py›
    • deploy-ops
    • health
    1. Call /health/readiness without credentials
    View test on GitHub ↗
  1009. ✓test_signed_token_for_another_application_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-1c986082d1a0 and the user e2e-jwt-user-1c986082d1a0 in it
    2. Create a team with team alias: e2e-jwt-1c986082d1a0 and team id: e2e-jwt-team-1c986082d1a0
    3. Get a Keycloak access token for e2e-jwt-user-1c986082d1a0 from the client litellm-e2e-other-app
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  1010. ✓test_signed_token_from_the_wrong_issuer_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-987280fd8649 and the user e2e-jwt-user-987280fd8649 in it
    2. Create a team with team alias: e2e-jwt-987280fd8649 and team id: e2e-jwt-team-987280fd8649
    3. Get a Keycloak access token for e2e-jwt-user-987280fd8649 from the client litellm-e2e-tests
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  1011. ✓test_tampered_signature_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-fc27d7d43e4a and the user e2e-jwt-user-fc27d7d43e4a in it
    2. Create a team with team alias: e2e-jwt-fc27d7d43e4a and team id: e2e-jwt-team-fc27d7d43e4a
    3. Get a Keycloak access token for e2e-jwt-user-fc27d7d43e4a from the client litellm-e2e-tests
    4. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  1012. ✓test_team_header_with_the_alias_of_a_team_the_caller_is_not_in_is_rejected_like_an_unknown_value TestJwtTeamHeadertest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-4e1c5366fa64 and the user e2e-jwt-user-4e1c5366fa64 in it
    2. Create a team with team alias: e2e-jwt-alias-4e1c5366fa64 and team id: e2e-jwt-team-4e1c5366fa64
    3. Get a Keycloak access token for e2e-jwt-user-4e1c5366fa64 from the client litellm-e2e-tests
    4. Create a team with team alias: e2e-jwt-alias-023b528c7977 and team id: e2e-jwt-other-023b528c7977
    5. Send a /chat/completions request to gpt-5.5 as team e2e-jwt-alias-023b528c7977 with the given token
    6. Send a /chat/completions request to gpt-5.5 as team e2e-jwt-unknown-73809280a630 with the given token
    View test on GitHub ↗
  1013. ✓test_team_header_with_the_team_alias_binds_the_same_team_as_the_team_id TestJwtTeamHeadertest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-f06f89b8befc and the user e2e-jwt-user-f06f89b8befc in it
    2. Create a team with team alias: e2e-jwt-alias-f06f89b8befc and team id: e2e-jwt-team-f06f89b8befc
    3. Get a Keycloak access token for e2e-jwt-user-f06f89b8befc from the client litellm-e2e-tests
    4. Send a /chat/completions request to gpt-5.5 as team e2e-jwt-team-f06f89b8befc with the given token
    5. Wait for the request's spend log in /spend/logs
    6. Send a /chat/completions request to gpt-5.5 as team e2e-jwt-alias-f06f89b8befc with the given token
    7. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  1014. ✓test_team_model_alias_is_listed_by_v1_models_under_the_same_token_that_routes_it[anthropic_shape] TestJwtTeamHeadertest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-d721f17e9da2 and the user e2e-jwt-user-d721f17e9da2 in it
    2. Create a team with team alias: e2e-jwt-aliased-d721f17e9da2, models: gpt-5.5, team id: e2e-jwt-team-d721f17e9da2 and model aliases: e2e-jwt-model-alias-d721f17e9da2: gpt-5.5
    3. Get a Keycloak access token for e2e-jwt-user-d721f17e9da2 from the client litellm-e2e-tests
    4. Send a /chat/completions request to e2e-jwt-model-alias-d721f17e9da2
    5. List the models from /v1/models with the given token, in the Anthropic shape: True
    View test on GitHub ↗
  1015. ✓test_team_model_alias_is_listed_by_v1_models_under_the_same_token_that_routes_it[openai_shape] TestJwtTeamHeadertest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-b21955651138 and the user e2e-jwt-user-b21955651138 in it
    2. Create a team with team alias: e2e-jwt-aliased-b21955651138, models: gpt-5.5, team id: e2e-jwt-team-b21955651138 and model aliases: e2e-jwt-model-alias-b21955651138: gpt-5.5
    3. Get a Keycloak access token for e2e-jwt-user-b21955651138 from the client litellm-e2e-tests
    4. Send a /chat/completions request to e2e-jwt-model-alias-b21955651138
    5. List the models from /v1/models with the given token, in the Anthropic shape: False
    View test on GitHub ↗
  1016. ✓test_token_naming_a_team_that_does_not_exist_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-bb16b5b3283e and the user e2e-jwt-user-bb16b5b3283e in it
    2. Get a Keycloak access token for e2e-jwt-user-bb16b5b3283e from the client litellm-e2e-tests
    3. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  1017. ✓test_unexpired_session_token_reaches_admin_route TestSessionTokentest_session_token_e2e.py›
    • proxy-auth
    1. List users from /user/list with the given key
    View test on GitHub ↗
  1018. ✓test_valid_token_for_an_existing_team_is_accepted_and_attributed TestJwtAuthtest_jwt_auth_e2e.py›
    • proxy-auth
    • nonstream
    • openai
    • gpt-5.5
    1. Create the Keycloak group e2e-jwt-team-94cd080efbec and the user e2e-jwt-user-94cd080efbec in it
    2. Create a team with team alias: e2e-jwt-94cd080efbec and team id: e2e-jwt-team-94cd080efbec
    3. Get a Keycloak access token for e2e-jwt-user-94cd080efbec from the client litellm-e2e-tests
    4. Send a /chat/completions request to gpt-5.5
    5. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  1019. ✓test_access_group_scopes_tool_selection TestMcpAccessGroupToolSelectiontest_mcp_access_group_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-ag-granted-8c216875f8fc
    4. Generate a virtual key for the user e2e-mcp-ag-other-09a3b8786256
    5. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    6. List the MCP tools the key can see from /mcp-rest/tools/list
    View test on GitHub ↗
  1020. ✓test_alias_grant_persists_verbatim_and_lists_tools TestMcpKeyGrantByAliastest_mcp_key_access_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-allowed-f053bc76e821
    4. Read the key's settings back from /key/info
    5. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    View test on GitHub ↗
  1021. ✓test_call_tool_denied_without_permission TestMcpKeyWithoutAccessIsDeniedtest_mcp_key_access_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-allowed-8fef5c73d38c
    4. Generate a virtual key for the user e2e-mcp-denied-95c8b59d5173
    5. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    6. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call
    7. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call and wait for a 403
    View test on GitHub ↗
  1022. ✓test_content_filter_blocks_banned_keyword_in_tool_args TestMcpToolCallGuardrailtest_mcp_guardrail_e2e.py›
    • guardrails
    • mcp
    1. Create the guardrail e2e-mcp-cf-d1b1396d3506 that blocks MCP tool calls containing "e2eblockedd1b1396d3506"
    2. Register the Datadog remote MCP server with its credentials from the environment
    3. Generate a virtual key for the user e2e-mcp-guard-d1b1396d3506
    4. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    5. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: tell me about e2eblockedd1b1396d3506, from: now-30m, to: now and max tokens: 500
    6. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: still about e2eblockedd1b1396d3506 #1, from: now-30m, to: now and max tokens: 500
    7. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: still about e2eblockedd1b1396d3506 #2, from: now-30m, to: now and max tokens: 500
    8. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: still about e2eblockedd1b1396d3506 #3, from: now-30m, to: now and max tokens: 500
    9. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: still about e2eblockedd1b1396d3506 #4, from: now-30m, to: now and max tokens: 500
    10. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call with query: e2e-clean-d1b1396d3506, from: now-30m, to: now and max tokens: 500
    View test on GitHub ↗
  1023. ✓test_filter_keeps_the_tool_the_prompt_needs TestMcpSemanticToolFiltertest_mcp_semantic_tool_filter_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Add a deployment named e2e-semantic-filter-embedding-a143a53f13e0 that calls openai/text-embedding-3-small
    4. Generate a virtual key for the user e2e-mcp-semantic-f4472cbe0b07
    5. List the MCP tools the key can see from /mcp-rest/tools/list
    6. POST /config/update
    7. Send a streaming /chat/completions request to gpt-5.5
    View test on GitHub ↗
  1024. ✓test_key_granted_a_toolset_lists_exactly_its_tools TestMcpToolsetEnforcementtest_mcp_toolset_enforcement_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-catalog-5efae1b65e9c
    4. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    5. List the MCP tools the key can see from /mcp-rest/tools/list
    6. Create an MCP toolset with the tools server id: a3e0a25d-34cf-4a9b-8a55-699d096e1303 and tool name: _dd_gff, server id: a3e0a25d-34cf-4a9b-8a55-699d096e1303 and tool name: _dd_guc
    7. Generate a virtual key for the user e2e-mcp-toolset-ecfebd1cb312
    8. Wait for /mcp-rest/tools/list to show the key exactly the expected tools on the MCP server
    View test on GitHub ↗
  1025. ✓test_list_tools_denied_without_permission TestMcpKeyWithoutAccessIsDeniedtest_mcp_key_access_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-allowed-22189b89483b
    4. Generate a virtual key for the user e2e-mcp-denied-38458ad69109
    5. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    6. List the MCP tools the key can see from /mcp-rest/tools/list
    View test on GitHub ↗
  1026. ✓test_native_tool_search_omits_oversized_description TestMcpToolSearchtest_mcp_tool_search_e2e.py›
    1. Add a deployment named e2e-mcp-tool-search-26de3d77326a that calls openai/text-embedding-3-large
    2. Read the MCP tool search settings from /get/mcp_tool_search_settings
    3. Update the MCP tool search settings with /update/mcp_tool_search_settings
    4. Register the Datadog remote MCP server with its credentials from the environment
    5. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    6. Generate a virtual key for the user e2e-mcp-tool-search-72d9c316c749
    7. List the MCP tools the key can see from /mcp-rest/tools/list
    8. Call the MCP virtual tool mcp_tool_search through /mcp-rest/tools/call
    View test on GitHub ↗
  1027. ✓test_route_restricted_health_matches_server_grants TestMcpHealthVisibilitytest_mcp_key_access_e2e.py›
    • mcp
    • mcp
    1. Register the Datadog remote MCP server with its credentials from the environment
    2. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    3. Generate a virtual key for the user e2e-mcp-allowed-8625d550cdbb
    4. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    5. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call
    6. Generate a virtual key with user id: e2e-mcp-health-2be5925f0add, allowed routes: /v1/mcp/server, /v1/mcp/server/health and object permission: mcp servers: d9b40ddf-a052-4116-b27d-c1f3562ad04d
    7. List the MCP servers the key can see from /v1/mcp/server
    8. Check the health of the MCP servers the key can see from /v1/mcp/server/health
    9. Generate a virtual key with user id: e2e-mcp-health-6060b8ff5a93, allowed routes: /v1/mcp/server, /v1/mcp/server/health and object permission: mcp servers: bd7b212c-eb89-4ddc-b423-f07fcf6d0fc3
    10. List the MCP servers the key can see from /v1/mcp/server
    11. Check the health of the MCP servers the key can see from /v1/mcp/server/health
    12. Generate a virtual key with user id: e2e-mcp-health-f53891439d2f, allowed routes: /v1/mcp/server, /v1/mcp/server/health and object permission: default settings
    13. List the MCP servers the key can see from /v1/mcp/server
    14. Check the health of the MCP servers the key can see from /v1/mcp/server/health
    View test on GitHub ↗
  1028. ✓test_search_logs_finds_seeded_completion TestDatadogMcpRoundTriptest_mcp_datadog_e2e.py›
    • mcp
    • mcp
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Call the management route /health/readiness/details
    2. Register the Datadog remote MCP server with its credentials from the environment
    3. Wait for every proxy replica to list the MCP server in /v1/mcp/server
    4. Generate a virtual key for the user e2e-dd-mcp-ccde131561c6
    5. Send a /chat/completions request to claude-haiku-4-5
    6. Wait for DataDog to ingest logs carrying the marker e2e-datadog-mcp-06a0b5a5469e, then watch for duplicates
    7. Wait for /mcp-rest/tools/list to show the MCP server's tool matching "search_datadog_logs"
    8. Call the MCP tool search_datadog_logs through /mcp-rest/tools/call
    View test on GitHub ↗
  1029. −test_chat_completion_uses_linear_with_authorization_bearer_header TestMcpChatCompletionOauthtest_mcp_chat_completion_oauth_e2e.py›
    • mcp
    • chat_completions
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • function_calling

    No steps recorded

    set E2E_LINEAR_STORAGE_STATE to a Linear session captured via mcp/linear_session_capture.py

    View test on GitHub ↗
  1030. −test_chat_completion_uses_linear_with_x_litellm_api_key_header TestMcpChatCompletionOauthtest_mcp_chat_completion_oauth_e2e.py›
    • mcp
    • chat_completions
    • nonstream
    • anthropic
    • claude-haiku-4-5
    • function_calling

    No steps recorded

    set E2E_LINEAR_STORAGE_STATE to a Linear session captured via mcp/linear_session_capture.py

    View test on GitHub ↗
  1031. ✓test_discovery_card_is_proxy_fronted TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the A2A agent e2e-a2a-eb8d984d1060 through /v1/agents
    3. Read the A2A agent's card from /a2a/{agent_id}/.well-known/agent-card.json with the given key
    View test on GitHub ↗
  1032. ✓test_malformed_protocol_version_rejected TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    1. Register the A2A agent e2e-a2a-bad-2ff7e13cc49b through /v1/agents
    View test on GitHub ↗
  1033. ✓test_message_send_runs_completion_bridge TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the A2A agent e2e-a2a-90bdad279aa1 through /v1/agents
    3. Send an A2A message to /a2a/{agent_id} with text: Reply with exactly the word PONG and nothing else
    4. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  1034. ✓test_pinned_v0_3_serves_flat_message_shape TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the A2A agent e2e-a2a-c8785d267e10 through /v1/agents
    3. Send an A2A message to /a2a/{agent_id} with text: Say hi in one word
    View test on GitHub ↗
  1035. ✓test_pinned_v1_0_serves_nested_message_shape TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the A2A agent e2e-a2a-63869ff90bef through /v1/agents
    3. Send an A2A message to /a2a/{agent_id} with text: Say hi in one word
    View test on GitHub ↗
  1036. ✓test_real_world_agent_replies_to_property_query TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    1. Generate a virtual key with user id: e2e-test-user
    2. Fetch a published A2A agent card from its /.well-known endpoint
    3. Register the A2A agent e2e-a2a-real-03249461fac8 through /v1/agents
    4. Send an A2A message to /a2a/{agent_id} with data: skill: search_properties and params: un locode: GBLON, service type: long_term and limit: 3
    View test on GitHub ↗
  1037. ✓test_register_persists TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    1. Register the A2A agent e2e-a2a-49903c76c4f5 through /v1/agents
    2. Read the A2A agent back from /v1/agents/{agent_id}
    View test on GitHub ↗
  1038. ✓test_semver_protocol_version_registers_and_serves TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    2. Register the A2A agent e2e-a2a-bd45136f96b7 through /v1/agents
    3. Read the A2A agent's card from /a2a/{agent_id}/.well-known/agent-card.json with the given key
    4. Send an A2A message to /a2a/{agent_id} with text: Say hi in one word
    View test on GitHub ↗
  1039. ✓test_unsupported_protocol_version_rejected TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    • agents-api
    • a2a
    1. Register the A2A agent e2e-a2a-bad-314fd0087ce9 through /v1/agents
    View test on GitHub ↗
  1040. −llm_translation/realtime/test_realtime_pipecat_audio_e2e.pyllm_translation/realtime/test_realtime_pipecat_audio_e2e.py›

    No steps recorded

    collection skipped