litellm-e2e · build #432

1507 passed, 2 failed, 59 skipped

1.105.1stable

Oct 13, 2026, 11:48 PM UTC ·d665dc1 ↗· build #432 ↗· 1.105.1 ↗· run history

All 1568 tests

  1. ✗test_bedrock_image_generation_returns_image TestImageGenerationtest_image_generation_e2e.py›
    1. Add a deployment named e2e-bedrock-image-b1fcf19821a5 that calls bedrock/amazon.nova-canvas-v1:0
    2. Generate a virtual key with user id: e2e-test-user

    openai.NotFoundError: Error code: 404 - {'error': {'message': 'litellm.NotFoundError: BedrockException - {"message":"This model version has reached the end of its life. Please refer to the AWS documentation for more details."}\n\nLiteLLM: model group \'e2e-bedrock-image-b1fcf19821a5\' failed with the error above and no fallback model group was found for it, so the request was not retried on another model. Fallbacks are configured for: gemini-2.5-flash. Add a fallbacks entry for that model group (Router fallbacks or proxy router_settings.fallbacks) to retry on another model.', 'type': 'invalid_request_error', 'param': None, 'code': '404'}}

    View test on GitHub ↗
  2. ✓test_anthropic_chat_returns_thinking_content TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-anthropic-thinking-9e69e9060d49 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-thinking-9e69e9060d49
    View test on GitHub ↗
  3. ✓test_anthropic_chat_returns_tool_call TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-anthropic-tool-2397c80f5384 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-tool-2397c80f5384
    View test on GitHub ↗
  4. ✓test_anthropic_chat_streams_real_content TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-anthropic-stream-42d249c28e4b that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-anthropic-stream-42d249c28e4b
    View test on GitHub ↗
  5. ✓test_anthropic_chat_streams_tool_call TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-anthropic-tool-stream-33b2dc31312b that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-anthropic-tool-stream-33b2dc31312b
    View test on GitHub ↗
  6. ✓test_anthropic_chat_structured_output_conforms_to_schema TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-anthropic-schema-7e2a858fe333 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-schema-7e2a858fe333
    View test on GitHub ↗
  7. ✓test_anthropic_chat_vision_describes_image TestAnthropicChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-anthropic-vision-583738d648a1 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-anthropic-vision-583738d648a1
    View test on GitHub ↗
  8. ✓test_anthropic_passthrough_nonstreaming_logs_costtest_passthrough_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  9. ✓test_anthropic_passthrough_streaming_logs_costtest_passthrough_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  10. ✓test_anthropic_passthrough_tool_call_logs_costtest_passthrough_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  11. ✓test_anthropic_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    1. Add a deployment named e2e-anthropic-cache-032eac2a0c3b that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  12. ✓test_anthropic_with_extended_thinking TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    1. Add a deployment named e2e-chat-tool-loop-855c59e4f0a8 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-855c59e4f0a8
    View test on GitHub ↗
  13. ✓test_array_input_returns_vectors TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    1. Add a deployment named e2e-embeddings-array-dd8961255c32 that calls openai/text-embedding-3-small
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  14. ✓test_audio_speech_returns_audio TestAudioSpeechtest_audio_speech_e2e.py›
    1. Add a deployment named e2e-speech-e8e5604c7c85 that calls openai/gpt-4o-mini-tts
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  15. ✓test_audio_speech_streams_audio_chunks TestAudioSpeechtest_audio_speech_e2e.py›
    1. Add a deployment named e2e-speech-4832c0cad5eb that calls openai/gpt-4o-mini-tts
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  16. ✓test_audio_transcriptions_returns_text TestAudioTranscriptionstest_audio_transcriptions_e2e.py›
    1. Add a deployment named e2e-transcribe-f018aab9a006 that calls openai/gpt-4o-mini-transcribe
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  17. ✓test_azure_foundry_chat_returns_content TestAzureFoundryChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-azure-foundry-chat-3f4bf6d3e1d5 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-azure-foundry-chat-3f4bf6d3e1d5
    View test on GitHub ↗
  18. ✓test_azure_openai_chat_returns_content TestAzureOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-azure-openai-chat-dd49f1d75786 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-azure-openai-chat-dd49f1d75786
    View test on GitHub ↗
  19. ✓test_azure_openai_chat_returns_tool_call TestAzureOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-azure-openai-tool-faebc0558001 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-azure-openai-tool-faebc0558001
    View test on GitHub ↗
  20. ✓test_basic_nonstream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    1. Add a deployment named e2e-azure-foundry-messages-e89268ae1e51 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-e89268ae1e51 and user id: e2e-test-user
    View test on GitHub ↗
  21. ✓test_basic_stream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    1. Add a deployment named e2e-azure-foundry-messages-64722ac9c847 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-64722ac9c847 and user id: e2e-test-user
    View test on GitHub ↗
  22. ✓test_batch_s3_keys_do_not_break_chat TestBedrockBatchDeploymentServesChattest_bedrock_provider_matrix_e2e.py›
    1. Add a deployment named e2e-bedrock-batch-chat-0b549110af7f that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  23. ✓test_bedrock_converse TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    1. Add a deployment named e2e-chat-tool-loop-f89b547dcffa that calls bedrock/converse/us.anthropic.claude-sonnet-5-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-f89b547dcffa
    View test on GitHub ↗
  24. ✓test_bedrock_converse_chat_returns_content TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-bedrock-chat-175e7dd67cfe that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-chat-175e7dd67cfe
    View test on GitHub ↗
  25. ✓test_bedrock_converse_chat_returns_reasoning TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-bedrock-thinking-bdca180ec4ab that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-thinking-bdca180ec4ab
    View test on GitHub ↗
  26. ✓test_bedrock_converse_chat_returns_tool_call TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-bedrock-tool-6b07bb9f4c65 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-tool-6b07bb9f4c65
    View test on GitHub ↗
  27. ✓test_bedrock_converse_chat_streams_real_content TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-bedrock-stream-374e61a096ae that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bedrock-stream-374e61a096ae
    View test on GitHub ↗
  28. ✓test_bedrock_converse_chat_vision_describes_image TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-bedrock-vision-58f790c9bc1f that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-vision-58f790c9bc1f
    View test on GitHub ↗
  29. ✓test_bedrock_converse_reads_a_pdf_sent_by_url TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-bedrock-document-3aca1c7b76a2 that calls bedrock/us.amazon.nova-2-lite-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-document-3aca1c7b76a2
    View test on GitHub ↗
  30. ✓test_bedrock_converse_with_extended_thinking TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    1. Add a deployment named e2e-chat-tool-loop-373f331632e7 that calls bedrock/converse/us.anthropic.claude-sonnet-4-6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-373f331632e7
    View test on GitHub ↗
  31. ✓test_bedrock_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    1. Add a deployment named e2e-embeddings-bedrock-9edb5e0ee7c7 that calls bedrock/amazon.titan-embed-text-v2:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  32. ✓test_bedrock_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    1. Add a deployment named e2e-bedrock-cache-685e847e3b70 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  33. ✓test_bedrock_request_id_header_surfaces TestBedrockResponseHeaderstest_bedrock_provider_matrix_e2e.py›
    1. Add a deployment named e2e-bedrock-headers-326366c17cff that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  34. ✓test_bedrock_request_id_header_surfaces_on_stream TestBedrockResponseHeaderstest_bedrock_provider_matrix_e2e.py›
    1. Add a deployment named e2e-bedrock-headers-stream-dfc838d89535 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bedrock-headers-stream-dfc838d89535
    View test on GitHub ↗
  35. ✓test_bedrock_rerank_scores_top_n TestReranktest_rerank_e2e.py›
    1. Add a deployment named e2e-bedrock-rerank-b0279ce0894c that calls bedrock/arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/rerank request to e2e-bedrock-rerank-b0279ce0894c for "What is the capital of the United States?"
    View test on GitHub ↗
  36. ✓test_bedrock_titan_embeds_token_array_input_as_its_decoded_text TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    1. Add a deployment named e2e-embeddings-titan-tokens-ea66ebfd3796 that calls bedrock/amazon.titan-embed-text-v2:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  37. ✓test_bridged_stream_delivers_content_finish_reason_and_done TestResponsesBridgeChatCompletionsStreamingtest_responses_bridge_streaming_e2e.py›
    1. Add a deployment named e2e-bridge-stream-10ecd6798179 that calls openai/gpt-5.3-codex
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bridge-stream-10ecd6798179
    View test on GitHub ↗
  38. ✓test_bridged_stream_reassembles_tool_call TestResponsesBridgeChatCompletionsStreamingtest_responses_bridge_streaming_e2e.py›
    1. Add a deployment named e2e-bridge-stream-f29ddce2e388 that calls openai/gpt-5.3-codex
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bridge-stream-f29ddce2e388
    View test on GitHub ↗
  39. ✓test_bridged_stream_shares_one_chunk_id TestResponsesBridgeChatCompletionsStreamingtest_responses_bridge_streaming_e2e.py›
    1. Add a deployment named e2e-bridge-stream-be8ef2a1671b that calls openai/gpt-5.3-codex
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bridge-stream-be8ef2a1671b
    View test on GitHub ↗
  40. ✓test_cache_read_tokens_bill_at_the_cache_read_rate TestTogetherChatCompletionstest_together_ai_e2e.py›
    1. Add a deployment named e2e-together-6001f032d976 that calls together_ai/deepseek-ai/DeepSeek-V4-Flash-0731
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-6001f032d976
    4. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  41. ✓test_caller_completion_window_bills_its_rates TestSailResponsestest_sail_e2e.py›
    1. Add a deployment named e2e-sail-d0c0b363149f that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  42. ✓test_calls_without_auth_is_denied TestRealtimeHttptest_realtime_http_e2e.py›

    No steps recorded

    View test on GitHub ↗
  43. ✓test_cancel_background_response TestBackgroundResponseCanceltest_responses_retrieve_e2e.py›
    1. Add a deployment named e2e-resp-cancel-24537e85491f that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  44. ✓test_cancel_background_streaming_response_by_streamed_id TestBackgroundResponseCanceltest_responses_retrieve_e2e.py›
    1. Add a deployment named e2e-resp-cancel-stream-76ea7b779921 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  45. ✓test_chat_returns_real_completion[claude-haiku-4-5-anthropic] TestChatCompletionsRegressiontest_chat_completions_regression_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to claude-haiku-4-5
    View test on GitHub ↗
  46. ✓test_chat_returns_real_completion[gemini-2.5-flash-gemini] TestChatCompletionsRegressiontest_chat_completions_regression_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  47. ✓test_chat_returns_real_completion[gpt-5.5-openai] TestChatCompletionsRegressiontest_chat_completions_regression_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  48. ✓test_chat_stream_is_sse_and_ends_with_done TestChatStreamContracttest_chat_stream_contract_e2e.py›
    1. Add a deployment named e2e-chat-stream-b6ff0fb5ec05 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-chat-stream-b6ff0fb5ec05
    View test on GitHub ↗
  49. ✓test_chat_template_kwargs_reach_together TestTogetherChatCompletionstest_together_ai_e2e.py›
    1. Add a deployment named e2e-together-504e85aa0151 that calls together_ai/Qwen/Qwen3.5-9B
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-504e85aa0151
    View test on GitHub ↗
  50. ✓test_client_secret_missing_auth_is_denied TestRealtimeHttptest_realtime_http_e2e.py›
    1. Add a deployment named e2e-realtime-http-f84f3e2485c9 that calls openai/gpt-realtime
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  51. ✓test_cohere_chat_returns_content TestCohereChattest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-cohere-chat-85bcb4165f94 that calls cohere/command-r-08-2024
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-cohere-chat-85bcb4165f94
    View test on GitHub ↗
  52. ✓test_cohere_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    1. Add a deployment named e2e-embeddings-cohere-937c42d20a0a that calls cohere/embed-v4.0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  53. ✓test_container_file_lifecycle_through_the_gateway TestOpenAIContainerFilestest_containers_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  54. ✓test_contents_are_counted_by_the_provider_endpoint[gemini-2.5-flash-vertex] TestGeminiContentsTokenCountingtest_token_counter_gemini_contents_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  55. ✓test_contents_are_counted_by_the_provider_endpoint[gemini-2.5-flash] TestGeminiContentsTokenCountingtest_token_counter_gemini_contents_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  56. ✓test_converse_empty_messages_returns_client_error TestBedrockNativetest_bedrock_native_e2e.py›
    1. Add a deployment named e2e-bedrock-native-ec6230aceded that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  57. ✓test_converse_invalid_model_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    1. Add a deployment named e2e-bedrock-native-add7009e3fe7 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  58. ✓test_converse_missing_messages_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    1. Add a deployment named e2e-bedrock-native-2b73342398e0 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  59. ✓test_converse_output_format_returns_schema_json_text TestBedrockMessagestest_messages_bedrock_e2e.py›
    1. Add a deployment named e2e-messages-bedrock-ee1b416c97ed that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  60. ✓test_converse_returns_assistant TestBedrockNativetest_bedrock_native_e2e.py›
    1. Add a deployment named e2e-bedrock-native-bfdaadfc91e6 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  61. ✓test_converse_stream_returns_chunks TestBedrockNativetest_bedrock_native_e2e.py›
    1. Add a deployment named e2e-bedrock-native-227609292033 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  62. ✓test_cost_header_and_spend_row_match_the_registry_price TestTogetherChatCompletionstest_together_ai_e2e.py›
    1. Add a deployment named e2e-together-254574a6e248 that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  63. ✓test_cost_header_matches_the_spend_log[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  64. ✓test_cost_header_matches_the_spend_log[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  65. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  66. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  67. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  68. ✓test_cost_header_matches_the_spend_log[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  69. ✓test_cost_header_matches_the_spend_log[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  70. ✓test_cost_header_matches_the_spend_log[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  71. ✓test_cost_header_matches_the_spend_log[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  72. ✓test_cost_header_matches_the_spend_log[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  73. ✓test_cost_header_matches_the_spend_log[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  74. ✓test_cost_header_matches_the_spend_log[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  75. ✓test_cost_header_matches_the_spend_log[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  76. ✓test_cost_header_matches_the_spend_log[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  77. ✓test_cost_header_matches_the_spend_log[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  78. ✓test_cost_header_matches_the_spend_log[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  79. ✓test_cost_header_matches_the_spend_log[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  80. ✓test_cost_header_matches_the_spend_log[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  81. ✓test_create_batch_missing_input_file_id_returns_error TestFilesBatchesContracttest_files_batches_contract_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  82. ✓test_create_client_secret TestRealtimeHttptest_realtime_http_e2e.py›
    1. Add a deployment named e2e-realtime-http-0e49ed8ab8d7 that calls openai/gpt-realtime
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  83. ✓test_create_list_retrieve_delete_lifecycle TestVectorStorestest_vector_stores_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  84. ✓test_credential_backed_messages TestCredentialBackedMessagestest_credential_messages_e2e.py›
    1. Save the provider credential e2e-cred-0f4a318dfd7c
    2. Add a deployment named e2e-cred-messages-0f4a318dfd7c that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  85. ✓test_custom_pricing_is_billed_at_configured_rate TestCustomPricingtest_custom_pricing_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named custom-priced-flash-4c878be794a0 that calls gemini/gemini-2.5-flash
    3. Send a /chat/completions request to custom-priced-flash-4c878be794a0
    View test on GitHub ↗
  86. ✓test_custom_pricing_is_isolated_from_sibling_deployment TestCustomPricingtest_custom_pricing_e2e.py›
    1. Add a deployment named custom-priced-flash-8ce4e4e90026 that calls gemini/gemini-2.5-flash
    2. Add a deployment named base-flash-651216ec0722 that calls gemini/gemini-2.5-flash
    3. List the deployments from /model/info
    View test on GitHub ↗
  87. ✓test_deleted_response_is_no_longer_retrievable TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    1. Add a deployment named e2e-resp-delete-8620df23e21e that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  88. ✓test_embeddings_call_logs_its_cost TestOpenAIPassthroughSpendtest_passthrough_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  89. ✓test_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    1. Add a deployment named e2e-embeddings-680e184d70db that calls openai/text-embedding-3-small
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  90. ✓test_empty_image_returns_error TestImageEdittest_image_edits_e2e.py›
    1. Add a deployment named e2e-image-edit-187081700f20 that calls openai/gpt-image-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  91. ✓test_empty_input_returns_client_error TestResponsestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-val-480872c1f01c that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  92. ✓test_empty_messages_returns_client_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    1. Add a deployment named e2e-chat-sec-6ed48562306b that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  93. ✓test_empty_prompt_returns_client_error TestImageGenerationtest_image_generation_e2e.py›
    1. Add a deployment named e2e-image-c9c9c1f823d3 that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  94. ✓test_empty_prompt_returns_error TestImageEdittest_image_edits_e2e.py›
    1. Add a deployment named e2e-image-edit-12685817a5e2 that calls openai/gpt-image-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  95. ✓test_file_attach_poll_and_search TestVectorStorestest_vector_stores_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  96. ✓test_flagged_model_keeps_prompt_cache_across_system_reminder TestAnthropicChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    1. Add a deployment named e2e-chat-midsys-14cdf597a829 that calls anthropic/claude-opus-4-8
    2. Generate a virtual key with models: e2e-chat-midsys-14cdf597a829 and user id: e2e-test-user
    View test on GitHub ↗
  97. ✓test_flagged_model_keeps_prompt_cache_across_system_reminder TestBedrockInvokeChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    1. Add a deployment named e2e-chat-midsys-c1c32c2497fa that calls bedrock/invoke/us.anthropic.claude-sonnet-5
    2. Generate a virtual key with models: e2e-chat-midsys-c1c32c2497fa and user id: e2e-test-user
    View test on GitHub ↗
  98. ✓test_gemini TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    1. Add a deployment named e2e-chat-tool-loop-b6ad286ff393 that calls gemini/gemini-3.5-flash-lite
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-b6ad286ff393
    View test on GitHub ↗
  99. ✓test_gemini_chat_returns_content_and_logs_cost TestGeminiChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-gemini-chat-161fd01d71cf that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-gemini-chat-161fd01d71cf
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  100. ✓test_gemini_passthrough_nonstreaming_logs_costtest_passthrough_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  101. ✓test_gemini_passthrough_streaming_logs_costtest_passthrough_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  102. ✓test_gemini_passthrough_tool_call_logs_costtest_passthrough_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  103. ✓test_generate_content_returns_response_cost_header TestGoogleNativeGenerateContenttest_google_native_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-google-native-9246bb462eb0 that calls gemini/gemini-2.5-flash
    View test on GitHub ↗
  104. ✓test_image_edit_returns_image TestImageEdittest_image_edits_e2e.py›
    1. Add a deployment named e2e-image-edit-dab0dc7a3927 that calls openai/gpt-image-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  105. ✓test_image_generation_returns_image TestImageGenerationtest_image_generation_e2e.py›
    1. Add a deployment named e2e-image-b4635448dfac that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  106. ✓test_input_items_list_the_stored_prompt TestStoredResponseLifecycletest_responses_retrieve_e2e.py›
    1. Add a deployment named e2e-resp-items-7faf8d77eeb2 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  107. ✓test_invalid_chunking_returns_error TestVectorStorestest_vector_stores_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  108. ✓test_invalid_max_completion_tokens_return_client_errors TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    1. Add a deployment named e2e-chat-sec-cee9fb2f6b9d that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  109. ✓test_invalid_n_returns_client_error TestImageGenerationtest_image_generation_e2e.py›
    1. Add a deployment named e2e-image-7bfb37bbcf5c that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  110. ✓test_invalid_role_returns_client_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    1. Add a deployment named e2e-chat-sec-7a8c9bf83a69 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  111. ✓test_invalid_size_returns_client_error TestImageGenerationtest_image_generation_e2e.py›
    1. Add a deployment named e2e-image-c88fbecf01d1 that calls openai/gpt-image-1-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  112. ✓test_invalid_temperatures_return_client_errors TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    1. Add a deployment named e2e-chat-sec-cef37658fda8 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  113. ✓test_invoke_invalid_temperature_returns_client_error TestBedrockNativetest_bedrock_native_e2e.py›
    1. Add a deployment named e2e-bedrock-native-0914badb3cdb that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  114. ✓test_invoke_missing_max_tokens_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    1. Add a deployment named e2e-bedrock-native-fd3bf70c5852 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  115. ✓test_invoke_missing_messages_returns_error TestBedrockNativetest_bedrock_native_e2e.py›
    1. Add a deployment named e2e-bedrock-native-1499a1764cd9 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  116. ✓test_invoke_regional_id_completes TestBedrockInvokeRegionalModelIdstest_bedrock_provider_matrix_e2e.py›
    1. Add a deployment named e2e-bedrock-invoke-b6c099d1c62f that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-invoke-b6c099d1c62f
    View test on GitHub ↗
  117. ✓test_invoke_regional_id_streams TestBedrockInvokeRegionalModelIdstest_bedrock_provider_matrix_e2e.py›
    1. Add a deployment named e2e-bedrock-invoke-stream-54fba7b57758 that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-bedrock-invoke-stream-54fba7b57758
    View test on GitHub ↗
  118. ✓test_invoke_returns_message TestBedrockNativetest_bedrock_native_e2e.py›
    1. Add a deployment named e2e-bedrock-native-170ecec0cc02 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  119. ✓test_invoke_stream_returns_chunks TestBedrockNativetest_bedrock_native_e2e.py›
    1. Add a deployment named e2e-bedrock-native-d74fcb1fcbb3 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  120. ✓test_messages_logs_cost_matching_the_response_header TestAnthropicMessagestest_messages_e2e.py›
    1. Add a deployment named e2e-messages-cost-944529becc27 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  121. ✓test_messages_output_format_returns_schema_json TestAnthropicMessagestest_messages_e2e.py›
    1. Add a deployment named e2e-messages-3a12e2939db3 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  122. ✓test_messages_returns_completion TestAnthropicMessagestest_messages_e2e.py›
    1. Add a deployment named e2e-messages-e87d4f4f65a5 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  123. ✓test_messages_streams_completion TestAnthropicMessagestest_messages_e2e.py›
    1. Add a deployment named e2e-messages-84732a9ebb35 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  124. ✓test_messages_tool_use TestAnthropicMessagestest_messages_e2e.py›
    1. Add a deployment named e2e-messages-806f67956a25 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  125. ✓test_missing_file_returns_error TestAudioTranscriptionstest_audio_transcriptions_e2e.py›
    1. Add a deployment named e2e-transcribe-3e3915a0db19 that calls openai/gpt-4o-mini-transcribe
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  126. ✓test_missing_input_returns_error TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    1. Add a deployment named e2e-embeddings-missin-5b2c1e808d90 that calls openai/text-embedding-3-small
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  127. ✓test_missing_messages_returns_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    1. Add a deployment named e2e-chat-sec-914f382ed262 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  128. ✓test_missing_model_returns_client_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    1. Add a deployment named e2e-chat-sec-8c198a4a4f03 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  129. ✓test_missing_model_returns_client_error TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  130. ✓test_missing_model_returns_client_error TestResponsestest_responses_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  131. ✓test_missing_model_returns_error TestAudioTranscriptionstest_audio_transcriptions_e2e.py›
    1. Add a deployment named e2e-transcribe-ce5d5fc642fe that calls openai/gpt-4o-mini-transcribe
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  132. ✓test_missing_model_returns_error TestAnthropicMessagestest_messages_e2e.py›
    1. Add a deployment named e2e-messages-6c55f508bb7e that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  133. ✓test_mistral TestChatToolResultRoundTriptest_chat_tool_round_trip_e2e.py›
    1. Add a deployment named e2e-chat-tool-loop-442a9d86cbb0 that calls mistral/mistral-medium-3.5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-tool-loop-442a9d86cbb0
    View test on GitHub ↗
  134. ✓test_mistral_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    1. Add a deployment named e2e-embeddings-mistral-2cb8f1b8cdd7 that calls mistral/mistral-embed
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  135. ✓test_model_info_reports_custom_pricing TestCustomPricingtest_custom_pricing_e2e.py›
    1. Add a deployment named custom-priced-flash-3e7b8848aa79 that calls gemini/gemini-2.5-flash
    2. List the deployments from /model/info
    View test on GitHub ↗
  136. ✓test_moderations_flags_violent_content TestModerationstest_moderations_e2e.py›
    1. Add a deployment named e2e-moderation-8c1a9e6614f8 that calls openai/omni-moderation-latest
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  137. ✓test_moderations_passes_benign_content TestModerationstest_moderations_e2e.py›
    1. Add a deployment named e2e-moderation-9e8813db2bfd that calls openai/omni-moderation-latest
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  138. ✓test_multi_turn_history_is_honored TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    1. Add a deployment named e2e-chat-sec-e7077e525919 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-chat-sec-e7077e525919
    View test on GitHub ↗
  139. ✓test_nova_sonic_response_create_completes TestNovaSonicRealtimerealtime/test_realtime_bedrock_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-nova-sonic-8e4fdef75a2c that calls bedrock/amazon.nova-2-sonic-v1:0
    View test on GitHub ↗
  140. ✓test_nova_stream_relays_text_usage_and_stop TestBedrockMessagestest_messages_bedrock_e2e.py›
    1. Add a deployment named e2e-messages-bedrock-ad561bb8e28b that calls bedrock/us.amazon.nova-2-lite-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  141. ✓test_ocr_reads_the_document_and_bills_its_pages[azure-ai-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    1. Add a deployment named ocr-content-azure-ai-image-4c59473b7b83 that calls azure_ai/mistral-document-ai-2512
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  142. ✓test_ocr_reads_the_document_and_bills_its_pages[cohere-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    1. Add a deployment named ocr-content-cohere-image-c35b7f5d9f78 that calls cohere/parse-v5.0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  143. ✓test_ocr_reads_the_document_and_bills_its_pages[mistral-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    1. Add a deployment named ocr-content-mistral-image-5fc24fbc5ea3 that calls mistral/mistral-ocr-latest
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  144. ✓test_ocr_reads_the_document_and_bills_its_pages[mistral-pdf] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    1. Add a deployment named ocr-content-mistral-pdf-6d8711d9d14b that calls mistral/mistral-ocr-latest
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  145. ✓test_ocr_reads_the_document_and_bills_its_pages[vertex-mistral-image] TestOcrDocumentContenttest_ocr_rust_e2e.py›
    1. Add a deployment named ocr-content-vertex-mistral-image-a8e55b30c57c that calls vertex_ai/mistral-ocr-2505
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  146. ✓test_openai_chat_logs_cost TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-openai-cost-1129d2a5cd1f that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-cost-1129d2a5cd1f
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  147. ✓test_openai_chat_reasoning_reports_reasoning_tokens TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-openai-reasoning-805f1aa90c5c that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-reasoning-805f1aa90c5c
    View test on GitHub ↗
  148. ✓test_openai_chat_returns_tool_call TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-openai-tool-14f0041bb710 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-tool-14f0041bb710
    View test on GitHub ↗
  149. ✓test_openai_chat_streams_real_content TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-openai-chat-40a8ad646e52 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-openai-chat-40a8ad646e52
    View test on GitHub ↗
  150. ✓test_openai_chat_streams_tool_call TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-openai-tool-stream-bee2ac2d0955 that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-openai-tool-stream-bee2ac2d0955
    View test on GitHub ↗
  151. ✓test_openai_chat_structured_output_conforms_to_schema TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-openai-schema-9ac191a544aa that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-schema-9ac191a544aa
    View test on GitHub ↗
  152. ✓test_openai_chat_vision_describes_image TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-openai-vision-5760c5dbc7ac that calls openai/gpt-4o
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-vision-5760c5dbc7ac
    View test on GitHub ↗
  153. ✓test_openai_family_model_id_completes_with_max_tokens TestBedrockOpenAIFamilyDefaultRoutetest_bedrock_provider_matrix_e2e.py›
    1. Add a deployment named e2e-bedrock-openai-family-89aee6336fe7 that calls bedrock/global.openai.gpt-6-sol
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-bedrock-openai-family-89aee6336fe7
    View test on GitHub ↗
  154. ✓test_openai_prefix_chat_returns_completion_and_logs_its_cost TestOpenAIProviderPrefixChattest_passthrough_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  155. ✓test_openai_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    1. Add a deployment named e2e-openai-cache-c2bf57d6892d that calls openai/gpt-5.6
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-openai-cache-c2bf57d6892d
    View test on GitHub ↗
  156. ✓test_openai_service_tier_is_echoed TestServiceTiertest_provider_features_e2e.py›
    1. Add a deployment named e2e-service-tier-059c3aa9469d that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-service-tier-059c3aa9469d
    View test on GitHub ↗
  157. ✓test_output_format_returns_schema_json TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    1. Add a deployment named e2e-azure-foundry-messages-52b3b6d02072 that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-52b3b6d02072 and user id: e2e-test-user
    View test on GitHub ↗
  158. ✓test_passthrough_denies_model_outside_key_allowlist TestPassthroughModelAllowlisttest_passthrough_e2e.py›
    1. Generate a virtual key with models: gemini-2.5-flash
    View test on GitHub ↗
  159. ✓test_passthrough_prefix_lists_batches_from_openai TestOpenAIPassthroughPrefixtest_passthrough_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  160. ✓test_passthrough_prefix_uploads_a_file_to_openai TestOpenAIPassthroughPrefixtest_passthrough_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  161. ✓test_pipecat_tool_smoke[azure]realtime/test_realtime_pipecat_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  162. ✓test_pipecat_tool_smoke[gemini]realtime/test_realtime_pipecat_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  163. ✓test_pipecat_tool_smoke[openai]realtime/test_realtime_pipecat_e2e.py›
    1. Add a deployment named openai-realtime-befc31e2c193 that calls openai/gpt-realtime-2
    2. Add a deployment named azure-realtime-2e8f714fbc6b that calls azure/gpt-realtime
    3. Add a deployment named gemini-realtime-51569783fde0 that calls gemini/gemini-3.1-flash-live-preview
    4. Add a deployment named vertex-realtime-c012c1607167 that calls vertex_ai/gemini-live-2.5-flash-native-audio
    5. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  164. ✓test_plain_call_returns_a_message TestSailMessagestest_sail_e2e.py›
    1. Add a deployment named e2e-sail-cbcc518451b1 that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  165. ✓test_polly_generative_voice_returns_mp3 TestAwsPollySpeechtest_audio_speech_e2e.py›
    1. Add a deployment named e2e-speech-polly-f5b0cd861550 that calls aws_polly/generative
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  166. ✓test_realtime_upgrade_reaches_openai_through_the_passthrough_prefix TestOpenAIPassthroughWebsockettest_passthrough_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  167. ✓test_reasoner_returns_reasoning_by_default TestDeepSeekReasoningDisabletest_deepseek_reasoning_e2e.py›
    1. Add a deployment named e2e-deepseek-reasoner-6405c3d82948 that calls deepseek/deepseek-v4-pro
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-deepseek-reasoner-6405c3d82948
    View test on GitHub ↗
  168. ✓test_reasoning_effort_none_disables_reasoning TestDeepSeekReasoningDisabletest_deepseek_reasoning_e2e.py›
    1. Add a deployment named e2e-deepseek-reasoner-eedb78a8ec43 that calls deepseek/deepseek-v4-pro
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-deepseek-reasoner-eedb78a8ec43
    View test on GitHub ↗
  169. ✓test_reasoning_effort_none_reaches_together TestTogetherChatCompletionstest_together_ai_e2e.py›
    1. Add a deployment named e2e-together-94ad7aea24be that calls together_ai/Qwen/Qwen3.5-9B
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-94ad7aea24be
    View test on GitHub ↗
  170. ✓test_reasoning_items_replay_into_structured_output_after_tool_call TestResponsesOpenAIHostedFeaturestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-reasoning-a565cb6f49f9 that calls openai/gpt-5.4-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  171. ✓test_reasoning_streams_as_reasoning_content_deltas TestTogetherChatCompletionstest_together_ai_e2e.py›
    1. Add a deployment named e2e-together-32060a8873a4 that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-together-32060a8873a4
    View test on GitHub ↗
  172. ✓test_reasoning_surfaces_as_reasoning_content TestTogetherChatCompletionstest_together_ai_e2e.py›
    1. Read the model cost map
    2. Add a deployment named e2e-together-4edd9ccd8d3f that calls together_ai/zai-org/GLM-5.3-Flash
    3. Generate a virtual key with user id: e2e-test-user
    4. Send a /chat/completions request to e2e-together-4edd9ccd8d3f
    View test on GitHub ↗
  173. ✓test_replayed_reasoning_content_reaches_together TestTogetherChatCompletionstest_together_ai_e2e.py›
    1. Add a deployment named e2e-together-70dfafcc6523 that calls together_ai/MiniMaxAI/MiniMax-M3
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-70dfafcc6523
    View test on GitHub ↗
  174. ✓test_reply_carries_assistant_text_and_usage[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  175. ✓test_reply_carries_assistant_text_and_usage[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  176. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Add a deployment named e2e-matrix-gpt-4o-mini-env_ref-05954b740f1c that calls openai/gpt-4o-mini
    2. Save the provider credential e2e-matrix-gpt-4o-mini-f8eb2e05a695
    3. Add a deployment named e2e-matrix-gpt-4o-mini-stored_credential-f8eb2e05a695 that calls openai/gpt-4o-mini
    4. Add a deployment named e2e-matrix-gpt-5.4-mini-env_ref-0776af81a780 that calls openai/gpt-5.4-mini
    5. Save the provider credential e2e-matrix-gpt-5.4-mini-d9730bb113a0
    6. Add a deployment named e2e-matrix-gpt-5.4-mini-stored_credential-d9730bb113a0 that calls openai/gpt-5.4-mini
    7. Add a deployment named e2e-matrix-claude-haiku-4-5-env_ref-2c2c84a87863 that calls anthropic/claude-haiku-4-5
    8. Save the provider credential e2e-matrix-claude-haiku-4-5-b0b08a9eb9d7
    9. Add a deployment named e2e-matrix-claude-haiku-4-5-stored_credential-b0b08a9eb9d7 that calls anthropic/claude-haiku-4-5
    10. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  177. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  178. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  179. ✓test_reply_carries_assistant_text_and_usage[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  180. ✓test_reply_carries_assistant_text_and_usage[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  181. ✓test_reply_carries_assistant_text_and_usage[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  182. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  183. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  184. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  185. ✓test_reply_carries_assistant_text_and_usage[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  186. ✓test_reply_carries_assistant_text_and_usage[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  187. ✓test_reply_carries_assistant_text_and_usage[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  188. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  189. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  190. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  191. ✓test_reply_carries_assistant_text_and_usage[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  192. ✓test_required_tool_arguments_and_correlated_result[nonstream] TestOpenAIMessagesToolContinuationtest_messages_e2e.py›
    1. Add a deployment named e2e-bridge-tool-a3032d153a1e that calls openai/gpt-5.6
    2. Generate a virtual key with models: e2e-bridge-tool-a3032d153a1e and user id: e2e-test-user
    View test on GitHub ↗
  193. ✓test_required_tool_arguments_and_correlated_result[stream] TestOpenAIMessagesToolContinuationtest_messages_e2e.py›
    1. Add a deployment named e2e-bridge-tool-9e28294ec382 that calls openai/gpt-5.6
    2. Generate a virtual key with models: e2e-bridge-tool-9e28294ec382 and user id: e2e-test-user
    View test on GitHub ↗
  194. ✓test_rerank_scores_top_n TestReranktest_rerank_e2e.py›
    1. Add a deployment named e2e-rerank-bcb01c90806a that calls cohere/rerank-v3.5
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/rerank request to e2e-rerank-bcb01c90806a for "What is the capital of the United States?"
    View test on GitHub ↗
  195. ✓test_response_format_json_schema_shapes_the_reply TestTogetherChatCompletionstest_together_ai_e2e.py›
    1. Add a deployment named e2e-together-770d75388779 that calls together_ai/Qwen/Qwen3.5-9B
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-770d75388779
    View test on GitHub ↗
  196. ✓test_responses_anthropic_returns_completion TestResponsestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-21c3d69c7930 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  197. ✓test_responses_anthropic_returns_function_call TestResponsestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-94858282df11 that calls anthropic/claude-haiku-4-5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  198. ✓test_responses_azure_openai_returns_completion TestResponsestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-azure-openai-33c1a30b7588 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  199. ✓test_responses_azure_openai_returns_function_call TestResponsestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-azure-openai-tool-5b57fd1c10b6 that calls azure/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  200. ✓test_responses_bedrock_returns_completion TestResponsestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-a8f27ea9379e that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  201. ✓test_responses_bedrock_returns_function_call TestResponsestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-a77d32533d55 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  202. ✓test_responses_logs_cost TestResponsestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-0b5cae63f014 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  203. ✓test_responses_returns_completion TestResponsestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-7efd75dc8166 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  204. ✓test_responses_returns_function_call TestResponsestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-d3282c67c1bd that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  205. ✓test_responses_streaming_returns_completion TestResponsestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-85edcab439f3 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  206. ✓test_responses_upgrade_is_accepted_on_the_openai_prefix TestOpenAIPassthroughWebsockettest_passthrough_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  207. ✓test_responses_vertex_returns_completion TestResponsestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-vertex-d6c2e1d95334 that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  208. ✓test_responses_vertex_returns_function_call TestResponsestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-vertex-tool-2d2fbbc5303b that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  209. ✓test_responses_vision_describes_image TestResponsestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-c4a28eb510a7 that calls openai/gpt-4o
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  210. ✓test_retrieve_invalid_batch_id_returns_error TestFilesBatchesContracttest_files_batches_contract_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  211. ✓test_rust_ocr_response[azure-ai] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    1. Add a deployment named rust-ocr-azure-ai-7fe836edb7d2 that calls azure_ai/mistral-document-ai-2512
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-azure-ai-7fe836edb7d2
    View test on GitHub ↗
  212. ✓test_rust_ocr_response[azure-document-intelligence] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    1. Add a deployment named rust-ocr-azure-document-intelligence-873fc565a26f that calls azure_ai/doc-intelligence/prebuilt-layout
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-azure-document-intelligence-873fc565a26f
    View test on GitHub ↗
  213. ✓test_rust_ocr_response[mistral] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    1. Add a deployment named rust-ocr-mistral-ee484c442885 that calls mistral/mistral-ocr-latest
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-mistral-ee484c442885
    View test on GitHub ↗
  214. ✓test_rust_ocr_response[vertex-mistral] TestRustOcrGatewaytest_ocr_rust_e2e.py›
    1. Add a deployment named rust-ocr-vertex-mistral-d7c3d4cc07d9 that calls vertex_ai/mistral-ocr-2505
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/ocr request to rust-ocr-vertex-mistral-d7c3d4cc07d9
    View test on GitHub ↗
  215. ✓test_service_account_key_reads_container_file_by_native_id TestAzureContainerFilestest_containers_e2e.py›
    1. Add a deployment named e2e-containers-decoy-b28babcd3525 that calls azure/gpt-5.4-nano
    2. Add a deployment named e2e-containers-b28babcd3525 that calls azure/gpt-5.4-nano
    View test on GitHub ↗
  216. ✓test_service_account_key_reads_container_file_created_by_a_streamed_response TestAzureContainerFilestest_containers_e2e.py›
    1. Add a deployment named e2e-containers-decoy-ed1d19e63a6f that calls azure/gpt-5.4-nano
    2. Add a deployment named e2e-containers-ed1d19e63a6f that calls azure/gpt-5.4-nano
    View test on GitHub ↗
  217. ✓test_service_tier_bills_the_matching_completion_window[auto-base] TestSailChatCompletionstest_sail_e2e.py›
    1. Add a deployment named e2e-sail-88fc67083532 that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  218. ✓test_service_tier_bills_the_matching_completion_window[balanced-balanced] TestSailChatCompletionstest_sail_e2e.py›
    1. Add a deployment named e2e-sail-e3f09ab5f5e4 that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  219. ✓test_shell_tool_stream_surfaces_shell_call_and_its_output TestResponsesOpenAIHostedFeaturestest_responses_e2e.py›
    1. Add a deployment named e2e-responses-shell-6e3d5bb1f042 that calls openai/gpt-5.5
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  220. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  221. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  222. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  223. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  224. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  225. ✓test_stream_delivers_text_usage_and_a_terminal_event[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  226. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  227. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  228. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  229. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  230. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  231. ✓test_stream_delivers_text_usage_and_a_terminal_event[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  232. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  233. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  234. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  235. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  236. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  237. ✓test_stream_delivers_text_usage_and_a_terminal_event[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  238. ✓test_stream_generate_content_frames_sse_the_way_google_sdks_expect TestGoogleNativeGenerateContenttest_google_native_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-google-native-a80cbb0b743a that calls gemini/gemini-2.5-flash
    View test on GitHub ↗
  239. ✓test_streamed_responses_call_logs_its_cost TestOpenAIPassthroughSpendtest_passthrough_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  240. ✓test_streams_text_deltas TestTogetherMessagestest_together_ai_e2e.py›
    1. Add a deployment named e2e-together-38bd7a760be8 that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /v1/messages request to e2e-together-38bd7a760be8
    View test on GitHub ↗
  241. ✓test_success_response_matches_chat_completion_contract TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    1. Add a deployment named e2e-chat-sec-677243ff9bd5 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  242. ✓test_temperature_boundaries_succeed TestChatCompletionsContracttest_chat_completions_contract_e2e.py›
    1. Add a deployment named e2e-chat-sec-5901f235d089 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  243. ✓test_text_completion_returns_text TestCompletionsEndpointtest_completions_endpoint_e2e.py›
    1. Add a deployment named e2e-completions-1363fff51024 that calls openai/gpt-5.4-nano
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  244. ✓test_text_conversation[azure]realtime/test_realtime_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  245. ✓test_text_conversation[gemini]realtime/test_realtime_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  246. ✓test_text_conversation[openai]realtime/test_realtime_e2e.py›
    1. Add a deployment named openai-realtime-1872fa0d01da that calls openai/gpt-realtime-2
    2. Add a deployment named azure-realtime-2517c1c85473 that calls azure/gpt-realtime
    3. Add a deployment named gemini-realtime-1eee85119e86 that calls gemini/gemini-3.1-flash-live-preview
    4. Add a deployment named vertex-realtime-ae2692a9258e that calls vertex_ai/gemini-live-2.5-flash-native-audio
    5. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  247. ✓test_text_conversation[vertex_ai]realtime/test_realtime_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  248. ✓test_thinking_disabled_disables_reasoning TestDeepSeekReasoningDisabletest_deepseek_reasoning_e2e.py›
    1. Add a deployment named e2e-deepseek-reasoner-79e9e4b8a129 that calls deepseek/deepseek-v4-pro
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-deepseek-reasoner-79e9e4b8a129
    View test on GitHub ↗
  249. ✓test_tool_call_is_returned TestTogetherChatCompletionstest_together_ai_e2e.py›
    1. Add a deployment named e2e-together-a76367e8de6a that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-a76367e8de6a
    View test on GitHub ↗
  250. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  251. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  252. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  253. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  254. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  255. ✓test_tool_call_is_returned_named_and_addressable[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  256. ✓test_tool_call_is_returned_named_and_addressable[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  257. ✓test_tool_call_is_returned_named_and_addressable[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  258. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  259. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  260. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  261. ✓test_tool_call_is_returned_named_and_addressable[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  262. ✓test_tool_call_is_returned_named_and_addressable[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  263. ✓test_tool_call_is_returned_named_and_addressable[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  264. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  265. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  266. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  267. ✓test_tool_call_is_returned_named_and_addressable[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  268. ✓test_tool_call_is_streamed TestTogetherChatCompletionstest_together_ai_e2e.py›
    1. Add a deployment named e2e-together-f70bc011d4af that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-together-f70bc011d4af
    View test on GitHub ↗
  269. ✓test_tool_call_round_trip[azure]realtime/test_realtime_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  270. ✓test_tool_call_round_trip[gemini]realtime/test_realtime_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  271. ✓test_tool_call_round_trip[openai]realtime/test_realtime_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  272. ✓test_tool_call_round_trip[vertex_ai]realtime/test_realtime_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  273. ✓test_tool_result_round_trip TestTogetherChatCompletionstest_together_ai_e2e.py›
    1. Add a deployment named e2e-together-e36173966315 that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-together-e36173966315
    View test on GitHub ↗
  274. ✓test_tool_result_round_trip TestTogetherMessagestest_together_ai_e2e.py›
    1. Add a deployment named e2e-together-de6c40aedc06 that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/messages request to e2e-together-de6c40aedc06
    View test on GitHub ↗
  275. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  276. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  277. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  278. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  279. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  280. ✓test_tool_result_round_trip_reaches_the_model[chat_completions-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  281. ✓test_tool_result_round_trip_reaches_the_model[messages-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  282. ✓test_tool_result_round_trip_reaches_the_model[messages-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  283. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  284. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  285. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  286. ✓test_tool_result_round_trip_reaches_the_model[messages-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  287. ✓test_tool_result_round_trip_reaches_the_model[responses-claude-haiku-4-5-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  288. ✓test_tool_result_round_trip_reaches_the_model[responses-claude-haiku-4-5-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  289. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-4o-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  290. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-4o-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  291. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-5.4-mini-env_ref] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  292. ✓test_tool_result_round_trip_reaches_the_model[responses-gpt-5.4-mini-stored_credential] TestConversationalMatrixtest_conversational_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  293. ✓test_tool_use_block_is_returned TestTogetherMessagestest_together_ai_e2e.py›
    1. Add a deployment named e2e-together-d38dd709c956 that calls together_ai/zai-org/GLM-5.3-Flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /v1/messages request to e2e-together-d38dd709c956
    View test on GitHub ↗
  294. ✓test_tool_use_nonstream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    1. Add a deployment named e2e-azure-foundry-messages-09d70f56206d that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-09d70f56206d and user id: e2e-test-user
    View test on GitHub ↗
  295. ✓test_tool_use_stream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py›
    1. Add a deployment named e2e-azure-foundry-messages-272bcee97f2b that calls azure_ai/claude-haiku-4-5
    2. Generate a virtual key with models: e2e-azure-foundry-messages-272bcee97f2b and user id: e2e-test-user
    View test on GitHub ↗
  296. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestAnthropicChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    1. Add a deployment named e2e-chat-midsys-ee076b7ae8fa that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-chat-midsys-ee076b7ae8fa and user id: e2e-test-user
    View test on GitHub ↗
  297. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestBedrockInvokeChatMidConversationSystemtest_chat_mid_conversation_system_e2e.py›
    1. Add a deployment named e2e-chat-midsys-93b1e7c30541 that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with models: e2e-chat-midsys-93b1e7c30541 and user id: e2e-test-user
    View test on GitHub ↗
  298. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestBedrockInvokeMidConversationSystemtest_messages_mid_conversation_system_e2e.py›
    1. Add a deployment named e2e-midsys-8a5bc9f7e2a2 that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with models: e2e-midsys-8a5bc9f7e2a2 and user id: e2e-test-user
    View test on GitHub ↗
  299. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestAzureFoundryMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›
    1. Add a deployment named e2e-midsys-ff8bdcc8d5a7 that calls azure_ai/claude-opus-4-7
    2. Generate a virtual key with models: e2e-midsys-ff8bdcc8d5a7 and user id: e2e-test-user
    View test on GitHub ↗
  300. ✓test_unflagged_model_converts_system_reminder_and_succeeds TestVertexMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›
    1. Add a deployment named e2e-midsys-4ebeb01b3d58 that calls vertex_ai/claude-sonnet-4-6
    2. Generate a virtual key with models: e2e-midsys-4ebeb01b3d58 and user id: e2e-test-user
    View test on GitHub ↗
  301. ✓test_unknown_service_tier_is_dropped_and_billed_asap[5] TestSailChatCompletionstest_sail_e2e.py›
    1. Add a deployment named e2e-sail-62540847ba31 that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  302. ✓test_unknown_service_tier_is_dropped_and_billed_asap[bogus] TestSailChatCompletionstest_sail_e2e.py›
    1. Add a deployment named e2e-sail-ba84f9162244 that calls sail/zai-org/GLM-5.3
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  303. ✓test_upload_without_purpose_returns_error TestFilesBatchesContracttest_files_batches_contract_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  304. ✓test_upstream_handshake_refusal_is_an_error_event_and_policy_close[azure-bad-key]realtime/test_realtime_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named azure-realtime-refused-33e717fb22be that calls azure/gpt-realtime
    View test on GitHub ↗
  305. ✓test_verbose_json_returns_word_timestamps TestWhisperTranscriptionFormatstest_audio_transcriptions_e2e.py›
    1. Add a deployment named e2e-whisper-verbose-525acd5e649d that calls openai/whisper-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  306. ✓test_vertex_chat_returns_content TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-vertex-chat-dfc24a3c5171 that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-vertex-chat-dfc24a3c5171
    View test on GitHub ↗
  307. ✓test_vertex_chat_returns_tool_call TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-vertex-tool-29f3cdc7de72 that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-vertex-tool-29f3cdc7de72
    View test on GitHub ↗
  308. ✓test_vertex_chat_streams_real_content TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-vertex-stream-381d25a0c62e that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a streaming /chat/completions request to e2e-vertex-stream-381d25a0c62e
    View test on GitHub ↗
  309. ✓test_vertex_chat_vision_describes_image TestVertexChatCompletionstest_chat_completions_regression_e2e.py›
    1. Add a deployment named e2e-vertex-vision-5c000328777d that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-vertex-vision-5c000328777d
    View test on GitHub ↗
  310. ✓test_vertex_embeddings_honor_requested_dimensions TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    1. Add a deployment named e2e-embeddings-vertex-dims-36cea4c7471a that calls vertex_ai/text-embedding-005
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  311. ✓test_vertex_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    1. Add a deployment named e2e-embeddings-vertex-a11ce8cc52f3 that calls vertex_ai/text-embedding-005
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  312. ✓test_vertex_multimodal_embeddings_honor_dimensions_and_are_costed TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py›
    1. Add a deployment named e2e-embeddings-vertex-mm-3863b17f01ff that calls vertex_ai/multimodalembedding@001
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  313. ✓test_vertex_passthrough_via_managed_model_logs_cost TestVertexPassthroughSpendTrackingtest_vertex_passthrough_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  314. ✓test_vertex_prompt_caching_reads_cache TestCacheControltest_cache_control.py›
    1. Add a deployment named e2e-vertex-cache-e0a768d87176 that calls vertex_ai/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    3. Wait for the request's spend log in /spend/logs
    4. List the deployments from /model/info
    View test on GitHub ↗
  315. ✓test_vtt_format_returns_webvtt_transcript TestWhisperTranscriptionFormatstest_audio_transcriptions_e2e.py›
    1. Add a deployment named e2e-whisper-vtt-34644c1dc091 that calls openai/whisper-1
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  316. −test_empty_input_returns_error TestAudioSpeechtest_audio_speech_e2e.py›

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on empty input instead of surfacing the provider 4xx

    View test on GitHub ↗
  317. −test_flagged_model_keeps_prompt_cache_across_system_reminder TestBedrockInvokeMidConversationSystemtest_messages_mid_conversation_system_e2e.py›

    No steps recorded

    LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

    View test on GitHub ↗
  318. −test_flagged_model_keeps_prompt_cache_across_system_reminder TestAzureFoundryMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›

    No steps recorded

    LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

    View test on GitHub ↗
  319. −test_flagged_model_keeps_prompt_cache_across_system_reminder TestVertexMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›

    No steps recorded

    LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

    View test on GitHub ↗
  320. −test_gemini_passthrough_returns_the_same_header_contract_as_the_managed_routetest_passthrough_e2e.py›

    No steps recorded

    stage red: product gap, native passthrough returns no x-litellm-response-cost or x-ratelimit-* headers

    View test on GitHub ↗
  321. −test_hosted_vllm_chat_returns_content TestHostedVllmChattest_chat_completions_regression_e2e.py›

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  322. −test_invalid_response_id_returns_error TestResponsesRetrievetest_responses_retrieve_e2e.py›

    No steps recorded

    stage red: product gap (LIT-5447), retrieving an unknown response id returns 400 (model=None) instead of 404

    View test on GitHub ↗
  323. −test_invalid_voice_returns_error TestAudioSpeechtest_audio_speech_e2e.py›

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on invalid voice instead of surfacing the provider 4xx

    View test on GitHub ↗
  324. −test_missing_document_returns_error TestRustOcrGatewaytest_ocr_rust_e2e.py›

    No steps recorded

    stage red: product gap, /v1/ocr 500s (aocr TypeError) on missing document instead of 400

    View test on GitHub ↗
  325. −test_missing_input_returns_error TestAudioSpeechtest_audio_speech_e2e.py›

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on missing input instead of 400

    View test on GitHub ↗
  326. −test_missing_input_returns_error TestModerationstest_moderations_e2e.py›

    No steps recorded

    stage red: product gap, /v1/moderations 500s (KeyError 'input') on missing input instead of 400

    View test on GitHub ↗
  327. −test_missing_input_returns_error TestResponsestest_responses_e2e.py›

    No steps recorded

    stage red: product gap, /v1/responses 500s (aresponses TypeError) on missing input instead of 400

    View test on GitHub ↗
  328. −test_missing_max_tokens_returns_error TestAnthropicMessagestest_messages_e2e.py›

    No steps recorded

    stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing max_tokens instead of 400

    View test on GitHub ↗
  329. −test_missing_messages_returns_error TestAnthropicMessagestest_messages_e2e.py›

    No steps recorded

    stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing messages instead of 400

    View test on GitHub ↗
  330. −test_missing_model_returns_error TestAudioSpeechtest_audio_speech_e2e.py›

    No steps recorded

    stage red: product gap, /v1/audio/speech 500s on missing model instead of 400

    View test on GitHub ↗
  331. −test_missing_prompt_returns_error TestImageGenerationtest_image_generation_e2e.py›

    No steps recorded

    stage red: product gap, /v1/images/generations 500s (aimage_generation TypeError) on missing prompt instead of 400

    View test on GitHub ↗
  332. −test_retrieve_invalid_id_returns_error TestVectorStorestest_vector_stores_e2e.py›

    No steps recorded

    stage red: product gap, retrieving a nonexistent vector store returns 2xx with an error envelope in the body instead of 404

    View test on GitHub ↗
  333. −test_search_missing_query_returns_error TestVectorStorestest_vector_stores_e2e.py›

    No steps recorded

    stage red: product gap, vector store search 500s (asearch TypeError) on missing query instead of 400

    View test on GitHub ↗
  334. −test_static_and_x_pass_headers_reach_upstream TestPassthroughHeaderstest_passthrough_headers_e2e.py›

    No steps recorded

    stage red: ALB sends custom passthrough paths to backend pods, which lack ANTHROPIC_API_KEY, so the os.environ header forwards unresolved and Anthropic 401s; routing user-defined paths to the gateway needs a product/infra decision (fixed path namespace or dynamic ingress)

    View test on GitHub ↗
  335. −test_store_and_retrieve_by_id TestResponsesRetrievetest_responses_retrieve_e2e.py›

    No steps recorded

    stage red: product gap (LIT-5446), retrieve returns a different id than the stored response (non-idempotent response-id re-encryption)

    View test on GitHub ↗
  336. −test_vertex_partner_model_streams_openai_shaped_chunks[vertex_ai/mistral-small-2503] TestVertexPartnerChatCompletionstest_chat_completions_regression_e2e.py›

    No steps recorded

    the e2e Vertex project has no access to mistral-small-2503 (404 publisher model not found)

    View test on GitHub ↗
  337. −test_vertex_partner_model_streams_openai_shaped_chunks[vertex_ai/openai/gpt-oss-120b-maas] TestVertexPartnerChatCompletionstest_chat_completions_regression_e2e.py›

    No steps recorded

    never served by the e2e Vertex project (60s read timeout, no headers)

    View test on GitHub ↗
  338. −test_web_search_server_tool_is_served TestBedrockWebSearchServerTooltest_bedrock_web_search_server_tool_e2e.py›

    No steps recorded

    stage red: environment gap, the e2e stack neither enables the websearch_interception callback nor declares a search backend, so the request reaches bedrock's transformation and takes its by-design 400. Unskip once the ephemeral stack ships the config in this module's docstring.

    View test on GitHub ↗
  339. ✓test_a_client_built_for_another_proxy_reads_management_routes_back_from_that_proxy TestReplicasFortest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  340. ✓test_a_route_no_replica_serves_is_refused_rather_than_read_back_vacuously TestReplicasFortest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  341. ✓test_actor_key_cleanup_reports_failure_and_continues TestBoundManagementCallertest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  342. ✓test_an_exported_list_wins_over_the_base_url_rule TestParseControlPlaneReplicaUrlstest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  343. ✓test_any_other_endpoints_follow_the_base_url_rule TestStackEndpointsControlReplicastest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  344. ✓test_collapses_repeated_gateway_addresses_to_one_replica TestParseReplicaUrlstest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  345. ✓test_control_and_data_replica_readers_keep_the_caller TestSplitCallerPropagationtest_proxy_client.py›
    1. Read the key's settings back from /key/info
    2. Read /key/info on every proxy replica until it settles
    3. Read /key/info on every proxy replica until they all agree
    4. Read /v1/models on every proxy replica until they all agree
    View test on GitHub ↗
  346. ✓test_credentials_are_absent_from_binding_and_header_diagnostics TestBoundManagementCallertest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  347. ✓test_direct_delegated_and_replica_reads_keep_the_bound_caller[dashboard_session] TestBoundManagementCallertest_proxy_client.py›
    1. Read the key's settings back from /key/info
    2. Read /key/info on every proxy replica until they all agree
    3. Read /key/info on every proxy replica until it settles
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  348. ✓test_direct_delegated_and_replica_reads_keep_the_bound_caller[direct_jwt] TestBoundManagementCallertest_proxy_client.py›
    1. Read the key's settings back from /key/info
    2. Read /key/info on every proxy replica until they all agree
    3. Read /key/info on every proxy replica until it settles
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  349. ✓test_direct_delegated_and_replica_reads_keep_the_bound_caller[virtual_key] TestBoundManagementCallertest_proxy_client.py›
    1. Read the key's settings back from /key/info
    2. Read /key/info on every proxy replica until they all agree
    3. Read /key/info on every proxy replica until it settles
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  350. ✓test_each_replica_gets_its_own_full_budget TestAwaitConvergedEverywheretest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  351. ✓test_expired_shaped_token_is_sent_once_without_renewal TestSplitCallerPropagationtest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  352. ✓test_explicit_override_wins_without_rebinding_or_changing_master TestBoundManagementCallertest_proxy_client.py›
    1. Read the key's settings back from /key/info
    View test on GitHub ↗
  353. ✓test_fails_on_the_replica_that_never_lists_the_model[gateway-1] TestAwaitServableEverywheretest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  354. ✓test_fails_on_the_replica_that_never_lists_the_model[gateway-2] TestAwaitServableEverywheretest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  355. ✓test_falls_back_to_the_data_plane_address_when_unset TestParseReplicaUrlstest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  356. ✓test_gateway_pods_behind_one_router_read_management_routes_back_from_the_router TestReplicasFortest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  357. ✓test_management_operations_send_the_selected_credential[dashboard_session-add_team_member]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  358. ✓test_management_operations_send_the_selected_credential[dashboard_session-block_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  359. ✓test_management_operations_send_the_selected_credential[dashboard_session-connection_test]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  360. ✓test_management_operations_send_the_selected_credential[dashboard_session-create_customer]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  361. ✓test_management_operations_send_the_selected_credential[dashboard_session-create_mcp_server]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  362. ✓test_management_operations_send_the_selected_credential[dashboard_session-create_org]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  363. ✓test_management_operations_send_the_selected_credential[dashboard_session-create_tag]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  364. ✓test_management_operations_send_the_selected_credential[dashboard_session-create_team]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  365. ✓test_management_operations_send_the_selected_credential[dashboard_session-create_user]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  366. ✓test_management_operations_send_the_selected_credential[dashboard_session-customer_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  367. ✓test_management_operations_send_the_selected_credential[dashboard_session-delete_customer]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  368. ✓test_management_operations_send_the_selected_credential[dashboard_session-delete_key_strict]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  369. ✓test_management_operations_send_the_selected_credential[dashboard_session-delete_mcp_server]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  370. ✓test_management_operations_send_the_selected_credential[dashboard_session-delete_model_strict]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  371. ✓test_management_operations_send_the_selected_credential[dashboard_session-delete_org]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  372. ✓test_management_operations_send_the_selected_credential[dashboard_session-delete_tag]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  373. ✓test_management_operations_send_the_selected_credential[dashboard_session-delete_team_member]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  374. ✓test_management_operations_send_the_selected_credential[dashboard_session-delete_team]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  375. ✓test_management_operations_send_the_selected_credential[dashboard_session-delete_user_strict]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  376. ✓test_management_operations_send_the_selected_credential[dashboard_session-delete_user]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  377. ✓test_management_operations_send_the_selected_credential[dashboard_session-generate_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  378. ✓test_management_operations_send_the_selected_credential[dashboard_session-key_alias_count]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  379. ✓test_management_operations_send_the_selected_credential[dashboard_session-key_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  380. ✓test_management_operations_send_the_selected_credential[dashboard_session-key_list]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  381. ✓test_management_operations_send_the_selected_credential[dashboard_session-llm_only_key]test_proxy_client.py›
    1. Generate a virtual key with allowed routes: llm_api_routes
    View test on GitHub ↗
  382. ✓test_management_operations_send_the_selected_credential[dashboard_session-org_info_status]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  383. ✓test_management_operations_send_the_selected_credential[dashboard_session-org_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  384. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.create_credential]test_proxy_client.py›
    1. Save the provider credential owned
    View test on GitHub ↗
  385. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.create_model]test_proxy_client.py›
    1. Add a deployment named owned that calls synthetic
    View test on GitHub ↗
  386. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.create_team]test_proxy_client.py›
    1. Create a team with team alias: owned
    View test on GitHub ↗
  387. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.create_toolset]test_proxy_client.py›
    1. Create an MCP toolset with the tools
    View test on GitHub ↗
  388. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.delete_credential]test_proxy_client.py›
    1. Delete the provider credential
    View test on GitHub ↗
  389. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.delete_customers]test_proxy_client.py›
    1. Delete the end users owned
    View test on GitHub ↗
  390. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.delete_key]test_proxy_client.py›
    1. Delete the virtual key
    View test on GitHub ↗
  391. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.delete_model]test_proxy_client.py›
    1. Delete the deployment
    View test on GitHub ↗
  392. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.delete_team]test_proxy_client.py›
    1. Delete the team
    View test on GitHub ↗
  393. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.delete_toolset]test_proxy_client.py›
    1. Delete the MCP toolset
    View test on GitHub ↗
  394. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.delete_user]test_proxy_client.py›
    1. Delete the internal user
    View test on GitHub ↗
  395. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.generate_key]test_proxy_client.py›
    1. Generate a virtual key with default settings
    View test on GitHub ↗
  396. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.key_info]test_proxy_client.py›
    1. Read the key's settings back from /key/info
    View test on GitHub ↗
  397. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.memory_summary]test_proxy_client.py›
    1. Read memory usage from /debug/memory/summary on every proxy replica
    View test on GitHub ↗
  398. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.model_cost_map]test_proxy_client.py›
    1. Read the model cost map
    View test on GitHub ↗
  399. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.model_info]test_proxy_client.py›
    1. List the deployments from /model/info
    View test on GitHub ↗
  400. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.probe]test_proxy_client.py›
    1. Call the management route /user/info
    View test on GitHub ↗
  401. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.spend_logs]test_proxy_client.py›
    1. Read /spend/logs
    View test on GitHub ↗
  402. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.update_model]test_proxy_client.py›
    1. Update a deployment's settings to model: synthetic
    View test on GitHub ↗
  403. ✓test_management_operations_send_the_selected_credential[dashboard_session-proxy.update_toolset]test_proxy_client.py›
    1. Update an MCP toolset with toolset id: owned
    View test on GitHub ↗
  404. ✓test_management_operations_send_the_selected_credential[dashboard_session-regenerate_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  405. ✓test_management_operations_send_the_selected_credential[dashboard_session-reset_key_spend]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  406. ✓test_management_operations_send_the_selected_credential[dashboard_session-tag_list]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  407. ✓test_management_operations_send_the_selected_credential[dashboard_session-team_info_status]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  408. ✓test_management_operations_send_the_selected_credential[dashboard_session-team_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  409. ✓test_management_operations_send_the_selected_credential[dashboard_session-team_list_ids]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  410. ✓test_management_operations_send_the_selected_credential[dashboard_session-update_key_models]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  411. ✓test_management_operations_send_the_selected_credential[dashboard_session-update_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  412. ✓test_management_operations_send_the_selected_credential[dashboard_session-update_mcp_server]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  413. ✓test_management_operations_send_the_selected_credential[dashboard_session-update_org]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  414. ✓test_management_operations_send_the_selected_credential[dashboard_session-update_team]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  415. ✓test_management_operations_send_the_selected_credential[dashboard_session-update_user]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  416. ✓test_management_operations_send_the_selected_credential[dashboard_session-user_count]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  417. ✓test_management_operations_send_the_selected_credential[dashboard_session-user_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  418. ✓test_management_operations_send_the_selected_credential[dashboard_session-user_list_ids]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  419. ✓test_management_operations_send_the_selected_credential[direct_jwt-add_team_member]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  420. ✓test_management_operations_send_the_selected_credential[direct_jwt-block_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  421. ✓test_management_operations_send_the_selected_credential[direct_jwt-connection_test]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  422. ✓test_management_operations_send_the_selected_credential[direct_jwt-create_customer]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  423. ✓test_management_operations_send_the_selected_credential[direct_jwt-create_mcp_server]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  424. ✓test_management_operations_send_the_selected_credential[direct_jwt-create_org]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  425. ✓test_management_operations_send_the_selected_credential[direct_jwt-create_tag]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  426. ✓test_management_operations_send_the_selected_credential[direct_jwt-create_team]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  427. ✓test_management_operations_send_the_selected_credential[direct_jwt-create_user]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  428. ✓test_management_operations_send_the_selected_credential[direct_jwt-customer_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  429. ✓test_management_operations_send_the_selected_credential[direct_jwt-delete_customer]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  430. ✓test_management_operations_send_the_selected_credential[direct_jwt-delete_key_strict]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  431. ✓test_management_operations_send_the_selected_credential[direct_jwt-delete_mcp_server]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  432. ✓test_management_operations_send_the_selected_credential[direct_jwt-delete_model_strict]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  433. ✓test_management_operations_send_the_selected_credential[direct_jwt-delete_org]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  434. ✓test_management_operations_send_the_selected_credential[direct_jwt-delete_tag]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  435. ✓test_management_operations_send_the_selected_credential[direct_jwt-delete_team_member]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  436. ✓test_management_operations_send_the_selected_credential[direct_jwt-delete_team]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  437. ✓test_management_operations_send_the_selected_credential[direct_jwt-delete_user_strict]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  438. ✓test_management_operations_send_the_selected_credential[direct_jwt-delete_user]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  439. ✓test_management_operations_send_the_selected_credential[direct_jwt-generate_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  440. ✓test_management_operations_send_the_selected_credential[direct_jwt-key_alias_count]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  441. ✓test_management_operations_send_the_selected_credential[direct_jwt-key_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  442. ✓test_management_operations_send_the_selected_credential[direct_jwt-key_list]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  443. ✓test_management_operations_send_the_selected_credential[direct_jwt-llm_only_key]test_proxy_client.py›
    1. Generate a virtual key with allowed routes: llm_api_routes
    View test on GitHub ↗
  444. ✓test_management_operations_send_the_selected_credential[direct_jwt-org_info_status]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  445. ✓test_management_operations_send_the_selected_credential[direct_jwt-org_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  446. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.create_credential]test_proxy_client.py›
    1. Save the provider credential owned
    View test on GitHub ↗
  447. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.create_model]test_proxy_client.py›
    1. Add a deployment named owned that calls synthetic
    View test on GitHub ↗
  448. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.create_team]test_proxy_client.py›
    1. Create a team with team alias: owned
    View test on GitHub ↗
  449. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.create_toolset]test_proxy_client.py›
    1. Create an MCP toolset with the tools
    View test on GitHub ↗
  450. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.delete_credential]test_proxy_client.py›
    1. Delete the provider credential
    View test on GitHub ↗
  451. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.delete_customers]test_proxy_client.py›
    1. Delete the end users owned
    View test on GitHub ↗
  452. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.delete_key]test_proxy_client.py›
    1. Delete the virtual key
    View test on GitHub ↗
  453. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.delete_model]test_proxy_client.py›
    1. Delete the deployment
    View test on GitHub ↗
  454. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.delete_team]test_proxy_client.py›
    1. Delete the team
    View test on GitHub ↗
  455. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.delete_toolset]test_proxy_client.py›
    1. Delete the MCP toolset
    View test on GitHub ↗
  456. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.delete_user]test_proxy_client.py›
    1. Delete the internal user
    View test on GitHub ↗
  457. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.generate_key]test_proxy_client.py›
    1. Generate a virtual key with default settings
    View test on GitHub ↗
  458. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.key_info]test_proxy_client.py›
    1. Read the key's settings back from /key/info
    View test on GitHub ↗
  459. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.memory_summary]test_proxy_client.py›
    1. Read memory usage from /debug/memory/summary on every proxy replica
    View test on GitHub ↗
  460. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.model_cost_map]test_proxy_client.py›
    1. Read the model cost map
    View test on GitHub ↗
  461. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.model_info]test_proxy_client.py›
    1. List the deployments from /model/info
    View test on GitHub ↗
  462. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.probe]test_proxy_client.py›
    1. Call the management route /user/info
    View test on GitHub ↗
  463. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.spend_logs]test_proxy_client.py›
    1. Read /spend/logs
    View test on GitHub ↗
  464. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.update_model]test_proxy_client.py›
    1. Update a deployment's settings to model: synthetic
    View test on GitHub ↗
  465. ✓test_management_operations_send_the_selected_credential[direct_jwt-proxy.update_toolset]test_proxy_client.py›
    1. Update an MCP toolset with toolset id: owned
    View test on GitHub ↗
  466. ✓test_management_operations_send_the_selected_credential[direct_jwt-regenerate_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  467. ✓test_management_operations_send_the_selected_credential[direct_jwt-reset_key_spend]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  468. ✓test_management_operations_send_the_selected_credential[direct_jwt-tag_list]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  469. ✓test_management_operations_send_the_selected_credential[direct_jwt-team_info_status]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  470. ✓test_management_operations_send_the_selected_credential[direct_jwt-team_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  471. ✓test_management_operations_send_the_selected_credential[direct_jwt-team_list_ids]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  472. ✓test_management_operations_send_the_selected_credential[direct_jwt-update_key_models]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  473. ✓test_management_operations_send_the_selected_credential[direct_jwt-update_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  474. ✓test_management_operations_send_the_selected_credential[direct_jwt-update_mcp_server]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  475. ✓test_management_operations_send_the_selected_credential[direct_jwt-update_org]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  476. ✓test_management_operations_send_the_selected_credential[direct_jwt-update_team]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  477. ✓test_management_operations_send_the_selected_credential[direct_jwt-update_user]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  478. ✓test_management_operations_send_the_selected_credential[direct_jwt-user_count]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  479. ✓test_management_operations_send_the_selected_credential[direct_jwt-user_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  480. ✓test_management_operations_send_the_selected_credential[direct_jwt-user_list_ids]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  481. ✓test_management_operations_send_the_selected_credential[master-add_team_member]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  482. ✓test_management_operations_send_the_selected_credential[master-block_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  483. ✓test_management_operations_send_the_selected_credential[master-connection_test]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  484. ✓test_management_operations_send_the_selected_credential[master-create_customer]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  485. ✓test_management_operations_send_the_selected_credential[master-create_mcp_server]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  486. ✓test_management_operations_send_the_selected_credential[master-create_org]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  487. ✓test_management_operations_send_the_selected_credential[master-create_tag]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  488. ✓test_management_operations_send_the_selected_credential[master-create_team]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  489. ✓test_management_operations_send_the_selected_credential[master-create_user]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  490. ✓test_management_operations_send_the_selected_credential[master-customer_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  491. ✓test_management_operations_send_the_selected_credential[master-delete_customer]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  492. ✓test_management_operations_send_the_selected_credential[master-delete_key_strict]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  493. ✓test_management_operations_send_the_selected_credential[master-delete_mcp_server]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  494. ✓test_management_operations_send_the_selected_credential[master-delete_model_strict]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  495. ✓test_management_operations_send_the_selected_credential[master-delete_org]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  496. ✓test_management_operations_send_the_selected_credential[master-delete_tag]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  497. ✓test_management_operations_send_the_selected_credential[master-delete_team_member]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  498. ✓test_management_operations_send_the_selected_credential[master-delete_team]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  499. ✓test_management_operations_send_the_selected_credential[master-delete_user_strict]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  500. ✓test_management_operations_send_the_selected_credential[master-delete_user]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  501. ✓test_management_operations_send_the_selected_credential[master-generate_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  502. ✓test_management_operations_send_the_selected_credential[master-key_alias_count]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  503. ✓test_management_operations_send_the_selected_credential[master-key_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  504. ✓test_management_operations_send_the_selected_credential[master-key_list]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  505. ✓test_management_operations_send_the_selected_credential[master-llm_only_key]test_proxy_client.py›
    1. Generate a virtual key with allowed routes: llm_api_routes
    View test on GitHub ↗
  506. ✓test_management_operations_send_the_selected_credential[master-org_info_status]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  507. ✓test_management_operations_send_the_selected_credential[master-org_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  508. ✓test_management_operations_send_the_selected_credential[master-proxy.create_credential]test_proxy_client.py›
    1. Save the provider credential owned
    View test on GitHub ↗
  509. ✓test_management_operations_send_the_selected_credential[master-proxy.create_model]test_proxy_client.py›
    1. Add a deployment named owned that calls synthetic
    View test on GitHub ↗
  510. ✓test_management_operations_send_the_selected_credential[master-proxy.create_team]test_proxy_client.py›
    1. Create a team with team alias: owned
    View test on GitHub ↗
  511. ✓test_management_operations_send_the_selected_credential[master-proxy.create_toolset]test_proxy_client.py›
    1. Create an MCP toolset with the tools
    View test on GitHub ↗
  512. ✓test_management_operations_send_the_selected_credential[master-proxy.delete_credential]test_proxy_client.py›
    1. Delete the provider credential
    View test on GitHub ↗
  513. ✓test_management_operations_send_the_selected_credential[master-proxy.delete_customers]test_proxy_client.py›
    1. Delete the end users owned
    View test on GitHub ↗
  514. ✓test_management_operations_send_the_selected_credential[master-proxy.delete_key]test_proxy_client.py›
    1. Delete the virtual key
    View test on GitHub ↗
  515. ✓test_management_operations_send_the_selected_credential[master-proxy.delete_model]test_proxy_client.py›
    1. Delete the deployment
    View test on GitHub ↗
  516. ✓test_management_operations_send_the_selected_credential[master-proxy.delete_team]test_proxy_client.py›
    1. Delete the team
    View test on GitHub ↗
  517. ✓test_management_operations_send_the_selected_credential[master-proxy.delete_toolset]test_proxy_client.py›
    1. Delete the MCP toolset
    View test on GitHub ↗
  518. ✓test_management_operations_send_the_selected_credential[master-proxy.delete_user]test_proxy_client.py›
    1. Delete the internal user
    View test on GitHub ↗
  519. ✓test_management_operations_send_the_selected_credential[master-proxy.generate_key]test_proxy_client.py›
    1. Generate a virtual key with default settings
    View test on GitHub ↗
  520. ✓test_management_operations_send_the_selected_credential[master-proxy.key_info]test_proxy_client.py›
    1. Read the key's settings back from /key/info
    View test on GitHub ↗
  521. ✓test_management_operations_send_the_selected_credential[master-proxy.memory_summary]test_proxy_client.py›
    1. Read memory usage from /debug/memory/summary on every proxy replica
    View test on GitHub ↗
  522. ✓test_management_operations_send_the_selected_credential[master-proxy.model_cost_map]test_proxy_client.py›
    1. Read the model cost map
    View test on GitHub ↗
  523. ✓test_management_operations_send_the_selected_credential[master-proxy.model_info]test_proxy_client.py›
    1. List the deployments from /model/info
    View test on GitHub ↗
  524. ✓test_management_operations_send_the_selected_credential[master-proxy.probe]test_proxy_client.py›
    1. Call the management route /user/info
    View test on GitHub ↗
  525. ✓test_management_operations_send_the_selected_credential[master-proxy.spend_logs]test_proxy_client.py›
    1. Read /spend/logs
    View test on GitHub ↗
  526. ✓test_management_operations_send_the_selected_credential[master-proxy.update_model]test_proxy_client.py›
    1. Update a deployment's settings to model: synthetic
    View test on GitHub ↗
  527. ✓test_management_operations_send_the_selected_credential[master-proxy.update_toolset]test_proxy_client.py›
    1. Update an MCP toolset with toolset id: owned
    View test on GitHub ↗
  528. ✓test_management_operations_send_the_selected_credential[master-regenerate_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  529. ✓test_management_operations_send_the_selected_credential[master-reset_key_spend]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  530. ✓test_management_operations_send_the_selected_credential[master-tag_list]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  531. ✓test_management_operations_send_the_selected_credential[master-team_info_status]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  532. ✓test_management_operations_send_the_selected_credential[master-team_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  533. ✓test_management_operations_send_the_selected_credential[master-team_list_ids]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  534. ✓test_management_operations_send_the_selected_credential[master-update_key_models]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  535. ✓test_management_operations_send_the_selected_credential[master-update_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  536. ✓test_management_operations_send_the_selected_credential[master-update_mcp_server]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  537. ✓test_management_operations_send_the_selected_credential[master-update_org]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  538. ✓test_management_operations_send_the_selected_credential[master-update_team]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  539. ✓test_management_operations_send_the_selected_credential[master-update_user]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  540. ✓test_management_operations_send_the_selected_credential[master-user_count]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  541. ✓test_management_operations_send_the_selected_credential[master-user_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  542. ✓test_management_operations_send_the_selected_credential[master-user_list_ids]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  543. ✓test_management_operations_send_the_selected_credential[virtual_key-add_team_member]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  544. ✓test_management_operations_send_the_selected_credential[virtual_key-block_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  545. ✓test_management_operations_send_the_selected_credential[virtual_key-connection_test]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  546. ✓test_management_operations_send_the_selected_credential[virtual_key-create_customer]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  547. ✓test_management_operations_send_the_selected_credential[virtual_key-create_mcp_server]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  548. ✓test_management_operations_send_the_selected_credential[virtual_key-create_org]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  549. ✓test_management_operations_send_the_selected_credential[virtual_key-create_tag]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  550. ✓test_management_operations_send_the_selected_credential[virtual_key-create_team]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  551. ✓test_management_operations_send_the_selected_credential[virtual_key-create_user]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  552. ✓test_management_operations_send_the_selected_credential[virtual_key-customer_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  553. ✓test_management_operations_send_the_selected_credential[virtual_key-delete_customer]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  554. ✓test_management_operations_send_the_selected_credential[virtual_key-delete_key_strict]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  555. ✓test_management_operations_send_the_selected_credential[virtual_key-delete_mcp_server]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  556. ✓test_management_operations_send_the_selected_credential[virtual_key-delete_model_strict]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  557. ✓test_management_operations_send_the_selected_credential[virtual_key-delete_org]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  558. ✓test_management_operations_send_the_selected_credential[virtual_key-delete_tag]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  559. ✓test_management_operations_send_the_selected_credential[virtual_key-delete_team_member]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  560. ✓test_management_operations_send_the_selected_credential[virtual_key-delete_team]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  561. ✓test_management_operations_send_the_selected_credential[virtual_key-delete_user_strict]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  562. ✓test_management_operations_send_the_selected_credential[virtual_key-delete_user]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  563. ✓test_management_operations_send_the_selected_credential[virtual_key-generate_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  564. ✓test_management_operations_send_the_selected_credential[virtual_key-key_alias_count]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  565. ✓test_management_operations_send_the_selected_credential[virtual_key-key_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  566. ✓test_management_operations_send_the_selected_credential[virtual_key-key_list]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  567. ✓test_management_operations_send_the_selected_credential[virtual_key-llm_only_key]test_proxy_client.py›
    1. Generate a virtual key with allowed routes: llm_api_routes
    View test on GitHub ↗
  568. ✓test_management_operations_send_the_selected_credential[virtual_key-org_info_status]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  569. ✓test_management_operations_send_the_selected_credential[virtual_key-org_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  570. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.create_credential]test_proxy_client.py›
    1. Save the provider credential owned
    View test on GitHub ↗
  571. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.create_model]test_proxy_client.py›
    1. Add a deployment named owned that calls synthetic
    View test on GitHub ↗
  572. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.create_team]test_proxy_client.py›
    1. Create a team with team alias: owned
    View test on GitHub ↗
  573. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.create_toolset]test_proxy_client.py›
    1. Create an MCP toolset with the tools
    View test on GitHub ↗
  574. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.delete_credential]test_proxy_client.py›
    1. Delete the provider credential
    View test on GitHub ↗
  575. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.delete_customers]test_proxy_client.py›
    1. Delete the end users owned
    View test on GitHub ↗
  576. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.delete_key]test_proxy_client.py›
    1. Delete the virtual key
    View test on GitHub ↗
  577. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.delete_model]test_proxy_client.py›
    1. Delete the deployment
    View test on GitHub ↗
  578. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.delete_team]test_proxy_client.py›
    1. Delete the team
    View test on GitHub ↗
  579. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.delete_toolset]test_proxy_client.py›
    1. Delete the MCP toolset
    View test on GitHub ↗
  580. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.delete_user]test_proxy_client.py›
    1. Delete the internal user
    View test on GitHub ↗
  581. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.generate_key]test_proxy_client.py›
    1. Generate a virtual key with default settings
    View test on GitHub ↗
  582. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.key_info]test_proxy_client.py›
    1. Read the key's settings back from /key/info
    View test on GitHub ↗
  583. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.memory_summary]test_proxy_client.py›
    1. Read memory usage from /debug/memory/summary on every proxy replica
    View test on GitHub ↗
  584. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.model_cost_map]test_proxy_client.py›
    1. Read the model cost map
    View test on GitHub ↗
  585. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.model_info]test_proxy_client.py›
    1. List the deployments from /model/info
    View test on GitHub ↗
  586. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.probe]test_proxy_client.py›
    1. Call the management route /user/info
    View test on GitHub ↗
  587. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.spend_logs]test_proxy_client.py›
    1. Read /spend/logs
    View test on GitHub ↗
  588. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.update_model]test_proxy_client.py›
    1. Update a deployment's settings to model: synthetic
    View test on GitHub ↗
  589. ✓test_management_operations_send_the_selected_credential[virtual_key-proxy.update_toolset]test_proxy_client.py›
    1. Update an MCP toolset with toolset id: owned
    View test on GitHub ↗
  590. ✓test_management_operations_send_the_selected_credential[virtual_key-regenerate_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  591. ✓test_management_operations_send_the_selected_credential[virtual_key-reset_key_spend]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  592. ✓test_management_operations_send_the_selected_credential[virtual_key-tag_list]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  593. ✓test_management_operations_send_the_selected_credential[virtual_key-team_info_status]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  594. ✓test_management_operations_send_the_selected_credential[virtual_key-team_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  595. ✓test_management_operations_send_the_selected_credential[virtual_key-team_list_ids]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  596. ✓test_management_operations_send_the_selected_credential[virtual_key-update_key_models]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  597. ✓test_management_operations_send_the_selected_credential[virtual_key-update_key]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  598. ✓test_management_operations_send_the_selected_credential[virtual_key-update_mcp_server]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  599. ✓test_management_operations_send_the_selected_credential[virtual_key-update_org]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  600. ✓test_management_operations_send_the_selected_credential[virtual_key-update_team]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  601. ✓test_management_operations_send_the_selected_credential[virtual_key-update_user]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  602. ✓test_management_operations_send_the_selected_credential[virtual_key-user_count]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  603. ✓test_management_operations_send_the_selected_credential[virtual_key-user_info]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  604. ✓test_management_operations_send_the_selected_credential[virtual_key-user_list_ids]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  605. ✓test_management_read_backs_poll_the_control_replicas_only TestReplicasFortest_proxy_client.py›
    1. Read /key/info on every proxy replica until they all agree
    View test on GitHub ↗
  606. ✓test_mcp_admin_routes_read_back_from_every_data_plane_replica TestReplicasFortest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  607. ✓test_monolith_reads_management_routes_back_from_every_replica TestReplicasFortest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  608. ✓test_names_the_replica_that_never_converges_with_its_last_read TestAwaitConvergedEverywheretest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  609. ✓test_names_the_replica_that_never_converges_with_what_it_last_served TestAwaitEverywheretest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  610. ✓test_partial_updates_preserve_explicit_null_at_the_http_boundary[server]test_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  611. ✓test_partial_updates_preserve_explicit_null_at_the_http_boundary[toolset]test_proxy_client.py›
    1. Update an MCP toolset with toolset id: owned
    View test on GitHub ↗
  612. ✓test_passes_once_every_replica_lists_the_model TestAwaitServableEverywheretest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  613. ✓test_polls_until_the_deadline_before_giving_up TestAwaitEverywheretest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  614. ✓test_split_deployment_reads_management_routes_back_from_the_control_plane TestReplicasFortest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  615. ✓test_splits_and_trims_the_gateway_addresses TestParseReplicaUrlstest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  616. ✓test_strict_key_cleanup_accepts_missing_only_when_requested TestBoundManagementCallertest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  617. ✓test_successful_team_and_model_polling_uses_the_bound_caller TestSplitCallerPropagationtest_proxy_client.py›
    1. Add a deployment named owned that calls synthetic
    View test on GitHub ↗
  618. ✓test_the_stacks_own_endpoints_take_its_exported_control_list TestStackEndpointsControlReplicastest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  619. ✓test_unset_with_a_split_control_plane_polls_its_base_alone TestParseControlPlaneReplicaUrlstest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  620. ✓test_unset_with_one_shared_base_follows_the_data_plane_replicas TestParseControlPlaneReplicaUrlstest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  621. ✓test_waits_for_the_lagging_replica_and_returns_every_settled_answer TestAwaitEverywheretest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  622. ✓test_waits_for_the_replica_that_lags_behind_the_write TestAwaitConvergedEverywheretest_proxy_client.py›

    No steps recorded

    View test on GitHub ↗
  623. ✓test_a_drained_group_does_not_block_a_different_group TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-mag-free-94bd35987d42 and user id: e2e-test-user
    View test on GitHub ↗
  624. ✓test_a_key_that_spent_nothing_is_blocked_by_the_shared_pool TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-mag-budget-94bd35987d42 and user id: e2e-test-user
    View test on GitHub ↗
  625. ✓test_alias_shares_rpm_bucket_with_model_group TestModelGroupAliasRateLimitratelimit/test_model_group_alias_rate_limit_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word 53fe2279db05"
    3. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word 7dbf0bef492b"
    4. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word dbe1dacd5e97"
    5. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word 010114bfea32"
    View test on GitHub ↗
  626. ✓test_bare_key_blocks_over_its_own_budget TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06
    View test on GitHub ↗
  627. ✓test_bare_key_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    View test on GitHub ↗
  628. ✓test_blocked_member_does_not_block_peer TestTeamMemberBudgetIsolationbudgets/test_team_member_budget_isolation_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: 06ebef6f-51d5-4064-bfdf-542e2d64740e and team id: f77608e8-be0b-4e1e-92f3-423da31b7936
    2. Generate a virtual key with user id: 2b005a66-9371-469f-8b66-b5b494a9d8ce and team id: f77608e8-be0b-4e1e-92f3-423da31b7936
    3. Send a /chat/completions request to gpt-5.5
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  629. ✓test_budget_crud_roundtripbudgets/test_budget_crud_e2e.py›
    • spend-budgets
    • budget_management
    1. Generate a virtual key with budget id: 6e05a571-a0ad-4313-a7fc-d04491b40132
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  630. ✓test_budget_delete_removes_itbudgets/test_budget_crud_e2e.py›
    • spend-budgets
    • budget_management

    No steps recorded

    View test on GitHub ↗
  631. ✓test_budget_duration_schedules_reset_on_keybudgets/test_budget_crud_e2e.py›
    • spend-budgets
    • key_management
    1. Generate a virtual key with max budget: 10.0 and budget duration: 30d
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  632. ✓test_budget_fallback_reroutes_anthropic_messages_to_openaibudgets/test_budget_fallback_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • anthropic
    • openai
    • claude-haiku-4-5
    • gpt-5.5
    1. Generate a virtual key with model max budget: claude-haiku-4-5: budget limit: 1e-06 and time period: 30d and budget fallbacks: claude-haiku-4-5: gpt-5.5
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  633. ✓test_burst_of_concurrent_calls_loses_no_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.6-luna
    1. Add a deployment named e2e-reconciliation-7830955182f7 that calls openai/gpt-5.6-luna
    2. Create a team with team alias: e2e-spend-3bea2161ebe7
    3. Generate a virtual key with models: e2e-reconciliation-7830955182f7 and team id: 059b5213-bc8c-4fcf-a602-30d2f2c08d62
    4. Send a /chat/completions request to e2e-reconciliation-7830955182f7
    5. Create a team with team alias: e2e-spend-1ff544bf1078
    6. Generate a virtual key with models: e2e-reconciliation-7830955182f7 and team id: 98ec278f-0b43-47ee-a3c6-8c5f332ea84b
    7. Send a /chat/completions request to e2e-reconciliation-7830955182f7
    8. Wait for at least 7 of the key's spend logs in /spend/logs
    9. Read the key's settings back from /key/info
    10. Wait for at least 7 of the key's spend logs in /spend/logs
    11. Read the key's settings back from /key/info
    View test on GitHub ↗
  634. ✓test_burst_rate_limit_does_not_freeze_fresh_key TestRedisCircuitBreakerPathratelimit/test_redis_circuit_breaker_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-cb-model-e9e865e4c851 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-cb-model-e9e865e4c851, key alias: e2e-cb-hot-457e8130c6a5 and rpm limit: 1
    3. Generate a virtual key with models: e2e-cb-model-e9e865e4c851 and key alias: e2e-cb-cool-43485c27cc1f
    4. Send a /chat/completions request to e2e-cb-model-e9e865e4c851 with the prompt "burst 535270a967aa"
    5. Send a /chat/completions request to e2e-cb-model-e9e865e4c851 with the prompt "burst 16501cdfebab"
    6. Send a /chat/completions request to e2e-cb-model-e9e865e4c851 with the prompt "burst 292f3ca61079"
    7. Send a /chat/completions request to e2e-cb-model-e9e865e4c851 with the prompt "burst 1e077b8563eb"
    8. Send a /chat/completions request to e2e-cb-model-e9e865e4c851 with the prompt "burst d56519e8300a"
    9. Send a /chat/completions request to e2e-cb-model-e9e865e4c851 with the prompt "burst 2cc341b1073d"
    10. Send a /chat/completions request to e2e-cb-model-e9e865e4c851 with the prompt "burst 3fad9e78a3c8"
    11. Send a /chat/completions request to e2e-cb-model-e9e865e4c851 with the prompt "burst c96afaf8ec0c"
    12. Send a /chat/completions request to e2e-cb-model-e9e865e4c851 with the prompt "burst 2d68e105d655"
    13. Send a /chat/completions request to e2e-cb-model-e9e865e4c851 with the prompt "burst d72f298d0fec"
    14. Send a /chat/completions request to e2e-cb-model-e9e865e4c851 with the prompt "burst 36a14a8ed8fb"
    15. Send a /chat/completions request to e2e-cb-model-e9e865e4c851 with the prompt "burst 9c50a71add70"
    16. Send a /chat/completions request to e2e-cb-model-e9e865e4c851 with the prompt "fresh 542636778ae1"
    View test on GitHub ↗
  635. ✓test_cache_hit_is_zero_cost_and_suffixedspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  636. ✓test_cache_hit_reduces_tpm_by_non_cached_only TestTpmExcludesCachedTokensratelimit/test_tpm_excludes_cached_tokens_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    • prompt_caching
    1. Add a deployment named e2e-tpm-cache-b98b45128462 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-tpm-cache-b98b45128462 and tpm limit: 100000
    View test on GitHub ↗
  637. ✓test_cache_write_tokens_billed_at_cache_creation_rate TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.6-luna
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named cache-write-priced-12e4564fef43 that calls openai/gpt-5.6-luna
    3. Send a /chat/completions request to cache-write-priced-12e4564fef43
    View test on GitHub ↗
  638. ✓test_capture_rate_reports_or_names_the_missing_billing_keyspend_tracking/test_spend_routes.py›

    No steps recorded

    View test on GitHub ↗
  639. ✓test_chat_completion_writes_nonzero_spend_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. List the deployments from /model/info
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to gemini-2.5-flash
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  640. ✓test_cold_counter_reseed_keeps_counter_equal_to_db_spendbudgets/test_spend_counter_reseed_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and max budget: 1.0
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  641. ✓test_component_cost_headers_sum_to_total TestCostHeadersspend_tracking/test_cost_headers_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.5
    1. List the deployments from /model/info
    2. Generate a virtual key with user id: e2e-test-user
    3. Add a deployment named header-priced-1ae79c4ca3a0 that calls openai/gpt-5.5
    View test on GitHub ↗
  642. ✓test_cost_breakdown_reports_component_costs TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.5
    • prompt_caching
    • reasoning
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named breakdown-priced-9fce4f65643d that calls openai/gpt-5.5
    3. Send a /chat/completions request to breakdown-priced-9fce4f65643d
    View test on GitHub ↗
  643. ✓test_each_model_on_a_shared_key_gets_its_own_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • gemini
    • claude-haiku-4-5
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Send a /chat/completions request to claude-haiku-4-5
    4. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  644. ✓test_edge_wired_chat_writes_nonzero_spend_rowspend_tracking/test_provider_edge_spend_e2e.py›
    • spend-budgets
    • chat_completions
    • nonstream
    • openai
    • openai/gpt-5.5
    1. List the deployments from /model/info
    2. Generate a virtual key with user id: e2e-test-user
    3. Add a deployment named e2e-edge-openai-9ee42a4d56d2 that calls openai/gpt-5.5
    4. Send a /chat/completions request to e2e-edge-openai-9ee42a4d56d2
    5. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  645. ✓test_embedding_writes_nonzero_spend_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • embeddings
    • nonstream
    • openai
    • openai-text-embedding-3-small
    1. Generate a virtual key with user id: e2e-test-user
    2. Send an /embeddings request to openai-text-embedding-3-small for "vectorize this sentence b4b39736899f"
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  646. ✓test_end_user_budget_blocks_attributed_calls TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5
    View test on GitHub ↗
  647. ✓test_end_user_header_attributes_responses_row[x-litellm-customer-id]spend_tracking/test_spend_tracking_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  648. ✓test_end_user_header_attributes_responses_row[x-litellm-end-user-id]spend_tracking/test_spend_tracking_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  649. ✓test_end_user_spend_attributed_on_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  650. ✓test_every_streamed_chunk_carries_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named tier-echo-stream-b0be3aa12718 that calls openai/gpt-5.5
    3. Send a streaming /chat/completions request to tier-echo-stream-b0be3aa12718
    View test on GitHub ↗
  651. ✓test_every_write_path_row_joins_the_key TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • spend_reporting
    • anthropic
    • gemini
    • openai
    • claude-haiku-4-5
    • gemini-2.5-flash
    • openai-gpt-4o-mini
    • openai-responses-codex
    • openai-text-embedding-3-small
    1. List the deployments from /model/info
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  652. ✓test_failure_call_writes_failure_status_rowspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.5
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-spend-failure-0bd2ff8cec00 that calls openai/gpt-5.5
    3. Send a /chat/completions request to e2e-spend-failure-0bd2ff8cec00
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  653. ✓test_failure_rows_share_normalized_error_across_provider_wordingspend_tracking/test_spend_tracking_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-norm-openai-3888b016508f that calls openai/gpt-5.5
    3. Send a /chat/completions request to e2e-norm-openai-3888b016508f
    4. Add a deployment named e2e-norm-anthropic-3888b016508f that calls anthropic/claude-haiku-4-5
    5. Send a /chat/completions request to e2e-norm-anthropic-3888b016508f
    6. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  654. ✓test_generous_mode_lets_priority_borrow_past_reservation TestDynamicRateLimitPriorityratelimit/test_dynamic_rate_limit_priority_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-dynpri-generous-28ba912a3727 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-dynpri-generous-28ba912a3727, key alias: e2e-dynpri-generous-dev-1361a1651483 and metadata: priority: dev
    3. Generate a virtual key with models: e2e-dynpri-generous-28ba912a3727, key alias: e2e-dynpri-generous-prod-3b77b161db6c and metadata: priority: prod
    4. Read the key's settings back from /key/info
    5. Send a /chat/completions request to e2e-dynpri-generous-28ba912a3727 with the prompt "reply with one word 5861e339c915"
    6. Send a /chat/completions request to e2e-dynpri-generous-28ba912a3727 with the prompt "reply with one word f2995128a9be"
    7. Send a /chat/completions request to e2e-dynpri-generous-28ba912a3727 with the prompt "reply with one word c56b65d4baad"
    8. Send a /chat/completions request to e2e-dynpri-generous-28ba912a3727 with the prompt "reply with one word 928450690ed4"
    9. Send a /chat/completions request to e2e-dynpri-generous-28ba912a3727 with the prompt "reply with one word f72e668a7316"
    View test on GitHub ↗
  655. ✓test_headers_report_limit_and_remaining TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5, tpm limit: 100000 and rpm limit: 5
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 5fc9d91744f7"
    View test on GitHub ↗
  656. ✓test_health_check_rows_keep_the_service_account_key TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • health
    • gemini
    • gemini-2.5-flash
    1. Read /spend/logs between 2026-10-14 00:12:22.337366+00:00 and 2026-10-14 00:14:23.073291+00:00
    2. Read /spend/logs between 2026-10-14 00:12:22.337366+00:00 and 2026-10-14 00:14:28.237429+00:00
    View test on GitHub ↗
  657. ✓test_intercepted_search_is_billed_under_the_request_session TestWebSearchInterceptionSessionspend_tracking/test_websearch_interception_session_e2e.py›
    1. Add a deployment named e2e-websearch-session-6f1abaf90163 that calls bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with models: e2e-websearch-session-6f1abaf90163 and user id: e2e-test-user
    3. Send a /v1/messages request to e2e-websearch-session-6f1abaf90163
    4. Wait for at least 2 of the session's spend logs in /spend/logs
    View test on GitHub ↗
  658. ✓test_key_budget_reset_at_advances_after_windowbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  659. ✓test_key_spend_blocks_at_capbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    View test on GitHub ↗
  660. ✓test_key_spend_equals_sum_of_logsspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Wait for at least 2 of the key's spend logs in /spend/logs
    4. Read the key's settings back from /key/info
    View test on GitHub ↗
  661. ✓test_key_with_budget_duration_schedules_reset_at_creationbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • key_management
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  662. ✓test_long_window_blocks_after_short_window_resetsbudgets/test_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with models: gpt-5.5 and budget limits: budget duration: 30s and max budget: 1e-09, budget duration: 1d and max budget: 5e-07
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  663. ✓test_member_budget_blocks_without_touching_teammates TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: fbd350ac-e59f-4bb8-aec7-cf720780321a and team id: a99ef662-fdce-4be1-b387-17246eb5ac49
    2. Generate a virtual key with user id: 8c701685-7816-4752-a76e-f51171074ac4 and team id: a99ef662-fdce-4be1-b387-17246eb5ac49
    View test on GitHub ↗
  664. ✓test_member_spend_attributed_to_team_and_user TestTeamMemberBudgetbudgets/test_team_member_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: c4e59759-9a0e-4a53-a5d1-b0e46f2b8631 and team id: ce4e4c71-7073-4b39-adfc-b333e2119544
    2. Send a /chat/completions request to claude-haiku-4-5
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  665. ✓test_member_spend_over_budget_is_blocked TestTeamMemberBudgetbudgets/test_team_member_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5

    No steps recorded

    View test on GitHub ↗
  666. ✓test_messages_bridge_keeps_cache_tokens TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • messages
    • nonstream
    • openai
    • openai/gpt-5.3-codex
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named bridge-cache-priced-ada96f86a8d9 that calls openai/gpt-5.3-codex
    3. Send a /v1/messages request to bridge-cache-priced-ada96f86a8d9
    View test on GitHub ↗
  667. ✓test_messages_stream_records_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named tier-messages-stream-c8713cffe6b2 that calls openai/gpt-5.5
    3. Send a streaming /v1/messages request to tier-messages-stream-c8713cffe6b2
    View test on GitHub ↗
  668. ✓test_missing_end_date_is_rejected TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  669. ✓test_missing_start_date_is_rejected TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  670. ✓test_model_group_shares_rpm_bucket_with_alias TestModelGroupAliasRateLimitratelimit/test_model_group_alias_rate_limit_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word 14de94398e7b"
    3. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word 58f03d9a139f"
    4. Send a /chat/completions request to e2e-alias-rl-alias with the prompt "reply with one word 29c5c391df21"
    5. Send a /chat/completions request to e2e-alias-rl-target with the prompt "reply with one word 007a1bd42b89"
    View test on GitHub ↗
  671. ✓test_model_max_budget_isolates_per_modelbudgets/test_model_max_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • gemini
    • claude-haiku-4-5
    • gemini-2.5-flash
    1. Generate a virtual key with model max budget: claude-haiku-4-5: budget limit: 1e-06 and time period: 30d and gemini-2.5-flash: budget limit: 1000.0 and time period: 30d
    View test on GitHub ↗
  672. ✓test_multi_window_key_resets_each_window_independentlybudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with budget limits: budget duration: 30s and max budget: 3e-06, budget duration: 1m and max budget: 1.0
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  673. ✓test_one_request_lands_the_same_spend_on_every_surface TestSpendSurfaceConsistencyspend_tracking/test_spend_surface_consistency_e2e.py›
    1. List the deployments from /model/info
    2. Add a deployment named e2e-spend-surfaces-d3739a625c2d that calls openai/gpt-5.6-luna
    3. Create a team with team alias: e2e-spend-surfaces-d3739a625c2d
    4. Send a /chat/completions request to e2e-spend-surfaces-d3739a625c2d
    5. Wait for the request's spend log in /spend/logs
    6. Read the key's settings back from /key/info
    View test on GitHub ↗
  674. ✓test_org_budget_blocks_keys_under_it TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with team id: 563739dd-61db-4b21-aac6-292fdb7da9f2
    View test on GitHub ↗
  675. ✓test_org_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with team id: a6db4d8a-24af-4f38-aef0-f045bdb63d23
    View test on GitHub ↗
  676. ✓test_personal_key_blocks_over_its_own_budget TestKeyBudgetBlocksAcrossKeyKindsbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and user id: 4702b543-987f-4917-9cf4-a28ad578f50d
    2. Generate a virtual key with user id: 4702b543-987f-4917-9cf4-a28ad578f50d
    View test on GitHub ↗
  677. ✓test_personal_key_resets_after_window TestKeyBudgetResetAcrossKeyKindsbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06, budget duration: 30s and user id: eb173124-a429-4109-a1f1-e930b1eb83f5
    View test on GitHub ↗
  678. ✓test_personal_key_user_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: 56f7d3b6-4e2f-430e-b62f-1be877585c08
    View test on GitHub ↗
  679. ✓test_pre_call_rejection_row_attributes_provider_and_model_idspend_tracking/test_spend_tracking_e2e.py›
    1. Add a deployment named e2e-spend-precall-da8b7d18606b that calls openai/gpt-5.5
    2. Generate a virtual key with models: e2e-spend-precall-da8b7d18606b and rpm limit: 1
    3. Send a /chat/completions request to e2e-spend-precall-da8b7d18606b
    4. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  680. ✓test_priority_tier_bills_priority_rates TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.6-luna
    • reasoning
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named tier-priced-4dac9124948d that calls openai/gpt-5.6-luna
    3. Send a /chat/completions request to tier-priced-4dac9124948d
    View test on GitHub ↗
  681. ✓test_request_tags_round_tripspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  682. ✓test_reset_wait_never_yields_non_budget_errorbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and budget duration: 30s
    View test on GitHub ↗
  683. ✓test_responses_stream_records_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named tier-responses-stream-bedf3b50970e that calls openai/gpt-5.5
    3. Send a streaming /v1/responses request to tier-responses-stream-bedf3b50970e
    View test on GitHub ↗
  684. ✓test_rpm_limit_blocks_over_limit TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and rpm limit: 3
    2. Read the key's settings back from /key/info
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word f97a8b38aaa3"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 83a68ec61889"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 0731805a5f2c"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 4847f25d45e4"
    View test on GitHub ↗
  685. ✓test_rpm_limit_one_blocks_second_call TestRedisBackedRateLimitratelimit/test_redis_backed_ratelimit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001
    1. Add a deployment named e2e-redis-rpm-ca43a87e9daa that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with models: e2e-redis-rpm-ca43a87e9daa, key alias: e2e-redis-rpm-bef0b0183dbb and rpm limit: 1
    3. Read the key's settings back from /key/info
    4. Send a /chat/completions request to e2e-redis-rpm-ca43a87e9daa with the prompt "ping d761a21f140c"
    5. Send a /chat/completions request to e2e-redis-rpm-ca43a87e9daa with the prompt "pong e4a95e74b3a0"
    View test on GitHub ↗
  686. ✓test_rpm_limit_resets_after_window TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and rpm limit: 1
    2. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 662786598521"
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word f2f6cecfc682"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 174cbcc50a7c"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 3edac291f51d"
    6. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 9afbf714037f"
    7. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 8fd34ba35f2e"
    8. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 66dc34db94ec"
    9. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 037157b1083d"
    10. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word d87f9cdd8b3b"
    11. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 0e3d80836b69"
    12. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word d35ef8e88f07"
    13. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 4a62e4ac0ee1"
    14. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 15fc9fb47b41"
    15. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 9a7610cfcb70"
    16. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 143af849940a"
    17. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 013bf273c123"
    View test on GitHub ↗
  687. ✓test_schema_listed_spend_routes_are_responsivespend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  688. ✓test_short_window_blocks_then_resetsbudgets/test_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with models: gpt-5.5 and budget limits: budget duration: 30s and max budget: 1e-09, budget duration: 1m and max budget: 1.0
    View test on GitHub ↗
  689. ✓test_soft_budget_does_not_blockbudgets/test_soft_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 1000.0 and soft budget: 1e-09
    View test on GitHub ↗
  690. ✓test_spend_calculate_returns_nonzero_costspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • spend_reporting
    • gemini
    • gemini-2.5-flash

    No steps recorded

    View test on GitHub ↗
  691. ✓test_spend_logs_by_key_return_every_row_with_the_alias TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • spend_reporting
    • anthropic
    • gemini
    • openai
    • claude-haiku-4-5
    • gemini-2.5-flash
    • openai-gpt-4o-mini
    • openai-responses-codex
    • openai-text-embedding-3-small
    1. Wait for at least 9 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  692. ✓test_spend_logs_endpoint_returns_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • spend_reporting
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  693. ✓test_spend_logs_v2_pagination_caps_pages_and_keeps_totalspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • spend_reporting
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  694. ✓test_spend_route_responsive[/customer/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  695. ✓test_spend_route_responsive[/customer/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • customer_management

    No steps recorded

    View test on GitHub ↗
  696. ✓test_spend_route_responsive[/end_user/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  697. ✓test_spend_route_responsive[/global/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  698. ✓test_spend_route_responsive[/global/activity/exceptions]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  699. ✓test_spend_route_responsive[/global/activity/exceptions/deployment]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  700. ✓test_spend_route_responsive[/global/activity/model]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  701. ✓test_spend_route_responsive[/global/all_end_users]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  702. ✓test_spend_route_responsive[/global/spend]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  703. ✓test_spend_route_responsive[/global/spend/all_tag_names]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  704. ✓test_spend_route_responsive[/global/spend/keys]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  705. ✓test_spend_route_responsive[/global/spend/logs]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  706. ✓test_spend_route_responsive[/global/spend/models]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  707. ✓test_spend_route_responsive[/global/spend/provider]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  708. ✓test_spend_route_responsive[/global/spend/report]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  709. ✓test_spend_route_responsive[/global/spend/tags]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  710. ✓test_spend_route_responsive[/global/spend/teams]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  711. ✓test_spend_route_responsive[/key/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • key_management

    No steps recorded

    View test on GitHub ↗
  712. ✓test_spend_route_responsive[/organization/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  713. ✓test_spend_route_responsive[/organization/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • organization_management

    No steps recorded

    View test on GitHub ↗
  714. ✓test_spend_route_responsive[/spend/keys]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting
    1. List the deployments from /model/info
    2. Add a deployment named openai-text-embedding-3-small that calls openai/text-embedding-3-small
    3. Add a deployment named openai-responses-codex that calls openai/gpt-5.3-codex
    4. Add a deployment named openai-gpt-4o-mini that calls openai/gpt-4o-mini
    View test on GitHub ↗
  715. ✓test_spend_route_responsive[/spend/logs]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  716. ✓test_spend_route_responsive[/spend/logs/session/ui]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  717. ✓test_spend_route_responsive[/spend/logs/ui]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  718. ✓test_spend_route_responsive[/spend/logs/v2]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  719. ✓test_spend_route_responsive[/spend/tags]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  720. ✓test_spend_route_responsive[/spend/users]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  721. ✓test_spend_route_responsive[/tag/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  722. ✓test_spend_route_responsive[/team/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  723. ✓test_spend_route_responsive[/team/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • team_management

    No steps recorded

    View test on GitHub ↗
  724. ✓test_spend_route_responsive[/user/daily/activity]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  725. ✓test_spend_route_responsive[/user/daily/activity/aggregated]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • spend_reporting

    No steps recorded

    View test on GitHub ↗
  726. ✓test_spend_route_responsive[/user/list]spend_tracking/test_spend_routes.py›
    • spend-budgets
    • user_management

    No steps recorded

    View test on GitHub ↗
  727. ✓test_streamed_call_records_and_bills_the_served_tier TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named tier-priced-stream-d45a1ce02427 that calls openai/gpt-5.6-luna
    3. Send a streaming /chat/completions request to tier-priced-stream-d45a1ce02427
    View test on GitHub ↗
  728. ✓test_streaming_cache_read_billed_at_cache_read_rate TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py›
    • spend-budgets
    • stream
    • openai
    • openai/gpt-5.5
    • prompt_caching
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named stream-cache-priced-8522ddecc1b4 that calls openai/gpt-5.5
    3. Send a /chat/completions request to stream-cache-priced-8522ddecc1b4
    4. Send a streaming /chat/completions request to stream-cache-priced-8522ddecc1b4
    View test on GitHub ↗
  729. ✓test_streaming_chat_completion_tracks_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • chat_completions
    • stream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to gemini-2.5-flash
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  730. ✓test_streaming_messages_via_responses_bridge_tracks_spendspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • messages
    • stream
    • openai
    • openai-responses-codex
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to openai-responses-codex
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  731. ✓test_tag_budget_blocks_tagged_requestsbudgets/test_tag_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  732. ✓test_tag_spend_matches_sum_of_tagged_logsspend_tracking/test_spend_tracking_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  733. ✓test_team_budget_blocks_every_team_key TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with team id: ea5c22ad-3e98-4d6c-9d83-8aed31af4c10
    View test on GitHub ↗
  734. ✓test_team_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with team id: 72ad202e-c5f0-4206-a0bf-20d8510713e7
    View test on GitHub ↗
  735. ✓test_team_key_blocks_over_its_own_budget TestKeyBudgetBlocksAcrossKeyKindsbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06 and team id: f6fc37c5-cb94-4263-ae8b-5b4e78844582
    2. Generate a virtual key with team id: f6fc37c5-cb94-4263-ae8b-5b4e78844582
    View test on GitHub ↗
  736. ✓test_team_key_resets_after_window TestKeyBudgetResetAcrossKeyKindsbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06, budget duration: 30s and team id: 2394f66d-75e9-4fda-bcb8-0ff86d595925
    View test on GitHub ↗
  737. ✓test_team_long_window_blocks_after_short_window_resetsbudgets/test_team_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and team id: 663c8340-7eca-4200-b486-591ec8d99457
    View test on GitHub ↗
  738. ✓test_team_member_budget_reset_at_advancesbudgets/test_budget_reset_advances_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: 6e81eda1-bd6d-44a1-932a-23ac24fd303c and team id: b7f069cd-3b32-44e2-81ae-d0b697be8ea5
    View test on GitHub ↗
  739. ✓test_team_member_budget_reset_keeps_advancingbudgets/test_team_member_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: f84b7028-547b-44ef-9d00-946624891987 and team id: ac97de65-367b-4ec9-b39a-d58f65de7320
    View test on GitHub ↗
  740. ✓test_team_member_key_blocks_over_its_own_budget TestKeyBudgetBlocksAcrossKeyKindsbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06, user id: c26df51a-ea40-4c8b-bfe7-af3229be8395 and team id: 95662196-8b5c-40dc-a15d-d3104977b1b3
    2. Generate a virtual key with user id: c26df51a-ea40-4c8b-bfe7-af3229be8395 and team id: 95662196-8b5c-40dc-a15d-d3104977b1b3
    View test on GitHub ↗
  741. ✓test_team_member_key_resets_after_window TestKeyBudgetResetAcrossKeyKindsbudgets/test_budget_reset_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with max budget: 3e-06, budget duration: 30s, user id: 77eccad7-6417-47db-ad07-21e7823365a0 and team id: 5976ce12-1706-4c32-8ed4-e2a47e1adb46
    View test on GitHub ↗
  742. ✓test_team_short_window_blocks_then_resetsbudgets/test_team_multi_window_budget_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and team id: e4e85ea3-3fa2-4ed5-8f78-ab969d616b2d
    View test on GitHub ↗
  743. ✓test_terminal_batch_cost_row_joins_the_retrieving_key TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • batches
    • batch
    • openai
    • openai-gpt-4o-mini
    1. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  744. ✓test_the_budget_read_reports_the_spend_drawn_against_the_pool TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • budget_management
    • openai
    • openai/gpt-5.4-nano

    No steps recorded

    View test on GitHub ↗
  745. ✓test_the_key_that_drained_the_pool_stays_blocked TestModelAccessGroupBudgetbudgets/test_model_access_group_budget_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • openai/gpt-5.4-nano
    1. Generate a virtual key with models: e2e-mag-budget-94bd35987d42
    2. Add a deployment named e2e-mag-budgeted-94bd35987d42 that calls openai/gpt-5.4-nano
    3. Add a deployment named e2e-mag-unbudgeted-94bd35987d42 that calls openai/gpt-5.4-nano
    View test on GitHub ↗
  746. ✓test_tpm_limit_blocks_over_limit TestKeyRateLimitsratelimit/test_rate_limit_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with models: claude-haiku-4-5 and tpm limit: 60
    2. Read the key's settings back from /key/info
    3. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 0ea14218be7c"
    4. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 7df8f190881b"
    5. Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word 432a74d50bf4"
    View test on GitHub ↗
  747. ✓test_user_budget_blocks_a_second_key TestUserBudgetAcrossKeysbudgets/test_user_budget_across_keys_e2e.py›
    • spend-budgets
    • nonstream
    • openai
    • gpt-5.5
    1. Generate a virtual key with user id: 0cc1bdf5-5e59-49d8-8208-ea288f8f2e1d
    View test on GitHub ↗
  748. ✓test_user_budget_enforced_across_their_personal_keys TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • claude-haiku-4-5
    1. Generate a virtual key with user id: e7932fe1-c2a5-44a5-a11f-eed50c8a9018
    2. Generate a virtual key with user id: e7932fe1-c2a5-44a5-a11f-eed50c8a9018 and team id: 815d862e-5596-4062-936e-fd865b71cff9
    View test on GitHub ↗
  749. ✓test_user_daily_activity_reports_alias_and_email TestKeyAttributionspend_tracking/test_key_attribution_e2e.py›
    • spend-budgets
    • spend_reporting
    • anthropic
    • gemini
    • openai
    • claude-haiku-4-5
    • gemini-2.5-flash
    • openai-gpt-4o-mini
    • openai-responses-codex
    • openai-text-embedding-3-small

    No steps recorded

    View test on GitHub ↗
  750. ✓test_valid_date_range_returns_results_and_metadata TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py›
    • spend-budgets
    • spend_reporting
    • openai
    • openai/gpt-5.6-luna
    1. List the deployments from /model/info
    2. Add a deployment named e2e-reconciliation-1f4524c171c8 that calls openai/gpt-5.6-luna
    3. Create a team with team alias: e2e-spend-3e6b5fb92771
    4. Generate a virtual key with models: e2e-reconciliation-1f4524c171c8 and team id: 071b55b7-9d89-49c8-b673-9b6a07c341cf
    5. Send a /chat/completions request to e2e-reconciliation-1f4524c171c8
    6. Create a team with team alias: e2e-spend-8a93869ef927
    7. Generate a virtual key with models: e2e-reconciliation-1f4524c171c8 and team id: 6fc12ab8-c3ee-4cea-beef-c8e1cb810fae
    8. Send a /chat/completions request to e2e-reconciliation-1f4524c171c8
    9. Wait for at least 7 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  751. −test_end_user_model_max_budget_enforces_per_model_rpmbudgets/test_model_max_budget_e2e.py›
    • spend-budgets
    • nonstream
    • gemini
    • gemini-2.5-flash

    No steps recorded

    stage red: product gap, end-user model_max_budget rpm_limit is stored but never enforced

    View test on GitHub ↗
  752. −test_strict_mode_blocks_saturated_priority_but_serves_the_other TestDynamicRateLimitPriorityratelimit/test_dynamic_rate_limit_priority_e2e.py›
    • spend-budgets
    • nonstream
    • anthropic
    • anthropic/claude-haiku-4-5-20251001

    No steps recorded

    LIT-5118: the stage proxy does not run the dynamic_rate_limiter_v3 callbacks + priority_reservation config this module's docstring requires (zero limiter log lines on any pod during the 2026-08-02 run), so strict enforcement can never engage there

    View test on GitHub ↗
  753. ✓test_access_group_info_reports_membership TestModelAccessGroupRoutestest_model_tag_accessgroup_e2e.py›
    1. Add a deployment named e2e-mgmt-agmodel-2467962049ed that calls openai/gpt-5.5
    View test on GitHub ↗
  754. ✓test_actor_subject_and_database_role[internal_user_viewer] TestJwtManagementtest_jwt_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  755. ✓test_actor_subject_and_database_role[internal_user] TestJwtManagementtest_jwt_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  756. ✓test_actor_subject_and_database_role[organization_admin] TestJwtManagementtest_jwt_management_e2e.py›
    1. Create a team with team alias: e2e-team-0ecdbc667feb and organization id: bcd511b4-1c71-4a3a-804c-43a73eb7ad7b
    View test on GitHub ↗
  757. ✓test_actor_subject_and_database_role[proxy_admin_viewer] TestJwtManagementtest_jwt_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  758. ✓test_actor_subject_and_database_role[proxy_admin] TestJwtManagementtest_jwt_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  759. ✓test_actor_subject_and_database_role[team_admin] TestJwtManagementtest_jwt_management_e2e.py›
    1. Create a team with team alias: e2e-team-065f95b1de5d and organization id: b9fbd2e7-fe49-4c74-9ceb-c98a1d137b51
    View test on GitHub ↗
  760. ✓test_actor_subject_and_database_role[team_member] TestJwtManagementtest_jwt_management_e2e.py›
    1. Create a team with team alias: e2e-team-66cfa70eba6a and organization id: a3e355b8-fac5-4f73-8ccb-f4a3012eea2a
    View test on GitHub ↗
  761. ✓test_actor_subject_and_database_role[unrelated_user] TestJwtManagementtest_jwt_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  762. ✓test_admin_creates_reads_updates_clears_and_deletes_a_key[direct_jwt] TestJwtManagementtest_jwt_management_e2e.py›
    1. Create a team with team alias: e2e-team-c43ec3d478b5 and organization id: 4dd9f548-501f-4edb-b3c2-bedad6134ace
    View test on GitHub ↗
  763. ✓test_admin_creates_reads_updates_clears_and_deletes_a_key[virtual_key] TestJwtManagementtest_jwt_management_e2e.py›
    1. Create a team with team alias: e2e-team-c69bdef4723d and organization id: fd91c172-d5d8-4fb6-affb-2f57271c76ed
    View test on GitHub ↗
  764. ✓test_admin_viewer_reads_but_cannot_update TestJwtManagementtest_jwt_management_e2e.py›
    1. Read the key's settings back from /key/info
    View test on GitHub ↗
  765. ✓test_approve_activates_submission_and_persists TestMcpServerSubmissiontest_config_misc_endpoints_e2e.py›
    1. Generate a virtual key with team id: 76700f9b-3408-4cc8-9c4a-a08fa722cebe
    View test on GitHub ↗
  766. ✓test_bedrock_mantle_responses_connection_succeeds TestModelTestConnectiontest_model_test_connection_e2e.py›

    No steps recorded

    View test on GitHub ↗
  767. ✓test_block_persists_to_key_info TestKeyRoutestest_management_e2e.py›
    1. Generate a virtual key with models: gemini-2.5-flash
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  768. ✓test_block_then_unblock_persists_to_model_info TestModelRoutestest_model_tag_accessgroup_e2e.py›
    1. Add a deployment named e2e-mgmt-model-block-748b691d8918 that calls openai/gpt-5.5
    View test on GitHub ↗
  769. ✓test_block_then_unblock_persists_to_team_info TestTeamManagementRoutestest_team_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  770. ✓test_bulk_update_applies_max_budget_to_target_key TestKeyManagementRoutestest_key_management_e2e.py›
    1. Generate a virtual key with models: gpt-5.5 and max budget: 5.0
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  771. ✓test_callbacks_list_reports_active_logging_callbacks TestInventoryRoutestest_config_misc_endpoints_e2e.py›

    No steps recorded

    View test on GitHub ↗
  772. ✓test_clearing_the_description_with_null_reads_back_null TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    1. Read /v1/mcp/server/4252aab1-63bb-4684-9c37-074488bd164f on every proxy replica until it settles
    View test on GitHub ↗
  773. ✓test_clearing_the_description_with_null_reads_back_null TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    1. Create an MCP toolset with the tools server id: 022d50de-28f4-4315-a51e-16e01a9cdbe4 and tool name: search_datadog_logs, server id: 022d50de-28f4-4315-a51e-16e01a9cdbe4 and tool name: get_datadog_metr
    2. Update an MCP toolset with toolset id: 2f17c904-7d64-4c8e-a4b5-2980e5da4949
    3. Read /v1/mcp/toolset/2f17c904-7d64-4c8e-a4b5-2980e5da4949 on every proxy replica until it settles
    View test on GitHub ↗
  774. ✓test_config_update_persists_router_setting_to_get TestRouterSettingstest_config_misc_endpoints_e2e.py›

    No steps recorded

    View test on GitHub ↗
  775. ✓test_create_echoes_every_field_written TestKeyLifecycletest_key_lifecycle_e2e.py›
    1. Add a deployment named e2e-key-lifecycle-a124ca37952b that calls gpt-4o-mini
    View test on GitHub ↗
  776. ✓test_create_persists_and_is_read_back TestFallbackManagementtest_config_misc_endpoints_e2e.py›
    1. Add a deployment named e2e-fallback-primary-16d99884b93e that calls openai/gpt-5.5
    2. Add a deployment named e2e-fallback-secondary-bc993538db4f that calls openai/gpt-5.5
    View test on GitHub ↗
  777. ✓test_create_persists_both_tools_under_the_exact_names_written TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    1. Create an MCP toolset with the tools server id: 1b6fab38-6fc9-498e-bbe2-a13f24262a1d and tool name: search_datadog_logs, server id: 1b6fab38-6fc9-498e-bbe2-a13f24262a1d and tool name: get_datadog_metr
    2. Read /v1/mcp/toolset/62b090db-0cb8-4d64-8cac-85e91100a0a2 on every proxy replica until it settles
    3. Read /v1/mcp/toolset on every proxy replica until it settles
    View test on GitHub ↗
  778. ✓test_create_persists_every_field_on_every_replica TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    1. Read /v1/mcp/server/fb0b8a55-88ef-4aa3-9014-92f8bb7930ac on every proxy replica until it settles
    View test on GitHub ↗
  779. ✓test_created_budget_appears_in_budget_list TestBudgetManagementtest_budget_customer_user_org_e2e.py›

    No steps recorded

    View test on GitHub ↗
  780. ✓test_created_key_appears_in_key_list_inventory TestKeyRoutestest_management_e2e.py›
    1. Generate a virtual key with key alias: e2e-mgmt-keylist-8c50daed2365
    View test on GitHub ↗
  781. ✓test_created_team_appears_in_team_list TestTeamRoutestest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  782. ✓test_created_users_appear_in_user_list TestUserRoutestest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  783. ✓test_creating_a_key_from_the_dashboard_persists_and_works TestDashboardKeyRoutestest_management_e2e.py›
    1. Read the key's settings back from /key/info
    View test on GitHub ↗
  784. ✓test_credential_migration_check_reports_residual_scan TestInventoryRoutestest_config_misc_endpoints_e2e.py›

    No steps recorded

    View test on GitHub ↗
  785. ✓test_customer_create_persists_to_info TestCustomertest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  786. ✓test_delete_forbidden_for_non_admin_key TestKeyManagementRoutestest_key_management_e2e.py›
    1. Generate a virtual key with models: gpt-5.5
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  787. ✓test_delete_persists_and_revokes_team_bound_key TestTeamRoutestest_management_e2e.py›
    1. Generate a virtual key with team id: abed68d8-edf1-4853-ac46-e19a8612b58f
    View test on GitHub ↗
  788. ✓test_delete_removes_from_model_info_catalog TestModelRoutestest_management_e2e.py›
    1. Add a deployment named e2e-mgmt-model-4b90b846c02d that calls openai/gpt-5.5
    2. List the deployments from /model/info
    View test on GitHub ↗
  789. ✓test_delete_removes_from_organization_info TestOrganizationRoutestest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  790. ✓test_delete_removes_the_customer TestCustomerManagementtest_budget_customer_user_org_e2e.py›

    No steps recorded

    View test on GitHub ↗
  791. ✓test_delete_removes_the_server_from_every_replica TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    1. Check that /v1/mcp/server/a02adbf3-b667-432f-bcb7-e13c2a4bdfe8 returns 404 on every proxy replica
    2. Read /v1/mcp/server on every proxy replica until it settles
    View test on GitHub ↗
  792. ✓test_delete_removes_the_toolset_from_every_replica TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    1. Create an MCP toolset with the tools server id: 9795d189-b9e5-419f-a16b-2c7e80e7333b and tool name: search_datadog_logs, server id: 9795d189-b9e5-419f-a16b-2c7e80e7333b and tool name: get_datadog_metr
    2. Delete the MCP toolset
    3. Check that /v1/mcp/toolset/80f844d8-bc3c-43f5-a38d-deece690c68f returns 404 on every proxy replica
    4. Read /v1/mcp/toolset on every proxy replica until it settles
    View test on GitHub ↗
  793. ✓test_delete_removes_the_user_from_inventory TestUserRoutestest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  794. ✓test_delete_revokes_info_and_chat_on_every_replica TestKeyLifecycletest_key_lifecycle_e2e.py›
    1. Read /key/info on every proxy replica until they all agree
    View test on GitHub ↗
  795. ✓test_delete_revokes_the_key_on_chat TestKeyRoutestest_management_e2e.py›
    1. Generate a virtual key with models: gemini-2.5-flash
    View test on GitHub ↗
  796. ✓test_editing_a_key_from_the_dashboard_persists_and_is_enforced TestDashboardKeyRoutestest_management_e2e.py›
    1. Generate a virtual key with models: gemini-2.5-flash, key alias: e2e-mgmt-uiedit-554367621ce6, tpm limit: 100 and rpm limit: 200
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  797. ✓test_end_user_new_persists_to_end_user_info TestCustomerManagementtest_budget_customer_user_org_e2e.py›

    No steps recorded

    View test on GitHub ↗
  798. ✓test_estimate_computes_cost_from_token_counts TestCostEstimatetest_config_misc_endpoints_e2e.py›

    No steps recorded

    View test on GitHub ↗
  799. ✓test_explicit_null_clears_the_budget_and_its_reset_time TestKeyLifecycletest_key_lifecycle_e2e.py›
    1. Read /key/info on every proxy replica until they all agree
    View test on GitHub ↗
  800. ✓test_gdpr_check_derives_verdict_from_the_request TestComplianceRoutestest_config_misc_endpoints_e2e.py›

    No steps recorded

    View test on GitHub ↗
  801. ✓test_generate_forbidden_for_non_admin_key TestKeyManagementRoutestest_key_management_e2e.py›
    1. Generate a virtual key with models: gpt-5.5
    View test on GitHub ↗
  802. ✓test_generate_persists_to_key_info_and_scopes_chat TestKeyRoutestest_management_e2e.py›
    1. Generate a virtual key with models: gemini-2.5-flash, key alias: e2e-mgmt-key-b867def343c9, tpm limit: 424242 and rpm limit: 424243
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  803. ✓test_health_reports_the_calling_key_healthy TestKeyManagementRoutestest_key_management_e2e.py›
    1. Generate a virtual key with models: gpt-5.5
    View test on GitHub ↗
  804. ✓test_info_reflects_the_fields_the_key_was_created_with TestKeyManagementRoutestest_key_management_e2e.py›
    1. Generate a virtual key with models: gpt-5.5, gemini-2.5-flash, key alias: e2e-mgmt-keyinfo-36d4831d2702, tpm limit: 131313 and rpm limit: 141414
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  805. ✓test_info_returns_created_team_fields TestTeamManagementRoutestest_team_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  806. ✓test_is_null_finds_the_budget_left_uncapped TestBudgetListV1test_budget_customer_user_org_e2e.py›

    No steps recorded

    View test on GitHub ↗
  807. ✓test_is_refused_for_a_non_admin_key TestBudgetListV1test_budget_customer_user_org_e2e.py›
    1. Generate a virtual key with default settings
    View test on GitHub ↗
  808. ✓test_key_delete_by_alias_writes_audit_row TestKeyDeletionAuditLogtest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  809. ✓test_key_delete_by_key_writes_audit_row TestKeyDeletionAuditLogtest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  810. ✓test_key_serves_its_model_and_is_denied_others TestKeyLifecycletest_key_lifecycle_e2e.py›

    No steps recorded

    View test on GitHub ↗
  811. ✓test_llm_only_key_forbidden_from_management_writes TestManagementRoutePermissionstest_management_e2e.py›
    1. Generate a virtual key with allowed routes: llm_api_routes
    View test on GitHub ↗
  812. ✓test_member_add_and_delete_persist_to_team_info TestTeamRoutestest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  813. ✓test_member_add_forbidden_to_plain_member TestTeamManagementRoutestest_team_management_e2e.py›
    1. Generate a virtual key with user id: cadcebf1-38a8-4463-b356-2ef09de7f1d7 and team id: 23b440ae-20c9-44d0-9995-a2b4f77712e6
    View test on GitHub ↗
  814. ✓test_member_add_records_membership TestOrganizationMembershiptest_budget_customer_user_org_e2e.py›

    No steps recorded

    View test on GitHub ↗
  815. ✓test_member_cannot_write_and_another_team_cannot_read_the_key TestJwtManagementtest_jwt_management_e2e.py›
    1. Create a team with team alias: e2e-jwt-15acedc51bfd and team id: e2e-jwt-team-15acedc51bfd
    2. Create a team with team alias: fadb9a1fa938 and team id: e2e-jwt-team-fadb9a1fa938
    View test on GitHub ↗
  816. ✓test_member_delete_forbidden_to_plain_member TestTeamManagementRoutestest_team_management_e2e.py›
    1. Generate a virtual key with user id: 918975d0-c5af-4c84-95c3-0a6ad735f67b and team id: b2ada9df-a91d-4067-807b-c17e9336b0f0
    View test on GitHub ↗
  817. ✓test_member_delete_persists_to_team_info TestTeamManagementRoutestest_team_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  818. ✓test_member_update_persists_role_and_budget TestTeamManagementRoutestest_team_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  819. ✓test_multi_group_actor_keeps_exact_memberships TestJwtManagementtest_jwt_management_e2e.py›
    1. Create a team with team alias: e2e-team-b304e04f8094 and organization id: e942b003-1c37-4865-977f-2313990a75d9
    2. Create a team with team alias: e2e-team-bf776d5ecba4 and organization id: 1dc90859-8f90-48d3-bafe-e2dcf4af8f1f
    View test on GitHub ↗
  820. ✓test_new_access_group_tags_the_deployment TestModelAccessGroupRoutestest_model_tag_accessgroup_e2e.py›
    1. Add a deployment named e2e-mgmt-agmodel-562553ed18b8 that calls openai/gpt-5.5
    View test on GitHub ↗
  821. ✓test_new_is_denied_to_non_admin_keys TestTeamManagementRoutestest_team_management_e2e.py›
    1. Generate a virtual key with default settings
    2. Generate a virtual key with user id: e049074f-211e-4f19-8b15-f751a0ae7a09
    View test on GitHub ↗
  822. ✓test_new_is_refused_for_a_non_admin_key TestBudgetManagementtest_budget_customer_user_org_e2e.py›
    1. Generate a virtual key with default settings
    View test on GitHub ↗
  823. ✓test_new_persists_mapping_and_is_read_back TestJwtKeyMappingtest_config_misc_endpoints_e2e.py›
    1. Generate a virtual key with default settings
    View test on GitHub ↗
  824. ✓test_new_persists_to_customer_info TestCustomerManagementtest_budget_customer_user_org_e2e.py›

    No steps recorded

    View test on GitHub ↗
  825. ✓test_new_persists_to_model_info_catalog TestModelRoutestest_management_e2e.py›
    1. Add a deployment named e2e-mgmt-model-348b6ff4edb2 that calls openai/gpt-5.5
    2. List the deployments from /model/info
    View test on GitHub ↗
  826. ✓test_new_persists_to_organization_info TestOrganizationRoutestest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  827. ✓test_new_persists_to_tag_list TestTagRoutestest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  828. ✓test_new_persists_to_team_info_and_binds_keys TestTeamRoutestest_management_e2e.py›
    1. Generate a virtual key with team id: f41d8ef9-2270-4273-b70d-f661ed780bf6
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  829. ✓test_new_persists_to_user_info TestUserRoutestest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  830. ✓test_new_user_is_readable_via_user_info TestUserManagementtest_budget_customer_user_org_e2e.py›

    No steps recorded

    View test on GitHub ↗
  831. ✓test_non_admin_key_cannot_add_global_model TestModelRoutestest_model_tag_accessgroup_e2e.py›
    1. Generate a virtual key with default settings
    2. List the deployments from /model/info
    View test on GitHub ↗
  832. ✓test_oidc_browser_profile_identity_mapping TestJwtManagementtest_jwt_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  833. ✓test_partial_setup_removes_previously_created_identities[group] TestJwtManagementtest_jwt_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  834. ✓test_partial_setup_removes_previously_created_identities[user] TestJwtManagementtest_jwt_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  835. ✓test_partial_update_changes_only_the_named_field TestKeyLifecycletest_key_lifecycle_e2e.py›
    1. Read /key/info on every proxy replica until they all agree
    View test on GitHub ↗
  836. ✓test_project_detachment_preserves_key_scope_and_refreshes_auth TestKeyManagementRoutestest_key_management_e2e.py›
    1. Add a deployment named e2e-detach-952cbd664864 that calls openai/synthetic-detachment
    2. Generate a virtual key with models: e2e-detach-952cbd664864, max budget: 5.0, team id: b2b11f51-17e8-4d83-b877-ffc5b9452866, project id: ff561d3c-ef0e-46a6-9eea-c33aebd5e513, organization id: 6f2ba712
    3. Read the key's settings back from /key/info
    View test on GitHub ↗
  837. ✓test_read_reflects_the_create_on_every_replica TestKeyLifecycletest_key_lifecycle_e2e.py›
    1. Read /key/info on every proxy replica until they all agree
    View test on GitHub ↗
  838. ✓test_refuses_a_sort_field_and_a_parameter_it_does_not_support TestBudgetListV1test_budget_customer_user_org_e2e.py›

    No steps recorded

    View test on GitHub ↗
  839. ✓test_regenerate_rotates_to_a_working_new_key TestKeyRegenerationtest_management_e2e.py›
    1. Generate a virtual key with models: gpt-5.5
    View test on GitHub ↗
  840. ✓test_regenerate_with_grace_period_keeps_old_key_until_revoked TestKeyRegenerationtest_management_e2e.py›
    1. Generate a virtual key with models: gpt-5.5
    View test on GitHub ↗
  841. ✓test_register_submits_pending_server TestMcpServerSubmissiontest_config_misc_endpoints_e2e.py›
    1. Generate a virtual key with team id: 6b2028ad-8b56-4778-a13f-67376e01bb98
    View test on GitHub ↗
  842. ✓test_reset_spend_zeroes_recorded_spend_and_lifts_the_budget_block TestKeyManagementRoutestest_key_management_e2e.py›
    1. Generate a virtual key with models: claude-haiku-4-5 and max budget: 3e-06
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  843. ✓test_sorts_pages_and_filters_the_budgets_it_created TestBudgetListV1test_budget_customer_user_org_e2e.py›

    No steps recorded

    View test on GitHub ↗
  844. ✓test_successful_actor_cleanup_removes_owned_state TestJwtManagementtest_jwt_management_e2e.py›
    1. Create a team with team alias: e2e-team-0dcac166caa5 and organization id: 00298f58-2108-49cb-8e18-628019ad397e
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  845. ✓test_tag_delete_removes_from_list TestTagRoutestest_model_tag_accessgroup_e2e.py›

    No steps recorded

    View test on GitHub ↗
  846. ✓test_tag_list_reports_created_tag TestTagRoutestest_model_tag_accessgroup_e2e.py›

    No steps recorded

    View test on GitHub ↗
  847. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[budget_duration] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    1. Generate a virtual key with user id: 75fd08cc-759f-4f40-b8ee-359255f63c5f
    View test on GitHub ↗
  848. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[max_budget] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    1. Generate a virtual key with user id: fb0af149-1cb8-46f1-bfca-43aa0134d970
    View test on GitHub ↗
  849. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[metadata] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    1. Generate a virtual key with user id: 307ff0cd-2c6c-4887-ba7b-8b157d579661
    View test on GitHub ↗
  850. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[models] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    1. Generate a virtual key with user id: 9f8bd86b-dfbe-42cc-a4c3-f7e86de412b2
    View test on GitHub ↗
  851. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[rpm_limit] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    1. Generate a virtual key with user id: 966c3dfb-b3b0-47d1-b0dd-a64a9bfccd6c
    View test on GitHub ↗
  852. ✓test_team_admin_cannot_change_a_setting_that_is_not_enabled[team_alias] TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    1. Generate a virtual key with user id: 8a446949-0a72-42b8-a3ad-85dbb9c9ac57
    View test on GitHub ↗
  853. ✓test_team_admin_cannot_change_any_team_setting TestTeamAdminWithNoEditableFieldstest_team_management_e2e.py›
    1. Generate a virtual key with user id: 7de635ff-1691-4ec6-b534-09ebbcafea68
    View test on GitHub ↗
  854. ✓test_team_admin_cannot_raise_an_org_team_budget_under_the_org_cap TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    1. Generate a virtual key with user id: 82795e26-b0b4-48da-9aec-62f0252372a4
    View test on GitHub ↗
  855. ✓test_team_admin_cannot_raise_or_remove_the_budget[raise] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    1. Generate a virtual key with user id: 14aa9eb3-7592-4b7d-a852-1024e9887cd6
    View test on GitHub ↗
  856. ✓test_team_admin_cannot_raise_or_remove_the_budget[remove] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    1. Generate a virtual key with user id: 778c7e3e-5c49-4d0f-8df2-c5852e01b559
    View test on GitHub ↗
  857. ✓test_team_admin_resending_the_budget_settings_keeps_the_next_budget_reset TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    1. Generate a virtual key with user id: e8b4d8d9-674b-4905-b153-862b92a4b512
    View test on GitHub ↗
  858. ✓test_team_admin_saves_a_new_rpm_limit_and_a_tighter_budget[first-budget] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    1. Generate a virtual key with user id: 6ca9b43d-137a-4a3a-acdd-55e1b0e102cd
    View test on GitHub ↗
  859. ✓test_team_admin_saves_a_new_rpm_limit_and_a_tighter_budget[lower] TestTeamAdminWithRpmLimitAndMaxBudgetEnabledtest_team_management_e2e.py›
    1. Generate a virtual key with user id: 65b369ad-4abb-44da-a2d2-6bd8cedf6017
    View test on GitHub ↗
  860. ✓test_team_admin_saves_the_settings_form_with_a_new_tpm_limit TestTeamAdminWithTpmLimitEnabledtest_team_management_e2e.py›
    1. Generate a virtual key with user id: 9086fcd0-5009-4c29-b230-c2e6a4fbda7c
    View test on GitHub ↗
  861. ✓test_team_delete_succeeds_for_team_larger_than_db_pool TestTeamRoutestest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  862. ✓test_team_delete_writes_audit_row_for_team_keys TestKeyDeletionAuditLogtest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  863. ✓test_team_member_delete_writes_audit_row_for_member_keys TestKeyDeletionAuditLogtest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  864. ✓test_tool_list_returns_catalog_with_consistent_total TestInventoryRoutestest_config_misc_endpoints_e2e.py›

    No steps recorded

    View test on GitHub ↗
  865. ✓test_two_actor_sets_keep_tenants_and_keys_isolated TestJwtManagementtest_jwt_management_e2e.py›
    1. Create a team with team alias: e2e-team-c2178aa0e864 and organization id: ebf11cab-e8b1-44fb-aec8-d84a82778a7c
    2. Create a team with team alias: e2e-team-fe0d35f42fea and organization id: baf28b8a-adb2-4105-aa0d-08cd340d1ba1
    3. Read the key's settings back from /key/info
    View test on GitHub ↗
  866. ✓test_unblock_flips_key_info_blocked_back TestKeyManagementRoutestest_key_management_e2e.py›
    1. Generate a virtual key with models: gpt-5.5
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  867. ✓test_update_accepts_per_model_budgets_including_punctuated_names TestBudgetManagementtest_budget_customer_user_org_e2e.py›

    No steps recorded

    View test on GitHub ↗
  868. ✓test_update_alias_persists_to_organization_info TestOrganizationRoutestest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  869. ✓test_update_forbidden_for_non_admin_key TestKeyManagementRoutestest_key_management_e2e.py›
    1. Generate a virtual key with models: gpt-5.5
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  870. ✓test_update_max_budget_persists_to_budget_info TestBudgetManagementtest_budget_customer_user_org_e2e.py›

    No steps recorded

    View test on GitHub ↗
  871. ✓test_update_models_persists_and_flips_enforcement TestKeyRoutestest_management_e2e.py›
    1. Generate a virtual key with models: gemini-2.5-flash
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  872. ✓test_update_persists_input_cost_to_model_info TestModelRoutestest_management_e2e.py›
    1. Add a deployment named e2e-mgmt-model-993db829a849 that calls gpt-4o-mini
    2. List the deployments from /model/info
    3. Update a deployment's settings to model: gpt-4o-mini and input cost per token: 2.22e-06
    4. List the deployments from /model/info
    View test on GitHub ↗
  873. ✓test_update_persists_to_team_info TestTeamRoutestest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  874. ✓test_update_persists_to_user_info TestUserRoutestest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  875. ✓test_updating_only_the_alias_keeps_every_other_field_on_every_replica TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›
    1. Read /v1/mcp/server/3eb683bf-e5e0-4f95-9a64-cbf29ad79908 on every proxy replica until it settles
    View test on GitHub ↗
  876. ✓test_updating_only_the_description_keeps_the_tools_and_name TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    1. Create an MCP toolset with the tools server id: e45f7d27-5935-4962-9bed-a42c1216697c and tool name: search_datadog_logs, server id: e45f7d27-5935-4962-9bed-a42c1216697c and tool name: get_datadog_metr
    2. Update an MCP toolset with toolset id: 53230d97-d75a-4e66-a189-b2d089cf52c6 and description: edited
    3. Read /v1/mcp/toolset/53230d97-d75a-4e66-a189-b2d089cf52c6 on every proxy replica until it settles
    View test on GitHub ↗
  877. ✓test_updating_the_tools_to_one_entry_reads_back_exactly_that_entry TestMcpToolsetLifecycletest_mcp_lifecycle_e2e.py›
    1. Create an MCP toolset with the tools server id: 15b7bf70-8c65-4e6f-b80b-74fff7904795 and tool name: search_datadog_logs, server id: 15b7bf70-8c65-4e6f-b80b-74fff7904795 and tool name: get_datadog_metr
    2. Update an MCP toolset with toolset id: 1a6eaa08-bd6c-4ca9-a7c3-eaea3abb5aa8 and tools: server id: 15b7bf70-8c65-4e6f-b80b-74fff7904795 and tool name: search_datadog_logs
    3. Read /v1/mcp/toolset/1a6eaa08-bd6c-4ca9-a7c3-eaea3abb5aa8 on every proxy replica until it settles
    View test on GitHub ↗
  878. ✓test_user_delete_writes_audit_row_for_user_keys TestKeyDeletionAuditLogtest_management_e2e.py›

    No steps recorded

    View test on GitHub ↗
  879. ✓test_workflow_runs_list_returns_consistent_count TestInventoryRoutestest_config_misc_endpoints_e2e.py›

    No steps recorded

    View test on GitHub ↗
  880. −test_created_server_is_listed_with_every_field TestMcpServerLifecycletest_mcp_lifecycle_e2e.py›

    No steps recorded

    product gap: GET /v1/mcp/server builds each row from the in-memory registry, whose _build_mcp_server_table sets description from mcp_info['description'], so the list reports the mcp_info description while GET /v1/mcp/server/{server_id} reports the stored description column. A server created with both set to different text reads back with two different descriptions depending on the route

    View test on GitHub ↗
  881. ✓test_a_failed_first_turn_is_reported_as_the_first_turn_probe_unit_tests/test_http_probe.py›

    No steps recorded

    View test on GitHub ↗
  882. ✓test_a_history_with_no_server_tool_block_is_not_a_pass_probe_unit_tests/test_http_probe.py›

    No steps recorded

    View test on GitHub ↗
  883. ✓test_a_pending_tool_use_is_answered_with_the_id_the_model_returned_probe_unit_tests/test_http_probe.py›

    No steps recorded

    View test on GitHub ↗
  884. ✓test_a_turn_truncated_before_the_result_block_is_not_a_pass_probe_unit_tests/test_http_probe.py›

    No steps recorded

    View test on GitHub ↗
  885. ✓test_a_turn_with_no_pending_tool_use_gets_a_plain_follow_up_probe_unit_tests/test_http_probe.py›

    No steps recorded

    View test on GitHub ↗
  886. ✓test_accepts_a_replayed_server_tool_pair_probe_unit_tests/test_http_probe.py›
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    View test on GitHub ↗
  887. ✓test_basic_messaging_non_streaming_anthropicbasic_messaging_non_streaming/test_anthropic.py›
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    View test on GitHub ↗
  888. ✓test_basic_messaging_non_streaming_azurebasic_messaging_non_streaming/test_azure.py›

    No steps recorded

    View test on GitHub ↗
  889. ✓test_basic_messaging_non_streaming_azure_openaibasic_messaging_non_streaming/test_azure_openai.py›
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    View test on GitHub ↗
  890. ✓test_basic_messaging_non_streaming_bedrock_conversebasic_messaging_non_streaming/test_bedrock_converse.py›
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    View test on GitHub ↗
  891. ✓test_basic_messaging_non_streaming_bedrock_invokebasic_messaging_non_streaming/test_bedrock_invoke.py›

    No steps recorded

    View test on GitHub ↗
  892. ✓test_basic_messaging_non_streaming_vertex_aibasic_messaging_non_streaming/test_vertex_ai.py›

    No steps recorded

    View test on GitHub ↗
  893. ✓test_basic_messaging_non_streaming_vertex_ai_gptbasic_messaging_non_streaming/test_vertex_ai_gpt.py›

    No steps recorded

    View test on GitHub ↗
  894. ✓test_basic_messaging_streaming_anthropicbasic_messaging_streaming/test_anthropic.py›

    No steps recorded

    View test on GitHub ↗
  895. ✓test_basic_messaging_streaming_azurebasic_messaging_streaming/test_azure.py›

    No steps recorded

    View test on GitHub ↗
  896. ✓test_basic_messaging_streaming_azure_openaibasic_messaging_streaming/test_azure_openai.py›

    No steps recorded

    View test on GitHub ↗
  897. ✓test_basic_messaging_streaming_bedrock_conversebasic_messaging_streaming/test_bedrock_converse.py›
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    View test on GitHub ↗
  898. ✓test_basic_messaging_streaming_bedrock_invokebasic_messaging_streaming/test_bedrock_invoke.py›

    No steps recorded

    View test on GitHub ↗
  899. ✓test_basic_messaging_streaming_vertex_aibasic_messaging_streaming/test_vertex_ai.py›

    No steps recorded

    View test on GitHub ↗
  900. ✓test_basic_messaging_streaming_vertex_ai_gptbasic_messaging_streaming/test_vertex_ai_gpt.py›

    No steps recorded

    View test on GitHub ↗
  901. ✓test_bedrock_503_is_retryable_but_not_rate_limit_shaped_driver_unit_tests/test_retry_classification.py›

    No steps recorded

    View test on GitHub ↗
  902. ✓test_count_tokens_anthropiccount_tokens/test_anthropic.py›
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7
    View test on GitHub ↗
  903. ✓test_count_tokens_azurecount_tokens/test_azure.py›
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5-azure
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5-azure
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7-azure
    View test on GitHub ↗
  904. ✓test_count_tokens_bedrock_conversecount_tokens/test_bedrock_converse.py›
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5-bedrock-converse
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5-bedrock-converse
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7-bedrock-converse
    View test on GitHub ↗
  905. ✓test_count_tokens_bedrock_invokecount_tokens/test_bedrock_invoke.py›
    1. Count tokens with /v1/messages/count_tokens for claude-haiku-4-5-bedrock-invoke
    2. Count tokens with /v1/messages/count_tokens for claude-sonnet-4-5-bedrock-invoke
    3. Count tokens with /v1/messages/count_tokens for claude-opus-4-7-bedrock-invoke
    View test on GitHub ↗
  906. ✓test_driver_error_message_is_classified_driver_unit_tests/test_retry_classification.py›

    No steps recorded

    View test on GitHub ↗
  907. ✓test_find_regressions_empty_old_matrix_is_safe_builder_unit_tests/test_matrix_builder.py›

    No steps recorded

    View test on GitHub ↗
  908. ✓test_find_regressions_flags_pass_to_fail_builder_unit_tests/test_matrix_builder.py›
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    View test on GitHub ↗
  909. ✓test_find_regressions_ignores_green_to_grey_builder_unit_tests/test_matrix_builder.py›

    No steps recorded

    View test on GitHub ↗
  910. ✓test_find_regressions_ignores_improvements_and_steady_green_builder_unit_tests/test_matrix_builder.py›

    No steps recorded

    View test on GitHub ↗
  911. ✓test_find_regressions_ignores_new_cells_without_baseline_builder_unit_tests/test_matrix_builder.py›

    No steps recorded

    View test on GitHub ↗
  912. ✓test_find_regressions_ignores_red_to_red_builder_unit_tests/test_matrix_builder.py›

    No steps recorded

    View test on GitHub ↗
  913. ✓test_find_regressions_matches_by_id_not_name_builder_unit_tests/test_matrix_builder.py›

    No steps recorded

    View test on GitHub ↗
  914. ✓test_find_regressions_reports_multiple_sorted_builder_unit_tests/test_matrix_builder.py›

    No steps recorded

    View test on GitHub ↗
  915. ✓test_passing_outcome_is_never_retryable_driver_unit_tests/test_retry_classification.py›

    No steps recorded

    View test on GitHub ↗
  916. ✓test_passthrough_anthropicpassthrough/test_anthropic.py›

    No steps recorded

    View test on GitHub ↗
  917. ✓test_passthrough_bedrock_conversepassthrough/test_bedrock_converse.py›

    No steps recorded

    View test on GitHub ↗
  918. ✓test_passthrough_bedrock_invokepassthrough/test_bedrock_invoke.py›

    No steps recorded

    View test on GitHub ↗
  919. ✓test_passthrough_vertex_aipassthrough/test_vertex_ai.py›

    No steps recorded

    View test on GitHub ↗
  920. ✓test_pdf_input_anthropicpdf_input/test_anthropic.py›

    No steps recorded

    View test on GitHub ↗
  921. ✓test_pdf_input_azurepdf_input/test_azure.py›

    No steps recorded

    View test on GitHub ↗
  922. ✓test_pdf_input_bedrock_conversepdf_input/test_bedrock_converse.py›

    No steps recorded

    View test on GitHub ↗
  923. ✓test_pdf_input_bedrock_invokepdf_input/test_bedrock_invoke.py›

    No steps recorded

    View test on GitHub ↗
  924. ✓test_pdf_input_vertex_aipdf_input/test_vertex_ai.py›

    No steps recorded

    View test on GitHub ↗
  925. ✓test_picks_the_newest_published_version_at_least_min_age_old_pr_gate_unit_tests/test_pr_gate_version_resolver.py›

    No steps recorded

    View test on GitHub ↗
  926. ✓test_prompt_caching_1h_anthropicprompt_caching_1h/test_anthropic.py›

    No steps recorded

    View test on GitHub ↗
  927. ✓test_prompt_caching_1h_azureprompt_caching_1h/test_azure.py›

    No steps recorded

    View test on GitHub ↗
  928. ✓test_prompt_caching_1h_bedrock_converseprompt_caching_1h/test_bedrock_converse.py›

    No steps recorded

    View test on GitHub ↗
  929. ✓test_prompt_caching_1h_bedrock_invokeprompt_caching_1h/test_bedrock_invoke.py›

    No steps recorded

    View test on GitHub ↗
  930. ✓test_prompt_caching_1h_vertex_aiprompt_caching_1h/test_vertex_ai.py›

    No steps recorded

    View test on GitHub ↗
  931. ✓test_prompt_caching_5m_anthropicprompt_caching_5m/test_anthropic.py›

    No steps recorded

    View test on GitHub ↗
  932. ✓test_prompt_caching_5m_azureprompt_caching_5m/test_azure.py›

    No steps recorded

    View test on GitHub ↗
  933. ✓test_prompt_caching_5m_bedrock_converseprompt_caching_5m/test_bedrock_converse.py›

    No steps recorded

    View test on GitHub ↗
  934. ✓test_prompt_caching_5m_bedrock_invokeprompt_caching_5m/test_bedrock_invoke.py›

    No steps recorded

    View test on GitHub ↗
  935. ✓test_prompt_caching_5m_vertex_aiprompt_caching_5m/test_vertex_ai.py›

    No steps recorded

    View test on GitHub ↗
  936. ✓test_raises_when_the_only_old_enough_version_is_unpublished_pr_gate_unit_tests/test_pr_gate_version_resolver.py›

    No steps recorded

    View test on GitHub ↗
  937. ✓test_reports_the_status_when_the_replayed_history_is_rejected_probe_unit_tests/test_http_probe.py›

    No steps recorded

    View test on GitHub ↗
  938. ✓test_shapes_are_classified_independently[[claude-opus-4-7-bedrock-converse] tool_search probe failed: status 503: {"error":{"message":"litellm.ServiceUnavailableError: BedrockException - {\\"message\\":\\"Bedrock is unable to process your request.\\"}"}}-False-True]_driver_unit_tests/test_retry_classification.py›
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    View test on GitHub ↗
  939. ✓test_shapes_are_classified_independently[claude CLI timed out after 120s-True-False]_driver_unit_tests/test_retry_classification.py›

    No steps recorded

    View test on GitHub ↗
  940. ✓test_shapes_are_classified_independently[status 400: {"error":"bad request"}-False-False]_driver_unit_tests/test_retry_classification.py›

    No steps recorded

    View test on GitHub ↗
  941. ✓test_shapes_are_classified_independently[status 429: {"error":{"message":"Rate limit reached"}}-True-False]_driver_unit_tests/test_retry_classification.py›

    No steps recorded

    View test on GitHub ↗
  942. ✓test_shapes_are_classified_independently[status 503 service unavailable-False-True]_driver_unit_tests/test_retry_classification.py›

    No steps recorded

    View test on GitHub ↗
  943. ✓test_shapes_are_classified_independently[status 529: {"type":"overloaded_error"}-False-True]_driver_unit_tests/test_retry_classification.py›

    No steps recorded

    View test on GitHub ↗
  944. ✓test_shapes_are_classified_independently[throttling exception from provider-True-False]_driver_unit_tests/test_retry_classification.py›

    No steps recorded

    View test on GitHub ↗
  945. ✓test_shapes_are_classified_independently[upstream overloaded, try again later-False-True]_driver_unit_tests/test_retry_classification.py›

    No steps recorded

    View test on GitHub ↗
  946. ✓test_skips_a_version_npm_has_unpublished_pr_gate_unit_tests/test_pr_gate_version_resolver.py›
    1. Add a deployment named claude-haiku-4-5 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named claude-sonnet-4-5 that calls anthropic/claude-sonnet-4-5
    3. Add a deployment named claude-opus-4-7 that calls anthropic/claude-opus-4-7
    4. Add a deployment named claude-haiku-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    5. Add a deployment named claude-sonnet-4-5-bedrock-invoke that calls bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    6. Add a deployment named claude-opus-4-7-bedrock-invoke that calls bedrock/us.anthropic.claude-opus-4-7
    7. Add a deployment named claude-haiku-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
    8. Add a deployment named claude-sonnet-4-5-bedrock-converse that calls bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0
    9. Add a deployment named claude-opus-4-7-bedrock-converse that calls bedrock/converse/us.anthropic.claude-opus-4-7
    10. Add a deployment named claude-haiku-4-5-vertex that calls vertex_ai/claude-haiku-4-5
    11. Add a deployment named claude-sonnet-4-5-vertex that calls vertex_ai/claude-sonnet-4-5
    12. Add a deployment named claude-opus-4-7-vertex that calls vertex_ai/claude-opus-4-7
    13. Add a deployment named claude-haiku-4-5-azure that calls azure_ai/claude-haiku-4-5
    14. Add a deployment named claude-sonnet-4-5-azure that calls azure_ai/claude-sonnet-4-5
    15. Add a deployment named claude-opus-4-7-azure that calls azure_ai/claude-opus-4-7
    16. Add a deployment named gpt-5-6-sol-openai that calls openai/gpt-5.6-sol
    17. Add a deployment named gpt-5-6-terra-openai that calls openai/gpt-5.6-terra
    18. Add a deployment named gpt-5-6-luna-openai that calls openai/gpt-5.6-luna
    19. Add a deployment named gpt-5-6-sol-azure-openai that calls azure/gpt-5.6-sol
    20. Add a deployment named gpt-5-6-terra-azure-openai that calls azure/gpt-5.6-terra
    21. Add a deployment named gpt-5-6-luna-azure-openai that calls azure/gpt-5.6-luna
    22. Add a deployment named gpt-5-6-sol-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-sol
    23. Add a deployment named gpt-5-6-terra-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-terra
    24. Add a deployment named gpt-5-6-luna-bedrock-mantle that calls bedrock_mantle/openai.gpt-5.6-luna
    View test on GitHub ↗
  947. ✓test_structured_outputs_anthropicstructured_outputs/test_anthropic.py›

    No steps recorded

    View test on GitHub ↗
  948. ✓test_structured_outputs_azurestructured_outputs/test_azure.py›

    No steps recorded

    View test on GitHub ↗
  949. ✓test_structured_outputs_bedrock_conversestructured_outputs/test_bedrock_converse.py›

    No steps recorded

    View test on GitHub ↗
  950. ✓test_structured_outputs_bedrock_invokestructured_outputs/test_bedrock_invoke.py›

    No steps recorded

    View test on GitHub ↗
  951. ✓test_structured_outputs_vertex_aistructured_outputs/test_vertex_ai.py›

    No steps recorded

    View test on GitHub ↗
  952. ✓test_thinking_anthropicthinking/test_anthropic.py›

    No steps recorded

    View test on GitHub ↗
  953. ✓test_thinking_azurethinking/test_azure.py›

    No steps recorded

    View test on GitHub ↗
  954. ✓test_thinking_bedrock_conversethinking/test_bedrock_converse.py›

    No steps recorded

    View test on GitHub ↗
  955. ✓test_thinking_bedrock_invokethinking/test_bedrock_invoke.py›

    No steps recorded

    View test on GitHub ↗
  956. ✓test_thinking_vertex_aithinking/test_vertex_ai.py›

    No steps recorded

    View test on GitHub ↗
  957. ✓test_thinking_with_tool_use_anthropicthinking_with_tool_use/test_anthropic.py›

    No steps recorded

    View test on GitHub ↗
  958. ✓test_thinking_with_tool_use_azurethinking_with_tool_use/test_azure.py›

    No steps recorded

    View test on GitHub ↗
  959. ✓test_thinking_with_tool_use_bedrock_conversethinking_with_tool_use/test_bedrock_converse.py›

    No steps recorded

    View test on GitHub ↗
  960. ✓test_thinking_with_tool_use_bedrock_invokethinking_with_tool_use/test_bedrock_invoke.py›

    No steps recorded

    View test on GitHub ↗
  961. ✓test_thinking_with_tool_use_vertex_aithinking_with_tool_use/test_vertex_ai.py›

    No steps recorded

    View test on GitHub ↗
  962. ✓test_tool_search_anthropictool_search/test_anthropic.py›
    1. Send a /v1/messages request to claude-haiku-4-5
    2. Send a /v1/messages request to claude-sonnet-4-5
    3. Send a /v1/messages request to claude-opus-4-7
    View test on GitHub ↗
  963. ✓test_tool_search_bedrock_conversetool_search/test_bedrock_converse.py›
    1. Send a /v1/messages request to claude-haiku-4-5-bedrock-converse
    2. Send a /v1/messages request to claude-sonnet-4-5-bedrock-converse
    3. Send a /v1/messages request to claude-opus-4-7-bedrock-converse
    View test on GitHub ↗
  964. ✓test_tool_search_bedrock_invoketool_search/test_bedrock_invoke.py›
    1. Send a /v1/messages request to claude-haiku-4-5-bedrock-invoke
    2. Send a /v1/messages request to claude-sonnet-4-5-bedrock-invoke
    3. Send a /v1/messages request to claude-opus-4-7-bedrock-invoke
    View test on GitHub ↗
  965. ✓test_tool_search_history_bedrock_invoketool_search/test_bedrock_invoke.py›
    1. Send a /v1/messages request to claude-haiku-4-5-bedrock-invoke
    2. Send a /v1/messages request to claude-sonnet-4-5-bedrock-invoke
    3. Send a /v1/messages request to claude-opus-4-7-bedrock-invoke
    View test on GitHub ↗
  966. ✓test_tool_use_anthropictool_use/test_anthropic.py›

    No steps recorded

    View test on GitHub ↗
  967. ✓test_tool_use_azuretool_use/test_azure.py›

    No steps recorded

    View test on GitHub ↗
  968. ✓test_tool_use_azure_openaitool_use/test_azure_openai.py›

    No steps recorded

    View test on GitHub ↗
  969. ✓test_tool_use_bedrock_conversetool_use/test_bedrock_converse.py›

    No steps recorded

    View test on GitHub ↗
  970. ✓test_tool_use_bedrock_invoketool_use/test_bedrock_invoke.py›

    No steps recorded

    View test on GitHub ↗
  971. ✓test_tool_use_streaming_anthropictool_use_streaming/test_anthropic.py›

    No steps recorded

    View test on GitHub ↗
  972. ✓test_tool_use_streaming_azuretool_use_streaming/test_azure.py›

    No steps recorded

    View test on GitHub ↗
  973. ✓test_tool_use_streaming_azure_openaitool_use_streaming/test_azure_openai.py›

    No steps recorded

    View test on GitHub ↗
  974. ✓test_tool_use_streaming_bedrock_conversetool_use_streaming/test_bedrock_converse.py›

    No steps recorded

    View test on GitHub ↗
  975. ✓test_tool_use_streaming_bedrock_invoketool_use_streaming/test_bedrock_invoke.py›

    No steps recorded

    View test on GitHub ↗
  976. ✓test_tool_use_streaming_vertex_aitool_use_streaming/test_vertex_ai.py›

    No steps recorded

    View test on GitHub ↗
  977. ✓test_tool_use_streaming_vertex_ai_gpttool_use_streaming/test_vertex_ai_gpt.py›

    No steps recorded

    View test on GitHub ↗
  978. ✓test_tool_use_vertex_aitool_use/test_vertex_ai.py›

    No steps recorded

    View test on GitHub ↗
  979. ✓test_tool_use_vertex_ai_gpttool_use/test_vertex_ai_gpt.py›

    No steps recorded

    View test on GitHub ↗
  980. ✓test_vision_anthropicvision/test_anthropic.py›

    No steps recorded

    View test on GitHub ↗
  981. ✓test_vision_azurevision/test_azure.py›

    No steps recorded

    View test on GitHub ↗
  982. ✓test_vision_bedrock_conversevision/test_bedrock_converse.py›

    No steps recorded

    View test on GitHub ↗
  983. ✓test_vision_bedrock_invokevision/test_bedrock_invoke.py›

    No steps recorded

    View test on GitHub ↗
  984. ✓test_vision_vertex_aivision/test_vertex_ai.py›

    No steps recorded

    View test on GitHub ↗
  985. ✓test_web_search_anthropicweb_search/test_anthropic.py›

    No steps recorded

    View test on GitHub ↗
  986. ✓test_web_search_azureweb_search/test_azure.py›

    No steps recorded

    View test on GitHub ↗
  987. ✓test_web_search_bedrock_converseweb_search/test_bedrock_converse.py›

    No steps recorded

    View test on GitHub ↗
  988. ✓test_web_search_bedrock_invokeweb_search/test_bedrock_invoke.py›

    No steps recorded

    View test on GitHub ↗
  989. ✓test_web_search_vertex_aiweb_search/test_vertex_ai.py›

    No steps recorded

    View test on GitHub ↗
  990. −test_basic_messaging_non_streaming_bedrock_mantlebasic_messaging_non_streaming/test_bedrock_mantle.py›

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  991. −test_basic_messaging_non_streaming_openaibasic_messaging_non_streaming/test_openai.py›

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  992. −test_basic_messaging_streaming_bedrock_mantlebasic_messaging_streaming/test_bedrock_mantle.py›

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  993. −test_basic_messaging_streaming_openaibasic_messaging_streaming/test_openai.py›

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  994. −test_count_tokens_vertex_aicount_tokens/test_vertex_ai.py›

    No steps recorded

    stage red: Vertex returns not supported for token counting for Claude aliases

    View test on GitHub ↗
  995. −test_long_context_1m_anthropiclong_context_1m/test_anthropic.py›

    No steps recorded

    stage red: 1M long_context not green on stage Anthropic path yet (200k sonnet / model alias)

    View test on GitHub ↗
  996. −test_long_context_1m_azurelong_context_1m/test_azure.py›

    No steps recorded

    stage red: 1M long_context not green on stage Azure Foundry deployments yet

    View test on GitHub ↗
  997. −test_long_context_1m_bedrock_converselong_context_1m/test_bedrock_converse.py›

    No steps recorded

    stage red: 1M long_context not green on stage Bedrock Converse deployments yet

    View test on GitHub ↗
  998. −test_long_context_1m_bedrock_invokelong_context_1m/test_bedrock_invoke.py›

    No steps recorded

    stage red: 1M long_context not green on stage Bedrock Invoke deployments yet

    View test on GitHub ↗
  999. −test_long_context_1m_vertex_ailong_context_1m/test_vertex_ai.py›

    No steps recorded

    stage red: 1M long_context not green on stage Vertex deployments yet

    View test on GitHub ↗
  1000. −test_passthrough_azurepassthrough/test_azure.py›

    No steps recorded

    stage red: /azure passthrough drops client headers (e.g. anthropic-version); product gap

    View test on GitHub ↗
  1001. −test_tool_search_azuretool_search/test_azure.py›

    No steps recorded

    stage red: Azure Foundry tool_search_server not supported in workspace for probed models

    View test on GitHub ↗
  1002. −test_tool_search_vertex_aitool_search/test_vertex_ai.py›

    No steps recorded

    stage red: Vertex rejects tool_search when deployment extra_headers inject context-1m beta; product/config

    View test on GitHub ↗
  1003. −test_tool_use_bedrock_mantletool_use/test_bedrock_mantle.py›

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  1004. −test_tool_use_openaitool_use/test_openai.py›

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  1005. −test_tool_use_streaming_bedrock_mantletool_use_streaming/test_bedrock_mantle.py›

    No steps recorded

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

    View test on GitHub ↗
  1006. −test_tool_use_streaming_openaitool_use_streaming/test_openai.py›

    No steps recorded

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

    View test on GitHub ↗
  1007. ✓test_a_success_status_without_a_deletion_confirmation_is_rejected[bGl0ZWxsbV9wcm94eTtmaWxlLTE=] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1008. ✓test_a_success_status_without_a_deletion_confirmation_is_rejected[file-1] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1009. ✓test_accepted_cancellation_waits_through_stale_provider_status[in_progress-batch-1] TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1010. ✓test_accepted_cancellation_waits_through_stale_provider_status[in_progress-bGl0ZWxsbV9wcm94eTtiYXRjaC0x] TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1011. ✓test_accepted_cancellation_waits_through_stale_provider_status[validating-batch-1] TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1012. ✓test_accepted_cancellation_waits_through_stale_provider_status[validating-bGl0ZWxsbV9wcm94eTtiYXRjaC0x] TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1013. ✓test_active_batch_is_cancelled_through_its_provider TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1014. ✓test_any_other_delete_failure_still_raises[failure0] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1015. ✓test_any_other_delete_failure_still_raises[failure1] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1016. ✓test_any_other_delete_failure_still_raises[failure2] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1017. ✓test_anything_but_still_cancelling_at_the_deadline_still_fails[last0-did not finish within 120.0s, last status in_progress] TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1018. ✓test_anything_but_still_cancelling_at_the_deadline_still_fails[last1-after cancellation failed: HTTP 403] TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1019. ✓test_azure_form_serializes_native_expiry_for_the_proxy TestAzureFileExpirytest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1020. ✓test_batch_create_over_rpm_returns_mapped_429 TestBatchRateLimitErrorMappingtest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-batch-rl-map-c375d15407d8, tpm limit: 1000000 and rpm limit: 2
    View test on GitHub ↗
  1021. ✓test_batch_key_model_access_deniedtest_batches_e2e.py›
    1. Generate a virtual key with models: openai-batch-ac49dd126388 and user id: e2e-test-user
    View test on GitHub ↗
  1022. ✓test_batch_lifecycle[azure-encoded]test_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1023. ✓test_batch_lifecycle[azure-model_param]test_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1024. ✓test_batch_lifecycle[azure-provider_fallback]test_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1025. ✓test_batch_lifecycle[azure-unified]test_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1026. ✓test_batch_lifecycle[bedrock-unified]test_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1027. ✓test_batch_lifecycle[openai-encoded]test_batches_e2e.py›
    1. Call the management route /health/liveliness
    2. Add a deployment named openai-batch-ac49dd126388 that calls openai/gpt-4o-mini
    3. Add a deployment named azure-batch-ac49dd126388 that calls azure/gpt-5.4-mini-batch
    4. Add a deployment named vertex-batch-ac49dd126388 that calls vertex_ai/gemini-2.5-flash
    5. Add a deployment named bedrock-batch-ac49dd126388 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    6. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1028. ✓test_batch_lifecycle[openai-model_param]test_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1029. ✓test_batch_lifecycle[openai-provider_fallback]test_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1030. ✓test_batch_lifecycle[openai-unified]test_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1031. ✓test_batch_lifecycle[vertex_ai-encoded]test_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1032. ✓test_batch_lifecycle[vertex_ai-model_param]test_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1033. ✓test_batch_lifecycle[vertex_ai-provider_fallback]test_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1034. ✓test_batch_lifecycle[vertex_ai-unified]test_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1035. ✓test_batch_still_cancelling_at_the_deadline_and_its_input_file_are_left_and_reported TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1036. ✓test_batch_that_completed_before_cleanup_deletes_output_and_error_files[False] TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1037. ✓test_batch_that_completed_before_cleanup_deletes_output_and_error_files[True] TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1038. ✓test_cancellation_conflict_is_accepted_only_when_batch_became_inactive[completed] TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1039. ✓test_cancellation_conflict_is_accepted_only_when_batch_became_inactive[in_progress] TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1040. ✓test_cancelling_batch_is_polled_until_terminal_without_cancelling_again TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1041. ✓test_cleanup_is_idempotent_when_file_is_already_deleted TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1042. ✓test_completed_batch_downloads_output_and_books_cost TestBatchTerminalStatetest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1043. ✓test_default_resource_cleanup_keeps_existing_best_effort_behavior TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1044. ✓test_delete_refused_because_a_batch_still_references_the_file_is_left_and_reported TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1045. ✓test_deletes_raw_files_through_the_upload_provider[azure-encoded] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1046. ✓test_deletes_raw_files_through_the_upload_provider[azure-model_param] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1047. ✓test_deletes_raw_files_through_the_upload_provider[azure-provider_fallback] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1048. ✓test_deletes_raw_files_through_the_upload_provider[azure-unified] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1049. ✓test_deletes_raw_files_through_the_upload_provider[bedrock-unified] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1050. ✓test_deletes_raw_files_through_the_upload_provider[openai-encoded] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1051. ✓test_deletes_raw_files_through_the_upload_provider[openai-model_param] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1052. ✓test_deletes_raw_files_through_the_upload_provider[openai-provider_fallback] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1053. ✓test_deletes_raw_files_through_the_upload_provider[openai-unified] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1054. ✓test_deletes_raw_files_through_the_upload_provider[vertex_ai-encoded] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1055. ✓test_deletes_raw_files_through_the_upload_provider[vertex_ai-model_param] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1056. ✓test_deletes_raw_files_through_the_upload_provider[vertex_ai-provider_fallback] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1057. ✓test_deletes_raw_files_through_the_upload_provider[vertex_ai-unified] TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1058. ✓test_endpoint_mismatch_fails_batch_and_cancel_conflicts TestBatchFailurePathstest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  1059. ✓test_enqueued_allowance_accepts_batch_over_key_rpm TestBatchEnqueuedTokenLimittest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-batch-enq-rpm-fc7cffca1bd7, rpm limit: 2 and metadata: batch enqueued token limit: 100000
    View test on GitHub ↗
  1060. ✓test_exhausted_allowance_blocks_until_cancel_refunds TestBatchEnqueuedTokenLimittest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-batch-enq-size-45c79e3a65c9 and metadata: batch enqueued token limit: 1
    2. Generate a virtual key with user id: e2e-batch-enq-refund-7f2f4b880dea and metadata: batch enqueued token limit: 36
    View test on GitHub ↗
  1061. ✓test_failed_delete_is_reported_after_remaining_resources_are_cleaned TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1062. ✓test_file_content_matches_upload TestBatchFileContenttest_batches_e2e.py›
    1. Add a deployment named e2e-file-content-1b7470be6eb0 that calls openai/gpt-4o-mini
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1063. ✓test_file_lifecycle_signs_s3_with_s3_credentials TestBedrockBatchSplitS3Credentialstest_batches_e2e.py›
    1. Add a deployment named bedrock-split-s3-batch-ac49dd126388 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1064. ✓test_file_upload_and_delete_outputstest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1065. ✓test_foreign_encoded_file_id_routes_by_file_model TestBatchFailurePathstest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1066. ✓test_gemini_file_upload TestGeminiFilestest_batches_e2e.py›
    1. Add a deployment named gemini-files-ac49dd126388 that calls gemini/gemini-2.5-flash
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1067. ✓test_inactive_batch_needs_no_cancellation[cancelled] TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1068. ✓test_inactive_batch_needs_no_cancellation[completed] TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1069. ✓test_inactive_batch_needs_no_cancellation[expired] TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1070. ✓test_inactive_batch_needs_no_cancellation[failed] TestBatchCancellationtest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1071. ✓test_list_page_cursors_address_only_the_callers_own_files TestOpenAIFilestest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-file-list-4e678914cf11
    View test on GitHub ↗
  1072. ✓test_malformed_jsonl_upload_rejected TestBatchFailurePathstest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1073. ✓test_managed_delete_accepts_the_deleted_file_object TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1074. ✓test_native_jsonl_round_trips_untouched_and_starts_a_batch TestVertexNativePassthroughtest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1075. ✓test_other_providers_keep_their_existing_upload_fields[bedrock] TestAzureFileExpirytest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1076. ✓test_other_providers_keep_their_existing_upload_fields[openai] TestAzureFileExpirytest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1077. ✓test_other_providers_keep_their_existing_upload_fields[vertex_ai] TestAzureFileExpirytest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1078. ✓test_passthrough_upload_is_rejected_outside_a_native_vertex_batch[no-target-model] TestVertexNativePassthroughtest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1079. ✓test_passthrough_upload_is_rejected_outside_a_native_vertex_batch[non-vertex-target-model] TestVertexNativePassthroughtest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1080. ✓test_passthrough_upload_is_rejected_outside_a_native_vertex_batch[openai-shaped-rows] TestVertexNativePassthroughtest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1081. ✓test_permanent_error_is_not_retried TestCleanupRetriestest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1082. ✓test_persistent_error_has_bounded_retries TestCleanupRetriestest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1083. ✓test_retrieve_round_trips_metadata TestOpenAIFilestest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1084. ✓test_success_response_must_confirm_deletion TestFileCleanuptest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1085. ✓test_transient_error_retries_and_returns_success[failure0] TestCleanupRetriestest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1086. ✓test_transient_error_retries_and_returns_success[failure1] TestCleanupRetriestest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1087. ✓test_transient_error_retries_and_returns_success[failure2] TestCleanupRetriestest_batch_cleanup.py›

    No steps recorded

    View test on GitHub ↗
  1088. ✓test_unified_batch_create_with_assume_role TestBedrockBatchAssumeRoletest_batches_e2e.py›
    1. Add a deployment named bedrock-sts-batch-ac49dd126388 that calls bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1089. ✓test_unified_create_and_retrieve_via_chained_gateway TestBatchSecondHoptest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named openai-batch-hop-ac49dd126388 that calls litellm_proxy/openai-batch-ac49dd126388
    View test on GitHub ↗
  1090. ✓test_unified_file_content_downloads[azure] TestBatchFileContenttest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1091. ✓test_unified_file_content_downloads[bedrock] TestBatchFileContenttest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1092. ✓test_unified_file_content_downloads[vertex_ai] TestBatchFileContenttest_batches_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1093. ✓test_unified_file_upload_and_batch_create_in_govcloud TestBedrockBatchGovCloudtest_batches_e2e.py›
    1. Add a deployment named bedrock-govcloud-batch-ac49dd126388 that calls bedrock/amazon.nova-lite-v1:0
    2. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1094. −test_batch_runs_to_completion_with_a_downloadable_output[model_query] TestHostedVllmBatchtest_batches_e2e.py›

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  1095. −test_batch_runs_to_completion_with_a_downloadable_output[target_model_names] TestHostedVllmBatchtest_batches_e2e.py›

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  1096. −test_failing_line_lands_in_the_error_file_not_the_batch_status TestHostedVllmBatchtest_batches_e2e.py›

    No steps recorded

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

    View test on GitHub ↗
  1097. −test_rate_limited_batch_create_leaves_no_unattributed_spend_rowtest_batches_e2e.py›

    No steps recorded

    LIT-5027: the path under test hangs. The batch rate limiter reads the input file to count tokens by awaiting litellm.afile_content with no timeout, so a slow Files API holds POST /v1/batches open past any client deadline (63.6s observed on stage against a 60s read timeout). The unattributed-spend-row contract below is never reached, so the test reports a timeout rather than the behavior it guards. Unskip once the fetch is bounded.

    View test on GitHub ↗
  1098. −test_uploaded_file_appears_in_list TestOpenAIFilestest_batches_e2e.py›

    No steps recorded

    LIT-4820 (https://linear.app/litellm-ai/issue/LIT-4820): GET /v1/files omits newly uploaded files. The upload succeeds and GET /v1/files/{id} returns the file, but it never appears in the listing; the returned set is stable with its newest entry ~10h old, on both the managed (/v1/files?model=) and provider-scoped (/openai/v1/files) routes. Skipped rather than weakened because the assertion below is the correct contract. Remove this marker when LIT-4820 is fixed; do not relax the assertion to make it pass.

    View test on GitHub ↗
  1099. ✓test_a_body_that_does_not_match_its_declared_boundary_stays_opaque TestMultipartIdentitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1100. ✓test_a_chunk_split_inside_an_event_survives_replay TestStreamingFidelitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1101. ✓test_a_chunked_non_sse_response_stays_buffered TestStreamingFidelitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1102. ✓test_a_declared_part_content_type_separates_otherwise_identical_uploads TestMultipartIdentityEdgestest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1103. ✓test_a_downstream_disconnect_mid_relay_records_only_the_delivered_chunks TestStreamingFidelitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1104. ✓test_a_file_part_without_a_filename_is_not_mistaken_for_a_plain_field TestMultipartIdentityEdgestest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1105. ✓test_a_filename_carrying_a_per_run_marker_keys_the_same_next_run TestMultipartIdentityEdgestest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1106. ✓test_a_fresh_boundary_replays_the_same_upload TestMultipartIdentitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1107. ✓test_a_length_change_the_canonicalizer_absorbs_does_not_move_the_key TestMultipartIdentityEdgestest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1108. ✓test_a_mid_frame_cut_tears_a_data_line_whose_marker_is_split_across_chunks TestStreamCuttest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1109. ✓test_a_mid_stream_upstream_failure_records_the_delivered_chunks_and_the_truncation TestStreamingFidelitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1110. ✓test_a_non_streamed_response_keeps_the_buffered_shape TestStreamingFidelitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1111. ✓test_a_redacted_field_still_matches_the_live_request_that_carried_the_secret TestMultipartIdentityEdgestest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1112. ✓test_a_repeated_field_cannot_collide_with_a_literal_indexed_name TestMultipartIdentityEdgestest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1113. ✓test_a_secret_named_field_never_reaches_the_stored_request TestMultipartIdentityEdgestest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1114. ✓test_a_separator_inside_a_filename_cannot_forge_a_different_split TestMultipartIdentityEdgestest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1115. ✓test_a_streamed_response_records_its_chunk_boundaries TestStreamingFidelitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1116. ✓test_a_structurally_different_upload_misses[extra-file-part] TestMultipartIdentitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1117. ✓test_a_structurally_different_upload_misses[file-content] TestMultipartIdentitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1118. ✓test_a_structurally_different_upload_misses[file-name] TestMultipartIdentitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1119. ✓test_a_structurally_different_upload_misses[form-field] TestMultipartIdentitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1120. ✓test_a_structurally_different_upload_misses[repeated-form-field] TestMultipartIdentitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1121. ✓test_a_truncated_recording_replays_as_a_truncated_stream TestStreamingFidelitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1122. ✓test_an_empty_declared_boundary_falls_back_instead_of_splitting_on_dashes TestMultipartIdentityEdgestest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1123. ✓test_bedrock_never_wires_a_bundle_because_the_edge_cannot_sign_into_one[record] TestApiBaseSeamtest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1124. ✓test_bedrock_never_wires_a_bundle_because_the_edge_cannot_sign_into_one[replay] TestApiBaseSeamtest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1125. ✓test_concurrent_streams_each_record_their_own_chunks TestStreamingFidelitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1126. ✓test_content_drift_returns_the_miss_status_naming_both_keys TestReplayModetest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1127. ✓test_exhausted_key_returns_the_miss_status TestReplayModetest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1128. ✓test_failed_provider_attempt_is_counted TestProviderRequestObservationtest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1129. ✓test_forwards_to_the_provider_and_writes_one_interaction_file TestRecordModetest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1130. ✓test_fully_consumed_recording_leaves_nothing TestReplayLeftovertest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1131. ✓test_identical_requests_replay_their_responses_in_recorded_order TestReplayModetest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1132. ✓test_inert_outside_replay_mode TestReplayLeftovertest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1133. ✓test_invalid_mode_raises_naming_the_value TestApiBaseSeamtest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1134. ✓test_live_counts_repeated_marker_calls_without_recording TestProviderRequestObservationtest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1135. ✓test_live_mode_returns_none TestApiBaseSeamtest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1136. ✓test_never_stores_headers_so_credentials_never_touch_disk TestRecordModetest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1137. ✓test_non_json_bodies_match_by_canonical_digest_without_storing_them TestReplayModetest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1138. ✓test_parallel_identical_calls_serve_each_recording_exactly_once TestConcurrentReplaytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1139. ✓test_partially_consumed_recording_names_the_leftover TestReplayLeftovertest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1140. ✓test_query_params_are_part_of_the_identity TestReplayModetest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1141. ✓test_record_and_replay_count_each_matching_call TestProviderRequestObservationtest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1142. ✓test_record_mode_boots_one_shared_edge_and_prepares_the_bundle TestApiBaseSeamtest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1143. ✓test_record_mode_relays_the_stream_chunked_like_replay_will TestStreamingFidelitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1144. ✓test_replay_of_a_stream_makes_no_provider_connection TestStreamingFidelitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1145. ✓test_replay_reproduces_the_recorded_split_points TestStreamingFidelitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1146. ✓test_replay_serves_a_directly_recorded_interaction TestHandleEdgeRequestPuretest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1147. ✓test_request_identity_ignores_auth_headers TestReplayModetest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1148. ✓test_serves_recorded_bytes_with_zero_provider_hits TestReplayModetest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1149. ✓test_several_file_parts_separate_when_their_contents_swap TestMultipartIdentitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1150. ✓test_strips_volatile_response_headers_and_serves_the_filtered_copy TestRecordModetest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1151. ✓test_test_without_recordings_has_no_leftover TestReplayLeftovertest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1152. ✓test_the_boundary_parameter_is_read_the_way_the_client_meant_it[lookalike-parameter] TestMultipartIdentityEdgestest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1153. ✓test_the_boundary_parameter_is_read_the_way_the_client_meant_it[uppercase-parameter] TestMultipartIdentityEdgestest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1154. ✓test_the_stored_request_carries_fields_and_file_identity_but_no_secrets TestMultipartIdentitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1155. ✓test_the_usage_chunk_replays_in_its_recorded_position TestStreamingFidelitytest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1156. ✓test_two_binary_field_values_of_one_length_stay_apart TestMultipartIdentityEdgestest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1157. ✓test_unknown_mount_404s_naming_the_known_mounts TestHandleEdgeRequestPuretest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1158. ✓test_unknown_mount_raises_naming_the_known_mounts TestApiBaseSeamtest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1159. ✓test_unreachable_provider_records_and_serves_a_502 TestRecordModetest_provider_edge.py›

    No steps recorded

    View test on GitHub ↗
  1160. ✓test_429_honors_server_reset_and_preserves_duplicate_eventstest_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1161. ✓test_all_attempts_failed_is_a_failure_not_a_passtest_span_selection.py›

    No steps recorded

    View test on GitHub ↗
  1162. ✓test_chat_completions_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-dd-chat-ffac33131355 and key alias: dd-chat-ffac33131355
    View test on GitHub ↗
  1163. ✓test_chat_completions_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-otel-trace-chat-73ca29caf3b0 and key alias: otel-trace-chat-73ca29caf3b0
    View test on GitHub ↗
  1164. ✓test_chat_completions_failure_writes_one_object TestS3LogDeliverytest_s3_log_e2e.py›
    1. Call the management route /health/readiness/details
    2. Add a deployment named s3-err-cb1995c48f32 that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key with models: s3-err-cb1995c48f32, user id: e2e-s3-err-key-39a089912000 and key alias: s3-err-key-39a089912000
    View test on GitHub ↗
  1165. ✓test_chat_completions_object_key_follows_the_partition_granularity TestS3LogDeliverytest_s3_log_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-s3-layout-75a4fa4656e6 and key alias: s3-layout-75a4fa4656e6
    View test on GitHub ↗
  1166. ✓test_chat_completions_stream_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-dd-stream-chat-1282c9a44274 and key alias: dd-stream-chat-1282c9a44274
    3. Send a streaming /chat/completions request to claude-haiku-4-5
    4. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1167. ✓test_chat_completions_stream_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-otel-stream-chat-cf9c1eeecbce and key alias: otel-stream-chat-cf9c1eeecbce
    3. Send a streaming /chat/completions request to claude-haiku-4-5
    View test on GitHub ↗
  1168. ✓test_chat_completions_stream_records_real_ttft TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-otel-ttft-chat-76ca14db76b0 and key alias: otel-ttft-chat-76ca14db76b0
    3. Send a streaming /chat/completions request to claude-haiku-4-5
    View test on GitHub ↗
  1169. ✓test_chat_completions_writes_one_success_object TestS3LogDeliverytest_s3_log_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-s3-chat-429fdf86818c and key alias: s3-chat-429fdf86818c
    View test on GitHub ↗
  1170. ✓test_chat_completions_writes_one_success_record TestGcsLogDeliverytest_gcs_log_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-gcs-chat-5f41072e409c and key alias: gcs-chat-5f41072e409c
    View test on GitHub ↗
  1171. ✓test_distinct_key_aliases_produce_distinct_series TestPrometheusPerKeyCardinalitytest_prometheus_cardinality_e2e.py›
    1. Generate a virtual key with models: gemini-2.5-flash, user id: e2e-e2e-prom-8864d3b86b07 and key alias: e2e-prom-8864d3b86b07
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Generate a virtual key with models: gemini-2.5-flash, user id: e2e-e2e-prom-69716d908cee and key alias: e2e-prom-69716d908cee
    4. Send a /chat/completions request to gemini-2.5-flash
    5. Generate a virtual key with models: gemini-2.5-flash, user id: e2e-e2e-prom-7ebab3c2440b and key alias: e2e-prom-7ebab3c2440b
    6. Send a /chat/completions request to gemini-2.5-flash
    7. Call the management route /metrics
    View test on GitHub ↗
  1172. ✓test_empty_polling_does_not_start_a_final_search_after_its_deadlinetest_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1173. ✓test_failed_chat_completions_emits_one_error_event TestDataDogFailureDeliverytest_datadog_log_e2e.py›
    1. Call the management route /health/readiness/details
    2. Add a deployment named dd-err-e4f2e6035a9e that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key with models: dd-err-e4f2e6035a9e, user id: e2e-dd-err-key-b8b3a9f1787d and key alias: dd-err-key-b8b3a9f1787d
    View test on GitHub ↗
  1174. ✓test_failed_chat_completions_error_span_attributes TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    1. Call the management route /health/readiness/details
    2. Add a deployment named otel-err-e5e292f20c01 that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key with models: otel-err-e5e292f20c01, user id: e2e-otel-err-364d5ba4c5e3 and key alias: otel-err-364d5ba4c5e3
    View test on GitHub ↗
  1175. ✓test_failed_messages_error_span_attributes TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    1. Call the management route /health/readiness/details
    2. Add a deployment named otel-err-672e7c56a92c that calls anthropic/claude-haiku-4-5
    3. Generate a virtual key with models: otel-err-672e7c56a92c, user id: e2e-otel-err-b225eece1697 and key alias: otel-err-b225eece1697
    View test on GitHub ↗
  1176. ✓test_failure_diagnostics_hide_credentials_without_changing_auth_headerstest_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1177. ✓test_invalid_reset_uses_search_interval[-1]test_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1178. ✓test_invalid_reset_uses_search_interval[]test_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1179. ✓test_invalid_reset_uses_search_interval[inf]test_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1180. ✓test_invalid_reset_uses_search_interval[invalid]test_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1181. ✓test_invalid_reset_uses_search_interval[nan]test_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1182. ✓test_late_delivery_cannot_pass_without_a_complete_settle_windowtest_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1183. ✓test_late_retry_cannot_receive_a_fresh_request_timeouttest_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1184. ✓test_messages_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-dd-messages-5ff4818d1fb2 and key alias: dd-messages-5ff4818d1fb2
    View test on GitHub ↗
  1185. ✓test_messages_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-otel-trace-messages-52f3e29b7b4b and key alias: otel-trace-messages-52f3e29b7b4b
    View test on GitHub ↗
  1186. ✓test_messages_stream_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-dd-stream-messages-d737c21207ae and key alias: dd-stream-messages-d737c21207ae
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1187. ✓test_messages_stream_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-otel-stream-messages-95566764df9b and key alias: otel-stream-messages-95566764df9b
    View test on GitHub ↗
  1188. ✓test_messages_stream_records_real_ttft TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-otel-ttft-messages-3ca2aabd3085 and key alias: otel-ttft-messages-3ca2aabd3085
    View test on GitHub ↗
  1189. ✓test_non_json_native_metadata_reaches_langsmith TestLangsmithBatchSerializationtest_langsmith_batch_serialization_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1190. ✓test_non_quota_failures_are_not_retried_or_treated_as_empty_results[-1]test_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1191. ✓test_non_quota_failures_are_not_retried_or_treated_as_empty_results[401]test_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1192. ✓test_non_quota_failures_are_not_retried_or_treated_as_empty_results[403]test_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1193. ✓test_non_quota_failures_are_not_retried_or_treated_as_empty_results[500]test_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1194. ✓test_other_operations_are_not_countedtest_span_selection.py›

    No steps recorded

    View test on GitHub ↗
  1195. ✓test_polling_quota_retries_share_the_original_deadlinetest_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1196. ✓test_queue_time_histogram_records_an_observation TestPrometheusRequestQueueTimetest_prometheus_queue_time_e2e.py›
    1. Generate a virtual key with models: gemini-2.5-flash, user id: e2e-e2e-queue-time-d89f27228549 and key alias: e2e-queue-time-d89f27228549
    2. Send a /chat/completions request to gemini-2.5-flash
    3. Call the management route /metrics
    View test on GitHub ↗
  1197. ✓test_rate_limit_wait_stops_at_deadline_without_issuing_another_requesttest_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1198. ✓test_responses_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: gpt-5.5, user id: e2e-dd-responses-2d66138c254c and key alias: dd-responses-2d66138c254c
    View test on GitHub ↗
  1199. ✓test_responses_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: gpt-5.5, user id: e2e-otel-trace-responses-70f4eba58693 and key alias: otel-trace-responses-70f4eba58693
    View test on GitHub ↗
  1200. ✓test_responses_stream_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: gpt-5.5, user id: e2e-dd-stream-responses-afcf4980d5fe and key alias: dd-stream-responses-afcf4980d5fe
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1201. ✓test_responses_stream_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: gpt-5.5, user id: e2e-otel-stream-responses-a0f25089e127 and key alias: otel-stream-responses-a0f25089e127
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1202. ✓test_responses_stream_records_real_ttft TestOtelTraceCompletenesstest_otel_trace_e2e.py›
    1. Call the management route /health/readiness/details
    2. Generate a virtual key with models: gpt-5.5, user id: e2e-otel-ttft-responses-73b737cc3ea8 and key alias: otel-ttft-responses-73b737cc3ea8
    View test on GitHub ↗
  1203. ✓test_retried_attempt_span_is_excludedtest_span_selection.py›

    No steps recorded

    View test on GitHub ↗
  1204. ✓test_retry_after_is_not_shortened_by_an_earlier_resettest_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1205. ✓test_served_span_is_the_only_one_when_nothing_was_retriedtest_span_selection.py›

    No steps recorded

    View test on GitHub ↗
  1206. ✓test_settlement_detects_a_duplicate_on_the_final_searchtest_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1207. ✓test_settlement_keeps_confirmed_events_through_empty_searchestest_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1208. ✓test_settlement_quota_retries_keep_the_remaining_readback_budgettest_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1209. ✓test_several_failed_attempts_still_leave_one_served_spantest_span_selection.py›

    No steps recorded

    View test on GitHub ↗
  1210. ✓test_team_callback_delivers_and_isolates TestTeamLangfuseCallbacktest_team_langfuse_callback_e2e.py›
    1. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-lf-team-key-5b60ea16c547, team id: 041cd6c1-e7e0-4f95-9697-27f248b35484 and key alias: lf-team-key-5b60ea16c547
    2. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-lf-solo-key-0f77fafef528 and key alias: lf-solo-key-0f77fafef528
    View test on GitHub ↗
  1211. ✓test_two_served_spans_still_failtest_span_selection.py›

    No steps recorded

    View test on GitHub ↗
  1212. ✓test_zero_reset_cannot_create_a_busy_retry_looptest_datadog_reader.py›

    No steps recorded

    View test on GitHub ↗
  1213. −test_chat_completions_delivers_one_call_with_spend TestWeaveLogDeliverytest_weave_log_e2e.py›

    No steps recorded

    stage red: product gap, key-scoped weave_otel spans are not delivered when the OTEL v2 callback is active

    View test on GitHub ↗
  1214. −test_failed_chat_completions_delivers_one_error_call TestWeaveLogDeliverytest_weave_log_e2e.py›

    No steps recorded

    stage red: product gap, key-scoped weave_otel spans are not delivered when the OTEL v2 callback is active

    View test on GitHub ↗
  1215. ✓test_a_flat_ceiling_fails_a_degraded_phase_that_is_cheaper_than_its_baseline TestAbsoluteBudgettest_phase_budget.py›

    No steps recorded

    View test on GitHub ↗
  1216. ✓test_a_missing_baseline_is_a_violation_rather_than_a_silent_pass TestRatioBudgettest_phase_budget.py›

    No steps recorded

    View test on GitHub ↗
  1217. ✓test_a_route_that_never_ran_is_absent_so_a_one_sided_run_cannot_pass_unnoticed TestPerEndpointtest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1218. ✓test_a_run_inside_every_budget_reports_nothing TestViolationstest_phase_budget.py›

    No steps recorded

    View test on GitHub ↗
  1219. ✓test_a_run_that_drove_no_traffic_reports_a_total_failure_ratio TestAggregatetest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1220. ✓test_a_run_with_no_failures_writes_no_csv_and_reports_no_errors TestErrorBreakdowntest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1221. ✓test_a_single_sample_has_no_interval_and_reports_zero TestCpuUtilizationtest_proxy_usage.py›

    No steps recorded

    View test on GitHub ↗
  1222. ✓test_a_value_exactly_at_the_ceiling_is_allowed TestAbsoluteBudgettest_phase_budget.py›

    No steps recorded

    View test on GitHub ↗
  1223. ✓test_a_value_past_the_ceiling_reports_the_measurement_and_the_ceiling TestAbsoluteBudgettest_phase_budget.py›

    No steps recorded

    View test on GitHub ↗
  1224. ✓test_a_value_under_the_ceiling_is_not_a_violation TestAbsoluteBudgettest_phase_budget.py›

    No steps recorded

    View test on GitHub ↗
  1225. ✓test_a_zero_measurement_is_not_a_violation TestAbsoluteBudgettest_phase_budget.py›

    No steps recorded

    View test on GitHub ↗
  1226. ✓test_all_planned_turns_completing_reports_zero_failures TestSummarizePlannedTurnstest_session_anomaly.py›

    No steps recorded

    View test on GitHub ↗
  1227. ✓test_an_empty_window_reports_zero_rather_than_indexing_nothing TestRssPercentilestest_proxy_usage.py›

    No steps recorded

    View test on GitHub ↗
  1228. ✓test_an_even_split_takes_the_lower_middle_sample_as_locust_itself_does TestPercentilestest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1229. ✓test_both_budget_shapes_report_together TestViolationstest_phase_budget.py›

    No steps recorded

    View test on GitHub ↗
  1230. ✓test_cost_per_request_separates_runs_that_cores_busy_reports_identically TestCpuUtilizationtest_proxy_usage.py›

    No steps recorded

    View test on GitHub ↗
  1231. ✓test_diagnosis_caps_the_list_and_says_how_many_it_left_out TestErrorBreakdowntest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1232. ✓test_diagnosis_leads_with_the_most_common_error TestErrorBreakdowntest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1233. ✓test_diagnosis_says_so_when_locust_recorded_nothing TestErrorBreakdowntest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1234. ✓test_each_route_keeps_its_own_requests_failures_and_median TestPerEndpointtest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1235. ✓test_every_blown_budget_is_reported_not_just_the_first TestViolationstest_phase_budget.py›

    No steps recorded

    View test on GitHub ↗
  1236. ✓test_exhausted_attempts_return_the_last_failure TestRetriedtest_session_anomaly.py›

    No steps recorded

    View test on GitHub ↗
  1237. ✓test_first_try_success_never_sleeps TestRetriedtest_session_anomaly.py›

    No steps recorded

    View test on GitHub ↗
  1238. ✓test_growth_exactly_at_the_ceiling_is_allowed TestRatioBudgettest_phase_budget.py›

    No steps recorded

    View test on GitHub ↗
  1239. ✓test_growth_past_the_ceiling_reports_both_values_and_the_ratio TestRatioBudgettest_phase_budget.py›

    No steps recorded

    View test on GitHub ↗
  1240. ✓test_growth_within_the_ceiling_is_not_a_violation TestRatioBudgettest_phase_budget.py›

    No steps recorded

    View test on GitHub ↗
  1241. ✓test_locust_failure_rows_become_the_error_breakdown TestErrorBreakdowntest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1242. ✓test_median_is_the_middle_sample_not_the_mean_a_slow_tail_would_drag TestPercentilestest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1243. ✓test_no_requests_reports_zero_cost_rather_than_dividing_by_zero TestCpuUtilizationtest_proxy_usage.py›

    No steps recorded

    View test on GitHub ↗
  1244. ✓test_no_samples_reports_zero_rather_than_dividing_by_an_empty_histogram TestPercentilestest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1245. ✓test_ordinary_locust_chatter_is_not_reported_as_a_warning TestGeneratorSaturationtest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1246. ✓test_partial_total_between_batch_flushes_is_not_accepted_as_final TestSettledSpendtest_session_anomaly.py›

    No steps recorded

    View test on GitHub ↗
  1247. ✓test_percentiles_merge_the_histograms_of_every_stats_entry TestPercentilestest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1248. ✓test_repeated_cpu_warnings_collapse_to_one_and_reach_the_diagnosis TestGeneratorSaturationtest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1249. ✓test_session_aborted_on_first_turn_counts_all_its_planned_turns_as_failed TestSummarizePlannedTurnstest_session_anomaly.py›

    No steps recorded

    View test on GitHub ↗
  1250. ✓test_several_stats_entries_for_one_route_fold_into_a_single_row TestPerEndpointtest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1251. ✓test_shrinking_is_never_a_violation TestRatioBudgettest_phase_budget.py›

    No steps recorded

    View test on GitHub ↗
  1252. ✓test_spend_that_never_becomes_nonzero_raises TestSettledSpendtest_session_anomaly.py›

    No steps recorded

    View test on GitHub ↗
  1253. ✓test_spend_that_never_stabilizes_raises TestSettledSpendtest_session_anomaly.py›

    No steps recorded

    View test on GitHub ↗
  1254. ✓test_summary_reports_every_percentile_in_human_units TestCpuUtilizationtest_proxy_usage.py›

    No steps recorded

    View test on GitHub ↗
  1255. ✓test_tail_percentiles_come_from_the_slow_end_of_the_histogram TestAggregatetest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1256. ✓test_the_summary_names_every_route_with_its_counts TestPerEndpointtest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1257. ✓test_the_tail_percentiles_reach_the_peak_the_median_hides TestRssPercentilestest_proxy_usage.py›

    No steps recorded

    View test on GitHub ↗
  1258. ✓test_the_tail_percentiles_reach_the_slow_samples_the_median_hides TestPercentilestest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1259. ✓test_the_unit_and_decimals_carry_into_the_message TestRatioBudgettest_phase_budget.py›

    No steps recorded

    View test on GitHub ↗
  1260. ✓test_throughput_spans_from_the_earliest_start_when_locust_reports_several_entries TestAggregatetest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1261. ✓test_throughput_spans_the_whole_window_and_latency_comes_from_the_histogram TestAggregatetest_locust_load.py›

    No steps recorded

    View test on GitHub ↗
  1262. ✓test_transient_failures_then_success_returns_the_success TestRetriedtest_session_anomaly.py›

    No steps recorded

    View test on GitHub ↗
  1263. ✓test_utilization_is_the_counter_delta_over_the_interval_not_the_counter_itself TestCpuUtilizationtest_proxy_usage.py›

    No steps recorded

    View test on GitHub ↗
  1264. ✓test_apply_guardrail_blocks_banned_and_allows_clean TestApplyGuardrailEndpointtest_apply_guardrail_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1265. ✓test_bedrock_post_call_blocks_denied_model_output TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  1266. ✓test_bedrock_post_call_blocks_denied_streamed_output_and_passes_clean_streams TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  1267. ✓test_bedrock_pre_call_blocks_harmful_prompt TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  1268. ✓test_bedrock_pre_call_blocks_on_messages TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1269. ✓test_bedrock_pre_call_blocks_on_responses TestBedrockGuardrailtest_bedrock_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1270. ✓test_blocks_execution_request_but_allows_explanation TestBlockCodeExecutionGuardrailtest_block_code_execution_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-blockcode-backend-b3e2742ca003 that calls gemini/gemini-2.5-flash
    3. Send a /chat/completions request to e2e-blockcode-backend-b3e2742ca003
    View test on GitHub ↗
  1271. ✓test_chat_blocks_category TestOpenAIModerationCategoryMatrixtest_openai_moderation_category_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-mod-cat-chat-1652e7e034c4 that calls gemini/gemini-2.5-flash
    3. Send a /chat/completions request to e2e-mod-cat-chat-1652e7e034c4
    View test on GitHub ↗
  1272. ✓test_child_condition_match_applies_child_and_inherited_parent_guardrails TestPolicyInheritedGuardrailtest_policy_inherited_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1273. ✓test_child_condition_miss_still_applies_inherited_parent_guardrail TestPolicyInheritedGuardrailtest_policy_inherited_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1274. ✓test_during_call_blocks_stream_before_first_chunk TestBedrockDuringCallStreamingtest_streaming_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  1275. ✓test_flag_returns_guardrail_information_for_the_guardrail_that_ran TestGuardrailInformationResponsetest_guardrail_information_response_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-guardrail-info-backend-a0c14af8c338 that calls openai/gpt-4.1-mini
    3. Send a /chat/completions request to e2e-guardrail-info-backend-a0c14af8c338
    View test on GitHub ↗
  1276. ✓test_global_guardrail_blocks_key_without_team_opt_out TestTeamDisableGlobalGuardrailtest_team_disable_global_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  1277. ✓test_http_failure_is_not_hidden_by_a_later_success[400]test_guardrails_client.py›

    No steps recorded

    View test on GitHub ↗
  1278. ✓test_http_failure_is_not_hidden_by_a_later_success[401]test_guardrails_client.py›

    No steps recorded

    View test on GitHub ↗
  1279. ✓test_http_failure_is_not_hidden_by_a_later_success[429]test_guardrails_client.py›

    No steps recorded

    View test on GitHub ↗
  1280. ✓test_http_failure_is_not_hidden_by_a_later_success[500]test_guardrails_client.py›

    No steps recorded

    View test on GitHub ↗
  1281. ✓test_key_attached_content_filter_blocks_banned_video_prompt TestKeyAttachedGuardrailOnVideostest_key_guardrail_video_e2e.py›
    1. Generate a virtual key with user id: e2e-guardrails-user and metadata: guardrails: e2e-video-filter-6815a0e04ec5
    2. Add a deployment named e2e-guard-video-41b50929afeb that calls vertex_ai/veo-3.1-fast-generate-001
    3. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  1282. ✓test_masking_run_is_recorded_on_the_spend_log TestPresidioSpendLogRecordtest_presidio_masking_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  1283. ✓test_messages_blocks_category TestOpenAIModerationCategoryMatrixtest_openai_moderation_category_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-mod-cat-msg-b21a2dbf3e2a that calls anthropic/claude-haiku-4-5
    3. Send a /v1/messages request to e2e-mod-cat-msg-b21a2dbf3e2a
    View test on GitHub ↗
  1284. ✓test_missing_exact_guardrail_returns_failure_evidence_at_deadline[]test_guardrails_client.py›

    No steps recorded

    View test on GitHub ↗
  1285. ✓test_missing_exact_guardrail_returns_failure_evidence_at_deadline[global-filter]test_guardrails_client.py›

    No steps recorded

    View test on GitHub ↗
  1286. ✓test_missing_exact_guardrail_returns_failure_evidence_at_deadline[tool-permission-sibling]test_guardrails_client.py›

    No steps recorded

    View test on GitHub ↗
  1287. ✓test_moderation_blocks_flagged_input TestOpenAIModerationGuardrailtest_openai_moderation_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-moderation-backend-95972b6995e5 that calls gemini/gemini-2.5-flash
    3. Send a /chat/completions request to e2e-moderation-backend-95972b6995e5
    View test on GitHub ↗
  1288. ✓test_moderation_blocks_flagged_input_on_messages TestOpenAIModerationGuardrailtest_openai_moderation_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-moderation-msg-backend-2659f87cc3b8 that calls gemini/gemini-2.5-flash
    3. Send a /v1/messages request to e2e-moderation-msg-backend-2659f87cc3b8
    View test on GitHub ↗
  1289. ✓test_non_stream_block_is_schema_valid_json TestResponsesPreCallBlocktest_responses_pre_call_block_stream_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-responses-block-afb5d3daa046 that calls openai/gpt-4.1-mini
    View test on GitHub ↗
  1290. ✓test_post_call_masks_pii_in_model_output TestPresidioPostCallMaskingtest_presidio_masking_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  1291. ✓test_pre_call_allows_permitted_tool TestToolPermissionPreCalltest_tool_permission_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1292. ✓test_pre_call_blocks_tool_outside_the_allow_list TestToolPermissionPreCalltest_tool_permission_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  1293. ✓test_pre_call_masks_pii_on_chat_completions TestPresidioPreCallMaskingtest_presidio_masking_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  1294. ✓test_pre_call_masks_pii_on_messages TestPresidioPreCallMaskingtest_presidio_masking_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /v1/messages request to gemini-2.5-flash
    View test on GitHub ↗
  1295. ✓test_responses_blocks_category TestOpenAIModerationCategoryMatrixtest_openai_moderation_category_matrix_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-mod-cat-resp-556632fedcb2 that calls openai/gpt-4o-mini
    View test on GitHub ↗
  1296. ✓test_spend_log_stores_masked_output_on_anthropic_messages TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1297. ✓test_spend_log_stores_masked_output_on_chat_completions TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1298. ✓test_spend_log_stores_masked_output_on_responses TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1299. ✓test_spend_log_stores_masked_output_on_streaming_anthropic_messages TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to gemini-2.5-flash
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1300. ✓test_spend_log_stores_masked_output_on_streaming_chat_completions TestPresidioSpendLogStoresMaskedOutputtest_presidio_masking_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to gemini-2.5-flash
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1301. ✓test_stream_block_is_sse_with_completed_assistant_message TestResponsesPreCallBlocktest_responses_pre_call_block_stream_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-responses-block-36c9ce2f401f that calls openai/gpt-4.1-mini
    View test on GitHub ↗
  1302. ✓test_team_with_disable_flag_bypasses_global_guardrail TestTeamDisableGlobalGuardrailtest_team_disable_global_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-guardrails-user and team id: 01b1654d-9f1b-482d-8895-fc004210bd6f
    2. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  1303. ✓test_ui_default_scope_masks_a_card_number_the_model_generates_on_chat_completions TestPresidioCreditCardOutputMaskingtest_presidio_masking_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  1304. ✓test_ui_default_scope_masks_a_card_number_the_model_generates_on_streaming_anthropic_messages TestPresidioCreditCardOutputMaskingtest_presidio_masking_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /v1/messages request to gemini-2.5-flash
    View test on GitHub ↗
  1305. ✓test_ui_default_scope_masks_a_card_number_the_model_generates_on_streaming_chat_completions TestPresidioCreditCardOutputMaskingtest_presidio_masking_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a streaming /chat/completions request to gemini-2.5-flash
    View test on GitHub ↗
  1306. ✓test_waits_for_requested_guardrail_after_an_unrelated_global_guardrailtest_guardrails_client.py›

    No steps recorded

    View test on GitHub ↗
  1307. ✓test_without_flag_response_has_no_guardrail_information TestGuardrailInformationResponsetest_guardrail_information_response_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-guardrail-info-backend-526ba59bb1b9 that calls openai/gpt-4.1-mini
    3. Send a /chat/completions request to e2e-guardrail-info-backend-526ba59bb1b9
    View test on GitHub ↗
  1308. −test_request_naming_an_unknown_guardrail_fails_closedtest_guardrail_dispatch_e2e.py›

    No steps recorded

    stage red: product gap, a request naming a guardrail the proxy does not serve is silently served unguarded (200) instead of failing closed

    View test on GitHub ↗
  1309. ✗test_failing_requests_do_not_grow_rss_or_stored_request TestReliabilityMemorytest_reliability_memory_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-memory-83e0276639dc that calls openai/gpt-5.5
    3. Add a deployment named reliability-memory-fb-83e0276639dc that calls openai/gpt-5.5
    4. Wait for the request's spend log in /spend/logs
    5. Read memory usage from /debug/memory/summary on every proxy replica

    AssertionError: proxy RSS grew 59.8 MB over a second batch of 300 failing requests (2 retries each plus a fallback) after an identical warmup batch, past the 48 MB budget: worker pid 1 on e2e-bk-432-litellm-gateway-7d594657f7-8mh5v behind http://e2e-bk-432-router.litellm.svc.cluster.local settled at 680.7 MB warm and 740.5 MB after; failing requests are leaking memory the way the v1.100.0 retry breadcrumbs did assert 59.77999999999997 <= 48.0 + where 59.77999999999997 = WorkerGrowth(warm=RssReading(replica='http://e2e-bk-432-router.litellm.svc.cluster.local', hostname='e2e-bk-432-litellm-gateway-7d594657f7-8mh5v', worker_pid=1, ram_usage_mb=680.7), after=RssReading(replica='http://e2e-bk-432-router.litellm.svc.cluster.local', hostname='e2e-bk-432-litellm-gateway-7d594657f7-8mh5v', worker_pid=1, ram_usage_mb=740.48)).growth_mb

    View test on GitHub ↗
  1310. ✓test_429_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Generate a virtual key with models: gpt-5.5, user id: e2e-test-user and rpm limit: 1
    3. Add a deployment named reliability-retry-429-1e43a633ddc8 that calls openai/gpt-5.5
    View test on GitHub ↗
  1311. ✓test_429_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Generate a virtual key with models: gpt-5.5, user id: e2e-test-user and rpm limit: 1
    3. Add a deployment named reliability-cooldown-429-9ccf06b4dd8d that calls openai/gpt-5.5
    View test on GitHub ↗
  1312. ✓test_5xx_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-5xx-upstream-866f2af5d6ab that calls openai/gpt-5.5
    3. Add a deployment named reliability-retry-5xx-3ccf39d0f462 that calls openai/reliability-5xx-upstream-866f2af5d6ab
    4. Add a deployment named reliability-retry-5xx-3ccf39d0f462 that calls openai/gpt-5.5
    View test on GitHub ↗
  1313. ✓test_5xx_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-fail-3c64c42142de that calls openai/gpt-5.5
    View test on GitHub ↗
  1314. ✓test_5xx_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cooldown-5xx-upstream-faaa4059e5b6 that calls openai/gpt-5.5
    3. Add a deployment named reliability-cooldown-5xx-1aba6e3813c5 that calls openai/reliability-cooldown-5xx-upstream-faaa4059e5b6
    4. Add a deployment named reliability-cooldown-5xx-1aba6e3813c5 that calls openai/gpt-5.5
    View test on GitHub ↗
  1315. ✓test_alias_api_key_never_overrides_the_tier_credential TestAliasParamForwardingtest_auto_router_regressions_e2e.py›
    1. Add a deployment named e2e-cred-tier-08758ee0f177 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named e2e-cred-alias-08758ee0f177 that calls auto_router/complexity_router
    3. Generate a virtual key with models: e2e-cred-alias-08758ee0f177, e2e-cred-tier-08758ee0f177 and user id: e2e-auto-router-regressions
    4. Send a /chat/completions request to e2e-cred-alias-08758ee0f177
    5. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1316. ✓test_auth_failure_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-retry-auth-dedcde221da0 that calls openai/gpt-5.5
    View test on GitHub ↗
  1317. ✓test_auth_failure_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cooldown-auth-b2ba22a38115 that calls openai/gpt-5.5
    View test on GitHub ↗
  1318. ✓test_body_tagged_chat_reaches_the_untagged_tier_after_marker_rewrite TestUntaggedTierDeploymentstest_auto_router_regressions_e2e.py›
    1. Generate a virtual key with models: e2e-autoroute-984870b7c23f, e2e-tier-984870b7c23f, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /chat/completions request to e2e-autoroute-984870b7c23f
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1319. ✓test_body_tagged_chat_routes_through_the_marker_to_its_tier TestTagSplitRoutingtest_auto_router_regressions_e2e.py›
    1. Add a deployment named e2e-autoroute-9dd91555bafd that calls anthropic/claude-sonnet-5
    2. Add a deployment named e2e-tier-9dd91555bafd that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-autoroute-9dd91555bafd that calls auto_router/complexity_router
    4. Generate a virtual key with models: e2e-autoroute-9dd91555bafd, e2e-tier-9dd91555bafd, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    5. Send a /chat/completions request to e2e-autoroute-9dd91555bafd
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1320. ✓test_body_tagged_responses_with_list_input_routes_to_the_tier TestResponsesApiTagRoutingtest_auto_router_regressions_e2e.py›
    1. Generate a virtual key with models: e2e-autoroute-05a2f5f10f91, e2e-tier-05a2f5f10f91, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1321. ✓test_cached_conversation_stays_on_deployment_holding_its_cache TestReliabilityPromptCachingAffinitytest_reliability_prompt_caching_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Read the router settings from /router/settings
    3. Add a deployment named reliability-cache-d1f29946d146 that calls anthropic/claude-haiku-4-5
    4. Add a deployment named reliability-cache-d1f29946d146 that calls openai/gpt-5.5
    View test on GitHub ↗
  1322. ✓test_client_hanging_up_never_benches_the_deployment TestReliabilityCancelOnDisconnecttest_reliability_cancel_on_disconnect_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Check whether the general setting cancel_on_disconnect is on
    3. Add a deployment named reliability-cooldown-disconnect-a8de17ed7c92 that calls azure/gpt-5.4-nano
    4. Add a deployment named reliability-cooldown-disconnect-a8de17ed7c92 that calls openai/gpt-5.5
    View test on GitHub ↗
  1323. ✓test_content_policy_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-policyfail-f1c91da9e3c7 that calls azure/gpt-5.4-nano
    View test on GitHub ↗
  1324. ✓test_context_window_refusal_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-retry-context-05402ba00673 that calls openai/gpt-3.5-turbo
    3. Add a deployment named reliability-retry-context-05402ba00673 that calls openai/gpt-5.5
    View test on GitHub ↗
  1325. ✓test_context_window_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-ctxfail-da783fd61c23 that calls openai/gpt-3.5-turbo
    View test on GitHub ↗
  1326. ✓test_cost_based_picks_cheapest_deployment TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cost-689f5ac1e1cb that calls openai/gpt-5.5
    View test on GitHub ↗
  1327. ✓test_exact_cache_returns_cached TestReliabilityCachetest_reliability_cache_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named e2e-cache-73f6d6fd3c97 that calls openai/gpt-5.6
    View test on GitHub ↗
  1328. ✓test_header_tagged_messages_routes_through_the_marker_to_an_untagged_tier TestUntaggedTierDeploymentstest_auto_router_regressions_e2e.py›
    1. Add a deployment named e2e-autoroute-984870b7c23f that calls auto_router/complexity_router
    2. Add a deployment named e2e-tier-984870b7c23f that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-autoroute-984870b7c23f that calls anthropic/claude-sonnet-5
    4. Generate a virtual key with models: e2e-autoroute-984870b7c23f, e2e-tier-984870b7c23f, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    5. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1329. ✓test_header_tagged_responses_with_string_input_routes_to_the_tier TestResponsesApiTagRoutingtest_auto_router_regressions_e2e.py›
    1. Add a deployment named e2e-autoroute-05a2f5f10f91 that calls anthropic/claude-sonnet-5
    2. Add a deployment named e2e-tier-05a2f5f10f91 that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-autoroute-05a2f5f10f91 that calls auto_router/complexity_router
    4. Generate a virtual key with models: e2e-autoroute-05a2f5f10f91, e2e-tier-05a2f5f10f91, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    5. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1330. ✓test_request_timeout_exceeds_deadline TestReliabilityTimeoutstest_reliability_timeouts_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-timeout-1ffae99db7df that calls openai/gpt-5.5
    View test on GitHub ↗
  1331. ✓test_responses_input_reaches_the_semantic_auto_router TestSemanticAutoRouterResponsestest_auto_router_regressions_e2e.py›
    1. Add a deployment named e2e-semantic-embedding-1ac6ed8d026f that calls openai/text-embedding-3-small
    2. Add a deployment named e2e-semantic-target-1ac6ed8d026f that calls anthropic/claude-haiku-4-5
    3. Add a deployment named e2e-semantic-fallback-1ac6ed8d026f that calls anthropic/claude-sonnet-5
    4. Add a deployment named e2e-semantic-router-1ac6ed8d026f that calls auto_router/e2e-semantic-router-1ac6ed8d026f
    5. Generate a virtual key with models: e2e-semantic-router-1ac6ed8d026f, e2e-semantic-target-1ac6ed8d026f, e2e-semantic-fallback-1ac6ed8d026f and user id: e2e-auto-router-regressions
    6. Wait for at least 2 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1332. ✓test_simple_shuffle_honors_weights TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-shuffle-f4ade071ad16 that calls openai/gpt-5.5
    View test on GitHub ↗
  1333. ✓test_stream_timeout_exceeds_deadline TestReliabilityTimeoutstest_reliability_timeouts_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-stream-timeout-0c383110b0d2 that calls openai/gpt-5.5
    View test on GitHub ↗
  1334. ✓test_tagged_call_straight_at_an_untagged_deployment_stays_denied TestUntaggedTierDeploymentstest_auto_router_regressions_e2e.py›
    1. Generate a virtual key with models: e2e-tier-984870b7c23f, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /chat/completions request to e2e-tier-984870b7c23f
    View test on GitHub ↗
  1335. ✓test_timeout_on_first_deployment_succeeds_on_retry TestReliabilityRetriestest_reliability_retries_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-retry-72f17265589b that calls openai/gpt-5.5
    View test on GitHub ↗
  1336. ✓test_timeout_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-tofail-988abb5ccf2c that calls openai/gpt-5.5
    View test on GitHub ↗
  1337. ✓test_timeout_trips_cooldown_then_recovers TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-cooldown-timeout-74e169760c17 that calls openai/gpt-5.5
    View test on GitHub ↗
  1338. ✓test_trivial_ask_behind_keyword_heavy_system_prompt_stays_on_the_cheap_tier TestComplexityHeuristicScopetest_auto_router_regressions_e2e.py›
    1. Add a deployment named e2e-heuristic-cheap-2bca66b556e1 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named e2e-heuristic-strong-2bca66b556e1 that calls openai/gpt-5.6
    3. Add a deployment named e2e-heuristic-router-2bca66b556e1 that calls auto_router/complexity_router
    4. Generate a virtual key with models: e2e-heuristic-router-2bca66b556e1, e2e-heuristic-cheap-2bca66b556e1, e2e-heuristic-strong-2bca66b556e1 and user id: e2e-auto-router-regressions
    5. Send a /chat/completions request to e2e-heuristic-router-2bca66b556e1
    6. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1339. ✓test_untagged_chat_is_always_served_by_the_plain_deployment TestTagSplitRoutingtest_auto_router_regressions_e2e.py›
    1. Generate a virtual key with models: e2e-autoroute-9dd91555bafd, e2e-tier-9dd91555bafd, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /chat/completions request to e2e-autoroute-9dd91555bafd
    3. Wait for at least 5 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1340. ✓test_untagged_messages_is_served_by_the_plain_deployment TestTagSplitRoutingtest_auto_router_regressions_e2e.py›
    1. Generate a virtual key with models: e2e-autoroute-9dd91555bafd, e2e-tier-9dd91555bafd, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Send a /v1/messages request to e2e-autoroute-9dd91555bafd
    3. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1341. ✓test_untagged_responses_is_served_by_the_plain_deployment TestResponsesApiTagRoutingtest_auto_router_regressions_e2e.py›
    1. Generate a virtual key with models: e2e-autoroute-05a2f5f10f91, e2e-tier-05a2f5f10f91, user id: e2e-auto-router-regressions and router settings: enable tag filtering: True
    2. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1342. ✓test_usage_based_picks_deployment_with_tpm_headroom TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Add a deployment named reliability-usage-5c55656d58d4 that calls openai/gpt-5.5
    View test on GitHub ↗
  1343. ✓test_workers_idle_under_rss_budget_before_traffic TestReliabilityMemorytest_reliability_memory_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1344. ✓test_zero_priced_alias_still_logs_spend_at_the_tier_rate TestStrategyAliasPricingtest_auto_router_regressions_e2e.py›
    1. Add a deployment named e2e-priced-tier-b25d1fcf5654 that calls anthropic/claude-haiku-4-5
    2. Add a deployment named e2e-priced-alias-b25d1fcf5654 that calls auto_router/complexity_router
    3. Generate a virtual key with models: e2e-priced-alias-b25d1fcf5654, e2e-priced-tier-b25d1fcf5654 and user id: e2e-auto-router-regressions
    4. Send a /chat/completions request to e2e-priced-alias-b25d1fcf5654
    5. Wait for at least 1 of the key's spend logs in /spend/logs
    View test on GitHub ↗
  1345. −test_latency_based_routes_around_deployment_that_times_out TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›

    No steps recorded

    LIT-7682: since #40229 the per-request routing_strategy override runs without the latency sampler, so latency-based has no signal to route on

    View test on GitHub ↗
  1346. −test_least_busy_avoids_deployment_with_request_in_flight TestReliabilityRoutingStrategiestest_reliability_routing_strategies_e2e.py›

    No steps recorded

    LIT-7682: since #40229 the per-request routing_strategy override runs without the in-flight counter, so least-busy has no signal to route on

    View test on GitHub ↗
  1347. −test_llm_classifier_runs_and_routes_by_semantic_tier TestComplexityRouterLlmClassifiertest_complexity_router_e2e.py›

    No steps recorded

    product bug LIT-4521: LLM classifier returns SIMPLE for short hard prompts (e.g. Is P equal to NP?); re-enable when classifier tier quality is fixed

    View test on GitHub ↗
  1348. −test_sibling_replica_serves_backup_within_redis_read_interval TestReliabilityCooldownstest_reliability_cooldowns_e2e.py›

    No steps recorded

    this cell trips a deployment through one gateway and reads the bench from another, so LITELLM_PROXY_REPLICA_URLS has to name at least two distinct gateways, got ('http://e2e-bk-432-router.litellm.svc.cluster.local',)

    View test on GitHub ↗
  1349. ✓test_a_kept_header_is_identity TestKeyDistinctnesstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1350. ✓test_a_volatile_header_is_not_identity TestKeyDistinctnesstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1351. ✓test_file_content_is_identity TestKeyDistinctnesstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1352. ✓test_form_fields_are_identity TestKeyDistinctnesstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1353. ✓test_generated_ids_in_the_path_do_not_change_the_key TestKeyStabilitytest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1354. ✓test_key_names_method_path_and_digest TestKeyShapetest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1355. ✓test_names_that_carry_credentials[api_base-False] TestSecretFieldstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1356. ✓test_names_that_carry_credentials[api_key-True] TestSecretFieldstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1357. ✓test_names_that_carry_credentials[aws_secret_access_key-True] TestSecretFieldstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1358. ✓test_names_that_carry_credentials[aws_session_token-True] TestSecretFieldstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1359. ✓test_names_that_carry_credentials[langfuse_secret_key-True] TestSecretFieldstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1360. ✓test_names_that_carry_credentials[max_completion_tokens-False] TestSecretFieldstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1361. ✓test_names_that_carry_credentials[model-False] TestSecretFieldstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1362. ✓test_names_that_carry_credentials[openai_api_key-True] TestSecretFieldstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1363. ✓test_names_that_carry_credentials[static_headers-True] TestSecretFieldstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1364. ✓test_names_that_carry_credentials[vertex_credentials-True] TestSecretFieldstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1365. ✓test_query_params_are_identity TestKeyDistinctnesstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1366. ✓test_requests_differing_only_inside_canonicalized_fields_stay_distinct TestKeyDistinctnesstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1367. ✓test_rewrites_exactly_the_volatile_shapes[0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef-<sha256>] TestPlaceholderstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1368. ✓test_rewrites_exactly_the_volatile_shapes[2026-08-19-<date>] TestPlaceholderstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1369. ✓test_rewrites_exactly_the_volatile_shapes[2026-08-19T20:57:13.363499+00:00-<timestamp>] TestPlaceholderstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1370. ✓test_rewrites_exactly_the_volatile_shapes[9f1c8a2e-4b3d-4f6a-8f2f-0a1b2c3d4e5f-<uuid>] TestPlaceholderstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1371. ✓test_rewrites_exactly_the_volatile_shapes[batch_688a8b7f9a08819096e0f7c88fcd07c5-<id>] TestPlaceholderstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1372. ✓test_rewrites_exactly_the_volatile_shapes[chatcmpl-C0LO6rRkfJlpJ2mqW9BHYo4Sm8FWl-<id>] TestPlaceholderstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1373. ✓test_rewrites_exactly_the_volatile_shapes[e2e-chat-stream-4d5152a995b7-e2e-chat-stream-<marker>] TestPlaceholderstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1374. ✓test_rewrites_exactly_the_volatile_shapes[file-XyZ12345abc-<id>] TestPlaceholderstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1375. ✓test_rewrites_exactly_the_volatile_shapes[gpt-4o-mini-gpt-4o-mini] TestPlaceholderstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1376. ✓test_rewrites_exactly_the_volatile_shapes[max_tokens-max_tokens] TestPlaceholderstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1377. ✓test_rewrites_exactly_the_volatile_shapes[Reply ok. 4d5152a995b7-Reply ok. <marker>] TestPlaceholderstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1378. ✓test_rewrites_exactly_the_volatile_shapes[sk-1234-sk-1234] TestPlaceholderstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1379. ✓test_rewrites_exactly_the_volatile_shapes[sk-3mCXCTGmYuEEIU2i2qmVE3Xq6tSK1O0X6ZIRP1Lpw8ZlbNjt-<key>] TestPlaceholderstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1380. ✓test_rewrites_exactly_the_volatile_shapes[zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz-zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] TestPlaceholderstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1381. ✓test_secret_set_versus_unset_stays_distinct TestKeyDistinctnesstest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1382. ✓test_serialization_order_is_not_identity TestKeyStabilitytest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1383. ✓test_volatile_material_does_not_change_the_key TestKeyStabilitytest_fixture_canonical.py›

    No steps recorded

    View test on GitHub ↗
  1384. ✓test_429_is_never_retried TestTransientRetryPolicytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1385. ✓test_a_non_streaming_outcome_carries_no_arrivals TestStreamEventArrivalstest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1386. ✓test_body_that_is_not_json_is_still_a_validation_failure TestClassifyEmptyBodytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1387. ✓test_each_event_is_stamped_at_the_moment_its_line_arrives TestStreamEventArrivalstest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1388. ✓test_empty_2xx_body_is_a_success TestClassifyEmptyBodytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1389. ✓test_malformed_guardrail_mode_remains_a_validation_failure[5] TestSpendLogDecodingtest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1390. ✓test_malformed_guardrail_mode_remains_a_validation_failure[mode1] TestSpendLogDecodingtest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1391. ✓test_malformed_guardrail_mode_remains_a_validation_failure[mode2] TestSpendLogDecodingtest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1392. ✓test_non_transient_status_returns_immediately[200] TestTransientRetryPolicytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1393. ✓test_non_transient_status_returns_immediately[201] TestTransientRetryPolicytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1394. ✓test_non_transient_status_returns_immediately[400] TestTransientRetryPolicytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1395. ✓test_non_transient_status_returns_immediately[401] TestTransientRetryPolicytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1396. ✓test_non_transient_status_returns_immediately[404] TestTransientRetryPolicytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1397. ✓test_non_transient_status_returns_immediately[422] TestTransientRetryPolicytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1398. ✓test_non_transient_status_returns_immediately[500] TestTransientRetryPolicytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1399. ✓test_non_transient_status_returns_immediately[502] TestTransientRetryPolicytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1400. ✓test_non_transient_status_returns_immediately[503] TestTransientRetryPolicytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1401. ✓test_non_transient_status_returns_immediately[504] TestTransientRetryPolicytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1402. ✓test_overloaded_529_retries_with_backoff_then_returns_the_success TestTransientRetryPolicytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1403. ✓test_partial_body_omits_unset_fields_and_sends_explicit_none_as_null TestWireBodytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1404. ✓test_persistent_transient_is_bounded_and_returns_the_last_response TestTransientRetryPolicytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1405. ✓test_plain_body_drops_none_fields TestWireBodytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1406. ✓test_qualification_disables_retries_and_restores_the_default TestTransientRetryPolicytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1407. ✓test_supported_guardrail_modes_preserve_neighbor_attribution_and_masked_response[mode2-False] TestSpendLogDecodingtest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1408. ✓test_supported_guardrail_modes_preserve_neighbor_attribution_and_masked_response[mode2-True] TestSpendLogDecodingtest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1409. ✓test_supported_guardrail_modes_preserve_neighbor_attribution_and_masked_response[mode3-False] TestSpendLogDecodingtest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1410. ✓test_supported_guardrail_modes_preserve_neighbor_attribution_and_masked_response[mode3-True] TestSpendLogDecodingtest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1411. ✓test_supported_guardrail_modes_preserve_neighbor_attribution_and_masked_response[mode4-False] TestSpendLogDecodingtest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1412. ✓test_supported_guardrail_modes_preserve_neighbor_attribution_and_masked_response[mode4-True] TestSpendLogDecodingtest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1413. ✓test_supported_guardrail_modes_preserve_neighbor_attribution_and_masked_response[None-False] TestSpendLogDecodingtest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1414. ✓test_supported_guardrail_modes_preserve_neighbor_attribution_and_masked_response[None-True] TestSpendLogDecodingtest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1415. ✓test_supported_guardrail_modes_preserve_neighbor_attribution_and_masked_response[post_call-False] TestSpendLogDecodingtest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1416. ✓test_supported_guardrail_modes_preserve_neighbor_attribution_and_masked_response[post_call-True] TestSpendLogDecodingtest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1417. ✓test_transient_set_is_only_statuses_the_proxy_cannot_emit TestTransientRetryPolicytest_e2e_http.py›

    No steps recorded

    View test on GitHub ↗
  1418. ✓test_attach_is_idempotent TestResultPropertiestest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1419. ✓test_bar-logging] TestPackageFromNodeid::test_package_is_the_first_dir_under_the_suite_root[logging/test_x.py::TestFootest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1420. ✓test_bar-logging] TestPackageFromNodeid::test_package_is_the_first_dir_under_the_suite_root[tests/e2e/logging/test_x.py::TestFootest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1421. ✓test_bar-quota_management] TestPackageFromNodeid::test_package_is_the_first_dir_under_the_suite_root[quota_management/spend_tracking/test_x.pytest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1422. ✓test_bar-root] TestPackageFromNodeid::test_package_is_the_first_dir_under_the_suite_root[tests/e2e/test_fixture_mode.pytest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1423. ✓test_both_invocation_shapes_collapse_to_the_same_components[./logging/test_x.py] TestSuitePartstest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1424. ✓test_both_invocation_shapes_collapse_to_the_same_components[logging/test_x.py] TestSuitePartstest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1425. ✓test_both_invocation_shapes_collapse_to_the_same_components[tests/e2e/logging/test_x.py] TestSuitePartstest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1426. ✓test_both_invocation_shapes_collapse_to_the_same_components[tests\\e2e\\logging\\test_x.py] TestSuitePartstest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1427. ✓test_every_test_carries_package_covers_and_source TestResultPropertiestest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1428. ✓test_ids_are_unique_order_preserving_and_non_empty_strings TestDedupeCoverstest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1429. ✓test_known_values_normalize-root] TestPackageFromNodeid::test_package_is_the_first_dir_under_the_suite_root[test_fixture_mode.py::TestParseFixtureModetest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1430. ✓test_line_is_emitted_one_based TestSourceFromLocationtest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1431. ✓test_nothing_linkable_yields_empty_rather_than_a_guess[-40] TestSourceFromLocationtest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1432. ✓test_nothing_linkable_yields_empty_rather_than_a_guess[../conftest.py-40] TestSourceFromLocationtest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1433. ✓test_nothing_linkable_yields_empty_rather_than_a_guess[/app/e2e/a2a/test_x.py-40] TestSourceFromLocationtest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1434. ✓test_nothing_linkable_yields_empty_rather_than_a_guess[a2a/test_x.py-None] TestSourceFromLocationtest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1435. ✓test_nothing_linkable_yields_empty_rather_than_a_guess[C:\\app\\e2e\\a2a\\test_x.py-40] TestSourceFromLocationtest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1436. ✓test_path_is_repo_relative_however_pytest_was_started[a2a/test_a2a_agent_e2e.py] TestSourceFromLocationtest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1437. ✓test_path_is_repo_relative_however_pytest_was_started[tests/e2e/a2a/test_a2a_agent_e2e.py] TestSourceFromLocationtest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1438. ✓test_top_level_suite_file_keeps_its_single_component TestSuitePartstest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1439. ✓test_top_level_suite_file_sits_directly_under_the_suite_root TestSourceFromLocationtest_junit_properties.py›

    No steps recorded

    View test on GitHub ↗
  1440. −test_suite_root_names_this_file_s_real_home TestSuiteRoottest_junit_properties.py›

    No steps recorded

    no checkout above this file (the runner image copies tests/e2e/ to /app/e2e)

    View test on GitHub ↗
  1441. ✓test_a_missing_admin_credential_fails_loudly_instead_of_skipping[ ]test_idp.py›

    No steps recorded

    View test on GitHub ↗
  1442. ✓test_a_missing_admin_credential_fails_loudly_instead_of_skipping[]test_idp.py›

    No steps recorded

    View test on GitHub ↗
  1443. ✓test_a_refused_create_fails_the_test_with_the_idps_own_wordstest_idp.py›

    No steps recorded

    View test on GitHub ↗
  1444. ✓test_cleanup_failure_is_visibletest_idp.py›

    No steps recorded

    View test on GitHub ↗
  1445. ✓test_connection_details_come_from_the_environmenttest_idp.py›

    No steps recorded

    View test on GitHub ↗
  1446. ✓test_create_without_a_resource_id_fails[]test_idp.py›

    No steps recorded

    View test on GitHub ↗
  1447. ✓test_create_without_a_resource_id_fails[http://keycloak/groups/]test_idp.py›

    No steps recorded

    View test on GitHub ↗
  1448. ✓test_created_id_is_the_last_segment_of_the_location_headertest_idp.py›

    No steps recorded

    View test on GitHub ↗
  1449. ✓test_expired_admin_credentials_do_not_abort_remaining_cleanupstest_idp.py›

    No steps recorded

    View test on GitHub ↗
  1450. ✓test_new_users_are_born_fully_set_uptest_idp.py›

    No steps recorded

    View test on GitHub ↗
  1451. ✓test_oidc_launcher_removes_client_on_exit_and_termination[group-False]test_idp.py›

    No steps recorded

    View test on GitHub ↗
  1452. ✓test_oidc_launcher_removes_client_on_exit_and_termination[normal-False]test_idp.py›

    No steps recorded

    View test on GitHub ↗
  1453. ✓test_oidc_launcher_removes_client_on_exit_and_termination[parent-False]test_idp.py›

    No steps recorded

    View test on GitHub ↗
  1454. ✓test_oidc_launcher_removes_client_on_exit_and_termination[parent-True]test_idp.py›

    No steps recorded

    View test on GitHub ↗
  1455. ✓test_partial_provisioning_removes_the_group_when_user_creation_failstest_idp.py›

    No steps recorded

    View test on GitHub ↗
  1456. ✓test_provisioning_records_zero_one_or_multiple_groups[groups0]test_idp.py›

    No steps recorded

    View test on GitHub ↗
  1457. ✓test_provisioning_records_zero_one_or_multiple_groups[groups1]test_idp.py›

    No steps recorded

    View test on GitHub ↗
  1458. ✓test_provisioning_records_zero_one_or_multiple_groups[groups2]test_idp.py›

    No steps recorded

    View test on GitHub ↗
  1459. ✓test_realm_urls_match_keycloaks_own_layouttest_idp.py›

    No steps recorded

    View test on GitHub ↗
  1460. ✓test_strict_cleanup_reports_each_failure_and_continuestest_idp.py›

    No steps recorded

    View test on GitHub ↗
  1461. ✓test_successful_provisioning_cleans_up_user_before_grouptest_idp.py›

    No steps recorded

    View test on GitHub ↗
  1462. ✓test_encrypted_stored_value_is_not_a_bearer_token TestSessionTokentest_session_token_e2e.py›
    1. Generate a virtual key with key alias: e2e-session-41a97959a43b and metadata: logging: callback name: langfuse and callback vars: default settings
    2. Read the key's settings back from /key/info
    View test on GitHub ↗
  1463. ✓test_expired_session_token_is_denied TestSessionTokentest_session_token_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1464. ✓test_expired_token_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    1. Create a team with team alias: e2e-jwt-19e38ea42ccf and team id: e2e-jwt-team-19e38ea42ccf
    2. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  1465. ✓test_liveness_reports_alive_without_auth TestHealthLifecycletest_health_lifecycle_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1466. ✓test_master_key_authenticates_and_grants_admin_route TestMasterKeyAuthtest_master_key_auth_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1467. ✓test_non_matching_master_key_is_denied TestMasterKeyAuthtest_master_key_auth_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1468. ✓test_plain_virtual_key_still_works_with_jwt_auth_enabled TestJwtAuthtest_jwt_auth_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  1469. ✓test_readiness_details_require_auth_and_expose_diagnostics TestHealthLifecycletest_health_lifecycle_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1470. ✓test_readiness_is_reachable_without_credentials TestHealthLifecycletest_health_lifecycle_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1471. ✓test_readiness_reports_connected_db TestHealthLifecycletest_health_lifecycle_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1472. ✓test_signed_token_for_another_application_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    1. Create a team with team alias: e2e-jwt-1103629aade8 and team id: e2e-jwt-team-1103629aade8
    2. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  1473. ✓test_signed_token_from_the_wrong_issuer_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    1. Create a team with team alias: e2e-jwt-c28121a5ad0e and team id: e2e-jwt-team-c28121a5ad0e
    2. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  1474. ✓test_tampered_signature_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    1. Create a team with team alias: e2e-jwt-9de76d6bdc01 and team id: e2e-jwt-team-9de76d6bdc01
    2. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  1475. ✓test_team_header_with_the_alias_of_a_team_the_caller_is_not_in_is_rejected_like_an_unknown_value TestJwtTeamHeadertest_jwt_auth_e2e.py›
    1. Create a team with team alias: e2e-jwt-alias-f01e9b4e07a9 and team id: e2e-jwt-team-f01e9b4e07a9
    2. Create a team with team alias: e2e-jwt-alias-bfbd5f2dd0a6 and team id: e2e-jwt-other-bfbd5f2dd0a6
    View test on GitHub ↗
  1476. ✓test_team_header_with_the_team_alias_binds_the_same_team_as_the_team_id TestJwtTeamHeadertest_jwt_auth_e2e.py›
    1. Create a team with team alias: e2e-jwt-alias-1b9083f58639 and team id: e2e-jwt-team-1b9083f58639
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  1477. ✓test_team_model_alias_is_listed_by_v1_models_under_the_same_token_that_routes_it[anthropic_shape] TestJwtTeamHeadertest_jwt_auth_e2e.py›
    1. Create a team with team alias: e2e-jwt-aliased-7465eb5da9e7, models: gpt-5.5, team id: e2e-jwt-team-7465eb5da9e7 and model aliases: e2e-jwt-model-alias-7465eb5da9e7: gpt-5.5
    2. Send a /chat/completions request to e2e-jwt-model-alias-7465eb5da9e7
    View test on GitHub ↗
  1478. ✓test_team_model_alias_is_listed_by_v1_models_under_the_same_token_that_routes_it[openai_shape] TestJwtTeamHeadertest_jwt_auth_e2e.py›
    1. Create a team with team alias: e2e-jwt-aliased-bcd863816ede, models: gpt-5.5, team id: e2e-jwt-team-bcd863816ede and model aliases: e2e-jwt-model-alias-bcd863816ede: gpt-5.5
    2. Send a /chat/completions request to e2e-jwt-model-alias-bcd863816ede
    View test on GitHub ↗
  1479. ✓test_token_naming_a_team_that_does_not_exist_is_rejected TestJwtAuthtest_jwt_auth_e2e.py›
    1. Send a /chat/completions request to gpt-5.5
    View test on GitHub ↗
  1480. ✓test_unexpired_session_token_reaches_admin_route TestSessionTokentest_session_token_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1481. ✓test_valid_token_for_an_existing_team_is_accepted_and_attributed TestJwtAuthtest_jwt_auth_e2e.py›
    1. Create a team with team alias: e2e-jwt-8d8a7b66a225 and team id: e2e-jwt-team-8d8a7b66a225
    2. Send a /chat/completions request to gpt-5.5
    3. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  1482. ✓test_allowed_model_is_permitted TestAccessControltest_access_control_e2e.py›
    1. Generate a virtual key with models: gemini-2.5-flash and user id: e2e-test-user
    View test on GitHub ↗
  1483. ✓test_bearer_invalid_token_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1484. ✓test_disallowed_model_is_denied_403 TestAccessControltest_access_control_e2e.py›
    1. Generate a virtual key with models: gemini-2.5-flash and user id: e2e-test-user
    View test on GitHub ↗
  1485. ✓test_empty_bearer_token_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1486. ✓test_group_grants_every_deployment_in_it[bare name the group's wildcard covers-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    1. Add a deployment named openai/gpt-5.4* that calls openai/gpt-5.4*
    2. Add a deployment named e2e-ag-member-d82dba3c44ed that calls openai/gpt-5.4-nano
    3. Add a deployment named e2e-ag-outsider-d82dba3c44ed that calls openai/gpt-5.4-nano
    4. Generate a virtual key with models: e2e-ag-d82dba3c44ed and user id: e2e-test-user
    View test on GitHub ↗
  1487. ✓test_group_grants_every_deployment_in_it[exactly-named deployment in the group-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    1. Generate a virtual key with models: e2e-ag-d82dba3c44ed and user id: e2e-test-user
    View test on GitHub ↗
  1488. ✓test_group_grants_every_deployment_in_it[provider-prefixed name the group's wildcard covers-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    1. Generate a virtual key with models: e2e-ag-d82dba3c44ed and user id: e2e-test-user
    View test on GitHub ↗
  1489. ✓test_group_grants_nothing_outside_it[deployment outside the group-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    1. Generate a virtual key with models: e2e-ag-d82dba3c44ed and user id: e2e-test-user
    View test on GitHub ↗
  1490. ✓test_group_grants_nothing_outside_it[name no provider claims-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    1. Generate a virtual key with models: e2e-ag-d82dba3c44ed and user id: e2e-test-user
    View test on GitHub ↗
  1491. ✓test_group_grants_nothing_outside_it[provider model outside the group's wildcard-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py›
    1. Generate a virtual key with models: e2e-ag-d82dba3c44ed and user id: e2e-test-user
    View test on GitHub ↗
  1492. ✓test_group_grants_the_team_nothing_outside_it TestTeamScopedToAccessGrouptest_model_access_group_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1493. ✓test_group_grants_the_teams_own_wildcard TestTeamScopedToAccessGrouptest_model_access_group_e2e.py›
    1. Generate a virtual key with team id: 9942ea09-7635-4a4a-8331-0415b08d9d3a
    2. Add a deployment named openai/gpt-5.6* that calls openai/gpt-5.6*
    3. Read /team/info?team_id=9942ea09-7635-4a4a-8331-0415b08d9d3a on every proxy replica until it settles
    View test on GitHub ↗
  1494. ✓test_llm_api_routes_group_grants_every_llm_endpoint TestAccessControltest_access_control_e2e.py›
    1. Generate a virtual key with allowed routes: llm_api_routes
    2. Send an /embeddings request to openai-text-embedding-3-small for "route group 52bb4a535d91"
    View test on GitHub ↗
  1495. ✓test_llm_only_key_forbidden_from_management_route_403 TestAccessControltest_access_control_e2e.py›
    1. Generate a virtual key with allowed routes: llm_api_routes
    View test on GitHub ↗
  1496. ✓test_missing_authorization_header_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1497. ✓test_not_bearer_scheme_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1498. ✓test_token_without_bearer_prefix_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1499. ✓test_unknown_model_returns_400 TestAccessControltest_access_control_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1500. ✓test_valid_key_allows_and_invalid_key_denied TestVirtualKeyAuthtest_access_control_e2e.py›
    1. Add a deployment named e2e-auth-chat-5c487097ca63 that calls anthropic/claude-haiku-4-5-20251001
    2. Generate a virtual key with user id: e2e-test-user
    3. Send a /chat/completions request to e2e-auth-chat-5c487097ca63
    View test on GitHub ↗
  1501. ✓test_a_streamed_response_round_trips_through_the_bundle TestRecordAndLoadtest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1502. ✓test_bundle_at_the_limit_is_still_fresh TestFreshnesstest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1503. ✓test_corrupt_manifest_is_unreadable TestFreshnesstest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1504. ✓test_fresh_directory_gets_a_fresh_manifest TestPrepareBundletest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1505. ✓test_interaction_filename_orders_and_slugs TestSlugstest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1506. ✓test_interactions_group_per_test TestRecordAndLoadtest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1507. ✓test_load_bundle_rejects_a_foreign_format_version TestRecordAndLoadtest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1508. ✓test_load_returns_interactions_in_recorded_order TestRecordAndLoadtest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1509. ✓test_missing_manifest_is_unreadable_with_recording_hint TestFreshnesstest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1510. ✓test_naive_recorded_at_is_read_as_utc TestFreshnesstest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1511. ✓test_record_wipes_the_previous_bundle_instead_of_reading_it TestPrepareBundletest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1512. ✓test_refuses_a_path_that_is_a_file TestPrepareBundletest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1513. ✓test_refuses_to_wipe_a_directory_that_is_not_a_bundle TestPrepareBundletest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1514. ✓test_same_tail_in_different_files_never_collides TestSlugstest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1515. ✓test_slug_for_test_is_deterministic TestSlugstest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1516. ✓test_stale_bundle_reports_age_and_limit TestFreshnesstest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1517. ✓test_unknown_format_version_is_unreadable TestFreshnesstest_fixture_bundle.py›

    No steps recorded

    View test on GitHub ↗
  1518. ✓test_fresh_replay_bundle_collects TestCollectionGatetest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1519. ✓test_invalid_mode_names_the_value_and_the_choices TestCollectionGatetest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1520. ✓test_known_values_normalize[ REPLAY -replay] TestParseFixtureModetest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1521. ✓test_known_values_normalize[-live] TestParseFixtureModetest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1522. ✓test_known_values_normalize[live-live] TestParseFixtureModetest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1523. ✓test_known_values_normalize[record-record] TestParseFixtureModetest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1524. ✓test_known_values_normalize[replay-replay] TestParseFixtureModetest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1525. ✓test_live_and_record_never_block_collection[] TestCollectionGatetest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1526. ✓test_live_and_record_never_block_collection[live] TestCollectionGatetest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1527. ✓test_live_and_record_never_block_collection[record] TestCollectionGatetest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1528. ✓test_live_mode_prints_nothing TestReportHeadertest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1529. ✓test_names_this_test_and_strips_the_phase TestCurrentTestKeytest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1530. ✓test_record_and_replay_name_the_bundle TestReportHeadertest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1531. ✓test_replay_with_no_bundle_says_how_to_record_one TestCollectionGatetest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1532. ✓test_sequence_is_a_pure_function_of_test_and_ordinal TestDeterministicMarkertest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1533. ✓test_stale_replay_bundle_fails_naming_its_age TestCollectionGatetest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1534. ✓test_unknown_value_is_invalid_with_the_original_spelling TestParseFixtureModetest_fixture_mode.py›

    No steps recorded

    View test on GitHub ↗
  1535. ✓test_cell_claimed_only_by_a_skipped_test_is_uncoveredtest_collector.py›

    No steps recorded

    View test on GitHub ↗
  1536. ✓test_collection_counts_only_markers_on_tests_that_would_runtest_collector.py›

    No steps recorded

    View test on GitHub ↗
  1537. ✓test_compute_coverage_counts_covered_p0_and_gapstest_collector.py›

    No steps recorded

    View test on GitHub ↗
  1538. ✓test_json_render_exposes_module_coverage_for_grafana_jobstest_collector.py›

    No steps recorded

    View test on GitHub ↗
  1539. ✓test_llm_cells_roll_up_by_core_endpointtest_collector.py›

    No steps recorded

    View test on GitHub ↗
  1540. ✓test_load_registry_rejects_duplicate_idstest_collector.py›

    No steps recorded

    View test on GitHub ↗
  1541. ✓test_logging_and_guardrail_roll_up_into_one_moduletest_collector.py›

    No steps recorded

    View test on GitHub ↗
  1542. ✓test_loki_render_exposes_exact_stdout_lines_for_lokitest_collector.py›

    No steps recorded

    View test on GitHub ↗
  1543. ✓test_orphan_marker_is_reported_not_countedtest_collector.py›

    No steps recorded

    View test on GitHub ↗
  1544. ✓test_prometheus_render_exposes_module_coverage_timeseriestest_collector.py›

    No steps recorded

    View test on GitHub ↗
  1545. ✓test_real_registry_loads_and_ids_are_uniquetest_collector.py›

    No steps recorded

    View test on GitHub ↗
  1546. ✓test_skipped_marker_outside_the_registry_is_still_an_orphantest_collector.py›

    No steps recorded

    View test on GitHub ↗
  1547. ✓test_text_render_uses_plain_coverage_languagetest_collector.py›

    No steps recorded

    View test on GitHub ↗
  1548. ✓test_access_group_scopes_tool_selection TestMcpAccessGroupToolSelectiontest_mcp_access_group_e2e.py›
    1. Read /v1/mcp/server on every proxy replica until it settles
    2. Generate a virtual key with user id: e2e-mcp-ag-granted-ae761e6796f0 and object permission: mcp access groups: e2e-mcp-grp-f560de17321b
    3. Generate a virtual key with user id: e2e-mcp-ag-other-3bf41cc2a9ab and object permission: mcp access groups: e2e-mcp-grp-absent-4da1630a159c
    View test on GitHub ↗
  1549. ✓test_alias_grant_persists_verbatim_and_lists_tools TestMcpKeyGrantByAliastest_mcp_key_access_e2e.py›
    1. Read /v1/mcp/server on every proxy replica until it settles
    2. Generate a virtual key with user id: e2e-mcp-allowed-a2b58e2e6ff4 and object permission: mcp servers: e2e_dd_mcp_133464326248
    3. Read the key's settings back from /key/info
    View test on GitHub ↗
  1550. ✓test_call_tool_denied_without_permission TestMcpKeyWithoutAccessIsDeniedtest_mcp_key_access_e2e.py›
    1. Read /v1/mcp/server on every proxy replica until it settles
    2. Generate a virtual key with user id: e2e-mcp-allowed-712265129c6b and object permission: mcp servers: ceb38eef-50d9-4899-bf6d-3e55f76da4b7
    3. Generate a virtual key with user id: e2e-mcp-denied-9530c1084987
    View test on GitHub ↗
  1551. ✓test_content_filter_blocks_banned_keyword_in_tool_args TestMcpToolCallGuardrailtest_mcp_guardrail_e2e.py›
    1. Generate a virtual key with user id: e2e-mcp-guard-e137c7e45d34 and object permission: mcp servers: adc2fd8d-1aee-4c6a-8252-d4c2f36ab9ed
    View test on GitHub ↗
  1552. ✓test_key_granted_a_toolset_lists_exactly_its_tools TestMcpToolsetEnforcementtest_mcp_toolset_enforcement_e2e.py›
    1. Read /v1/mcp/server on every proxy replica until it settles
    2. Generate a virtual key with user id: e2e-mcp-catalog-dd8b1806cb80 and object permission: mcp servers: 2df0ca57-87cc-46e4-b03c-211489cf94ce
    3. Create an MCP toolset with the tools server id: 2df0ca57-87cc-46e4-b03c-211489cf94ce and tool name: _dd_gff, server id: 2df0ca57-87cc-46e4-b03c-211489cf94ce and tool name: _dd_guc
    4. Generate a virtual key with user id: e2e-mcp-toolset-4919c8d10a78 and object permission: mcp servers: 2df0ca57-87cc-46e4-b03c-211489cf94ce and mcp toolsets: 0fbd2d37-9dab-4bf8-b61b-7961bcba1e69
    View test on GitHub ↗
  1553. ✓test_list_tools_denied_without_permission TestMcpKeyWithoutAccessIsDeniedtest_mcp_key_access_e2e.py›
    1. Read /v1/mcp/server on every proxy replica until it settles
    2. Generate a virtual key with user id: e2e-mcp-allowed-aa1a5473b5da and object permission: mcp servers: cc62ce0a-7ae4-408f-b3ab-da7edb8e2dfb
    3. Generate a virtual key with user id: e2e-mcp-denied-9d3375a90375
    View test on GitHub ↗
  1554. ✓test_route_restricted_health_matches_server_grants TestMcpHealthVisibilitytest_mcp_key_access_e2e.py›
    1. Read /v1/mcp/server on every proxy replica until it settles
    2. Generate a virtual key with user id: e2e-mcp-allowed-ae14ce844e08 and object permission: mcp servers: bd70f244-2d4b-4ef5-8f71-7f547a8e985c
    3. Generate a virtual key with user id: e2e-mcp-health-dd1276c40222, allowed routes: /v1/mcp/server, /v1/mcp/server/health and object permission: mcp servers: bd70f244-2d4b-4ef5-8f71-7f547a8e985c
    4. Generate a virtual key with user id: e2e-mcp-health-33eaf99ae804, allowed routes: /v1/mcp/server, /v1/mcp/server/health and object permission: mcp servers: cda25c57-68df-41ad-a59c-444dd24f3c02
    5. Generate a virtual key with user id: e2e-mcp-health-53bb49b4e7bc, allowed routes: /v1/mcp/server, /v1/mcp/server/health and object permission: default settings
    View test on GitHub ↗
  1555. ✓test_search_logs_finds_seeded_completion TestDatadogMcpRoundTriptest_mcp_datadog_e2e.py›
    1. Call the management route /health/readiness/details
    2. Read /v1/mcp/server on every proxy replica until it settles
    3. Generate a virtual key with models: claude-haiku-4-5, user id: e2e-dd-mcp-0e2440cdc240 and object permission: mcp servers: 7195438e-9252-4948-adab-a725664ba346
    4. Send a /chat/completions request to claude-haiku-4-5
    View test on GitHub ↗
  1556. −test_chat_completion_uses_linear_with_authorization_bearer_header TestMcpChatCompletionOauthtest_mcp_chat_completion_oauth_e2e.py›

    No steps recorded

    set E2E_LINEAR_STORAGE_STATE to a Linear session captured via mcp/linear_session_capture.py

    View test on GitHub ↗
  1557. −test_chat_completion_uses_linear_with_x_litellm_api_key_header TestMcpChatCompletionOauthtest_mcp_chat_completion_oauth_e2e.py›

    No steps recorded

    set E2E_LINEAR_STORAGE_STATE to a Linear session captured via mcp/linear_session_capture.py

    View test on GitHub ↗
  1558. ✓test_discovery_card_is_proxy_fronted TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1559. ✓test_malformed_protocol_version_rejected TestA2AAgentLifecycletest_a2a_agent_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1560. ✓test_message_send_runs_completion_bridge TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    2. Wait for the request's spend log in /spend/logs
    View test on GitHub ↗
  1561. ✓test_pinned_v0_3_serves_flat_message_shape TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1562. ✓test_pinned_v1_0_serves_nested_message_shape TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1563. ✓test_real_world_agent_replies_to_property_query TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1564. ✓test_register_persists TestA2AAgentLifecycletest_a2a_agent_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1565. ✓test_semver_protocol_version_registers_and_serves TestA2AAgentLifecycletest_a2a_agent_e2e.py›
    1. Generate a virtual key with user id: e2e-test-user
    View test on GitHub ↗
  1566. ✓test_unsupported_protocol_version_rejected TestA2AAgentLifecycletest_a2a_agent_e2e.py›

    No steps recorded

    View test on GitHub ↗
  1567. −llm_translation/realtime/test_realtime_pipecat_audio_e2e.pyllm_translation/realtime/test_realtime_pipecat_audio_e2e.py›

    No steps recorded

    collection skipped

  1568. ✓test_readers_share_exclusive_waits_and_a_waiting_exclusive_beats_later_readerstest_stack_lock.py›

    No steps recorded

    View test on GitHub ↗