litellm-e2e · build #46

643 passed, 3 failed, 58 skipped

1.99.0.dev2nightly

Aug 21, 2026, 3:01 AM UTC ·418c7c6 ↗· build #46 ↗· 1.99.0.dev2 ↗· run history

All 704 tests

  1. ✗test_unflagged_model_converts_system_reminder_and_succeeds TestBedrockInvokeMidConversationSystemtest_messages_mid_conversation_system_e2e.py›

    Failed: e2e-midsys-0a506627e9d6: prompt cache never became readable in full within 60.0s (last usage: input_tokens=3 output_tokens=6 cache_creation_input_tokens=1609 cache_read_input_tokens=5101)

  2. ✗test_unflagged_model_converts_system_reminder_and_succeeds TestAzureFoundryMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›

    Failed: e2e-midsys-c29308893a33: prompt cache never became readable in full within 60.0s (last usage: input_tokens=6 output_tokens=8 cache_creation_input_tokens=1812 cache_read_input_tokens=6602)

  3. ✗test_unflagged_model_converts_system_reminder_and_succeeds TestVertexMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›

    Failed: e2e-midsys-45b8fa9f8b98: prompt cache never became readable in full within 60.0s (last usage: input_tokens=3 output_tokens=6 cache_creation_input_tokens=1409 cache_read_input_tokens=4802)

  4. ✓test_anthropic_passthrough_nonstreaming_logs_costtest_passthrough_e2e.py
  5. ✓test_anthropic_passthrough_streaming_logs_costtest_passthrough_e2e.py
  6. ✓test_anthropic_passthrough_tool_call_logs_costtest_passthrough_e2e.py
  7. ✓test_array_input_returns_vectors TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py
  8. ✓test_audio_speech_returns_audio TestAudioSpeechtest_audio_speech_e2e.py
  9. ✓test_audio_speech_streams_audio_chunks TestAudioSpeechtest_audio_speech_e2e.py
  10. ✓test_audio_transcriptions_returns_text TestAudioTranscriptionstest_audio_transcriptions_e2e.py
  11. ✓test_basic_nonstream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py
  12. ✓test_basic_stream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py
  13. ✓test_bedrock_converse_chat_returns_content TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py
  14. ✓test_bedrock_converse_chat_returns_reasoning TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py
  15. ✓test_bedrock_converse_chat_returns_tool_call TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py
  16. ✓test_bedrock_converse_chat_streams_real_content TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py
  17. ✓test_bedrock_converse_chat_vision_describes_image TestBedrockConverseChatCompletionstest_chat_completions_regression_e2e.py
  18. ✓test_bedrock_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py
  19. ✓test_bedrock_image_generation_returns_image TestImageGenerationtest_image_generation_e2e.py
  20. ✓test_bedrock_prompt_caching_reads_cache TestCacheControltest_cache_control.py
  21. ✓test_bedrock_rerank_scores_top_n TestReranktest_rerank_e2e.py
  22. ✓test_bridged_stream_delivers_content_finish_reason_and_done TestResponsesBridgeChatCompletionsStreamingtest_responses_bridge_streaming_e2e.py
  23. ✓test_bridged_stream_reassembles_tool_call TestResponsesBridgeChatCompletionsStreamingtest_responses_bridge_streaming_e2e.py
  24. ✓test_bridged_stream_shares_one_chunk_id TestResponsesBridgeChatCompletionsStreamingtest_responses_bridge_streaming_e2e.py
  25. ✓test_calls_without_auth_is_denied TestRealtimeHttptest_realtime_http_e2e.py
  26. ✓test_chat_returns_real_completion[claude-haiku-4-5-anthropic] TestChatCompletionsRegressiontest_chat_completions_regression_e2e.py
  27. ✓test_chat_returns_real_completion[gemini-2.5-flash-gemini] TestChatCompletionsRegressiontest_chat_completions_regression_e2e.py
  28. ✓test_chat_returns_real_completion[gpt-5.5-openai] TestChatCompletionsRegressiontest_chat_completions_regression_e2e.py
  29. ✓test_chat_stream_is_sse_and_ends_with_done TestChatStreamContracttest_chat_stream_contract_e2e.py
  30. ✓test_client_secret_missing_auth_is_denied TestRealtimeHttptest_realtime_http_e2e.py
  31. ✓test_cohere_chat_returns_content TestCohereChattest_chat_completions_regression_e2e.py
  32. ✓test_converse_empty_messages_returns_client_error TestBedrockNativetest_bedrock_native_e2e.py
  33. ✓test_converse_invalid_model_returns_error TestBedrockNativetest_bedrock_native_e2e.py
  34. ✓test_converse_missing_messages_returns_error TestBedrockNativetest_bedrock_native_e2e.py
  35. ✓test_converse_returns_assistant TestBedrockNativetest_bedrock_native_e2e.py
  36. ✓test_converse_stream_returns_chunks TestBedrockNativetest_bedrock_native_e2e.py
  37. ✓test_create_client_secret TestRealtimeHttptest_realtime_http_e2e.py
  38. ✓test_create_list_retrieve_delete_lifecycle TestVectorStorestest_vector_stores_e2e.py
  39. ✓test_credential_backed_messages TestCredentialBackedMessagestest_credential_messages_e2e.py
  40. ✓test_custom_pricing_is_billed_at_configured_rate TestCustomPricingtest_custom_pricing_e2e.py
  41. ✓test_custom_pricing_is_isolated_from_sibling_deployment TestCustomPricingtest_custom_pricing_e2e.py
  42. ✓test_embeddings_call_logs_its_cost TestOpenAIPassthroughSpendtest_passthrough_e2e.py
  43. ✓test_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py
  44. ✓test_empty_image_returns_error TestImageEdittest_image_edits_e2e.py
  45. ✓test_empty_input_returns_client_error TestResponsestest_responses_e2e.py
  46. ✓test_empty_messages_returns_client_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py
  47. ✓test_empty_prompt_returns_client_error TestImageGenerationtest_image_generation_e2e.py
  48. ✓test_empty_prompt_returns_error TestImageEdittest_image_edits_e2e.py
  49. ✓test_file_attach_poll_and_search TestVectorStorestest_vector_stores_e2e.py
  50. ✓test_gemini_chat_returns_content_and_logs_cost TestGeminiChatCompletionstest_chat_completions_regression_e2e.py
  51. ✓test_gemini_passthrough_nonstreaming_logs_costtest_passthrough_e2e.py
  52. ✓test_gemini_passthrough_streaming_logs_costtest_passthrough_e2e.py
  53. ✓test_gemini_passthrough_tool_call_logs_costtest_passthrough_e2e.py
  54. ✓test_generate_content_returns_response_cost_header TestGoogleNativeGenerateContenttest_google_native_e2e.py
  55. ✓test_image_edit_returns_image TestImageEdittest_image_edits_e2e.py
  56. ✓test_image_generation_returns_image TestImageGenerationtest_image_generation_e2e.py
  57. ✓test_invalid_chunking_returns_error TestVectorStorestest_vector_stores_e2e.py
  58. ✓test_invalid_max_completion_tokens_return_client_errors TestChatCompletionsContracttest_chat_completions_contract_e2e.py
  59. ✓test_invalid_n_returns_client_error TestImageGenerationtest_image_generation_e2e.py
  60. ✓test_invalid_role_returns_client_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py
  61. ✓test_invalid_size_returns_client_error TestImageGenerationtest_image_generation_e2e.py
  62. ✓test_invalid_temperatures_return_client_errors TestChatCompletionsContracttest_chat_completions_contract_e2e.py
  63. ✓test_invoke_invalid_temperature_returns_client_error TestBedrockNativetest_bedrock_native_e2e.py
  64. ✓test_invoke_missing_max_tokens_returns_error TestBedrockNativetest_bedrock_native_e2e.py
  65. ✓test_invoke_missing_messages_returns_error TestBedrockNativetest_bedrock_native_e2e.py
  66. ✓test_invoke_returns_message TestBedrockNativetest_bedrock_native_e2e.py
  67. ✓test_invoke_stream_returns_chunks TestBedrockNativetest_bedrock_native_e2e.py
  68. ✓test_messages_logs_cost_matching_the_response_header TestAnthropicMessagestest_messages_e2e.py
  69. ✓test_messages_returns_completion TestAnthropicMessagestest_messages_e2e.py
  70. ✓test_messages_streams_completion TestAnthropicMessagestest_messages_e2e.py
  71. ✓test_messages_tool_use TestAnthropicMessagestest_messages_e2e.py
  72. ✓test_missing_file_returns_error TestAudioTranscriptionstest_audio_transcriptions_e2e.py
  73. ✓test_missing_input_returns_error TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py
  74. ✓test_missing_messages_returns_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py
  75. ✓test_missing_model_returns_client_error TestChatCompletionsContracttest_chat_completions_contract_e2e.py
  76. ✓test_missing_model_returns_client_error TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py
  77. ✓test_missing_model_returns_client_error TestResponsestest_responses_e2e.py
  78. ✓test_missing_model_returns_error TestAudioTranscriptionstest_audio_transcriptions_e2e.py
  79. ✓test_missing_model_returns_error TestAnthropicMessagestest_messages_e2e.py
  80. ✓test_model_info_reports_custom_pricing TestCustomPricingtest_custom_pricing_e2e.py
  81. ✓test_moderations_flags_violent_content TestModerationstest_moderations_e2e.py
  82. ✓test_moderations_passes_benign_content TestModerationstest_moderations_e2e.py
  83. ✓test_multi_turn_history_is_honored TestChatCompletionsContracttest_chat_completions_contract_e2e.py
  84. ✓test_nova_sonic_response_create_completes TestNovaSonicRealtimerealtime/test_realtime_bedrock_e2e.py
  85. ✓test_openai_chat_logs_cost TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py
  86. ✓test_openai_chat_reasoning_reports_reasoning_tokens TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py
  87. ✓test_openai_chat_returns_tool_call TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py
  88. ✓test_openai_chat_streams_real_content TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py
  89. ✓test_openai_chat_streams_tool_call TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py
  90. ✓test_openai_chat_structured_output_conforms_to_schema TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py
  91. ✓test_openai_chat_vision_describes_image TestOpenAIChatCompletionstest_chat_completions_regression_e2e.py
  92. ✓test_openai_service_tier_is_echoed TestServiceTiertest_provider_features_e2e.py
  93. ✓test_passthrough_denies_model_outside_key_allowlist TestPassthroughModelAllowlisttest_passthrough_e2e.py
  94. ✓test_passthrough_prefix_lists_batches_from_openai TestOpenAIPassthroughPrefixtest_passthrough_e2e.py
  95. ✓test_passthrough_prefix_uploads_a_file_to_openai TestOpenAIPassthroughPrefixtest_passthrough_e2e.py
  96. ✓test_pipecat_tool_smoke[azure]realtime/test_realtime_pipecat_e2e.py
  97. ✓test_pipecat_tool_smoke[gemini]realtime/test_realtime_pipecat_e2e.py
  98. ✓test_pipecat_tool_smoke[openai]realtime/test_realtime_pipecat_e2e.py
  99. ✓test_reasoner_returns_reasoning_by_default TestDeepSeekReasoningDisabletest_deepseek_reasoning_e2e.py
  100. ✓test_reasoning_effort_none_disables_reasoning TestDeepSeekReasoningDisabletest_deepseek_reasoning_e2e.py
  101. ✓test_rerank_scores_top_n TestReranktest_rerank_e2e.py
  102. ✓test_responses_anthropic_returns_completion TestResponsestest_responses_e2e.py
  103. ✓test_responses_anthropic_returns_function_call TestResponsestest_responses_e2e.py
  104. ✓test_responses_bedrock_returns_completion TestResponsestest_responses_e2e.py
  105. ✓test_responses_bedrock_returns_function_call TestResponsestest_responses_e2e.py
  106. ✓test_responses_logs_cost TestResponsestest_responses_e2e.py
  107. ✓test_responses_returns_completion TestResponsestest_responses_e2e.py
  108. ✓test_responses_returns_function_call TestResponsestest_responses_e2e.py
  109. ✓test_responses_streaming_returns_completion TestResponsestest_responses_e2e.py
  110. ✓test_responses_vision_describes_image TestResponsestest_responses_e2e.py
  111. ✓test_retrieve_invalid_batch_id_returns_error TestFilesBatchesContracttest_files_batches_contract_e2e.py
  112. ✓test_rust_ocr_response[azure-ai] TestRustOcrGatewaytest_ocr_rust_e2e.py
  113. ✓test_rust_ocr_response[azure-document-intelligence] TestRustOcrGatewaytest_ocr_rust_e2e.py
  114. ✓test_rust_ocr_response[mistral] TestRustOcrGatewaytest_ocr_rust_e2e.py
  115. ✓test_rust_ocr_response[vertex-mistral] TestRustOcrGatewaytest_ocr_rust_e2e.py
  116. ✓test_stream_generate_content_frames_sse_the_way_google_sdks_expect TestGoogleNativeGenerateContenttest_google_native_e2e.py
  117. ✓test_streamed_responses_call_logs_its_cost TestOpenAIPassthroughSpendtest_passthrough_e2e.py
  118. ✓test_success_response_matches_chat_completion_contract TestChatCompletionsContracttest_chat_completions_contract_e2e.py
  119. ✓test_temperature_boundaries_succeed TestChatCompletionsContracttest_chat_completions_contract_e2e.py
  120. ✓test_text_completion_returns_text TestCompletionsEndpointtest_completions_endpoint_e2e.py
  121. ✓test_text_conversation[azure]realtime/test_realtime_e2e.py
  122. ✓test_text_conversation[gemini]realtime/test_realtime_e2e.py
  123. ✓test_text_conversation[openai]realtime/test_realtime_e2e.py
  124. ✓test_text_conversation[vertex_ai]realtime/test_realtime_e2e.py
  125. ✓test_thinking_disabled_disables_reasoning TestDeepSeekReasoningDisabletest_deepseek_reasoning_e2e.py
  126. ✓test_tool_call_round_trip[azure]realtime/test_realtime_e2e.py
  127. ✓test_tool_call_round_trip[gemini]realtime/test_realtime_e2e.py
  128. ✓test_tool_call_round_trip[openai]realtime/test_realtime_e2e.py
  129. ✓test_tool_call_round_trip[vertex_ai]realtime/test_realtime_e2e.py
  130. ✓test_tool_use_nonstream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py
  131. ✓test_tool_use_stream TestAzureFoundryMessagestest_messages_azure_foundry_e2e.py
  132. ✓test_upload_without_purpose_returns_error TestFilesBatchesContracttest_files_batches_contract_e2e.py
  133. ✓test_vertex_embeddings_returns_vector TestEmbeddingsEndpointtest_embeddings_endpoint_e2e.py
  134. ✓test_vertex_prompt_caching_reads_cache TestCacheControltest_cache_control.py
  135. −test_create_batch_missing_input_file_id_returns_error TestFilesBatchesContracttest_files_batches_contract_e2e.py›

    stage red: product gap, /v1/batches 500s (acreate_batch TypeError) on missing input_file_id instead of 400

  136. −test_empty_input_returns_error TestAudioSpeechtest_audio_speech_e2e.py›

    stage red: product gap, /v1/audio/speech 500s on empty input instead of surfacing the provider 4xx

  137. −test_flagged_model_keeps_prompt_cache_across_system_reminder TestBedrockInvokeMidConversationSystemtest_messages_mid_conversation_system_e2e.py›

    LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

  138. −test_flagged_model_keeps_prompt_cache_across_system_reminder TestAzureFoundryMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›

    LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

  139. −test_flagged_model_keeps_prompt_cache_across_system_reminder TestVertexMidConversationSystemtest_messages_mid_conversation_system_native_providers_e2e.py›

    LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

  140. −test_gemini_passthrough_returns_the_same_header_contract_as_the_managed_routetest_passthrough_e2e.py›

    stage red: product gap, native passthrough returns no x-litellm-response-cost or x-ratelimit-* headers

  141. −test_hosted_vllm_chat_returns_content TestHostedVllmChattest_chat_completions_regression_e2e.py›

    set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

  142. −test_invalid_response_id_returns_error TestResponsesRetrievetest_responses_retrieve_e2e.py›

    stage red: product gap (LIT-5447), retrieving an unknown response id returns 400 (model=None) instead of 404

  143. −test_invalid_voice_returns_error TestAudioSpeechtest_audio_speech_e2e.py›

    stage red: product gap, /v1/audio/speech 500s on invalid voice instead of surfacing the provider 4xx

  144. −test_missing_document_returns_error TestRustOcrGatewaytest_ocr_rust_e2e.py›

    stage red: product gap, /v1/ocr 500s (aocr TypeError) on missing document instead of 400

  145. −test_missing_input_returns_error TestAudioSpeechtest_audio_speech_e2e.py›

    stage red: product gap, /v1/audio/speech 500s on missing input instead of 400

  146. −test_missing_input_returns_error TestModerationstest_moderations_e2e.py›

    stage red: product gap, /v1/moderations 500s (KeyError 'input') on missing input instead of 400

  147. −test_missing_input_returns_error TestResponsestest_responses_e2e.py›

    stage red: product gap, /v1/responses 500s (aresponses TypeError) on missing input instead of 400

  148. −test_missing_max_tokens_returns_error TestAnthropicMessagestest_messages_e2e.py›

    stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing max_tokens instead of 400

  149. −test_missing_messages_returns_error TestAnthropicMessagestest_messages_e2e.py›

    stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing messages instead of 400

  150. −test_missing_model_returns_error TestAudioSpeechtest_audio_speech_e2e.py›

    stage red: product gap, /v1/audio/speech 500s on missing model instead of 400

  151. −test_missing_prompt_returns_error TestImageGenerationtest_image_generation_e2e.py›

    stage red: product gap, /v1/images/generations 500s (aimage_generation TypeError) on missing prompt instead of 400

  152. −test_retrieve_invalid_id_returns_error TestVectorStorestest_vector_stores_e2e.py›

    stage red: product gap, retrieving a nonexistent vector store returns 2xx with an error envelope in the body instead of 404

  153. −test_search_missing_query_returns_error TestVectorStorestest_vector_stores_e2e.py›

    stage red: product gap, vector store search 500s (asearch TypeError) on missing query instead of 400

  154. −test_static_and_x_pass_headers_reach_upstream TestPassthroughHeaderstest_passthrough_headers_e2e.py›

    stage red: ALB sends custom passthrough paths to backend pods, which lack ANTHROPIC_API_KEY, so the os.environ header forwards unresolved and Anthropic 401s; routing user-defined paths to the gateway needs a product/infra decision (fixed path namespace or dynamic ingress)

  155. −test_store_and_retrieve_by_id TestResponsesRetrievetest_responses_retrieve_e2e.py›

    stage red: product gap (LIT-5446), retrieve returns a different id than the stored response (non-idempotent response-id re-encryption)

  156. −test_vertex_passthrough_via_managed_model_logs_cost TestVertexPassthroughSpendTrackingtest_vertex_passthrough_e2e.py›

    set VERTEXAI_PROJECT (the project the vertex deployment bills)

  157. −test_web_search_server_tool_is_served TestBedrockWebSearchServerTooltest_bedrock_web_search_server_tool_e2e.py›

    stage red: environment gap, the e2e stack neither enables the websearch_interception callback nor declares a search backend, so the request reaches bedrock's transformation and takes its by-design 400. Unskip once the ephemeral stack ships the config in this module's docstring.

  158. ✓test_a_failed_first_turn_is_reported_as_the_first_turn_probe_unit_tests/test_http_probe.py
  159. ✓test_a_history_with_no_server_tool_block_is_not_a_pass_probe_unit_tests/test_http_probe.py
  160. ✓test_a_pending_tool_use_is_answered_with_the_id_the_model_returned_probe_unit_tests/test_http_probe.py
  161. ✓test_a_turn_truncated_before_the_result_block_is_not_a_pass_probe_unit_tests/test_http_probe.py
  162. ✓test_a_turn_with_no_pending_tool_use_gets_a_plain_follow_up_probe_unit_tests/test_http_probe.py
  163. ✓test_accepts_a_replayed_server_tool_pair_probe_unit_tests/test_http_probe.py
  164. ✓test_basic_messaging_non_streaming_anthropicbasic_messaging_non_streaming/test_anthropic.py
  165. ✓test_basic_messaging_non_streaming_azurebasic_messaging_non_streaming/test_azure.py
  166. ✓test_basic_messaging_non_streaming_azure_openaibasic_messaging_non_streaming/test_azure_openai.py
  167. ✓test_basic_messaging_non_streaming_bedrock_conversebasic_messaging_non_streaming/test_bedrock_converse.py
  168. ✓test_basic_messaging_non_streaming_bedrock_invokebasic_messaging_non_streaming/test_bedrock_invoke.py
  169. ✓test_basic_messaging_non_streaming_vertex_aibasic_messaging_non_streaming/test_vertex_ai.py
  170. ✓test_basic_messaging_non_streaming_vertex_ai_gptbasic_messaging_non_streaming/test_vertex_ai_gpt.py
  171. ✓test_basic_messaging_streaming_anthropicbasic_messaging_streaming/test_anthropic.py
  172. ✓test_basic_messaging_streaming_azurebasic_messaging_streaming/test_azure.py
  173. ✓test_basic_messaging_streaming_azure_openaibasic_messaging_streaming/test_azure_openai.py
  174. ✓test_basic_messaging_streaming_bedrock_conversebasic_messaging_streaming/test_bedrock_converse.py
  175. ✓test_basic_messaging_streaming_bedrock_invokebasic_messaging_streaming/test_bedrock_invoke.py
  176. ✓test_basic_messaging_streaming_vertex_aibasic_messaging_streaming/test_vertex_ai.py
  177. ✓test_basic_messaging_streaming_vertex_ai_gptbasic_messaging_streaming/test_vertex_ai_gpt.py
  178. ✓test_count_tokens_anthropiccount_tokens/test_anthropic.py
  179. ✓test_count_tokens_azurecount_tokens/test_azure.py
  180. ✓test_count_tokens_bedrock_conversecount_tokens/test_bedrock_converse.py
  181. ✓test_count_tokens_bedrock_invokecount_tokens/test_bedrock_invoke.py
  182. ✓test_find_regressions_empty_old_matrix_is_safe_builder_unit_tests/test_matrix_builder.py
  183. ✓test_find_regressions_flags_pass_to_fail_builder_unit_tests/test_matrix_builder.py
  184. ✓test_find_regressions_ignores_green_to_grey_builder_unit_tests/test_matrix_builder.py
  185. ✓test_find_regressions_ignores_improvements_and_steady_green_builder_unit_tests/test_matrix_builder.py
  186. ✓test_find_regressions_ignores_new_cells_without_baseline_builder_unit_tests/test_matrix_builder.py
  187. ✓test_find_regressions_ignores_red_to_red_builder_unit_tests/test_matrix_builder.py
  188. ✓test_find_regressions_matches_by_id_not_name_builder_unit_tests/test_matrix_builder.py
  189. ✓test_find_regressions_reports_multiple_sorted_builder_unit_tests/test_matrix_builder.py
  190. ✓test_passthrough_anthropicpassthrough/test_anthropic.py
  191. ✓test_passthrough_bedrock_conversepassthrough/test_bedrock_converse.py
  192. ✓test_passthrough_bedrock_invokepassthrough/test_bedrock_invoke.py
  193. ✓test_passthrough_vertex_aipassthrough/test_vertex_ai.py
  194. ✓test_pdf_input_anthropicpdf_input/test_anthropic.py
  195. ✓test_pdf_input_azurepdf_input/test_azure.py
  196. ✓test_pdf_input_bedrock_conversepdf_input/test_bedrock_converse.py
  197. ✓test_pdf_input_bedrock_invokepdf_input/test_bedrock_invoke.py
  198. ✓test_pdf_input_vertex_aipdf_input/test_vertex_ai.py
  199. ✓test_prompt_caching_1h_anthropicprompt_caching_1h/test_anthropic.py
  200. ✓test_prompt_caching_1h_azureprompt_caching_1h/test_azure.py
  201. ✓test_prompt_caching_1h_bedrock_converseprompt_caching_1h/test_bedrock_converse.py
  202. ✓test_prompt_caching_1h_bedrock_invokeprompt_caching_1h/test_bedrock_invoke.py
  203. ✓test_prompt_caching_1h_vertex_aiprompt_caching_1h/test_vertex_ai.py
  204. ✓test_prompt_caching_5m_anthropicprompt_caching_5m/test_anthropic.py
  205. ✓test_prompt_caching_5m_azureprompt_caching_5m/test_azure.py
  206. ✓test_prompt_caching_5m_bedrock_converseprompt_caching_5m/test_bedrock_converse.py
  207. ✓test_prompt_caching_5m_bedrock_invokeprompt_caching_5m/test_bedrock_invoke.py
  208. ✓test_prompt_caching_5m_vertex_aiprompt_caching_5m/test_vertex_ai.py
  209. ✓test_reports_the_status_when_the_replayed_history_is_rejected_probe_unit_tests/test_http_probe.py
  210. ✓test_structured_outputs_anthropicstructured_outputs/test_anthropic.py
  211. ✓test_structured_outputs_azurestructured_outputs/test_azure.py
  212. ✓test_structured_outputs_bedrock_conversestructured_outputs/test_bedrock_converse.py
  213. ✓test_structured_outputs_bedrock_invokestructured_outputs/test_bedrock_invoke.py
  214. ✓test_structured_outputs_vertex_aistructured_outputs/test_vertex_ai.py
  215. ✓test_thinking_anthropicthinking/test_anthropic.py
  216. ✓test_thinking_azurethinking/test_azure.py
  217. ✓test_thinking_bedrock_conversethinking/test_bedrock_converse.py
  218. ✓test_thinking_bedrock_invokethinking/test_bedrock_invoke.py
  219. ✓test_thinking_vertex_aithinking/test_vertex_ai.py
  220. ✓test_thinking_with_tool_use_anthropicthinking_with_tool_use/test_anthropic.py
  221. ✓test_thinking_with_tool_use_azurethinking_with_tool_use/test_azure.py
  222. ✓test_thinking_with_tool_use_bedrock_conversethinking_with_tool_use/test_bedrock_converse.py
  223. ✓test_thinking_with_tool_use_bedrock_invokethinking_with_tool_use/test_bedrock_invoke.py
  224. ✓test_thinking_with_tool_use_vertex_aithinking_with_tool_use/test_vertex_ai.py
  225. ✓test_tool_search_anthropictool_search/test_anthropic.py
  226. ✓test_tool_search_bedrock_conversetool_search/test_bedrock_converse.py
  227. ✓test_tool_search_bedrock_invoketool_search/test_bedrock_invoke.py
  228. ✓test_tool_search_history_bedrock_invoketool_search/test_bedrock_invoke.py
  229. ✓test_tool_use_anthropictool_use/test_anthropic.py
  230. ✓test_tool_use_azuretool_use/test_azure.py
  231. ✓test_tool_use_azure_openaitool_use/test_azure_openai.py
  232. ✓test_tool_use_bedrock_conversetool_use/test_bedrock_converse.py
  233. ✓test_tool_use_bedrock_invoketool_use/test_bedrock_invoke.py
  234. ✓test_tool_use_streaming_anthropictool_use_streaming/test_anthropic.py
  235. ✓test_tool_use_streaming_azuretool_use_streaming/test_azure.py
  236. ✓test_tool_use_streaming_azure_openaitool_use_streaming/test_azure_openai.py
  237. ✓test_tool_use_streaming_bedrock_conversetool_use_streaming/test_bedrock_converse.py
  238. ✓test_tool_use_streaming_bedrock_invoketool_use_streaming/test_bedrock_invoke.py
  239. ✓test_tool_use_streaming_vertex_aitool_use_streaming/test_vertex_ai.py
  240. ✓test_tool_use_streaming_vertex_ai_gpttool_use_streaming/test_vertex_ai_gpt.py
  241. ✓test_tool_use_vertex_aitool_use/test_vertex_ai.py
  242. ✓test_tool_use_vertex_ai_gpttool_use/test_vertex_ai_gpt.py
  243. ✓test_vision_anthropicvision/test_anthropic.py
  244. ✓test_vision_azurevision/test_azure.py
  245. ✓test_vision_bedrock_conversevision/test_bedrock_converse.py
  246. ✓test_vision_bedrock_invokevision/test_bedrock_invoke.py
  247. ✓test_vision_vertex_aivision/test_vertex_ai.py
  248. ✓test_web_search_anthropicweb_search/test_anthropic.py
  249. ✓test_web_search_azureweb_search/test_azure.py
  250. ✓test_web_search_bedrock_converseweb_search/test_bedrock_converse.py
  251. ✓test_web_search_bedrock_invokeweb_search/test_bedrock_invoke.py
  252. ✓test_web_search_vertex_aiweb_search/test_vertex_ai.py
  253. −test_basic_messaging_non_streaming_bedrock_mantlebasic_messaging_non_streaming/test_bedrock_mantle.py›

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

  254. −test_basic_messaging_non_streaming_openaibasic_messaging_non_streaming/test_openai.py›

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

  255. −test_basic_messaging_streaming_bedrock_mantlebasic_messaging_streaming/test_bedrock_mantle.py›

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

  256. −test_basic_messaging_streaming_openaibasic_messaging_streaming/test_openai.py›

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

  257. −test_count_tokens_vertex_aicount_tokens/test_vertex_ai.py›

    stage red: Vertex returns not supported for token counting for Claude aliases

  258. −test_long_context_1m_anthropiclong_context_1m/test_anthropic.py›

    stage red: 1M long_context not green on stage Anthropic path yet (200k sonnet / model alias)

  259. −test_long_context_1m_azurelong_context_1m/test_azure.py›

    stage red: 1M long_context not green on stage Azure Foundry deployments yet

  260. −test_long_context_1m_bedrock_converselong_context_1m/test_bedrock_converse.py›

    stage red: 1M long_context not green on stage Bedrock Converse deployments yet

  261. −test_long_context_1m_bedrock_invokelong_context_1m/test_bedrock_invoke.py›

    stage red: 1M long_context not green on stage Bedrock Invoke deployments yet

  262. −test_long_context_1m_vertex_ailong_context_1m/test_vertex_ai.py›

    stage red: 1M long_context not green on stage Vertex deployments yet

  263. −test_passthrough_azurepassthrough/test_azure.py›

    stage red: /azure passthrough drops client headers (e.g. anthropic-version); product gap

  264. −test_tool_search_azuretool_search/test_azure.py›

    stage red: Azure Foundry tool_search_server not supported in workspace for probed models

  265. −test_tool_search_vertex_aitool_search/test_vertex_ai.py›

    stage red: Vertex rejects tool_search when deployment extra_headers inject context-1m beta; product/config

  266. −test_tool_use_bedrock_mantletool_use/test_bedrock_mantle.py›

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

  267. −test_tool_use_openaitool_use/test_openai.py›

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

  268. −test_tool_use_streaming_bedrock_mantletool_use_streaming/test_bedrock_mantle.py›

    Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

  269. −test_tool_use_streaming_openaitool_use_streaming/test_openai.py›

    OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

  270. ✓test_bare_key_blocks_over_its_own_budget TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py
  271. ✓test_bare_key_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py
  272. ✓test_blocked_member_does_not_block_peer TestTeamMemberBudgetIsolationbudgets/test_team_member_budget_isolation_e2e.py
  273. ✓test_budget_crud_roundtripbudgets/test_budget_crud_e2e.py
  274. ✓test_budget_delete_removes_itbudgets/test_budget_crud_e2e.py
  275. ✓test_budget_duration_schedules_reset_on_keybudgets/test_budget_crud_e2e.py
  276. ✓test_budget_fallback_reroutes_anthropic_messages_to_openaibudgets/test_budget_fallback_e2e.py
  277. ✓test_burst_of_concurrent_calls_loses_no_spendspend_tracking/test_spend_tracking_e2e.py
  278. ✓test_burst_rate_limit_does_not_freeze_fresh_key TestRedisCircuitBreakerPathratelimit/test_redis_circuit_breaker_e2e.py
  279. ✓test_cache_hit_is_zero_cost_and_suffixedspend_tracking/test_spend_tracking_e2e.py
  280. ✓test_cache_hit_reduces_tpm_by_non_cached_only TestTpmExcludesCachedTokensratelimit/test_tpm_excludes_cached_tokens_e2e.py
  281. ✓test_cache_write_tokens_billed_at_cache_creation_rate TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py
  282. ✓test_chat_completion_writes_nonzero_spend_rowspend_tracking/test_spend_tracking_e2e.py
  283. ✓test_cold_counter_reseed_keeps_counter_equal_to_db_spendbudgets/test_spend_counter_reseed_e2e.py
  284. ✓test_component_cost_headers_sum_to_total TestCostHeadersspend_tracking/test_cost_headers_e2e.py
  285. ✓test_cost_breakdown_reports_component_costs TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py
  286. ✓test_each_model_on_a_shared_key_gets_its_own_rowspend_tracking/test_spend_tracking_e2e.py
  287. ✓test_edge_wired_chat_writes_nonzero_spend_rowspend_tracking/test_provider_edge_spend_e2e.py
  288. ✓test_embedding_writes_nonzero_spend_rowspend_tracking/test_spend_tracking_e2e.py
  289. ✓test_end_user_budget_blocks_attributed_calls TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py
  290. ✓test_end_user_spend_attributed_on_rowspend_tracking/test_spend_tracking_e2e.py
  291. ✓test_failure_call_writes_failure_status_rowspend_tracking/test_spend_tracking_e2e.py
  292. ✓test_generous_mode_lets_priority_borrow_past_reservation TestDynamicRateLimitPriorityratelimit/test_dynamic_rate_limit_priority_e2e.py
  293. ✓test_headers_report_limit_and_remaining TestKeyRateLimitsratelimit/test_rate_limit_e2e.py
  294. ✓test_key_budget_reset_at_advances_after_windowbudgets/test_budget_reset_advances_e2e.py
  295. ✓test_key_spend_blocks_at_capbudgets/test_budget_reset_advances_e2e.py
  296. ✓test_key_spend_equals_sum_of_logsspend_tracking/test_spend_tracking_e2e.py
  297. ✓test_key_with_budget_duration_schedules_reset_at_creationbudgets/test_budget_reset_advances_e2e.py
  298. ✓test_long_window_blocks_after_short_window_resetsbudgets/test_multi_window_budget_e2e.py
  299. ✓test_member_budget_blocks_without_touching_teammates TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py
  300. ✓test_member_spend_attributed_to_team_and_user TestTeamMemberBudgetbudgets/test_team_member_budget_e2e.py
  301. ✓test_member_spend_over_budget_is_blocked TestTeamMemberBudgetbudgets/test_team_member_budget_e2e.py
  302. ✓test_messages_bridge_keeps_cache_tokens TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py
  303. ✓test_missing_end_date_is_rejected TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py
  304. ✓test_missing_start_date_is_rejected TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py
  305. ✓test_model_max_budget_isolates_per_modelbudgets/test_model_max_budget_e2e.py
  306. ✓test_multi_window_key_resets_each_window_independentlybudgets/test_budget_reset_advances_e2e.py
  307. ✓test_org_budget_blocks_keys_under_it TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py
  308. ✓test_org_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py
  309. ✓test_personal_key_blocks_over_its_own_budget TestKeyBudgetBlocksAcrossKeyKindsbudgets/test_budget_enforcement_e2e.py
  310. ✓test_personal_key_resets_after_window TestKeyBudgetResetAcrossKeyKindsbudgets/test_budget_reset_e2e.py
  311. ✓test_personal_key_user_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py
  312. ✓test_priority_tier_bills_priority_rates TestServiceTierPricingspend_tracking/test_service_tier_pricing_e2e.py
  313. ✓test_request_tags_round_tripspend_tracking/test_spend_tracking_e2e.py
  314. ✓test_reset_wait_never_yields_non_budget_errorbudgets/test_budget_reset_advances_e2e.py
  315. ✓test_rpm_limit_blocks_over_limit TestKeyRateLimitsratelimit/test_rate_limit_e2e.py
  316. ✓test_rpm_limit_one_blocks_second_call TestRedisBackedRateLimitratelimit/test_redis_backed_ratelimit_e2e.py
  317. ✓test_rpm_limit_resets_after_window TestKeyRateLimitsratelimit/test_rate_limit_e2e.py
  318. ✓test_schema_listed_spend_routes_are_responsivespend_tracking/test_spend_routes.py
  319. ✓test_short_window_blocks_then_resetsbudgets/test_multi_window_budget_e2e.py
  320. ✓test_soft_budget_does_not_blockbudgets/test_soft_budget_e2e.py
  321. ✓test_spend_calculate_returns_nonzero_costspend_tracking/test_spend_tracking_e2e.py
  322. ✓test_spend_logs_endpoint_returns_spendspend_tracking/test_spend_tracking_e2e.py
  323. ✓test_spend_logs_v2_pagination_caps_pages_and_keeps_totalspend_tracking/test_spend_tracking_e2e.py
  324. ✓test_spend_route_responsive[/customer/daily/activity]spend_tracking/test_spend_routes.py
  325. ✓test_spend_route_responsive[/customer/list]spend_tracking/test_spend_routes.py
  326. ✓test_spend_route_responsive[/end_user/daily/activity]spend_tracking/test_spend_routes.py
  327. ✓test_spend_route_responsive[/global/activity]spend_tracking/test_spend_routes.py
  328. ✓test_spend_route_responsive[/global/activity/exceptions]spend_tracking/test_spend_routes.py
  329. ✓test_spend_route_responsive[/global/activity/exceptions/deployment]spend_tracking/test_spend_routes.py
  330. ✓test_spend_route_responsive[/global/activity/model]spend_tracking/test_spend_routes.py
  331. ✓test_spend_route_responsive[/global/all_end_users]spend_tracking/test_spend_routes.py
  332. ✓test_spend_route_responsive[/global/spend/all_tag_names]spend_tracking/test_spend_routes.py
  333. ✓test_spend_route_responsive[/global/spend/provider]spend_tracking/test_spend_routes.py
  334. ✓test_spend_route_responsive[/global/spend/report]spend_tracking/test_spend_routes.py
  335. ✓test_spend_route_responsive[/global/spend/teams]spend_tracking/test_spend_routes.py
  336. ✓test_spend_route_responsive[/key/list]spend_tracking/test_spend_routes.py
  337. ✓test_spend_route_responsive[/organization/daily/activity]spend_tracking/test_spend_routes.py
  338. ✓test_spend_route_responsive[/organization/list]spend_tracking/test_spend_routes.py
  339. ✓test_spend_route_responsive[/spend/keys]spend_tracking/test_spend_routes.py
  340. ✓test_spend_route_responsive[/spend/logs]spend_tracking/test_spend_routes.py
  341. ✓test_spend_route_responsive[/spend/logs/session/ui]spend_tracking/test_spend_routes.py
  342. ✓test_spend_route_responsive[/spend/logs/ui]spend_tracking/test_spend_routes.py
  343. ✓test_spend_route_responsive[/spend/logs/v2]spend_tracking/test_spend_routes.py
  344. ✓test_spend_route_responsive[/spend/tags]spend_tracking/test_spend_routes.py
  345. ✓test_spend_route_responsive[/spend/users]spend_tracking/test_spend_routes.py
  346. ✓test_spend_route_responsive[/tag/daily/activity]spend_tracking/test_spend_routes.py
  347. ✓test_spend_route_responsive[/team/daily/activity]spend_tracking/test_spend_routes.py
  348. ✓test_spend_route_responsive[/team/list]spend_tracking/test_spend_routes.py
  349. ✓test_spend_route_responsive[/user/daily/activity]spend_tracking/test_spend_routes.py
  350. ✓test_spend_route_responsive[/user/daily/activity/aggregated]spend_tracking/test_spend_routes.py
  351. ✓test_spend_route_responsive[/user/list]spend_tracking/test_spend_routes.py
  352. ✓test_streaming_cache_read_billed_at_cache_read_rate TestCacheCostAccountingspend_tracking/test_cache_cost_accounting_e2e.py
  353. ✓test_streaming_chat_completion_tracks_spendspend_tracking/test_spend_tracking_e2e.py
  354. ✓test_streaming_messages_via_responses_bridge_tracks_spendspend_tracking/test_spend_tracking_e2e.py
  355. ✓test_tag_budget_blocks_tagged_requestsbudgets/test_tag_budget_e2e.py
  356. ✓test_tag_spend_matches_sum_of_tagged_logsspend_tracking/test_spend_tracking_e2e.py
  357. ✓test_team_budget_blocks_every_team_key TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py
  358. ✓test_team_budget_resets_after_window TestBudgetResetPerLevelbudgets/test_budget_reset_e2e.py
  359. ✓test_team_key_blocks_over_its_own_budget TestKeyBudgetBlocksAcrossKeyKindsbudgets/test_budget_enforcement_e2e.py
  360. ✓test_team_key_resets_after_window TestKeyBudgetResetAcrossKeyKindsbudgets/test_budget_reset_e2e.py
  361. ✓test_team_long_window_blocks_after_short_window_resetsbudgets/test_team_multi_window_budget_e2e.py
  362. ✓test_team_member_budget_reset_at_advancesbudgets/test_budget_reset_advances_e2e.py
  363. ✓test_team_member_budget_reset_keeps_advancingbudgets/test_team_member_budget_reset_e2e.py
  364. ✓test_team_member_key_blocks_over_its_own_budget TestKeyBudgetBlocksAcrossKeyKindsbudgets/test_budget_enforcement_e2e.py
  365. ✓test_team_member_key_resets_after_window TestKeyBudgetResetAcrossKeyKindsbudgets/test_budget_reset_e2e.py
  366. ✓test_team_short_window_blocks_then_resetsbudgets/test_team_multi_window_budget_e2e.py
  367. ✓test_tpm_limit_blocks_over_limit TestKeyRateLimitsratelimit/test_rate_limit_e2e.py
  368. ✓test_user_budget_blocks_a_second_key TestUserBudgetAcrossKeysbudgets/test_user_budget_across_keys_e2e.py
  369. ✓test_user_budget_enforced_across_their_personal_keys TestBudgetBlocksPerLevelbudgets/test_budget_enforcement_e2e.py
  370. ✓test_valid_date_range_returns_results_and_metadata[1] TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py
  371. ✓test_valid_date_range_returns_results_and_metadata[30] TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py
  372. ✓test_valid_date_range_returns_results_and_metadata[7] TestTeamDailyActivityspend_tracking/test_team_daily_activity_e2e.py
  373. −test_end_user_model_max_budget_enforces_per_model_rpmbudgets/test_model_max_budget_e2e.py›

    stage red: product gap, end-user model_max_budget rpm_limit is stored but never enforced

  374. −test_spend_route_responsive[/global/spend]spend_tracking/test_spend_routes.py›

    LIT-5211: on a fresh database the proxy's startup view creation can lose the race against schema migrations, leaving MonthlyGlobalSpend/DailyTagSpend/Last30d* views missing and these routes 500ing until the views exist

  375. −test_spend_route_responsive[/global/spend/keys]spend_tracking/test_spend_routes.py›

    LIT-5211: on a fresh database the proxy's startup view creation can lose the race against schema migrations, leaving MonthlyGlobalSpend/DailyTagSpend/Last30d* views missing and these routes 500ing until the views exist

  376. −test_spend_route_responsive[/global/spend/logs]spend_tracking/test_spend_routes.py›

    LIT-5211: on a fresh database the proxy's startup view creation can lose the race against schema migrations, leaving MonthlyGlobalSpend/DailyTagSpend/Last30d* views missing and these routes 500ing until the views exist

  377. −test_spend_route_responsive[/global/spend/models]spend_tracking/test_spend_routes.py›

    LIT-5211: on a fresh database the proxy's startup view creation can lose the race against schema migrations, leaving MonthlyGlobalSpend/DailyTagSpend/Last30d* views missing and these routes 500ing until the views exist

  378. −test_spend_route_responsive[/global/spend/tags]spend_tracking/test_spend_routes.py›

    LIT-5211: on a fresh database the proxy's startup view creation can lose the race against schema migrations, leaving MonthlyGlobalSpend/DailyTagSpend/Last30d* views missing and these routes 500ing until the views exist

  379. −test_strict_mode_blocks_saturated_priority_but_serves_the_other TestDynamicRateLimitPriorityratelimit/test_dynamic_rate_limit_priority_e2e.py›

    LIT-5118: the stage proxy does not run the dynamic_rate_limiter_v3 callbacks + priority_reservation config this module's docstring requires (zero limiter log lines on any pod during the 2026-08-02 run), so strict enforcement can never engage there

  380. ✓test_access_group_info_reports_membership TestModelAccessGroupRoutestest_model_tag_accessgroup_e2e.py
  381. ✓test_approve_activates_submission_and_persists TestMcpServerSubmissiontest_config_misc_endpoints_e2e.py
  382. ✓test_block_persists_to_key_info TestKeyRoutestest_management_e2e.py
  383. ✓test_block_then_unblock_persists_to_model_info TestModelRoutestest_model_tag_accessgroup_e2e.py
  384. ✓test_block_then_unblock_persists_to_team_info TestTeamManagementRoutestest_team_management_e2e.py
  385. ✓test_bulk_update_applies_max_budget_to_target_key TestKeyManagementRoutestest_key_management_e2e.py
  386. ✓test_callbacks_list_reports_active_logging_callbacks TestInventoryRoutestest_config_misc_endpoints_e2e.py
  387. ✓test_config_update_persists_router_setting_to_get TestRouterSettingstest_config_misc_endpoints_e2e.py
  388. ✓test_create_persists_and_is_read_back TestFallbackManagementtest_config_misc_endpoints_e2e.py
  389. ✓test_created_budget_appears_in_budget_list TestBudgetManagementtest_budget_customer_user_org_e2e.py
  390. ✓test_created_key_appears_in_key_list_inventory TestKeyRoutestest_management_e2e.py
  391. ✓test_created_team_appears_in_team_list TestTeamRoutestest_management_e2e.py
  392. ✓test_created_users_appear_in_user_list TestUserRoutestest_management_e2e.py
  393. ✓test_credential_migration_check_reports_residual_scan TestInventoryRoutestest_config_misc_endpoints_e2e.py
  394. ✓test_customer_create_persists_to_info TestCustomertest_management_e2e.py
  395. ✓test_delete_forbidden_for_non_admin_key TestKeyManagementRoutestest_key_management_e2e.py
  396. ✓test_delete_persists_and_revokes_team_bound_key TestTeamRoutestest_management_e2e.py
  397. ✓test_delete_removes_from_model_info_catalog TestModelRoutestest_management_e2e.py
  398. ✓test_delete_removes_from_organization_info TestOrganizationRoutestest_management_e2e.py
  399. ✓test_delete_removes_the_customer TestCustomerManagementtest_budget_customer_user_org_e2e.py
  400. ✓test_delete_removes_the_user_from_inventory TestUserRoutestest_management_e2e.py
  401. ✓test_delete_revokes_the_key_on_chat TestKeyRoutestest_management_e2e.py
  402. ✓test_end_user_new_persists_to_end_user_info TestCustomerManagementtest_budget_customer_user_org_e2e.py
  403. ✓test_estimate_computes_cost_from_token_counts TestCostEstimatetest_config_misc_endpoints_e2e.py
  404. ✓test_gdpr_check_derives_verdict_from_the_request TestComplianceRoutestest_config_misc_endpoints_e2e.py
  405. ✓test_generate_forbidden_for_non_admin_key TestKeyManagementRoutestest_key_management_e2e.py
  406. ✓test_generate_persists_to_key_info_and_scopes_chat TestKeyRoutestest_management_e2e.py
  407. ✓test_health_reports_the_calling_key_healthy TestKeyManagementRoutestest_key_management_e2e.py
  408. ✓test_info_reflects_the_fields_the_key_was_created_with TestKeyManagementRoutestest_key_management_e2e.py
  409. ✓test_info_returns_created_team_fields TestTeamManagementRoutestest_team_management_e2e.py
  410. ✓test_is_null_finds_the_budget_left_uncapped TestBudgetListV1test_budget_customer_user_org_e2e.py
  411. ✓test_is_refused_for_a_non_admin_key TestBudgetListV1test_budget_customer_user_org_e2e.py
  412. ✓test_llm_only_key_forbidden_from_management_writes TestManagementRoutePermissionstest_management_e2e.py
  413. ✓test_member_add_and_delete_persist_to_team_info TestTeamRoutestest_management_e2e.py
  414. ✓test_member_add_forbidden_to_plain_member TestTeamManagementRoutestest_team_management_e2e.py
  415. ✓test_member_add_records_membership TestOrganizationMembershiptest_budget_customer_user_org_e2e.py
  416. ✓test_member_delete_forbidden_to_plain_member TestTeamManagementRoutestest_team_management_e2e.py
  417. ✓test_member_delete_persists_to_team_info TestTeamManagementRoutestest_team_management_e2e.py
  418. ✓test_member_update_persists_role_and_budget TestTeamManagementRoutestest_team_management_e2e.py
  419. ✓test_new_access_group_tags_the_deployment TestModelAccessGroupRoutestest_model_tag_accessgroup_e2e.py
  420. ✓test_new_is_denied_to_non_admin_keys TestTeamManagementRoutestest_team_management_e2e.py
  421. ✓test_new_is_refused_for_a_non_admin_key TestBudgetManagementtest_budget_customer_user_org_e2e.py
  422. ✓test_new_persists_mapping_and_is_read_back TestJwtKeyMappingtest_config_misc_endpoints_e2e.py
  423. ✓test_new_persists_to_customer_info TestCustomerManagementtest_budget_customer_user_org_e2e.py
  424. ✓test_new_persists_to_model_info_catalog TestModelRoutestest_management_e2e.py
  425. ✓test_new_persists_to_organization_info TestOrganizationRoutestest_management_e2e.py
  426. ✓test_new_persists_to_tag_list TestTagRoutestest_management_e2e.py
  427. ✓test_new_persists_to_team_info_and_binds_keys TestTeamRoutestest_management_e2e.py
  428. ✓test_new_persists_to_user_info TestUserRoutestest_management_e2e.py
  429. ✓test_new_user_is_readable_via_user_info TestUserManagementtest_budget_customer_user_org_e2e.py
  430. ✓test_non_admin_key_cannot_add_global_model TestModelRoutestest_model_tag_accessgroup_e2e.py
  431. ✓test_refuses_a_sort_field_and_a_parameter_it_does_not_support TestBudgetListV1test_budget_customer_user_org_e2e.py
  432. ✓test_regenerate_rotates_to_a_working_new_key TestKeyRegenerationtest_management_e2e.py
  433. ✓test_register_submits_pending_server TestMcpServerSubmissiontest_config_misc_endpoints_e2e.py
  434. ✓test_sorts_pages_and_filters_the_budgets_it_created TestBudgetListV1test_budget_customer_user_org_e2e.py
  435. ✓test_tag_delete_removes_from_list TestTagRoutestest_model_tag_accessgroup_e2e.py
  436. ✓test_tag_list_reports_created_tag TestTagRoutestest_model_tag_accessgroup_e2e.py
  437. ✓test_tool_list_returns_catalog_with_consistent_total TestInventoryRoutestest_config_misc_endpoints_e2e.py
  438. ✓test_unblock_flips_key_info_blocked_back TestKeyManagementRoutestest_key_management_e2e.py
  439. ✓test_update_alias_persists_to_organization_info TestOrganizationRoutestest_management_e2e.py
  440. ✓test_update_forbidden_for_non_admin_key TestKeyManagementRoutestest_key_management_e2e.py
  441. ✓test_update_max_budget_persists_to_budget_info TestBudgetManagementtest_budget_customer_user_org_e2e.py
  442. ✓test_update_models_persists_and_flips_enforcement TestKeyRoutestest_management_e2e.py
  443. ✓test_update_persists_input_cost_to_model_info TestModelRoutestest_management_e2e.py
  444. ✓test_update_persists_to_team_info TestTeamRoutestest_management_e2e.py
  445. ✓test_update_persists_to_user_info TestUserRoutestest_management_e2e.py
  446. ✓test_workflow_runs_list_returns_consistent_count TestInventoryRoutestest_config_misc_endpoints_e2e.py
  447. −test_update_accepts_per_model_budgets_including_punctuated_names TestBudgetManagementtest_budget_customer_user_org_e2e.py›

    stage red: product gap, /budget/update 500s on any model_max_budget (prisma Json arg + unquoted GraphQL interpolation)

  448. ✓test_a_kept_header_is_identity TestKeyDistinctnesstest_fixture_canonical.py
  449. ✓test_a_volatile_header_is_not_identity TestKeyDistinctnesstest_fixture_canonical.py
  450. ✓test_file_content_is_identity TestKeyDistinctnesstest_fixture_canonical.py
  451. ✓test_form_fields_are_identity TestKeyDistinctnesstest_fixture_canonical.py
  452. ✓test_generated_ids_in_the_path_do_not_change_the_key TestKeyStabilitytest_fixture_canonical.py
  453. ✓test_key_names_method_path_and_digest TestKeyShapetest_fixture_canonical.py
  454. ✓test_names_that_carry_credentials[api_base-False] TestSecretFieldstest_fixture_canonical.py
  455. ✓test_names_that_carry_credentials[api_key-True] TestSecretFieldstest_fixture_canonical.py
  456. ✓test_names_that_carry_credentials[aws_secret_access_key-True] TestSecretFieldstest_fixture_canonical.py
  457. ✓test_names_that_carry_credentials[aws_session_token-True] TestSecretFieldstest_fixture_canonical.py
  458. ✓test_names_that_carry_credentials[langfuse_secret_key-True] TestSecretFieldstest_fixture_canonical.py
  459. ✓test_names_that_carry_credentials[max_completion_tokens-False] TestSecretFieldstest_fixture_canonical.py
  460. ✓test_names_that_carry_credentials[model-False] TestSecretFieldstest_fixture_canonical.py
  461. ✓test_names_that_carry_credentials[openai_api_key-True] TestSecretFieldstest_fixture_canonical.py
  462. ✓test_names_that_carry_credentials[static_headers-True] TestSecretFieldstest_fixture_canonical.py
  463. ✓test_names_that_carry_credentials[vertex_credentials-True] TestSecretFieldstest_fixture_canonical.py
  464. ✓test_query_params_are_identity TestKeyDistinctnesstest_fixture_canonical.py
  465. ✓test_requests_differing_only_inside_canonicalized_fields_stay_distinct TestKeyDistinctnesstest_fixture_canonical.py
  466. ✓test_rewrites_exactly_the_volatile_shapes[0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef-<sha256>] TestPlaceholderstest_fixture_canonical.py
  467. ✓test_rewrites_exactly_the_volatile_shapes[2026-08-19-<date>] TestPlaceholderstest_fixture_canonical.py
  468. ✓test_rewrites_exactly_the_volatile_shapes[2026-08-19T20:57:13.363499+00:00-<timestamp>] TestPlaceholderstest_fixture_canonical.py
  469. ✓test_rewrites_exactly_the_volatile_shapes[9f1c8a2e-4b3d-4f6a-8f2f-0a1b2c3d4e5f-<uuid>] TestPlaceholderstest_fixture_canonical.py
  470. ✓test_rewrites_exactly_the_volatile_shapes[batch_688a8b7f9a08819096e0f7c88fcd07c5-<id>] TestPlaceholderstest_fixture_canonical.py
  471. ✓test_rewrites_exactly_the_volatile_shapes[chatcmpl-C0LO6rRkfJlpJ2mqW9BHYo4Sm8FWl-<id>] TestPlaceholderstest_fixture_canonical.py
  472. ✓test_rewrites_exactly_the_volatile_shapes[e2e-chat-stream-4d5152a995b7-e2e-chat-stream-<marker>] TestPlaceholderstest_fixture_canonical.py
  473. ✓test_rewrites_exactly_the_volatile_shapes[file-XyZ12345abc-<id>] TestPlaceholderstest_fixture_canonical.py
  474. ✓test_rewrites_exactly_the_volatile_shapes[gpt-4o-mini-gpt-4o-mini] TestPlaceholderstest_fixture_canonical.py
  475. ✓test_rewrites_exactly_the_volatile_shapes[max_tokens-max_tokens] TestPlaceholderstest_fixture_canonical.py
  476. ✓test_rewrites_exactly_the_volatile_shapes[Reply ok. 4d5152a995b7-Reply ok. <marker>] TestPlaceholderstest_fixture_canonical.py
  477. ✓test_rewrites_exactly_the_volatile_shapes[sk-1234-sk-1234] TestPlaceholderstest_fixture_canonical.py
  478. ✓test_rewrites_exactly_the_volatile_shapes[sk-3mCXCTGmYuEEIU2i2qmVE3Xq6tSK1O0X6ZIRP1Lpw8ZlbNjt-<key>] TestPlaceholderstest_fixture_canonical.py
  479. ✓test_rewrites_exactly_the_volatile_shapes[zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz-zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] TestPlaceholderstest_fixture_canonical.py
  480. ✓test_secret_set_versus_unset_stays_distinct TestKeyDistinctnesstest_fixture_canonical.py
  481. ✓test_serialization_order_is_not_identity TestKeyStabilitytest_fixture_canonical.py
  482. ✓test_volatile_material_does_not_change_the_key TestKeyStabilitytest_fixture_canonical.py
  483. ✓test_batch_create_over_rpm_returns_mapped_429 TestBatchRateLimitErrorMappingtest_batches_e2e.py
  484. ✓test_batch_key_model_access_deniedtest_batches_e2e.py
  485. ✓test_batch_lifecycle[azure-encoded]test_batches_e2e.py
  486. ✓test_batch_lifecycle[azure-model_param]test_batches_e2e.py
  487. ✓test_batch_lifecycle[azure-provider_fallback]test_batches_e2e.py
  488. ✓test_batch_lifecycle[azure-unified]test_batches_e2e.py
  489. ✓test_batch_lifecycle[bedrock-unified]test_batches_e2e.py
  490. ✓test_batch_lifecycle[openai-encoded]test_batches_e2e.py
  491. ✓test_batch_lifecycle[openai-model_param]test_batches_e2e.py
  492. ✓test_batch_lifecycle[openai-provider_fallback]test_batches_e2e.py
  493. ✓test_batch_lifecycle[openai-unified]test_batches_e2e.py
  494. ✓test_batch_lifecycle[vertex_ai-encoded]test_batches_e2e.py
  495. ✓test_batch_lifecycle[vertex_ai-model_param]test_batches_e2e.py
  496. ✓test_batch_lifecycle[vertex_ai-provider_fallback]test_batches_e2e.py
  497. ✓test_batch_lifecycle[vertex_ai-unified]test_batches_e2e.py
  498. ✓test_enqueued_allowance_accepts_batch_over_key_rpm TestBatchEnqueuedTokenLimittest_batches_e2e.py
  499. ✓test_exhausted_allowance_blocks_until_cancel_refunds TestBatchEnqueuedTokenLimittest_batches_e2e.py
  500. ✓test_file_content_matches_upload TestBatchFileContenttest_batches_e2e.py
  501. ✓test_file_upload_and_delete_outputstest_batches_e2e.py
  502. ✓test_gemini_file_upload TestGeminiFilestest_batches_e2e.py
  503. ✓test_list_page_cursors_address_only_the_callers_own_files TestOpenAIFilestest_batches_e2e.py
  504. ✓test_retrieve_round_trips_metadata TestOpenAIFilestest_batches_e2e.py
  505. ✓test_unified_batch_create_with_assume_role TestBedrockBatchAssumeRoletest_batches_e2e.py
  506. −test_rate_limited_batch_create_leaves_no_unattributed_spend_rowtest_batches_e2e.py›

    LIT-5027: the path under test hangs. The batch rate limiter reads the input file to count tokens by awaiting litellm.afile_content with no timeout, so a slow Files API holds POST /v1/batches open past any client deadline (63.6s observed on stage against a 60s read timeout). The unattributed-spend-row contract below is never reached, so the test reports a timeout rather than the behavior it guards. Unskip once the fetch is bounded.

  507. −test_unified_file_and_batch_create TestHostedVllmBatchtest_batches_e2e.py›

    hosted_vllm batch/files needs a live vLLM server (HOSTED_VLLM_API_BASE) not provisioned in the e2e environment; re-enable when available (LIT-3266)

  508. −test_uploaded_file_appears_in_list TestOpenAIFilestest_batches_e2e.py›

    LIT-4820 (https://linear.app/litellm-ai/issue/LIT-4820): GET /v1/files omits newly uploaded files. The upload succeeds and GET /v1/files/{id} returns the file, but it never appears in the listing; the returned set is stable with its newest entry ~10h old, on both the managed (/v1/files?model=) and provider-scoped (/openai/v1/files) routes. Skipped rather than weakened because the assertion below is the correct contract. Remove this marker when LIT-4820 is fixed; do not relax the assertion to make it pass.

  509. ✓test_all_attempts_failed_is_a_failure_not_a_passtest_span_selection.py
  510. ✓test_chat_completions_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py
  511. ✓test_chat_completions_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py
  512. ✓test_chat_completions_stream_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py
  513. ✓test_chat_completions_stream_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py
  514. ✓test_chat_completions_stream_records_real_ttft TestOtelTraceCompletenesstest_otel_trace_e2e.py
  515. ✓test_distinct_key_aliases_produce_distinct_series TestPrometheusPerKeyCardinalitytest_prometheus_cardinality_e2e.py
  516. ✓test_failed_chat_completions_error_span_attributes TestOtelTraceCompletenesstest_otel_trace_e2e.py
  517. ✓test_messages_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py
  518. ✓test_messages_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py
  519. ✓test_messages_stream_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py
  520. ✓test_messages_stream_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py
  521. ✓test_messages_stream_records_real_ttft TestOtelTraceCompletenesstest_otel_trace_e2e.py
  522. ✓test_other_operations_are_not_countedtest_span_selection.py
  523. ✓test_queue_time_histogram_records_an_observation TestPrometheusRequestQueueTimetest_prometheus_queue_time_e2e.py
  524. ✓test_responses_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py
  525. ✓test_responses_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py
  526. ✓test_responses_stream_emits_one_log_event TestDataDogLogDeliverytest_datadog_log_e2e.py
  527. ✓test_responses_stream_exports_complete_trace TestOtelTraceCompletenesstest_otel_trace_e2e.py
  528. ✓test_responses_stream_records_real_ttft TestOtelTraceCompletenesstest_otel_trace_e2e.py
  529. ✓test_retried_attempt_span_is_excludedtest_span_selection.py
  530. ✓test_served_span_is_the_only_one_when_nothing_was_retriedtest_span_selection.py
  531. ✓test_several_failed_attempts_still_leave_one_served_spantest_span_selection.py
  532. ✓test_two_served_spans_still_failtest_span_selection.py
  533. ✓test_a_run_that_drove_no_traffic_reports_a_total_failure_ratio TestAggregatetest_locust_load.py
  534. ✓test_a_run_with_no_failures_writes_no_csv_and_reports_no_errors TestErrorBreakdowntest_locust_load.py
  535. ✓test_all_planned_turns_completing_reports_zero_failures TestSummarizePlannedTurnstest_session_anomaly.py
  536. ✓test_an_even_split_takes_the_lower_middle_sample_as_locust_itself_does TestSerialLatencytest_locust_load.py
  537. ✓test_diagnosis_caps_the_list_and_says_how_many_it_left_out TestErrorBreakdowntest_locust_load.py
  538. ✓test_diagnosis_leads_with_the_most_common_error TestErrorBreakdowntest_locust_load.py
  539. ✓test_diagnosis_says_so_when_locust_recorded_nothing TestErrorBreakdowntest_locust_load.py
  540. ✓test_exhausted_attempts_return_the_last_failure TestRetriedtest_session_anomaly.py
  541. ✓test_first_try_success_never_sleeps TestRetriedtest_session_anomaly.py
  542. ✓test_locust_failure_rows_become_the_error_breakdown TestErrorBreakdowntest_locust_load.py
  543. ✓test_median_is_the_middle_sample_not_the_mean_a_slow_tail_would_drag TestSerialLatencytest_locust_load.py
  544. ✓test_median_merges_the_histograms_of_every_stats_entry TestSerialLatencytest_locust_load.py
  545. ✓test_no_samples_reports_zero_rather_than_dividing_by_an_empty_histogram TestSerialLatencytest_locust_load.py
  546. ✓test_ordinary_locust_chatter_is_not_reported_as_a_warning TestGeneratorSaturationtest_locust_load.py
  547. ✓test_partial_total_between_batch_flushes_is_not_accepted_as_final TestSettledSpendtest_session_anomaly.py
  548. ✓test_repeated_cpu_warnings_collapse_to_one_and_reach_the_diagnosis TestGeneratorSaturationtest_locust_load.py
  549. ✓test_session_aborted_on_first_turn_counts_all_its_planned_turns_as_failed TestSummarizePlannedTurnstest_session_anomaly.py
  550. ✓test_spend_that_never_becomes_nonzero_raises TestSettledSpendtest_session_anomaly.py
  551. ✓test_spend_that_never_stabilizes_raises TestSettledSpendtest_session_anomaly.py
  552. ✓test_throughput_spans_from_the_earliest_start_when_locust_reports_several_entries TestAggregatetest_locust_load.py
  553. ✓test_throughput_spans_the_whole_window_and_latency_comes_from_the_histogram TestAggregatetest_locust_load.py
  554. ✓test_transient_failures_then_success_returns_the_success TestRetriedtest_session_anomaly.py
  555. ✓test_content_drift_returns_the_miss_status_naming_both_keys TestReplayModetest_provider_edge.py
  556. ✓test_exhausted_key_returns_the_miss_status TestReplayModetest_provider_edge.py
  557. ✓test_forwards_to_the_provider_and_writes_one_interaction_file TestRecordModetest_provider_edge.py
  558. ✓test_fully_consumed_recording_leaves_nothing TestReplayLeftovertest_provider_edge.py
  559. ✓test_identical_requests_replay_their_responses_in_recorded_order TestReplayModetest_provider_edge.py
  560. ✓test_inert_outside_replay_mode TestReplayLeftovertest_provider_edge.py
  561. ✓test_invalid_mode_raises_naming_the_value TestApiBaseSeamtest_provider_edge.py
  562. ✓test_live_mode_returns_none TestApiBaseSeamtest_provider_edge.py
  563. ✓test_never_stores_headers_so_credentials_never_touch_disk TestRecordModetest_provider_edge.py
  564. ✓test_non_json_bodies_match_by_canonical_digest_without_storing_them TestReplayModetest_provider_edge.py
  565. ✓test_parallel_identical_calls_serve_each_recording_exactly_once TestConcurrentReplaytest_provider_edge.py
  566. ✓test_partially_consumed_recording_names_the_leftover TestReplayLeftovertest_provider_edge.py
  567. ✓test_query_params_are_part_of_the_identity TestReplayModetest_provider_edge.py
  568. ✓test_record_mode_boots_one_shared_edge_and_prepares_the_bundle TestApiBaseSeamtest_provider_edge.py
  569. ✓test_replay_serves_a_directly_recorded_interaction TestHandleEdgeRequestPuretest_provider_edge.py
  570. ✓test_request_identity_ignores_auth_headers TestReplayModetest_provider_edge.py
  571. ✓test_serves_recorded_bytes_with_zero_provider_hits TestReplayModetest_provider_edge.py
  572. ✓test_strips_volatile_response_headers_and_serves_the_filtered_copy TestRecordModetest_provider_edge.py
  573. ✓test_test_without_recordings_has_no_leftover TestReplayLeftovertest_provider_edge.py
  574. ✓test_unknown_mount_404s_naming_the_known_mounts TestHandleEdgeRequestPuretest_provider_edge.py
  575. ✓test_unknown_mount_raises_naming_the_known_mounts TestApiBaseSeamtest_provider_edge.py
  576. ✓test_unreachable_provider_records_and_serves_a_502 TestRecordModetest_provider_edge.py
  577. ✓test_5xx_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py
  578. ✓test_alias_api_key_never_overrides_the_tier_credential TestAliasParamForwardingtest_auto_router_regressions_e2e.py
  579. ✓test_body_tagged_chat_reaches_the_untagged_tier_after_marker_rewrite TestUntaggedTierDeploymentstest_auto_router_regressions_e2e.py
  580. ✓test_body_tagged_chat_routes_through_the_marker_to_its_tier TestTagSplitRoutingtest_auto_router_regressions_e2e.py
  581. ✓test_body_tagged_responses_with_list_input_routes_to_the_tier TestResponsesApiTagRoutingtest_auto_router_regressions_e2e.py
  582. ✓test_exact_cache_returns_cached TestReliabilityCachetest_reliability_cache_e2e.py
  583. ✓test_header_tagged_messages_routes_through_the_marker_to_an_untagged_tier TestUntaggedTierDeploymentstest_auto_router_regressions_e2e.py
  584. ✓test_header_tagged_responses_with_string_input_routes_to_the_tier TestResponsesApiTagRoutingtest_auto_router_regressions_e2e.py
  585. ✓test_request_timeout_exceeds_deadline TestReliabilityTimeoutstest_reliability_timeouts_e2e.py
  586. ✓test_responses_input_reaches_the_semantic_auto_router TestSemanticAutoRouterResponsestest_auto_router_regressions_e2e.py
  587. ✓test_stream_timeout_exceeds_deadline TestReliabilityTimeoutstest_reliability_timeouts_e2e.py
  588. ✓test_tagged_call_straight_at_an_untagged_deployment_stays_denied TestUntaggedTierDeploymentstest_auto_router_regressions_e2e.py
  589. ✓test_timeout_routes_to_fallback TestReliabilityFallbackstest_reliability_fallbacks_e2e.py
  590. ✓test_trivial_ask_behind_keyword_heavy_system_prompt_stays_on_the_cheap_tier TestComplexityHeuristicScopetest_auto_router_regressions_e2e.py
  591. ✓test_untagged_chat_is_always_served_by_the_plain_deployment TestTagSplitRoutingtest_auto_router_regressions_e2e.py
  592. ✓test_untagged_messages_is_served_by_the_plain_deployment TestTagSplitRoutingtest_auto_router_regressions_e2e.py
  593. ✓test_untagged_responses_is_served_by_the_plain_deployment TestResponsesApiTagRoutingtest_auto_router_regressions_e2e.py
  594. ✓test_zero_priced_alias_still_logs_spend_at_the_tier_rate TestStrategyAliasPricingtest_auto_router_regressions_e2e.py
  595. −test_llm_classifier_runs_and_routes_by_semantic_tier TestComplexityRouterLlmClassifiertest_complexity_router_e2e.py›

    product bug LIT-4521: LLM classifier returns SIMPLE for short hard prompts (e.g. Is P equal to NP?); re-enable when classifier tier quality is fixed

  596. ✓test_allowed_model_is_permitted TestAccessControltest_access_control_e2e.py
  597. ✓test_bearer_invalid_token_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py
  598. ✓test_disallowed_model_is_denied_403 TestAccessControltest_access_control_e2e.py
  599. ✓test_empty_bearer_token_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py
  600. ✓test_group_grants_every_deployment_in_it[bare name the group's wildcard covers-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py
  601. ✓test_group_grants_every_deployment_in_it[exactly-named deployment in the group-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py
  602. ✓test_group_grants_every_deployment_in_it[provider-prefixed name the group's wildcard covers-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py
  603. ✓test_group_grants_nothing_outside_it[deployment outside the group-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py
  604. ✓test_group_grants_nothing_outside_it[name no provider claims-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py
  605. ✓test_group_grants_nothing_outside_it[provider model outside the group's wildcard-<lambda>] TestKeyScopedToAccessGrouptest_model_access_group_e2e.py
  606. ✓test_group_grants_the_team_nothing_outside_it TestTeamScopedToAccessGrouptest_model_access_group_e2e.py
  607. ✓test_group_grants_the_teams_own_wildcard TestTeamScopedToAccessGrouptest_model_access_group_e2e.py
  608. ✓test_llm_only_key_forbidden_from_management_route_403 TestAccessControltest_access_control_e2e.py
  609. ✓test_missing_authorization_header_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py
  610. ✓test_not_bearer_scheme_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py
  611. ✓test_token_without_bearer_prefix_is_denied TestChatAuthHeaderstest_chat_auth_headers_e2e.py
  612. ✓test_unknown_model_returns_400 TestAccessControltest_access_control_e2e.py
  613. ✓test_valid_key_allows_and_invalid_key_denied TestVirtualKeyAuthtest_access_control_e2e.py
  614. ✓test_fresh_replay_bundle_collects TestCollectionGatetest_fixture_mode.py
  615. ✓test_invalid_mode_names_the_value_and_the_choices TestCollectionGatetest_fixture_mode.py
  616. ✓test_known_values_normalize[ REPLAY -replay] TestParseFixtureModetest_fixture_mode.py
  617. ✓test_known_values_normalize[-live] TestParseFixtureModetest_fixture_mode.py
  618. ✓test_known_values_normalize[live-live] TestParseFixtureModetest_fixture_mode.py
  619. ✓test_known_values_normalize[record-record] TestParseFixtureModetest_fixture_mode.py
  620. ✓test_known_values_normalize[replay-replay] TestParseFixtureModetest_fixture_mode.py
  621. ✓test_live_and_record_never_block_collection[] TestCollectionGatetest_fixture_mode.py
  622. ✓test_live_and_record_never_block_collection[live] TestCollectionGatetest_fixture_mode.py
  623. ✓test_live_and_record_never_block_collection[record] TestCollectionGatetest_fixture_mode.py
  624. ✓test_live_mode_prints_nothing TestReportHeadertest_fixture_mode.py
  625. ✓test_names_this_test_and_strips_the_phase TestCurrentTestKeytest_fixture_mode.py
  626. ✓test_record_and_replay_name_the_bundle TestReportHeadertest_fixture_mode.py
  627. ✓test_replay_with_no_bundle_says_how_to_record_one TestCollectionGatetest_fixture_mode.py
  628. ✓test_sequence_is_a_pure_function_of_test_and_ordinal TestDeterministicMarkertest_fixture_mode.py
  629. ✓test_stale_replay_bundle_fails_naming_its_age TestCollectionGatetest_fixture_mode.py
  630. ✓test_unknown_value_is_invalid_with_the_original_spelling TestParseFixtureModetest_fixture_mode.py
  631. ✓test_bundle_at_the_limit_is_still_fresh TestFreshnesstest_fixture_bundle.py
  632. ✓test_corrupt_manifest_is_unreadable TestFreshnesstest_fixture_bundle.py
  633. ✓test_fresh_directory_gets_a_fresh_manifest TestPrepareBundletest_fixture_bundle.py
  634. ✓test_interaction_filename_orders_and_slugs TestSlugstest_fixture_bundle.py
  635. ✓test_interactions_group_per_test TestRecordAndLoadtest_fixture_bundle.py
  636. ✓test_load_returns_interactions_in_recorded_order TestRecordAndLoadtest_fixture_bundle.py
  637. ✓test_missing_manifest_is_unreadable_with_recording_hint TestFreshnesstest_fixture_bundle.py
  638. ✓test_naive_recorded_at_is_read_as_utc TestFreshnesstest_fixture_bundle.py
  639. ✓test_record_wipes_the_previous_bundle_instead_of_reading_it TestPrepareBundletest_fixture_bundle.py
  640. ✓test_refuses_a_path_that_is_a_file TestPrepareBundletest_fixture_bundle.py
  641. ✓test_refuses_to_wipe_a_directory_that_is_not_a_bundle TestPrepareBundletest_fixture_bundle.py
  642. ✓test_same_tail_in_different_files_never_collides TestSlugstest_fixture_bundle.py
  643. ✓test_slug_for_test_is_deterministic TestSlugstest_fixture_bundle.py
  644. ✓test_stale_bundle_reports_age_and_limit TestFreshnesstest_fixture_bundle.py
  645. ✓test_unknown_format_version_is_unreadable TestFreshnesstest_fixture_bundle.py
  646. ✓test_429_is_never_retried TestTransientRetryPolicytest_e2e_http.py
  647. ✓test_non_transient_status_returns_immediately[200] TestTransientRetryPolicytest_e2e_http.py
  648. ✓test_non_transient_status_returns_immediately[201] TestTransientRetryPolicytest_e2e_http.py
  649. ✓test_non_transient_status_returns_immediately[400] TestTransientRetryPolicytest_e2e_http.py
  650. ✓test_non_transient_status_returns_immediately[401] TestTransientRetryPolicytest_e2e_http.py
  651. ✓test_non_transient_status_returns_immediately[404] TestTransientRetryPolicytest_e2e_http.py
  652. ✓test_non_transient_status_returns_immediately[422] TestTransientRetryPolicytest_e2e_http.py
  653. ✓test_non_transient_status_returns_immediately[500] TestTransientRetryPolicytest_e2e_http.py
  654. ✓test_non_transient_status_returns_immediately[502] TestTransientRetryPolicytest_e2e_http.py
  655. ✓test_non_transient_status_returns_immediately[503] TestTransientRetryPolicytest_e2e_http.py
  656. ✓test_non_transient_status_returns_immediately[504] TestTransientRetryPolicytest_e2e_http.py
  657. ✓test_overloaded_529_retries_with_backoff_then_returns_the_success TestTransientRetryPolicytest_e2e_http.py
  658. ✓test_persistent_transient_is_bounded_and_returns_the_last_response TestTransientRetryPolicytest_e2e_http.py
  659. ✓test_transient_set_is_only_statuses_the_proxy_cannot_emit TestTransientRetryPolicytest_e2e_http.py
  660. ✓test_cell_claimed_only_by_a_skipped_test_is_uncoveredtest_collector.py
  661. ✓test_collection_counts_only_markers_on_tests_that_would_runtest_collector.py
  662. ✓test_compute_coverage_counts_covered_p0_and_gapstest_collector.py
  663. ✓test_json_render_exposes_module_coverage_for_grafana_jobstest_collector.py
  664. ✓test_llm_cells_roll_up_by_core_endpointtest_collector.py
  665. ✓test_load_registry_rejects_duplicate_idstest_collector.py
  666. ✓test_logging_and_guardrail_roll_up_into_one_moduletest_collector.py
  667. ✓test_loki_render_exposes_exact_stdout_lines_for_lokitest_collector.py
  668. ✓test_orphan_marker_is_reported_not_countedtest_collector.py
  669. ✓test_prometheus_render_exposes_module_coverage_timeseriestest_collector.py
  670. ✓test_real_registry_loads_and_ids_are_uniquetest_collector.py
  671. ✓test_skipped_marker_outside_the_registry_is_still_an_orphantest_collector.py
  672. ✓test_text_render_uses_plain_coverage_languagetest_collector.py
  673. ✓test_discovery_card_is_proxy_fronted TestA2AAgentLifecycletest_a2a_agent_e2e.py
  674. ✓test_malformed_protocol_version_rejected TestA2AAgentLifecycletest_a2a_agent_e2e.py
  675. ✓test_message_send_runs_completion_bridge TestA2AAgentLifecycletest_a2a_agent_e2e.py
  676. ✓test_pinned_v0_3_serves_flat_message_shape TestA2AAgentLifecycletest_a2a_agent_e2e.py
  677. ✓test_pinned_v1_0_serves_nested_message_shape TestA2AAgentLifecycletest_a2a_agent_e2e.py
  678. ✓test_real_world_agent_replies_to_property_query TestA2AAgentLifecycletest_a2a_agent_e2e.py
  679. ✓test_register_persists TestA2AAgentLifecycletest_a2a_agent_e2e.py
  680. ✓test_semver_protocol_version_registers_and_serves TestA2AAgentLifecycletest_a2a_agent_e2e.py
  681. ✓test_unsupported_protocol_version_rejected TestA2AAgentLifecycletest_a2a_agent_e2e.py
  682. ✓test_apply_guardrail_blocks_banned_and_allows_clean TestApplyGuardrailEndpointtest_apply_guardrail_e2e.py
  683. ✓test_bedrock_pre_call_blocks_harmful_prompt TestBedrockGuardrailtest_bedrock_guardrail_e2e.py
  684. ✓test_blocks_execution_request_but_allows_explanation TestBlockCodeExecutionGuardrailtest_block_code_execution_guardrail_e2e.py
  685. ✓test_chat_blocks_category TestOpenAIModerationCategoryMatrixtest_openai_moderation_category_matrix_e2e.py
  686. ✓test_global_guardrail_blocks_key_without_team_opt_out TestTeamDisableGlobalGuardrailtest_team_disable_global_guardrail_e2e.py
  687. ✓test_messages_blocks_category TestOpenAIModerationCategoryMatrixtest_openai_moderation_category_matrix_e2e.py
  688. ✓test_moderation_blocks_flagged_input TestOpenAIModerationGuardrailtest_openai_moderation_guardrail_e2e.py
  689. ✓test_responses_blocks_category TestOpenAIModerationCategoryMatrixtest_openai_moderation_category_matrix_e2e.py
  690. ✓test_team_with_disable_flag_bypasses_global_guardrail TestTeamDisableGlobalGuardrailtest_team_disable_global_guardrail_e2e.py
  691. ✓test_access_group_scopes_tool_selection TestMcpAccessGroupToolSelectiontest_mcp_access_group_e2e.py
  692. ✓test_list_tools_denied_without_permission TestMcpKeyWithoutAccessIsDeniedtest_mcp_key_access_e2e.py
  693. −test_call_tool_denied_without_permission TestMcpKeyWithoutAccessIsDeniedtest_mcp_key_access_e2e.py›

    LIT-5052: the control call proving a granted key CAN invoke the tool sends a `telemetry` argument that Datadog's search_datadog_logs tool now rejects, so it errors with 'unexpected additional properties ["telemetry"]' and the denial assertion is never reached. `telemetry` was never a documented Datadog parameter; the test relied on the server ignoring unknown properties. Unskip once the argument is dropped.

  694. −test_chat_completion_uses_linear_with_authorization_bearer_header TestMcpChatCompletionOauthtest_mcp_chat_completion_oauth_e2e.py›

    set E2E_LINEAR_STORAGE_STATE to a Linear session captured via mcp/linear_session_capture.py

  695. −test_chat_completion_uses_linear_with_x_litellm_api_key_header TestMcpChatCompletionOauthtest_mcp_chat_completion_oauth_e2e.py›

    set E2E_LINEAR_STORAGE_STATE to a Linear session captured via mcp/linear_session_capture.py

  696. −test_content_filter_blocks_banned_keyword_in_tool_args TestMcpToolCallGuardrailtest_mcp_guardrail_e2e.py›

    LIT-5052: the control call sends a `telemetry` argument that Datadog's search_datadog_logs tool now rejects, so the clean-argument half of this test errors with 'unexpected additional properties ["telemetry"]' and the guardrail block it exists to prove is never exercised. `telemetry` was never a documented Datadog parameter; the test relied on the server ignoring unknown properties. Unskip once the argument is dropped.

  697. −test_search_logs_finds_seeded_completion TestDatadogMcpRoundTriptest_mcp_datadog_e2e.py›

    LIT-5052: this test sends a `telemetry` argument that Datadog's search_datadog_logs tool now rejects, so every tool call fails validation with 'unexpected additional properties ["telemetry"]' before the round-trip assertion is reached. `telemetry` was never a documented Datadog parameter; the test relied on the server ignoring unknown properties. Unskip once the argument is dropped.

  698. ✓test_liveness_reports_alive_without_auth TestHealthLifecycletest_health_lifecycle_e2e.py
  699. ✓test_master_key_authenticates_and_grants_admin_route TestMasterKeyAuthtest_master_key_auth_e2e.py
  700. ✓test_non_matching_master_key_is_denied TestMasterKeyAuthtest_master_key_auth_e2e.py
  701. ✓test_readiness_details_require_auth_and_expose_diagnostics TestHealthLifecycletest_health_lifecycle_e2e.py
  702. ✓test_readiness_is_reachable_without_credentials TestHealthLifecycletest_health_lifecycle_e2e.py
  703. ✓test_readiness_reports_connected_db TestHealthLifecycletest_health_lifecycle_e2e.py
  704. −llm_translation.realtime.test_realtime_pipecat_audio_e2ellm_translation.realtime.test_realtime_pipecat_audio_e2e›

    collection skipped