1436 passed, 0 failed, 58 skipped

stage red: product gap, /v1/audio/speech 500s on empty input instead of surfacing the provider 4xx

LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

LIT-4873: a mid-conversation role='system' reminder invalidates the prompt cache on the vertex_ai / azure_ai / bedrock_invoke Messages paths, while the same request preserves it both direct to Anthropic and through litellm's first-party anthropic path. Product bug, not a test defect: the assertion here is correct and must be restored unchanged with the fix

stage red: product gap, native passthrough returns no x-litellm-response-cost or x-ratelimit-* headers

set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

stage red: product gap (LIT-5447), retrieving an unknown response id returns 400 (model=None) instead of 404

stage red: product gap, /v1/audio/speech 500s on invalid voice instead of surfacing the provider 4xx

stage red: product gap, /v1/ocr 500s (aocr TypeError) on missing document instead of 400

stage red: product gap, /v1/audio/speech 500s on missing input instead of 400

stage red: product gap, /v1/moderations 500s (KeyError 'input') on missing input instead of 400

stage red: product gap, /v1/responses 500s (aresponses TypeError) on missing input instead of 400

stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing max_tokens instead of 400

stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing messages instead of 400

stage red: product gap, /v1/audio/speech 500s on missing model instead of 400

stage red: product gap, /v1/images/generations 500s (aimage_generation TypeError) on missing prompt instead of 400

stage red: product gap, retrieving a nonexistent vector store returns 2xx with an error envelope in the body instead of 404

stage red: product gap, vector store search 500s (asearch TypeError) on missing query instead of 400

stage red: ALB sends custom passthrough paths to backend pods, which lack ANTHROPIC_API_KEY, so the os.environ header forwards unresolved and Anthropic 401s; routing user-defined paths to the gateway needs a product/infra decision (fixed path namespace or dynamic ingress)

stage red: product gap (LIT-5446), retrieve returns a different id than the stored response (non-idempotent response-id re-encryption)

stage red: environment gap, the e2e stack neither enables the websearch_interception callback nor declares a search backend, so the request reaches bedrock's transformation and takes its by-design 400. Unskip once the ephemeral stack ships the config in this module's docstring.

product gap: GET /v1/mcp/server builds each row from the in-memory registry, whose _build_mcp_server_table sets description from mcp_info['description'], so the list reports the mcp_info description while GET /v1/mcp/server/{server_id} reports the stored description column. A server created with both set to different text reads back with two different descriptions depending on the route

Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

stage red: Vertex returns not supported for token counting for Claude aliases

stage red: 1M long_context not green on stage Anthropic path yet (200k sonnet / model alias)

stage red: 1M long_context not green on stage Azure Foundry deployments yet

stage red: 1M long_context not green on stage Bedrock Converse deployments yet

stage red: 1M long_context not green on stage Bedrock Invoke deployments yet

stage red: 1M long_context not green on stage Vertex deployments yet

stage red: /azure passthrough drops client headers (e.g. anthropic-version); product gap

stage red: Azure Foundry tool_search_server not supported in workspace for probed models

stage red: Vertex rejects tool_search when deployment extra_headers inject context-1m beta; product/config

Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

Bedrock Mantle GPT-5.6 cells are opt-in; set COMPAT_MANTLE_CELLS=1 once the AWS account is allowlisted for the openai.gpt-5.6-* models

OpenAI GPT-5.6 cells are opt-in; set COMPAT_OPENAI_GPT_CELLS=1 to run them (stage suite timeouts under concurrent load)

stage red: product gap, end-user model_max_budget rpm_limit is stored but never enforced

LIT-5118: the stage proxy does not run the dynamic_rate_limiter_v3 callbacks + priority_reservation config this module's docstring requires (zero limiter log lines on any pod during the 2026-08-02 run), so strict enforcement can never engage there

set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

set HOSTED_VLLM_API_BASE (the live vLLM server this deployment targets)

LIT-5027: the path under test hangs. The batch rate limiter reads the input file to count tokens by awaiting litellm.afile_content with no timeout, so a slow Files API holds POST /v1/batches open past any client deadline (63.6s observed on stage against a 60s read timeout). The unattributed-spend-row contract below is never reached, so the test reports a timeout rather than the behavior it guards. Unskip once the fetch is bounded.

LIT-4820 (https://linear.app/litellm-ai/issue/LIT-4820): GET /v1/files omits newly uploaded files. The upload succeeds and GET /v1/files/{id} returns the file, but it never appears in the listing; the returned set is stable with its newest entry ~10h old, on both the managed (/v1/files?model=) and provider-scoped (/openai/v1/files) routes. Skipped rather than weakened because the assertion below is the correct contract. Remove this marker when LIT-4820 is fixed; do not relax the assertion to make it pass.

stage red: product gap, key-scoped weave_otel spans are not delivered when the OTEL v2 callback is active

stage red: product gap, key-scoped weave_otel spans are not delivered when the OTEL v2 callback is active

LANGSMITH_API_KEY is not provisioned in the e2e CI environment