Lorenze Jay a1033e4bfe Fix structured output leaks in tool-calling loops (#5897) ...

* Fix structured output leaks in tool-calling loops

* addressing comments

* drop scripts

* Update Gemini agent tests to include structured output with thoughts and bump model version to 2.5-flash

* merge

* Update Anthropic test cases to use new model and tool structure

- Changed the model from "claude-3-5-haiku-20241022" to "claude-sonnet-4-6" in the test setup.
- Updated the request and response formats in the YAML test cassette to reflect the new tool structure and improved content formatting.
- Adjusted the expected response body to match the new output format from the assistant, including changes in tool usage and response details.
- Increased rate limit values in the response headers for better testing scenarios.

* adjusted bedrock cassettes

* adjusting cassettes for bedrock

* fix test

* Update VCR configuration to use 'host' instead of 'bedrock_host' for request matching

2026-05-27 13:20:53 -07:00

..

test_agent_custom_max_iterations.yaml

feat: deprecate CrewAgentExecutor, default Crew agents to AgentExecutor (#5745)

2026-05-12 11:22:13 -07:00

test_agent_error_on_parsing_tool.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_agent_execute_task_basic.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_agent_execute_task_with_context.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_agent_execute_task_with_custom_llm.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_agent_execute_task_with_planning_refine.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_agent_execute_task_with_planning.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_agent_execute_task_with_tool.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_agent_execute_task_without_planning.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_agent_execution_with_specific_tools.yaml

Lorenze/imp/native tool calling (#4258)

2026-01-22 17:44:03 -08:00

test_agent_execution_with_tools.yaml

Lorenze/imp/native tool calling (#4258)

2026-01-22 17:44:03 -08:00

test_agent_execution.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_agent_kickoff_multi_step_task_with_planning.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_agent_kickoff_with_mcp_tools.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_agent_kickoff_with_planning_disabled.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_agent_kickoff_with_planning.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_agent_kickoff_with_platform_tools.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_agent_kickoff_without_planning.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_agent_knowledege_with_crewai_knowledge.yaml

fix: ensure otel span is closed

2025-12-05 13:23:26 -05:00

test_agent_max_iterations_stops_loop.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_agent_moved_on_after_max_iterations.yaml

Lorenze/imp/native tool calling (#4258)

2026-01-22 17:44:03 -08:00

test_agent_output_when_guardrail_returns_base_model.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_agent_powered_by_new_o_model_family_that_allows_skipping_tool.yaml

Lorenze/imp/native tool calling (#4258)

2026-01-22 17:44:03 -08:00

test_agent_powered_by_new_o_model_family_that_uses_tool.yaml

Lorenze/imp/native tool calling (#4258)

2026-01-22 17:44:03 -08:00

test_agent_respect_the_max_rpm_set_over_crew_rpm.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_agent_respect_the_max_rpm_set.yaml

Lorenze/imp/native tool calling (#4258)

2026-01-22 17:44:03 -08:00

test_agent_step_callback.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_agent_use_specific_tasks_output_as_context.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_agent_with_knowledge_sources_extensive_role.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_agent_with_knowledge_sources_generate_search_query.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_agent_with_knowledge_sources_with_query_limit_and_score_threshold_default.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_agent_with_knowledge_sources_with_query_limit_and_score_threshold.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_agent_with_knowledge_sources.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_agent_with_knowledge_with_no_crewai_knowledge.yaml

fix: ensure otel span is closed

2025-12-05 13:23:26 -05:00

test_agent_with_only_crewai_knowledge.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_agent_without_max_rpm_respects_crew_rpm.yaml

Lorenze/imp/native tool calling (#4258)

2026-01-22 17:44:03 -08:00

test_agent_without_max_rpm_respet_crew_rpm.yaml

feat: native multimodal file handling; openai responses api

2026-01-23 15:13:25 -05:00

test_disabling_cache_for_agent.yaml

Lorenze/imp/native tool calling (#4258)

2026-01-22 17:44:03 -08:00

test_do_not_allow_crewai_trigger_context_for_first_task_hierarchical.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_ensure_first_task_allow_crewai_trigger_context_is_false_does_not_inject.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_first_task_auto_inject_trigger.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_get_knowledge_search_query.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_guardrail_is_called_using_callable.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_guardrail_is_called_using_string.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_guardrail_reached_attempt_limit.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_handle_context_length_exceeds_limit_cli_no.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_lite_agent_created_with_correct_parameters[False].yaml

feat: native multimodal file handling; openai responses api

2026-01-23 15:13:25 -05:00

test_lite_agent_created_with_correct_parameters[True].yaml

feat: native multimodal file handling; openai responses api

2026-01-23 15:13:25 -05:00

test_lite_agent_inside_flow_sync.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_lite_agent_inside_flow_with_tools.yaml

Lorenze/enh decouple executor from crew (#4209)

2026-01-20 21:44:45 -08:00

test_lite_agent_kickoff_async_inside_flow.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_lite_agent_output_includes_messages.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_lite_agent_returns_usage_metrics_async.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_lite_agent_returns_usage_metrics.yaml

feat: native multimodal file handling; openai responses api

2026-01-23 15:13:25 -05:00

test_lite_agent_standalone_still_works.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_lite_agent_structured_output.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_lite_agent_with_tools.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_llm_call_with_all_attributes.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_llm_call_with_error.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_llm_call.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_multiple_agents_in_same_flow.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_native_tool_calling_error_handling.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

test_task_allow_crewai_trigger_context_no_payload.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_task_allow_crewai_trigger_context.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_task_without_allow_crewai_trigger_context.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

TestAgentA2AKickoff.test_agent_kickoff_delegates_to_a2a.yaml

feat: add a2a liteagent, auth, transport negotiation, and file support

2026-01-30 17:10:00 -05:00

TestAgentA2AKickoff.test_agent_kickoff_handles_multi_turn_conversation.yaml

feat: add a2a liteagent, auth, transport negotiation, and file support

2026-01-30 17:10:00 -05:00

TestAgentA2AKickoff.test_agent_kickoff_returns_lite_agent_output.yaml

feat: add a2a liteagent, auth, transport negotiation, and file support

2026-01-30 17:10:00 -05:00

TestAgentA2AKickoff.test_agent_kickoff_with_calculator_skill.yaml

feat: add a2a liteagent, auth, transport negotiation, and file support

2026-01-30 17:10:00 -05:00

TestAgentA2AKickoff.test_agent_kickoff_with_conversation_skill.yaml

feat: add a2a liteagent, auth, transport negotiation, and file support

2026-01-30 17:10:00 -05:00

TestAgentA2AKickoff.test_agent_kickoff_with_failed_a2a_endpoint.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

TestAgentA2AKickoff.test_agent_kickoff_with_list_messages.yaml

feat: add a2a liteagent, auth, transport negotiation, and file support

2026-01-30 17:10:00 -05:00

TestAgentA2AKickoff.test_agent_without_a2a_works_normally.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

TestAgentA2AKickoffAsync.test_agent_kickoff_async_delegates_to_a2a.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

TestAgentA2AKickoffAsync.test_agent_kickoff_async_with_calculator.yaml

feat: add a2a liteagent, auth, transport negotiation, and file support

2026-01-30 17:10:00 -05:00

TestAgentExecutorPlanning.test_agent_kickoff_with_planning_stores_plan_in_state.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

TestAgentExecutorPlanning.test_agent_kickoff_without_planning_skips_plan_generation.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

TestAgentExecutorPlanning.test_executor_state_contains_plan_after_planning.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

TestAgentExecutorPlanning.test_planning_creates_minimal_steps_for_multi_step_task.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

TestAgentExecutorPlanning.test_planning_disabled_skips_planning.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

TestAgentExecutorPlanning.test_planning_handles_sequential_dependency_task.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

TestAnthropicNativeToolCalling.test_anthropic_agent_with_native_tool_calling.yaml

Lorenze/imp/native tool calling (#4258)

2026-01-22 17:44:03 -08:00

TestAnthropicNativeToolCalling.test_anthropic_parallel_native_tool_calling_test_agent_kickoff.yaml

supporting parallel tool use (#4513)

2026-02-19 14:07:28 -08:00

TestAnthropicNativeToolCalling.test_anthropic_parallel_native_tool_calling_test_crew.yaml

supporting parallel tool use (#4513)

2026-02-19 14:07:28 -08:00

TestAzureNativeToolCalling.test_azure_agent_with_native_tool_calling.yaml

supporting parallel tool use (#4513)

2026-02-19 14:07:28 -08:00

TestAzureNativeToolCalling.test_azure_parallel_native_tool_calling_test_agent_kickoff.yaml

supporting parallel tool use (#4513)

2026-02-19 14:07:28 -08:00

TestAzureNativeToolCalling.test_azure_parallel_native_tool_calling_test_crew.yaml

supporting parallel tool use (#4513)

2026-02-19 14:07:28 -08:00

TestBedrockNativeToolCalling.test_bedrock_agent_kickoff_with_tools_mocked.yaml

Fix structured output leaks in tool-calling loops (#5897)

2026-05-27 13:20:53 -07:00

TestBedrockNativeToolCalling.test_bedrock_parallel_native_tool_calling_test_agent_kickoff.yaml

Fix structured output leaks in tool-calling loops (#5897)

2026-05-27 13:20:53 -07:00

TestBedrockNativeToolCalling.test_bedrock_parallel_native_tool_calling_test_crew.yaml

Fix structured output leaks in tool-calling loops (#5897)

2026-05-27 13:20:53 -07:00

TestGeminiNativeToolCalling.test_gemini_agent_with_native_tool_calling.yaml

supporting parallel tool use (#4513)

2026-02-19 14:07:28 -08:00

TestGeminiNativeToolCalling.test_gemini_parallel_native_tool_calling_test_agent_kickoff.yaml

supporting parallel tool use (#4513)

2026-02-19 14:07:28 -08:00

TestGeminiNativeToolCalling.test_gemini_parallel_native_tool_calling_test_crew.yaml

supporting parallel tool use (#4513)

2026-02-19 14:07:28 -08:00

TestMaxUsageCountWithNativeToolCalling.test_max_usage_count_limit_enforced_in_native_tool_calling.yaml

Lorenze/imp/native tool calling (#4258)

2026-01-22 17:44:03 -08:00

TestMaxUsageCountWithNativeToolCalling.test_max_usage_count_tracked_in_native_tool_calling.yaml

Lorenze/imp/native tool calling (#4258)

2026-01-22 17:44:03 -08:00

TestMaxUsageCountWithNativeToolCalling.test_tool_usage_increments_after_successful_execution.yaml

Lorenze/imp/native tool calling (#4258)

2026-01-22 17:44:03 -08:00

TestNativeToolCallingTokenUsage.test_openai_native_tool_calling_token_usage.yaml

Lorenze/imp/native tool calling (#4258)

2026-01-22 17:44:03 -08:00

TestOpenAINativeToolCalling.test_openai_agent_with_native_tool_calling.yaml

supporting parallel tool use (#4513)

2026-02-19 14:07:28 -08:00

TestOpenAINativeToolCalling.test_openai_parallel_native_tool_calling_test_agent_kickoff.yaml

supporting parallel tool use (#4513)

2026-02-19 14:07:28 -08:00

TestOpenAINativeToolCalling.test_openai_parallel_native_tool_calling_test_crew.yaml

supporting parallel tool use (#4513)

2026-02-19 14:07:28 -08:00

TestOpenAINativeToolCalling.test_openai_parallel_native_tool_calling_tool_hook_parity_agent_kickoff.yaml

supporting parallel tool use (#4513)

2026-02-19 14:07:28 -08:00

TestOpenAINativeToolCalling.test_openai_parallel_native_tool_calling_tool_hook_parity_crew.yaml

supporting parallel tool use (#4513)

2026-02-19 14:07:28 -08:00

TestReasoningEffort.test_reasoning_effort_high_runs_full_observation_pipeline.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

TestReasoningEffort.test_reasoning_effort_low_skips_decide_and_replan.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

TestResponseFormatWithKickoff.test_kickoff_no_response_format_returns_raw_text.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

TestResponseFormatWithKickoff.test_kickoff_response_format_with_planning_and_tools.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00

TestResponseFormatWithKickoff.test_kickoff_response_format_without_planning.yaml

Lorenze/feat/plan execute pattern (#4817)

2026-03-15 18:33:17 -07:00