feat: implement knowledge retrieval events in Agent (#2727)

* feat: implement knowledge retrieval events in Agent

This commit introduces a series of knowledge retrieval events in the Agent class, enhancing its ability to handle knowledge queries. New events include KnowledgeRetrievalStartedEvent, KnowledgeRetrievalCompletedEvent, KnowledgeQueryGeneratedEvent, KnowledgeQueryFailedEvent, and KnowledgeSearchQueryCompletedEvent. The Agent now emits these events during knowledge retrieval processes, allowing for better tracking and handling of knowledge queries. Additionally, the console formatter has been updated to handle these new events, providing visual feedback during knowledge retrieval operations.

* refactor: update knowledge query handling in Agent

This commit refines the knowledge query processing in the Agent class by renaming variables for clarity and optimizing the query rewriting logic. The system prompt has been updated in the translation file to enhance clarity and context for the query rewriting process. These changes aim to improve the overall readability and maintainability of the code.

* fix: add missing newline at end of en.json file

* fix broken tests

* refactor: rename knowledge query events and enhance retrieval handling

This commit renames the KnowledgeQueryGeneratedEvent to KnowledgeQueryStartedEvent to better reflect its purpose. It also updates the event handling in the EventListener and ConsoleFormatter classes to accommodate the new event structure. Additionally, the retrieval knowledge is now included in the KnowledgeRetrievalCompletedEvent, improving the overall knowledge retrieval process.

* docs for transparancy

* refactor: improve error handling in knowledge query processing

This commit refactors the knowledge query handling in the Agent class by changing the order of checks for LLM compatibility. It now logs a warning and emits a failure event if the LLM is not an instance of BaseLLM before attempting to call the LLM. Additionally, the task_prompt attribute has been removed from the KnowledgeQueryFailedEvent, simplifying the event structure.

* test: add unit test for knowledge search query and VCR cassette

This commit introduces a new test, `test_get_knowledge_search_query`, to verify that the `_get_knowledge_search_query` method in the Agent class correctly interacts with the LLM using the appropriate prompts. Additionally, a VCR cassette is added to record the interactions with the OpenAI API for this test, ensuring consistent and reliable test results.

src/crewai/utilities/events/utils/console_formatter.py

@@ -783,3 +783,202 @@ class ConsoleFormatter:
             self.update_lite_agent_status(
                 self.current_lite_agent_branch, lite_agent_role, status, **fields
             )
+
+    def handle_knowledge_retrieval_started(
+        self,
+        agent_branch: Optional[Tree],
+        crew_tree: Optional[Tree],
+    ) -> Optional[Tree]:
+        """Handle knowledge retrieval started event."""
+        if not self.verbose:
+            return None
+
+        branch_to_use = agent_branch or self.current_lite_agent_branch
+        tree_to_use = branch_to_use or crew_tree
+
+        if branch_to_use is None or tree_to_use is None:
+            # If we don't have a valid branch, use crew_tree as the branch if available
+            if crew_tree is not None:
+                branch_to_use = tree_to_use = crew_tree
+            else:
+                return None
+
+        knowledge_branch = branch_to_use.add("")
+        self.update_tree_label(
+            knowledge_branch, "🔍", "Knowledge Retrieval Started", "blue"
+        )
+
+        self.print(tree_to_use)
+        self.print()
+        return knowledge_branch
+
+    def handle_knowledge_retrieval_completed(
+        self,
+        agent_branch: Optional[Tree],
+        crew_tree: Optional[Tree],
+        retrieved_knowledge: Any,
+    ) -> None:
+        """Handle knowledge retrieval completed event."""
+        if not self.verbose:
+            return None
+
+        branch_to_use = self.current_lite_agent_branch or agent_branch
+        tree_to_use = branch_to_use or crew_tree
+
+        if branch_to_use is None and tree_to_use is not None:
+            branch_to_use = tree_to_use
+
+        if branch_to_use is None or tree_to_use is None:
+            if retrieved_knowledge:
+                knowledge_text = str(retrieved_knowledge)
+                if len(knowledge_text) > 500:
+                    knowledge_text = knowledge_text[:497] + "..."
+
+                knowledge_panel = Panel(
+                    Text(knowledge_text, style="white"),
+                    title="📚 Retrieved Knowledge",
+                    border_style="green",
+                    padding=(1, 2),
+                )
+                self.print(knowledge_panel)
+                self.print()
+            return None
+
+        knowledge_branch_found = False
+        for child in branch_to_use.children:
+            if "Knowledge Retrieval Started" in str(child.label):
+                self.update_tree_label(
+                    child, "✅", "Knowledge Retrieval Completed", "green"
+                )
+                knowledge_branch_found = True
+                break
+
+        if not knowledge_branch_found:
+            for child in branch_to_use.children:
+                if (
+                    "Knowledge Retrieval" in str(child.label)
+                    and "Started" not in str(child.label)
+                    and "Completed" not in str(child.label)
+                ):
+                    self.update_tree_label(
+                        child, "✅", "Knowledge Retrieval Completed", "green"
+                    )
+                    knowledge_branch_found = True
+                    break
+
+        if not knowledge_branch_found:
+            knowledge_branch = branch_to_use.add("")
+            self.update_tree_label(
+                knowledge_branch, "✅", "Knowledge Retrieval Completed", "green"
+            )
+
+        self.print(tree_to_use)
+
+        if retrieved_knowledge:
+            knowledge_text = str(retrieved_knowledge)
+            if len(knowledge_text) > 500:
+                knowledge_text = knowledge_text[:497] + "..."
+
+            knowledge_panel = Panel(
+                Text(knowledge_text, style="white"),
+                title="📚 Retrieved Knowledge",
+                border_style="green",
+                padding=(1, 2),
+            )
+            self.print(knowledge_panel)
+
+        self.print()
+
+    def handle_knowledge_query_started(
+        self,
+        agent_branch: Optional[Tree],
+        task_prompt: str,
+        crew_tree: Optional[Tree],
+    ) -> None:
+        """Handle knowledge query generated event."""
+        if not self.verbose:
+            return None
+
+        branch_to_use = self.current_lite_agent_branch or agent_branch
+        tree_to_use = branch_to_use or crew_tree
+        if branch_to_use is None or tree_to_use is None:
+            return None
+
+        query_branch = branch_to_use.add("")
+        self.update_tree_label(
+            query_branch, "🔎", f"Query: {task_prompt[:50]}...", "yellow"
+        )
+
+        self.print(tree_to_use)
+        self.print()
+
+    def handle_knowledge_query_failed(
+        self,
+        agent_branch: Optional[Tree],
+        error: str,
+        crew_tree: Optional[Tree],
+    ) -> None:
+        """Handle knowledge query failed event."""
+        if not self.verbose:
+            return
+
+        tree_to_use = self.current_lite_agent_branch or crew_tree
+        branch_to_use = self.current_lite_agent_branch or agent_branch
+
+        if branch_to_use and tree_to_use:
+            query_branch = branch_to_use.add("")
+            self.update_tree_label(query_branch, "❌", "Knowledge Query Failed", "red")
+            self.print(tree_to_use)
+            self.print()
+
+        # Show error panel
+        error_content = self.create_status_content(
+            "Knowledge Query Failed", "Query Error", "red", Error=error
+        )
+        self.print_panel(error_content, "Knowledge Error", "red")
+
+    def handle_knowledge_query_completed(
+        self,
+        agent_branch: Optional[Tree],
+        crew_tree: Optional[Tree],
+    ) -> None:
+        """Handle knowledge query completed event."""
+        if not self.verbose:
+            return None
+
+        branch_to_use = self.current_lite_agent_branch or agent_branch
+        tree_to_use = branch_to_use or crew_tree
+
+        if branch_to_use is None or tree_to_use is None:
+            return None
+
+        query_branch = branch_to_use.add("")
+        self.update_tree_label(query_branch, "✅", "Knowledge Query Completed", "green")
+
+        self.print(tree_to_use)
+        self.print()
+
+    def handle_knowledge_search_query_failed(
+        self,
+        agent_branch: Optional[Tree],
+        error: str,
+        crew_tree: Optional[Tree],
+    ) -> None:
+        """Handle knowledge search query failed event."""
+        if not self.verbose:
+            return
+
+        tree_to_use = self.current_lite_agent_branch or crew_tree
+        branch_to_use = self.current_lite_agent_branch or agent_branch
+
+        if branch_to_use and tree_to_use:
+            query_branch = branch_to_use.add("")
+            self.update_tree_label(query_branch, "❌", "Knowledge Search Failed", "red")
+            self.print(tree_to_use)
+            self.print()
+
+        # Show error panel
+        error_content = self.create_status_content(
+            "Knowledge Search Failed", "Search Error", "red", Error=error
+        )
+        self.print_panel(error_content, "Search Error", "red")