[FEATURE] Add WebFetch to embedded agent tools

[stephanj]

The embedded Agent Mode tools are missing a "WebFetch" tool so the agent can get the content from a web page / URL.

@claude investigate feature request.

[claude]

Claude finished @stephanj's task in 1m 38s —— View job

---

Investigation Complete: WebFetch Tool for Agent Mode

I've completed the investigation of adding a WebFetch tool to the embedded Agent Mode. Here are my findings:

---

Current State

Agent Mode Tools (BuiltInToolProvider.java:src/main/java/com/devoxx/genie/service/agent/tool/BuiltInToolProvider.java)

The agent currently has 7 built-in tools:

Read Tools:

• read_file - Read file contents from the project
• list_files - List files and directories
• search_files - Search for regex patterns in project files

Write Tools:

• write_file - Create new files
• edit_file - Modify existing files by replacing text
• run_command - Execute terminal commands (30s timeout)

Advanced:

• parallel_explore - Launch multiple sub-agents for concurrent exploration (when enabled)

Existing Web Capabilities

The plugin already has web search functionality, but it's NOT exposed as an agent tool:

• WebSearchPromptExecutionService (src/main/java/com/devoxx/genie/service/prompt/websearch/WebSearchPromptExecutionService.java:1) uses LangChain4j's WebSearchContentRetriever
• Supports Google Custom Search and Tavily search engines
• Currently only available as a separate execution strategy (via /search command or toggle), not as an agentic tool
• Configuration UI exists in Settings → Web Search

---

Implementation Options

There are two distinct approaches to consider:

Option 1: WebFetch Tool (Direct URL Fetching)

Add a new web_fetch tool that allows the agent to directly fetch content from a specific URL:

Specification:

ToolSpecification.builder()
    .name("web_fetch")
    .description("Fetch and read the content from a specific URL")
    .parameters(JsonObjectSchema.builder()
        .addStringProperty("url", "The URL to fetch content from")
        .required("url")
        .build())
    .build()

Pros:

• Direct, precise fetching of specific URLs
• Useful when the agent needs to read documentation, API responses, or specific web pages
• Lower cost than search (no multiple results to process)

Cons:

• Requires the agent to already know the URL
• No search/discovery capability
• Need to handle various content types (HTML, JSON, XML, etc.)
• Requires HTTP client implementation or LangChain4j web content integration

Option 2: WebSearch Tool (Search Engine Integration)

Expose the existing WebSearchPromptExecutionService as an agent tool:

Specification:

ToolSpecification.builder()
    .name("web_search")
    .description("Search the web for information using Google or Tavily. Returns summarized results.")
    .parameters(JsonObjectSchema.builder()
        .addStringProperty("query", "The search query")
        .addIntegerProperty("max_results", "Maximum number of results to return (default: 5)")
        .build())
    .build()

Pros:

• Agent can discover information without knowing specific URLs
• Already implemented in the codebase (just needs to be wrapped as a tool)
• Uses existing configuration (Google Custom Search / Tavily API keys)
• Returns synthesized, LLM-processed results

Cons:

• Requires API keys and external services
• More expensive (search + LLM processing)
• Results are summaries, not raw content

---

Recommendations

I recommend implementing both tools, as they serve different purposes:

1. Start with web_search tool (easier to implement)

   • Wrap the existing WebSearchPromptExecutionService
   • Add it to BuiltInToolProvider conditionally (only when web search is enabled in settings)
   • Follow the pattern of parallel_explore for conditional tool registration

2. Then add web_fetch tool (requires new implementation)

   • Create WebFetchToolExecutor implementing ToolExecutor
   • Use Java's HttpClient or LangChain4j's document loaders
   • Handle HTML-to-text conversion (could use Jsoup or similar)
   • Add URL validation and safety checks
   • Consider adding timeout and size limits

---

Implementation Pattern

Based on the codebase analysis, here's how to add a new tool:

1. Create Tool Executor (e.g., WebFetchToolExecutor.java)

   public class WebFetchToolExecutor implements ToolExecutor {
       @Override
       public String execute(ToolExecutionRequest request, Object memoryId) {
           // Implementation
       }
   }

2. Register in BuiltInToolProvider.java

   // In constructor, add:
   tools.put(
       ToolSpecification.builder()
           .name("web_fetch")
           .description("...")
           .parameters(...)
           .build(),
       new WebFetchToolExecutor(project)
   );

3. Add Settings (optional, if configuration needed)

   • Update DevoxxGenieStateService for any new settings
   • Add UI components in ui/settings/agent/AgentSettingsComponent.java

4. Add Tests

   • Create WebFetchToolExecutorTest.java following existing test patterns

---

Dependencies Available

The project already has:

• LangChain4j web search engines: langchain4j-web-search-engine-google-custom and langchain4j-web-search-engine-tavily
• HTTP client capabilities via langchain4j-http-client-jdk
• Potential document loaders in LangChain4j core

---

Next Steps

Would you like me to:

1. Implement the web_search tool by wrapping the existing WebSearchPromptExecutionService?
2. Implement the web_fetch tool for direct URL fetching?
3. Implement both tools?
4. Create a detailed implementation plan first for review?

Please let me know your preference, and I can proceed with the implementation.

---