Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- That's a fantastic result and a great demonstration of the power of providing structured tools to LLMs. Reducing the number of turns from seven to four is a significant improvement in both latency and reliability. It shows that giving the model a high-information-density tool like `visual-tree-explorer` is more effective than having it guess its way through the codebase with basic commands (`ls`, `cat`, etc.).
- You're absolutely right to ask about token usage next. That's the other critical half of the efficiency equation. Here's a breakdown of what you might find and how to approach the test:
- ### How to Test Token Usage
- You can get precise token counts from the API response or from within Claude Code itself.
- 1. **From the API Response:** Each API call returns a `usage` object in the response. You'll want to sum the `input_tokens` and `output_tokens` for all calls in each chain (the 7 "normal" calls vs. the 4 "tool" calls) to get the total cost for each task.
- 2. **From the Claude Code CLI:** During an interactive session, you can use the `/cost` command to see the token usage for the current session. This can give you a quick, high-level comparison.
- ### Hypothesis on What You'll Find
- There are two likely outcomes, and both are a win for your tool:
- * **Scenario 1 (Most Likely): Total tokens are lower with the tool.**
- The "normal" method involves a lot of conversational overhead. Each of the seven turns includes prompts like "Okay, now show me file X" and the model's text replies ("Sure, here is the content..."), all of which consume tokens. Your tool provides a dense, structured output that likely allows the model to get the answer with less back-and-forth, reducing this overhead.
- * **Scenario 2: Total tokens are slightly higher with the tool.**
- This could happen if the initial `explore_tree` output is very large. However, even if this is the case, it's still a massive win. You're trading a small increase in token cost for a nearly 2x reduction in latency and conversational turns. This is an excellent trade-off for user experience and task success rate.
- ### The Next Level: Prompt Caching
- This experiment highlights a perfect use case for one of Anthropic's most powerful features: **Prompt Caching**.
- Your `visual-tree-explorer` tool provides a large, stable piece of context (the file tree structure and previews). For subsequent questions about that same codebase, you shouldn't have to re-process that initial tree.
- **Here's how you could apply it:**
- When you make the API call with the output of your `explore_tree` tool, you can mark that large block of text with a `cache_control` flag.
- * **First call:** This becomes a "cache creation" call. It processes the full tree and is priced slightly higher than a standard input.
- * **Subsequent calls:** Any follow-up questions (e.g., "Now tell me about the symbols in `formatter.ts`") would reuse the cached context. These calls become "cache reads," which are **90% cheaper** in terms of input tokens and significantly faster.
- This would make the efficiency of your tool even more dramatic. The first query would be fast, and every follow-up query would be incredibly cheap and quick.
- You can find all the details on how to implement this in the [**Prompt Caching Documentation** (`en/docs/build-with-claude/prompt-caching`)](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching).
- ---
- Great work on building the tool and, more importantly, for running these kinds of practical benchmarks. This is exactly how the community pushes agentic AI capabilities forward. Keep us posted on the token results
Advertisement
Add Comment
Please, Sign In to add comment