Improving tool performance

Tips to improve tool and agent performance.

When testing agents, multiple iterations of updating tool / agent descriptions, instructions, or API documentation may be required. Consider these guidelines when you update your tool to be more accurate and reliable:

  • Validate that the Agent instructions include terminology or acronyms that you will likely include in your prompt.
  • Confirm all tool dependencies that are necessary for successful execution as part of the Agent or Tool Instruction. For example, "This tool requires the user’s account ID, which can be retrieved from the RetrieveID tool".
  • Create clear instructions for the Agent to navigate every step similar to how a user will execute each step.
  • Continuously test your Tools with varying levels of complexity such as these:
    • Level 1: Start a new chat for every invocation to disqualify any influence from conversation history.
    • Level 2: Invoke Tools several times within the same conversation to allow tool selection to be impacted by the results of previous inputs.
    • Level 3: Invoke Tools several times within the same conversation without clear guidance, such as these:
      • Intentionally being ambiguous in your intent
      • Providing the chatbot with incomplete information like missing product ID to complete the search
      • Changing topics unexpectedly in the middle of the conversation

This table shows the examples to track Tool performance testing:

Table 1. Example Level 1-2
Input Expected Output Expected Tools Used
What is my job title? Your current job title is “___”.
  • OS_UserProfileAgent
  • OS_UserProfile
Update my locale to en-US Your locale has been successfully updated to en-US.
  • OS_UserProfileAgent
  • OS_UpdateUserProfile
Table 2. Example Level 3
Input Expected Output Expected Tools Used
I forget what people are supposed to call me at work. Your job title is “___”.
  • OS_UserProfileAgent
  • OS_UserProfile
English is not my first language, and I can’t understand the Portal. I can help you with that. Could you please let me know which language you would like to change to?
  • OS_UserProfileAgent
  • OS_UpdateUserProfile
Spanish The language preference has been successfully updated to Spanish. Please log out and log back in to see the changes reflected in the UI.
  • OS_UserProfileAgent
  • OS_UpdateUserProfile