Creating evaluations

An evaluation can have multiple owners. When you create an evaluation, the system automatically assigns you as the first owner. You can add additional owners to run evaluations, create or manage evaluation schedules, and view sensitive execution data such as Ground Truth, Model Generated Output, and Agent Trajectory data.

Note: Non-owners can view general evaluation information but cannot run evaluations or access sensitive details.
  1. Select Infor OS > GenAI > Factory > Evaluation.
  2. Click + Add.
  3. Specify a name for the evaluation.
  4. Select the endpoint where the evaluation applies:
    • GenAI Assistant
    • Embedded Experience
  5. Specify these fields:
    System Prompt
    Define a system prompt for the evaluation. Applicable only for Embedded Experience.
    System Cache
    Enable the toggle for caching the system prompt. Applicable only for Embedded Experience.
    Owners
    Specify the names within the tenant of people who can manage this evaluation.
  6. Under Evaluation Parameters, click + Add and specify these:
    Input
    The user prompt to evaluate.
    Ground Truth
    The expected response from the GenAI Assistant.
    Judge Model
    Model selector for defining model used for evaluation job.
    Agent Trajectory Ground Truth
    Toggle for enabling Agent and Tool efficiency metric.
  7. Enable any relevant Scoring toggles:
    Answer Correctness
    The factual accuracy of the generated output, independent of the defined ground truth.
    Answer Similarity
    The degree of alignment between the generated output and the defined ground truth.
    Answer Relevance
    The measure of how appropriate and useful the generated output is with respect to the user’s intent.
    Agent Trajectory
    The measure of how efficiently an Agent or Tool was used.
  8. Click Add at the bottom of the Add Evaluation Parameter window to save.

    If you enabled Agent Trajectory, click the Generate evaluation trace button to generate the trajectory.

    After you click the Generate evaluation trace button, the overview screen is displayed. The value in the Status field shows the state of trajectory generation:

    • In progress: the trajectory is being generated.
    • Ready for review: the trajectory can be reviewed and edited.
    • Final: the reviewer approved the trajectory.
    Note: If you did not select Agent trajectory ground truth, the status of the evaluation is Final after you save it.