1. Define the current and expected behavior
Choose a concrete task. Record the current approach, useful output, common errors and the consequences of an incorrect result.
From a behavior problem to a tested change
The work should show whether the proposed change helps the intended task, including important failures.
Choose a concrete task. Record the current approach, useful output, common errors and the consequences of an incorrect result.
Confirm the intended use and access with the responsible data owner. Clean the relevant examples and expected results, and keep separate material for evaluation.
Test a suitable change to instructions, information retrieval, model choice or adaptation. Document what is being compared and what will make the experiment stop.
Compare on the agreed evaluation set rather than only development examples. Include review needs, response time and operating cost in the decision.
Test the model within the actual application path using controlled records. Keep the tested configuration, operating instructions, release recovery and change checks together.
The task comparison and application checks establish what was learned. Prepare the evaluation brief.