Supermarket Evaluation

Overall Performance

Success Rate (SR): Cart-addition rate overall and by interaction turn.
$S R = \frac{100}{N} \sum_{e = 1}^{N} C_{e}$
$S R (t) = \frac{100}{N} \sum_{e = 1}^{N} 1 [C_{e} = 1 \land T_{e} \leq t]$
N is the number of episodes, Cₑ indicates a nonempty final cart, and Tₑ is the episode’s interaction-turn count. SR(t) is the percentage of episodes completing a purchase by turn t.
Turn: Mean number of recorded interaction turns per episode.
$T_{e} = | ℛ_{e} |$
$\overset{―}{T} = \frac{1}{N} \sum_{e = 1}^{N} T_{e}$
ℛₑ is the set of recorded interaction turns in episode e. Tₑ is its turn count, and T̄ is the mean across N episodes.

Dialogue Quality

Informativeness: Provides specific, relevant information supporting the next decision or action.
${\overset{―}{s}}_{d} = \frac{1}{| E_{d} |} \sum_{e \in E_{d}} s_{e, d}$
$s_{e, d} \in {1, 2, 3, 4, 5}$
When information or guidance is needed, provide concrete, relevant content that supports a decision or next action rather than vague statements.
sₑ,ᵈ is the questionnaire or judge rating for dimension d in episode e; Eᵈ is the set of rated episodes. The scale is 1 Very poor, 2 Poor, 3 Acceptable, 4 Good, and 5 Excellent.
Naturalness: Uses natural, concise and coherent shopping dialogue.
${\overset{―}{s}}_{d} = \frac{1}{| E_{d} |} \sum_{e \in E_{d}} s_{e, d}$
$s_{e, d} \in {1, 2, 3, 4, 5}$
Assess wording, tone and pacing against everyday shopping conversation and basic common sense, without system jargon or formulaic phrasing.
sₑ,ᵈ is the questionnaire or judge rating for dimension d in episode e; Eᵈ is the set of rated episodes. The scale is 1 Very poor, 2 Poor, 3 Acceptable, 4 Good, and 5 Excellent.
Groundedness: Keeps claims consistent with available scene, product and dialogue evidence.
${\overset{―}{s}}_{d} = \frac{1}{| E_{d} |} \sum_{e \in E_{d}} s_{e, d}$
$s_{e, d} \in {1, 2, 3, 4, 5}$
Avoid unsupported assertions, contradictions, or treating information unavailable to the user as something the user already knows.
sₑ,ᵈ is the questionnaire or judge rating for dimension d in episode e; Eᵈ is the set of rated episodes. The scale is 1 Very poor, 2 Poor, 3 Acceptable, 4 Good, and 5 Excellent.
Preference Alignment: Respects stated needs, budget, preferences and constraints.
${\overset{―}{s}}_{d} = \frac{1}{| E_{d} |} \sum_{e \in E_{d}} s_{e, d}$
$s_{e, d} \in {1, 2, 3, 4, 5}$
Consistently respect the user’s expressed needs and constraints, and adapt after rejection or hesitation.
sₑ,ᵈ is the questionnaire or judge rating for dimension d in episode e; Eᵈ is the set of rated episodes. The scale is 1 Very poor, 2 Poor, 3 Acceptable, 4 Good, and 5 Excellent.
Proactiveness: Moves the user toward completing the shopping task.
${\overset{―}{s}}_{d} = \frac{1}{| E_{d} |} \sum_{e \in E_{d}} s_{e, d}$
$s_{e, d} \in {1, 2, 3, 4, 5}$
When reasonable opportunities occur, questions, guidance, comparisons, recommendations and confirmations should advance the task rather than cause stalls or aimless actions.
sₑ,ᵈ is the questionnaire or judge rating for dimension d in episode e; Eᵈ is the set of rated episodes. The scale is 1 Very poor, 2 Poor, 3 Acceptable, 4 Good, and 5 Excellent.
Non-repetition: Avoids unjustified repetition and changes strategy when progress stalls.
${\overset{―}{s}}_{d} = \frac{1}{| E_{d} |} \sum_{e \in E_{d}} s_{e, d}$
$s_{e, d} \in {1, 2, 3, 4, 5}$
Avoid repeating the same question, route, product or wording without new justification; change strategy when interaction continues to stall.
sₑ,ᵈ is the questionnaire or judge rating for dimension d in episode e; Eᵈ is the set of rated episodes. The scale is 1 Very poor, 2 Poor, 3 Acceptable, 4 Good, and 5 Excellent.
Overall Satisfaction: Rates the complete interaction and overall shopping experience.
${\overset{―}{s}}_{d} = \frac{1}{| E_{d} |} \sum_{e \in E_{d}} s_{e, d}$
$s_{e, d} \in {1, 2, 3, 4, 5}$
Consider need fulfillment, process efficiency, trustworthiness and the overall experience. This is an independently assigned rating for the complete interaction.
sₑ,ᵈ is the questionnaire or judge rating for dimension d in episode e; Eᵈ is the set of rated episodes. The scale is 1 Very poor, 2 Poor, 3 Acceptable, 4 Good, and 5 Excellent.
Pearson r: Linear association between paired assistant-level mean scores.
$r = \frac{\sum_{i = 1}^{n} (x_{i} - \overset{―}{x}) (y_{i} - \overset{―}{y})}{\sqrt{\sum_{i = 1}^{n} (x_{i} - \overset{―}{x})^{2} \sum_{i = 1}^{n} (y_{i} - \overset{―}{y})^{2}}}$
xᵢ and yᵢ are paired scores for the same assistant under two evaluation conditions; x̄ and ȳ are their respective means.
Spearman ρ: Rank agreement between paired assistant-level mean scores.
$ρ = corr (rank (x), rank (y))$
rank(x) and rank(y) are the rankings of the paired assistant-level scores under the two evaluation conditions.

Recommendation Utility

R@5: Agreement between top-five recommendations and final-cart products.
$q_{e t} = \frac{| unique (R_{e t} [1 : 5]) \cap P_{e} |}{| P_{e} |}$
$s_{e} (q) = {\begin{matrix} M_{e}^{- 1} \sum_{t = 1}^{M_{e}} q_{e t}, & | P_{e} | M_{e} > 0, \\ 0, & otherwise, \end{matrix}$
$R @ 5 = \frac{100}{N} \sum_{e = 1}^{N} s_{e} (q)$
Pₑ is the final-cart product set, Rₑₜ is the recommendation list at turn t, and Mₑ is the number of lists in episode e. qₑₜ is the list’s recall, sₑ(q) is the episode score, and N is the number of episodes. Non-purchases score zero.
NDCG@10: Top-ten ranking quality with final-cart products as relevance targets.
$z_{e t} = \frac{\sum_{r = 1}^{10} {r e l}_{e t r} / \log_{2} (r + 1)}{\sum_{r = 1}^{min (10, | P_{e} |)} 1 / \log_{2} (r + 1)}$
$s_{e} (z) = {\begin{matrix} M_{e}^{- 1} \sum_{t = 1}^{M_{e}} z_{e t}, & | P_{e} | M_{e} > 0, \\ 0, & otherwise, \end{matrix}$
$N D C G @ 10 = \frac{100}{N} \sum_{e = 1}^{N} s_{e} (z)$
relₑₜᵣ is the binary relevance of rank r to the final cart Pₑ. zₑₜ is the list’s normalized discounted gain, Mₑ is the number of lists, sₑ(z) is the episode score, and N is the number of episodes. Non-purchases score zero.

Navigation Effectiveness

Action Following Rate (AFR): Agreement between the intended and actual legal user action types.
${A F R}_{e} = \frac{\sum_{t \in V_{e}} 1 [{\hat{a}}_{e t} = a_{e t}]}{| V_{e} |}$
$A F R = \frac{100}{| E_{V} |} \sum_{e \in E_{V}} {A F R}_{e}$
Vₑ is the set of events with a legal suggested action. âₑₜ and aₑₜ are the suggested and actual action types, respectively. Eⱽ is the set of episodes with eligible events.
Local Navigation Alignment (LNSR): Joint agreement on action type and comparable local parameters.
${L N S R}_{e} = \frac{\sum_{t \in D_{e}} 1 [{\hat{a}}_{e t} = a_{e t}] b_{e t}}{| D_{e} |}$
$L N S R = \frac{100}{| E_{D} |} \sum_{e \in E_{D}} {L N S R}_{e}$
Dₑ is the set of navigation events with comparable local parameters. âₑₜ and aₑₜ are the suggested and actual action types; bₑₜ indicates whether their parameters match. Eᴰ is the set of episodes with eligible events.
Cumulative Goal Visibility (CGV): Share of eligible navigation goals visible by a given horizon.
$u_{g, s} = 1 [v i s i b l e (g, t_{g} + s)]$
${C G V}_{h} = \frac{100}{| G |} \sum_{g \in G} {max}_{0 \leq s \leq h} u_{g, s}$
G is the set of eligible navigation goals, t_g is the turn when goal g is given, and u_g,s indicates whether it is visible s turns later. h is the observation horizon; End extends to the episode’s end.