A sample is useful when you know what it represents.

Imagine an operation handling 10,000 conversations in a month. It reviews a random sample of 200. That is 2 per cent coverage. The sample can provide useful evidence about the wider service, provided the selection and assessment support the conclusion. It cannot identify what happened in each of the other 9,800 conversations. These figures are illustrative, not a description of an ellio client.

The distinction matters because quality teams answer several different questions. Is the service improving overall? Which customers need help now? What should an adviser learn? Which process keeps failing? A review designed to estimate general performance may be unsuitable for finding a rare but serious problem. Decide which decision the evidence needs to support before deciding how much to review.

A representative sample helps describe the service within its stated limits. A targeted review deliberately concentrates on something: disputed decisions, repeat contacts, difficult handovers or a particular digital-agent behaviour. Both are useful. They answer different questions and should remain distinguishable in the reporting.

In the illustrative operation, suppose the team also reviews 50 conversations connected to complaints. Combining those cases with the random 200 and publishing one failure rate would change what the number represents. The targeted cases were selected because something might be wrong. Report their findings as a separate investigation, then use appropriate evidence to establish whether the same issue is widespread.

Segmentation helps explain change. If voice, chat, email, service types or customer groups have different characteristics, an overall average can move because their proportions changed. Keep enough detail to understand the movement without producing a report nobody can use.

Automated assessment can make a wider review practical. It can also repeat a poor interpretation at greater scale. A system might reward the presence of a phrase while missing that the promised action never happened. It might classify a planned follow-up as failed resolution, or mistake an adviser questioning an incorrect record for a knowledge gap.

Build the assessment around observable criteria. State what counts as meeting the requirement, which evidence supports the decision and when a conclusion is unavailable. Compare automated judgements with reviewed examples, including ordinary cases, difficult exceptions and cases where reviewers initially disagree. Resolve the reason for disagreement before treating either score as the reference answer.

Look at missed issues as well as incorrect flags. A reassuring agreement rate can hide a system that performs well on easy cases and misses the concern you most need it to detect. The test set needs to include those situations, and the report needs to show their results separately.

To find missed issues, examine some conversations the system cleared, using a selection designed for that purpose. Reviewing only flagged cases can tell you how often an alert was justified, but cannot reveal everything the system overlooked. Keep a separate set of examples for evaluation when changing the assessment. Reusing only the examples that shaped the change can give an overly comfortable result. Repeat the comparison when the service, guidance or assessment changes, and investigate whether performance differs by channel, language or work type. The amount of checking should reflect the consequence of a wrong judgement.

An adviser can give an appropriate answer, follow the process and still depend on another team to complete the task. A digital agent can end a conversation successfully while leaving the customer to return later. Review what can be known at the end of the interaction, then revisit the outcome after the relevant observation period.

Human input within a digital conversation needs careful interpretation. In ellio's service, a digital agent can consult an expert and retain the interaction; a required handover transfers responsibility with context, while digital guidance continues. Quality assessment asks whether the right expertise informed the decision and whether the task was completed. It should distinguish consultation from transfer and examine the reason for each.

For an order enquiry, that might mean checking whether the promised delivery update arrived and whether the customer had to ask again. For a payment arrangement, the relevant outcome emerges over a different period. The service needs definitions suited to the work, with planned follow-ups distinguished from avoidable repeat contact.

This protects the quality judgement from hindsight. A later failure should trigger investigation into what changed or did not happen. It should not automatically rewrite an earlier decision that was appropriate on the evidence available at the time.

A finding about unclear explanation may belong in coaching. A finding about a misleading bill belongs with the team able to change the bill. A missing system action may require engineering or process ownership. Sending every finding to the adviser creates work without necessarily addressing the cause.

Each material finding needs a next step, an owner and a way to check the effect. Connect the finding to the knowledge and intelligence behind the action, timing and channel, as well as the conversation itself. An unsuitable recommendation may originate before anyone speaks. Reviewed outcomes and expert corrections then inform improvements to shared knowledge, guidance and predictive models. Keep the immediate customer remedy visible alongside that learning.

Review capacity also belongs in the design. A screening system that generates more alerts than specialists can examine creates a queue of unresolved uncertainty. Prioritise by consequence and urgency, monitor the queue and narrow the scope if the service cannot respond responsibly to what it finds.

Choose one recurring customer need and one decision the quality report is meant to support. Map the eligible interaction population, the current review selection and the people using the findings. Check whether missing channels, incomplete records or uneven case selection change the meaning of the headline score.

Then review a representative set and a separate set selected for a known concern. Ask reviewers to explain their decisions from the evidence. Trace a small number of findings into coaching, a customer remedy or a process change. Revisit whether those actions occurred and whether they addressed the issue.

The purpose is to improve the relationship between evidence and action. Sampling remains useful. Wider assessment can add value. Neither approach removes the need for a clear question, dependable judgement and someone responsible for what happens next.