01 / Quality & compliance
Define quality for the intended use.
Quality is not a single score that travels unchanged between projects. A transcript used for search, a subtitle file used for viewing and a reference transcript used for model evaluation have different failure modes. Start by describing what an error would prevent the downstream user from doing.
Translate that description into a task-specific rubric. Define the unit being judged, the evidence reviewers may use and the treatment of ambiguous cases. Record which defects are merely cosmetic and which require escalation. This playbook is a planning framework, not a certification or a promise of a particular project result.
- Correctness
- Does the output faithfully represent the available source or satisfy the stated task?
- Completeness
- Are required segments, fields and relevant context present?
- Usability
- Can the intended person or system use the delivery without avoidable repair?
02 / Quality & compliance
Calibrate people before comparing scores.
Give reviewers the same small set of representative examples and ask them to apply the draft rubric independently. Discuss differences in interpretation before expanding the workload. Calibration is especially useful where meaning, domain knowledge or cultural context influences a judgment.
Keep the resolution, not just the winning label. An adjudication note should explain which evidence mattered and which rule applies next time. Update the guidance and version it. When you add a new task type or specialist domain, repeat the relevant calibration instead of assuming earlier alignment will transfer.
03 / Quality & compliance
Put different checks at different gates.
Use deterministic checks for things that can be tested reliably: required fields, permitted label values, timestamp ordering, duplicate IDs and file integrity. Keep those results separate from judgments about meaning, relevance or safety.
Route unclear, high-impact or disputed items to an appropriately qualified reviewer. Decide in advance when a second review is necessary and who can adjudicate. An unresolved item should remain visibly unresolved; a tidy-looking export is not a reason to conceal uncertainty.
| Gate | Question | Evidence to retain |
|---|---|---|
| Input validation | Is this source usable and in scope? | Rejection reason and source ID |
| First-pass review | Does the output follow the rubric? | Label or correction with reviewer notes |
| Specialist / QA review | Does this case need additional judgment? | Issue category and escalation rationale |
| Adjudication | How is a material disagreement resolved? | Decision, justification and guidance version |
| Release | Does the batch meet the agreed acceptance rules? | Validation report and approval record |
04 / Quality & compliance
Sample deliberately. Report limitations.
A convenient sample may miss the exact conditions that make a dataset difficult. Define how review samples are selected across relevant slices, such as locale, domain, source type or acoustic condition. Combine representative sampling with targeted review of known high-risk cases, and label the two purposes separately.
Report the denominator, the measurement method and exclusions alongside an error rate or score. If you deliberately oversample difficult cases, do not present the result as an unqualified estimate for the whole population. Keep comparisons consistent across versions and call out changes in test composition.
- Coverage
- Which slices were checked, and which remain untested?
- Error taxonomy
- Which error types recur, and what should change because of them?
- Decision
- Is the evidence sufficient to accept, correct, re-review or pause the batch?
05 / Quality & compliance
Keep compliance questions separate from quality scores.
A well-labeled dataset does not automatically establish that its collection, use or sharing is permitted. Maintain the relevant source, permission, usage and handling records alongside the quality documentation. Assign an owner to unresolved questions and confirm the requirements for your specific project with the appropriate legal, privacy or security team.
Ask which contractual commitments and supporting documents are available for your engagement. Do not infer a certification, regulatory status, processing location or retention rule from a quality workflow diagram. Those details need their own current evidence.
Further reading
Further reading on a voluntary framework for managing AI risk; a reference, not a claim of Verbalscripts certification.