Research · · Verified
What Share of Offshore Quality Findings Repeat in Consecutive Reviews?
A paired-review method for measuring repeat findings, separating a persistent instruction gap from a new defect with the same label.
On this page
Research question and decision supported
The question is what share of quality findings in one review period also appear in the next review of the same task type or the same worker. The decision supported is whether the correction process is working, whether the finding taxonomy is too broad, and whether the reviewer is recording the same issue under different labels. A repeat finding is not automatically a failure, because a complex task can produce the same minor note twice. The method measures persistence, and the interpretation depends on the severity and the cause. This distinction keeps the team from treating every repeated label as evidence of neglect.
State whether the comparison covers the same workers, the same task types, or both. A worker-level comparison answers a different question from a task-level comparison. Report the unit of analysis beside the rate so the reader does not assume the wrong scope. The decision should follow the unit that produced the finding.
Define a repeat finding
A repeat finding is the same defect, in the same task type, with the same underlying cause, observed in two consecutive review periods. A different defect with a similar label should not count, and a new instance of the same defect after a documented fix should be classified separately. Write the definition with two examples and one counterexample so reviewers can apply it. Record the finding identifier, the task, the cause code, the severity, and the review period. Link the second finding to the first rather than relying on the text description. If the link is uncertain, mark it as a possible repeat and report it separately.
Keep a short example bank beside the definition. Include a true repeat, a similar-looking but different defect, and a recurrence after a clean period. Review the bank with the reviewers at the start of each period. Examples are more effective than a longer written rule when the taxonomy is abstract.
Build paired review periods
Select two consecutive review periods with comparable volume and task mix. Use the same sampling rule in both periods so a change in the sample is not mistaken for a change in quality. If the team reviews every item in one period and a sample in the next, report that difference before comparing the rates. Keep the reviewer group stable where possible, and record any change in the rubric or the acceptance standard. Build a pair for each task type and each worker included in both periods. The analysis unit should be the task and finding, not the total number of notes.
Check that the review volume and the sampling method are comparable before calculating the rate. If the second period used a larger sample, report the rate by task and not only in total. Note any change in the reviewer group or the acceptance standard. A paired design is only paired if the surrounding method is stable.
Separate repeat from recurrence
A repeat is the same defect persisting across periods. A recurrence is the same defect appearing again after a period without it. The two have different causes and different responses. A repeat may indicate that the instruction was never corrected, while a recurrence may indicate that the fix decayed or the worker returned to an old habit. Code the sequence explicitly and show both counts in the report. Do not combine them into one persistence rate without explaining the choice. If the team only has two periods, state that recurrence cannot be distinguished from a first repeat.
Show both counts in the same table with a short definition for each. A recurrence after a clean period may indicate that a correction did not stick, while a repeat may indicate that no correction was made. The response differs even when the visible defect looks the same. Use the sequence, not just the label, to decide what to fix.
Measure the repeat share by severity
Report the share of findings that repeat, broken out by severity and cause. A high repeat rate among low-severity formatting notes is a different problem from a high rate among high-consequence errors. Show the number of findings, the number of repeats, and the number of tasks reviewed in each period. Use the task as the denominator when the review samples a fixed number of items per worker. Add the median interval between the first and second finding. A single aggregate percentage can hide a small number of serious repeats behind many minor ones, so keep the severity split visible.
Read the high-severity repeats in full before publishing the aggregate. A small number of serious repeats can be more important than a larger number of minor ones. Check whether the second finding was discovered by the same reviewer or a different one. The discovery path can reveal whether the first correction was ever applied.
Compare by cause and reviewer
Group repeats by cause code such as unclear instruction, missing source, access problem, training gap, or execution error. If most repeats cite unclear instruction, the correction belongs with the instruction owner rather than the worker. If most cite execution, check whether the worker received a specific correction and enough practice. Compare reviewers to see whether one person applies a broader definition or records more notes per item. Do not assume that a higher finding count means a stricter reviewer, because the sample may differ. Discuss the pattern with the reviewers and revise the examples before changing the score.
Ask reviewers to explain one repeat and one non-repeat in their own words. Their explanation often reveals a definition problem that the codes hide. Compare the explanations across reviewers and look for a shared ambiguity. If the ambiguity is real, fix the rubric before changing the coaching.
Test the finding taxonomy
A repeat rate can be inflated by a taxonomy that forces different problems into one label. Take ten repeats and ask two reviewers to code them independently from the original evidence. If they disagree, the label is doing too much work. Split the category or add an example that makes the boundary clear. Also check whether the same defect is recorded as two separate findings in one review, which would distort the denominator. Keep a version of the taxonomy and report which version was used. A measure is only as useful as the categories behind it.
Run the independent coding exercise on findings from both periods, not only the second. A taxonomy that changed between periods can create false repeats. Record the version used for each period and the date of the change. If the change was substantial, report the periods separately rather than as one trend.
Use the result to change one correction
Choose the most consequential repeat pattern and write a specific correction. If the same missing field repeats, change the intake form or the checklist rather than reminding the worker again. If the same judgment repeats, add a worked example and a short review with the owner. If the same reviewer produces the repeat finding, check the rubric and the calibration sample. Assign an owner and a review date, then repeat the paired analysis with the same definitions. Do not add a new layer of review before confirming that the original correction was actually delivered.
Confirm that the earlier correction was actually delivered before designing a new one. Check the training record, the updated instruction, and the worker's access to the current version. A repeat that follows an undelivered correction is a distribution problem. Fix the distribution first, then measure again before adding more training.
Limitations
The method depends on finding records that are complete and consistently coded. Reviewers may change their standards between periods, and a new rubric can create apparent repeats that are not comparable. A two-period comparison cannot show whether a finding would have persisted longer without the correction. Small samples and multiple task types make the rate unstable, and the same worker may handle a different mix in each period. Report the sample, the taxonomy version, and any reviewer changes. Treat the result as evidence for a targeted correction, not as a final judgment about quality.
Finding records may be incomplete, and a reviewer may record the same issue at different levels of detail. The paired method assumes the periods are comparable, which is often only partly true. A repeat rate is not a measure of intent or effort. Use it to locate a persistent instruction or review problem, and keep the interpretation close to the actual evidence.