Can an Offshore Team Reproduce a Redaction Decision From Written Rules?

A redaction rule is only useful if different workers apply it the same way to the same file. If two trained people reach different decisions from the same written instruction, the rule is not yet a reliable control. Reproducibility testing turns that concern into a measurable question. This report describes a method for asking an offshore team to redact the same samples independently and comparing the results against the written rule and each other.

Research question and decision supported

The question is whether two or more workers, using only the current written rules and approved examples, make the same redaction decisions on the same sample. The decision supported is whether the rule set is ready for live work or needs clearer categories, better examples, or a stronger second-review step. A second decision is whether a particular data category should be removed by default rather than left to judgment. The test does not measure whether a person is careful in general. It measures whether the written material determines the decision for a defined sample.

Define what reproduction means

Reproduction has two levels. The first is outcome agreement: workers remove the same values or leave the same values in place. The second is reason agreement: workers cite the same rule for the decision. Outcome agreement alone can hide a lucky match, while reason agreement alone can hide different outputs from the same stated rationale. Score both. A sample can also have more than one acceptable answer if the rule explicitly allows a range. In that case, define the acceptable set before the test and score against it. Without a pre-defined standard, the exercise becomes a debate about preferences rather than a test of the rule.

Record the acceptable set in the answer key before anyone starts. Put each item in one of three states: must remove, may leave with a reason, or must leave. The may-leave state is where most disagreement lives, so write a short justification for it. A rule that permits both outcomes is not necessarily bad, but the team should know that the choice is intentional.

Build the rule set under test

Use the exact rule set that workers see in production, including the category list, the examples, and the escalation path. Do not give the test group a cleaned-up version that is clearer than the live document. Record the version and date. If the live material is spread across several documents, include all of them and note where they conflict. The test should reflect the real task, including the need to find the relevant rule. If a rule is known only to a supervisor, it is not part of the written standard and should not be supplied during the exercise.

Keep the test packet identical for every participant, including the order of the samples. A different order can prime attention toward one category and distort the comparison. State whether participants may use search or ask a clarification question, and record every clarification. The same mechanics on every attempt make the agreement rate meaningful rather than an artifact of how the session was run.

Select representative samples

Choose samples that cover the categories in the rule and the formats the team handles. Include a document, a spreadsheet, a screenshot, and a file with hidden metadata if those are common. Add at least one borderline case where a value could be identifying only in combination with another field. Remove live personal data from the test set and replace it with realistic fictional values. Keep the source of each sample and the expected decision in a key that the participants do not see. The set should be small enough to complete in one session and varied enough to expose ambiguity.

Run independent decisions

Give each participant the same instructions, the same time limit, and the same sample set. Ask them to record what they removed, what they left, and the rule they applied. Do not let participants discuss the samples during the exercise, because the point is independent application. If the task normally involves a second reviewer, run the first pass independently and then observe the review conversation separately. Capture the time taken and any question the participant had to guess at. A question that cannot be answered from the written material is a finding about the rule, not a failure by the participant.

Keep the exercise short enough that fatigue does not become the main variable. Forty to sixty minutes is usually enough for six to ten samples. Give participants a short break if the set is longer. Note the time each person finished so a very fast result can be checked for missed categories rather than praised as efficiency.

Score outcome and reason agreement

Build a simple matrix with the expected outcome, the participant's outcome, and the cited rule. Calculate agreement by item and by participant. For each disagreement, code whether the rule was absent, ambiguous, conflicting, or missed. Include the cases where participants agreed with each other but not with the expected answer, because a shared misreading is a serious rule defect. Report the agreement rate with the number of items and participants. A high outcome rate with a low reason rate suggests that people are copying a habit rather than applying the rule.

Include a short comment field for each decision so the scorer can see the reasoning. Do not average the outcome and reason scores into one number, because the two answer different questions. Report the item-level results beside the participant-level totals. A single participant who disagrees on one high-risk item can matter more than several low-risk agreements.

Weight the items by consequence when you summarize the result, but show the unweighted count too. A redaction error on a public document and one on an internal draft do not carry the same risk. If the team does not have a consequence rating, use a simple high and standard split. Explain the weighting in the report so the rate is not mistaken for a pure average.

Diagnose the disagreements

Read the disagreements together with the participants and the rule owner. Ask which sentence produced the different readings. Look for categories that overlap, examples that contradict the text, and terms that mean different things to different teams. Check whether a participant had to use knowledge from outside the written rule, such as a prior client instruction. If the rule owner cannot explain the intended answer from the document, the rule is unclear by definition. Record the diagnosis before rewriting anything, so the repair responds to the actual source of disagreement.

Re-test after the repair

Revise the rule, the examples, or the category list based on the diagnosis. Keep a copy of the old version so the change can be compared. Run a second test with a fresh sample set and, where possible, the same participants. Do not reuse the exact items, because memory will inflate agreement. Compare the outcome and reason rates with the first round. A modest improvement may be enough if the remaining disagreements are genuine judgment calls that the escalation path handles. Re-test again after a major format or system change.

Keep the old and new answer keys side by side when the expected decision changes. A change in the key may reflect a real policy update or an accidental revision by the test owner. Have the rule owner confirm the new key before the second round. Without that check, the re-test can show improvement simply because the answer key moved closer to what participants already did.

Limitations

The test measures agreement on a sample, not the overall quality of every live decision. Participants may be more careful during an exercise than during a busy shift. The expected answer is itself a judgment, and the rule owner may be wrong about the intended standard. Small teams produce unstable rates, and a single ambiguous item can change the result. State the sample size, the rule version, and the scoring method. Use the test as a diagnostic for the written rule, not as a certification that the team will always redact correctly.

Sources

1. U.S. Bureau of Labor Statistics, Occupational Employment and Wage Statistics
2. U.S. Department of Labor, Fact Sheet 22: Hours Worked
3. U.S. Small Business Administration, Hire and Manage Employees
4. NIST Cybersecurity Framework 2.0
5. NIST SP 800-53 Security and Privacy Controls
6. CISA Cyber Essentials
7. APQC Process Classification Framework
8. ISO 9001 Quality Management Principles
9. NIST Privacy Framework
10. FTC Data Security Guidance