1The problem is not capability
The public debate about generative models has largely been conducted in the currency of capability. Can the model pass the bar exam, write the essay, read the radiograph? This framing has produced a great deal of argument and very little guidance, because capability was never the binding constraint on professional adoption.
The binding constraint is that these systems fail in a particular and inconvenient way. A model that has no information about a question does not decline to answer it. It produces an answer with the same fluency, formatting and apparent confidence as one it has strong grounds for. This behaviour is well documented across model families and task types, and is now conventionally described as hallucination.1 Its practical significance is that the output carries no usable signal about its own reliability. Fluency and correctness are produced by the same mechanism and are therefore not separable by inspection.
Two incidents illustrate the range. In Mata v. Avianca, counsel submitted a federal filing containing judicial opinions that did not exist, generated by a chatbot and not independently checked; the court imposed sanctions.2 At the other end of the severity scale, a widely reported 2024 failure of an AI search summary recommended adding adhesive to pizza sauce. The second is merely absurd. The first cost money and professional standing. What the two share is structural: in each case the output was plausible on its face, and nothing in its presentation distinguished it from a correct one.
This is why the useful question is not what the model can do. It is where, in a given workflow, a wrong answer would survive undetected and cause harm.
2Wrangling, defined
An AI wrangler is a domain practitioner who treats generative output as a proposal rather than a result, and who has determined in advance which proposals require independent verification, by what means, and at what cost. Wrangling is the design and maintenance of that verification structure.
Three features distinguish this from prompt engineering. First, it is domain-bound: the failure modes that matter and the tolerances that apply are properties of the field, not of the model, and cannot be supplied by a technologist alone. Second, it is structural rather than conversational: the object of design is the workflow, including the points at which the human is required to intervene, not the phrasing of an individual request. Third, it is adversarial: the wrangler's stance toward the output is that of a reviewer looking for the error, not a user hoping there is none.
The economic content of the role is an asymmetry. For a broad class of tasks, generation has become very cheap while verification has not. Drafting a contract clause, a literature summary or a function now costs seconds. Establishing that the clause is enforceable in the relevant jurisdiction, that the cited papers exist and say what is claimed, or that the function is correct and secure costs very nearly what it always did. Where that ratio is extreme, the model relocates effort rather than removing it, and the total saving depends entirely on how efficiently the verification is organised.
This gives a criterion sharper than "is the model good enough". A task is safely delegable when verification is cheaper than production and independent of the model that produced the output. Compiling and testing generated code satisfies both conditions, which is why software adoption ran ahead of other fields. Confirming a clinical judgement satisfies neither, which is why it has not.
3Five domains
The same structure recurs across professions, with the locus of verification shifting according to where the cost of an undetected error is concentrated.
3.1 Research
The characteristic failure is the fabricated or misattributed citation: a reference whose authors, year or venue do not correspond to any real work, presented in correct bibliographic form. Because the format is right, the error survives casual review and propagates into whatever cites it. The wrangling response separates the two things a model is being asked for. Synthesis and hypothesis generation are accepted as drafts. Every bibliographic claim is resolved against a real record before it is relied upon. The rule is blunt but effective: a citation that has not been opened has not been checked.
3.2 Education
The instinctive response to models that can complete assignments has been detection, which places the institution in an arms race it is unlikely to win and which imposes a serious cost when the detector is wrong. The wrangling response changes what is assessed. If the artefact can be generated, the artefact is no longer evidence of learning, and assessment moves to the process: the critique, the correction, the justification of a choice against a source. An assignment that asks a student to locate and repair the errors in a machine-generated argument tests exactly the competence this essay is about, and is not completed by the tool under examination.
3.3 Medicine
Here the asymmetry is at its most severe, since verification requires the clinical judgement the model was meant to supply, and an undetected error can be irreversible. Two properties make the models genuinely useful nonetheless: they are strong at pattern recognition over high-dimensional data, and they do not tire. Two properties make unsupervised use untenable: their outputs reflect the distribution of their training data, so they are systematically biased toward common presentations, and they express no reliable measure of their own uncertainty. The wrangling posture follows directly. The model is used to widen the differential and to flag what might have been missed, which is a recall-oriented task where a false positive is cheap. It is not used to narrow the differential, which is a precision-oriented task where a false negative is not.
3.4 Law
Legal work presents an unusually clean version of the problem, because the ground truth is external, authoritative and machine-queryable. Whether a case exists, remains good law, and stands for the proposition cited are all questions with definite answers available from a citator. The corresponding discipline is therefore mechanical and non-negotiable, and its violation is what produced the sanctions described in section 1. The instructive point is that the cheapest possible verification step, resolving each citation against a database, was the one omitted.
3.5 Engineering and the sciences
Software is the domain where verification is most nearly automatic. A compiler, a test suite and a static analyser will reject a large class of defects without human attention, which is precisely why generated code was adopted first and most widely. The residue is what those tools do not catch: architectural coherence, security properties that depend on context, and maintainability. The practitioner's work shifts accordingly, from producing implementation toward specifying interfaces and reviewing what has been produced against them.
Scientific applications introduce a failure mode that verification cannot address at all. Generative models for molecular design have accelerated legitimate work substantially,3 and the same models, with the optimisation objective inverted, will propose toxic compounds just as readily. This was demonstrated deliberately and reported responsibly in 2022, when a drug-discovery model was re-run to reward rather than penalise predicted toxicity and generated tens of thousands of candidate molecules in a matter of hours.4 No amount of checking the output helps here, because the output is correct. The control has to sit upstream, in access to the model and in the governance of what it is asked to optimise.
| Domain | Delegated to the model | Characteristic failure | Cost of verification |
|---|---|---|---|
| Research | Synthesis, hypothesis generation | Fabricated or misattributed citation | Low, and fully external |
| Education | Materials, practice items, feedback | Assessment measures the tool, not the student | Moderate; requires redesign, not detection |
| Medicine | Triage, differential expansion | Bias toward common presentations | High; needs the expertise being assisted |
| Law | Discovery, summarisation, drafting | Confident citation of non-existent authority | Low, and fully external |
| Engineering | Boilerplate, tests, translation | Plausible code with latent defects | Low for correctness, high for design |
| Molecular design | Candidate generation | Dual use; the output is correct | Not applicable; control must precede generation |
The pattern in the last column is worth stating explicitly. The domains that adopted these tools fastest are not the ones where the models are strongest. They are the ones where checking is cheapest and does not depend on the model. That is a claim about verification infrastructure, not about intelligence.
4What follows
If the argument holds, three consequences follow that are worth separating from the general enthusiasm.
The first is that expertise becomes more valuable rather than less, but its function changes. Evaluating a proposed answer requires the same knowledge as producing one, and often more, since it must be done without the scaffolding that producing it would have provided. What erodes is the market for producing routine artefacts. What appreciates is the judgement required to accept or reject them. This is an uncomfortable transition, because the erased work is how practitioners have traditionally acquired the judgement.
The second is that the returns to these systems are bounded by verification cost, not by model quality. An organisation that deploys a capable model without a verification structure has not reduced its costs; it has converted a known cost into an unknown liability. This predicts, correctly so far, that adoption tracks the availability of cheap external ground truth rather than the difficulty of the task.
The third is that wrangling is teachable, and is not currently taught. It consists of identifiable components: knowing the failure modes of the class of model in use, knowing which claims in one's own field are externally checkable and by what means, designing workflows so that unverified output cannot reach a consequential decision, and maintaining the discipline to treat fluent text as unevidenced until it is evidenced. None of this requires the ability to train a model. All of it requires domain expertise, which is why it cannot be delegated to a technical function.
5Conclusion
The framing of generative models as either replacements for professionals or as ordinary tools both miss what is distinctive about them. They are unusually productive and unusually unreliable at the same time, and they do not distinguish between the two states in anything a user can observe. The professional consequence is not that work disappears. It is that the work relocates, from producing the artefact to establishing that the artefact is sound.
That relocated work is what I have called wrangling. It is not glamorous, it is not automatable by the system that created the need for it, and at present it is largely improvised by individuals rather than designed by institutions. The professions that formalise it will get the benefit of these systems. The professions that do not will get the liability instead.
6References
- Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y., Madotto, A., and Fung, P. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 2023.
- Mata v. Avianca, Inc., No. 22-cv-1461 (S.D.N.Y. 2023). Opinion and order on sanctions, 22 June 2023.
- Jumper, J., Evans, R., Pritzel, A., et al. Highly accurate protein structure prediction with AlphaFold. Nature, 596:583-589, 2021.
- Urbina, F., Lentzos, F., Invernizzi, C., and Ekins, S. Dual use of artificial-intelligence-powered drug discovery. Nature Machine Intelligence, 4:189-191, 2022.
The 2024 search-summary incident referred to in section 1 is described from contemporaneous press reporting rather than a primary source, and is offered as illustration only.