Annotation, Coding, and Classification as the Core Use Case
The most direct expression of this topic is a cluster of papers that treat LLMs as annotators or classifiers substituting for, or augmenting, human coders. Yang2026-tq uses a human-in-the-loop LLM pipeline simply to find the relevant literature — estimating the prevalence of social-media-data studies across 59 journals — treating the LLM as a scalable screening instrument rather than an analytic engine. Ober2026-vd pushes further into interpretive territory, combining topic modeling with LLM-generated cluster labels to refine a human-derived qualitative codebook from focus-group transcripts, explicitly arguing that topic modeling’s mathematical transparency should anchor LLM labeling so that opacity doesn’t creep into thematic analysis. Meher2025-qb moves from annotation to domain-specific classification proper: fine-tuning Llama 3.1 via QLoRA to classify terrorism event types, showing that parameter-efficient adaptation can outperform base LLMs by large margins on rare categories while running on consumer hardware — a methodological roadmap for political scientists rather than a one-off application. DiGiuseppe2025-es extends the annotation paradigm to measurement, proposing LLM-paired comparisons to scale open-ended survey responses into continuous latent traits, addressing the perennial tradeoff between rich but costly open-ended coding and impoverished closed-ended items. Tan2024-vl surveys this entire subfield, organizing it around generation, assessment, and downstream utilization of LLM-produced labels — providing the taxonomic scaffolding against which the other papers in this cluster can be read as instances.
Running beneath all of these is the question Brown2025-jk answers directly: do LLM annotators systematically misrepresent demographic subgroups on contentious classification tasks? Its finding — that bias is dataset-specific rather than model-specific, and dwarfed by item difficulty (label entropy) — complicates any simple story about LLMs replacing or ignoring particular human perspectives, and offers an evaluation template (compare against annotator disagreement, not just ground truth) that the other coding papers in this collection implicitly need but do not always supply.
Fine-Tuning, Reproduction, and Automated Review: Scaling Up the Scholarly Task
A second strand asks what happens when LLMs are pointed not at raw text but at the scholarly workflow itself — reproducing findings or reviewing manuscripts. Alizadeh2026-es benchmarks frontier coding agents (Claude Code, Codex) on reproducing 221 published social-science results, finding startlingly high accuracy but also documenting failure modes central to any human-in-the-loop design: agents become sycophantic under confirmatory prompting, and providing paper PDFs improves accuracy while corrupting agents’ ability to correctly flag non-reproducible tasks. Jayaram2026-wd extends the automation frontier to peer review itself, describing Google’s Paper Assistant Tool and its four-level taxonomy of AI roles in review (from author tool to full automation); its pilot deployment at STOC and ICML shows real detection of substantive errors alongside persistent hallucination and grounding problems — an empirical illustration of exactly the trust/verification tension Alizadeh2026-es surfaces analytically. Read together, these two papers suggest that “human-in-the-loop” for high-stakes scholarly verification is less a fixed design choice than a moving target, since agent capability, prompt framing, and provided context all shift where the loop needs to close.
Auditing the Infrastructure Beneath Computational Methods
A third, more skeptical strand interrogates the data and platform infrastructure that computational methods — LLM-based or otherwise — depend on, and by extension warns against treating any pipeline as a neutral instrument. Rieder2025-ju audits YouTube’s Data API and finds it “forgetful by design,” with severe temporal decay and irreproducibility that undermine any downstream classification or annotation task built on it. Ulloa2024-jm shows that web-scraped news content diverges systematically from what users actually see, with errors concentrated in ways that bias substantive conclusions rather than adding random noise. Balluff2026-if generalizes this skepticism to LLM adoption itself, arguing that convenience has outpaced reflexivity: smaller, open models often match LLM performance at a fraction of the environmental and reproducibility cost, and corporate control over model updates threatens the very replicability that methods papers like Meher2025-qb and Alizadeh2026-es depend on. Together these three papers function as a standing methodological check on the enthusiasm elsewhere in this set — a reminder that fine-tuning accuracy or annotation agreement scores are only as trustworthy as the data infrastructure and model provenance underneath them.
Human-in-the-Loop as a Design Principle, Not a Caveat
Matias2025-px provides the clearest normative argument for why human involvement should be constitutive of AI evaluation rather than a compliance afterthought, identifying five evaluation stages — equipoise, measurement, explanation, inference, interpretation — where lived-experience participation improves scientific rigor, not just legitimacy. This reframes the “human-in-the-loop” language used more instrumentally elsewhere (e.g., codebook refinement in Ober2026-vd, sycophancy checks in Alizadeh2026-es) as part of a broader epistemic claim: participatory design is often the only way to catch errors that purely technical evaluation misses, as in the cited Allegheny Family Screening Tool reanalysis.
The Boundary Case: LLMs as Interactive Interlocutors
A final cluster — Costello2024-bg, Szabo2026-rd, Dubey2026-bl, and Rauchfleisch2026-fa — sits at the edge of “research workflow” proper: these papers deploy LLM-powered chatbots not to annotate or classify scholarly material but as the intervention under study, engaging members of the public in dialogue about conspiracy beliefs, misinformation, or contested news. Yet they belong here because they generate exactly the kind of human-in-the-loop evaluation evidence the rest of this topic needs: Costello2024-bg demonstrates that personalized, evidence-dense LLM dialogue durably shifts entrenched beliefs, challenging assumptions about the limits of corrective information; Szabo2026-rd shows conversational format itself — adaptability, trust-building, minimized friction — matters as much as argument content; Dubey2026-bl finds that trust and perceived usefulness, not source skepticism, govern acceptance of LLM-based balanced-news tools even among conspiracy believers; and Rauchfleisch2026-fa delivers a pointed methodological lesson for transparency design broadly — disclosing an AI’s identity does little, while disclosing its intent substantially blunts persuasion. Read against the annotation and review papers above, this last finding is a useful cautionary parallel: just as identity labels are a weak transparency mechanism for persuasive chatbots, surface-level “human-in-the-loop” gestures (a reviewer skimming LLM output, a label reading “AI-assisted”) may be similarly weak safeguards unless the loop is designed around what the system is actually trying to do and where it is most likely to fail.