Ferrara, E. (2026). The Generative AI Paradox: GenAI and the erosion of trust, the corrosion of information verification, and the demise of truth. arXiv [cs.CY].
Summary
Ferrara argues that the gravest risk posed by generative AI is not the isolated deepfake but the rise of synthetic realities: coherent, interactive, and personalized information environments in which content, identity, and social interaction are jointly fabricated. Rather than treating GenAI as merely producing “more misinformation,” the paper formalizes it as a systemic threat to the shared epistemic ground societies depend on. The central thesis — the Generative AI Paradox — holds that as synthetic media becomes ubiquitous, rational actors may discount all digital evidence, empowering strategic denial and imposing an “epistemic tax” on institutions. The paper is conceptual and agenda-setting, offering a layered framework, an expanded harm taxonomy, a case bank of recent incidents, and a defense-in-depth mitigation stack oriented toward a new research program on “epistemic security.”
Key Contributions
- Formalizes synthetic reality as a four-layer stack: content, identity, interaction, and institutions — mapping attack surfaces to defensive levers.
- Expands GenAI harm taxonomies by adding epistemic and institutional integrity as a cross-cutting harm category.
- Identifies seven qualitative shifts distinguishing GenAI from prior deception tech: cost collapse, throughput, customization, micro-segmentation, synthetic interaction, provenance gap, and trust erosion.
- Compiles a mechanism-focused case bank of 2023–2025 incidents linked to stack layers.
- Proposes a layered mitigation framework (provenance, platform governance, institutional redesign, public resilience, policy accountability).
- Sets a research agenda for epistemic security with candidate metrics (authenticity coverage, correction latency, manipulation susceptibility, verification load, attribution stability).
Methods
The paper is theoretical and synthetic rather than empirical. It develops a conceptual framework (the four-layer stack), performs a taxonomic expansion of harm categories, and conducts a mechanism analysis of the qualitative differences GenAI introduces. To ground the framework, Ferrara constructs a curated case bank of representative 2023–2025 incidents, selected for documentation quality, mechanism diversity, and linkage to the stack layers. Mitigation strategies and proposed measurement constructs are then mapped back onto the framework.
Findings
- Five recurring case categories illustrate synthetic-reality harms: high-conviction impersonation fraud (e.g., the ~$25M Arup Hong Kong deepfake video-conference scam), election-adjacent synthetic outreach (AI-generated Biden robocalls), non-consensual synthetic sexual imagery (the Taylor Swift incident), fabricated everyday documentation (AI-generated receipts/invoices), and compromised generative pipelines (malicious models, backdoors, data poisoning).
- A common operational pattern: cheap high-conviction artifact production → insertion at workflow choke points → scale-driven exposure → lagging correction → institutional absorption of verification costs.
- Detection and watermarking are brittle in open ecosystems (compression, re-encoding, adversarial perturbation, unauthenticated generation), producing a persistent provenance gap.
- Trust erosion yields dual failure modes — credulity (believing fakes) and cynicism (dismissing truths) — both exploitable via plausible deniability.
- Harms are unevenly distributed, burdening marginalized communities and those without access to authenticated channels.
Connections
This paper’s epistemic-security framing complements empirical work on the actual persuasive limits and reach of AI-generated content, such as Hackenburg2025-dj and DeVerna2025-dl, as well as psychological inoculation approaches to resilience discussed in van-der-Linden2026-jt. Its emphasis on trust erosion and the discounting of digital evidence resonates with scholarship on the “liar’s dividend” and information disorder more broadly, including Hameleers2026-mc and work on data voids and manipulation by Marwick2025-ov. The provenance and platform-governance dimensions link it to studies of detection and moderation at scale like Pierri2025-hm.
Podcast
A research-radio episode discusses this paper: 🎧 MP3 · Spotify · Apple Podcasts