Ferrara, E. (2026). The Generative AI Paradox: GenAI and the erosion of trust, the corrosion of information verification, and the demise of truth. arXiv [cs.CY].
Summary
Ferrara argues that the dominant framing of generative AI risk — isolated deepfakes and discrete synthetic artifacts — badly understates what is actually at stake. The deeper threat is the emergence of synthetic realities: coherent, interactive, personalized information environments in which content, identity, and social interaction are jointly fabricated at low cost and high throughput. The paper formalizes this as a four-layer stack (content, identity, interaction, institutions), expands existing GenAI harm taxonomies to foreground epistemic and institutional integrity, catalogs 2023–2025 incidents that exemplify the pattern, and proposes a layered mitigation strategy plus a research agenda for “epistemic security.” It culminates in the Generative AI Paradox: as synthetic media saturates the information environment, rational actors may discount all digital evidence, raising the social cost of truth itself and empowering strategic denial.
Key Contributions
- A four-layer formalization of synthetic reality — content, identity, interaction, institutions — mapping attack surfaces to defensive levers.
- An expanded GenAI harms taxonomy that adds epistemic/institutional integrity as a cross-cutting category alongside personal, financial, informational, and socio-technical harms.
- Seven qualitative shifts distinguishing GenAI from prior deception tech: cost collapse, scale/throughput, customization, micro-segmentation, automated social engineering, provenance gaps, and trust erosion.
- A mechanism-focused case bank of 2023–2025 incidents linking documented harms to specific stack layers.
- A defense-in-depth mitigation framework spanning provenance, platform governance, institutional process redesign, public resilience, and policy.
- A research agenda for epistemic security, including candidate metrics (authenticity coverage, correction latency, manipulation susceptibility, verification load, attribution stability) and interactive-manipulation benchmarks.
- Formulation of the Generative AI Paradox as a testable systemic-discounting thesis.
Methods
The paper is conceptual and synthesis-driven rather than empirical. Ferrara develops a theoretical framework (the four-layer synthetic-reality stack), conducts taxonomic expansion of harm categories, performs a mechanism analysis of how GenAI differs qualitatively from earlier deception technologies, and constructs a curated case bank of 2023–2025 incidents selected for documentation quality, mechanism diversity, and coverage across stack layers. Mitigation strategies and research priorities are then mapped back onto the framework.
Findings
- Five recurring case archetypes exemplify synthetic-reality harms: (A) high-conviction impersonation fraud (e.g., the Arup ~$25M deepfake video-conference scam), (B) election-adjacent synthetic outreach (e.g., AI-generated Biden robocalls), (C) non-consensual synthetic sexual imagery (e.g., the Taylor Swift incident), (D) fabricated everyday documentation (AI receipts, invoices, expense fraud), and (E) compromised generative pipelines (malicious model uploads, backdoors, data poisoning).
- A common operational pattern recurs across cases: cheap high-conviction artifact production, insertion at workflow choke points, scale-driven exposure, lagging correction, and institutions absorbing the verification externalities.
- Detection and watermarking are structurally brittle in open ecosystems due to compression, re-encoding, adversarial perturbations, and unauthenticated generation — producing a durable provenance gap.
- Trust erosion produces dual failure modes — credulity (believing fakes) and cynicism (dismissing truths) — both exploitable for plausible deniability.
- Harms are unevenly distributed: marginalized communities and those without authenticated channels bear disproportionate verification burdens.
Connections
This paper sits upstream of much of the empirical work on GenAI-enabled influence and persuasion, providing a framework that organizes findings on AI-driven persuasion at scale Hackenburg2025-dj and on generative-AI fingerprints in coordinated influence operations Rothut2026-or. Its emphasis on trust erosion and the dual credulity/cynicism failure mode connects to work on synthetic-media skepticism and the “liar’s dividend” Hameleers2026-mc, while its claim that GenAI is qualitatively, not just quantitatively, different speaks directly to debates over whether AI meaningfully reshapes the misinformation landscape DeVerna2025-dl. The epistemic-security agenda also resonates with broader assessments of the misinformation problem’s scope and structure Budak2024-ef.
Podcast
A research-radio episode discusses this paper: 🎧 MP3 · Spotify · Apple Podcasts