The Generative AI Paradox: GenAI and the erosion of trust, the corrosion of information verification, and the demise of truth

Summary

Ferrara argues that the defining risk of generative AI is not the standalone deepfake but the rise of synthetic realities: coherent, interactive, personalized information environments in which content, identity, and social interaction are jointly fabricated. The paper formalizes synthetic reality as a four-layer stack (content → identity → interaction → institutions), expands the taxonomy of GenAI harms to foreground epistemic and institutional integrity, and identifies seven qualitative shifts that distinguish GenAI from prior deception technologies. Drawing on a case bank of 2023–2025 incidents, it advances the Generative AI Paradox: as synthetic media saturates the information ecosystem, rational actors may discount all digital evidence, imposing an epistemic tax on truth itself and empowering strategic denial.

Key Contributions

• A four-layer synthetic reality stack (content, identity, interaction, institutions) that maps attack surfaces to defensive levers.
• Expanded GenAI harm taxonomy adding epistemic/institutional integrity as a cross-cutting category.
• Seven qualitative shifts distinguishing GenAI from earlier manipulation tools: cost collapse, throughput, customization, micro-segmentation, synthetic interaction, provenance gap, and trust erosion.
• A mechanism-focused case bank of 2023–2025 incidents linking documented harms to stack layers.
• A layered, defense-in-depth mitigation framework spanning provenance, platform governance, institutional process redesign, public resilience, and policy accountability.
• A research agenda for epistemic security with candidate metrics: authenticity coverage, correction latency, manipulation susceptibility, verification load, and attribution stability.
• Formulation of the Generative AI Paradox as a testable thesis about systemic discounting of digital evidence.

Methods

The paper is conceptual and synthetic rather than empirical. Ferrara (1) develops a theoretical framework formalizing synthetic reality as a layered socio-technical stack, (2) performs taxonomic expansion of GenAI harm categories, (3) conducts mechanism analysis identifying seven shifts relative to prior deception technologies, (4) constructs a curated case bank of 2023–2025 incidents selected for documentation quality, mechanism diversity, and stack-layer coverage, and (5) synthesizes mitigation strategies and measurement constructs mapped onto the stack.

Findings

• Five recurring case categories illustrate synthetic reality harms: high-conviction impersonation fraud (Arup ~$25M Hong Kong deepfake call), election-adjacent synthetic outreach (AI Biden robocalls), non-consensual synthetic sexual imagery (Taylor Swift incident), fabricated everyday documentation (AI receipts/invoices), and compromised generative pipelines (malicious model uploads, backdoors, data poisoning).
• A shared operational pattern recurs: cheap high-conviction artifact production → insertion at workflow choke points → scale-driven exposure → lagging correction → institutional absorption of verification costs.
• Detection and watermarking are brittle in open ecosystems due to compression, re-encoding, adversarial perturbations, and unauthenticated generation — producing a persistent provenance gap.
• Trust erosion yields dual failure modes — credulity (believing fakes) and cynicism (dismissing truths) — both exploitable via plausible deniability.
• Harms are unevenly distributed: marginalized communities and those without authenticated channels bear disproportionate verification burdens.

Connections

This paper provides a high-level theoretical scaffold for empirical work on AI-generated political content and persuasion such as Hackenburg2025-dj, DeVerna2025-dl, and Triedman2025-uy, and complements taxonomic and ecosystem-level mapping efforts like Nenno2025-xa and Marwick2025-ov. Its dual-failure-mode argument (credulity and cynicism) speaks directly to “liar’s dividend” and trust-erosion findings explored in Hameleers2026-mc and Schroeder2026-im, while its concerns about synthetic identity and interaction layers connect to work on AI personas and astroturfing such as Tornberg2025-ir and Tornberg2026-lc. Broader debates about whether GenAI represents a categorical shift or continuity with prior misinformation dynamics — engaged by Budak2024-ef and Simon2023-style skeptical accounts — are precisely what Ferrara’s seven-shifts argument is designed to adjudicate.

Podcast

A research-radio episode discusses this paper: Listen