Week 3 Generators Slides

Practical Work in AI, WS26

Hands-on with Deepfake Generators:
Face Swap and Voice Cloning

Week №03
Oleksandr Shytyk | K12429859
Supervisor: Shah Nawaz
Institute of Computational Perception

1. Task and Selected Generators

Task from the last meeting

Pick 2–3 generators from SpoofCeleb or ILLUSION.
Understand how their pipelines work.
Include face manipulation: audio and visual.
Measure the quality of the generated content.

Setup constraints

Apple M5, 16 GB, no CUDA GPU.
One isolated conda environment per generator.
Pinned versions; no releases from the last two weeks.

Selected generators

Face swap + voice clone reproduce ILLUSION's four classes: RA-RV, RA-FV, FA-RV, FA-FV

Generator	Modality	Source paper	Why this one
Roop (inswapper_128) via FaceFusion	Face swap, video	ILLUSION, Set A	One-shot, no training, runs without CUDA
XTTS-v2	Voice-cloning TTS	ILLUSION, Set A; MLAAD	Used in two of the three papers; EN and DE
Chatterbox V3 (TTS + VC)	TTS and voice conversion	Bilingual SpoofCeleb; MLAAD	Your TTS system; one codebase covers TTS and VC

Sources: ILLUSION, App. B.1; Bilingual SpoofCeleb, §3; MLAAD v11, Table IV.

2. How the Generators Work

Face swapILLUSION · Set A

ArcFace turns one source photo into a 512-number identity vector.
Each target frame: face detected, aligned, cropped to 128×128.
The generator redraws the crop with the source identity.
Pasted back with a soft mask, frame by frame.

XTTS-v2ILLUSION · MLAAD

A short reference recording becomes speaker latents.
A GPT-2-style decoder predicts audio tokens one by one.
HiFi-GAN turns the tokens into a 24 kHz waveform.
Sampling is random; the model decides when to stop.

Chatterbox V3SpoofCeleb · MLAAD

T3 language model: text → speech tokens (content, prosody).
S3Gen flow matching renders them in the reference timbre.
A vocoder makes the waveform; a Perth watermark is added.
VC skips T3 and re-voices real source speech.

Takeaway:Each artifact follows from the pipeline: 128 px crop → blur and seams; no temporal model → overly stable identity; autoregressive decoding → missing stop token.

Sources: ILLUSION, App. B.1; XTTS (arXiv 2406.04904); Chatterbox repository and model card.

3. Face Manipulation

Four pairs: source, target and swap — Rows: 4 pairs · columns: source | target | swap

Real Actor 04, original

Fake Actor 02 → actor 04

Observed artifacts

Pose, expression and hair come from the target.
Face is soft: drawn at 128×128, then upscaled.
Pair 02→04: dark mouth, seam at the cheek.

Takeaway:Identity transfers (ArcFace 0.90 vs 0.92 for real); texture does not (face BRISQUE 33.8 → 43.7).

Data: RAVDESS actors 01–04 (CC BY-NC-SA); same-sex pairs, one source frame per pair (ILLUSION protocol).

4. Voice Cloning

Same sentence, same speaker: actor 01

EN: “Kids are talking by the door.” · DE: “Die Kinder unterhalten sich an der Tür.”

Sample	Condition	Length
Real recording	Held-out take of actor 01, never used as a reference	1.6 s
XTTS-v2	Cloned from ~15 s of actor 01's other recordings	2.0 s
Chatterbox TTS	Cloned from the same reference	1.7 s
Chatterbox VC	Actor 03's real speech re-voiced as actor 01	1.8 s
Chatterbox TTS, DE	Cross-lingual: actor 01 never spoke German	2.8 s
XTTS-v2, DE	Failure: keeps talking after the sentence (actor 03)	9.5 s

Note:Volume normalised for this call; raw XTTS output is 10–20× louder than the real recordings.

Data: RAVDESS, Thorsten-Voice (CC0). Reference: six S2 clips per speaker; S1 held out as ground truth.

5. Audio-Visual Composition

ILLUSION Set A protocol for one target: actor 03

32 clips in total: RA-RV 4 · RA-FV 4 · FA-RV 12 · FA-FV 12

RA RV
Original

RA FV
Actor 01's face

FA RV
Actor 01's cloned voice

FA FV
Actor 01's face and voice

Controls

Identical encoding: H.264 CRF 23, AAC 48 kHz.
Fake audio loudness-matched and onset-aligned.

Limitation

No lip-sync, as in ILLUSION.
In FA-* clips, mouth and audio do not match.

Chatterbox TTS shown as the fake audio; XTTS and Chatterbox VC versions exist for every target.

6. Quality Metrics

Audio: mean per file, real second take as baseline

Generator	WER ↓	Speaker sim. ↑	MCD, dB ↓	Sec / char	FAD ↓ (real-vs-real)
Real (2nd take)	0.009	0.968	1.91	0.058	–
XTTS-v2	0.019	0.924	3.29	0.101	2.33 (0.26)
Chatterbox TTS	0.010	0.941	2.78	0.058	1.59 (0.40)
Chatterbox VC	0.000	0.917	2.87	0.053	Sample too small

Face swap: 137 face crops per side

Clips	BRISQUE face ↓	ID sim. to source	ID stability over time	FID face ↓ (real-vs-real)
Real (RA-RV)	33.8	0.922 (same person)	0.977	–
Swap (RA-FV)	43.7	0.900	0.987	36.0 (5.3)

Takeaway:Chatterbox is more realistic than XTTS, and no fake matches a second real recording.

From the papers: BRISQUE, FAD (ILLUSION); Whisper WER (Bilingual SpoofCeleb). Added: speaker similarity, MCD, sec/char, FID.

7. Lessons for Detection

What the metrics show

Whisper misses XTTS overruns (WER = 0).
ArcFace is fooled; texture metrics are not.
German clones lose 0.09–0.12 similarity.

Shortcuts a detector could learn

Loudness: XTTS is 10–20× louder.
Watermark: every Chatterbox file carries Perth.
Codec: differs unless re-encoded.
Lip-sync: audio does not match the mouth.

Link to the three unseen conditions

Condition	Observation
Unseen language	Cross-lingual fakes keep quality but lose speaker similarity
Unseen generator	Each generator fails differently: XTTS overruns, Chatterbox is unusually clean
Unseen identity	Not covered yet: only five speakers

Takeaway:The detection signal sits in texture and spectral detail, not in content or identity.

Shortcuts were controlled in this set by identical encoding and loudness matching.