- Entry date
- 10 May 2023
- Category
- Models
- Access
- 🔓 Public
Audio in, a synchronised talking face out — and a hard early lesson about how unforgiving real-time inference is for generative media.
- audio → face
- end-to-end synthesis pipeline
- pre-trained
- models used, not trained from scratch
Honest evaluation
The end-to-end pipeline worked on pre-trained models as claimed; the harder real-time-latency question was surfaced honestly, not solved, and no claim to have solved it is made.
What would prove or disprove it further
What would prove or disprove it further: the pipeline claim is already closed by the working demo. The latency observation is the part still open and genuinely falsifiable — rerun the same pipeline with today's inference stack and measure the actual audio-to-frame latency against a "feels live" bar, which would show directly whether the lesson this project banked has since been resolved by faster hardware and runtimes, or whether it still holds.
The evidence — full reasoning behind the verdict
Verdict: proven. The claim was to build a working, end-to-end speech-to-face animation pipeline on pre-trained models, and it worked — audio in, synchronised face animation out, a real running research implementation rather than a slide-deck concept.
No novel model or from-scratch training is claimed, so the bar for "proven" here is honestly modest: does audio-in produce synchronised face-out, using existing pre-trained models wired together correctly? It does. The genuinely useful finding sits alongside that proven claim rather than inside it — real-time inference for generative media is hard, a latency gap between "eventually correct" and "live-feeling" that this project surfaced rather than closed.
Not every research thread continues. This one was picked up, worked for a month, and then honestly set down — which is exactly why it belongs in the record.
What it was
A speech-driven facial animation synthesis pipeline: an end-to-end system taking audio as input and producing synchronised face animation as output, built using pre-trained models rather than training a model from scratch.
What we built
ResearchA working pipeline connecting audio processing to a pre-trained facial animation model, in Python, using TensorFlow/PyTorch tooling to run inference and produce synchronised output frames from a speech signal. It worked — a real, running research implementation, not a slide-deck concept.
What we learned — including the honest negative(s)
The real lesson was about latency, not about faces. Real-time inference for generative media turned out to be genuinely difficult — the gap between "the model produces correct output eventually" and "the model produces correct output fast enough to feel live" is large, and closing it is a pipeline-design problem as much as a modelling one. That's a lesson about interaction-shaped systems generally, and it resurfaces later in anything where a human is waiting on the other end: agents, voice, real-time dashboards.
The honest edge is plain: this used pre-trained models, not a model built from scratch, and it never became a product. It's kept as an honest data point on a specific question — how far into generative media did this portfolio actually go? — and the honest answer is: briefly, then it moved on.
Where it went / status
Set down after May 2023, without a follow-on product. The direction of the portfolio shifted toward text and code generation shortly after, and much later toward GPU-native inference work — where the same underlying lesson about the cost of real-time inference reappears, at a completely different scale and with a completely different substrate.
Nothing about the pipeline itself was carried forward — no code, no model weights, no product. What carried forward was the lesson: that generative media has a latency bar most text-generation work doesn't, and that lesson was banked well before it was needed again.
What is still open — kept visible
The honest edges, next to the wins. This is what turns 🔬 into 🟢 — honestly.
- Built on pre-trained models, not a from-scratch model or a shipped product.
- Real-time inference for generative media proved genuinely hard — a latency lesson that recurs in every later interaction-shaped project.
- The portfolio's excursion into generative media was brief; it moved toward text, code, and much later GPU-native inference instead.
Sources
- vocabotics project audit — Animated Face (speech-driven facial animation pipeline), May 2023vocabotics internal project history · as of May 2023