An AI audio guide system for museums

For most museums, going digital has always meant a brutal trade-off. Hardware audio guides are expensive to rent, service, and restock, and they ship in a handful of languages with no personalization. Actually digitizing a collection is worse: writing descriptions and recording voice-overs across every language a museum's visitors might speak is months of curator and voice-actor work that smaller institutions simply can't staff or fund. So foreign visitors stand in front of labels they can't read, and the museum walks away with no idea how anyone actually moved through the exhibition.

Codigee's insight was to knock out both bottlenecks at once: let AI do the production work that used to take months, and let each visitor's own phone be the guide instead of a rental device. A visitor snaps a photo of any artwork, computer vision recognizes it in roughly 100 milliseconds, and AI-generated narration plays in their language — no QR codes, no hardware, no per-language recording sessions. To prove the loop before scaling it, we shipped the Ambassador app first as the MVP: the tool a museum's own staff use to photograph works and trigger AI description and narration generation, validating the entire pipeline against real objects fast.

What we delivered is one connected ecosystem, not a single app. A Flutter visitor app (iOS and Android, offline-capable for thick museum walls, UI in 28 languages) and the Ambassador app build from a single codebase as two flavors, while a self-service Next.js admin panel lets curators add a work, generate its description and narration in one click, translate, publish, and read analytics like scans, most-viewed works, and visit heatmaps. Underneath sits the brain: a Bun and Elysia backend on PostgreSQL with multi-tenant isolation per museum and Redis/BullMQ queues orchestrating the AI work. That brain feeds two custom GPU services we built from scratch — a fast artwork-recognition model we trained and tuned ourselves, and a triple-engine, Whisper-verified text-to-speech system — taking the platform from concept to a working product in about three months.

Before a single visitor scans a painting, someone has to populate the collection — and that someone is the museum's ambassador. The Ambassador app puts the entire content-creation loop in one hand: walk the gallery, photograph each work, trigger AI generation of its description and multilingual narration, then review and approve the result before it goes live. There's no curator writing catalog essays for months and no voice actor booking studio time — the ambassador captures the artwork, the backend's AI job queues do the heavy lifting, and a draft description plus studio-grade audio come back for a quick human check. It collapses weeks of digitization labor into a photograph-and-approve workflow.

We shipped the Ambassador app as the MVP on purpose, because it sits exactly where the technical risk lives. A polished consumer app is worthless if the AI pipeline behind it can't produce content a museum trusts — so we proved that loop end-to-end first: photo in, our trained recognition model finds the artwork, an LLM writes the description, triple-engine TTS renders the narration, Whisper verifies it, human approval out. Putting that pipeline in a real ambassador's hands let us validate Museo's most uncertain mechanics against an actual collection before investing in B2C subscriptions, geolocation, and offline polish. It's de-risking by sequencing: prove the engine, then build the car around it.

The MVP was never throwaway code. The Ambassador app is a second flavor of the exact same Flutter codebase that becomes the visitor app — one project, two build flavors, with shared auth, networking, and architecture. That means everything we built to validate the content loop carried straight into the production user experience instead of being rebuilt, and it keeps two distinct apps in sync from a single source of truth. For the client, MVP-first cost almost nothing in wasted effort and bought a fully de-risked AI pipeline before the consumer app even existed.

Digitizing a collection has always meant months of curator writing and voice-actor sessions, repeated language by language — work most museums can't staff, never mind the IT team needed to run the software around it. The admin panel collapses that into a self-service workflow a single curator can drive from a browser. Adding a work, generating its description and narration, translating it, and publishing it to every visitor's phone are all point-and-click operations. The museum holds the keys: we built it specifically so they operate it without us, and the interface itself ships in 20 languages because the people running the institution aren't always English speakers either.

Each artwork begins with a photo and one click that fires the full AI content pipeline — an LLM drafts the description, a web-scraper enriches the metadata, auto-translation fans it out across languages, and the triple-engine TTS system returns studio-grade narration, every clip auto-verified by a Whisper speech-to-text round-trip before it can ship. Curators stay firmly in control: they edit copy, retune individual translations, and approve audio inside the panel, then publish when it's right. A versioning system tracks which descriptions and recordings have gone stale after a prompt change and flags them for regeneration — version control for museum content, so a growing collection never silently drifts out of date.

For the first time, the museum can see how people actually move through the exhibition. The panel surfaces scan counts, most-viewed works, and visit heatmaps that reveal which rooms pull crowds and which get skipped — the behavioral data hardware audio guides never captured. A collection-completeness view shows exactly which works still lack a description or audio, while a translation matrix maps coverage language by language so gaps are obvious at a glance rather than discovered by a confused visitor. Underneath, the backend's multi-tenant isolation keeps each museum's content and analytics strictly its own, so the same self-service panel scales cleanly from a single gallery to a national institution.

Museo has a lot of surfaces that all need the same source of truth: the visitor app scanning artwork, the ambassador app populating content, the self-service admin panel, and the recognition, description, and text-to-speech pipelines running on GPU servers. We built the backend on Bun and Elysia precisely because it sits in the hot path of all of them — fast enough to stay out of the way of a ~100ms recognition response, and strict enough to keep a fast-moving multi-client product honest. Drizzle ORM models the data over PostgreSQL 18, and the API is fully type-safe end to end via Eden Treaty and OpenAPI, so the Flutter clients and the Next.js panel consume the exact same contracts the server defines. When a field changes on the backend, the clients know at compile time — not in a bug report from production.

Because Museo is a SaaS platform, the hardest invariant is that no museum can ever see another museum's data. Multi-tenant isolation is enforced at the data layer: every query is scoped to its museum, so a scan, an analytics heatmap, or a content edit only ever resolves inside the right tenant's boundary. On top of that, the backend owns the cross-cutting concerns the rest of the system shouldn't reinvent — Better Auth handles accounts and sessions across Google, Apple, and OTP with bearer tokens for the apps; PostGIS powers the geolocation queries behind the visitor app's "museums nearby"; and S3 holds every photo and audio clip the platform generates. One place to authenticate, one place to locate, one place to store.

The AI is the part that can't be synchronous, and the backend is what makes that invisible. Recognizing an artwork, writing a description, translating it, and rendering studio-grade narration on GPU workers are slow, bursty, failure-prone jobs — so the backend runs them through Redis-backed BullMQ queues instead of blocking a request. It enqueues work, brokers it to the recognition, description, and TTS pipelines, tracks state, retries, and hands results back when they're ready — exactly what lets a curator click "generate" once and walk away. That same queue layer drives the content versioning system: when a prompt changes and descriptions or audio go stale, the backend knows what to regenerate and schedules it. The apps and the AI services never talk to each other directly — they talk to the brain, and the brain keeps them in sync.

No single model speaks every language well, so we don't pretend one does. Our text-to-speech runs as a triple-engine system with smart per-language routing: Chatterbox Turbo handles English at the lowest latency, Chatterbox Multilingual covers 20+ European languages, and Qwen3-TTS takes Chinese, Japanese, and Korean. The router picks the right engine per language automatically, so a curator clicks "generate" once and the backend chooses the best voice path under the hood — no model juggling, no per-language setup on the museum's side.

Raw model output isn't shippable, so we engineered the pipeline around its failure modes. Long passages are chunked before synthesis to stop the models from hallucinating or drifting on extended text, and every chunk is cleaned with audio post-processing — a noise gate and silence trimming — so the narration sounds deliberate, not machine-stitched. Then comes the part most TTS pipelines skip: verification. Every clip is fed back through Whisper in a speech-to-text round-trip, the transcription is compared against the original script, and any clip that drifts is flagged for a human to listen before it ever reaches a visitor. Generation is automated; correctness is never assumed.

All of this is our own code on serverless GPU — PyTorch models orchestrated by the backend's job queues, not a third-party TTS API we resell. The result is studio-grade narration in 20+ languages with no human voice actors, no recording booths, and no months-long localization project. A museum can populate an entire multilingual collection in the time it once took to schedule a single recording session — and every minute of audio that ships has already been checked against its script.

We solved it the way human recognition actually works: by understanding what is in the image, not by reading a code off the wall. Every artwork in a museum's collection is learned by a vision model we trained and tuned on that museum's own pieces. When a visitor takes a photo, the same model recognizes the work on the fly — it answers "which artwork is this?" from the image itself, so a bad angle, a glare, or a partly cut-off frame does not break it the way a barcode or QR code would.

Doing it at speed is exactly what the model is tuned for: it returns the right artwork in roughly 100ms, scoped strictly to that museum so two galleries with similar pieces never bleed into each other. Because it recognizes the artwork itself rather than a rigid code, it holds up in the real world: a canvas shot from the side, a sculpture under warm spotlights, a frame partly cut off, a collection of thousands. The same robustness that makes it forgiving for visitors is what makes it deployable — no curator has to print, stick, and maintain a tag on every wall.

The last challenge is making real-time computer vision feel instant in a live gallery. The recognition model runs on GPU and is kept ready ahead of demand, so it responds the moment a visitor lifts their phone — no cold-start lag, no spinner in front of the painting. The result is studio-quality AI recognition that feels immediate to the visitor and runs efficiently enough for any institution, from a single gallery to a national museum, to rely on every day.