Jia Lin

Mosaic app screens: camera, a playing soundscape for Starry Night Over the Rhône, and sound layers

The mission

Making visual art explorable for blind and low-vision audiences through sound.

Alt text reduces a painting to a single sentence. The experience ends there.

As Mosaic’s founding product designer and UX researcher, I led need-finding with blind and low-vision artists, designed the product and its sound design system from zero, and ran our co-design and museum studies.

We designed AI-generated soundscapes that express an artwork’s mood as well as its content, then tested them with visitors in a live exhibition.

3blind and low-vision artists shaped the concept in need-finding interviews
2rounds of testing: 1-on-1 co-design, then a group study in a museum
0 → 1product and sound design system, designed from zero
2industry partners in discussions — Microsoft Seeing AI and the Meta smart glasses team

01 · Empathize

Alt text tells you what’s in a painting, then moves on.

We interviewed three blind and low-vision artists and musicians about how they experience visual art today, and where current tools fall short.

BlindChildren’s author & mosaic artist
Low visionEngineer & musician
Low visionFlutist

What we heard

Loss of instant perception: sighted visitors take in a whole painting at once.
Current AI tools describe, but lack explanatory depth.
Traditional alt text feels “final,” not exploratory.

02 · Define

Blind and low-vision visitors lack real-time, explorable, multi-sensory ways to interpret visual art.

How might we

…recreate the immediacy of visual perception with immersive soundscapes, so blind and low-vision visitors can explore art on their own?

03 · Ideate

Mapping the space: how literal, and how much control?

We plotted sound concepts on two axes: literal vs. interpretive audio, and how much control the listener has over what they hear.

Interpretive / emotionalLiteral / factualLow controlHigh control

Layered sound
Mood-based soundscapes
Object-based accuracy
Single-track literal audio

04 · Prototype

Snap a photo, hear the painting, ask it questions.

The first iteration turned a phone photo of an artwork into a layered soundscape, with an LLM-powered voice assistant for follow-up questions.

System architecture: camera, composing the soundscape, and the soundscape player

How it works: snap a photo of the artwork, hear it as a soundscape, ask it anything

05 · Co-design

When sound and description disagreed, trust broke instantly.

In 1-on-1 co-design sessions, participants explored two artworks: alt text first, then the soundscape. They adjusted volumes, toggled layers, and talked through reactions and moments of confusion.

“But the alt text didn’t say there was an ocean…”

“The seagulls are too loud — it breaks the mood.”

Key insight

What users kept asking for: audio that expresses the art, sound that belongs to the piece.

06 · Iterate

A sound design system that stays faithful to the description.

We rebuilt the pipeline so every sound traces back to something in the description, and softened harsh or overly literal effects that broke immersion.

Sound design system: composing, the layer table, and the mix

07 · Museum study

Testing in the gallery: the “Buttons On!” exhibition at the Museum of Craft and Design.

Participants moved freely between 5–6 artworks at their own pace. For each piece they heard the alt text, then the soundscape, and described what they imagined, comparing interpretations as a group while we watched where immersion held or broke.

From the day

A visitor listening to a button sculpture in the gallery.

Button-covered garments on display — “Buttons On!” at the Museum of Craft and Design.

What we learned

Sounds sparked curiosity: “I’d ask, ‘tell me about the water in the scene.’”
Freedom to interpret mattered: “You have the ability to decide whether you like the art or not… that’s what a sighted person does.”
Museums want their own audio: “They should be able to upload sound bites or exhibit-specific content.”
People noticed stylistic choices: “I’d be curious what sound representations of Cubist buildings would be.”

08 · What’s next

From a phone app to the gallery itself.

Exhibit modesMuseums upload custom sounds, context, and artist notes for curated shows.
Style-aligned soundAudio that captures Impressionism, Cubism, or abstraction: the artistic style and the mood.
On-site wearablesA lightweight version for museum headsets or smart glasses, hands-free.