Building a user observation system with an AI design process

Intuit founder Scott Cook watched users work in their homes to build deep empathy for the problems they faced. Follow-Me-Homes are a core part of Intuit’s DNA and origin story. For the customer success team, the people doing that work are the Experts, and their experience directly shapes the customers they serve. Intuit has invested in Immersion Studios, where five to ten Experts travel to the office each week to work on site. The team is invited to observe and meet them, but some Experts make the trip and nobody comes.

A leader sitting in the studio said she wished she could put a GoPro on her head and walk around, so other leaders could see what she was seeing. Then she started talking about her kids watching Twitch. She was imagining a world where the team could hop into the world of our Experts and see their struggles and their successes. How could we remove the barriers to spending time with the people doing the work?

That leader’s experience led to an exploratory project on how to virtualize the studio: adding cameras for recording, and making it possible to visit Experts at home by sending out observation camera kits. Real faces and real feeling, but still a small sample, with questions like how to handle personally identifiable information still open.

Another project was already running to build an internal RUM system, real user monitoring, so technical teams could watch replays of Experts using their core tools. Any session, with PII masked. Powerful and built for scale, but it lacked the human element.

There was no designer on the RUM project and its lead asked for my help. The virtual Follow-Me-Home project was running in my group with a contract designer and a design technologist, and I was asked to guide them.

I pitched combining the two. RUM as the foundation, unlocking every session and handling PII, with the human layer on top: invite the Expert to turn on their webcam, capture the room audio, and fold in the systems that already record calls, chats, and screenshares so playback carries the conversation and a live transcript. The webcam is always an invitation, and Experts can decline, mute, or stop at any point.

To set the foundation for bringing the projects together, I did a pass of the RUM system to align it with our design system. Working directly in the code with Claude and Cursor, I brought the experience into line with Intuit’s patterns. I set experience-oriented rules for the AI agents: use an existing component wherever one exists, and when something genuinely new is needed, compose it out of components that already do. New primitives are a last resort, not a starting point.

A Slack conversation with our engineering lead turned up a new design problem. A change to how we process PII meant loading states were going to show up more frequently. The thread already held the constraints and the reasoning, so I pointed Claude at it and turned it into a Jira epic, a story, and a design exploration task. I assigned it to the contract designer minutes later. AI as a design operations tool tightens the gap between spotting a problem and getting it into a team member’s hands.

The session view is where you actually watch. Everything you need is in view at once: the replay, and the tooling around it like the transcript. This is the main window into the Expert experience. The current design is a v1, and I am looking forward to revisiting it to explore better handling of white space, bringing captions into the video, separating out the technical tooling in the panel, and adding capabilities like comments.

Where design enters the process changes from feature to feature. For the issues detail view, engineering showed me a v1 for rework. I pulled the summary data into a section, added a plain-language description written for a human reader, migrated the chart to our data-vis library, and expanded the information about the technical assets each issue touches.

Working on it in the code with AI let me uncover things that would otherwise have been easy to miss, like the chart generating its bars from the wrong bucketing of data. Getting in front of technical issues like these is not flashy, but it is some of the most meaningful experience improvement we can make.

Issues, the grouping of problems the system identifies, were being created with IDs that hard-coded the type into the ID. I argued for an identifier agnostic of categorization, so we can mature the labels and groupings over time as we learn. How the system is built shapes the experience it can support. I am proud of a relationship with engineering where design gets to influence something as deep in the plumbing as an ID format.

The transcript scrolls with the call. I looked at how YouTube, Zoom, and Spotify’s lyrics solve the same problem, then built it in code against mock data and dialed it in.

Scroll turned out to need decisions I would not historically have known to make until after seeing a first coded version: when focus moves to the next line, when the transcript scrolls up, where to land when you jump to a new call, what to do when someone has scrolled manually but the current line is still in view. These only get design thinking when you are experiencing them. Nobody would have filed a bug about any of it. That is the win from bringing design deeper into development.

I have been running interviews as the alpha opens up, using AI to prepare the sessions, work back through the recordings and transcripts, and cut the moments worth showing other people. The clearest signal so far is appetite: everyone we have shown it to wants access. A useful note came from a session with someone on our accessibility team, who asked for the transcript as captions over the replay rather than beside it. Better for them, better for anyone watching.

This is a current project. We have just started inviting users into a closed alpha, and I am eager to learn from how they use it. I am excited to bring the team closer to the people we serve.