Most write-ups of an AI workflow build list the clever parts. This one is a real build for a Melbourne technology business: a tool that listens to a live event and maps the conversation on the big screen while it happens. Here is what the AI does, where a person still signs off, and what we deliberately kept away from the model.
What this AI workflow build replaced
The brief was a tool for in-person events: panels, fireside chats, internal sessions.
The manual version is familiar. An audience Q&A tool only knows what people type into it. The record of the night is whatever someone wrote down, and the write-up gets assembled afterwards. The first attendee recap on this project was exactly that, curated by hand.
The goal was a room that can see what it is talking about while it is talking, and an organiser who leaves with a record they can hand out. From the kickoff call to the first real room took under three weeks.
How the AI workflow runs, step by step
Building an AI workflow like this is mostly plumbing. The AI sits at a handful of points and rules do the rest, which is the pattern we describe in our guide to AI workflow automation in Australia.
- Listen. A laptop records the room in chunks of about 20 seconds. Each chunk is transcribed, then discarded. No audio is stored.
- Extract. Every 30 seconds a language model reads the new transcript and returns the topics it heard, in a strict structured format.
- Match. Each new topic is compared with the topics already on the board using embeddings, which score how close two phrases are in meaning. At 0.80 similarity or above it counts as another mention of an existing topic. At 0.55 or above it is linked to that topic as a satellite. Below that it becomes a new bubble. The board holds ten top-level topics at most.
- Size. How big a bubble gets is not an AI decision. It is a formula: three points per mention, two per vote, one per comment.
- Join. The audience scans a QR code to vote, comment and propose topics from their phones. A topic from the floor that keeps growing before the stage touches it is the signal to the moderator.
- Write up. Afterwards the whole transcript is read into a one-page summary and a take-home document.
Where a human stays in the loop
Every AI implementation needs an answer to one question: who catches it when the model is wrong? This build has three.
- The host corrects the board. The model only heard the audio. The host was in the room. So the host can rename a topic the model misheard, or hide one that was never said.
- The organiser moderates. Audience posts pass an automated check, and the organiser can hide anything from the screen.
- A person reads what leaves the room. For now, someone on the product team reads each AI-written take-home document before the organiser can hand it out. The organiser keeps their own board, transcript and summary throughout. Only the documents going to other people wait.
If the AI pipeline stops mid-event, the audience voting layer keeps working without it. That was in the first spec.
What this AI workflow build deliberately does not automate
Good AI workflow design is as much about what the model is not allowed to do. Five things we kept away from it:
- Who should meet whom. Guests can opt in to introductions. The matching is arithmetic. The model only writes the sentence explaining a match the arithmetic already made.
- Consent. A guest is introduced only if they tick the box themselves. An organiser cannot tick it for them, and it was never switched on in bulk for earlier registrations.
- Naming people. The take-home document names no attendee by default.
- Trusting what the audience types. Audience text is fenced off in every prompt, so the model treats a comment as content, not as instructions.
- Choosing the model. An early version let the organiser pick a model tier. The default was the cheapest and nobody ever changed it, so two better options sat unused. We removed the setting. Nobody should be weighing up language models thirty seconds before their event.
What building an AI workflow taught us
Three things only showed up once we measured real sessions. This is the Adapt and Keep Watch end of the ATTACK Method.
The summary was reading 18% of the night
The original summariser read the first 24,000 characters of the transcript. Measured against three real sessions, that covered 18%, 36% and 48% of the transcript. Every summary was a summary of the opening. We rebuilt it to read the whole transcript in windows, four at a time. A transcript of roughly 136,000 characters now reads in 19 seconds, and on one event the new summary surfaced a theme that existed only in the 64% the old one never read.
Quiet looked the same as broken
On a lunch-table test in a noisy restaurant, transcription worked the whole way through: 62 chunks in 22 minutes. But the board showed nothing from the conversation for the first eight minutes. The model was right: the early talk was chatter about the tool itself, with no topics in it. The defect was that healthy-and-quiet looked identical to broken. The big screen now carries a small readout that says whether the tool is listening and how long since it last heard a topic. It is a warning, not an error, on purpose: an error for correct behaviour trains people to ignore the banner.
A token limit quietly switched a feature off
The model we use spends hidden reasoning tokens out of the same limit as its answer. One call was capped so tightly that it spent the whole limit thinking and returned nothing, so automatic theme grouping did nothing for about two weeks. Another cap cut off one transcript window in three. Both now have headroom, and a test checks every limit.
We also measured before tuning. Perfect de-duplication of topics was worth about 8%, and a looser threshold fused topics that were genuinely different. The thresholds stayed.
That is the shape of our AI implementation work: a small first version in front of real users, then measurement.
Try this today
Take one AI idea you have for your business and draw two columns. In the left, list the steps where a wrong answer is cheap and easy to spot, like a misheard topic on a screen. In the right, list the steps where a wrong answer reaches a customer or makes a decision about a person. Let the AI own the left column. Put a named person or a plain rule against every line in the right. If you cannot yet, that is the work to do before anyone builds anything.