Running our own inference stack as a small company

Renting a hosted model would have been faster and, at our volume, probably cheaper. We chose the harder path because the alternative meant handing every conversation to a company whose business depends on knowing things about people.

The decision

The default for a small team is obvious: call somebody else’s API, ship in a week, worry about it later. The problem is that "later" never comes with leverage. Once your product depends on a provider, their retention policy is your retention policy, their terms of service are your privacy promise, and any change they make lands on your users without your consent.

So we operate the stack ourselves. We deploy the model, tune the serving layer, own the hardware budget, and answer for every byte that touches it. That is a real cost, and it shows up in three places: capacity, latency and the number of hours two people can spend on infrastructure instead of product.

Capacity planning without a hyperscaler

Hosted providers hide the awkward truth that inference capacity is lumpy. A model either fits on the accelerator you have or it does not, and the gap between "comfortable" and "queueing" is much narrower than a dashboard suggests. Our planning is deliberately conservative:

Cost per answer

Owning hardware turns a variable cost into a fixed one, which is uncomfortable early and pleasant later. Below a certain volume we are paying for silicon that idles. Above it, every additional answer is nearly free, and our margin stops being a function of somebody else’s price list.

That shape is why the pricing is what it is. Free tiers are limited not to upsell you but because a free request costs us real electricity on a machine we bought. We would rather be transparent about that than pretend it is magic.

What it cost us

Debuggability, mostly. A managed provider gives you dashboards, traces and someone to page. We gave that up, and we also chose not to keep the transcripts that would make debugging easiest. Diagnosing a bad answer with a thirty-day excerpt and a latency number is slower than reading the full conversation — and that is the trade we accept in exchange for having nothing worth stealing.

What is next

Better batching for long-running Flows, so a background job cannot starve a person waiting on a chat turn, and a clearer capacity signal in the app. We will publish the numbers once they are stable enough to be worth publishing.

Approve-first actions, and undo after

Every real-world action Budulai takes is shown as a plain-language plan before it runs. What we learned is that people read the first step, skim the second, and trust the rest — which changes how you have to design the whole thing.

The premise

An assistant that can send email, move files and drive a Mac is only safe if the person stays in the loop. The obvious answer is a confirmation dialog. The obvious answer is also useless, because a dialog that appears every time is a dialog nobody reads.

So we ask a narrower question: what is the smallest number of moments where a human decision genuinely changes the outcome, and how do we make those moments impossible to skim past?

What people actually read

Watching real approvals, the pattern was consistent. The first line gets read carefully. The second gets read. Anything past step three is scanned for a familiar shape — an app name, a file name — and approved on vibes. Length is not the problem; position is.

When to ask twice

Some actions cannot be walked back: an email that has left, a file that is gone, a message posted somewhere public. For those, one approval is not enough, and a second identical dialog is just friction. The second prompt has to add information — who exactly is receiving this, what exactly is attached — so that confirming requires reading something new.

Destructive shell commands are handled differently again: they are refused outright by a safety filter before they ever reach an approval screen, even for people who have set that capability to "always".

Undo is a feature, not an apology

Approval is a prediction; undo is a correction. Anything reversible ships with the reversal built in — a draft returns to drafts, a moved file goes back, a scheduled Flow can be unscheduled mid-run. Making undo cheap is what lets us keep the approval flow short: you do not need to interrogate a plan when getting it wrong costs one tap.

Open question

We still do not know the right default for people in their first week. Ask about everything and the app feels bureaucratic; ask about less and you are teaching someone to trust a system they have not yet seen work. Right now we start cautious and let people loosen it. We are not certain that is correct.

Seven variables, one number

Whether someone keeps their HRT depends on supply, prescription expiry, lab timing, pharmacy delays, insurance approval, paperwork and the calendar. Hormone Helper collapses all seven into a single figure — days of runway — and then shows its work.

Why one number

Ask someone on long-term hormone therapy what they are worried about and the answer is never "my adherence rate". It is "am I going to run out". Every existing tracker answers a different question — what did you take, did you take it on time — which is a record of the past when what people need is a forecast.

So the app opens with a forecast: how many days of medicine you have, given everything currently known. One number, in plain language, on the first screen.

The seven inputs

Against the black box

A single number is only trustworthy if you can see where it came from. Tap the figure and the app shows the chain: which input is currently binding, what would extend it, and how many days each fix would buy. A prediction people cannot audit is a prediction they will ignore the first time it disagrees with them.

Calm, not alarming

Low supply is an amber flag with a suggested action, never a red alert. The people using this app are frequently already anxious about access, and an interface that panics on their behalf makes the situation worse rather than more manageable. The best empty state here is nothing needing you at all.

Still open

The math needs clinical review before real regimens depend on it, and how much of the chain can be genuinely automated depends on pharmacy and lab integrations we do not have yet. Until then, the app drafts the messages and you send them.

Reading a medication label on the phone

Typing a regimen is where setup dies. Photographing the vial should replace it — but only if the app is honest about which fields it actually read and which ones it guessed.

The setup problem

Every health tracker asks for the same thing up front: medication, dose, concentration, frequency, route. It is five fields of pharmacological detail entered by someone who just wants the app to work, and it is where most people abandon the process entirely.

The label already has all of it. So the first screen is a camera, and everything else is a fallback.

Three kinds of label

Read on the phone, then deleted

Parsing happens on the device. The photo is never uploaded, and it is deleted once the fields are populated — a medication label is an identity document, a diagnosis and a pharmacy record in one image, and there is no version of storing it that we are comfortable with.

Per-field confidence

The interesting design problem is not accuracy, it is uncertainty. A parser that returns five fields silently is dangerous, because the one it got wrong looks exactly like the four it got right. So every field carries a state — read, matched or check — and the interface pulls the uncertain one to the top rather than burying it in a filled-in form.

In practice concentration is the field that most often needs a human. Handwritten decimals and unfamiliar units are where the parser is least sure, and that is precisely the value that determines every dose calculation downstream.

Reconciliation

Nobody adds a medication on day one. They add it on day forty-one, having taken a dose that morning. Assume day one and every number downstream is wrong — so before saving, the app asks how long you have been on it and what you took today. Two questions, and the count is correct from the start.

Still open

Handwriting remains the weak point, and we have not yet decided how far to push on-device recognition versus simply asking. Asking is unglamorous and always correct; we are inclined to keep asking until the parser earns the benefit of the doubt.