- Building
Nightshift is the night shift for an AI phone agent. Every night it reads the calls, sends the failures to a Claude Code agent on HarnessRouter that traces each one to the instruction that caused it and makes the smallest fix, runs a safety check (the AI and call-recording disclosure can never change), re-tests on fresh calls with the odds it's luck, and waits for a person to approve. I built it on Godfrey, my after-hours receptionist for plumbers (demo shop Sunset Plumbing, made up). Solo, for Berkeley.
Day 1 (Mon): the whole loop in Python. First night: the agent read the 5 worst calls, found 3 causes, changed 4 lines in under a minute. Night 2 got rejected: one case (a cut hand) dropped 2/4 to 0/4, so Nightshift now refuses any fix where one case drops by 2 or more, whatever the total says.
Day 2 (Tue): full check of night 1's fix: 140/178 to 153/178 (79% to 86%), about 5% odds it's luck. Added "any agent" mode: give it any bot's instructions and transcripts and the agent does the triage too. Four friends called the real demo line to break it; their calls became permanent tests.
Day 3 (Wed): Nightshift blocked its own fix. The fix from the friends' calls went 31/44 to 24/44 (a burst-pipe call that always passed fell to 1 in 4), so it never reached the phone. Added regression memory: every later proof re-tests the call that broke the last fix.
Day 4 (Thu): next fix, 24 to 26 of 48, odds it's luck about 42%. Not approved. The burst-pipe call held 4/4. Made Demo Day.
- Blocked
- 402 no_key on the first task. HarnessRouter's free plan covers agent time, not the model. Fixed with Bring Your Own Key.
- A number I almost posted. First re-test said 27/33 to 30/33, but "before" reused the calls that had failed, so it was rigged to look good. Switched to fresh calls only.
- 81 of 178. The wifi dropped mid-run and 83 calls crashed before the AI spoke. Nightshift scored them as failures. Now crashes are set aside and only those re-run.
- Noise. One call per case lies: the burst-pipe case scored 1/4 one day and 3/4 the next with no change. Added re-checks that run a case 4 times.
- Privacy. A friend gave his first name as a one-word answer and it slipped past the scrubber. Now it learns names from context, and every upload gets a last check.
- The Mac slept on battery and froze a 2am proof. The run now keeps the Mac awake until it's done.
- Budget: $2/day of simulated calls, capped in code.
- Loved
- The Claude Code harness reads a pile of call transcripts and comes back with causes, not just edits. On night 1 it found 3 causes and changed 4 lines in under a minute.
- It pushes back. It flagged a Mandarin-speaker call as conflicting with an existing rule instead of forcing a fix, and on real calls it said which automatic flags were wrong and why.
- Every fix traces to a harness run (chrn_2714f128e6d744ad8c8fe0fd4844c88e), so I can show exactly which run wrote which line.
- Cheap: an "any agent" review is about 9 cents.
- Want
- Run Nightshift on two other voice-agent teams' real calls this month, not just mine.
- A simulator for other people's agents, so their fixes can be proven, not just proposed.
- From HarnessRouter: a clearer error when there's no model key (402 no_key cost me time on day 1), and a run link I can paste into a report so anyone can check the evidence.
Live: https://nightshift-bighack.vercel.app · Code: https://github.com/lucasmasen/nightshift
Loading