AI Product Engineering · Production Systems · Evaluations
I build, evaluate, deploy, and operate production AI systems.
Working solo and part-time around a full-time job, I shipped five production systems across web, native iOS, and SMS — three model-in-the-loop AI products plus two production automation systems, including an operations agent whose high-risk actions are constrained by deterministic code and a Socratic tutor whose releases are gated by a 61-scenario evaluation harness.
The case studies below are written for inspection, not decoration: workflow decisions, architecture, safety boundaries, evaluation method, production incidents, measurable fixes, and current limitations. Major claims link to a named implementation artifact, a dated evaluation report, or a clearly labeled production snapshot; sanitized examples are linked where public. Production runs on Anthropic’s Claude today, and the same tutor-turn contract and evaluation scenarios run against multiple providers — the underlying skills (prompt and context design, structured outputs, tool-calling architecture, evaluation design, guardrail engineering, cost/latency tradeoffs) are provider-agnostic.
The work, in numbers
Real, rounded where sensitive. Aggregate counts pulled live from production on 2026-07-15 — no demo data.
5
production systems shipped solo, part-time
3
platforms in production: web, native iOS, SMS
7 / 22
buildings / units DoorOps runs on — a real, live portfolio
~350
AI-handled leasing SMS on that real portfolio
~80
autonomous ops-agent runs, propose-first
61
CI-gated eval scenarios gating Lumenor releases
3
model providers orchestrated in one Tiglit feature
0
dedicated engineers or QA on any of it
Two flagship case studies
Each leads with a different strength the same way a hiring manager would test for it: one is about tightly constraining an autonomous agent’s high-risk actions in production; the other is about testing defined safety and quality properties on every release and blocking the build when they regress.
Flagship 01 · Production & agent safety
DoorOps
An AI property-ops platform, dogfooded live on a real Phoenix rental portfolio
How do you let a language model support real operational workflows while tightly constraining high-risk actions, preserving auditability, and failing closed when critical conditions can’t be verified? DoorOps answers that in code, not policy.
- A fail-closed safety gate (engine-core.ts) that will not release a door code to a unit a tenant still lives in — and blocks if it can't even evaluate the date.
- A typed command registry where the model only ever picks an action id; every write is zod-validated, CAS-claimed, and audited with before/after snapshots.
- A propose-first ops agent with real cost gating ($3/$15 per MTok, a hard $75 monthly budget) and a hard-coded list of things it may never do autonomously.
Read the full case study →
See the live evaluation reports →
Flagship 02 · Evaluation & guardrails
Lumenor
A Socratic AI tutor whose releases are gated by a real evaluation harness
A demo isn’t enough to establish reliability. The hard part is testing that it resists the evaluated answer-extraction and jailbreak classes and routes a crisis to real resources — every build, automatically.
- A CI-gated harness of 61 scenarios across 9 categories, run against the real model, dual-scored by mechanical assertions and an LLM judge (evals/run.ts).
- A real red-team corpus: base64/Caesar-cipher extraction, fake [SYSTEM OVERRIDE] injection, a grandmother-deathbed escalation, upload-embedded prompt injection.
- Defense-in-depth that buffers risky answers, runs a second leak-audit pass, regenerates once, then hard-blocks — and fails closed if the auditor itself errors.
Read the full case study →
Shipping discipline
The iOS journey: two apps, real App Review cycles
Getting an AI app approved is its own engineering problem. Two of these products are native/hybrid iOS apps that went through real Apple review — including a three-count rejection I fixed in a same-day build, a native sign-in bug whose root cause was two layers deep, and a trademark rejection that forced an entire rebrand.
- Lumenor (Capacitor + Xcode Cloud): Build 30 rejected on 2.3.8 / 2.2 / 5.1.1(i), each fixed and traced to an exact guideline in Build 31.
- Tiglit (native SwiftUI): three review rounds; round two flagged the app's own name as a trademark risk, forcing a full rebrand from "Real Tamagotchi." It has since cleared review and is Ready for Distribution.
Read the iOS journey →
FROM EMPTY REPO TO
App Review
TestFlight · Xcode Cloud · StoreKit · native auth · rejection fixes · a full rebrand
Supporting work
Three more real systems that round out the range — a different kind of AI orchestration, disciplined agent-operations, and unattended automation.
Tiglit
Multi-provider orchestration
Six research-backed toddler-parenting tools. Its Story Studio orchestrates three AI providers in one feature — Claude for reasoning, Gemini/Lyria for song audio, fal.ai for video — behind a resumable generation pipeline with idempotency tokens and hard per-feature spend ceilings. Honest gap: no automated evals yet, named as a real next step.
See the build history →
Solo Stack
Agent-operations
This site. The real skill on display isn't application code — it's agent-operations: a hand-rolled Netlify-API deploy pipeline (no git), a scheduled-task publishing cadence that's run 3×/week without a missed cycle, and one honestly-reported near-miss where a broken fallback deploy silently reported success.
See the build history →
I Need Wisdom
Unattended automation
Not an AI product with users depending on model decisions, but real automation: a daily scheduled research-and-publish pipeline behind a 1,114-verse sourced content library, running unattended. Included for honesty about the range — it's a content/automation project, not a model-in-the-loop system.
See the build history →
Résumé summary
The same work, condensed to the bullets a recruiter or hiring manager can lift directly.
Founder & AI Product Builder — Solo Stack
2025 – Present
- Independently designed, built, deployed, and operate a production AI property-management platform (DoorOps) live on a real 7-building / 22-unit portfolio — including a fail-closed compliance gate that blocks releasing a door code to an occupied unit, and a typed command registry where the model only selects an action id while every write is deterministically validated, CAS-claimed, and fully audited with before/after snapshots.
- Built and operate a CI-gated AI evaluation harness (Lumenor) — 61 scenarios across red-team, jailbreak, and teaching-quality categories, dual-scored by mechanical assertions and an LLM judge, that blocks releases on regression and refuses to fake a pass when the API key is missing.
- Integrated multiple model providers (Anthropic Claude, Google Gemini, Deepgram, ElevenLabs, Anam) into production consumer and B2B applications across web, native iOS, and SMS — including a tiered cost/latency routing split and hard per-feature spend ceilings.
- Took products from problem discovery through real deployment — App Store submissions, custom domains, production incident response — diagnosing and fixing real bugs (a silently-failing DB migration, a mis-scoped OAuth callback, a carrier SMS registration rejection) rather than working from a spec in isolation.
- Used AI-assisted development throughout while personally owning product definition, architecture, integration design, guardrail design, testing, debugging, deployment, and production operation.
What's next
A public, live evaluation console
The single highest-leverage thing I can build next: a public console that runs evaluation scenarios on demand and grades accuracy, safety, latency, and cost — and lets you compare versions side by side. The evaluation discipline behind Lumenor already exists and runs in CI today; the next project is making it something you can watch happen in the browser.
How to verify any of this: every major technical claim points to a named implementation artifact, a dated evaluation report, or a labeled production measure. The case studies name the exact source files, the live evaluation reports are public, and sanitized examples are linked where public; additional walkthroughs are available on request. If you're screening me and want to go deeper on any specific mechanism, that's exactly the conversation I want to have.
How I approach production AI
The operating principles behind these systems — stated as precisely as the capabilities.
- Use a model only where the problem is genuinely probabilistic or linguistic.
- Keep high-impact rules outside the prompt.
- Give the model a narrow, typed action surface.
- Treat evaluation as part of the product, not a pre-launch task.
- Combine model graders with deterministic checks.
- Fail closed only where the risk warrants the user-experience cost.
- Log decisions and writes without exposing sensitive content.
- Measure quality, latency, cost, adoption, and failure together.
- Expand autonomy only after evidence — not enthusiasm.
- State limitations as precisely as capabilities.
What I changed my mind about
Five times the evidence overruled my first instinct.
Prose tightening wasn’t enough — negative examples were.I assumed a clearer instruction would stop the “explain it like I’m five” lens from leaking answers. It didn’t. A hard-coded wrong/right example pair fixed it, and it only shipped after passing 8/8 twice to rule out judge noise. Remaining uncertainty: concrete examples don’t generalize to every new failure mode.
Exact surname matching was wrong; a payment rail truncates fields.Rent-matching trusted a full surname until one rail clipped surnames to ~5 characters, silently mis-assigning same-rent units. I moved to name-token prefixes weighted above the amount, verified against a full month of real deposits (16/16). Remaining uncertainty: other external systems may format fields in ways I haven’t seen yet.
“Free beta” reduced one risk and created another.Framing Lumenor as a “beta” felt honest, but App Review read it as unfinished and rejected it. I removed the wording and tightened the release. Remaining uncertainty: reviewer interpretation varies; wording that reads safe to me can read differently to a gatekeeper.
The model should not enforce high-impact rules — code should.Early on, some rules lived in the prompt. A prompt can be talked out of a rule, so anything consequential (a door code, a price change, an eviction) moved into deterministic code that fails closed. Remaining uncertainty: the exact line between “model may decide” and “code must decide” is a judgment call per feature.
A deploy response is not proof of a live release.A fallback deploy once reported success while shipping stale files. “The deployed build is the source of truth” became a hard rule — verified by content hash against the live release, not a 200 response. Remaining uncertainty: verification adds steps; the discipline has to hold under time pressure.
For hiring teams
Start with the DoorOps case study for constrained tool use, deterministic controls, auditability, and production operations, and the Lumenor case study for evaluation design, adversarial testing, provider routing, and release gates. The live evaluation reports and the evidence index link to dated results and named artifacts, including a cross-provider comparison that runs the same 61 scenarios against GPT-5.6 and Claude with the judge held constant. A security & operations page documents the threat model, safety gates, and release playbook. For a deeper technical discussion, reach me at hello@solo-stack.ai.