kestrelsnest-blog/2025-09-17-the-reality-of-production-when-hope-meets-live-users.md at main

Files

Eric Wagoner 8ac2a6ad8c Add callouts to LocallyGrown series for enhanced readability

Applied strategic callouts across all 4 posts in the LocallyGrown series:
- Info, success, warning, danger, tip, quote, and example callouts
- Improved visual hierarchy and information highlighting
- Better mobile readability with structured content blocks

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-09-23 10:02:52 -04:00

20 KiB

Raw Permalink Blame History

title, description, date, preview, draft, tags, categories, lastmod, keywords, slug

title

description

date

preview

draft

TL;DR

{{< callout type="warning" title="The Production Reality Check" >}} I launched with 3,000 passing tests, weeks of manual checks on production data, and a two-month beta—and still hit two weeks of real-world bugs:

DNS delays, fee miscalculations, role-permission gaps
Stripe edge cases, fragile email templates
A thousand small cuts
Result: 314 commits in 18 days to keep markets running

This is what broke, why, and what I'd do differently. {{< /callout >}}

Launch Morning: The DNS Disaster

August 14, 2025, Thursday morning. What should have been a five-minute DNS switch became a multi-hour nightmare. Years ago, I had set up Cloudflare for the domain, a detail completely forgotten after years of not needing to touch DNS. The changes I made weren't propagating. The "coming soon" page stayed up while I diagnosed what was happening.

What should have been done by 3 AM dragged on past dawn. By the time I untangled the Cloudflare mess and got DNS propagating correctly, I was already exhausted. But finally, the new LocallyGrown.net was live.

I'd taken two days off from my regular job, planning to relax, handle a few tech support requests, maybe squash a bug or two. Mostly relax.

Instead, I got maybe eight hours of sleep total over the next four days.

The first manager email arrived within hours. Then another. Then another. By evening, my inbox was a cascade of increasingly urgent discoveries:

Desktop views showing blank pages while mobile worked fine
The three dollar flat fee disaster: every Stripe payment charging $3 instead of 3%
Growers locked out of editing their own products
Invoices no longer grouped by grower, breaking the physical workflow of how volunteers distributed food
Some customers charged zero, others charged twice
"Internal server error" when adding milk to cart
Decimal amounts wouldn't save, so managers couldn't adjust balances for $12.50 worth of extras

One manager was getting calls, texts, and emails from confused customers. By evening of the first day, she had only 7 orders when she usually had dozens. Customers were placing duplicate orders thinking the first hadn't gone through. Growers couldn't reactivate products after vacation mode. The harvest summary page showed negative quantities (-1 egg for -$5).

But there was one bright spot: "The mobile checkout is so much better than before," one customer told their manager. At least something was working better.

The real testing had just begun.

Bug Category #1: The Stupid Math Mistakes

The Most Embarrassing Bug:

Market credit card fees weren't being calculated as percentages; they were being added as flat amounts.

// What I wrote (wrong):
const surcharge = market.customerPercentage; // If 3%, adds $3.00

// What it should have been:
const surcharge = subtotal * (market.customerPercentage / 100); // 3% of subtotal

How This Slipped Through:

My test data was too convenient:

$100 order with 3% fee = $103 total ✓
Looked correct, but for entirely wrong reasons
The three dollars was a flat fee, not 3% of $100

On real orders:

$47.50 order with 3% fee = $50.50 (wrong - added $3)
Should have been $48.93 (3% = $1.43)

Markets were overcharging on small orders, undercharging on large ones.

My Git logs from that first week are littered with fixes for math bugs. Markets can customize their pricing structure in an almost infinite number of ways, and even with thousands of tests I didn't capture everything I needed to.

Bug Category #2: Role-Based Permission Failures

What I Thoroughly Tested: Manager accounts doing manager things.

What I Didn't Test Enough: Grower-only accounts, volunteer-only accounts, customer-with-no-special-role accounts.

The bugs were everywhere, all in the direction of being too restrictive:

Growers couldn't edit their own products (Aug 15)
Volunteers couldn't access the volunteer functions they needed (Aug 28)
Growers couldn't remove items from harvest lists (Aug 22)
Non-grower admins couldn't add products (Aug 17)
Growers couldn't access their Sales History (Aug 27)

Root cause: I validated backend rules but missed UI visibility checks for non-manager roles.

<!-- The broken check -->
{#if user.isAdmin || user.isManager}
  <button>Edit Product</button>
{/if}

<!-- Should have been -->
{#if user.isAdmin || user.isManager || (user.isGrower && product.growerId === user.growerId)}
  <button>Edit Product</button>
{/if}

My manual testing had a fatal flaw: I tried to test every kind of account, but I mostly spent my time viewing the site as a manager would. Managers can do everything. I ran out of time before I could test every kind of account with every button and control in the system. The back end logic was correct and tested, but that doesn't help when the button you need to click isn't even visible.

Bug Category #3: Payment Processing Disasters

The Stripe integration broke in spectacular ways. The first disaster happened while we were still in the beta period, when I had created a whole set of safeguards that allowed for testing every aspect of the payment process but simulated the final step of charging the card. I'd accidentally left one path unguarded, and a manager charged several of her customers for copies of their real orders. It was painful, and we hadn't even launched yet.

LocallyGrown’s payment flow is unlike typical e-commerce. When customers place their orders, often times the produce is still in the field. It's not unusual for orders to be received five days after they're placed, and there are many, many reasons why what gets delivered doesn't exactly match what was ordered. The simplest way to handle this is to verify the card, save it as a payment method tied to the customer, and then charge that card after the order is picked up and everything is verified and reconciled.

Stripe allows me to do all this, though it's not their typical way of handling payments. On the old system, I was using an extremely early version of their APIs, but they have added a lot of complexity since then and that simplicity was not available to me. I tested everything I could, using the test environment they provide as well as running countless small real charges to my personal cards. It seemed as ready as it could ever be, but once the public started using the system, the bugs started appearing.

Every time one was reported, it became the highest priority. It was imperative that I get a fix in place before anyone else was affected, and with many markets running cards on different schedules, I had to act quickly. Luckily there weren't many, but when they happened I had to drop everything and fix them.

Bug Category #4: Communication Breakdown

My application sends thousands of emails every day. Customers need to be kept informed of what to expect when they receive their order. Growers need to know what to harvest. Managers expect a constant flow of information delivered to their inbox. All of this information is available in real time on the website, but when you're riding a tractor or setting up canopies in a parking lot, sometimes pulling up an email is the best way to get the information you need.

I'd completely revamped the email system to be more flexible, to allow greater insight into deliverability, and to be friendlier in tone. It was a big change, and I was proud of it. But it also meant that I had to re-write all of the templates from scratch, include both plain text and formatted versions of each, and make sure they called the right variables in the right places.

And of course I missed a few things here and there that looked good to my eyes but were noticed immediately when used by the markets. In a few places I used the old Rails variable names, so a gap appeared in an email instead of, say, a product name. Or in the process of changing the layout of a nicely formatted email I omitted a column so the growers didn't have the contact information for a customer they were used to seeing.

None of these were major problems, and they were easy to fix. But, they piled up on top of everything else and became another thing for my customers to be frustrated with.

Bug Category #5: Death by a Thousand Cuts

The Git history tells the rest: CSV exports with wrong columns. iPhone photos that wouldn't upload. Infinite scroll that wasn't infinite. Password resets that didn't reset. Bots flooding the logs so I couldn't see real errors.

Every subsystem had issues. Every feature had edge cases. Every workflow had something I'd missed.

The Human Cost

Market managers were drowning. One manager sent me 8 detailed bug reports in the first week alone. She was simultaneously:

Fielding calls from confused customers
Helping growers who couldn't access their products
Manually adjusting incorrect charges
Trying to run a physical market with broken invoices

The invoice bug wasn't just about data display. When items weren't grouped by grower, volunteers had to walk back and forth across the market instead of going down the line. Physical workflow, broken by code.

The "can't use decimals" bug meant managers couldn't close out their markets. They literally couldn't mark orders complete because they couldn't enter $12.50 for extras, only $12 or $13.

This wasn't just a website with bugs. This was dozens of live food systems, hundreds of growers, thousands of customers depending on me to keep their local food networks running. Real businesses. Real money. Real communities.

The Two-Week Bug Avalanche

{{< callout type="danger" title="The Two-Week Bug Avalanche" >}} The Numbers:

314 commits from August 14 to September 1
58 pull requests merged
105 commits with "fix", "bug", or "critical" in the first week alone

Look at the PR names from August 18:

"morning bugs"
"afternoon fixes"
"evening bugs"
"night bugs" {{< /callout >}}

I was naming my branches by time of day because there were too many to give them meaningful names. I was working quickly and fixing things as quickly as they were coming in, but they just kept coming.

My Personal Reality

For the most part, my customers (the market managers) were understanding. They could see I was responding quickly and decisively. A few were angry at the surprise instability, and might still be angry.

I was overwhelmed. My morale was at an all-time low. My disappointment in myself for not catching so many bugs that were obvious in hindsight was immense. Every bug report felt like a personal failure. Every "this used to work" comment cut deep.

I've been in the software industry long enough to see many, many rollouts go horribly wrong. Goodwill can be destroyed overnight, and some large companies never recover. As a consulting developer, I've even been called in to help fix the messes caused by these disasters. I had to keep reminding myself that I've seen much worse, and even if it didn't go as planned I was capable of fixing it.

And I was also determined to get it done. There was no rolling back, so I had to go forward. I had to dig my way out and redeem myself and the choices I made along the way.

What kept me going was knowing this wasn't about me. It was about the communities depending on this system.

The Unsung Heroes

The market managers became my QA team in production. They had every right to be furious. Instead, they sent detailed bug reports with screenshots, exact reproduction steps, and impact assessments.

{{< callout type="example" title="Evolution of Bug Reports" >}} One manager's emails evolved into increasingly detailed technical reports:

Day 1: "The total amount due to growers is displaying incorrectly"
Day 3: "When I export the checks .csv the amounts are correct but the display shows $85 instead of $73.95 after the 13% fee"
Day 5: "I can replicate this. It only happens when you select specific growers like 3C or BMB unless you also choose honey or baked goods category" {{< /callout >}}

They were debugging complex interaction patterns while running physical markets. They discovered edge cases I never would have found. They showed patience I didn't think possible.

By Week 3, the emails changed tone. The bug reports became less frequent. The managers stopped apologizing for "all the emails."

Success wasn't about the code being perfect. It was about the system becoming invisible again: managers could run their markets without fighting the software.

Lessons From The Bug Apocalypse

1. Test Data Can Hide the Truth

Using convenient test values ($100 orders, 10% fees) hides calculation bugs. Real-world data is messy: $47.83 orders with 2.75% fees reveal rounding errors immediately. I had an extensive library of test data, but most of the products had nice round numbers that were easy to verify but hid the bugs.

Seed fixtures with non-round prices and odd percentages; add randomized totals to catch rounding errors.

2. Your Primary Account Blinds You

Testing mainly with your admin account means you'll miss most permission bugs. You need test accounts for every role combination, and you need to use them repeatedly, even when you just change a single button.

Automate smoke flows per role; assert on control visibility, not just API responses.

3. Mock APIs Aren't Reality

Stripe test mode worked perfectly with clean test cards. Production Stripe used with real cards and long-standing accounts worked differently. I needed to better understand the newer APIs and how they differed from the old simple ones I was using.

Mirror prod settings in a staging account; record/replay webhooks and handle retries/duplicate events.

4. The Correct Backend + Broken Frontend = Broken System

A single wrong Svelte condition can make perfect backend logic useless.

Add component tests for visibility conditions and null/“unlimited” states.

5. Email Templates Are Code

They need testing. They need error handling. They need to handle missing data gracefully. "Hello undefined" is not acceptable.

Snapshot test rendered templates with missing fields; fail builds on “undefined” placeholders.

6. The Frontend Testing Gap

My backend had 3000 tests. Rock solid. But Svelte 5's testing story in 2025? Still evolving.

@testing-library/svelte had experimental Svelte 5 support with known bugs. The mount() function threw "not available on the server" errors in jsdom. Testing $derived runes didn't detect reactivity changes. Cypress and Playwright worked for basic flows but couldn't properly test Svelte 5's new reactive primitives.

Ideally, I'd have automated tests running as manager, grower, volunteer, and customer accounts at various screen sizes. More critically, these would have caught the permission-based bugs where essential buttons and controls were hidden from the very users who needed them: growers who couldn't edit their own products, volunteers locked out of their tools. That's still the goal. But with the testing tools available and the Rails system dying, I had to choose: wait for the ecosystem to mature, or launch with manual testing only for the frontend.

I chose to launch.

Until tooling stabilizes, lean on Playwright smoke suites across roles and screen sizes; treat them as gate checks.

7. Edge Cases Aren't Edge Cases

Products with unlimited inventory, markets with 0% fees, users with multiple roles. These aren't edge cases. They're Tuesday. When I offer near-unlimited flexibility, I'm taking on the responsibility to keep the freedom I've given working.

Promote “edge” scenarios to first-class test data and UX checks.

8. Beta Testing Has Limits

The handful of managers who beta tested provided invaluable feedback. But they weren't a representative sample. I set up perfect copies of every market with real data, gave two months of access, sent regular updates. Still, most managers (including those with the most complex customizations) waited until launch day.

The lesson: Beta testing helps, but it's a self-selecting sample. The users with the most complex workflows are often too busy to test until they have no choice.

Identify high-risk markets and schedule guided sessions; don’t rely on opt-in.

9. Launch and Fix > Wait for Perfect

The old Rails system was dying. Every day I delayed was another day it might fail completely. Launching with bugs I could fix was better than not launching at all.

Pair fast rollback-like mitigations (feature flags, circuit breakers) with rapid patch cadence.

Where We Are Now

{{< callout type="success" title="Four Weeks Later" >}} It's been four weeks since launch. I fixed more bugs last night. Reports are still coming in, but it's a trickle now instead of a flood.

But here's what matters: The system works. Markets are running. Orders are flowing. Communities are getting their local food.

Yes, it was messier than I'd hoped. Yes, I made mistakes. But I also:

Successfully migrated 23 years of Rails code to modern infrastructure
Kept dozens of markets running without any data loss
Fixed 314 commits worth of bugs without breaking production
Maintained trust by responding quickly and transparently {{< /callout >}}

I'm not in crisis management mode anymore. I'm able to think about (and even implement) new features. The system is stable enough that I can look forward, not just fix what's broken.

The migration isn't complete, but we're getting there. And more importantly: we're still here.

What's Next: How One Developer Used Every Tool Available

From day one of this migration, I used the current generation of programming tools to accomplish what would have been impossible alone. Enhanced IDE autocomplete, AI-powered code generation, groups of specialized agents I created and orchestrated. These weren't crutches; they were force multipliers.

But let's talk about that loaded term: "AI." Like "farming," it's a broad term that covers both the exploitative and the ethical. Most food comes from factory farms that destroy communities and environments. But LocallyGrown.net exists to support the farmers who grow food responsibly. Similarly, many AI companies are built on stolen art and exploited resources. But there are also thoughtfully-created development tools that learned from code developers explicitly shared to help others.

I was the architect, the decision maker, the one who read and understood every line of code before it was committed. But I typed with more than just my fingers, using tools that amplified rather than replaced my expertise.

Part 5 will cover:

Why "AI" (like "farming") isn't a monolith, and why that distinction matters
How modern development tools made a solo migration of 23 years of code possible
The reality of being the human in the loop: reviewing, correcting, and owning every line
The ethics of using AI tools while opposing AI exploitation
Building a workflow that let me multiply my efforts without losing control

This is the story of using every available responsible tool to save a system that feeds communities. Not because I couldn't code, but because I could code smarter, and because sometimes the right tool for the job happens to use machine learning.

This is part four of a series documenting the rescue and modernization of LocallyGrown.net.

The Series

From Accidental Discovery to Agricultural Infrastructure (2002-2011)
The 23-Year Rescue Mission: Saving Agricultural Innovation from Technical Extinction
The Architecture Challenge: Translating 19 Years of Rails Logic to Modern SvelteKit
The Reality of Production: When Hope Meets Live Users ← You are here
Lessons from the Solo Developer Using Modern Tools
The Future: Building on Modern Foundations

20 KiB Raw Permalink Blame History Unescape Escape