For fourteen weeks a systemd user timer on a box in my house generated three articles a day, rendered them to static HTML, committed them to git, pushed, deployed to Vercel, and queued the social posts. It ran on 81 of the 85 possible days between May 6 and July 29. It published 278 articles.
As of this morning the newsletter has zero subscribers. The social queue has posted nothing, ever. Fifteen sponsorship emails produced zero replies.
The software worked. The publication failed. Those are separate claims, and going through the logs to write this was the first time I took the second one seriously.
What Apollo actually was
The pipeline is 29 scripts totalling 4,101 lines across scripts/*.mjs, scripts/*.sh and scripts/lib/*.mjs, plus about 390 lines of Vercel serverless handlers for subscribe/unsubscribe/health. Three systemd --user timers drove it:
- 13:30 UTC — draft generation. Three desk briefs in, three drafts out, running on OPENAI_MODEL defaulting to gpt-5.5.
- 14:00 UTC — publish. apollo-publisher.mjs --approve-all, then hydrate-autopost-queue.mjs, fetch-social-embeds.mjs, deploy.sh, and autopost.mjs --publish chained as ExecStartPost steps.
- 14:30 UTC — email digest.
Generation was the reliable part. Across 82 recorded runs the generator logged 237 successes against 9 failures — 75 runs of `[daily-drafts] 3 succeeded, 0 failed`. In fourteen weeks it produced 179,258 body words and 1,196 minutes of claimed reading time.
There was no human editorial gate. approveDueDrafts() flips every article with status 'draft' to 'approved' unconditionally when --approve-all is passed, and --approve-all is exactly how the systemd unit invokes it. The only quality gate, scripts/validate-site.mjs, ran inside Vercel's build — which happens after deploy.sh has already committed and pushed. Every check in the system ran downstream of the point of no return.
The duplicate slug that froze the site for five days
From June 4 through June 8, fieldsignal.tech served the same build. The git log looked perfect the entire time: Apollo committed and pushed eleven files every day at 14:00 UTC. Nothing alerted. Nothing looked wrong from any angle I was actually looking at.
The build had been failing on one line:
Duplicate article slug in content/articles.json: sports-rights-enforcement-stackOn June 4 the generator wrote a near-duplicate of the June 3 article and slugified to the same slug. The two drafts had different ids — fs-2026-06-03-sports-rights-enforcement-stack and fs-2026-06-04-rights-enforcement-stack — and ingestDraftOutputs() dedupes on the id:
const existing = new Set(articles.articles.map((article) => article.id));
// ...
if (existing.has(draft.id)) continue;Both ids were new, so both entries were appended. validate-site.mjs caught the collision correctly — just inside Vercel, after the push. And because Vercel keeps the last good deployment live when a build fails, the site did not go down. It just stopped moving. A frozen site and a healthy site look identical to a casual visitor, and identical to me.
The two headlines were 'Sports rights are not a bundle anymore. They are an enforcement stack.' and 'Sports rights are no longer just distribution. They are enforcement.' The machine wrote the same article twice and had no way to notice.
The fix took 66 deletions: drop the June 4 entry, keep the canonical June 3 one so no URL changed. Deploy went Ready.
It came back in three hours, and the real fix was in three places
Apollo's next run re-introduced the identical duplicate. I had committed the fix at 20:52 on June 8; the 14:00 run on June 9 broke production again, and I re-fixed it at 17:12 — about three hours after the automated run put it back.
Three mechanisms were compounding, and I had only addressed zero of them:
- The stale source draft was still sitting in output/drafts/. That directory is gitignored, so the git-side fix could not see it, and the next run re-ingested it.
- ingestDraftOutputs() still deduped by id, so the re-ingested draft was still 'new'.
- deploy.sh's push-race recovery runs `git rebase -X ours origin/main`, which re-favors Apollo's articles.json over any hand-fix that landed in between.
Making it stick required removing the entry from content/articles.json AND the matching item from content/autopost-queue.json AND deleting the gitignored source draft — 84 deletions across two committed files plus one file git could not see — then committing after Apollo's latest commit so the rebase strategy would not undo it.
The durable fix is one line: dedupe on slug, not id. I wrote it up as landmine L1 in the platform blueprint on June 9. It has still not shipped. The id-only dedupe is on line 33 of apollo-publisher.mjs today.
Three weeks of failed builds with no logs
The last good production deploy before the outage was June 14, 14:00:06 UTC. After that, production deployments came back ● Error with a build duration of 0ms. Not a failing build — a build that died before it started, so `vercel inspect --logs` returns nothing at all. Fourteen consecutive ● Error production deploys are still retained, running from June 29 through July 8; the ones from June 18 to June 28 have aged out of Vercel's retention, so I can't give a hard total.
What I recorded at the time was BUILD_FAILED with 'Resource provisioning failed'. I could not re-verify that string today — only the ● Error status and the 0ms duration survive — so take it as what I saw then, not as re-confirmed.
`npm run build` ran clean locally throughout. It was never the code. It looked like a platform or account-level block on the Hobby plan, and the most likely trigger was that I was actively running sponsorship sales off a free-tier project. It cleared on July 8 through the dashboard: last ● Error at 23:07:28, first ● Ready at 23:49:16, 42 minutes later. Production has been green since, with the twenty most recent deploys all Ready at 11-18 seconds.
Roughly three weeks of daily articles sat in git and never reached the live site. The engineering failure is the same one as the duplicate slug: deploy.sh commits and pushes but never verifies that Vercel actually built. A dead deploy pipeline produces exactly the same signal as a healthy one. That gap is landmine L3 in my own blueprint. It is also still open.
Two publishers, one branch
Earlier, in May, I had a stranger version of the same problem. A GitHub Action was scheduled at 13:30 UTC and the local systemd timer pushed at 14:00 UTC. The Action won the ref every single time:
! [rejected] main -> main (fetch first)
error: failed to push some refs
hint: Updates were rejected because the remote contains work that you do not
hint: have locally. This is usually caused by another repository pushing to
hint: the same ref.Eleven consecutive rejections in the publish log. The two publishers were doing different jobs, which is what made it bite: the Action pushed but generated no new articles, and the local job generated all the articles but could never push them. Twelve commits landed from the Action between May 7 and May 18 while eleven local commits piled up unpushed.
I merged the histories on May 18 with local content taking precedence, regenerated index/archive/feed/sitemap/podcast deterministically from content/articles.json against the unified 59-article set, and disabled the Action's schedule:
# Daily schedule disabled 2026-05-18: local systemd timer
# (field-signal-apollo-publish.timer) is the sole automated publisher.
# The schedule was racing with the local job and causing push rejections.
# schedule:
# - cron: "30 13 * * *"Then I made deploy.sh race-safe with a fetch / rebase -X ours / re-render / amend / retry-push path. That is the exact mechanism that resurrected the duplicate slug three weeks later. The fix for May caused the recurrence in June, and I did not connect the two until I sat down to write this.
The distribution half never ran
content/autopost-queue.json holds 285 social packages. Every one has status 'queued'. Every article got a 4-6 post X thread and a LinkedIn post written for it, for fourteen weeks.
For the first two months, scripts/autopost.mjs — the only script that actually posts — was not in any Apollo timer. The queue hydrated daily and drained never. When I finally wired it into the systemd unit on July 8, it attempted 129 posts and published zero, because no write credentials existed:
{"webhook":{"skipped":"SOCIAL_WEBHOOK_URL not set"},"x":{"skipped":"X_USER_ACCESS_TOKEN not set"},"linkedin":{"skipped":"LINKEDIN_ACCESS_TOKEN or LINKEDIN_AUTHOR_URN not set"}}129 log entries of 'no channel published (all skipped/errored)'. output/sent-log.jsonl has 207 lines and 207 of them are seeded backfill skips. The one thing I did right was making the poster retry-safe — it only marks an item sent when a channel actually published — so the log says 'Not marking … as sent' 129 times instead of quietly recording 129 phantom posts.
The newsletter had its own version of this. The subscribe API had been migrated to Supabase on May 6; the sender still read the Upstash Redis set. The Supabase project itself was paused. Anyone who subscribed would have been written to a dead store the sender never looked at.
I did not notice for two months, because the failure mode was indistinguishable from the actual state of the business. The log line is the same whether the store is empty or simply the wrong store:
[send-daily-email] no subscribersEighty consecutive runs of that. Zero runs that ever logged a subscriber loaded. I moved both handlers back onto Upstash on July 8 and cut Supabase entirely. The newsletter stack has been correct and fully provisioned ever since, and has served zero people.
Selling adjacency to an audience that didn't exist
This is the part I would most like back. On May 18 I sent 15 cold sponsorship emails, roughly six minutes apart, nine tier-1 and six tier-2. All 15 rows in the tracker show outcome 'sent' and an empty reply_received_at. Zero replies.
The sponsor page priced a founding slot at $500 for 3 months against a $2,000 standard rate, and promised placement in '≈65 issues over 3 months' of a memo that 'goes out every weekday to the people writing checks for sports technology.' The subscriber count at the time was zero. I am not going to name the companies — they ignored an email, which was the correct response.
The machine only knew one sentence
Reading 271 titles in one sitting is a specific kind of unpleasant. 116 of them — 43% — use the identical construction 'X is not Y. It is Z.' Widen it to any negation pivot (is not / are not / is no longer / did not) and it's 151 of 271, 56%. 34 titles contain 'stack', 28 contain 'layer'. The word 'operator' appears in the body of 236 of 271 articles.
On the final day it published these two, side by side: 'FIFA is not privatizing the World Cup. It is selling the tournament OS.' and 'FIFA is not just selling the World Cup. It is building a rightsco.'
The generator also had a habit of substituting a CJK token mid-sentence and then truncating, which shipped straight to live SEO meta because nothing in the pipeline scanned article fields for non-ASCII. Google was served deks ending '…sponsor adjacency, and human sign-off before anything is放' and '…approval history before they触'. Eight corrupted fields across six articles, cleaned by hand on June 9 using each article's own whyItMatters field to reconstruct the intent. A sanitizeNonAscii backstop went into the generator afterward; a re-scan today finds zero remaining artifacts.
And all 278 machine-written articles carry my personal byline in three places — the visible byline link, the JSON-LD author Person object, and `<meta name="author" content="Pranav Patel" />`. All three are hardcoded string literals in the template, not data fields. There was no place in the system to say a machine wrote this even if I had wanted to.
I had already been told
On June 1, two months before writing this, I commissioned a deep-research pass on my own site. It got it right immediately:
This niche is newsletter-led, not SEO-led… The newsletter is the core asset that makes every other improvement (and every revenue path) work.
The dominant risk is shipping unreviewed AI output under the masthead + inflating thin page count.
A newsletter memo from the same day said it in one line: 'The real bottleneck is distribution, not tooling.' Eight days later I committed a platform blueprint that listed the pipeline's own defects under the heading 'Known landmines (design around these — they currently ship bugs to live)' — L1 id-not-slug dedup, L2 CJK artifacts, L3 silent Vercel build failure, L4 rebase -X ours clobbering hand-fixes, L5 no X/LinkedIn creds. I wrote that list. Most of it is still unfixed.
Then I went and built a third desk. The Markets desk launched June 10 with three articles, produced one more on July 8, and stopped forever, because the cron that would have made it self-sustaining was never wired. The Cars desk got its automation on July 8 and immediately started producing daily for 20 days.
Here is the honest read. I kept working on the generator because the generator gave me feedback. A failing build hands you an error string and a stack trace. A missing audience hands you nothing at all — the logs look the same on day one and day ninety-eight. So I optimized the half of the system that could tell me I was wrong, and never built the half that could tell me whether any of it mattered.
What I don't know, and what replaces it
There are numbers I still can't state. Vercel Web Analytics and Speed Insights are live and their scripts return 200, but the counts only ever lived in the dashboard and I never pulled them; same for Search Console impressions and clicks. I have no cost log at all for fourteen weeks of daily generation against the OpenAI API. And the JSON-LD still points sameAs at an X account the pipeline never posted to once. The numbers I'm certain of are the ones I opened with: 278 articles, zero subscribers.
fieldsignal.tech becomes a personal engineering journal. Same name, same domain, flat homepage — newest first, tags do the sorting, no desks in the navigation. /sports and /cars survive as demoted automated desks under a disclosed institutional byline instead of mine. Markets is done.
The 278 articles can't just be re-bylined into the new thing. The prose is wrong at the sentence level; you cannot get from 'X is not Y. It is Z.' to a paragraph about a rebase strategy resurrecting a slug collision. And the material I'd actually want to publish was already written — 379 notes in a local vault and a pile of session transcripts, none of it on the site, while the machine shipped 179,258 words that nobody asked for.
The timers are still armed. They fired at 14:00 today and published two articles about FIFA.