Chris' Blog - Kubernetes, Platform Engineering, and DevOps

Connecting My Local AI to GrowthBook with MCP (Part 6)

Sun, 06 Sep 2026 00:00:00 GMT

I use GrowthBook feature flags for the site's interactive demos. One flag selects a scene, another controls an effect, and targeting rules can change what a particular visitor sees. I wanted to ask the site's chat a question like “What is dotc-fx set to?” and have it look up the answer.

The model was already running locally through Ollama, with the Cloudflare Tunnel from Part 3 connecting it to the site. I added a Docker service that exposes GrowthBook reads through Model Context Protocol, then wired the chat's Pages Functions to call it.

The first flag lookup returned FIRE. Getting that value into a normal chat answer involved flag evaluation, two routes through the same tunnel, and a couple of differences between testing in Node and running in Cloudflare Pages.

How the Request Travels

Model Context Protocol, or MCP, lets an application discover a server's tools and call them using defined input schemas. My server offers operations for listing permitted flags and evaluating a flag by key.

The Pages Function acts as the MCP client. It discovers those tools and includes their definitions in the request to Ollama. When the model asks for a flag lookup, Pages calls the MCP server, adds the result to the conversation, and asks the model to produce the answer.

Browser: Neon GPT or Station Intelligence
                 |
                 v
Cloudflare Pages: /api/llm or /api/ask
       |                           |
       v                           v
llm.chrishouse.io/v1     llm.chrishouse.io/mcp
       |                           |
       +---- Cloudflare Access ----+
                     |
           Existing Cloudflare Tunnel
                     |
              Desktop at home
              /             \
      Ollama :11434      Docker MCP :8792
      Qwen model               |
                               v
                      GrowthBook SDK endpoint

The site's article search still supplies background from posts, and Redis memory supplies conversation context. The flag tool adds a lookup at the time of the question. That matters for a value I can change in GrowthBook after publishing an article about it.

Building the MCP Server

I used Microsoft's MCP builder agent skill as a guide for the tool definitions, validation, and transport. The service is a TypeScript application under services/home-mcp/ in the blog repository, with its own dependencies, tests, and Dockerfile.

It uses the MCP TypeScript SDK's v1.x line and Streamable HTTP. Each request gets a fresh server and transport instance, with JSON responses. That gives the Pages Function an HTTP endpoint it can call with fetch.

The server registers three read operations:

Tool	Result
`growthbook_list_flags`	A paginated list of permitted flag keys and their evaluated values
`growthbook_get_flag`	The evaluated value of one permitted flag
`house_get_capabilities`	The service's configured integrations and availability

Each tool has a Zod input schema and an output schema. Results include structured JSON and a text representation. Both the server and the Pages client enforce allowlists: the server limits which flags can be read, and Pages limits which tool names it will execute.

The GrowthBook connection uses the public SDK payload and allows ten existing dotc-* flags. The service only needs read access to the configuration used by those demos.

Evaluating a Flag

A GrowthBook flag can have targeting rules and experiment assignments, so returning its defaultValue would miss part of the configuration. I pass the payload and attributes through the GrowthBook SDK evaluator.

The default evaluation context is:

{
  "id": "station-ai",
  "plan": "free"
}

The tools also accept id, plan, and dotId attributes. For example, the color demo targets by dotId. To investigate what a particular dot sees, the lookup needs that dot's identifier. The default context gives a consistent answer when the question supplies only a flag key.

This abbreviated response came from a local call using the real GrowthBook SDK endpoint:

{
  "key": "dotc-fx",
  "value": "FIRE",
  "on": true,
  "source": "force",
  "attributes": {
    "id": "station-ai",
    "plan": "free"
  }
}

The attributes and source help explain the returned value. They also make it possible to compare evaluations for different targeting contexts in the Flag Lab.

The full response includes fetched_at, when the service fetched the payload, and source_updated_at, the timestamp supplied by GrowthBook. The service caches payloads for fifteen seconds. After that, a failed refresh produces an error. Missing flags and keys outside the allowlist also produce errors, while a flag evaluated to false remains a successful lookup.

Running the Service in Docker

The Compose definition is in docker-compose.mcp.yml. These are the relevant settings:

services:
  home-mcp:
    build: ./services/home-mcp
    container_name: chrishouse-home-mcp
    restart: unless-stopped
    init: true
    env_file: .env.home-mcp
    ports:
      - "127.0.0.1:8792:8792"
    read_only: true
    cap_drop: [ALL]
    security_opt: [no-new-privileges:true]

The Docker image compiles the TypeScript and runs the application as the non-root node user. Docker publishes port 8792 on the host's loopback interface, where the local cloudflared service can reach it. The container has no host filesystem or Docker socket mounted.

From the repository root:

node scripts/configure-home-mcp.mjs
docker compose -f docker-compose.mcp.yml up --build -d
npm --prefix services/home-mcp ci
npm --prefix services/home-mcp run verify

The setup helper creates a random bearer token in the ignored .env.home-mcp file and writes the matching token and local URL to .dev.vars. The verification command checks the running MCP service.

For local development, Gatsby serves the frontend on port 8000 and the Pages runtime serves the API on 8788. Restart the Pages process after changing .dev.vars so it loads the new credentials.

Reusing the Existing Tunnel

Ollama already had a route at llm.chrishouse.io/v1. I added /mcp to the same hostname and pointed it at the Docker service. The relevant ingress rules are:

ingress:
  - hostname: llm.chrishouse.io
    path: ^/mcp/?$
    service: http://127.0.0.1:8792
    originRequest:
      httpHostHeader: localhost:8792

  - hostname: llm.chrishouse.io
    path: ^/v1/.*
    service: http://localhost:11434
    originRequest:
      httpHostHeader: localhost:11434

  - hostname: llm.chrishouse.io
    service: http_status:403

  # Other hostname rules remain above the final catch-all.
  - service: http_status:404

Cloudflared matches rules in order. The /mcp rule belongs above the hostname's 403 rule. This is an excerpt from a shared tunnel configuration; its other hostname routes stay in place.

I validated the configuration and restarted the Windows service to load it. The commands, run from Administrator PowerShell, are:

cloudflared tunnel --config "$env:USERPROFILE\.cloudflared\config.yml" ingress validate
Restart-Service Cloudflared

Pages uses HOME_MCP_URL and HOME_MCP_TOKEN for the MCP connection. The repository's helper can upload those two secrets to the site's Pages project:

node scripts/configure-home-mcp.mjs --cloudflare

Cloudflare Access supplies the outer authentication layer. Because MCP and Ollama share an origin here, the Pages client reuses the existing Ollama Access service credentials. The MCP server also checks its own bearer token, validates the host, and rejects browser Origin headers. Both sets of credentials stay in Pages Functions; the browser sends its chat requests to the site's API.

Connecting the Chat

Neon GPT enables the tool flow with home_tools: true in its request to /api/llm. Station Intelligence uses the same integration in /api/ask, alongside article retrieval.

The tool loop lives in functions/api/lib/home-mcp.js. It initializes MCP, discovers permitted tools, and translates their schemas into the tool definitions used by the chat completion API. A requested tool call goes through the client's allowlist before execution. Its result then becomes another message in the model's conversation.

I limited the loop to three calls per round and two rounds before the final answer. Individual MCP requests have an eight-second timeout, with a two-minute deadline around the completion flow. If the service is unavailable, the model receives that failure so it can explain why a current flag value could not be retrieved.

Qwen Sometimes Returns the Tool Request as Text

During testing, qwen2.5-coder:14B sometimes put this JSON in the assistant's text instead of the API's tool_calls field:

{
  "name": "growthbook_get_flag",
  "arguments": {
    "key": "dotc-fx"
  }
}

The original handler displayed the JSON and never performed the lookup. I added support for a whole-message JSON tool request whose name matches an offered tool. It follows the same validation and execution path as a native tool call. Requiring the entire message to match keeps ordinary prose with embedded JSON examples out of that path.

Pages Exposed a Fetch Difference

The MCP client initially used redirect: "error". Its Node test passed, but the Pages runtime rejected that fetch option. Switching to redirect: "manual" and explicitly rejecting redirects fixed the request in Pages.

This also handles an Access login redirect as a connection failure. The MCP client stops at that response instead of following it with the bearer token. Sending a request through the local Pages runtime caught the issue that the Node test had missed.

A Flag Lookup Through the Chat API

With Docker, Ollama, and the local Pages runtime running, a request to /api/llm returned this abbreviated response:

{
  "content": "The current value of the `dotc-scene` flag is `\"fx\"`.",
  "backend": "house",
  "home_tools": {
    "status": "connected",
    "calls": [
      { "name": "growthbook_get_flag", "ok": true }
    ]
  }
}

The home_tools.calls entry records the successful lookup alongside the answer. A request through Station Intelligence's /api/ask also called growthbook_get_flag and returned FIRE for dotc-fx.

The repository includes a chat verification script for exercising the integration with Ollama:

node services/home-mcp/scripts/verify-chat.mjs "Read the current dotc-fx flag using your tool."

For a flag question, I can now trace the answer through the tool call to its evaluated value and attributes. If the result differs from what a demo displays, those attributes are the first place to look: the tool's default station-ai context and the visitor's context can match different GrowthBook rules.

One Hostname, Two Origins: Cloudflare Load Balancer Cutover

Fri, 04 Sep 2026 00:00:00 GMT

We have a platform at work living on two sets of infrastructure at once. There is the original one, which everybody knows the name of and half the internal tooling has hardcoded. Then there is the new one, which got stood up beside it with the word prod wedged into its hostname because at the time nobody had a better idea. Nobody likes that name. The plan is that the original hostname survives, the new infrastructure ends up underneath it, and prod quietly disappears from anything a human ever types.

Most of the work here is deciding what a hostname is allowed to mean, and who gets to depend on it. The load balancer is just what lets me change the answer without changing the name. So before touching anything that matters, I built the whole shape on my own domain, where the blast radius is me.

The Goal

Three things:

service.chrishouse.io is the only name anyone sees, permanently. Whatever serves it can change underneath without anybody noticing.
Behind it, two independent origins, service-east and service-west, standing in for old infrastructure and new.
Moving production from one to the other is one deliberate action, it takes seconds, and it reverses just as fast.

The thing I explicitly did not want is a percentage ramp. Ten percent, then fifty, then ninety sounds sophisticated, but it only means anything if both backends are genuinely interchangeable to a user: shared database, no diverging session state. If they are not, a 50/50 split is two different worlds served at random. What I wanted was a switch. This side is live now, with an undo button next to it.

The Building Blocks

Cloudflare Load Balancing is the steering layer. It runs about $5 a month at the entry tier with two origins included, which is exactly two origins more than I need to prove the point. The object model is small: a load balancer attached to a hostname, holding an ordered list of pools, each pool holding endpoints, with a monitor health-checking them.

Cloudflare Tunnel connects the origins. Both of mine are containers on the desktop under my desk. No port forwarding, no static IP, no inbound anything. Cloudflare's own guidance is one tunnel per data centre and one pool per tunnel, which maps onto east and west without requiring any creativity.

Two nginx containers are the origins, standing in for the two application stacks. They do no routing and no load balancing of their own. All of that happens at the edge, and these are just the things at the far end that answer. Identical config, different environment variables, so the only honest difference between them is the label each one prints about itself. If the two sides were not interchangeable, the demo would be lying.

x-origin: &origin
  image: nginx:1.27-alpine
  restart: unless-stopped
  volumes:
    - ./nginx/default.conf.template:/etc/nginx/templates/default.conf.template:ro

services:
  east:
    <<: *origin
    environment:
      REGION: east
      STACK: legacy
      RELEASE: v1
    ports: ["8081:80"]

  west:
    <<: *origin
    environment:
      REGION: west
      STACK: new
      RELEASE: v2
    ports: ["8082:80"]

There is no application behind either one. They exist to answer a request and admit which side they are, and that admission is four lines of nginx config. This is a stub, not something you would ever put in a real config:

location = /api/whoami {
    default_type application/json;
    return 200 '{"region":"${REGION}","stack":"${STACK}","hostSeen":"$host"}';
}

${REGION} and ${STACK} get substituted once at container start by the nginx image's envsubst entrypoint, so they say what the container is. $host is an nginx runtime variable evaluated per request, so it says what the container was asked for. Those two being able to disagree is the entire point.

The Architecture

                  service.chrishouse.io
                           |
                  Cloudflare Load Balancer
                  steering: off (failover order)
                           |
              +------------+------------+
              v                         v
        pool: east                pool: west
        (live)                    (standby)
              |                         |
     <uuid>.cfargotunnel.com   <uuid>.cfargotunnel.com
              |                         |
        cloudflared               cloudflared
              |                         |
        nginx :80                 nginx :80
        legacy / v1               new / v2

The one setting that makes this work is traffic steering set to Off. Cloudflare offers dynamic, geo, proximity and least-outstanding-requests steering, and every one of them hands the routing decision to a latency measurement. Off does something better. It uses the pool list in order, and the first healthy pool takes everything. The order of that list is the switch.

The Host Header

Each endpoint in a pool carries a host header, and what you put there decides the entire character of the migration.

Set it to the origin's own name and you get override mode. The visitor types service.chrishouse.io, the backend receives service-east.chrishouse.io. It works, and it is the easy option, because the backend needs to know nothing.

The trouble is what a backend does with the hostname it thinks it has. A 302, a cookie Domain attribute, an OIDC redirect_uri, a link in a notification email. All of them get built from the name the app believes it is serving. In override mode an app will happily hand your users the internal hostname, which is the name you were trying to retire in the first place.

Preserve mode is the other option. Put the public name in the host header on every pool, and the backend sees service.chrishouse.io no matter which side is serving. It never learns it moved.

That costs something. Every backend has to actually answer to the final hostname before you can point the final hostname at it, and how much work that is depends on how the backend is reached.

A backend exposed as a public HTTPS origin needs two things per app: a routing rule for the hostname, and that hostname on the certificate. On Kubernetes that is a host: rule on the Ingress plus the name on the cert-manager SAN list, or hostnames: on the HTTPRoute if you are on Gateway API, where the parent Gateway's listener also has to permit it.

A backend behind a Cloudflare tunnel only needs the routing half. Cloudflare terminates TLS at the edge, the tunnel is encrypted, and cloudflared talks plain HTTP to the service inside the cluster, so there is no public certificate to issue for that hostname at all. Cert-manager drops out of the critical path entirely, which is a smaller change than it first looks like.

What you do need is a rule on the tunnel:

service.chrishouse.io       -> http://east:80
service-east.chrishouse.io  -> http://east:80
(any host)                  -> http_status:404

Worth knowing: the Zero Trust dashboard will not let you add a rule without a hostname, because in that UI a route is a published hostname and it creates a DNS record to match. The catch-all slot is already occupied by the terminating http_status:404 that cloudflared requires. Extra rules go in through the API. I would use the API anyway. Naming the hostname explicitly is the move you will make on every real app, so the rehearsal may as well practise it rather than take a wildcard shortcut.

The proof is one field. Here is the origin describing itself, through the load balancer, in preserve mode:

{
  "region": "west",
  "stack": "new",
  "release": "v2",
  "hostSeen": "service.chrishouse.io"
}

hostSeen is $host straight out of nginx, whatever the container was actually asked for. When that reads as the public name, the abstraction is real. When it reads as the origin's own name, you have a leak waiting to surface in somebody's password reset email.

The Cutover

With steering off, promoting a side means reordering a list. Having that as a script rather than a click path is what makes it something other people can run:

await cf("PATCH", `/zones/${zone.id}/load_balancers/${lb.id}`, {
  default_pools: [idOf[target], idOf[standby]],
  fallback_pool: idOf[target],
})

$ node cutover.js west
cut over: east -> west   (fallback now west)

$ node cutover.js
live:     west
order:    west -> east
fallback: west

Under eight seconds from command to the new side serving, and node cutover.js east puts it back just as quickly. No DNS record changes, so there is no TTL to wait out and no client cache holding a stale answer. The decision happens at Cloudflare's edge on every request, which is the entire argument for a load balancer over swapping a CNAME.

Note the fallback pool moving along with the primary. The fallback is where traffic goes when everything reads unhealthy, and by far the likeliest cause of that is a monitor you misconfigured, not two independently dead stacks. It should always point at whichever side you currently trust most, which means it changes when the primary changes. Doing that by hand is a step people forget until an incident reminds them.

A script in a runbook still assumes somebody finds the runbook, holds an API token, and types the right argument at 2am. The version I actually want is this wrapped as a Day-2 action in Port, sitting on the service entity next to everything else you can do to it. The input is a dropdown with two values. The backend runs the same cutover.js from a workflow, so what the portal triggers is the same thing I would run by hand. What the portal adds is not the logic, it is the guardrails: who is allowed to press it, and what gets written down when they do. Moving production between two sets of infrastructure is exactly the kind of operation that should leave a record of who did it and when, and a terminal on my laptop leaves none.

When the Hostname Already Exists

The demo had it easy. I invented service.chrishouse.io and made it a load balancer from nothing. The real migration does not get that. The hostname already exists, already has a DNS record, and already has users on it right now. A Cloudflare load balancer owns a hostname, so the record and the load balancer cannot both have it.

The way through is to make the takeover a no-op. Point the load balancer's first pool at exactly what the DNS record already points at. Same origin, same host header, same response. The hostname changes what kind of object it is, and nothing else about the request path moves at all. Nothing has migrated yet. All you have changed is who decides where traffic goes, which is the thing you need in place before you can migrate anything.

How risky that moment is comes down to one question: is the hostname proxied today?

If it is already orange-clouded, this is close to free. Clients resolve to Cloudflare anycast addresses now and will resolve to Cloudflare anycast addresses afterwards. Nothing a client can observe changes, and there is no TTL to wait out. If it is grey-clouded and pointing straight at an ingress IP, clients would move from your address to Cloudflare's, which is full propagation exposure with a slow rollback. That case deserves its own change on its own timeline: lower the TTL, orange-cloud it, let it settle for a week, and only then treat the load balancer as a separate step.

The sequence:

Save exactly what the record is today. Type, content, proxied flag, TTL. That is your rollback and there is no other copy of it.
Get an origin address for legacy. If the record is a CNAME, its target already works. If it is an A record, the IP does. A DNS-only alias like myapp-legacy keeps the pool config readable and lets you repoint legacy later without touching the pool.
Build the pool in preserve mode from the start, with the public hostname in the host header. This is the good part. Legacy already answers to that name and its certificate already covers it, so legacy needs no changes whatsoever. Every bit of work lands on the new cluster.
Create the pool and monitor with no load balancer attached, and confirm the pool reads healthy. Pools are account-level objects, so you can validate the health check before anything is load-bearing.
Rehearse on a parallel hostname. Pools are reusable across load balancers, so a throwaway name gives you a full dress rehearsal against the real origins.
Create the real load balancer with only the legacy pool in it.
Leave it alone for days. Nothing has migrated.
Then put the public hostname on the new cluster's routing rules, and on its certificate too if it is reached as a public origin rather than through a tunnel. Add it as a second pool on standby, and reorder when you are ready.

Steps 1 through 5 change nothing. Step 8 reverses in seconds. Step 6 is the only one-way door, and only in the sense that undoing it means recreating a DNS record by hand instead of reordering a list. That is what step 1 is for.

Design Decisions

Why failover order instead of weights? Because a weighted ramp is a promise about equivalence I could not honestly make for the real migration. Two backends only blend if they share state. Failover order gives me one live side and one standby, which is what I actually wanted.

Why tunnels instead of public origins? Both origins are containers on a desktop in my house. A tunnel means no open ports, no static IP, and an origin address that only Cloudflare can route to. It also matters that the endpoint has to be the <uuid>.cfargotunnel.com hostname directly. A friendlier CNAME pointing at it is explicitly unsupported, which is a fun twenty minutes if you assume otherwise.

Why one monitor for both pools? A monitor defines how to check. The address and host header come from each endpoint, so one monitor means both sides are judged by an identical standard, which is the only way a comparison between them means anything. It also halves the health-check volume you are billed for.

Why adaptive routing on? Failover across pools lets a request move to a healthy pool immediately instead of waiting for the monitor to notice. With a 60 second check interval, that is the difference between a minute of errors and none.

Why not just change DNS? Because rollback is the whole point. A DNS change propagates at the mercy of every resolver and browser between you and your users, so the undo button takes as long as the button did. Edge-side steering reverses in seconds, and that is what lets you attempt a cutover during business hours instead of at midnight.

Why not put the routing in nginx? spydergsx, a former colleague I still use as a sounding board, put this better than the rest of the post does: "you don't want routing baked into a config file, or your rollback becomes a config edit and reload instead of an 8-second API call." Rolling back at the edge is the same single API call the change was. Rolling back a config file is an edit, a commit, a pipeline and a reload, on every box holding a copy of it.

Two Things Worth Knowing Before You Try It

Failover is not cutover. In active-passive, when the primary pool recovers, traffic automatically returns to it. I watched this happen live. East went unhealthy, west took over, east recovered, and traffic moved back on its own without me touching anything. Which means if you migrate by making the old side unhealthy, you have not migrated. You have taken a temporary detour that ends the moment somebody fixes the old health check. Reordering the pools is the only durable move.

Change one side at a time. The host header is used by the health monitor too, so changing it on both pools at once takes both pools unhealthy simultaneously, and failover has nowhere to go when the failure is correlated. Adaptive routing, the fallback pool, none of it saves you. Change the standby side, verify it, then cut over. Do it in that order and the safety mechanisms you are paying for can actually engage.

Conclusion

The shape is small enough to hold in your head. One permanent public name at the edge, an ordered list of pools underneath it, and backends that answer to the public name rather than to their own. Cutting over is a list reorder. Rolling back is the same reorder backwards. The internal hostname never reaches a user's address bar or their bookmarks.

The rehearsal cost $5 and an evening, and it has already changed the real plan three times over. The new infrastructure has to answer to the final hostname before anything gets pointed at it. The two sides get migrated one at a time. And the load balancer goes live with only the legacy pool in it, so the day the hostname changes hands is a day when nothing else happens. All three are obvious in hindsight and none of them were in the original plan.

If you have a migration coming where a hostname needs to outlive the infrastructure under it, build the two-origin version first on a domain nobody depends on. The failure modes are identical and considerably cheaper.

A Real Drum for a Browser Rhythm Game (ESP32 + Web Serial)

Tue, 11 Aug 2026 00:00:00 GMT

There is a small game on this site called Rhythm Hands. It is a learning toy for my kid: notes scroll across a staff, a yellow playhead sweeps over them, and you tap in time. Blue notes are the left hand (spacebar), red notes are the right (enter), and there are touch pads for a tablet. It teaches quarter notes and ti-ti eighths without ever showing a child a settings screen.

Tapping a spacebar is fine. Hitting an actual drum is better. So this post gives the game a physical input: a piezo sensor glued to a drum surface, an ESP32 reading the knock, and the hit landing in the browser as if you'd pressed the key. No app rewrite, no cloud, no server. Field report, mistakes left in, because the mistakes are the useful part.

A quick demo of my Rhythm Hands project. Watch on YouTube.

The Parts

An ESP-WROOM-32 DevKit V1 (the cheap dual-core board with the CP2102 USB bridge). I also had a XIAO ESP32C3 on the bench, but the C3 has no GPIO34/35, and I'd already wired the sensor to 34, so the WROOM won.
A HiLetgo piezoelectric vibration sensor module — a brass piezo disk on a little board, three pins: S (signal), +, -. Passive-ish: a knock produces a voltage spike on S.
A USB cable, and a laptop running Chrome.

That is the entire bill of materials. The interesting decisions are all in software.

Why Web Serial, Not WiFi

The ESP32 can talk WiFi, so the obvious-looking design is: firmware POSTs hit events to a little local server, or opens a WebSocket to the browser. I didn't do that, and I'd argue you shouldn't either for this.

A drum hit is a latency-critical, false-trigger-sensitive event. Every hop you add — router, DHCP lease, a server process to babysit, mDNS name resolution, a socket that silently drops after the laptop sleeps — is a hop that adds jitter or a 3am "it just stopped working." For a thing a child whacks, "plug the USB in and it works" beats "make sure the ESP32 joined the 2.4GHz SSID and the server is running."

Web Serial (navigator.serial, Chrome/Edge) reads the USB serial port straight from the page. Sub-millisecond transport, no network, no server, and the debounce logic lives on the microcontroller where it belongs. The one cost is a user gesture — the browser makes you click a button and pick the port — which is a feature, not a bug, for anything touching hardware.

So the contract is dead simple. The firmware prints a line per hit:

L:1873

and the page reads L and fires a blue hit. That's the whole protocol.

Wiring

HiLetgo piezo module      ESP32 DevKit V1
  S  (signal)  ---------> GPIO34   (ADC1, input-only)
  -  (GND)     ---------> GND
  +  (VCC)     ---------> 3V3

GPIO34 is one of the ESP32's input-only pins (GPIO34–39). They have no internal pull resistors and no output drivers, which is exactly what you want for an analog sensor — nothing fighting the piezo. Do not use an ADC2 pin (0, 2, 4, 12–15, 25–27) for this; ADC2 is unavailable whenever WiFi is active and behaves strangely even when it isn't.

One honest note I learned the hard way below: on this module, + to 3V3 matters. Skip it and the signal pin sits dead flat.

The Firmware

The whole job is: read the analog pin, decide a knock happened, and don't let one whack count as five. A struck piezo rings — one hit is a decaying burst of oscillation, not a single clean spike — so there are two pieces of debounce:

A retrigger lockout: after an accepted hit, ignore the pin for 80ms so the ring counts once.
A short peak scan: once we cross threshold, follow the signal for ~12ms to report its true peak. That number is only for tuning; the game ignores it.

struct DrumPad {
  const char *id;          // token the game reads: "L" (blue) or "R" (red)
  uint8_t pin;             // ADC-capable GPIO the sensor's signal is on
  int threshold;           // raw 12-bit value (0-4095) a hit must exceed
  unsigned long lastHitAt; // millis() of last accepted hit (debounce)
};

// Only the connected pad is listed so an unconnected input-only pin
// can't pick up noise and fire phantom hits.
DrumPad pads[] = {
  { "L", 34, 350, 0 },
  // { "R", 35, 350, 0 },   // uncomment once a red piezo is on GPIO35
};

const int padCount = sizeof(pads) / sizeof(pads[0]);
const unsigned long retriggerMs = 80;   // min gap between hits on one pad
const unsigned long peakScanMs  = 12;   // how long to track the peak

void setup() {
  Serial.begin(115200);
  analogReadResolution(12);              // 0-4095
  for (int i = 0; i < padCount; i++) {
    pinMode(pads[i].pin, INPUT);
    analogSetPinAttenuation(pads[i].pin, ADC_11db);  // full ~0-3.3V range
  }
}

void loop() {
  const unsigned long now = millis();
  for (int i = 0; i < padCount; i++) {
    DrumPad &pad = pads[i];
    const int value = analogRead(pad.pin);

    // below threshold, or still inside the post-hit lockout -> ignore
    if (value <= pad.threshold || now - pad.lastHitAt < retriggerMs) continue;

    // follow the spike briefly to capture its true peak
    int peak = value;
    const unsigned long started = millis();
    while (millis() - started < peakScanMs) {
      const int sample = analogRead(pad.pin);
      if (sample > peak) peak = sample;
      delayMicroseconds(300);
    }

    if (peak > pad.threshold) {
      Serial.print(pad.id);
      Serial.print(':');
      Serial.println(peak);          // e.g. "L:1873\n"
      pad.lastHitAt = millis();
    }
  }
  delay(1);
}

Notice the second pad is commented out. I'm running one sensor for now, and an unconnected input-only GPIO floats — leave GPIO35 in the scan list with nothing on it and it'll happily invent red hits from electrical noise. List only what's wired. When the second disk goes on, uncomment one line and reflash.

I flashed it with the arduino-cli that ships inside Arduino IDE 2.x, so there was nothing new to install:

CLI="/c/Users/you/AppData/Local/Programs/Arduino IDE/resources/app/lib/backend/resources/arduino-cli.exe"
"$CLI" compile --fqbn esp32:esp32:esp32 rhythm-hands-esp32-drum-pads
"$CLI" upload -p COM6 --fqbn esp32:esp32:esp32 rhythm-hands-esp32-drum-pads

The Receiver, in the Browser

This is the part I like: the game already speaks keypress and pointer events, so I didn't touch the game logic at all. Web Serial just calls the same press("L") the spacebar does.

Opening the port is a button click, at 115200 baud to match the sketch:

const connectSerial = useCallback(async () => {
  if (typeof navigator === "undefined" || !("serial" in navigator)) {
    setSerial({ status: "unsupported", message: "Use Chrome or Edge for ESP32 USB" })
    return
  }
  const port = await navigator.serial.requestPort()   // the user gesture
  await port.open({ baudRate: 115200 })
  serialPortRef.current = port
  serialStopRef.current = false
  setSerial({ status: "connected", message: "ESP32 live" })
  readSerial(port)
}, [readSerial])

The read loop decodes bytes and splits on newlines, buffering any partial line for next time — serial gives you a byte stream, not tidy messages, so you have to reassemble lines yourself:

const readSerial = useCallback(async (port) => {
  const decoder = new TextDecoder()
  while (port.readable && !serialStopRef.current) {
    const reader = port.readable.getReader()
    try {
      while (!serialStopRef.current) {
        const { value, done } = await reader.read()
        if (done) break
        serialBufferRef.current += decoder.decode(value, { stream: true })
        const lines = serialBufferRef.current.split(/\r?\n/)
        serialBufferRef.current = lines.pop() || ""   // keep the partial line
        lines.forEach(handleSerialLine)
      }
    } finally {
      reader.releaseLock()
    }
  }
}, [handleSerialLine])

And the parser is deliberately forgiving. It takes the first token off the line and maps a small vocabulary to a hand, so L, L:1873, LEFT, BLUE, 1 all mean the same thing. That way I can change the firmware's mind about wording later without touching the browser:

const handleSerialLine = useCallback((rawLine) => {
  const line = rawLine.trim().toUpperCase()
  if (!line) return
  const [token] = line.split(/[\s,:;=]+/).filter(Boolean)
  const hand =
    token === "L" || token === "LEFT" || token === "BLUE" || token === "B" || token === "1" ? "L"
    : token === "R" || token === "RIGHT" || token === "RED" || token === "2" ? "R"
    : null
  if (hand) press(hand)   // exactly what the spacebar calls
}, [press])

press("L") is the game's existing input path — it flashes the pad, finds the nearest unplayed note within the timing window, grades it perfect/great/ok, updates the combo. The drum is now indistinguishable from the keyboard as far as the game is concerned. That's the goal: the transport is a detail, the game doesn't know or care.

Two Bugs That Ate The Afternoon

It did not work on the first try. It never does. Both failures were the honest kind — the ones where the tool is telling you the truth and you're misreading it.

1. "Wrong boot mode detected (0x13)"

First upload:

A fatal error occurred: Failed to connect to ESP32: Wrong boot mode
detected (0x13)! The chip needs to be in download mode.

These cheap DevKit clones often have a flaky auto-reset circuit (the little cap-and-transistor network that lets esptool drop the chip into the bootloader over DTR/RTS). When it doesn't work, you do it by hand. The trick that finally stuck: hold the BOOT button down through the entire Connecting...... phase, not the tap-EN-and-release dance. Holding IO0 low the whole time forces download mode even while esptool pulses EN. Once you see Writing at 0x... with a percentage, let go. Every reflash since has been the same little ritual.

2. The sensor that read a perfect, dead zero

This one was embarrassing and worth it. After flashing, I listened on the port and tapped. Nothing. Zero lines. I lowered the threshold — still nothing. I started doubting the wiring.

So I flashed a throwaway diagnostic sketch that just streams the raw ADC 20 times a second, and watched the number:

void loop() {
  int peak = 0; unsigned long t = millis();
  while (millis() - t < 50) { int s = analogRead(34); if (s > peak) peak = s; }
  Serial.println(peak);
}

Baseline: 0. Tapping: 0. A flat zero — not drifting noise, an actual grounded-looking zero. That is not "no signal reaching the pin," that's "the pin is being held at 0V." I went hunting for a short... and then realized the actual bug was two-layered:

I hadn't connected + to 3V3. This particular module needs power; without it the signal output just sits at rail-bottom.
And on the run where I thought I was proving it dead, I wasn't actually tapping during the capture window. A passive piezo idles at 0. A flat zero from an untouched sensor is the correct, expected reading. I had built a test that couldn't tell "broken" apart from "you didn't hit it."

Powered it, tapped for real, and the numbers told the whole story at once:

signal: 554
signal: 4095
signal: 733
signal: 2816
...
PEAK reading over 20s: 4095
non-zero(>20) samples: 303

Idle floor of 0, firm hits railing the ADC at 4095, light taps landing in the few-hundreds. That's a gorgeous signal — and it also handed me the threshold for free. My initial guess of 550 was needlessly deaf; with a noise floor of literally zero there's enormous headroom, so I dropped it to 350 to catch lighter taps and never looked back. Reflashed the real sketch, listened again, and got clean L:871, L:1982, L:4095 lines pouring out with every hit.

The lesson I keep re-learning: when a sensor reads nothing, build the test that shows you the raw value before you start rewiring. Half the time the sensor is fine and your test is lying to you.

Where It Landed

Load the game in Chrome, click CONNECT ESP32, pick the port, and whack the drum. It fires the blue pad exactly like the spacebar — same timing window, same combo counter, same little green PERFECT.

It's blue-only for the moment, which covers the early one-hand levels. The two-handed levels want a second disk on GPIO35 and one uncommented line, and because the browser's parser already understands R, that's a firmware-only change. The nicer future version drops the hands entirely — one drum that just means "I hit now," letting the game pick whichever note is nearest. But that's a different post. This one just needed to make a real drum play a browser game, and it does.

A Voice Assistant on My Desk That Talks to My Local LLM (Part 5)

Sun, 26 Jul 2026 00:00:00 GMT

By the end of Part 4, the site had a mouth from Part 3 (a local Qwen on the desktop under my desk) and a hippocampus (Redis memory that remembers you between visits). It still lived entirely inside a browser tab. This part gives it a body: a small round screen on my actual desk that listens when I tap it, ships my voice off to be transcribed, asks the same local Qwen the website uses, and reads the answer back out loud.

The hardware is a Waveshare ESP32-S3-Touch-AMOLED-1.75. It is a lovely little board with a genuinely thin paper trail once you get past "blink the screen." If you googled that board name and landed here, this is the post I wanted when I started. Fair warning, same as the rest of the series: this is a field report, written while the mistakes are still warm.

The Goal

Three things, because that format keeps working:

A physical object on the desk that answers questions using my LLM, running on my hardware, not a cloud assistant renting me my own voice back.
Have it respect the same backend flag from Part 3. When I flip the GrowthBook toggle between my house box and my friend's box, the desk gadget follows, no reflash.
Do the speech-to-text and text-to-speech without standing up yet another server. The thinking stays local. The transcription and the voice can be somebody else's free compute.

House rules from the rest of the series still hold: nothing on my network exposed without a token, and the part that reasons about my questions runs on my own GPU.

The Board, Honestly

Here is what is actually on it:

A 466x466 round AMOLED, driven by a CO5300 controller over a 4-lane QSPI bus.
A CST9217 capacitive touch panel on I2C.
An ESP32-S3 with 16MB flash and 8MB PSRAM, which matters, because you render a full-frame canvas in PSRAM.
And the part that cost me a full day: the audio path is two separate chips, and almost every tutorial treats it like one.

That last point is the whole reason this post has a hardware section. The display and touch are well-trodden. The audio is where people (me) lose a day.

The Pin Map

Every pin, so you can skip the datasheet archaeology. These are the values I run in config.h:

Display (CO5300, QSPI)
  SCLK 38   D0 4   D1 5   D2 6   D3 7   CS 12   RST 39
  466 x 466, column offset 6

Touch (CST9217, I2C @ 0x5A)
  SDA 15   SCL 14   INT 11   RST 40

Audio I2S (shared by both codecs)
  MCLK 42   BCLK 9   WS/LRCLK 45   DOUT 8 (to speaker)   DIN 10 (from mic)
  PA enable 46  (NS4150 amplifier)

Codecs on the same I2C bus as touch (SDA 15 / SCL 14)
  ES8311  @ 0x18   output only  (DAC -> NS4150 -> speaker)
  ES7210  @ 0x40   input only   (dual mics -> ADC -> I2S DIN)

The I2C bus is shared three ways: touch, the output codec, and the input codec. The I2S bus is shared two ways: the ESP is the clock master, and both codecs are slaves hanging off the same MCLK, BCLK, and word-select lines.

The Two Chips Nobody Draws

This is the single fact that would have saved me the most time, so it gets its own heading.

The ES8311 is a mono codec, and on this board it is wired for output. It takes I2S data on GPIO8, converts it to analog, and hands it to an NS4150 class-D amplifier that drives the speaker. The ES8311 has a microphone input on paper. On this board it is not the microphone. Do not spend an afternoon configuring its ADC. I did.

The ES7210 is a separate 4-channel audio ADC (Everest Semiconductor datasheet), and it is the microphone. The board's two mics feed the ES7210, which serializes them onto the I2S bus and drives GPIO10, the ESP's I2S data-in pin. It is its own I2C device at address 0x40, with its own register map, entirely independent of the ES8311.

So the mental model is: GPIO8 goes down to the speaker chip, GPIO10 comes up from the microphone chip, and they are different silicon. The ESP32 generates one set of clocks (MCLK 42, BCLK 9, WS 45) and both chips slave to them. Once that clicked, everything else was downhill.

The Architecture

  Tap the orb
     |
  ES7210 mic ADC --I2S DIN(10)--> ESP32-S3
     |                               |
     |                        build a 16 kHz WAV in PSRAM
     |                               |
     |                     HTTPS POST /api/assistant
     v                               v
  ES8311 <--I2S DOUT(8)--    Cloudflare Pages Function
  speaker                          |   Workers AI Whisper  (speech -> text)
     ^                             |   local Qwen via the Part 3 tunnel  (the answer)
     |                             |   Workers AI aura-1   (text -> speech, 16 kHz WAV)
     +--------- audio/wav ---------+

Whisper and aura-1 run on Cloudflare Workers AI, which has a free daily allowance and needs no box of mine. The reasoning step in the middle, the part that actually answers the question, hits the same local Qwen the website uses. The GrowthBook flag from Part 3 decides house or fever, and the device inherits that decision for free because it calls through the same function the browser does.

Doing It Wrong First

My first recordings were pure silence. Not quiet audio. Digital zero, every sample. Whisper, handed four seconds of nothing, does the polite thing and hallucinates: it returned "you", and once, memorably, "have a great time and let me know when you are back."

The debugging went in stages, each one a small archaeology dig:

I configured the ES8311's microphone. Wrong chip. See above.
I found the ES7210, ported a driver, and still read zeros. The chip answered on I2C at 0x40, so it was alive. It just was not producing audio.
I put the raw microphone peak into the device's telemetry, because opening the USB serial port resets the ESP32-S3 and I kept missing the boot logs. Watching the peak sit at exactly 0 over the network told me the ADC was clocked wrong, not wired wrong.

The actual bug was one byte. The ES7210 has a MAINCLK register (0x02) that sets the ADC clock divider, and for 16 kHz audio with the master clock this board runs (16000 times 256, which is 4.096 MHz), that register has to enable the clock doubler and the DLL on top of the divider. The correct value is 0x01 | (1 << 6) | (1 << 7), which is 0xC1. I had written 0x01. With the wrong divider the ADC ran unclocked and produced flat zeros, and, being an ADC, it reported no error about it. Silence is a valid output.

The fix came from the board's own repository. Waveshare ships an esp-idf/05_Spec_Analyzer example that reads the microphone for an FFT display, and it delegates to Espressif's esp-bsp ES7210 driver. That driver has a clock coefficient table, and the row for 4.096 MHz at 16 kHz spells out every divider bit. Porting that table's values verbatim turned the peak from 0 to a live, breathing noise floor on the first flash. If you are fighting this chip, the driver is the source of truth, and the coefficient table is the part to copy exactly.

One more from the same fight: do not hand-gate the clock-off register (0x01) trying to be clever about power. The esp-bsp sequence leaves it alone. So should you.

The Two-Core Rule

The ESP32-S3 has two cores, and on a display device you must respect the division of labor or the screen stutters. I render the full round canvas out of PSRAM at roughly 15 frames a second on core 1. Every blocking network call, and TLS is very blocking, lives on core 0 behind a mutex-guarded snapshot. The first time I put an HTTPS poll inside the render loop, the display froze solid for the length of the request, and it was obvious within one visit that the draw loop and the radio could not share a thread.

The voice recording earns its own task for a related reason. When you tap the orb, the "listening" cue appears instantly on core 1, but if the actual capture is queued behind a stack of network polls it can start seconds late, after you have already finished talking. Giving the recorder a dedicated task drops the tap-to-capture gap to about 60 milliseconds, so the moment the screen says listening, the microphone is genuinely open. Half my early transcription failures were not audio quality at all. They were timing, my voice landing in the silence before the window opened.

Making It Talk Back

Getting audio out was its own small puzzle. Cloudflare's aura-1 text-to-speech returns MP3 by default, and the ESP32 has no cheap MP3 decoder I wanted to carry. The trick is that aura-1 accepts format parameters: ask for encoding: linear16, container: wav, and sample_rate: 16000, and it hands back a plain PCM WAV at exactly the rate the ES8311 already plays. The device streams that WAV straight off the wire into the speaker with no decoding step.

I return it from the Pages Function as a binary audio/wav body with the transcript and answer riding in response headers, rather than base64 inside JSON, which would have tripled the payload for no reason. On the device, the reply gets read into PSRAM and clocked out to the codec.

The last snag was comedy: the first spoken answers sounded like a walkie-talkie. Full-scale synthesized speech was overdriving the tiny speaker into a rasp. Scaling the samples to about 60 percent gave the amplifier headroom and the voice cleaned right up. The synthesized ping and whoosh sounds had always been fine because they were generated quieter, which is exactly the clue I ignored for an hour.

Design Decisions

Why speech-to-text and text-to-speech live on Workers AI, not the house box. The local GPU has one job worth protecting: running the model that answers questions. Whisper and aura on a free tier keep that card free for the model people actually talk to, and the transcription and the voice are commodity work. The reasoning stays home.

Why the device respects the flag instead of pinning a backend. Same answer as Part 3. One toggle should move everything. The desk gadget calling through the same function as the browser means it can never disagree with the website about which LLM is live.

Why a tap and not a wake word. An always-listening microphone on my desk is a thing I do not want, and a tap is a clean act of consent. The orb turns to listening only when I ask it to.

Why the microphone peak lives in telemetry. Because opening the native USB serial port resets this board, and I kept missing the one boot log I needed. A number in the heartbeat I could watch over the network turned a guessing game into a measurement. That single move is what actually cracked the silence bug.

The Gotchas, So You Skip Them

Board-specific, hard-won, in the order they bit me:

The microphone is the ES7210, a different chip from the ES8311. The speaker codec cannot record on this board. Configure the ADC, not the DAC.
The MAINCLK register must match your master clock. For 16 kHz at 4.096 MHz MCLK, register 0x02 is 0xC1, not 0x01. Wrong divider means clean, error-free silence.
Copy the esp-bsp ES7210 coefficient table verbatim. The board's 05_Spec_Analyzer example is the working reference. Do not improvise the clock bits.
MCLK has to be running before the ES7210 will take its configuration. Bring the I2S clocks up first, then talk to the chip.
Mute the amplifier (PA on GPIO46) while recording. The idle class-D amp couples hiss straight into the ADC and buries your voice under a wall of noise the size of your signal.
Average the two microphone channels, do not sum them. Summing clips on loud input, and clipped audio makes Whisper hallucinate confident nonsense.
DIN is GPIO10, DOUT is GPIO8, and they land on different chips. Cross them in your head and you will chase silence on one end and confusion on the other.
Keep all networking off the render core. Any blocking call in the draw loop freezes the screen for its full duration.

Conclusion

Part 3 gave the site a mouth, Part 4 gave it a memory, and this part gives it a body sitting on my desk. It listens on a tap, transcribes on somebody else's free GPU, asks the same local Qwen the website asks, and speaks the answer back through a speaker the size of a coin, for a running cost of zero dollars a month plus one ESP32.

The honest status: the voice loop works end to end, the reasoning is genuinely mine, and the hardware surprises were all in the audio path that the docs wave past. If you have this exact board and a weekend, the pin map and the one clock byte above are the two things that would have given me back my lost day. The rest is the fun part.

Giving the Site a Memory with Redis Agent Memory Server (Part 4)

Wed, 15 Jul 2026 00:00:00 GMT

By the end of Part 3, the site had its own LLM running on the desktop under my desk, reachable through a Cloudflare Tunnel, with a feature flag to fall back to my friend's box. One problem remained: every conversation started from zero. You could tell the chat your name, your favorite database, your deepest fears about Ingress controllers, and thirty seconds later it had the memory of a goldfish with a busy schedule.

Then I found agent-memory-server, an open source memory service from the Redis team. This post is how I wired it into the site's chats, running entirely on my own hardware, and the two or three ways I got it wrong before I got it right. Fair warning up front: I am maybe a week into using this thing. Consider this a field report from early territory, written down while the mistakes are still fresh.

The Goal

Three things again, because that format keeps working:

Give the site's AI features durable memory. Tell the chat something once, phrase it as "remember that ...", and have it recalled days later, in a different conversation, with no login and no account.
Run the whole memory stack on the house box. Redis, the memory service, and every model call it makes should hit my Ollama, with no OpenAI key anywhere in the chain.
The real endgame: first watch how it learns about a visitor, then flip that around and use it to help the visitor learn faster. The k8s exam prep tool covers KCNA, CKA, and CKAD, and a coach that actually remembers which domains you keep bombing is a better coach than one that meets you fresh every visit.

Same house rules as the rest of the series: no keys in the browser, and nothing on my network exposed without a token.

The Building Blocks

agent-memory-server gives an AI agent two kinds of memory. Working memory is the conversation itself, per session: you PUT the messages, it holds them with a TTL. Long-term memory is the interesting part. A background process reads the working memory, uses an LLM to extract durable facts ("user prefers dark mode", "user lives in Memphis"), deduplicates them, embeds them, and indexes them for semantic search. You never write long-term memories by hand. You hand it conversations and it does the remembering.

Redis is the storage under all of it: the sessions, the extracted memories, and the vector index the semantic search runs on. I am running the redis-stack image in Docker next to the memory server.

Ollama, still the workhorse from Part 3. The memory server needs two models of its own: an embedding model for the vector search and a generation model for the fact extraction. Both point at the same Ollama that serves the chat, which means the entire memory system runs without a single external API call.

The whole stack is one Docker Compose file:

services:
  redis-memory:
    image: redis/redis-stack-server:latest
  agent-memory:
    image: redislabs/agent-memory-server:latest
    environment:
      REDIS_URL: redis://redis-memory:6379
      OLLAMA_API_BASE: http://host.docker.internal:11434
      GENERATION_MODEL: ollama/qwen2.5:3b
      EMBEDDING_MODEL: ollama/nomic-embed-text

The Architecture

Browser (Neon GPT, Station Intelligence, exam coach)
  │  POST /api/llm or /api/ask   { messages, memory: { userId, sessionId } }
  ▼
Cloudflare Pages Function
  │  recall:  POST /v1/memory/prompt      (before generating)
  │  store:   PUT  /v1/working-memory/…   (after answering, in the background)
  ▼
memory.chrishouse.io   (Cloudflare Tunnel + Access service token)
  ▼
agent-memory-server ──► Redis (sessions, facts, vectors)
        │
        └──► Ollama (extraction + embeddings, same box as the chat model)

The tunnel and Access setup is a straight copy of Part 3: a second hostname on the same tunnel, path-restricted to the API, behind its own service token. Anyone hitting memory.chrishouse.io without the token gets a 403 from Cloudflare's edge before the request ever reaches my house.

Identity is an anonymous id the browser mints once and keeps in localStorage. No accounts, no emails, no names unless you volunteer one to the chat. The memory is scoped to that id, so your remembered facts are yours and the next visitor gets their own clean slate.

Doing It Wrong First

My first integration treated the memory server like a database. Every chat turn, my code wrote its own long-term records: one episodic record of the exchange, plus a "durable fact" record whenever someone said "remember that ...". For recall, it ran four separate searches (semantic, keyword, hybrid, and a topic-filtered one) and stitched the results into the prompt by hand.

It worked, in the sense that a car with square wheels technically moves. What I actually built was a duplicate factory. The server was extracting memories from the conversations on its own, like it is designed to, while my code wrote overlapping records next to them. Within a day of testing, the store contained "chris likes pineapple pizza" four times, a server-merged memory that said pepperoni instead, and an episodic transcript of me asking about pizza. The site held contradictory pizza beliefs simultaneously, which is very human of it, and also useless.

The fix was deleting code. The idiomatic pattern is embarrassingly small, so here it is as pseudo code instead of a wall of JavaScript:

store(turn):
    PUT working-memory(sessionId, messages)
    # that's it. the server extracts, dedups, and indexes in the background

recall(question):
    POST memory/prompt(query, sessionId, semantic search over this user)
    # returns the session context plus relevant long-term memories,
    # already formatted to drop into the system prompt

One write, one read. Half the GPU work, no duplicate records, and the server's own deduplication actually gets a chance to do its job. If you take one thing from this post: do not hand-write long-term memories. The extraction pipeline is the product.

The Extraction Model Matters More Than I Expected

The memory server's extraction step is an LLM call, and my first configuration pointed it at the model I already had loaded: qwen2.5-coder, the code-tuned model that answers the chat. It extracted exactly zero memories. Not bad memories. Zero. A code model asked to distill "remember that my keyboard is a HHKB and I live in Memphis" into facts just stares at you and thinks about brackets.

Swapping to a general instruct model fixed extraction immediately, and then created a new problem: the 7B version plus the 14B chat model plus the embedding model added up to more VRAM than the card has, and Ollama started playing musical chairs with 16GB of memory. At one point the chat model got stuck mid-eviction and the whole endpoint hung. The 3B instruct model extracts nearly as well, fits alongside everything else, and the musical chairs stopped.

The lesson generalizes: extraction quality is a function of the model doing the extracting, and it is a separate decision from your chat model. Small general instruct beats large code-tuned at this job by an infinite margin, because anything beats zero.

What It Powers Today

"Remember that ..." in the chats. Tell Neon GPT or Station Intelligence to remember something, come back tomorrow in a fresh conversation, and ask about it. The recall goes through the same memory prompt as everything else, so facts surface even when you ask sideways ("what keyboard do I use?" finds a memory that never contained those exact words).

The station recognizes regulars. When Station Intelligence boots and memory exists for your browser id, the greeting is composed from what it knows. Mine mentions blue fish, for reasons I have fully earned.

The exam coach. This is the part I care about. Finish a quiz or practice exam on k8s exam prep and the page logs a summary to memory: score, domains, where you were weak. The coach button recalls that history and prescribes what to drill today. Next to it, a "drill weak spots" button builds your next quiz directly from the questions you have missed and your lowest-mastery domains, and those results feed back into memory. The loop closes: study, get remembered, get coached, drill the gap, repeat.

That third one is the experiment I wanted to run all along. The site learning my pizza order is a party trick. The site remembering that I cannot tell a NetworkPolicy from a hole in the ground, and quietly building tomorrow's study session around that fact, is the thing that might actually be worth the Redis container.

Early Days, Honestly

A week in, here is what is rough:

Contradictions accumulate. Telling the chat "I no longer like pineapple pizza" adds a new memory next to the old one. The server has forget and compact endpoints built for exactly this, and I have not wired them up yet. Right now the model gets both facts and has to sort out the timeline itself.
One enrichment step fails on my Redis. The topic/entity tagging task uses a command that only exists in Redis 8, and the redis-stack image I deployed predates it. Core memory works fine (store, search, recall are all healthy), but every extraction logs an error for the metadata step. The fix is moving to the redis:8 image, which now has vector search built in.
Extraction is eventually consistent. Facts land seconds after the conversation, on a debounce, after a background model call. Say "remember X" and immediately ask about X in a new session and you can beat the pipeline. In practice nobody chats that way, but tests do.
My own bug was the best one. After the refactor, every "remember that ..." reported a failed write, because my handler still checked a response field the old code returned and the new code did not. The writes were succeeding the whole time. The site was gaslighting itself.

None of these have made me regret the setup. All of them are the kind of thing you only learn by running it.

Design Decisions

Why memory lives behind the Pages Function. The browser never talks to the memory server. The function resolves the identity, loads memory into the prompt, and stores the turn afterward in the background, so a slow memory call never delays the answer. If the memory server is down, everything fails open and the chat just answers without memory. The site loses its memory gracefully, like the rest of us.

Why the identity is anonymous. An id in localStorage costs nothing, needs no consent screen beyond what analytics already covers, and gives exactly the scoping memory needs. Accounts would make the memory better bound and the site worse.

Why extraction runs on a small model. Covered above, but as a decision: the extraction model is infrastructure, and infrastructure should be the smallest thing that does the job, especially when it shares a GPU with the model people are actually talking to.

Why I am writing this now instead of after a month. Because the early mistakes are the useful content. A month from now the duplicate factory and the code-model-extracts-nothing discovery would be smoothed over into "configure it correctly", and you would step on both rakes yourself.

The Gotchas, So You Skip Them

Do not write long-term memories by hand. PUT working memory and let the server extract. Hand-written records fight the deduplication and double your GPU load.
Use a general instruct model for extraction. Code models extract nothing. Check your VRAM budget: chat model + extraction model + embeddings all resident at once.
Session and user ids have a minimum length. Short test ids get silently replaced by fallbacks, and then your writes and reads disagree about identity. An hour of my life, gone.
Check the response shape after upgrades. The memory prompt endpoint returns message content as objects, and my code assumed strings. Fail-open code hides that kind of bug until you go looking.
Old Redis, new commands. If extraction logs unknown-command errors, your Redis predates what the memory server expects. Core features may still work, which makes it easy to miss.

Conclusion

The stack from Parts 1 through 3 gave the site a mouth. This part gives it a hippocampus, one Docker Compose file and a tunnel hostname away, running on the same GPU as everything else for the same monthly cost of zero dollars.

I am early on this, and the honest status is: durable memory works, recall is better than I expected, hygiene needs work, and the contradiction problem is real but survivable. The experiment that matters is the exam coach. If a site with memory can watch you study, notice what you get wrong, and shape what you practice next, that is a genuinely different thing from a chat widget with a good personality. Ask me in a month whether it made me faster at the CKA. The site will remember either way.

Running Your Own LLM Endpoint with Cloudflare Tunnels and Ollama (Part 3)

Sun, 12 Jul 2026 00:00:00 GMT

This is the third stop in a series that started with a friend's GPU. Part 1 pointed OpenCode at his local Qwen. Part 2 wired this blog's AI features to that same box through a Cloudflare AI Gateway, with Turnstile keeping bots off his GPU. It all worked great, right up until his machine went dark for a weekend and every AI feature on the site started returning shrugs.

His box can go down whenever it wants, and I do not get a vote. So I wanted my own copy of the model running on my own hardware, ready to take over when his machine naps. This post is how I did that with Ollama and a Cloudflare Tunnel, locked down so I am the only one who gets to spend my electricity.

The Goal

Three things, in plain terms:

Serve a Qwen model from the desktop under my desk, the one with the RTX 4080 Super in it.
Reach it from blog.chrishouse.io, which runs on Cloudflare's edge and has no idea what my home network looks like. No port forwarding, no static IP, no exposing my home address to the internet.
Flip between my box and my friend's box on demand, without a redeploy, and see instantly whether the one I picked is actually alive.

The whole thing had to keep the existing rule from Part 1: no keys in the browser, and nobody but my own site gets to use the GPU.

The Building Blocks

Ollama runs local models and hands you an OpenAI-compatible endpoint at http://localhost:11434/v1. That compatibility is the whole trick. From the blog's point of view a box under my desk looks identical to any other POST /v1/chat/completions provider, which is exactly what Part 1's proxy already speaks.

Cloudflare Tunnel (the cloudflared daemon) makes an outbound-only connection from my PC to Cloudflare's edge and publishes a local port at a public hostname. My home IP stays hidden, my router stays closed, and llm.chrishouse.io resolves to a machine that never accepted an inbound connection in its life.

Cloudflare Access sits in front of that hostname and rejects anyone without a valid service token. A public tunnel to an unauthenticated Ollama is a free GPU for the entire internet, so this part is not optional.

GrowthBook holds a single feature flag, llm-backend, whose value is house or fever. The blog reads it to decide which upstream to call. Same tool that already runs the flag demos on this site, now doing something I actually depend on.

The Architecture

Browser (Neon GPT)
  │  POST /api/llm   { messages, backend }
  ▼
Cloudflare Pages Function  (origin gate → session check → sanitize → pick backend)
  │
  ├── backend = "fever" ─► Cloudflare AI Gateway ─► LiteLLM ─► friend's Qwen   (Part 1)
  │
  └── backend = "house" ─► https://llm.chrishouse.io
                              │  CF-Access-Client-Id / CF-Access-Client-Secret
                              ▼
                          Cloudflare Access   (403 without the token)
                              │
                              ▼
                          cloudflared tunnel  (Windows service)
                              │  http host header rewritten to localhost:11434
                              ▼
                          Ollama ─► Qwen (my RTX 4080 Super)

The flag decides which branch runs. The proxy holds both sets of credentials server-side and picks one per request. The browser never learns either endpoint.

Standing Up Ollama

Installing Ollama and pulling a model is two commands, and I already had a few models on disk. The part worth writing down is which model to serve, because I got this wrong first.

Qwen3 is a reasoning model. Left alone it burns a few hundred tokens thinking before it says hello. Part 1 disabled that with chat_template_kwargs: { enable_thinking: false }, and my friend's LiteLLM honored it. Ollama's OpenAI-compatible endpoint does not. I tried every switch (think:false, the /no_think prompt token, the enable_thinking kwarg) and Ollama's /v1 route ignored all of them. The answer kept landing in a reasoning field with content sitting there empty, which the blog then rendered as a blank message.

So I served a model that does not think out loud:

ollama pull qwen2.5-coder:14b

At a Q4 quant it is about 9 GB, which sits entirely in the 4080 Super's 16 GB of VRAM with room to spare, and it answers in well under a second. Coder-tuned, but for a general chat widget it holds a conversation fine. A quick sanity check straight against Ollama:

curl -s http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen2.5-coder:14b","messages":[{"role":"user","content":"say hi"}],"stream":false}'

Real content, finish_reason: stop, no empty replies. Good enough to put behind a tunnel.

The Tunnel

cloudflared needs a login (a browser SSO handshake against your Cloudflare account), then a named tunnel and a DNS route:

cloudflared tunnel login
cloudflared tunnel create theblog-llm
cloudflared tunnel route dns theblog-llm llm.chrishouse.io

Then a config file. This is where the second gotcha lives. Recent Ollama refuses any request whose Host header is not localhost, a sensible anti-drive-by measure. Through the tunnel Ollama sees Host: llm.chrishouse.io and answers with a fast, confusing 403. The fix is one line that rewrites the host header back to what Ollama expects:

tunnel: theblog-llm
credentials-file: C:\Users\Chris\.cloudflared\<tunnel-uuid>.json
ingress:
  - hostname: llm.chrishouse.io
    service: http://localhost:11434
    originRequest:
      httpHostHeader: localhost:11434
  - service: http_status:404

cloudflared tunnel run theblog-llm and the hostname goes live. To survive reboots it runs as a Windows service. One warning there: a bare cloudflared service install registers the service with no run arguments, so it starts, prints its help text, and exits half a second later. I set an explicit command on the service so it runs my tunnel with my config:

"C:\Users\Chris\tools\cloudflared.exe" --config "C:\Users\Chris\.cloudflared\config.yml" tunnel run theblog-llm

Automatic startup, comes back after a reboot, done.

Locking It Down with Access

At this point llm.chrishouse.io is a public URL pointed at an LLM with no authentication. Anyone who finds it gets free inference on my GPU. Cloudflare Access closes that with a service token, which is a client ID and secret built for machine-to-machine calls.

I created the service token, a self-hosted Access application on the hostname, and a policy that allows only that token. The blog's proxy presents the token on every call to the house backend:

const headers = { "Content-Type": "application/json", Authorization: `Bearer ${key}` }
if (env.LLM_ACCESS_CLIENT_ID_HOUSE && env.LLM_ACCESS_CLIENT_SECRET_HOUSE) {
  headers["CF-Access-Client-Id"] = env.LLM_ACCESS_CLIENT_ID_HOUSE
  headers["CF-Access-Client-Secret"] = env.LLM_ACCESS_CLIENT_SECRET_HOUSE
}

The proof is two curls. Without the token, Access stops the request at the edge before it ever reaches my house:

$ curl -s -o /dev/null -w '%{http_code}\n' https://llm.chrishouse.io/v1/chat/completions -d '{}'
403

$ curl -s https://llm.chrishouse.io/v1/chat/completions \
    -H "CF-Access-Client-Id: <id>" -H "CF-Access-Client-Secret: <secret>" \
    -d '{"model":"qwen2.5-coder:14b","messages":[{"role":"user","content":"reply OK"}],"stream":false}'
{"choices":[{"message":{"content":"OK"}}], ...}

403 for the world, 200 for the blog. That is the whole security model in two lines of output.

GrowthBook Is the Toggle

The switch itself is a single GrowthBook flag named llm-backend, and its value is either house or fever. That flag is the whole point. Nothing in the browser decides which model you talk to, and I never touch an environment variable to change it. I flip the flag and the entire site follows.

A small endpoint reads the flag and writes it through the GrowthBook Admin API, so the choice is shared state instead of a setting stuck in one person's browser tab. Change it once and everyone hitting the site moves to the same backend at the same time. When my friend's box comes back online, one flip sends everyone back to it.

Behind the flag, the proxy still has to turn the word house or fever into a real endpoint with the right credentials. I kept the naming boring: the base environment variables are the fever backend from Part 1, and a parallel set with a _HOUSE suffix is my box. A tiny resolver maps the flag value to one of them:

resolve(flag):
    if flag is "house":            # my box
        endpoint = the *_HOUSE vars (local Qwen via the tunnel)
        auth     = Cloudflare Access service token
    else:                          # fever, the Part 1 path
        endpoint = the base vars (feverdreams gateway)
        auth     = Cloudflare AI Gateway token

    return endpoint, model, auth headers

Same two lines of config, two different locks. Fever carries the AI Gateway token from Part 1, house carries the Access service token, and the flag is the only thing that decides which one runs. Adding my box was a config change, and the Part 1 path never moved.

Where You Can Test It

Go poke at Neon GPT. Up in the header there is a House / Fever toggle sitting next to the connection dot.

House routes your message to the Qwen on my desk, through the tunnel, through Access.
Fever routes it to my friend's box, the Part 1 path.

Flip the toggle and two things happen. It writes the GrowthBook flag, then it fires a tiny health-check ping at the backend you picked and repaints the dot: green if that model answered, red if it did not. So if you switch to House and the dot goes red, my desktop is off, or Ollama crashed, or I am playing a game and told it to stop hogging the GPU. The chat degrades to a clear "that model is offline" message instead of hanging.

It is a real switch on a real thing. Be nice to my electric bill.

Design Decisions

Why a tunnel instead of port forwarding? Port forwarding means opening my router to the internet and handing out my home IP, then hoping nothing else on my network has a bad day. The tunnel is outbound-only, hides the IP, and gives me a real hostname with a real certificate for free. Less attack surface and less DNS busywork.

Why a service token instead of a login page? The only caller is a server, my Pages Function. Human login flows are the wrong shape for that. A service token is two secret headers, it lives in the Pages environment next to the other keys, and it never involves a browser. Access checks it at the edge, so a bad request dies in Cloudflare's datacenter and never touches my house.

Why a feature flag instead of just editing an env var? Because an env var edit is a redeploy, and redeploys are slow when a model is down and the chat is broken right now. The flag moves everyone in one API call, and I can flip it from my phone. It also means the fallback is a product decision I can make live, not a code path I have to ship.

Why a non-reasoning model? Because Ollama's /v1 endpoint would not let me turn Qwen3's thinking off, and a reasoning model that spends its whole token budget thinking returns an empty chat bubble. A coder-tuned 14B that fits fully in VRAM and answers immediately is the better trade for a chat toy, and it sidesteps the whole fight.

Why keep my friend's box as the default? His is a beefier 27B and it is usually up. Mine is the safety net for when it is not. Default to the better model, fall back to the reliable one, flip in a second when reality disagrees.

The Gotchas, So You Skip Them

A short list of things that ate my afternoon:

Ollama 403 through the tunnel. Host header. Rewrite it to localhost:11434 in the ingress config.
Empty chat replies. Reasoning model. Ollama's /v1 ignores every thinking switch, so serve a non-reasoning model or read the reasoning field as a fallback.
Service starts then dies. cloudflared service install leaves no run arguments. Set the service command explicitly.
A zombie worker. wrangler pages dev runs the real worker as a separate workerd process. A dev server I killed days ago left one holding the port, so every "fresh" restart was quietly serving stale environment variables and swearing my new config did not exist. Kill the orphaned workerd, confirm the port is actually free, then start over. I lost an hour to this one and I want that hour back.

Conclusion

The shape from Part 1 held up. An OpenAI-compatible upstream, a thin Pages Function for secrets and sanitizing, and now a second upstream that happens to be a computer in my house. Ollama makes the model boring, the tunnel makes it reachable without opening my network, Access makes it mine alone, and one GrowthBook flag turns "which model" into a switch instead of a deploy.

If you have a decent GPU sitting idle and a public site that wants an LLM, this is a weekend of work and zero dollars a month. Serve the model, tunnel it out, lock it behind a token, and give yourself a flag so the day your dependency goes dark, you are one toggle away from your own hardware picking up the slack.

Cloudflare AI Gateway and Turnstile: Fronting a Friend's Local Qwen Endpoint (Part 2)

Thu, 02 Jul 2026 00:00:00 GMT

This blog has a handful of LLM-powered features: Neon GPT (a no-login chat), the world generator in Blocks, and the live slide remixer in the arcade's PowerPoint Karaoke game. None of them call a commercial API. The model is a Qwen instance running on a friend's hardware, served through LiteLLM, and every request from this site reaches it through a Cloudflare AI Gateway that I own.

That arrangement creates a specific problem: the endpoint burns my friend's electricity and GPU time, and the frontend is a public static site with no accounts. Anyone can open dev tools, find the API call, and script it. The answer ended up being three layers: an AI Gateway for visibility and control, a thin Cloudflare Pages Function so no keys ever ship to the browser, and Turnstile to make sure a real browser on my site is asking.

The Building Blocks

Cloudflare AI Gateway is a proxy that sits between your application and an LLM provider. You create a gateway in the Cloudflare dashboard, point your app at the gateway URL instead of the provider, and every request flows through Cloudflare on its way upstream. In exchange you get analytics (request counts, tokens, latency, errors), full request/response logging, response caching, rate limiting, and retry/fallback behavior, all configured per-gateway without touching application code. It supports the big commercial providers, but it will also front any OpenAI-compatible endpoint, which is what makes it useful here: the "provider" behind my gateway is a friend's machine.

Cloudflare Turnstile is Cloudflare's CAPTCHA replacement. The widget runs in the browser (there's an invisible mode with no puzzle to solve), produces a short-lived token, and your server verifies that token against the siteverify API. A passing token means a real browser executed the challenge on a page with your sitekey. It's free, and it's the entire reason curl can't talk to my LLM proxy.

LiteLLM is an open-source proxy that exposes 100+ model backends behind one OpenAI-compatible API. My friend runs it in front of a locally hosted Qwen3 model (a 27B running an NVFP4 quant). From my side of the wire, it looks exactly like the OpenAI chat completions API: POST /v1/chat/completions, same request shape, same response shape. That compatibility is what lets the AI Gateway and my Pages Function treat a hobby GPU box like any other provider.

The Architecture

Browser (Neon GPT, Blocks, arcade games)
  │  POST /api/llm         + X-LLM-Session header
  ▼
Cloudflare Pages Function  (origin allowlist → rate limit → session check → sanitize)
  │  Authorization: Bearer <gateway key>
  │  cf-aig-authorization: Bearer <AI Gateway token>
  ▼
Cloudflare AI Gateway      (logging, analytics, caching, kill switch)
  │
  ▼
LiteLLM (friend's box)     (OpenAI-compatible proxy)
  │
  ▼
Qwen3 27B (local GPU)

And the session side-channel that feeds the X-LLM-Session header:

Browser ── invisible Turnstile widget ──> turnstile token
  │  POST /api/llm-session { turnstileToken }
  ▼
Pages Function ── siteverify ──> Cloudflare
  │  (valid → mint HMAC session token, 10 min TTL)
  ▼
Browser caches token, attaches it to every /api/llm call

Nothing secret ever reaches the browser. The gateway URL, the upstream key, the AI Gateway token, the Turnstile secret, and the session-signing secret all live as encrypted Cloudflare environment variables.

Setting Up the AI Gateway

In the Cloudflare dashboard: AI → AI Gateway → Create Gateway. That gives you a gateway endpoint. Because the upstream here is a custom OpenAI-compatible server rather than a named provider, requests address it in passthrough style. The gateway forwards to the configured upstream and records everything that goes through.

Two settings matter for this build:

Authenticated Gateway. By default anyone who discovers your gateway URL can send requests through it (and to your upstream). Turning on authentication makes the gateway reject any request that doesn't carry a cf-aig-authorization header with a token you generate in the dashboard.
Logs. Request/response logging is per-gateway. For a setup where someone else pays the GPU bill, the log view is the accountability layer. I can see exactly what traffic I sent him, token counts included.

The server-side call that goes through the gateway lives in a Pages Function (functions/api/llm.js):

const base = env.LLM_GATEWAY_URL.replace(/\/+$/, "")
const upstreamHeaders = { "Content-Type": "application/json", Authorization: `Bearer ${env.LLM_GATEWAY_KEY}` }
// When routing through a Cloudflare AI Gateway (authenticated), add its gateway token.
if (env.LLM_AIG_TOKEN) upstreamHeaders["cf-aig-authorization"] = `Bearer ${env.LLM_AIG_TOKEN}`

upstream = await fetch(`${base}/chat/completions`, {
  method: "POST",
  headers: upstreamHeaders,
  body: JSON.stringify(payload),
})

Two credentials, two different locks: Authorization is the LiteLLM key the upstream expects, cf-aig-authorization is the AI Gateway's own token. The LLM_AIG_TOKEN check means the same code runs with or without a gateway in the path. Unset it and the function calls the provider directly.

One Qwen-specific detail in the payload: Qwen3 is a reasoning model, and by default it will happily spend hundreds of tokens thinking before it answers. For interactive features that's latency you can feel, so thinking is off unless a caller explicitly asks for it:

const payload = {
  model: env.LLM_MODEL || body.model || "qwen-3.6-27b-nvfp4",
  messages: safeMessages,
  temperature: clamp(numOr(body.temperature, 0.9), 0, 2),
  max_tokens: clamp(numOr(body.max_tokens, 256), 1, 2048),
  stream: false,
  // Qwen3 is a reasoning model, so disable thinking for fast, direct answers
  chat_template_kwargs: { enable_thinking: body.think === true },
}

The Pages Function Proxy

Cloudflare Pages automatically deploys anything in /functions as a serverless function, so /api/llm is one file with no infrastructure. It does four jobs before anything reaches the gateway.

Origin gating. Only requests originating from this site (or localhost during dev) are allowed:

if (!origin || !allowed.has(origin)) return json({ error: "Forbidden" }, 403, headers)

Rate limiting. Cloudflare's rate limiting binding is wired per-IP, written so it no-ops until the binding is configured:

if (env.RATE_LIMITER && typeof env.RATE_LIMITER.limit === "function") {
  const ip = request.headers.get("CF-Connecting-IP") || origin || "anon"
  const { success } = await env.RATE_LIMITER.limit({ key: ip })
  if (!success) return json({ error: "Rate limited, slow down." }, 429, headers)
}

Input capping. Whatever the client sends gets clamped before it can cost anything: last 16 messages only, 6,000 characters per message, max_tokens capped at 2048, roles whitelisted:

const safeMessages = messages.slice(-16).map((m) => ({
  role: ["system", "user", "assistant"].includes(m.role) ? m.role : "user",
  content: String(m.content == null ? "" : m.content).slice(0, 6000),
}))

Response normalization. Every caller on the site gets the same tiny shape back, { content, usage }, regardless of what the upstream returns, with a fallback to reasoning_content for the times Qwen answers inside its thinking block.

Turnstile as the Abuse Gate

Origin checks stop drive-by scripts, but Origin is just a header, and anything outside a browser can forge it. Turnstile is the piece that can't be forged, and it's wired as a two-endpoint flow so the widget only runs occasionally instead of on every request.

Client side, an invisible widget is rendered off-screen once and executed on demand (src/lib/turnstile.js):

widgetId = window.turnstile.render(host, {
  sitekey: SITEKEY,
  size: "flexible",
  action: "share",
  callback: (token) => { pendingResolver(token); pendingResolver = null },
})
// later:
window.turnstile.execute(widgetId, { action: "share" })

The token goes to /api/llm-session, which verifies it server-side and mints a short-lived session:

async function verifyTurnstile(secret, token, ip) {
  const form = new URLSearchParams()
  form.append("secret", secret)
  form.append("response", token)
  if (ip) form.append("remoteip", ip)
  const r = await fetch("https://challenges.cloudflare.com/turnstile/v0/siteverify", { method: "POST", body: form })
  const j = await r.json()
  return !!j.success
}

A passing verification returns an HMAC-signed token, <expiry>.<nonce>.<signature>, signed with WebCrypto, valid for 10 minutes:

async function mintToken(secret) {
  const exp = Date.now() + TTL_SECONDS * 1000
  const nonce = b64url(crypto.getRandomValues(new Uint8Array(8)))
  const payload = `${exp}.${nonce}`
  const sig = await sign(secret, payload) // HMAC-SHA256 via crypto.subtle
  return `${payload}.${sig}`
}

The shared client (src/utils/llm.js) caches that session token, attaches it to every /api/llm call as X-LLM-Session, and transparently re-mints when it expires or gets a 401. /api/llm verifies the signature and expiry on each request. No state, no KV, just math:

if (env.TURNSTILE_SECRET && env.LLM_SESSION_SECRET) {
  const ok = await verifySession(request.headers.get("X-LLM-Session"), env.LLM_SESSION_SECRET)
  if (!ok) return json({ error: "Unauthorized, no valid session" }, 401, headers)
}

The chain of custody: you can't get a session without passing Turnstile, you can't pass Turnstile without a real browser on an allowed origin, and you can't call the LLM without a session. A forger with curl fails at step one.

One design decision worth copying: the whole gate fails open. If TURNSTILE_SECRET or LLM_SESSION_SECRET is unset, /api/llm-session hands back a placeholder token and /api/llm skips the check. The client flow is identical in dev (where there's no Turnstile) and prod, and misconfiguring a secret degrades to "no bot gate" instead of "site-wide outage."

Neon GPT

Neon GPT is the most direct consumer: a no-login chat UI at /tools/chat that talks straight through this stack. There's nothing special about its API usage. It calls the same askLLM() helper as everything else:

const reply = await askLLM(payload, { maxTokens: 1400, temperature: 0.7, timeout: 45000 })

The user never logs in, never sees a CAPTCHA, and never learns the endpoint. The invisible Turnstile run happens on first message, the session rides along for 10 minutes, and re-minting is invisible too. The same helper powers the Blocks world generator and the arcade games, so every AI feature on the site inherits the gate for free.

Design Decisions

Why an AI Gateway between two hobby systems? Because the alternative is my code calling my friend's box with zero visibility for either of us. The gateway gives both sides an audit trail neither has to build: request logs, token counts, latency, error rates. It's also a kill switch. If something on my site goes haywire, the gateway can rate-limit or shut off the traffic without him touching his server. And since the gateway is authenticated, even the gateway URL leaking doesn't expose his endpoint.

Why Turnstile instead of accounts? The features are toys: a chat page and some arcade games. Forcing login would kill them, and standing up auth infrastructure to protect a friend's GPU is disproportionate. Turnstile is free, invisible in this configuration, and moves the cost of abuse from "free curl loop" to "defeating a browser challenge every 10 minutes."

Why the HMAC session layer instead of verifying Turnstile per request? Turnstile tokens are single-use, and running the widget before every LLM call would add latency to each message. Exchanging one Turnstile pass for a 10-minute signed session amortizes the challenge across a whole play session. Verification is a stateless HMAC check: no session store, no KV reads, just crypto.subtle.verify in the function.

Why a Pages Function at all, when the AI Gateway is already a proxy? Keys. The browser can't hold the LiteLLM key or the gateway token, so something server-side has to inject them. Once that function exists, it's also the right place for origin checks, input caps, and response normalization, things the gateway doesn't know about my app.

Why LiteLLM on the far end? It makes the local model boring. Qwen behind LiteLLM speaks the same protocol as everything else, so if the box is down, swapping LLM_GATEWAY_URL to any other OpenAI-compatible endpoint is a config change, not a code change.

Conclusion

The whole thing is two Pages Functions, one client helper, one dashboard gateway, and zero servers on my side. A public static site gets LLM features backed by a friend's local model, with the keys server-side, the traffic observable by both parties, and the abuse surface reduced to "must be a real browser, on my site, at most this fast, at most this big."

If you're fronting any self-hosted model for public use, the pattern generalizes: OpenAI-compatible upstream, AI Gateway for observability and the kill switch, a thin function for secrets and caps, and Turnstile exchanged for a signed session so the challenge cost is paid once per session instead of per request.

OpenCode Part 1: A Terminal AI Agent You Actually Control

Thu, 25 Jun 2026 00:00:00 GMT

This is Part 1 of a series on self-hosting an LLM. Part 2 fronts a friend's local Qwen for a public site, Part 3 runs your own model on a Cloudflare Tunnel, and Part 4 gives it a memory with Redis.

GitHub Copilot's pricing has gotten complicated. What used to be a straightforward $10/month individual subscription has expanded into a tiered structure — Pro, Pro+, Max — ranging from $10 to $100/month, all moving toward a credit-based consumption model where heavy usage costs more on top of the base plan. New self-serve signups for Copilot Business were even temporarily paused in April 2026. On the enterprise side, if no user-level budget is configured, individual developers have no visibility into what they're personally consuming — only org and enterprise admins can see aggregate usage through the billing dashboard. You can be burning credits on long agent sessions with no idea what it's costing until the invoice lands.

OpenCode is an open source AI coding agent that runs in your terminal — and it's a reasonable answer to that problem. You bring your own model, and it tracks token usage and cost against every session locally. For custom or self-hosted providers you define your own per-token pricing. GitHub officially added Copilot subscription support for it earlier this year, so if you already have a paid Copilot plan you can authenticate into OpenCode with those credentials and use it without any additional setup.

Here's how I have it configured.

The Config

OpenCode lives at ~/.config/opencode/opencode.jsonc. Everything — providers, models, plugins, permissions, MCP servers — is in one file.

{
  "$schema": "https://opencode.ai/config.json",

  // Cross-repo references — use @alias in any session to bring in context
  "reference": {
    "api":          { "path": "~/dev/my-api" },
    "infra":        { "path": "~/dev/my-infra" },
    "shared-libs":  { "path": "~/dev/shared-libs" }
  },

  "lsp": true,
  "formatter": true,
  "share": "disabled",
  "snapshot": true,

  "provider": {
    "friends-gateway": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "AI Gateway",
      "options": {
        "baseURL": "https://llm.azure_fakeness.app/v1",
        "apiKey": "<api-key>"
      },
      "models": {
        "claude-sonnet-4-5": {
          "name": "Claude Sonnet (gateway)",
          "cost": {
            "input": 3.00,
            "output": 15.00,
            "cache_read": 0.30,
            "cache_write": 3.75
          },
          "limit": { "context": 200000, "output": 16000 }
        }
      }
    },
    "local": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama (local)",
      "options": {
        "baseURL": "http://127.0.0.1:11434/v1",
        "apiKey": "ollama"
      },
      "models": {
        "qwen3.6:27b-mlx": {
          "name": "Qwen3.6 27B MLX (local)",
          "limit": { "context": 128000, "output": 8192 }
        }
      }
    }
  },

  "model": "friends-gateway/claude-sonnet-4-5",

  "plugin": ["opencode-vibeguard", "@tarquinen/opencode-dcp"],

  "permission": {
    "bash": "ask",
    "edit": "ask",
    "webfetch": "ask",
    "read": "allow",
    "glob": "allow",
    "grep": "allow"
  },

  "mcp": {
    "github": {
      "type": "remote",
      "url": "https://api.githubcopilot.com/mcp/"
    },
    "your-observability-tool": {
      "type": "remote",
      "url": "https://mcp.yourtool.com/mcp",
      "headers": {
        "Authorization": "Bearer <token>"
      }
    }
  }
}

Cross-Repo References

The reference block lets you give repos an alias and pull them in by name during any session:

how does @infra handle network rules between environments?

compare how @api and @shared-libs both handle auth middleware

OpenCode treats the referenced repo as additional context for that session. You can ask questions that span multiple codebases without switching directories or opening separate sessions. For any project where work is spread across repos, this alone is worth the setup.

Providers — Including a Friend's Gateway

OpenCode uses the AI SDK under the hood, so any OpenAI-compatible endpoint works as a provider. That includes hosted APIs from Anthropic, OpenAI, and Google — but also any custom gateway someone is running.

If you have a generous friend hosting a model AI gateway on a Blackwell 6000 they have laying around, you can point OpenCode at it the same way you'd point it at any API:

"friends-gateway": {
  "npm": "@ai-sdk/openai-compatible",
  "name": "Friend's Gateway",
  "options": {
    "baseURL": "https://llm.feverdreams.app/v1",
    "apiKey": "<their-key>"
  },
  "models": {
    "claude-sonnet-4-5": {
      "name": "Claude Sonnet (gateway)",
      "cost": { "input": 3.00, "output": 15.00 },
      "limit": { "context": 200000, "output": 16000 }
    }
  }
}

The cost values are whatever you've agreed to pay — or zero if they're sharing access. OpenCode doesn't validate pricing against any external source, it just multiplies tokens by whatever you define. Which is the whole point.

The local Ollama provider works the same way — same config pattern, just pointed at localhost. Part 2 covers what's worth running locally on Apple Silicon.

Switching providers mid-session is a keybind. No restart needed.

Plugins

Two plugins worth knowing about:

opencode-vibeguard — scans outgoing prompts for secrets, JWTs, API keys, and credentials before they leave. Any match gets replaced with a placeholder before the request goes out. Useful if you work in a codebase where sensitive values end up in config files.

@tarquinen/opencode-dcp — dynamic context pruning. Compresses and prunes context as sessions grow so you don't hit the limit mid-conversation on a long debugging session. The cache read numbers in the usage table below are large partly because of this plugin keeping context alive across turns efficiently.

MCP Servers

MCP (Model Context Protocol) servers let OpenCode call external tools mid-session — querying your observability platform, pulling issue data, reading a service catalog, checking deployment status. Most major platforms have MCP servers now: GitHub, Linear, Jira, Datadog, PagerDuty, Grafana, Notion, Slack. If it has an API, there's probably an MCP server for it.

Cost Tracking

Every session's token usage and cost gets written to a local SQLite database at ~/.local/share/opencode/opencode.db. For custom providers, the cost field reflects whatever per-token pricing you defined in the config — OpenCode multiplies tokens by your rates, no external validation.

For GitHub Copilot specifically: token counts are real, pulled from the API response. The dollar figure is OpenCode's own estimate using Anthropic's public per-token pricing via the AI SDK — it is not GitHub's AIC (AI Credits). If you want to know your actual AIC consumption, that's still in the GitHub billing dashboard. What OpenCode gives you is a relative measure — how many tokens a session consumed and roughly what that would cost at list price. Good enough for spotting runaway sessions.

If GitHub's AIC is coming out higher than OpenCode's estimate, that's the signal. OpenCode is calculating against Anthropic's public list price — if Copilot costs more than that for the same tokens, you're paying a premium for the subscription wrapper. A direct gateway gives you both numbers to compare.

I wrote a small Python script that reads that database and outputs a markdown summary. It runs via a custom /usage command:

---
description: Show token and cost usage stats
---
!`python3 ~/bin/opencode-usage-md`

Type /usage in any session:

Period	Sessions	Cost	In Tokens	Out Tokens	Cache Read	Cache Write
Today	1	$0.17	24	2,074	103,049	28,093
This Week	12	$10.96	966	120,717	16,109,350	1,150,924
This Month	35	$24.47	362,404	278,251	27,766,983	3,320,768
All Time	64	$28.79	364,598	568,755	86,375,370	8,588,282

And the most recent sessions:

Time	Cost	Out	Cache Read	Session
06-24 17:26	$0.20	5,724	115,965	Explore React router changes
06-24 15:53	$3.68	37,506	6,403,169	Security Scan and fix.
06-24 15:34	$1.81	24,614	2,606,697	Incident write-up

The script is straightforward — a few SQL queries against the local database:

#!/usr/bin/env python3
import sqlite3
from datetime import datetime
from pathlib import Path

DB = Path.home() / ".local/share/opencode/opencode.db"
conn = sqlite3.connect(DB)
cur = conn.cursor()

def agg(since_ms):
    r = cur.execute(
        "SELECT COUNT(*) n, SUM(cost) cost, SUM(tokens_input) ti, SUM(tokens_output) to_, "
        "SUM(tokens_cache_read) cr, SUM(tokens_cache_write) cw FROM session WHERE time_updated >= ?",
        (since_ms,)
    ).fetchone()
    return r[0] or 0, r[1] or 0.0, r[2] or 0, r[3] or 0, r[4] or 0, r[5] or 0

The session table has everything. Cost by project, week-over-week trends, alerting when a session goes over a threshold — it's all queryable.

Getting Started

brew install opencode

If you have a Copilot subscription and want to try it first:

/connect

Select GitHub Copilot, complete the device login, done. If you want to use your own provider, add it to ~/.config/opencode/opencode.jsonc and switch models with Ctrl+M.

Part 2 covers running Ollama locally, which models are worth running on Apple Silicon, and how the local provider hooks into the same /usage tracking — except the cost column reads $0.00.

GitHub's Agentic Workflows

Tue, 23 Jun 2026 00:00:00 GMT

There's a category of work that lives in every repo that nobody actually wants to do. Updating the README after the API changes. Fixing twelve broken links that crept in over six months. Regenerating the architecture slides before the quarterly review. It's not hard. It's just relentless and invisible until someone notices it's wrong.

GitHub Agentic Workflows are GitHub's answer to that. They're currently in public preview.

What They Are

Regular GitHub Actions automation is deterministic. You define steps, a runner executes them, done. It doesn't reason—it does what you told it.

Agentic workflows are Markdown files that live in .github/workflows/ alongside your regular YAML. Instead of scripted steps, you describe what you want in natural language. When the workflow runs, a coding agent—Copilot CLI, Claude Code, or OpenAI Codex—reads the repo, figures out what to do, and opens a PR.

Traditional Workflow                Agentic Workflow
─────────────────────────           ──────────────────────────
Trigger → Fixed Steps → Done        Trigger → Agent reads repo
                                               Agent forms plan
                                               Agent edits files
                                               Agent runs tests
                                               Agent opens PR
                                               You review + merge

The format is a Markdown file with a YAML frontmatter block. Frontmatter handles the trigger, permissions, tools, and what the agent is allowed to output. The Markdown body is the instruction:

---
on:
  schedule: daily

permissions:
  contents: read
  issues: read
  pull-requests: read

safe-outputs:
  create-issue:
    title-prefix: "[report] "
    labels: [automated]

tools:
  github:
---

# Daily Repo Status Report

Create a daily status report for maintainers. Include recent activity,
open PRs worth reviewing, and any CI failures that need attention.
Keep it concise and link to the relevant issues/PRs.

The safe-outputs block is what keeps it from going rogue. Write operations—creating PRs, posting comments, opening issues—have to be explicitly listed. Anything not in that list the agent can't do. PRs are never auto-merged.

Setting One Up

GitHub Agentic Workflows are in public preview. Once you're in, install the CLI extension:

gh extension install github/gh-aw

The easiest way to write your first workflow is to ask a coding agent to do it:

Generate a workflow that checks for broken links in my docs weekly.
Use the instructions at https://github.com/github/gh-aw/blob/main/create.md

It drafts the Markdown file and a corresponding .lock.yml that GitHub Actions actually executes. Review both, then compile and push:

gh aw compile
git add .github/workflows/check-links.md .github/workflows/check-links.lock.yml
git commit -m "add: broken link check workflow"
git push

AGENTS.md

Drop an AGENTS.md in the repo root. The agent reads it at the start of every task—what it can touch, what it can't, any style rules:

# AGENTS.md

## What you can change
- `docs/` — All Markdown documentation
- `README.md` — Top-level readme
- `CHANGELOG.md` — Add entries, never modify existing ones

## What you must not touch
- `environments/` — Terraform configs, changes here deploy to production
- `.github/workflows/` — Workflow changes require human review

## Style
- Sentence case for headings
- External links must use HTTPS
- Code examples must be runnable before including them

This is also where structured documentation frameworks like Diataxis pay off. If your docs follow a consistent structure—tutorials, how-tos, reference, explanation—you can tell the agent exactly which sections are safe to update automatically. Reference tables: fine. Conceptual explanation: probably leave that to a human.

Scheduling

Any trigger GitHub Actions supports works in the frontmatter:

---
on:
  schedule:
    - cron: '0 9 * * 1'  # Every Monday at 9am
  workflow_dispatch:

permissions:
  contents: read
  pull-requests: write

safe-outputs:
  create-pull-request:
    title-prefix: "[automated] "
    base-branch: main
    labels: [documentation, automated]

tools:
  github:
  shell:
---

# Weekly Documentation Refresh

Review all documentation in `docs/` for:
- Broken or outdated external links
- Version numbers that don't match `package.json`
- API endpoints or config keys that no longer exist in the codebase

Fix what's mechanical. Flag anything requiring judgment with a
`<!-- TODO: verify -->` comment. Open a pull request with all changes
and a summary of what was updated and why.

Some Things Worth Building

Broken Link Repair

---
on:
  schedule:
    - cron: '0 8 * * 0'
  workflow_dispatch:

permissions:
  contents: read
  pull-requests: write

safe-outputs:
  create-pull-request:
    title-prefix: "[automated] "
    labels: [documentation, automated]

tools:
  shell:
  github:
---

# Broken Link Repair

Install markdown-link-check. Scan all Markdown files for broken external links.
For each broken link: find the new URL if the content moved, remove the hyperlink
but keep the text if the page is gone, or add a <!-- TODO: verify --> comment if
the entire reference is outdated. Open a pull request with all fixes.

Documentation Freshness

Docs drift. API docs reference endpoints that were renamed. Setup guides reference package versions from two releases back. Nobody notices until someone follows the guide and it breaks.

---
on:
  schedule:
    - cron: '0 9 * * 1'
  workflow_dispatch:

permissions:
  contents: read
  pull-requests: write

safe-outputs:
  create-pull-request:
    title-prefix: "[automated] "
    labels: [documentation, automated]

tools:
  shell:
  github:
---

# Documentation Freshness Check

Review all documentation in docs/ against the current codebase. Find anywhere
the docs describe behavior, APIs, or configuration that no longer matches the
actual code. Update the docs to match. Do not rewrite prose—only fix factual
inaccuracies. Include a list of every change with before/after.

If your docs follow Diataxis—tutorials, how-tos, reference, explanation—you can scope it to just the reference section and keep the risk low.

PowerPoint Updates

The agent can edit .pptx files directly using python-pptx. Point it at your architecture deck and the current infrastructure config and it'll update slide text to match.

---
on:
  workflow_dispatch:

permissions:
  contents: read
  pull-requests: write

safe-outputs:
  create-pull-request:
    title-prefix: "[automated] "
    labels: [slides, automated]

tools:
  shell:
  github:
---

# Architecture Slide Refresh

Install python-pptx. Open `docs/slides/architecture.pptx` and read the current
slide content. Compare service names, topology, and component labels against the
current infrastructure configuration in `environments/`.

For each slide where the content no longer matches reality:
- Update service names, counts, and connection labels to match current config
- Do not change layout, styling, or colors—only text content

Save the updated file. Open a PR with before/after notes for each changed slide.

Changelog Generation

---
on:
  workflow_dispatch:

permissions:
  contents: read
  pull-requests: write

safe-outputs:
  create-pull-request:
    title-prefix: "[automated] "
    labels: [changelog, automated]

tools:
  shell:
  github:
---

# Changelog Generation

Look at the git log since the last tagged release. Group commits by type:
features, fixes, breaking changes, internal. Generate a new CHANGELOG.md entry
following the existing format. Do not modify any previous entries.

Security Advisory Response

---
on:
  issues:
    types: [opened]

permissions:
  contents: read
  issues: write
  pull-requests: write

safe-outputs:
  create-pull-request:
    title-prefix: "[security] "
    labels: [security]
  add-issue-comment: true

tools:
  shell:
  github:
---

# Security Advisory Response

A security advisory has been filed. Review whether this repo uses the affected
package and the affected API surface. If affected: describe the exposure in
plain language, propose a fix, implement it if mechanical, open a PR with a
security label. If not affected, comment on the issue with an explanation.

README Hygiene

---
on:
  schedule:
    - cron: '0 9 * * 1'

permissions:
  contents: read
  pull-requests: write

safe-outputs:
  create-pull-request:
    title-prefix: "[automated] "
    labels: [documentation, automated]

tools:
  shell:
  github:
---

# README Hygiene

Review README.md against the current state of the repo. Update the getting started
section to match current setup steps, fix any command-line examples that no longer
work, and update the architecture overview if the major components have changed.
Do not change the introduction, contributing guidelines, or overall structure.

Where It Falls Down

It can't reach your production systems or query live observability data unless you wire that up via tools in the frontmatter. It'll make confident decisions on open-ended architecture questions, which isn't always what you want. And the vaguer the instruction, the more variable the output—same as any AI task.

The more specific the instruction and the more bounded the scope, the more predictable the result.

Platform AI Part 4: Building a Claude Code Plugin to Develop the AI Itself

Sun, 01 Feb 2026 00:00:00 GMT

This is Part 4 of the Platform AI series. Part 1 covers building the basic chatbot. Part 2 adds agentic tool use. Part 3 introduces the multi-agent system.

The AI assistant from the first three parts of this series has grown into a real system. Six specialized agents, a hallucination guard with four verification layers, 81 end-to-end tests, tool retry logic, parallel execution, budget tracking, and a self-evaluation pipeline. Making changes to it now requires understanding how all those pieces fit together.

I found myself repeating the same workflow every time I needed to fix a bug or add a feature. Export a chat session from the assistant to see what went wrong. Read through the codebase to find the relevant files. Make changes. Run the e2e suite. Check for regressions. That process is exactly the kind of structured, multi-step workflow that AI excels at.

So I built a Claude Code plugin to do it for me.

Live: https://chrishouse.io/tools/ai

What Is a Claude Code Skill?

Claude Code is Anthropic's CLI tool for using Claude directly in your terminal. It reads your codebase, edits files, runs commands, and understands project context. But out of the box, it's general-purpose. It doesn't know that my project has a specific agent architecture, that tests live in e2e/, or that every async function needs try/catch with bracket-prefixed logging.

Skills (also called slash commands) let you inject domain-specific expertise into Claude Code. A skill is a markdown file that acts as a system prompt, activated by typing /skill-name in the CLI. When you invoke a skill, Claude Code loads that markdown as additional context, effectively turning a general-purpose AI into a specialist for your project.

The skill lives at .claude/commands/ai-dev-studio.md in the repo. There's also a plugin registration that tells Claude Code this skill exists:

{
  "name": "ai-dev-studio",
  "version": "1.0.0",
  "description": "AI Development Studio - Multi-agent system for building and improving the AI DevOps assistant",
  "author": {
    "name": "Chris House",
    "email": "chris@platformlab.dev"
  }
}

The AI Dev Studio

The skill is called /ai-dev-studio and it transforms Claude Code into a six-agent development team. Each agent has a specific role in the development lifecycle:

User Request
     │
     ▼
┌─────────────┐
│  Classify    │ ── New Feature? Bug Fix? Improvement? Review?
└─────────────┘
     │
     ▼
┌─────────────────────────────────────────────────┐
│                Agent Pipeline                    │
│                                                  │
│  ARCHITECT → DEVELOPER → QA → REVIEWER → PRODUCT│
│                                                  │
│  Bug fix:    DEBUGGER → DEVELOPER → QA           │
│  Improvement: REVIEWER → DEVELOPER → QA          │
└─────────────────────────────────────────────────┘

When I type /ai-dev-studio and paste a chat session export or describe a bug, the skill classifies the request and activates the right agents in sequence.

The Six Agents

Architect

Architect handles design work. When I ask it to add a new feature, it reads the existing codebase first, studies how similar features are built, then produces a blueprint with specific file paths, line numbers, data flows, and integration points. It knows about the BaseAgent pattern, the agent orchestrator, the tool definition conventions, and the module structure.

Developer

Developer writes the actual code. It follows the existing conventions exactly because the skill prompt includes specific code style examples from the codebase. Named functions instead of arrow functions. Try/catch with bracket-prefixed console logs like [ModuleName]. Zod schemas for tool inputs. The skill enforces these patterns so the output matches the rest of the codebase without manual cleanup.

QA

QA writes Playwright e2e tests and runs them. The skill includes the exact test fixture pattern from e2e/fixtures.js so QA knows to call clearBrowserState() first, use aiAssistant.sendMessage() with appropriate timeouts, and check for both positive results and error states. After writing tests, it runs them and reports results.

Debugger

Debugger is the investigator. When a test fails or a session export shows unexpected behavior, Debugger reads the error, traces the execution path through the code, forms hypotheses ranked by confidence, and proposes minimal fixes. It uses a structured format with root cause analysis, evidence, and validation steps.

Reviewer

Reviewer checks for code quality, security, DRY violations, and convention adherence. It produces prioritized findings: P0 (must fix), P1 (should fix), P2 (nice to have). The skill includes specific checklists for pattern adherence, error handling, security (input validation, injection risks), and code quality.

Product

Product is the final quality gate. It reviews user-facing elements like error messages, success states, and response formatting against what the skill calls "Apple-level quality standards." The idea is that every user-visible string should be clear, actionable, and helpful rather than generic.

The Session Export Workflow

The most powerful pattern is feeding the plugin exported chat sessions from the AI assistant itself. The assistant's frontend has an export button that dumps the full session as JSON, including every message, tool call, tool result, agent routing decision, thinking blocks, and warnings.

When I paste that export into /ai-dev-studio, the Debugger agent can trace exactly what happened:

User asked: "check my clusters status"
  → Triage agent selected (confidence: 0.70)
  → get_cluster_status called → ERROR (4 times)
  → Model generated response with cluster data table
  → No fabrication warning shown

Problem: All tools failed but the model fabricated data

This is how I found and fixed the fabrication detector issues. The session showed that get_cluster_status failed four times with timeouts, but the assistant still presented a complete table with cluster names, regions, versions, and node counts. All fabricated. The Debugger agent traced the issue through the hallucination guard's four verification layers, identified why the detection missed it (markdown tables aren't code blocks, so the config-pattern matching didn't trigger), and proposed the exact fix.

In a later session, the IaC agent gave legitimate YAML suggestions for improving ArgoCD configurations, but the fabrication detector flagged them as hallucinations because no tools were called. The Debugger traced that false positive to the processHallucinationGuard function where the logic assumed "no tools + YAML = fabrication" without considering that advisory agents work from pre-loaded context.

Both fixes, across three files, came from the same workflow: export session, paste into /ai-dev-studio, let the agents trace the problem.

How It Knows the Codebase

The skill prompt is 600 lines of markdown that encodes the project's architecture, conventions, and patterns. It includes the agent capability system:

const AgentCapability = {
  READ_ONLY: 'read_only',     // Triage, Advisor
  SUGGEST: 'suggest',          // Suggestions requiring confirmation
  EXECUTE: 'execute',          // Operator - confirmation required
  AUTONOMOUS: 'autonomous'     // Rare
};

The BaseAgent class pattern showing how to define tools, canHandle, system prompts, and handoff logic. The DynamicStructuredTool pattern for defining new tools with Zod schemas. The e2e test fixture pattern. And the meta-tool pattern for composite operations like diagnose_pod_crash.

This context means Claude Code doesn't have to rediscover these patterns every time. When the Developer agent writes a new agent, it follows the exact pattern from the skill. When QA writes tests, it uses the exact fixture API.

Request Classification

The skill routes requests to different agent pipelines based on keywords:

Request Type	Keywords	Pipeline
New Feature	"add", "create", "build"	Architect, Developer, QA, Reviewer, Product
Bug Fix	"fix", "broken", "error"	Debugger, Developer, QA
Improvement	"improve", "optimize", "refactor"	Reviewer, Developer, QA
Test Coverage	"test", "e2e", "playwright"	QA
Architecture	"design", "architect", "plan"	Architect
Code Review	"review", "check", "quality"	Reviewer, Product

This mirrors how the AI assistant itself routes requests to its own agents. There's something recursive about an AI development tool that uses the same multi-agent routing pattern as the AI it's building.

Context Preservation Between Agents

When agents hand off to each other, the skill specifies a structured handoff format:

### Handoff: Debugger → Developer

**Work Completed**:
- Traced fabrication detection through 4 layers of hallucination guard
- Identified false positive trigger at response.js:135

**Key Findings**:
- Advisory agents (iac, advisor) legitimately produce YAML without tools
- The "no tools + config = fabrication" heuristic is too aggressive

**Artifacts**:
- Root cause analysis with file:line references

**Next Steps**:
- Add agent type exemption for advisory agents
- Broaden suggestion detection patterns in guard.js

This means the Developer agent gets exactly what it needs to implement the fix without re-reading the entire codebase. The Debugger already did the investigation and narrowed down the specific lines that need to change.

Quality Gates

Each agent phase has a quality gate before handing off:

Architect: Blueprint must be complete with specific file paths and line numbers
Developer: Code must parse without errors
QA: Tests must be valid and pass
Debugger: Root cause must have supporting evidence
Reviewer: All P0 issues must be addressed
Product: User-facing strings must be clear and actionable

If a gate fails, the pipeline loops back. QA failure goes to Debugger for diagnosis, then back to Developer for the fix, then back to QA.

Real Example: Fixing the Fabrication Detector

Here's how a real bug fix flowed through the plugin. I exported a session where the IaC agent's YAML suggestions triggered a false fabrication warning, and pasted it into /ai-dev-studio.

Debugger analyzed the session export and traced the issue through three code paths:

response.js:135 - The "no tools + suspicious content" check didn't account for advisory agents
guard.js:384-389 - The suggestion detection only matched narrow K8s manifest patterns (apiVersion, kind), missing ArgoCD patterns like retry:, syncOptions:, finalizers:
The all-tools-failed case where the model fabricated data presented in markdown tables rather than code blocks

Developer implemented three targeted fixes:

Added isAdvisoryAgent check in response.js to exempt IaC and Advisor agents
Added iacSuggestionPatterns array in guard.js with 14 ArgoCD/Helm-specific patterns
Added toolCallsAttempted counter in ai-chat.js to detect when all tool calls failed but the response contains structured data

QA ran the full 81-test e2e suite. Result: 81 passed, 3 flaky (API rate limits), 1 failed (pre-existing wrangler network issue). No regressions from the changes.

Three files changed, 51 lines added, 7 removed. The entire flow from session export to validated fix happened in a single Claude Code conversation.

Agents Building Agents

The most interesting aspect of this setup is that it's agents all the way down. The AI assistant running at /tools/ai uses six specialized agents (Triage, Operator, Debugger, IaC, Advisor, Network) to handle user requests. The development tool that builds and maintains that assistant uses six different specialized agents (Architect, Developer, QA, Debugger, Reviewer, Product) coordinated through a Claude Code skill.

Both systems use the same core patterns: intent classification to route to the right specialist, structured handoffs to preserve context, capability levels to control what each agent can do, and quality gates to catch issues before they ship.

The difference is scope. The runtime agents work with Kubernetes clusters and infrastructure. The development agents work with the codebase that defines those runtime agents. But the architecture is the same.

Setting It Up

The plugin requires two files in your repo:

Plugin registration at .claude/plugins/ai-dev-studio/.claude-plugin/plugin.json:

{
  "name": "ai-dev-studio",
  "version": "1.0.0",
  "description": "AI Development Studio for the AI DevOps assistant"
}

Skill definition at .claude/commands/ai-dev-studio.md:

---
name: ai-dev-studio
description: Multi-agent AI Development Studio...
---

# AI Development Studio

You are the AI Development Studio...
[600 lines of agent specs, patterns, and workflows]

Then in Claude Code, type /ai-dev-studio followed by your request. The skill activates and Claude Code becomes the development studio.

Conclusion

Building a development tool for an AI system using the same multi-agent patterns as the AI system itself felt like the natural evolution of this project. The AI assistant at /tools/ai manages Kubernetes infrastructure through specialized agents. The Claude Code plugin at /ai-dev-studio manages the assistant's codebase through specialized agents. Same architecture, different domain.

The practical value is real. Exporting a broken session from the assistant, pasting it into the plugin, and getting a traced root cause with a validated fix across multiple files in a single conversation is a workflow I now use regularly. The plugin knows the codebase patterns deeply enough that the Developer agent's output matches the existing code style without manual cleanup, and the QA agent runs the actual e2e suite to catch regressions before they ship.

The 600-line skill definition is just markdown describing how the codebase works and what quality standards to follow. The leverage you get from having an AI that understands your specific architecture, conventions, and testing patterns, compounds with every bug fix and feature you build through it.

Platform AI Part 3: Building a Multi-Agent System for DevOps

Sun, 18 Jan 2026 00:00:00 GMT

This is Part 3 of the Platform AI series. Part 1 covers building the basic chatbot. Part 2 adds agentic tool use for AKS management.

The single-agent approach from Part 2 worked fine for straightforward requests. Ask it to check cluster status, it calls the right tool. Ask it to stop a cluster, it confirms and executes. But as I started using it for real troubleshooting sessions, the cracks showed up fast.

A question like "why is my pod crashing?" requires a different mindset than "start the hub cluster." The first needs investigation, hypothesis forming, log analysis. The second needs careful execution with pre-flight checks and confirmation. Cramming both behaviors into a single system prompt creates a confused agent that either over-investigates simple operations or rushes through debugging without proper analysis.

The solution was obvious once I saw the pattern. Instead of one agent trying to be everything, build a team of specialists and let them collaborate.

Live: https://chrishouse.io/tools/ai

The Multi-Agent Architecture

The new system routes requests to one of five specialized agents, each optimized for a specific type of work.

┌─────────────────┐     ┌─────────────────────┐     ┌──────────────────┐
│   User Message  │────>│  Agent Orchestrator │────>│  Selected Agent  │
└─────────────────┘     └─────────────────────┘     └──────────────────┘
                               │                           │
                               │ Intent Classification     │ Specialized
                               │ Confidence Scoring        │ System Prompt
                               │ Handoff Management        │ Filtered Tools
                               v                           v
                        ┌─────────────────────────────────────────────┐
                        │              Agent Types                     │
                        ├─────────────────────────────────────────────┤
                        │  Triage    - Status checks, overview        │
                        │  Debugger  - Investigation, root cause      │
                        │  Operator  - Safe mutations, verification   │
                        │  Advisor   - Best practices, recommendations│
                        │  IaC       - Terraform, Crossplane, GitOps  │
                        └─────────────────────────────────────────────┘

Each agent has its own system prompt, a filtered set of tools, and rules for when to hand off to another specialist. The orchestrator sits in front, classifying intent and routing requests to the right agent.

The Five Agents

Triage Agent

Triage handles quick status checks and overview requests. When someone asks "what's running?" or "show me the pods," they probably want a fast answer, not a deep investigation. Triage gathers the requested data and presents it clearly.

Triage has read-only tools like get_cluster_status, get_pods, get_deployments, and get_events. It cannot execute mutations. If someone asks Triage to restart a deployment, it recognizes that requires the Operator agent and initiates a handoff.

Debugger Agent

Debugger is the investigator. When something is broken and you need to understand why, Debugger formulates hypotheses and tests them systematically. It looks at pod descriptions, pulls logs from current and previous containers, checks events, and correlates findings.

The Debugger system prompt explicitly instructs it to think in terms of root cause analysis. It gathers context first, forms a hypothesis about what might be wrong, tests that hypothesis with targeted tool calls, and presents findings with confidence levels.

Operator Agent

Operator handles mutations. Starting clusters, stopping clusters, deleting pods, restarting deployments. These actions have real consequences, so Operator is built around safety. Every mutation requires user confirmation. Operator runs pre-flight checks before actions and verification checks after.

The confirmation flow is baked into the agent itself. When Operator decides to call stop_cluster, it first verifies the cluster is actually running, warns about impacts like "12 pods will be terminated," and waits for explicit user approval before executing.

Advisor Agent

Advisor provides recommendations and best practices. When someone asks "should I use a DaemonSet or Deployment for this?" or "what's the best way to handle secrets?", Advisor draws on Kubernetes patterns and platform engineering principles to give guidance.

Advisor has no mutation tools at all. It can query current state for context but cannot make changes. Its role is purely consultative.

IaC Agent

IaC handles Infrastructure as Code questions. Terraform modules, Crossplane compositions, Helm charts, GitOps workflows. It knows how to read HCL syntax, understands Crossplane XRDs, and can help with ArgoCD application definitions.

IaC also integrates with a RAG system that indexes the actual repository. When someone asks about a specific Terraform module, IaC can search the codebase and reference real files rather than hallucinating generic examples.

Agent Selection

The orchestrator classifies each incoming message and selects the most appropriate agent. This happens through pattern matching and confidence scoring.

// Intent patterns for routing
const INTENT_PATTERNS = {
  debug: [
    /why.*(fail|crash|error|not working|stuck)/i,
    /debug/i,
    /troubleshoot/i,
    /investigate/i,
    /what.*(wrong|issue|problem)/i,
    /crashloop/i,
    /diagnose/i,
    /root cause/i,
    /analyze.*(logs?|error|issue)/i,
  ],

  operate: [
    /\b(start|stop|restart|scale|delete|rollout)\b/i,
    /\b(deploy|upgrade|rollback)\b/i,
    /apply.*changes?/i,
    /execute/i,
  ],

  advise: [
    /best practice/i,
    /recommend/i,
    /should I/i,
    /how should/i,
    /what.*(approach|strategy)/i,
    /optimize/i,
  ],

  triage: [
    /status/i,
    /overview/i,
    /health/i,
    /what.*(running|deployed|happening)/i,
    /show.*all/i,
    /list/i,
    /get.*pods?/i,
    /check/i,
  ]
};

Each pattern match adds to the score for that intent. The highest scoring intent determines which agent handles the request.

export function classifyIntent(message) {
  const scores = {};

  for (const [intent, patterns] of Object.entries(INTENT_PATTERNS)) {
    scores[intent] = 0;
    for (const pattern of patterns) {
      if (pattern.test(message)) {
        scores[intent]++;
      }
    }
  }

  // Find highest scoring intent
  let maxIntent = 'triage'; // Default to triage
  let maxScore = 0;

  for (const [intent, score] of Object.entries(scores)) {
    if (score > maxScore) {
      maxScore = score;
      maxIntent = intent;
    }
  }

  // Map intent to agent
  const agentMap = {
    debug: 'debugger',
    operate: 'operator',
    advise: 'advisor',
    triage: 'triage'
  };

  const confidence = maxScore > 0 ? Math.min(maxScore / 3, 1) : 0.3;

  return {
    intent: maxIntent,
    confidence,
    suggestedAgent: agentMap[maxIntent],
    scores
  };
}

A message like "why is my pod crashing and restarting?" hits multiple debug patterns. "Crashing" matches the crash pattern, "why" matches the why pattern, and the combination creates high confidence for the Debugger agent.

The Agent Orchestrator

The orchestrator is the traffic controller. It maintains state about which agent is currently active, handles agent selection, and manages handoffs between agents.

export class AgentOrchestrator {
  constructor(tools, options = {}) {
    this.tools = tools;
    this.agents = {};

    // Initialize all agents
    for (const [name, AgentClass] of Object.entries(AGENT_TYPES)) {
      this.agents[name] = new AgentClass(tools, options);
    }

    this.currentAgent = null;
    this.handoffHistory = [];
    this.onAgentSwitch = options.onAgentSwitch || null;
  }

  selectAgent(message, context = {}) {
    const scores = [];

    for (const [name, agent] of Object.entries(this.agents)) {
      const assessment = agent.canHandle(message, context);
      scores.push({
        agent: name,
        confidence: assessment.confidence,
        canHandle: assessment.canHandle,
        reason: assessment.reason
      });
    }

    // Sort by confidence
    scores.sort((a, b) => b.confidence - a.confidence);
    const best = scores[0];

    // If coming from handoff, respect the suggested agent
    if (context.handoff?.targetAgent && this.agents[context.handoff.targetAgent]) {
      return {
        agent: context.handoff.targetAgent,
        confidence: 0.9,
        reason: `Handoff from ${context.handoff.fromAgent}: ${context.handoff.reason}`,
        alternatives: scores.filter(s => s.agent !== context.handoff.targetAgent).slice(0, 2)
      };
    }

    return {
      agent: best.agent,
      confidence: best.confidence,
      reason: best.reason,
      alternatives: scores.slice(1, 3)
    };
  }
}

Each agent implements a canHandle method that returns a confidence score. The Debugger agent, for example, looks for investigation-related keywords and returns high confidence when it sees patterns indicating a troubleshooting session.

// From debugger-agent.js
canHandle(message, context = {}) {
  let score = 0;

  for (const pattern of DEBUGGER_PATTERNS) {
    if (pattern.test(message)) {
      score += 0.25;
    }
  }

  // Coming from handoff with issue to investigate
  if (context.handoff?.findings?.length > 0) {
    score += 0.4;
  }

  // Explicit debug keywords
  if (/\b(debug|investigate|root cause|diagnose)\b/i.test(message)) {
    score += 0.3;
  }

  const confidence = Math.min(1, score);

  return {
    canHandle: confidence > 0.3,
    confidence,
    reason: confidence > 0.5 ? 'Request indicates investigation needed' : 'May need debugging'
  };
}

Agent Handoffs

Sometimes an agent realizes mid-conversation that another specialist would be better suited. The Triage agent might discover a crashing pod while checking status and hand off to Debugger. The Debugger might identify a fix and hand off to Operator to execute it.

Handoffs preserve context. When Debugger hands off to Operator, it passes along what it discovered, what tools it already called, and what action it recommends. The receiving agent gets this context injected into its system prompt so it can continue intelligently rather than starting from scratch.

handleHandoff(fromAgent, toAgent, reason, context = {}) {
  const sourceAgent = this.agents[fromAgent];
  const targetAgent = this.agents[toAgent];

  // Build handoff context
  const handoffContext = {
    fromAgent,
    toAgent,
    reason,
    timestamp: Date.now(),
    ...sourceAgent.buildHandoffContext(
      context.workDone || [],
      context.findings || [],
      context.toolResults || []
    )
  };

  // Record handoff
  this.handoffHistory.push(handoffContext);

  // Switch to new agent
  this.currentAgent = toAgent;

  if (this.onAgentSwitch) {
    this.onAgentSwitch({
      agent: toAgent,
      confidence: 0.9,
      reason: `Handoff: ${reason}`,
      handoff: true,
      from: fromAgent
    });
  }

  return handoffContext;
}

Each agent has rules about when to suggest a handoff. The Operator agent, for example, hands off to Debugger if an action fails and needs investigation.

// From operator-agent.js
suggestHandoff(message, toolResults = [], context = {}) {
  // If action failed, suggest debugger
  const failedActions = toolResults.filter(r =>
    !r.success || r.data?.error || r.error
  );

  if (failedActions.length > 0) {
    return {
      shouldHandoff: true,
      targetAgent: 'debugger',
      reason: `Action failed: ${failedActions[0].error}. Debugger can investigate.`,
      context: { failedActions }
    };
  }

  // If user asks for explanation after action
  if (/why|should|best\s*practice|recommend/i.test(message)) {
    return {
      shouldHandoff: true,
      targetAgent: 'advisor',
      reason: 'User asking for recommendations/best practices',
      context: { completedActions: toolResults.filter(r => r.success) }
    };
  }

  return null;
}

Investigation Workflows

The Debugger agent is where multi-agent really shines. A simple "why is my pod crashing?" triggers a structured investigation.

The agent first gathers context. It calls describe_pod to get the pod specification, conditions, and container statuses. It pulls current logs with get_pod_logs and previous container logs to catch errors that happened before the last restart. It checks recent events for warnings like OOMKilled or ImagePullBackOff.

All of this happens in a single reasoning loop. The agent decides which tools to call, executes them, analyzes the results, and either continues investigating or presents its findings.

// Debugger agent system prompt (excerpt)
`You are the Debugger Agent for Platform AI. Your role is to:

1. **Gather Context**: Get pod status, logs, events, resource usage
2. **Form Hypotheses**: Based on symptoms, identify possible causes
3. **Test Hypotheses**: Use targeted queries to confirm or rule out causes
4. **Present Findings**: Explain root cause with evidence

## Investigation Process

When investigating an issue:
1. Start with describe_pod to understand current state
2. Pull logs (current + previous) for error messages
3. Check events for warnings (OOMKilled, ImagePullBackOff, etc.)
4. Correlate findings to identify root cause
5. Present diagnosis with confidence level

## When to Suggest Handoff
- **→ Operator**: If you've identified a fix (restart, scale, delete)
- **→ Advisor**: If user asks for best practices to prevent recurrence
- **→ Triage**: If issue resolved and user wants status overview`

The result is an agent that thinks like a platform engineer. It does not just dump logs and hope the user figures it out. It correlates evidence, identifies patterns, and explains what went wrong and why.

The Base Agent Pattern

All five agents extend a common base class that provides shared functionality. This keeps the architecture consistent and makes adding new agents straightforward.

export class BaseAgent {
  constructor(options = {}) {
    if (new.target === BaseAgent) {
      throw new Error('BaseAgent is abstract and cannot be instantiated directly');
    }

    this.name = options.name || 'BaseAgent';
    this.description = options.description || '';
    this.capability = options.capability || AgentCapability.READ_ONLY;
    this.priority = options.priority || AgentPriority.MEDIUM;
    this.tools = options.tools || [];
    this.maxIterations = options.maxIterations || 5;
    this.icon = options.icon || '🤖';
    this.color = options.color || '#6366f1';
  }

  // Must be implemented by subclasses
  buildSystemPrompt(context = {}) {
    throw new Error('buildSystemPrompt must be implemented by subclass');
  }

  canHandle(message, context = {}) {
    throw new Error('canHandle must be implemented by subclass');
  }

  // Optional overrides
  suggestHandoff(message, toolResults = [], context = {}) {
    return null;
  }

  preprocessMessage(message, context = {}) {
    return message;
  }

  postprocessResponse(response, toolResults = [], context = {}) {
    return response;
  }
}

The capability system controls what agents can do. READ_ONLY agents like Triage and Advisor cannot execute mutations. EXECUTE agents like Operator require confirmation for destructive actions. This is enforced at the tool level, so even if a prompt injection tried to get Advisor to delete a pod, the tool simply would not be available.

export const AgentCapability = {
  READ_ONLY: 'read_only',       // Can only query/read
  SUGGEST: 'suggest',           // Can suggest actions
  EXECUTE: 'execute',           // Can execute with confirmation
  AUTONOMOUS: 'autonomous'      // Can execute without confirmation (rare)
};

Confirmation Flow for Mutations

The Operator agent has a specific confirmation flow for dangerous operations. Every mutation goes through the same pattern: verify current state, present impact, wait for explicit confirmation, execute, verify result.

// Actions that require confirmation
const CONFIRMATION_REQUIRED = {
  start_cluster: { level: 'high', message: 'Starting a cluster will incur compute costs' },
  stop_cluster: { level: 'high', message: 'Stopping will make the cluster unavailable' },
  delete_pod: { level: 'medium', message: 'Pod will be recreated by its controller' },
  restart_deployment: { level: 'medium', message: 'Will trigger rolling restart of all pods' },
  scale_nodepool: { level: 'high', message: 'Will change the number of VMs (affects cost)' }
};

// Pre-flight checks for each action
const PREFLIGHT_CHECKS = {
  start_cluster: ['get_cluster_status'],
  stop_cluster: ['get_cluster_status', 'get_pods'],
  delete_pod: ['describe_pod'],
  restart_deployment: ['get_deployments', 'get_pods'],
  scale_nodepool: ['list_nodepools', 'get_pods']
};

When Operator decides to stop a cluster, it first calls get_cluster_status and get_pods as pre-flight checks. If the cluster is already stopped, it reports that and does not ask for confirmation. If pods are running, it warns the user how many will be terminated.

getConfirmationDetails(toolName, args = {}) {
  const config = CONFIRMATION_REQUIRED[toolName];
  if (!config) return null;

  let description = `${toolName}: `;

  switch (toolName) {
    case 'start_cluster':
      description = `Start cluster "${args.cluster_name}"`;
      break;
    case 'stop_cluster':
      description = `Stop cluster "${args.cluster_name}"`;
      break;
    case 'delete_pod':
      description = `Delete pod "${args.pod_name}" in ${args.namespace}`;
      break;
    case 'restart_deployment':
      description = `Restart deployment "${args.deployment_name}" in ${args.namespace}`;
      break;
  }

  return {
    action: toolName,
    description,
    level: config.level,
    warning: config.message,
    args
  };
}

Real-Time Feedback with SSE

The frontend shows which agent is active and when switches happen. This is handled through Server-Sent Events that the orchestrator emits on state changes.

export function createOrchestratorWithSSE(tools, emitEvent, options = {}) {
  return new AgentOrchestrator(tools, {
    ...options,
    onAgentSwitch: (selection) => {
      emitEvent('agent_switch', {
        agent: selection.agent,
        reason: selection.reason,
        confidence: selection.confidence,
        from: selection.from,
        handoff: selection.handoff || false,
        timestamp: Date.now()
      });
    }
  });
}

The React frontend listens for these events and updates the UI. Users see a small indicator showing which agent is handling their request. When a handoff happens, they see it transition smoothly from Debugger to Operator, for example.

This transparency builds trust. Users understand that when they ask an investigation question, they get the investigation specialist. When they ask to execute something, they get the operations specialist.

Security Layers

Before any message reaches an agent, it passes through security filters. The system blocks requests for secrets, credentials, and sensitive data. It also catches off-topic requests before they waste model tokens.

const BANNED_PATTERNS = [
  /\bsecret[s]?\b/i,
  /\bpassword[s]?\b/i,
  /\bcredential[s]?\b/i,
  /\bapi[_-]?key[s]?\b/i,
  /\bkubeconfig\b/i,
  /kubectl\s+get\s+secret/i,
  /kubectl\s+describe\s+secret/i,
];

function containsSensitiveRequest(text) {
  return BANNED_PATTERNS.some(pattern => pattern.test(text));
}

Off-topic detection prevents people from using the assistant for general chat, math homework, or prompt injection attempts.

const OFF_TOPIC_PATTERNS = [
  // Math
  /^\s*\d+\s*[\+\-\*\/x×÷]\s*\d+/i,
  /calculate|compute|solve.*\d/i,

  // Jailbreak attempts
  /ignore\s*(your|previous|all)\s*(instructions?|rules?)/i,
  /pretend\s*(you('re)?|to\s*be)/i,
  /you\s*are\s*now\s*(a|an|no\s*longer)/i,

  // General knowledge
  /what\s*(is|are)\s*(the\s*)?(sun|moon|stars?|planets?)/i,
  /write\s*(me\s*)?(a\s*)?(poem|story|essay)/i,
];

const OFF_TOPIC_REDIRECT = "I'm Platform AI - I only help with Kubernetes and infrastructure. Try asking about cluster status, deployments, pods, or Azure resources!";

These filters run before the orchestrator even sees the message. A request for secrets gets rejected immediately with a clear explanation of the security policy. An off-topic request gets a polite redirect without consuming any Claude API tokens.

Technical Decisions

Why five agents instead of three or ten?

Five covers the major interaction patterns I observed in my own usage. Status checks (Triage), troubleshooting (Debugger), operations (Operator), guidance (Advisor), and infrastructure code (IaC). More agents would mean more routing complexity without clear benefit. Fewer agents would mean cramming multiple personas into single prompts.

Why pattern-based routing instead of LLM classification?

Speed and cost. Pattern matching happens in microseconds without an API call. LLM classification would add latency and token cost to every request. The pattern approach handles 90% of cases correctly, and for edge cases, each agent's canHandle method provides a secondary check.

Why hand off context explicitly instead of sharing conversation history?

Token efficiency. A full conversation history grows quickly and most of it is not relevant to the receiving agent. By passing structured handoff context, the receiving agent gets what it needs to continue without paying for irrelevant earlier messages.

Why capability levels on agents?

Defense in depth. Even if prompt injection somehow bypassed the security filters and convinced an agent to try deleting a pod, READ_ONLY agents simply do not have the tool available. The capability level is enforced at tool binding time, not at prompt time.

What's Next

The handoff system could become smarter with learned routing. Instead of static patterns, the system could learn from successful interactions which agent combinations work best for which types of requests.

And there is still the state machine and meta-tool system to cover, but that is another article.

Conclusion

Going from a single chatbot to a multi-agent system changed how I think about AI assistants. The single-agent approach forces you to write one prompt that handles everything. That works for simple cases but creates confused behavior for complex workflows.

The multi-agent approach lets each specialist excel at its job. Triage is fast and focused. Debugger is methodical and thorough. Operator is safe and verified. They hand off to each other when needed, preserving context and building on each other's work.

The implementation is not complicated. Each agent is maybe 200-300 lines of code. The orchestrator is another 300. The real work is in thinking through the interaction patterns and defining clear boundaries between agents.

The assistant lives at /tools/ai on this blog if you want to see it in action. Ask it to check cluster status and notice the Triage agent. Ask why something is failing and watch the Debugger take over. Ask to restart a deployment and see the Operator's confirmation flow. The agents are visible in the UI, so you can follow along as they work.

Platform AI Part 2: Adding Agentic Tool Use for AKS Management

Wed, 14 Jan 2026 00:00:00 GMT

This is Part 2 of the Platform AI series. Part 1 covers building the basic chatbot with Claude API and Cloudflare.

Most AI assistants answer questions. This one executes actions. Platform AI is an agentic assistant built into my blog that can query live cluster status, start and stop AKS clusters, and handle infrastructure operations, all through natural conversation.

Live: https://chrishouse.io/tools/ai

What It Does

Platform AI is a specialized DevOps assistant that goes beyond answering questions. It can query real-time power state, node count, and Kubernetes version across all my AKS clusters. When I need to bring up a stopped cluster, I just ask it to start one and it walks me through a confirmation flow before executing. The same goes for stopping clusters, which includes automatic Crossplane webhook cleanup so the cluster doesn't get stuck on restart.

Beyond cluster management, it can list pods across namespaces with their status and restart counts, fetch logs from any container, show deployment replica health, and pull recent Kubernetes events for troubleshooting. All responses are tailored to my actual infrastructure since it has context about my clusters, namespaces, and ArgoCD apps. I've also scoped it to only answer questions about Kubernetes, Docker, Terraform, Helm, ArgoCD, Istio, and other IaC topics so it stays focused.

The assistant uses Claude's tool use capability to execute real actions against Azure, not just generate text about what you could do.

The Architecture

┌─────────────────┐     ┌─────────────────────┐     ┌──────────────────┐
│   React Chat    │────>│  Cloudflare Pages   │────>│   Claude API     │
│   Component     │     │  Function           │     │   (Haiku)        │
└─────────────────┘     └─────────────────────┘     └──────────────────┘
                               │                           │
                               │ Tool Calls                │
                               v                           │
                       ┌─────────────────────┐            │
                       │   Azure Function    │<───────────┘
                       │   (Managed ID)      │
                       └─────────────────────┘
                               │
                               v
                       ┌─────────────────────┐
                       │   AKS Clusters      │
                       │   (Hub + Spokes)    │
                       └─────────────────────┘

Components

The frontend is a React component with Firebase handling chat UI, message history, and session persistence. Requests go through a Cloudflare Pages Function that handles PAT authentication, routes tool calls, and formats responses. I'm using Claude 3 Haiku as the AI engine since it's fast and handles tool use decisions well. The backend is an Azure Function with Managed Identity so I don't have to store any credentials for cluster operations. Firebase Realtime DB stores context caching, usage stats, and chat history.

LangChain Integration

I recently refactored the Cloudflare function to use LangChain.js instead of calling the Claude API directly. This cleaned up the code significantly and made tool definitions more structured.

Why LangChain?

The raw Claude API works fine, but LangChain gives me a few things I wanted. Tool definitions use Zod schemas for input validation, which catches bad inputs before they hit my Azure Function. The message handling is cleaner with typed classes like HumanMessage, AIMessage, and ToolMessage. And the agentic loop for tool execution is more maintainable.

Dependencies

{
  "@langchain/anthropic": "^0.3.18",
  "@langchain/core": "^0.3.40",
  "zod": "^3.24.2"
}

Tool Definition with Zod

Each tool is now a DynamicStructuredTool with a Zod schema:

import { DynamicStructuredTool } from '@langchain/core/tools';
import { z } from 'zod';

const getClusterStatusTool = new DynamicStructuredTool({
  name: 'get_cluster_status',
  description: 'Get the current status and power state of AKS clusters.',
  schema: z.object({
    cluster_name: z.string().optional()
      .describe('Specific cluster name to check. If omitted, returns all clusters.')
  }),
  func: async ({ cluster_name }) => {
    const result = await api('status');
    // Filter by cluster_name if provided...
    return JSON.stringify(result);
  }
});

The Zod schema validates input types and provides descriptions that Claude uses to understand what each parameter does. If someone tries to pass an invalid type, Zod catches it before the function runs.

Message Handling

LangChain provides typed message classes that make the conversation flow clearer:

import { HumanMessage, AIMessage, SystemMessage, ToolMessage } from '@langchain/core/messages';

const messages = [
  new SystemMessage(systemPrompt),
  ...history.map(msg =>
    msg.role === 'user'
      ? new HumanMessage(msg.content)
      : new AIMessage(msg.content)
  ),
  new HumanMessage(userMessage)
];

const response = await model.invoke(messages);

When Claude calls a tool, it returns a response with tool call blocks. I execute the tool, then add a ToolMessage with the result:

if (response.tool_calls?.length > 0) {
  for (const toolCall of response.tool_calls) {
    const tool = tools.find(t => t.name === toolCall.name);
    const result = await tool.func(toolCall.args);

    messages.push(new AIMessage({ content: '', tool_calls: [toolCall] }));
    messages.push(new ToolMessage({
      content: result,
      tool_call_id: toolCall.id
    }));
  }
  // Call model again to synthesize final response
  const finalResponse = await model.invoke(messages);
}

Benefits I've Noticed

The code is more readable now that tool definitions are self-contained objects with their schema and function together. Type safety from Zod has caught a few issues during development. And the message handling is less error-prone since I'm not manually constructing JSON objects.

Tool Definitions

The assistant has seven tools available when infrastructure context is enabled.

Cluster Management

The get_cluster_status tool returns power state, Kubernetes version, and node count for all clusters. The start_cluster tool starts a stopped AKS cluster but requires confirmation first. And stop_cluster stops a running cluster with automatic Crossplane webhook cleanup, also requiring confirmation.

Kubernetes Introspection

For looking inside clusters, get_pods lists pods with their namespace, status, ready count, and restart count. The get_pod_logs tool fetches container logs with optional tail line limits. get_deployments shows deployment status and replica health. And get_events pulls recent cluster events which is useful for debugging.

Tool Execution Loop

When Claude decides to use a tool, the response includes tool call blocks. The Cloudflare function executes each tool, collects the results, and sends them back to Claude for a final response:

// Claude returns tool_calls when it wants to call tools
if (response.tool_calls?.length > 0) {
  for (const toolCall of response.tool_calls) {
    const result = await executeToolCall(toolCall.name, toolCall.args, env);
    toolResults.push(new ToolMessage({
      content: JSON.stringify(result),
      tool_call_id: toolCall.id
    }));
  }

  // Send results back to Claude for final response synthesis
  messages.push(...toolResults);
  const finalResponse = await model.invoke(messages);
}

Example: get_cluster_status Response

{
  "clusters": [
    {
      "name": "aks-mgmt-hub",
      "resourceGroup": "rg-landing-zone-hub",
      "powerState": "Running",
      "kubernetesVersion": "1.32",
      "nodeCount": 3
    }
  ],
  "collectedAt": "2026-01-14T21:10:27.274Z"
}

Confirmation Flow

Start and stop operations require explicit user confirmation. When I ask to start a cluster, the AI returns a message asking me to confirm with buttons. Only after I click confirm does it actually execute the operation. This prevents accidental cluster operations from a misunderstood request.

Stop operations include automatic Crossplane webhook removal. This prevents a common issue where Crossplane's validating webhooks block API calls when the cluster restarts since the webhook endpoints aren't running yet.

Smart Webhook Handling

When stopping a cluster that runs Crossplane, the Azure Function first gets the admin kubeconfig for the target cluster. It then deletes both the crossplane ValidatingWebhookConfiguration and MutatingWebhookConfiguration before proceeding with the cluster stop.

This prevents the cluster from getting stuck in a broken state on restart where the API server can't process requests because it's waiting for webhook responses from pods that aren't running yet.

The Azure Function Backend

The backend is a Node.js Azure Function that uses Managed Identity for zero-credential authentication to Azure and Kubernetes APIs.

Endpoints

The function exposes several endpoints. GET /status returns just cluster power states for quick checks. GET /full returns complete context for the AI including namespaces, apps, and XRDs. GET /inventory shows namespaces, deployments, and services. GET /argocd returns ArgoCD application sync status. GET /crossplane shows XRDs, compositions, and active claims. GET /istio returns VirtualServices and Gateways.

For operations, POST /start-cluster starts a stopped AKS cluster. POST /stop-cluster stops a running cluster with webhook cleanup. POST /get-pods lists pods with status and restart counts. POST /get-logs fetches pod container logs. POST /get-deployments lists deployments with replica status. And POST /get-events gets recent Kubernetes events.

Key Dependencies

The function uses @azure/identity with DefaultAzureCredential for Managed Identity authentication, @azure/arm-containerservice for AKS control plane operations, and @kubernetes/client-node for Kubernetes API access.

How It Works

Authentication happens automatically through DefaultAzureCredential which uses the Function App's Managed Identity. For cluster discovery, it lists all AKS clusters in the subscription via Azure Resource Manager. To access Kubernetes APIs on running clusters, it fetches the admin kubeconfig and queries directly. Start and stop operations are fire-and-forget since they take 2-5 minutes to complete.

The function collects data from multiple Kubernetes APIs in parallel:

const [inventory, argocd, crossplane, istio] = await Promise.all([
  collectInventory(kc, clusters),
  collectArgoCD(kc),
  collectCrossplane(kc),
  collectIstio(kc)
]);

No secrets are stored in code or config. The Managed Identity has Contributor access to the AKS clusters, and the Function App's system-assigned identity is granted cluster-admin via Azure RBAC.

Live Infrastructure Context

The assistant doesn't just know Kubernetes, it knows my Kubernetes. When infra context is enabled, every response is informed by my actual cluster names, resource groups, versions, and power states. It knows about my namespaces including system ones and platform namespaces for ArgoCD, Crossplane, and Istio. It sees ArgoCD app sync status, health, and destinations. It understands my Crossplane setup with XRDs, active claims, and provisioned resources. And it knows about Istio gateways, VirtualServices, and traffic routing.

Context comes from three sources with automatic fallback. First it tries live data from the Azure Function querying clusters in real-time. If that fails, it falls back to cached data in Firebase from hourly GitHub Actions snapshots. And if everything else fails, there's a hardcoded static fallback.

Scoped to DevOps

The assistant refuses off-topic questions:

"I can only help with Kubernetes, Docker, AKS, Azure CLI, Istio, Terraform, Helm, ArgoCD, Flux, and Infrastructure as Code questions. Please ask about those topics."

This keeps responses focused and prevents the model from hallucinating outside its configured expertise.

UI Features

Formatted Tool Results

Tool outputs render with visual indicators to make them easy to scan. Cluster status shows green or red power state indicators. Pod lists display namespace, status, ready count, and restart count. Deployments show replica health at a glance. Events display type icons for warnings versus normal events with truncated messages. Logs render in code blocks for easy reading. Start and stop operations show the cluster name, previous state, and webhook removal details. Errors surface clearly with warning indicators.

Context Window Management

The chat stores up to 40 messages but only sends the last 20 to Claude using a sliding window approach. A pie chart in the footer shows storage usage. Under 40 messages, the pie fills proportionally as you chat. At 40 messages, the pie fills solid blue and a compact button appears on hover. Clicking compact removes the oldest 20 messages while keeping the most recent 20.

This means I can have long conversations without constantly managing context. The API always sees recent messages, and I only need to compact occasionally.

Session Persistence

Chats persist to Firebase, keyed by a hash of the user's PAT. I can switch between conversations, delete old ones, or continue where I left off.

Security Model

Access to the API requires a Personal Access Token stored client-side. Azure authentication uses Managed Identity on the Azure Function so there are no stored credentials. Destructive actions like starting or stopping clusters require explicit user confirmation through the UI. The assistant refuses non-DevOps questions to keep it scoped. And there's per-user token tracking with visible usage stats.

What's Next

I'm thinking about adding ArgoCD sync triggers from chat, cost queries to see how much a cluster cost this month, Crossplane claim creation for provisioning infrastructure through conversation, and maybe kubectl exec support for running commands inside pods for debugging.

The foundation is set for expanding what the assistant can do while keeping the confirmation flow for anything destructive.

Platform AI Part 1: Building a Personal AI Chatbot with Claude and Cloudflare

Sun, 11 Jan 2026 00:00:00 GMT

This is Part 1 of the Platform AI series. Part 2 adds agentic tool use for managing AKS clusters.

I wanted a specialized AI assistant that could answer my DevOps and Infrastructure as Code questions without the distraction of general-purpose chatbots. The goal was simple: build a focused tool that knows Kubernetes, Docker, AKS, Azure CLI, Istio, Terraform, Helm, and related IaC tools inside and out, and politely declines everything else.

What started as a weekend project turned into a surprisingly elegant solution using Claude's API, React, Cloudflare Pages Functions, and Firebase Realtime Database for persistent chat history. The result is a clean, minimal chat interface that feels like Claude in VS Code. No fluff, just helpful answers to infrastructure questions.

The assistant lives at /tools/ai on this blog and includes persistent multi-chat conversations, real-time usage tracking, configurable AI parameters (temperature and max tokens), fullscreen mode for deep-dive sessions, and a visual design that stays out of your way while you work through deployment issues or troubleshoot pod crashes.

Try it now →

Architecture Overview

The stack is straightforward. The frontend is a Gatsby static site with a React chat component. The backend is a single Cloudflare Pages Function running serverless. Claude 3 Haiku handles all the AI heavy lifting via the Anthropic API, and Firebase Realtime Database stores persistent chat history.

Authentication uses simple PAT validation where the backend checks the token against an environment variable. Scope control happens through a system prompt that enforces topic restrictions.

The backend acts as a thin proxy between the frontend and Claude API, validating the PAT and injecting the system prompt that restricts Claude's responses to only Kubernetes, Docker, AKS, Azure CLI, Istio, Terraform, Helm, and Infrastructure as Code topics. Firebase Realtime Database stores chat conversations and usage statistics, keyed by a SHA-256 hash of the user's PAT for privacy.

Backend Implementation

Cloudflare Pages Function

The backend is a single file at functions/api/ai-chat.js. Cloudflare Pages automatically discovers functions in the /functions directory and maps them to routes.

The function handles PAT validation against an environment variable, forwards messages to Claude API with the system prompt, extracts usage statistics from the API response, calculates cost estimates based on token usage, and returns the response with usage metadata.

Authentication and Parameters:

const { messages, pat, temperature = 0.3, maxTokens = 2048 } = await request.json();

if (!pat || pat !== env.ASSISTANT_PAT) {
  return new Response(JSON.stringify({ error: 'Invalid access token' }), {
    status: 401,
    headers: { ...corsHeaders, 'Content-Type': 'application/json' }
  });
}

The backend accepts configurable temperature (defaults to 0.3 for consistent technical responses) and maxTokens (defaults to 2048) from the frontend, with validation to ensure they're within safe ranges (temperature: 0-1, maxTokens: 256-4096).

Scope Enforcement System Prompt:

const systemPrompt = `You are a specialized DevOps and Infrastructure as Code assistant focused exclusively on Kubernetes, Docker, AKS (Azure Kubernetes Service), Azure CLI (az), Istio, Terraform, Helm, and related IaC tools.

STRICT RULES:
1. ONLY answer questions about:
   - Kubernetes (kubectl, resources, YAML, concepts, troubleshooting)
   - Docker (Dockerfile, images, containers, compose, build)
   - AKS (Azure Kubernetes Service)
   - Azure CLI (az commands for AKS, ACR, and related services)
   - Istio (service mesh, VirtualServices, DestinationRules, traffic management)
   - Terraform (HCL, providers, state management, modules, workspaces)
   - Helm (charts, values, releases, templating, repositories)
   - Infrastructure as Code tools (Ansible, Pulumi, CloudFormation, ARM templates when related to Kubernetes/Azure)
   - GitOps tools (ArgoCD, Flux when related to K8s deployments)

2. For ANY other topics, respond with:
   "I can only help with Kubernetes, Docker, AKS, Azure CLI, Istio, Terraform, Helm, and Infrastructure as Code questions. Please ask about those topics."

3. When answering in-scope questions:
   - Provide practical, actionable answers
   - Include relevant command examples in code blocks
   - Be concise but thorough
   - Explain WHY when helpful (not just HOW)
   - Use proper formatting for commands, YAML, and HCL`;

Claude API Call:

const anthropicResponse = await fetch('https://api.anthropic.com/v1/messages', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'x-api-key': env.ANTHROPIC_API_KEY,
    'anthropic-version': '2023-06-01'
  },
  body: JSON.stringify({
    model: 'claude-3-haiku-20240307',
    max_tokens: Math.min(Math.max(maxTokens, 256), 4096),
    temperature: Math.min(Math.max(temperature, 0), 1),
    system: systemPrompt,
    messages: messages
  })
});

Usage Tracking:

const usage = data.usage; // { input_tokens, output_tokens }

const inputCost = (usage.input_tokens / 1000000) * 3;
const outputCost = (usage.output_tokens / 1000000) * 15;
const totalCost = inputCost + outputCost;

return new Response(JSON.stringify({
  content,
  usage: {
    input_tokens: usage.input_tokens,
    output_tokens: usage.output_tokens,
    total_tokens: usage.input_tokens + usage.output_tokens,
    estimated_cost: totalCost.toFixed(4)
  }
}));

Frontend Implementation

React Chat Component

The chat interface lives in src/components/tools/AiAssistant.js with ~680 lines of React code managing state, API calls, Firebase integration, and UI rendering.

State Management:

const [pat, setPat] = useState(() =>
  localStorage.getItem('ai_assistant_pat') || ''
)
const [patHash, setPatHash] = useState(null)
const [chats, setChats] = useState({})
const [currentChatId, setCurrentChatId] = useState(null)
const [messages, setMessages] = useState([])
const [input, setInput] = useState('')
const [loading, setLoading] = useState(false)
const [sessionUsage, setSessionUsage] = useState({
  messages: 0, tokens: 0, cost: 0
})
const [totalUsage, setTotalUsage] = useState({
  messages: 0, tokens: 0, cost: 0
})
const [temperature, setTemperature] = useState(() =>
  parseFloat(localStorage.getItem('ai_assistant_temperature')) || 0.3
)
const [maxTokens, setMaxTokens] = useState(() =>
  parseInt(localStorage.getItem('ai_assistant_max_tokens')) || 2048
)

The PAT, temperature, and maxTokens persist in localStorage. Chat history and usage statistics persist in Firebase Realtime Database.

PAT Hashing for Privacy:

async function hashPat(pat) {
  const encoder = new TextEncoder()
  const data = encoder.encode(pat)
  const hashBuffer = await crypto.subtle.digest('SHA-256', data)
  const hashArray = Array.from(new Uint8Array(hashBuffer))
  return hashArray.map(b => b.toString(16).padStart(2, '0')).join('')
}

The PAT is hashed using SHA-256 before being used as a Firebase database key. This ensures that even if the database is compromised, the actual PAT values remain private.

API Call Flow:

const sendMessage = async () => {
  let chatId = currentChatId
  if (!chatId) {
    chatId = Date.now().toString()
    setCurrentChatId(chatId)
  }

  const newMessages = [...messages, { role: 'user', content: input }]
  setMessages(newMessages)
  setLoading(true)

  try {
    const response = await fetch('/api/ai-chat', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ messages: newMessages, pat, temperature, maxTokens })
    })

    const data = await response.json()
    const finalMessages = [...newMessages, {
      role: 'assistant',
      content: data.content
    }]
    setMessages(finalMessages)

    // Track usage
    let newSessionUsage = sessionUsage
    if (data.usage) {
      newSessionUsage = {
        messages: sessionUsage.messages + 1,
        tokens: sessionUsage.tokens + data.usage.total_tokens,
        cost: sessionUsage.cost + parseFloat(data.usage.estimated_cost)
      }
      setSessionUsage(newSessionUsage)
      updateStatsInFirebase(data.usage)
    }

    // Save chat to Firebase
    const chatData = {
      id: chatId,
      title: generateChatTitle(finalMessages),
      messages: finalMessages,
      usage: newSessionUsage,
      createdAt: chats[chatId]?.createdAt || Date.now(),
      updatedAt: Date.now()
    }
    saveChatToFirebase(chatId, chatData)
  } catch (err) {
    setError(err.message)
  }
}

Firebase Integration & Chat History

Firebase Realtime Database stores all chat conversations and usage statistics, keyed by the SHA-256 hash of your PAT. When you enter your PAT, the app loads your complete chat history from Firebase. Every message is automatically saved, so you can close the browser and come back later to continue any conversation.

The database structure is simple: chats are stored under ai-assistant/sessions/{patHash}/chats/{chatId} with messages, usage stats, and timestamps. Aggregate usage statistics are stored separately under ai-assistant/stats/{patHash}.

Chat History Sidebar:

A separate AiChatHistory component displays all previous conversations in a right sidebar. It lists all chats sorted by most recently updated and shows the chat title (first 40 characters of the first user message). Dates use relative formatting like "Today", "Yesterday", or "X days ago". The currently active chat gets highlighted, each chat has a delete button, and there's a new chat button in the sidebar header.

Configurable AI Parameters

The UI includes sliders in the settings panel to adjust Claude's behavior.

The temperature slider (0 to 1, step 0.1) controls response randomness and creativity. It defaults to 0.3 for consistent, focused technical answers. Lower values between 0 and 0.3 give you more deterministic responses, which works better for kubectl commands and YAML configs. Higher values between 0.7 and 1.0 are more creative and useful for brainstorming solutions.

The max tokens slider (256 to 4096, step 256) controls maximum response length. The default is 2048 tokens, roughly 1500 words. Lower values give you shorter, more concise answers. Higher values let Claude write longer, more detailed explanations.

The context window slider (10 to 50 messages, step 5) controls how many recent messages are sent to the API. It defaults to 20 messages. Lower values like 10 or 15 reduce context but cost less and respond faster. Higher values like 30 or 50 give more context for complex troubleshooting sessions. The full conversation history is always preserved locally and in Firebase regardless of this setting.

All settings persist in localStorage and are sent with each API request. The backend validates and clamps temperature and maxTokens to safe ranges before passing them to the Claude API.

UI Implementation:

<input
  type="range"
  min="0"
  max="1"
  step="0.1"
  value={temperature}
  onChange={(e) => setTemperature(parseFloat(e.target.value))}
  className="ai-chat-settings-slider"
/>

<input
  type="range"
  min="256"
  max="4096"
  step="256"
  value={maxTokens}
  onChange={(e) => setMaxTokens(parseInt(e.target.value))}
  className="ai-chat-settings-slider"
/>

<input
  type="range"
  min="10"
  max="50"
  step="5"
  value={contextWindow}
  onChange={(e) => setContextWindow(parseInt(e.target.value))}
  className="ai-chat-settings-slider"
/>

Sliding Window Context Management

The assistant implements a sliding window to prevent conversations from becoming too expensive as they grow longer. Without this, each new message would include the entire conversation history, causing token usage to compound exponentially.

How it works:

// Apply sliding window: only send last N messages to API
const messagesToSend = newMessages.length <= contextWindow
  ? newMessages
  : newMessages.slice(-contextWindow)

const response = await fetch('/api/ai-chat', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ messages: messagesToSend, pat, temperature, maxTokens })
})

The full conversation history remains stored locally and in Firebase, but only the most recent messages (configurable, default 20) are sent to the Claude API. This keeps costs constant per message since each API call uses roughly the same number of tokens regardless of conversation length. Responses come back faster because smaller context means quicker processing. It also prevents exponential growth where a 50-message conversation would send 1,275 messages total to the API (1+2+3...+50). With a 20-message window, it only sends 695 messages total. Your full conversation is never lost, it's just not all sent to the API each time.

Deployment Configuration

Environment Variables:

Local development uses .dev.vars:

ANTHROPIC_API_KEY=TOP_SECRET...
ASSISTANT_PAT=YOURPAT

Production uses Cloudflare secrets (set via dashboard or CLI).

Firebase Setup:

Create a Firebase project and enable Realtime Database. Head to the Firebase Console and create a new project or use an existing one. Enable Realtime Database in the "Build" section and set the database rules for public read/write since we're using PAT hashing for privacy:

{
  "rules": {
    ".read": true,
    ".write": true
  }
}

Add your Firebase config to src/lib/firebase.js:

const firebaseConfig = {
  apiKey: "YOUR_API_KEY",
  authDomain: "your-project.firebaseapp.com",
  databaseURL: "https://your-project-default-rtdb.firebaseio.com",
  projectId: "your-project"
}

The Firebase SDK is modular and tree-shakeable, adding only ~15-20KB gzipped to the bundle size.

wrangler.jsonc:

{
  "name": "blog",
  "compatibility_date": "2025-12-25",
  "pages_build_output_dir": "./public"
}

Build and Deploy:

# Build Gatsby site
npm run build

# Deploy to Cloudflare Pages
npx wrangler pages deploy public --project-name=blog

The Cloudflare Pages deployment automatically serves the Gatsby static site and makes the Pages Function available at /api/ai-chat.

Usage Statistics

The UI displays real-time usage information in multiple locations.

The footer (always visible after the first message) shows current chat session stats including message count, total tokens, and cost, plus all-time cumulative stats across all chats.

The settings panel displays all-time total usage statistics, the last message's token usage and cost, and an estimate of remaining messages in a $5 API balance.

In fullscreen mode, session stats appear in the header for quick reference.

Each chat conversation tracks its own usage in terms of messages, tokens, and cost. Total usage gets aggregated across all chats and persisted in Firebase. Token counts come from the Claude API response, so they're exact rather than estimates. Cost calculation uses Claude 3 Haiku pricing at $0.25 per million input tokens and $1.25 per million output tokens.

When switching between chats, the session usage stats update to reflect that specific conversation's usage.

Cost Analysis

Based on initial testing, a typical question/answer exchange uses approximately 500-1500 tokens total, costing between $0.0005 and $0.002 per message. A $5 prepaid balance on the Claude API should provide several hundred interactions depending on response length.

Sliding window prevents cost escalation:

Without the sliding window, costs would compound exponentially as conversations grow. Message 1 uses roughly 500 tokens. Message 2 jumps to 1000 tokens because it includes the previous message. By message 10 you're at 5000 tokens, message 20 hits 10,000 tokens, and message 50 balloons to 25,000 tokens per API call.

With the default 20-message sliding window, messages 1 through 20 grow naturally up to around 10,000 tokens. But message 21 and beyond stay constant at roughly 10,000 tokens per API call. This means a 50-message conversation costs roughly the same as two 25-message conversations instead of exponentially more. The sliding window is the difference between spending $0.50 on a long troubleshooting session versus $5 or more.

I went with Claude 3 Haiku (claude-3-haiku-20240307) for the model. I originally planned to use Claude 3.5 Sonnet, but it's not available on the prepaid API tier. Haiku turned out to be sufficient for Kubernetes and Docker questions at a lower cost, and the response times are faster than Sonnet anyway.

There's no rate limiting implemented currently. PAT access control is sufficient for personal use, though I could add rate limiting via Cloudflare Workers KV or Durable Objects if this ever needs to support more users.

Technical Decisions

Why Cloudflare Pages Functions instead of direct API calls:

Going through a backend keeps the API key secure, enables PAT validation, and allows injection of the system prompt without exposing it client-side. Plus it's serverless, so there's no infrastructure to manage.

Why PAT instead of OAuth:

This is a personal use case with no multi-user requirements. A simple PAT implementation means no additional dependencies or auth providers to deal with.

Why Firebase Realtime Database for persistence:

The free tier is generous at 1GB storage and 10GB per month bandwidth. Firebase has real-time sync capabilities that aren't used yet but are there if needed. The key-value structure is perfect for chat storage with no server or database management required. The modular SDK only adds about 15-20KB gzipped to the bundle. I considered Cloudflare KV or Durable Objects but they're more complex for this use case.

Why hash PAT before using as database key:

SHA-256 is a one-way hash, so the PAT can't be recovered from the database even if it gets compromised. Each user's data is isolated by their unique hash without needing an additional authentication or encryption layer.

Why system prompt for scope enforcement:

Claude 3 Haiku is excellent at following instructions. No complex filtering logic needed. It works reliably across the entire conversation context as long as you give it a specific topic list rather than general categories.

Why Claude 3 Haiku instead of Sonnet:

Already covered this above, but the short version is Sonnet isn't available on prepaid, Haiku is sufficient for the use case, and it's faster and cheaper. Match the model capability to your task requirements.

Why sliding window for context management:

The sliding window prevents exponential token growth in long conversations and maintains constant cost per message after reaching the window size. I considered sending the full conversation (simple but expensive for long sessions), conversation compacting or summarization (complex and loses detail), or manual context reset (annoying user experience). The sliding window balances cost, performance, and UX without requiring manual intervention. Users can adjust the window size from 10 to 50 messages based on their needs, and the full history is still preserved for reference and chat switching.

What's Next

Future improvements could include conversation search and filtering, exporting chat history to markdown or JSON, code syntax highlighting in responses, and sharing individual conversations via URL.

Conclusion

Building this took maybe 4-5 hours spread across a weekend for the initial version, plus another few hours to add Firebase persistence and multi-chat support. The result is a specialized AI assistant that lives at /tools/ai on this blog, answers DevOps questions without distraction, remembers all your conversations, tracks its own costs, and feels like a native part of my workflow.

The architecture is elegantly simple: a static React frontend, a serverless Cloudflare Function as a thin proxy, Claude's API doing the heavy lifting, and Firebase Realtime Database storing chat history. No authentication servers, no complex infrastructure, no container orchestration. Just four components working together.

What surprised me most was how well the system prompt enforcement works. I expected to need keyword filtering or some kind of ML-based topic classification. Nope, just clear instructions in natural language, and Claude 3 Haiku follows them reliably. The Firebase integration was similarly straightforward. Hash the PAT for privacy, save chats on each message, load them on mount. Done.

The persistent chat history turned out to be more valuable than I expected. Being able to revisit past troubleshooting sessions or reference previous kubectl commands makes the tool feel less like a chatbot and more like a personal knowledge base that happens to answer questions.

If you're building something similar, here's my advice: start simple, match the model to the task, and don't over-engineer. Add features iteratively based on actual usage. The best tools are the ones that disappear, letting you focus on the actual problem you're trying to solve.

Building Golden Paths with Backstage: Part 3 - Production from Day One

Sat, 03 Jan 2026 00:00:00 GMT

In Part 1, we built the foundation: a shared development cluster with namespace isolation and a quick-start template that gets developers from zero to deployed in under 5 minutes. In Part 2, we added preview environments that automatically spin up for every pull request.

Now it's time for production.

The Wrong Approach (Twice)

My first attempt at a production template was basically a copy of the quick-start template that pointed to a different cluster. Create a new repository, scaffold a new application, deploy to production. That was obviously wrong.

My second attempt was a "promote to production" template. A developer who already had a service in dev would run a separate template that created production infrastructure for their existing repo. Better, but still wrong.

Here's what I realized: if you know a service will eventually need production infrastructure, why not create it from the start?

The Insight: Production is Just Configuration

The difference between development and production isn't the code or the repository. It's configuration:

Aspect	Development	Production
Cluster	shared-dev-cluster	aks-app-spoke
Namespace	team-dev	team-prod
Image tag	`latest`	`v1.0.0` (semver)
Replicas	2	3+
HPA	disabled	enabled
URL	`{service}.chrishouse.io`	`{service}-prod.chrishouse.io`
Routing	Hub → East-West Gateway	Hub → Spoke Gateway

If all of this is just YAML configuration, why can't we generate it all at once?

One Template, Both Environments

Instead of two separate templates (quick-start + promote), I merged them into a single golden path template with an optional production toggle:

parameters:
  - title: Service Details
    properties:
      service_name:
        title: Service Name
        type: string
      team:
        title: Team
        type: string
        ui:field: OwnerPicker
      description:
        title: Description
        type: string

  - title: Production Configuration
    properties:
      includeProduction:
        title: Include Production Infrastructure
        type: boolean
        default: true
        description: Create production ArgoCD app, namespace, and ingress routes

      prodReplicas:
        title: Production Replicas
        type: integer
        default: 3
        minimum: 2

      enableHPA:
        title: Enable Horizontal Pod Autoscaling
        type: boolean
        default: true

The magic is in that includeProduction checkbox. Default is true, but developers can uncheck it if they're just experimenting.

What Gets Created

When a developer runs the template with production enabled, a single PR contains everything:

infra/
├── quickstart-services/my-service/
│   ├── namespace-claim.yaml        # Dev namespace
│   ├── namespace-claim-prod.yaml   # Prod namespace
│   ├── argocd-application.yaml     # Dev ArgoCD app (tracks latest)
│   ├── argocd-application-prod.yaml # Prod ArgoCD app (tracks semver)
│   └── README.md
└── kubernetes/
    └── istio-hub/
        ├── virtualservice-my-service-dev.yaml   # Dev routing
        └── virtualservice-my-service-prod.yaml  # Prod routing

Six files. One PR. Both environments ready.

The template itself has separate folders (gitops-manifests-dev/ and gitops-manifests-prod/) as building blocks that get conditionally included, but the output is unified - everything lands in the same PR.

Compare this to the two-template approach:

Run quick-start template → PR #1
Wait for approval and merge
Develop for a while
Run promote template → PR #2
Wait for approval and merge

Now it's just:

Run golden path template → PR #1
Done

Conditional Steps in Backstage

The template uses conditional steps to skip production resources when not needed:

steps:
  # Always runs - creates dev infrastructure
  - id: create-gitops-dev
    name: Create Dev GitOps Configuration
    action: fetch:template
    input:
      url: ./gitops-manifests-dev
      targetPath: ./gitops/quickstart-services/${{ parameters.service_name }}

  # Only runs if production is enabled
  - id: create-gitops-prod
    name: Create Prod GitOps Configuration
    if: ${{ parameters.includeProduction }}
    action: fetch:template
    input:
      url: ./gitops-manifests-prod
      targetPath: ./gitops/quickstart-services/${{ parameters.service_name }}

  # Only runs if production is enabled
  - id: create-hub-vs
    name: Create Hub Ingress Route
    if: ${{ parameters.includeProduction }}
    action: fetch:template
    input:
      url: ./hub-virtualservice
      targetPath: ./gitops/kubernetes/istio-hub

The if clause makes steps conditional. When includeProduction is false, the production steps are skipped entirely.

The Production ArgoCD Application

The production app uses ArgoCD Image Updater for automatic version detection:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-service-prod
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/image-list: "app=ghcr.io/crh225/my-service"
    argocd-image-updater.argoproj.io/app.update-strategy: "semver"
    argocd-image-updater.argoproj.io/app.allow-tags: "regexp:^v[0-9]+\\.[0-9]+\\.[0-9]+$"
spec:
  source:
    repoURL: https://github.com/crh225/my-service
    targetRevision: main
    path: helm
    helm:
      parameters:
        - name: image.tag
          value: "v1.0.0"  # Initial version
        - name: replicaCount
          value: "3"
        - name: autoscaling.enabled
          value: "true"
  destination:
    name: aks-app-spoke
    namespace: team-prod

This creates a trunk-based release flow:

Development: Push to main → builds latest → auto-deploys
Production: Create GitHub Release → builds v1.x.x → Image Updater detects and deploys

Traffic Flow

Both dev and prod environments get external URLs through the Istio mesh. The key difference is which cluster receives the traffic.

Development Traffic

https://my-service.chrishouse.io
         │
         ▼
┌─────────────────────────────────────────┐
│ Hub Cluster (aks-mgmt-hub)              │
│                                         │
│ main-gateway                            │
│     │                                   │
│     ▼                                   │
│ VirtualService: my-service-dev-vs       │
│     │                                   │
│     ▼                                   │
│ ServiceEntry: shared-dev.internal       │
│     → shared-dev east-west gateway IP   │
└─────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────┐
│ Shared Dev Cluster                      │
│                                         │
│ cross-network-gateway (east-west)       │
│     │                                   │
│     ▼                                   │
│ VirtualService: my-service              │
│     │                                   │
│     ▼                                   │
│ my-service.team-dev.svc.cluster.local   │
└─────────────────────────────────────────┘

Production Traffic

https://my-service-prod.chrishouse.io
         │
         ▼
┌─────────────────────────────────────────┐
│ Hub Cluster (aks-mgmt-hub)              │
│                                         │
│ main-gateway                            │
│     │                                   │
│     ▼                                   │
│ VirtualService: my-service-prod-vs      │
│     │                                   │
│     ▼                                   │
│ ServiceEntry: spoke.internal            │
│     → app-spoke gateway IP              │
└─────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────┐
│ App Spoke Cluster (aks-app-spoke)       │
│                                         │
│ cross-network-gateway (east-west)       │
│     │                                   │
│     ▼                                   │
│ VirtualService: my-service              │
│     │                                   │
│     ▼                                   │
│ my-service.team-prod.svc.cluster.local  │
└─────────────────────────────────────────┘

Both services are immediately accessible at their URLs after ArgoCD syncs. No port-forwarding, no VPN - just HTTPS to a real domain.

Why This Matters

This change is more significant than it might appear:

1. Reduced Cognitive Load

Developers don't need to learn two templates. There's one golden path that handles everything. The mental model is simpler: "run the template, get a service."

2. Production-Ready from Day One

Even if a developer doesn't release to production immediately, the infrastructure is waiting. When they're ready, it's just:

Create a GitHub Release with tag v1.0.0
CI builds the image
ArgoCD deploys to production

No additional PRs, no waiting for platform team approval again.

3. Consistent Configuration

When dev and prod are created together, they're guaranteed to be consistent. Same Helm chart, same value patterns, same team ownership. No drift between what was created in dev months ago and what gets created for prod now.

4. Faster Time to Production

The two-template approach had a hidden cost: the second template was often run weeks or months after the first. By then, developers had forgotten the exact configuration, team names had changed, and the promotion became a mini-project.

With everything created upfront, the path to production is just a git tag away.

5. Optional Complexity

The checkbox makes it optional. Experimenting with something that will never go to production? Uncheck the box. Building a real service? Leave it checked (the default).

The Complete Golden Path

The journey from idea to production is now:

Stage	Action	What Happens
Create	Run golden path template	Dev + Prod infrastructure created
Develop	Push to main	Auto-deploys to dev cluster
Test	Open PR	Preview environment spins up
Release	Create GitHub Release v1.0.0	Production deploys automatically

One template. One PR. Both environments. Production when you're ready.

Developer Experience

From the developer's perspective:

Open Backstage and select "Golden Path - Node.js Microservice"
Enter service details (name, team, description)
Configure production (replicas, HPA) or uncheck to skip
Click create

The PR includes everything. After merge:

Dev deploys immediately on every push to main
Production waits for a GitHub Release

When ready for production:

Go to your repository
Create a Release with tag v1.0.0
Wait 2-3 minutes
Service is live at {service}-prod.chrishouse.io

Series Progress

Part 1: Foundation - Shared clusters, namespace isolation, golden path template
Part 2: Preview Environments - Ephemeral PR environments with Istio multi-cluster routing
Part 3: Production from Day One (this post) - Optional production infrastructure in the initial template

Key Takeaways

Merge, don't separate. If dev and prod use the same code, create their infrastructure together.
Make it optional. A checkbox is all you need. Default to "yes" for real services.
Reduce the number of workflows. Every additional template is cognitive overhead. One template that does everything is better than specialized templates.
Production-ready doesn't mean production-deployed. Create the infrastructure upfront; deploy when ready via git tags.
Time to production matters. The faster developers can go from idea to production, the more value they deliver. Remove every unnecessary step.

The golden path is now truly golden: one template creates everything a service needs, from development through production. The only thing left is writing the code.

Implementation Repository

Full implementation: github.com/crh225/ARMServicePortal

Key files:

Golden path template: backstage/templates/nodejs-quickstart/template.yaml
Dev manifests: backstage/templates/nodejs-quickstart/gitops-manifests-dev/
Prod manifests: backstage/templates/nodejs-quickstart/gitops-manifests-prod/
Hub VirtualService template: backstage/templates/nodejs-quickstart/hub-virtualservice/
Spoke VirtualService template: backstage/templates/nodejs-quickstart/spoke-virtualservice/

Adding Auth0 Authentication to Backstage: A Complete Guide

Fri, 26 Dec 2025 00:00:00 GMT

Backstage's default guest authentication works well for development, but it's not designed for production use. The dangerouslyAllowOutsideDevelopment: true flag exists as a workaround, but as the name suggests, it's not ideal for real deployments. Auth0's free tier offers a straightforward way to add proper identity management without the complexity of enterprise SSO solutions.

What we're building:

Auth0 OAuth integration with Backstage
User entity resolution (matching Auth0 emails to catalog users)
Production-ready Kubernetes deployment with Azure Key Vault secrets
Guest fallback only in development mode

Prerequisites

Backstage app (new backend system)
Auth0 account (free tier works)
For production: Kubernetes cluster with CSI Secret Store Driver

Part 1: Auth0 Setup

Create an Auth0 Application

Log into Auth0 Dashboard
Navigate to Applications > Applications > Create Application
Select Regular Web Application
Name it something like "Backstage"

Configure Callback URLs

In your Auth0 application settings, add:

Allowed Callback URLs:

http://localhost:7007/api/auth/auth0/handler/frame
https://backstage.yourdomain.com/api/auth/auth0/handler/frame

Allowed Logout URLs:

http://localhost:3000
https://backstage.yourdomain.com

Allowed Web Origins:

http://localhost:3000
https://backstage.yourdomain.com

Note Your Credentials

From the Auth0 application settings, grab:

Domain (e.g., dev-xxxxx.us.auth0.com)
Client ID
Client Secret

Part 2: Backend Configuration

Install the Auth0 Provider Module

cd packages/backend
yarn add @backstage/plugin-auth-backend-module-auth0-provider

Register the Module

In packages/backend/src/index.ts, add the Auth0 module:

import { createBackend } from '@backstage/backend-defaults';

const backend = createBackend();

// Core plugins
backend.add(import('@backstage/plugin-app-backend'));
backend.add(import('@backstage/plugin-catalog-backend'));
backend.add(import('@backstage/plugin-scaffolder-backend'));

// Auth plugins
backend.add(import('@backstage/plugin-auth-backend'));
// Auth0 provider - production authentication
backend.add(import('@backstage/plugin-auth-backend-module-auth0-provider'));
// Guest provider - development convenience only
backend.add(import('@backstage/plugin-auth-backend-module-guest-provider'));

// ... other plugins

backend.start();

Configure app-config.yaml

Add the Auth0 configuration:

auth:
  environment: development
  session:
    secret: ${AUTH_SESSION_SECRET}
  providers:
    # Guest provider for local development only
    guest: {}
    # Auth0 OAuth provider
    auth0:
      development:
        clientId: ${AUTH_AUTH0_CLIENT_ID}
        clientSecret: ${AUTH_AUTH0_CLIENT_SECRET}
        domain: ${AUTH_AUTH0_DOMAIN}
        signIn:
          resolvers:
            # Match Auth0 email with Backstage User entity profile email
            - resolver: emailMatchingUserEntityProfileEmail
            # Fallback: Match email local part (before @) with User entity name
            - resolver: emailLocalPartMatchingUserEntityName

The sign-in resolvers determine how Auth0 users get mapped to Backstage User entities. The first resolver that finds a match is used.

Part 3: Frontend Configuration

Create the Auth0 API Reference

Backstage doesn't export an auth0AuthApiRef from the core packages, so you'll need to create one yourself. This API reference connects the frontend to the Auth0 OAuth flow. In packages/app/src/apis.ts:

import {
  createApiRef,
  ApiRef,
  OpenIdConnectApi,
  ProfileInfoApi,
  BackstageIdentityApi,
  SessionApi,
  configApiRef,
  discoveryApiRef,
  oauthRequestApiRef,
  createApiFactory,
} from '@backstage/core-plugin-api';
import { OAuth2 } from '@backstage/core-app-api';

// Create Auth0 API reference (not exported from core-plugin-api)
export const auth0AuthApiRef: ApiRef<
  OpenIdConnectApi & ProfileInfoApi & BackstageIdentityApi & SessionApi
> = createApiRef({
  id: 'internal.auth.auth0',
});

export const apis: AnyApiFactory[] = [
  // ... existing APIs

  // Auth0 OAuth API factory
  createApiFactory({
    api: auth0AuthApiRef,
    deps: {
      discoveryApi: discoveryApiRef,
      oauthRequestApi: oauthRequestApiRef,
      configApi: configApiRef,
    },
    factory: ({ discoveryApi, oauthRequestApi, configApi }) =>
      OAuth2.create({
        discoveryApi,
        oauthRequestApi,
        provider: {
          id: 'auth0',
          title: 'Auth0',
          icon: () => null,
        },
        environment: configApi.getOptionalString('auth.environment'),
        defaultScopes: ['openid', 'profile', 'email'],
      }),
  }),
];

Configure the SignInPage

In packages/app/src/App.tsx, configure the sign-in page to use the Auth0 provider:

import { SignInPage } from '@backstage/core-components';
import { auth0AuthApiRef } from './apis';

const app = createApp({
  // ... other config
  components: {
    SignInPage: props => (
      <SignInPage
        {...props}
        providers={[
          {
            id: 'auth0',
            title: 'Auth0',
            message: 'Sign in with Auth0',
            apiRef: auth0AuthApiRef,
          },
        ]}
        title="Your App Name"
      />
    ),
  },
});

The built-in SignInPage component handles the OAuth popup flow automatically. When users click the sign-in button, a popup window opens for Auth0 authentication, then redirects back to your app once complete.

Part 4: User Entity Setup

Auth0 users must match Backstage User entities for sign-in to work. Create or update examples/org.yaml:

---
apiVersion: backstage.io/v1alpha1
kind: User
metadata:
  name: chris
spec:
  profile:
    displayName: Chris House
    email: chris@example.com  # Must match Auth0 user email
  memberOf: [platform-team]
---
apiVersion: backstage.io/v1alpha1
kind: Group
metadata:
  name: platform-team
  description: Platform engineering team
spec:
  type: team
  children: []

Configure Catalog to Load Users

In app-config.yaml, ensure User entities are allowed:

catalog:
  rules:
    # Include User and Group in global rules
    - allow: [Component, System, API, Resource, Location, Template, User, Group]
  locations:
    - type: file
      target: ./examples/org.yaml
      rules:
        - allow: [User, Group]

Part 5: Environment Variables

Local Development

Create a .env file (add to .gitignore):

AUTH_AUTH0_DOMAIN=dev-xxxxx.us.auth0.com
AUTH_AUTH0_CLIENT_ID=your-client-id
AUTH_AUTH0_CLIENT_SECRET=your-client-secret
AUTH_SESSION_SECRET=a-random-32-character-string-here

Backstage doesn't auto-load .env files. Either:

Use dotenv-cli: yarn add -D dotenv-cli and run dotenv yarn start-backend
Set environment variables directly in your shell

For PowerShell, add to your profile:

$env:AUTH_AUTH0_DOMAIN="dev-xxxxx.us.auth0.com"
$env:AUTH_AUTH0_CLIENT_ID="your-client-id"
$env:AUTH_AUTH0_CLIENT_SECRET="your-client-secret"
$env:AUTH_SESSION_SECRET="your-session-secret"

Part 6: Production Deployment

Add Secrets to Azure Key Vault

az keyvault secret set --vault-name your-vault --name AUTH-AUTH0-DOMAIN --value "dev-xxxxx.us.auth0.com"
az keyvault secret set --vault-name your-vault --name AUTH-AUTH0-CLIENT-ID --value "your-client-id"
az keyvault secret set --vault-name your-vault --name AUTH-AUTH0-CLIENT-SECRET --value "your-client-secret"
az keyvault secret set --vault-name your-vault --name AUTH-SESSION-SECRET --value "$(openssl rand -hex 32)"

Configure SecretProviderClass

Update your CSI Secret Store configuration:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-backstage-secrets
  namespace: backstage
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    clientID: "your-managed-identity-client-id"
    keyvaultName: "your-vault"
    tenantId: "your-tenant-id"
    objects: |
      array:
        # ... existing secrets
        # Auth0 authentication secrets
        - |
          objectName: AUTH-AUTH0-DOMAIN
          objectType: secret
        - |
          objectName: AUTH-AUTH0-CLIENT-ID
          objectType: secret
        - |
          objectName: AUTH-AUTH0-CLIENT-SECRET
          objectType: secret
        - |
          objectName: AUTH-SESSION-SECRET
          objectType: secret
  secretObjects:
    - secretName: backstage-secrets
      type: Opaque
      data:
        # ... existing mappings
        # Auth0 secrets
        - objectName: AUTH-AUTH0-DOMAIN
          key: AUTH_AUTH0_DOMAIN
        - objectName: AUTH-AUTH0-CLIENT-ID
          key: AUTH_AUTH0_CLIENT_ID
        - objectName: AUTH-AUTH0-CLIENT-SECRET
          key: AUTH_AUTH0_CLIENT_SECRET
        - objectName: AUTH-SESSION-SECRET
          key: AUTH_SESSION_SECRET

Update Deployment/Rollout

Add environment variables to your pod spec:

env:
  # Auth0 authentication secrets
  - name: AUTH_AUTH0_DOMAIN
    valueFrom:
      secretKeyRef:
        name: backstage-secrets
        key: AUTH_AUTH0_DOMAIN
  - name: AUTH_AUTH0_CLIENT_ID
    valueFrom:
      secretKeyRef:
        name: backstage-secrets
        key: AUTH_AUTH0_CLIENT_ID
  - name: AUTH_AUTH0_CLIENT_SECRET
    valueFrom:
      secretKeyRef:
        name: backstage-secrets
        key: AUTH_AUTH0_CLIENT_SECRET
  - name: AUTH_SESSION_SECRET
    valueFrom:
      secretKeyRef:
        name: backstage-secrets
        key: AUTH_SESSION_SECRET

Production Config

In app-config.production.yaml:

auth:
  environment: production
  session:
    secret: ${AUTH_SESSION_SECRET}
  providers:
    # No guest provider in production
    auth0:
      production:
        clientId: ${AUTH_AUTH0_CLIENT_ID}
        clientSecret: ${AUTH_AUTH0_CLIENT_SECRET}
        domain: ${AUTH_AUTH0_DOMAIN}
        sessionDuration: { hours: 24 }
        signIn:
          resolvers:
            - resolver: emailMatchingUserEntityProfileEmail
            - resolver: emailLocalPartMatchingUserEntityName

catalog:
  locations:
    # Load users from GitHub in production
    - type: url
      target: https://github.com/your-org/your-repo/blob/main/backstage/examples/org.yaml
      rules:
        - allow: [User, Group]

Testing the Integration

Start Backstage:

yarn start-backend  # Terminal 1
yarn start          # Terminal 2

Verify User entity loaded:

http://localhost:7007/api/catalog/entities?filter=kind=user

Click Auth0 login and sign in with a user whose email matches a User entity
Check the catalog to verify you're signed in as the correct user

Troubleshooting

There are several moving parts involved in getting Auth0 working with Backstage. Here are some common issues and their solutions.

This means Auth0 authenticated successfully, but no matching User entity was found. Check:

User entity exists in catalog (check /api/catalog/entities?filter=kind=user)
Email matches exactly between Auth0 profile and User entity spec.profile.email
Catalog rules include User and Group types
Wait for catalog sync after adding new User entities

"Auth0 misconfigured"

Check:

Environment variables are set (not just in .env file)
Callback URL in Auth0 matches your Backstage URL
Domain doesn't include https:// prefix

Guest option still shows in production

Verify NODE_ENV=production is set in your deployment. The conditional spread only hides guest when this is set.

User entities not loading from GitHub URLs

If your catalog logs show:

Unable to read url, NotAllowedError: Reading from 'https://raw.githubusercontent.com/...' is not allowed

You need to add raw.githubusercontent.com to the backend reading allow list:

backend:
  # ... other config
  reading:
    allow:
      - host: raw.githubusercontent.com

This is separate from the GitHub integration. The catalog reader needs its own permission to fetch from raw GitHub URLs.

Local file paths not loading User entities

File-based catalog locations use paths relative to the config file location, which can sometimes be confusing. Using GitHub raw URLs tends to be more straightforward:

catalog:
  locations:
    # Use raw GitHub URLs instead of local file paths
    - type: url
      target: https://raw.githubusercontent.com/your-org/your-repo/main/backstage/examples/entities.yaml
      rules:
        - allow: [User, Group]

GitHub rate limiting

If catalog processing fails with 403 errors, you're hitting GitHub's rate limit. Set GITHUB_TOKEN in your environment:

integrations:
  github:
    - host: github.com
      token: ${GITHUB_TOKEN}

Even for public repos, an authenticated token gets 5000 requests/hour vs 60 for anonymous.

SQLite database errors in development

If you see errors about connection.filename not being supported, use the in-memory database:

backend:
  database:
    client: better-sqlite3
    connection: ':memory:'  # Not { filename: './file.db' }

The new backend system uses a Knex wrapper that handles database connections differently than earlier versions.

Plugins blocking backend startup

Some plugins (like ArgoCD) will block startup if their external services aren't configured. For local development, comment them out:

// argocd plugin - requires argocd config, disabled for local dev
// backend.add(import('@roadiehq/backstage-plugin-argo-cd-backend'));

Summary

Auth0's free tier provides a good middle ground between guest authentication and enterprise SSO. The main components are:

Backend module - handles the OAuth flow and token exchange
Frontend API reference - connects the sign-in page to Auth0
Sign-in resolvers - map Auth0 users to catalog User entities
Environment configuration - keeps development and production settings separate

This approach works for both local development and production Kubernetes deployments, with secrets managed through Azure Key Vault.

Building Golden Paths with Backstage: Part 2 - Preview Environments

Tue, 23 Dec 2025 00:00:00 GMT

In Part 1, we built the foundation: a shared development cluster, namespace isolation, and a Backstage template that deploys a running service in under 5 minutes.

Today we add something developers actually love: preview environments.

If you've ever waited for a staging environment to free up so you could test your changes, or merged a PR only to find it broke something obvious that would have been caught with a quick manual test, you understand the problem. Shared staging environments create bottlenecks. Developers queue up behind each other, or worse, deploy over each other's changes and then spend time debugging phantom issues.

Preview environments solve this by giving every pull request its own isolated deployment with a unique URL. Code reviewers can click a link in the PR and see the actual running application. Not screenshots, not local recordings, the real thing. QA can test changes before they hit the main branch. Product managers can review features without asking developers to deploy something special for them.

The concept is simple: PR opens, environment spins up, PR closes, environment disappears. The implementation? That's where it gets interesting.

Here's what we built, and what broke along the way.

What We're Building

When a developer opens a pull request:

GitHub Actions builds the PR branch into a container image
A GitOps workflow creates an ephemeral namespace
ArgoCD deploys the preview environment
A unique URL is generated: pricing-api-pr-1-red.chrishouse.io
The PR gets a comment with the preview link
When the PR closes, everything is cleaned up automatically

The hard part isn't the workflow. It's the routing.

The Architecture Challenge

Here's where our design decisions from Part 1 created an interesting puzzle. We have a hub-spoke cluster topology:

Hub Cluster: Handles all external ingress, runs ArgoCD, Crossplane, Backstage
Dev Spoke Cluster: Runs application workloads, has no external ingress

This separation is intentional. The hub cluster is the control plane: it manages infrastructure, handles GitOps, and serves as the single entry point from the internet. The spoke cluster runs actual workloads, isolated from the management plane. It's a common pattern for enterprise Kubernetes deployments where you want to keep your cattle separate from your pets.

But preview environments run on the dev spoke. And DNS points to the hub.

Traffic flow needs to be:

Internet → Hub Istio Gateway → ??? → Dev Spoke → Preview Pod

That middle part is the question mark we need to solve. How do you route traffic from one cluster to another when they're on different networks? The clusters can talk to each other through VNet peering, but Kubernetes services don't automatically span clusters. The hub's Istio gateway has no native way to forward traffic to a service running in a completely different cluster.

This is a solved problem in the Kubernetes ecosystem. There are several approaches, but each comes with tradeoffs. Let's walk through what we tried.

Phase 1: The Preview Workflow

Before tackling the routing problem, let's set up the workflow that will trigger everything. This part is relatively straightforward: a GitHub Actions workflow that fires on pull request events.

The workflow lives in the Backstage template skeleton, which means every service created through the golden path automatically gets preview environment support. Developers don't have to configure anything; it just works.

# .github/workflows/preview.yml
name: Preview Environment

on:
  pull_request:
    types: [opened, synchronize, reopened, closed]

env:
  REGISTRY: ghcr.io
  SERVICE_NAME: pricing-api
  TEAM_NAME: red
  GITOPS_REPO: crh225/ARMServicePortal

jobs:
  deploy-preview:
    if: github.event.action != 'closed'
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      pull-requests: write

    steps:
      - uses: actions/checkout@v3

      - name: Set environment variables
        id: vars
        run: |
          echo "pr_number=${{ github.event.pull_request.number }}" >> $GITHUB_OUTPUT
          echo "namespace=${TEAM_NAME}-dev-pr-${{ github.event.pull_request.number }}" >> $GITHUB_OUTPUT
          echo "hostname=${SERVICE_NAME}-pr-${{ github.event.pull_request.number }}-${TEAM_NAME}.chrishouse.io" >> $GITHUB_OUTPUT
          echo "image_tag=pr-${{ github.event.pull_request.number }}" >> $GITHUB_OUTPUT

      - name: Build and push preview image
        uses: docker/build-push-action@v4
        with:
          push: true
          tags: ghcr.io/${{ github.repository }}:pr-${{ github.event.pull_request.number }}

      - name: Create GitOps manifests
        run: |
          # Creates NamespaceClaim and ArgoCD Application
          # Commits to ARMServicePortal repo
          # ArgoCD discovers and deploys

The workflow creates two files in the platform repository:

NamespaceClaim - Creates the isolated namespace:

apiVersion: platform.chrishouse.io/v1alpha1
kind: NamespaceClaim
metadata:
  name: red-dev-pr-1
  labels:
    ephemeral: "true"
    pr-number: "1"
spec:
  parameters:
    targetCluster: shared-dev-cluster
    namespaceName: red-dev-pr-1
    enableResourceQuota: true

ArgoCD Application - Deploys the preview:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: pricing-api-pr-1
  namespace: argocd
  labels:
    ephemeral: "true"
spec:
  source:
    repoURL: https://github.com/crh225/pricing-api
    targetRevision: feature-branch
    path: helm
    helm:
      parameters:
        - name: image.tag
          value: pr-1
        - name: istio.preview.enabled
          value: "true"
        - name: istio.preview.hostname
          value: pricing-api-pr-1-red.chrishouse.io
  destination:
    name: shared-dev-cluster
    namespace: red-dev-pr-1

The workflow uses a personal access token (GITOPS_TOKEN) to commit to the platform repository. This feels like a hack but is actually the standard GitOps pattern: your application repo triggers changes in your infrastructure repo, and your GitOps tool (ArgoCD in our case) picks up those changes and applies them.

This part worked immediately. Within a couple minutes of opening a PR, pods were running and the service was created. The Crossplane NamespaceClaim created the isolated namespace, ArgoCD deployed the application, and we had a working preview environment.

Except nobody could reach it. Now comes routing.

Phase 2: The Routing Problem

This is the part that took the longest to solve. Not because the concepts are hard, but because cloud networking has a way of surprising you with limitations that seem arbitrary until you understand the underlying infrastructure.

Attempt 1: Internal LoadBalancer

The obvious first attempt: put an Istio east-west gateway on the dev spoke with an internal (private) LoadBalancer. The hub cluster is on the same Azure VNet peering, so it should be able to reach an internal IP in the dev spoke's VNet.

service:
  type: LoadBalancer
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"

Hub cluster gets the internal IP. ServiceEntry points to it. VirtualService routes preview traffic.

Result:

$ curl http://10.1.0.158/health
# ... hangs ...
# 100% packet loss

Root cause: Azure internal LoadBalancers are not accessible across VNet peering by default. The hub cluster (10.0.0.0/16) cannot reach the dev spoke's internal LoadBalancer (10.1.0.158 in 10.1.0.0/16).

This was frustrating because everything looked correct. VNet peering was connected and showing "Connected" status in the Azure portal. NSG rules allowed VirtualNetwork traffic in both directions. Route tables looked fine. I spent a good hour checking every setting.

The issue is that Azure internal LoadBalancers use a different networking path than regular VNet traffic. They're implemented with a load balancer frontend IP that lives in a special Azure networking plane, and that plane doesn't traverse VNet peering without additional configuration. The VM-to-VM traffic works fine over peering; the traffic to the LoadBalancer frontend IP doesn't.

Azure networking strikes again.

Attempt 2: Azure Private Link Service

The Azure-recommended solution for this exact problem is Private Link. You create a Private Link Service that fronts the internal LoadBalancer, then create a Private Endpoint in the hub VNet that connects to that service. Traffic flows through Azure's backbone, never touching the public internet, and you get a private IP in the hub VNet that routes to the dev spoke's LoadBalancer.

Pros: Proper Azure-native cross-VNet LoadBalancer access. Fully private. Works reliably.

Cons: Adds complexity (two more Azure resources to manage), cost (~$7/month for the private endpoint), and another moving part that can break. For production environments handling sensitive traffic, this is the right answer. For dev preview environments where I'm trying to minimize costs, it felt like overkill.

Attempt 3: NodePort with Hardcoded IPs

I considered bypassing the LoadBalancer entirely. Kubernetes NodePort services expose a port on every node's IP address. Since VNet peering does work for node-to-node traffic, I could add all the dev spoke's node IPs to the ServiceEntry.

endpoints:
  - address: "10.1.0.4"  # node1
  - address: "10.1.0.5"  # node2
  - address: "10.1.0.6"  # node3

Pros: Works with VNet peering. No additional Azure services needed.

Cons: Hardcoded IPs. The moment the cluster scales up, scales down, or nodes get replaced (which happens regularly with AKS upgrades), this breaks. I'd have to build automation to keep the ServiceEntry in sync with the node pool, and that felt like fighting Kubernetes rather than working with it. Not acceptable for anything beyond a quick test.

Attempt 4: Istio Multi-Cluster with Remote Secrets

Istio was designed for exactly this scenario. Istio's multi-cluster support allows service meshes to span multiple Kubernetes clusters, with traffic routed seamlessly between them. It is a way how large organizations run Istio across regions, clouds, and network boundaries.

This is what worked.

Phase 3: Istio Multi-Cluster

The Concept

Istio multi-cluster is one of those features that sounds complex but solves a real problem elegantly. At its core, it allows Istio's control plane (istiod) to discover and route to services running in other clusters, as if they were local services.

The key insight is that cross-cluster communication doesn't have to be complicated if your service mesh understands the topology. Instead of manually configuring routing rules and endpoints, you tell Istio "here's another cluster you should know about" and it handles the rest.

The key components:

Remote Secrets: Kubeconfig credentials that allow istiod in one cluster to query the Kubernetes API of another cluster. Once istiod can list services and endpoints in the remote cluster, it can route traffic there.
East-West Gateway: A dedicated ingress point for cross-cluster traffic. Unlike the north-south gateway that handles external traffic, the east-west gateway handles internal mesh traffic between clusters.
Network Topology Labels: Labels on the istio-system namespace that tell Istio which network each cluster belongs to. This helps Istio understand when traffic needs to cross a network boundary.

Implementation

Step 1: Label namespaces with network topology

# Hub cluster
kubectl label namespace aks-istio-system \
  topology.istio.io/network=hub-network

# Dev spoke
kubectl label namespace istio-system \
  topology.istio.io/network=shared-dev-network

Step 2: Create remote secrets

Remote secrets allow istiod in one cluster to discover services running in another cluster. The Istio documentation covers this well. Use istioctl create-remote-secret to generate the secret for each cluster, then apply it to the other cluster's Istio namespace.

See the Istio multi-cluster installation guide for the complete setup process.

Step 3: Change east-west gateway to public LoadBalancer

Here's where we make the pragmatic choice. The internal LoadBalancer didn't work due to Azure's networking model, and Private Link adds cost and complexity. So we use a public LoadBalancer instead.

# istio-eastwest-gateway-argocd-app.yaml
service:
  type: LoadBalancer
  annotations:
    # Removed: azure-load-balancer-internal annotation
    service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz/ready

Result: East-west gateway gets public IP 52.255.217.180.

This means the east-west gateway is technically internet-accessible. I'll discuss the security implications later, but the short version is: for a dev environment with no sensitive data, it's an acceptable tradeoff.

Why public instead of private?

This is a cost/complexity tradeoff. The "proper" Azure solution would be:

Create an Azure Private Link Service exposing the east-west gateway
Create a Private Endpoint in the hub VNet
Route through the private endpoint

That adds ~$7/month for the private endpoint, plus complexity. For a personal lab environment running on my credit card, the public LoadBalancer at ~$3.65/month is acceptable.

Security implications:

The east-west gateway is now internet-accessible on port 80. However:

It only routes traffic to services with explicit VirtualService configurations
Services require the correct Host header to match
No sensitive data is exposed without intentional configuration
This is a dev environment for preview URLs, not production traffic

For production environments, I'd recommend:

Azure Private Link for fully private cross-cluster routing
Cilium ClusterMesh when Azure CNI adds support (blocked by GitHub issue #5194)
IP whitelisting on the east-west gateway if public access is required

Step 4: Configure hub routing

ServiceEntry tells the hub where to find the dev spoke:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: shared-dev-cluster
  namespace: istio-ingress
spec:
  hosts:
    - shared-dev.internal
  location: MESH_EXTERNAL
  ports:
    - number: 80
      name: http
      protocol: HTTP
  resolution: STATIC
  endpoints:
    - address: "52.255.217.180"  # East-west gateway public IP

VirtualService routes preview traffic to the ServiceEntry:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: preview-envs-vs
  namespace: istio-ingress
spec:
  hosts:
    - "*.chrishouse.io"
  gateways:
    - main-gateway
  http:
    - match:
        - headers:
            ":authority":
              regex: ".*-pr-[0-9]+-[a-z0-9]+\\.chrishouse\\.io"
      route:
        - destination:
            host: shared-dev.internal
            port:
              number: 80

The regex matches preview URLs like pricing-api-pr-1-red.chrishouse.io.

Step 5: Update Helm template for preview VirtualServices

The application's VirtualService needs to reference both the hub gateway and the cross-network gateway:

# helm/templates/virtualservice.yaml
spec:
  hosts:
    - {{ .Values.istio.preview.hostname }}
  gateways:
    - {{ .Values.istio.gateway }}
    {{- if .Values.istio.preview.enabled }}
    - istio-system/cross-network-gateway
    {{- end }}

Verification

The moment of truth. After all this configuration, does it actually work?

$ curl https://pricing-api-pr-1-red.chrishouse.io/health
{"status":"healthy","service":"pricing-api","timestamp":"2025-12-23T15:13:18.004Z"}

That healthy response took way too long to see, but there it is. Traffic is flowing from the internet, through the hub cluster, across the cluster boundary, and into the preview environment running on the dev spoke.

Here's the full traffic flow:

DNS resolution: Browser looks up pricing-api-pr-1-red.chrishouse.io, gets 48.194.61.98 (hub Istio ingress)
TLS termination: Hub's Istio gateway terminates TLS using the *.chrishouse.io wildcard certificate
Pattern matching: Hub's VirtualService sees the hostname matches the preview regex pattern
Cross-cluster routing: Request is forwarded to ServiceEntry shared-dev.internal
External routing: ServiceEntry resolves to 52.255.217.180 (dev spoke's east-west gateway)
Service routing: East-west gateway routes to the pricing-api service based on the Host header
Response: The whole chain reverses, response arrives at the browser

Phase 4: The TLS Certificate Issue

Just when I thought we were done, there was one more surprise waiting.

The Problem

Testing the preview URL with the original naming scheme:

$ curl https://pricing-api-pr-1.red.chrishouse.io/health
curl: (60) SSL certificate problem: unable to get local issuer certificate

The routing works (we verified that with HTTP), but HTTPS is failing with a certificate error?

The original URL pattern was {service}-pr-{number}.{team}.chrishouse.io, something like pricing-api-pr-1.red.chrishouse.io. That's a two-level subdomain: pricing-api-pr-1 under red under chrishouse.io.

The existing wildcard certificate is *.chrishouse.io. And here's the thing about wildcard certificates that trips people up: they only match one level of subdomain, not arbitrary depth.

pricing-api.chrishouse.io → matches *.chrishouse.io ✓
anything.chrishouse.io → matches *.chrishouse.io ✓
pricing-api-pr-1.red.chrishouse.io → does NOT match *.chrishouse.io ✗

The wildcard only replaces the single * portion. It doesn't recursively match nested subdomains.

Options Considered

Option A: Per-team wildcard certs (*.red.chrishouse.io)

I could create a wildcard certificate for each team's subdomain. *.red.chrishouse.io would match pricing-api-pr-1.red.chrishouse.io just fine.

Rejected. That means managing N certificates where N is the number of teams. Each new team requires provisioning a new certificate, configuring it in the gateway, and keeping track of renewals. Certificate management is already tedious; multiplying it doesn't help.

Option B: SAN certificate with all patterns

A single certificate with Subject Alternative Names (SANs) for each team pattern: *.chrishouse.io, *.red.chrishouse.io, *.blue.chrishouse.io, etc.

Rejected. Same problem: the certificate needs to be regenerated every time a new team is added. Plus there are limits on SAN entries, and it adds operational overhead.

Option C: Change URL pattern to single-level subdomain

Instead of {service}-pr-{number}.{team}.chrishouse.io, use {service}-pr-{number}-{team}.chrishouse.io. The team name becomes part of the single subdomain rather than its own level.

Accepted. It's the simplest solution and works with the existing wildcard certificate without any changes to certificate management.

The Fix

Changed URL pattern from:

{service}-pr-{number}.{team}.chrishouse.io  (two-level)

To:

{service}-pr-{number}-{team}.chrishouse.io  (single-level)

Updated files:

backstage/templates/nodejs-quickstart/skeleton/.github/workflows/preview.yml
backstage/templates/nodejs-quickstart/skeleton/helm/values.yaml
infra/kubernetes/istio-hub/virtualservice-preview-envs.yaml
pricing-api/.github/workflows/preview.yml

New regex pattern:

headers:
  ":authority":
    regex: ".*-pr-[0-9]+-[a-z0-9]+\\.chrishouse\\.io"

DNS Update

The final piece of the puzzle was DNS. I updated the wildcard DNS record *.chrishouse.io to point to the hub Istio ingress at 48.194.61.98. This means any subdomain that doesn't have a more specific A record will resolve to the hub gateway.

Importantly, existing services with explicit A records (like argohub.chrishouse.io and backstage.chrishouse.io pointing to nginx-ingress at 20.253.73.108) are unaffected. DNS resolution prefers more specific records over wildcards, so those services continue to work exactly as before.

Phase 5: Automatic Cleanup

Preview environments are only useful if they don't accumulate. Without cleanup, you'd end up with dozens of abandoned namespaces consuming cluster resources, each one a forgotten artifact of a PR from three months ago.

When a PR is merged or closed, the preview environment should be deleted automatically. This is the "ephemeral" part of ephemeral environments.

cleanup-preview:
  if: github.event.action == 'closed'
  runs-on: ubuntu-latest
  steps:
    - name: Checkout GitOps repository
      uses: actions/checkout@v3
      with:
        repository: crh225/ARMServicePortal
        token: ${{ secrets.GITOPS_TOKEN }}
        path: gitops

    - name: Remove preview environment manifests
      working-directory: gitops
      run: |
        rm -rf infra/quickstart-services/${SERVICE_NAME}/preview-pr-${PR_NUM}

    - name: Commit and push cleanup
      run: |
        git add .
        git commit -m "Cleanup preview environment for PR #${PR_NUM}"
        git push

The cleanup follows the same GitOps pattern as creation. The workflow deletes the manifests from Git, commits the change, and lets ArgoCD handle the rest. ArgoCD has prune: true in its sync policy, which means when it detects that a resource exists in the cluster but not in Git, it deletes it.

This is one of the elegant things about GitOps: cleanup is just another commit. Namespace, pods, services, VirtualService, all removed automatically when the ArgoCD Application manifest disappears from the repository. No custom cleanup scripts, no cron jobs scanning for orphaned resources, no manual intervention.

The Final Architecture

Internet
    │
    ▼
┌─────────────────────────────────────────┐
│ Hub Cluster (aks-mgmt-hub)              │
│                                         │
│ ┌─────────────────────────────────────┐ │
│ │ Istio Gateway (48.194.61.98)        │ │
│ │ - TLS termination (*.chrishouse.io) │ │
│ │ - VirtualService regex matching     │ │
│ └──────────────┬──────────────────────┘ │
│                │                        │
│ ┌──────────────▼──────────────────────┐ │
│ │ ServiceEntry (shared-dev.internal)  │ │
│ │ → 52.255.217.180                    │ │
│ └─────────────────────────────────────┘ │
└─────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────┐
│ Dev Spoke (aks-shared-dev)              │
│                                         │
│ ┌─────────────────────────────────────┐ │
│ │ East-West Gateway (52.255.217.180)  │ │
│ │ - Public LoadBalancer               │ │
│ └──────────────┬──────────────────────┘ │
│                │                        │
│ ┌──────────────▼──────────────────────┐ │
│ │ VirtualService (pricing-api)        │ │
│ │ - Routes to service in namespace    │ │
│ └──────────────┬──────────────────────┘ │
│                │                        │
│ ┌──────────────▼──────────────────────┐ │
│ │ pricing-api Service                 │ │
│ │ Namespace: red-dev-pr-1             │ │
│ └─────────────────────────────────────┘ │
└─────────────────────────────────────────┘

View interactive diagram →

What We Learned

Building this feature took longer than expected, but most of the time wasn't spent on the preview workflow itself (that was straightforward). The complexity was in the cross-cluster routing, and specifically in working around Azure's networking limitations.

What Worked

GitOps-driven preview environments: The pattern of PR workflow → GitOps commit → ArgoCD deploy → cleanup commit is clean and reliable. Every state change is recorded in Git, which makes debugging and auditing trivial.
Istio multi-cluster with remote secrets: Once configured, this just works. Istio handles service discovery across clusters without any per-service configuration.
Single-level subdomain URLs: A simple URL pattern change avoided certificate complexity entirely.
Automatic cleanup: GitOps makes cleanup as reliable as deployment. If the manifest isn't in Git, the resource doesn't exist.

What Didn't Work (Initially)

Azure internal LoadBalancers: Not accessible across VNet peering without Private Link. This was a frustrating discovery because everything else about VNet peering works fine.
Two-level subdomain URLs: Wildcard certs only match one level. This is well-documented behavior, but easy to forget when designing URL schemes.

Key Decisions

Decision	Choice	Rationale
Cross-cluster routing	Istio multi-cluster	Industry standard, mTLS, service discovery
East-west gateway	Public LoadBalancer	Azure internal LBs not accessible cross-VNet; cheaper than Private Link
URL pattern	Single-level subdomain	Works with existing wildcard cert
Cleanup trigger	PR close event	Immediate, no TTL complexity

Cost vs Security Tradeoff

This is a personal lab environment running on my Azure subscription (and credit card). The architecture choices reflect that:

Option	Monthly Cost	Complexity	Security
Public LoadBalancer (chosen)	~$3.65	Low	Dev-acceptable
Azure Private Link	~$10.65	Medium	Production-ready
VPN/ExpressRoute	$50+	High	Enterprise-grade

For production workloads, we could use Azure Private Link. For a dev environment where the only exposed services are ephemeral preview deployments with no sensitive data, the public LoadBalancer is a reasonable tradeoff.

The east-west gateway only routes traffic to services with explicit VirtualService configurations. It's not an open proxy.

Operational Costs

East-west gateway public IP: ~$3.65/month (Azure)
Additional Istio overhead: Minimal (already running Istio on both clusters)
Network egress: Hub → dev spoke traffic stays within Azure

Implementation Repository

Full implementation: github.com/crh225/ARMServicePortal

Key files:

Preview workflow template: backstage/templates/nodejs-quickstart/skeleton/.github/workflows/preview.yml
Hub VirtualService: infra/kubernetes/istio-hub/virtualservice-preview-envs.yaml
ServiceEntry: infra/kubernetes/istio-hub/service-entry-shared-dev.yaml
East-west gateway: infra/cluster-bootstrap-shared-dev/istio-eastwest-gateway-argocd-app.yaml
VirtualService Helm template: backstage/templates/nodejs-quickstart/skeleton/helm/templates/virtualservice.yaml

Next in series: Part 3 will cover cost visibility, showing developers the real cost of their applications directly in Backstage with Azure Cost Management integration and resource tagging.

Building Golden Paths with Backstage: Part 1 - Foundation

Mon, 22 Dec 2025 00:00:00 GMT

This is the first in a multi-part series documenting the actual implementation of a golden path for microservices deployment. Not the theory, but the real commands, the actual errors, and what it takes to go from empty cluster to working self-service platform.

There's no shortage of conference talks about platform engineering. Plenty of blog posts explaining why golden paths matter. What's harder to find is someone walking through the actual implementation: the YAML files, the permission errors, the "why isn't this working" moments that don't make it into the polished demos.

That's what this series is. By the end of today's work: a developer fills in 4 fields in Backstage, and 5 minutes later has a running microservice deployed to Kubernetes with health checks responding.

Here's how we built it.

What We're Building

A self-service platform where developers can deploy microservices without:

Manually creating Kubernetes namespaces
Configuring CI/CD pipelines
Setting up container registries
Writing ArgoCD applications
Requesting infrastructure tickets

The goal: minimize the distance between "I want to deploy a service" and "my service is running in production."

Phase 1: Shared Development Cluster Foundation

The Problem

Every team wants their own Kubernetes cluster. Understandable: clean isolation, no noisy neighbors, full control.

It's also expensive, operationally complex, and usually overkill for development workloads.

The alternative: one shared development cluster with proper namespace isolation. Teams get their own space, resource limits prevent noisy neighbor problems, and operations stays sane. The tradeoff is that you need to actually implement the isolation properly, which is what this section covers.

The Infrastructure Stack

Cluster:

Azure Kubernetes Service (AKS)
Named aks-shared-dev
Single cluster for all development teams

Namespace Management:

Crossplane for declarative namespace provisioning
Custom Resource Definition: XNamespaceClaim
Teams request namespaces, Crossplane creates them with isolation

Isolation Mechanisms:

# Resource quotas per team namespace
resourceQuota:
  requestsCpu: "2"
  requestsMemory: "4Gi"
  limitsCpu: "4"
  limitsMemory: "8Gi"
  pods: 50

# Default limits for containers
limitRange:
  cpuRequest: "100m"
  cpuLimit: "500m"
  memoryRequest: "128Mi"
  memoryLimit: "512Mi"

# Network policies enabled
enableNetworkPolicy: true

Each team gets a namespace with hard resource limits. No single team can consume the entire cluster. This sounds obvious, but without these limits, one team's runaway process or misconfigured HPA can starve everyone else.

Why Crossplane Instead of kubectl?

We could just run kubectl create namespace red-dev and be done. It's one command. Why introduce another abstraction layer?

I had this debate with myself for longer than I'd like to admit. The answer comes down to what happens after day one. Creating a namespace is easy. Creating a namespace with the right resource quotas, limit ranges, network policies, RBAC bindings, and labels every single time, consistently, without forgetting anything? That's where things fall apart.

Instead, we use Crossplane to create a NamespaceClaim:

apiVersion: platform.chrishouse.io/v1alpha1
kind: NamespaceClaim
metadata:
  name: red-dev
  namespace: crossplane-system
  labels:
    team: red
    environment: dev
    created-by: backstage
spec:
  parameters:
    targetCluster: shared-dev-cluster
    namespaceName: red-dev
    enableResourceQuota: true
    enableNetworkPolicy: true
    enableLimitRange: true

Why the extra abstraction?

Consistency - Every namespace gets resource quotas, limit ranges, and network policies automatically
GitOps - Namespace configuration is declarative and version-controlled
Self-service - Backstage templates can create namespaces without cluster credentials
Auditability - Clear record of who requested what and when

The Crossplane composition handles the actual Kubernetes API calls. Teams just declare what they need. The composition is essentially a contract: you give me a namespace name and a team label, I give you a fully configured namespace with all the guardrails in place.

Verification

After setting up the cluster and Crossplane:

$ kubectl get namespaceclaim -n crossplane-system
NAME      SYNCED   READY   AGE
red-dev   True     True    15m

$ kubectl get namespace red-dev -o yaml
apiVersion: v1
kind: Namespace
metadata:
  labels:
    team: red
    environment: dev
    managed-by: crossplane
  name: red-dev

$ kubectl get resourcequota -n red-dev
NAME                AGE
red-dev-quota       15m

$ kubectl get limitrange -n red-dev
NAME                AGE
red-dev-limits      15m

Namespace exists. Quotas applied. Limits enforced. This might not look like much, but it's the foundation everything else builds on. Get this wrong, and you'll be debugging resource contention issues and namespace configuration drift for months.

Phase 1 complete.

Phase 2: Microservice Deployment Template

The Goal

Now we have a place to put things. The next question: how do developers actually get their code running there?

The developer experience should be:

Click "Create Component" in Backstage
Fill in: Service Name, Team, Description, Owner
Wait 5 minutes
Service is running

Everything else happens automatically. No terminal. No YAML editing. No waiting for someone in ops to process a ticket.

What "Everything Else" Means

That "everything else" is doing a lot of heavy lifting. Behind that 4-field form, the platform needs to:

Create GitHub repository with proper structure
Configure repository permissions for GHCR (GitHub Container Registry)
Set up CI/CD pipeline to build and publish container images
Create namespace in shared dev cluster (via Crossplane)
Create ArgoCD application manifest
Commit GitOps configuration to platform repo
Let ArgoCD discover and deploy the service

That's seven separate systems that need to coordinate correctly. Any one of them failing silently would leave developers confused about why their service isn't running. This is where the abstraction of "fill in a form" starts looking a lot more complex than it sounds.

Let's break it down.

The Backstage Template

Core structure:

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: nodejs-microservice-quickstart
  title: Node.js Microservice (Quick Start)
spec:
  parameters:
    - title: Service Information
      properties:
        service_name:
          title: Service Name
          type: string
        team:
          title: Team Name
          type: string
        description:
          title: Description
          type: string
        owner:
          title: Owner
          type: string

Four input fields. That's it. Everything else is either defaulted or derived from these values. The temptation to add more fields is strong, but every additional input is friction. Resist it.

Template Steps

Step 1: Scaffold Repository

- id: fetch-template
  name: Fetch Template
  action: fetch:template
  input:
    url: ./skeleton
    values:
      serviceName: ${{ parameters.service_name }}
      team: ${{ parameters.team }}
      description: ${{ parameters.description }}

Backstage copies the template skeleton and replaces variables.

The skeleton includes:

Node.js application with Express
Dockerfile for container builds
GitHub Actions workflow for CI/CD
Helm chart for Kubernetes deployment
Health check endpoints (/health, /ready)

Step 2: Create GitHub Repository

- id: publish-github
  name: Publish to GitHub
  action: publish:github
  input:
    repoUrl: github.com?owner=crh225&repo=${{ parameters.service_name }}
    defaultBranch: main

Step 3: Configure Repository Permissions (Critical Step)

This is where it gets interesting, and where I wasted more time than I'd like to admit.

GitHub Container Registry requires specific repository permissions. By default, new repositories have workflow permissions set to read. This seems fine until you try to push a container image. The build passes, the login succeeds, and then:

denied: permission_denied: write_package

The error message is unhelpful. You'll search for PAT token issues, GHCR authentication problems, Docker login failures. None of that is the problem. The problem is a single checkbox in the repository settings that nobody told you about.

We could fix this manually for each repository, but that defeats the purpose of automation. Better: automate it with a custom Backstage action that configures permissions immediately after repository creation.

Custom action: armportal:github:configure-repo

export const configureGitHubRepoAction = (options?: { token?: string }) => {
  return createTemplateAction({
    id: 'armportal:github:configure-repo',
    async handler(ctx) {
      const { repoUrl, token } = ctx.input;
      const [, owner, repo] = repoUrl.match(/github\.com\?.*owner=([^&]+).*repo=([^&]+)/);

      const octokit = new Octokit({ auth: token });

      // Set workflow permissions to 'write'
      await octokit.request('PUT /repos/{owner}/{repo}/actions/permissions/workflow', {
        owner,
        repo,
        default_workflow_permissions: 'write',
        can_approve_pull_request_reviews: true,
      });

      // Enable GitHub Actions
      await octokit.request('PUT /repos/{owner}/{repo}/actions/permissions', {
        owner,
        repo,
        enabled: true,
        allowed_actions: 'all',
      });
    },
  });
};

Used in template:

- id: configure-repo
  name: Configure Repository Permissions
  action: armportal:github:configure-repo
  input:
    repoUrl: github.com?owner=crh225&repo=${{ parameters.service_name }}

Now GHCR publishing works automatically. Every new repository gets the right permissions from the start. No manual intervention, no mysterious failures on the first build.

Step 4: Create GitOps Configuration

- id: create-gitops-pr
  name: Create GitOps PR
  action: armportal:create-pr
  input:
    repoUrl: github.com?owner=crh225&repo=ARMServicePortal
    branch: add-${{ parameters.service_name }}
    title: "Add ${{ parameters.service_name }} to platform"
    files:
      - path: infra/crossplane/claims/aks-shared-dev/${{ parameters.team }}-dev/namespace-claim.yaml
        content: ${{ steps['prepare-gitops'].output.namespaceClaim }}
      - path: infra/quickstart-services/${{ parameters.service_name }}/argocd-application.yaml
        content: ${{ steps['prepare-gitops'].output.argoApplication }}

This creates a pull request in the platform repository with:

Namespace claim for Crossplane
ArgoCD application manifest

Merge the PR. GitOps takes over. This separation matters: the template creates the intent (a PR), but a human approves the actual deployment. You can make this automatic if you want, but having that gate gives teams a moment to review what's about to happen.

The CI/CD Pipeline

The template generates a complete GitHub Actions workflow for each new service. Nothing fancy here, just the standard build-and-push pattern:

name: Build and Push to GHCR

on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - uses: actions/checkout@v3

      - name: Convert repository name to lowercase
        id: repo
        run: echo "repository=${{ github.repository }}" | tr '[:upper:]' '[:lower:]' >> $GITHUB_OUTPUT

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: ghcr.io/${{ steps.repo.outputs.repository }}:latest

Key detail: lowercase repository name. GHCR requires it, and GitHub repository names can have uppercase letters. This mismatch will cause failures if you forget to normalize. The workflow handles it with that tr command, but I learned this the hard way after debugging "image not found" errors for an embarrassingly long time.

ArgoCD Service Discovery

Problem: Every time we create a new service, we'd need to manually register it with ArgoCD. That's exactly the kind of manual step that makes developers go around the platform.

Solution: App-of-Apps pattern. ArgoCD watches a directory. When new files appear, it creates applications automatically.

Created infra/argocd/apps/quickstart-services-app.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: quickstart-services
  namespace: argocd
spec:
  project: applications
  source:
    repoURL: https://github.com/crh225/ARMServicePortal.git
    targetRevision: main
    path: infra/quickstart-services
    directory:
      recurse: true
      include: '*/argocd-application.yaml'
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

ArgoCD watches infra/quickstart-services/. When it finds a new argocd-application.yaml, it creates the application automatically. The template commits the application manifest, and within a few minutes ArgoCD notices the new file and deploys the service.

No manual ArgoCD registration required. No "please add my app to ArgoCD" tickets.

The First Error: ServiceMonitor CRD Missing

Everything was working. Repository created, permissions configured, image built and pushed. ArgoCD picked up the application manifest and started syncing.

Then it failed. ArgoCD showed:

OutOfSync
Missing

Error: failed to discover server resources for group version monitoring.coreos.com/v1:
the server could not find the requested resource

The template included a ServiceMonitor resource for Prometheus metrics, because of course you want metrics for your services. But Prometheus Operator wasn't installed on the cluster yet. The CRD didn't exist. ArgoCD couldn't apply something that the cluster didn't understand.

This is a common trap with golden path templates: including "best practices" resources that depend on infrastructure that doesn't exist yet. The fix is simple, but the lesson is important. Don't put aspirational resources in your default template. :D

Fix: Disable ServiceMonitor by default in template.

# helm/values.yaml
serviceMonitor:
  enabled: false  # Set to true after installing Prometheus Operator

Re-sync. Success. Enable it later when Prometheus is actually running.

Verification: pricing-api Deployment

Time to test the whole pipeline end-to-end. I created pricing-api via the Backstage template, filled in the four fields, and waited:

$ kubectl get pods -n red-dev
NAME                          READY   STATUS    RESTARTS   AGE
pricing-api-6f4b8d9c7-8xk2p   1/1     Running   0          2m
pricing-api-6f4b8d9c7-m7n4q   1/1     Running   0          2m

$ kubectl port-forward -n red-dev svc/pricing-api 8080:80
Forwarding from 127.0.0.1:8080 -> 3000

$ curl http://localhost:8080/health
{"status":"healthy","timestamp":"2025-12-22T15:30:00.000Z"}

Two pods running. Health checks responding. The entire chain worked: Backstage created the repo, configured GHCR permissions, GitHub Actions built and pushed the image, the GitOps PR got merged, ArgoCD deployed the application.

Time from Backstage form submission to running service: 4 minutes, 37 seconds.

That number matters. If this took 30 minutes, developers would start looking for shortcuts. If it took 2 days waiting for approvals, they'd definitely route around the platform. Under 5 minutes is fast enough that the golden path becomes the path of least resistance.

Phase 2 complete.

View interactive diagrams →

What Actually Happened

Starting from an empty shared development cluster, we built:

Infrastructure:

Crossplane namespace provisioning with resource quotas
Shared AKS cluster for all development teams
Namespace isolation with limits and network policies

Automation:

Backstage template for Node.js microservices
Custom action for automatic GHCR permissions
CI/CD pipeline for container builds
ArgoCD App-of-Apps for service discovery
GitOps workflow for all deployments

Developer Experience:

4 input fields in Backstage
~5 minutes to running service
No manual configuration required
No cluster credentials needed
No infrastructure tickets

What worked:

End-to-end automation from form to deployment
Automatic GHCR configuration
Service discovery via App-of-Apps pattern
Resource isolation in shared cluster

What didn't (initially):

ServiceMonitor CRD dependency
Repository name case sensitivity in GHCR
ArgoCD not watching quickstart-services directory

All fixed. System working. Each of these issues took time to debug, but once fixed, they stay fixed. Every developer who uses the template after this benefits from the lessons learned.

What's Next

This gets us from zero to deployed service. A developer can fill out a form and have a running microservice in under 5 minutes. But we're missing some important pieces:

Networking:

External ingress (currently ClusterIP only)
Istio service mesh for mTLS
DNS configuration

Developer Workflow:

Preview environments per pull request
Automated testing in pipelines
Rollback mechanisms

Security:

Policy enforcement (OPA/Kyverno)
Secret management (External Secrets Operator)
Image scanning

Those are the next phases. Each one adds complexity, and each one needs to be optional until it's required.

The principle remains: make the golden path faster than the alternative.

If creating a service this way takes 5 minutes, and the manual alternative takes days of tickets and approvals, developers will use the platform.

If the platform becomes slower or more restrictive than doing it manually, they'll route around it.

That's the balance we're building toward.

Implementation Repository

Full implementation available at: github.com/crh225/ARMServicePortal

Key files referenced:

Backstage template: backstage/templates/nodejs-quickstart/template.yaml
Custom GitHub action: backstage/plugins/arm-portal-backend/src/scaffolder/actions/configureGitHubRepo.ts
App-of-Apps: infra/argocd/apps/quickstart-services-app.yaml
Crossplane composition: infra/crossplane/platform/namespace-composition.yaml

Next in series: Part 2 will cover preview environments: automatically creating ephemeral deployments for every pull request, with DNS, TLS certificates, and automatic cleanup.

Platform Engineering Golden Paths: Common Patterns

Sun, 21 Dec 2025 00:00:00 GMT

I have been thinking about how to build Self-service developer platforms, and deal with trade-offs: how much infrastructure complexity to expose, how to serve different user needs, and how to balance flexibility with standardization.

Looking at various implementations—from open-source tools like Backstage¹ to commercial platforms like Heroku² and Vercel³—some recurring patterns emerge.

The Three-Tier Abstraction Model

Many platforms structure their interfaces around three levels of abstraction, each serving different use cases.

Tier 1: Minimal Input Templates

Backstage's software templates¹ demonstrate this approach—forms with minimal required fields:

Service Type: Web Service
Name: customer-api
Team: payments

[Create]

Behind this simplicity, the platform makes decisions about:

Repository structure
CI/CD configuration
Deployment target
Monitoring setup

Tier 2: Configurable Options

The same tools often support optional advanced fields:

Service: customer-api
Team: payments

▼ Advanced Options
  Environment: Production
  Resources: Medium (2 CPU, 4GB)
  Database: PostgreSQL - Standard

[Create]

Progressive disclosure—showing complexity only when requested—appears across many platform UIs.

Tier 3: Full Infrastructure Templates

At the detailed end, templates expose networking, scaling, cluster configuration, and other infrastructure parameters.

These tend to be used less frequently, primarily by platform teams or for specialized workloads.

Shared Development Clusters

AWS Proton's documentation⁴ discusses environment strategies. A common pattern that emerges:

Shared development cluster:

Namespace-based isolation
Resource quotas per team
Lower operational overhead
Automatic cleanup of inactive namespaces

Dedicated clusters:

Reserved for production
Or teams with specific isolation requirements

The trade-off: shared environments reduce costs and complexity, but require clear resource boundaries.

Progressive Disclosure

GitHub Codespaces⁵ provides a clear example—default simplicity with optional depth:

Repository: myapp
Branch: main
[Create]

▼ Advanced options
  Machine type: 2-core, 8GB
  Region: US West
  Timeout: 30 minutes

Most users never expand the advanced section. Those who need it can find it.

Environment Lifecycles

Heroku² and Vercel³ both automate environment progression:

Pull requests create preview environments
Main branch deploys to staging
Production requires manual promotion

The platform determines where code runs based on git workflow, removing environment selection as a decision point.

Cost Visibility

Some platforms surface cost estimates before resource creation:

Estimated monthly cost:
  Compute: $150
  Database: $75
  Monitoring: $31
  Total: $256/month

[ ] Budget approved
[Create]

Whether this changes behavior is unclear, but it makes cost visible at decision time rather than after the fact.

Persona-Based Templates

Different templates for different needs:

Quick start:

3 input fields
All defaults
5 minutes to deployment

Configurable:

8-10 options
Recommended defaults shown
15 minutes to deployment

Full control:

All infrastructure parameters
For platform engineers or specialized needs

This acknowledges that not everyone comes to the platform with the same goals.

Making Standard Paths Faster

One approach that appears in platform engineering writing⁶: make the golden path fast, make deviation possible but slower.

Following standard patterns might take minutes. Requesting custom infrastructure might require architectural review and take days.

This nudges toward standards without blocking exceptions.

Preview Environments

Vercel³ automates this completely:

PR #42 opened
  → Deployed to preview-pr-42.vercel.app
  → URL commented on PR

PR merged or closed
  → Preview environment deleted

Internal platforms can implement similar patterns using namespace isolation in shared clusters.

Composition Over Custom Building

Looking across implementations, most platform descriptions emphasize orchestrating existing tools:

GitHub Actions for CI/CD
Prometheus or Datadog for monitoring
Managed databases from cloud providers

Rather than building custom versions of commodity tools.

Starting Simple, Adding Complexity

The documented implementation pattern tends to be:

Build minimal template
Launch to small group
Gather feedback
Add optional complexity
Broader rollout

Though whether teams actually follow this sequence or jump to comprehensive solutions is variable.

Common Elements

Across various platform implementations, certain patterns recur:

Three-tier abstraction: Simple default, configurable middle tier, full control for specialists.

Shared development infrastructure: One shared cluster for development, dedicated clusters for production or special cases.

Progressive disclosure: Hide complexity by default, make it available on request.

Automated lifecycles: Environments tied to git workflow rather than manual selection.

Cost awareness: Show estimated costs before provisioning.

What These Patterns Address

These approaches seem aimed at similar problems:

Supporting many developers without proportionally growing platform teams
Reducing time from "I have an idea" to "I have a running service"
Balancing standardization with flexibility
Making costs visible

How well they work depends on organizational context, but the patterns show up often enough to be worth examining.

References

1. Backstage Software Templates - backstage.io/docs/features/software-templates/

2. Heroku Review Apps and Pipeline Promotion - devcenter.heroku.com/articles/pipelines

3. Vercel Preview Deployments - vercel.com/docs/deployments/preview-deployments

4. AWS Proton Environment Templates and Strategies - docs.aws.amazon.com/proton/latest/userguide/ag-environments.html

5. GitHub Codespaces Configuration and Machine Types - docs.github.com/en/codespaces/setting-up-your-project-for-codespaces/adding-a-dev-container-configuration

6. CNCF Platform Engineering Maturity Model - tag-app-delivery.cncf.io/whitepapers/platform-eng-maturity-model/

Building a Hub-Spoke AKS Architecture with Istio Service Mesh

Sat, 20 Dec 2025 00:00:00 GMT

After running a single-cluster Kubernetes setup for a while, I recently migrated to a hub-spoke architecture. This post covers why I made the change, how the traffic routing works, and the key decisions around where to place the Istio ingress gateways.

Why Hub-Spoke?

The single-cluster approach is simple but has limitations:

Blast radius - A misconfiguration or resource exhaustion affects everything
Scaling constraints - Platform services compete with applications for resources
Upgrade risk - Cluster upgrades put everything at risk simultaneously

The hub-spoke pattern separates concerns:

Hub cluster - Runs platform/control plane services (ArgoCD, Crossplane, Backstage)
Spoke cluster(s) - Runs application workloads (APIs, frontends, microservices)

The Architecture

Here's what my setup looks like:

View Interactive Diagram →

                              +---------------------------+
                              |   Cloudflare (DNS/CDN)    |
                              +-------------+-------------+
                                            |
               +----------------------------+----------------------------+
               |                                                         |
               v                                                         v
+---------------------------+                             +---------------------------+
|  Hub Traffic              |                             |  Spoke Traffic            |
|  backstage.chrishouse.io  |                             |  portal.chrishouse.io     |
|  argocd.chrishouse.io     |                             |  portal-api.chrishouse.io |
+---------------------------+                             |  blog.chrishouse.io       |
               |                                          +---------------------------+
               v                                                         |
+--------------------------------------+                                 |
|  AKS Hub Cluster (aks-mgmt-hub)      |                                 |
|  +--------------------------------+  |                                 |
|  | Istio Ingress Gateway         |   |                                 |
|  | (Hub Services Only)           |   |                                 |
|  +--------------------------------+  |                                 |
|                                      |                                 |
|  ArgoCD | Crossplane | Cert-Manager  |                                 |
|  Backstage | Argo Rollouts           |                                 |
+--------------------------------------+                                 |
               |                                                         |
               | Manages via ArgoCD                                      |
               v                                                         v
+------------------------------------------------------------------------+
|  AKS Spoke Cluster (aks-app-spoke)                                     |
|  +------------------------------------------------------------------+  |
|  | Istio Ingress Gateway                                            |  |
|  | (Application Traffic - Direct from Cloudflare)                   |  |
|  +------------------------------------------------------------------+  |
|                                                                        |
|  portal-api (Node.js) | blog (Gatsby) | frontend (React)               |
+------------------------------------------------------------------------+

Key Design Decision: Decentralized Ingress

The critical decision was where to place the Istio ingress gateways. There are two patterns:

Centralized (Hub Ingress)

Internet -> Hub Gateway -> Routes to Spoke clusters

Decentralized (Spoke Ingress) - What I chose

Internet -> Each cluster's own Gateway

I went with decentralized ingress for one main reason: fault isolation. If the hub cluster goes down (maintenance, failed upgrade, resource issues), my applications remain accessible. The hub is a control plane, not a data plane.

Cluster Breakdown

Hub Cluster Services

Service	Purpose
ArgoCD	GitOps controller - manages deployments to all clusters
Crossplane	Infrastructure as Code - provisions cloud resources
Cert-Manager	TLS certificate automation via Let's Encrypt
Backstage	Developer portal and service catalog
Argo Rollouts	Progressive delivery controller

Spoke Cluster Platform Services

Service	Purpose
Cert-Manager	Independent TLS certificates for spoke ingress
Istio Service Mesh	Traffic management and mTLS

Spoke Cluster Workloads

Application	Description
portal-api	Node.js backend API
portal-frontend	React SPA
blog	Static site (Gatsby)

Istio Configuration

Each cluster runs its own Istio service mesh with an ingress gateway. The spoke cluster handles HTTPS termination with its own TLS certificates:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: external-gateway
  namespace: istio-ingress
spec:
  selector:
    istio: aks-istio-ingressgateway-external
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "portal.chrishouse.io"
        - "portal-api.chrishouse.io"
        - "blog.chrishouse.io"
      tls:
        httpsRedirect: true
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - "portal.chrishouse.io"
        - "portal-api.chrishouse.io"
        - "blog.chrishouse.io"
      tls:
        mode: SIMPLE
        credentialName: wildcard-tls

The wildcard-tls secret is created by cert-manager using a wildcard certificate for *.chrishouse.io. This means the spoke cluster is fully independent for TLS - it doesn't rely on the hub for certificate management.

Each application gets a VirtualService that routes traffic:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: portal-api-vs
  namespace: istio-ingress
spec:
  hosts:
    - "portal-api.chrishouse.io"
  gateways:
    - internal-gateway
  http:
    - route:
        - destination:
            host: portal-api.portal-api.svc.cluster.local
            port:
              number: 80

Removing Redundant Ingress Resources

One issue I ran into: my Helm charts still had nginx Ingress resources defined, even though Istio handles all traffic routing. This caused ArgoCD to show applications as "Progressing" indefinitely.

Why? ArgoCD's health check for Ingress resources waits for a load balancer IP to be assigned. Since nginx-ingress wasn't assigning IPs (Istio handles traffic instead), the Ingress stayed in a pending state forever.

The fix was simple - disable the Ingress in Helm values:

# values.yaml
ingress:
  enabled: false  # Istio VirtualService handles routing

And remove the Ingress from Kustomize resources:

# kustomization.yaml
resources:
  - namespace.yaml
  - serviceaccount.yaml
  - configmap.yaml
  - deployment.yaml
  - service.yaml
  # - ingress.yaml  # Removed - using Istio

Traffic Flow Explained

DNS: Cloudflare manages DNS for *.chrishouse.io, pointing to the spoke cluster's external IP
TLS: Terminated at the Istio ingress gateway using wildcard certificates issued by cert-manager (each cluster manages its own certs)
Service Mesh: Istio routes to the correct service based on VirtualService rules
mTLS: All pod-to-pod traffic within the mesh is encrypted

For hub services (backstage.chrishouse.io):

Cloudflare -> Hub Istio Gateway -> Backstage Pod

For spoke services (portal-api.chrishouse.io):

Cloudflare -> Spoke Istio Gateway -> Portal API Pod

The hub is never in the path for spoke traffic.

ArgoCD Multi-Cluster Management

ArgoCD on the hub manages applications across both clusters. Each Application specifies its destination:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: portal-api
  namespace: argocd
spec:
  destination:
    name: aks-app-spoke  # Target cluster
    namespace: portal-api
  source:
    repoURL: https://github.com/crh225/ARMServicePortal.git
    path: infra/kubernetes/portal-api
    targetRevision: main

The hub cluster is registered as a destination in ArgoCD, allowing centralized management while keeping workloads distributed.

Pros and Cons

Advantages

Fault isolation - Hub issues don't affect running applications
Independent scaling - Clusters scale based on their workload type
Cleaner upgrades - Upgrade hub without touching production apps
Security boundaries - Platform credentials isolated from app workloads

Tradeoffs

Complexity - Two clusters to manage instead of one
Cost - Additional control plane costs (though node pools can be sized appropriately)
Networking - Cross-cluster communication requires additional configuration
Observability - Metrics and logs spread across clusters

When to Use This Pattern

Good fit:

Multiple teams deploying to Kubernetes
High availability requirements for applications
Frequent platform upgrades
Compliance requirements for separation of concerns

Overkill for:

Single small application
Development/testing environments
Cost-sensitive projects with low traffic

The hub-spoke pattern provides a solid foundation for scaling the platform as needs grow.

Ultimately I chose this pattern also for cost. I want to be able to have my apps and blog available and shut down the hub when not in use and active development.

A key improvement was making the spoke cluster fully independent with its own cert-manager and TLS certificates. This means the spoke can serve HTTPS traffic even when the hub is completely offline - true fault isolation for production workloads.

Resources

Repo-Local AI Governance: Why We Need a Standard for Constraining AI Agents

Wed, 17 Dec 2025 00:00:00 GMT

AI coding assistants have evolved from autocomplete to autonomous agents. They read your codebase, run shell commands, create files, and make architectural decisions. But there's a problem: there's no standard way to constrain them at the repository level.

Every company is inventing their own solution. Claude uses CLAUDE.md. Cursor has .cursorrules. GitHub Copilot reads from various config files. None of them are compatible, and none of them are designed for governance—they're designed for context.

This is a governance problem masquerading as a documentation problem.

The Problem: Hallucinated Authority

AI assistants don't know what they're not supposed to do. Without explicit constraints, they'll:

Run destructive commands directly instead of going through your established workflows
Auto-merge PRs "to be helpful"
Modify secrets, state files, or protected configurations
Bypass approval gates that exist for good reasons
Invent execution paths that don't exist in your system

Prompts don't solve this. They're ephemeral—they disappear after the conversation. Every new session starts from zero. What you need is something that persists with the repository and encodes system invariants that must remain true regardless of what a user asks.

The Current Landscape

Here's what major AI tools use today:

Tool	Config File	Purpose
Claude Code	`CLAUDE.md`	Project context and instructions
Cursor	`.cursorrules`	Editor behavior and prompts
GitHub Copilot	Various	Workspace settings
Aider	`.aider.conf.yml`	Tool configuration
Continue	`.continuerc.json`	Extension settings

Notice the pattern: configuration, not governance.

These files tell the AI how to behave (formatting, language preferences, context). They don't tell it what it must never do or who has authority over what decisions.

What's Missing: Architecture Contracts for AI

Consider a platform with established workflows—CI/CD pipelines, approval gates, environment promotions. You need your AI assistant to understand:

All changes flow through Git—no direct mutations to production systems
Certain environments require human approvals
State files and secrets are immutable from the AI's perspective
Execution paths are fixed and cannot be reordered or skipped
The AI can analyze failures but cannot initiate recovery procedures autonomously

A config file with "please follow our processes" doesn't cut it. You need a structured contract that encodes:

Context: What is this system and what are its boundaries?
Intent: What tradeoffs were made and why?
Rules: What must never happen?
Workflows: What are the valid execution paths?

A Proposed Pattern: `.ai/` Directory

The solution is a dedicated directory for AI governance—tool-agnostic, plain markdown, readable by any AI assistant that scans your repo:

.ai/
├── context.md     # What is this system?
├── intent.md      # What tradeoffs were made?
├── rules.md       # What must never happen?
└── workflows.md   # What are valid execution paths?

context.md

Defines the system boundaries and operating model:

## Control Boundaries

Decision-making authority resides with humans at these points:

- Configuration curation (what can be changed)
- Policy definition (what constraints apply per environment)
- Pull request approval (whether a change proceeds)
- Production promotion approval (multi-reviewer gate)

Automation owns execution within those boundaries but cannot
bypass approval gates, modify policy configurations, or
introduce new execution paths.

intent.md

Encodes design philosophy and accepted tradeoffs:

## Primary Goals

The system optimizes for:

- **Safety over speed**: All changes flow through Git with review gates
- **Auditability**: Every change creates a traceable PR
- **Self-service within guardrails**: Users get autonomy; policies enforce boundaries

### Accepted Tradeoffs

- Changes take minutes (PR workflow) rather than seconds (direct execution)
- Policy violations block requests rather than warn
- Production changes require multiple approvals even for low-risk changes

rules.md

The hard constraints—things that must never happen:

## AI Guardrails

### What an AI Assistant Must Never Modify

- State files or backend storage
- Secrets or credential stores
- Policy configurations that relax security constraints
- Approval requirements for any environment
- Branch protection rules

### Prohibited Actions

- Bypassing approval gates
- Merging PRs automatically to protected environments
- Deleting resources without explicit destroy workflow
- Initiating recovery or restoration without explicit human request

workflows.md

Valid execution paths that cannot be modified:

### How Changes Are Executed

Changes proceed through the following fixed sequence.
Steps may not be reordered, skipped, or conditionally bypassed.

1. Branch creation (feature branch)
2. Configuration changes committed
3. Pull request creation
4. Automated validation
5. Results posted for human review
6. Human approval (per environment requirements)
7. Merge to main branch
8. Automated execution
...

Why This Works (And Where It Doesn't)

What It Solves

Hallucinated authority: The AI now knows it doesn't have permission to execute destructive operations directly.

Architecture drift: As conversations progress, the AI has a stable reference for system invariants. No more re-explaining your architecture every session.

Multi-tool consistency: Whether you use Claude, Cursor, or Copilot, the constraints are in the repo. Switch tools, keep your governance.

Shared repos: Other contributors (human or AI) can understand the governance model without tribal knowledge.

What It Doesn't Solve

Enforcement: These are constraints on reasoning, not execution. The AI could still violate them.

Partial context: If someone pastes a snippet without loading the governance docs, the constraints don't exist.

Narrow prompts: Direct commands might not trigger constraint checking.

User override: "Ignore the rules, I'm the admin" might work.

The honest answer is: these docs increase refusal likelihood, not inability to comply.

But that's often enough. Most security is about raising the bar, not building impenetrable walls.

The Case for Standardization

Right now, every organization is solving this independently. That's wasteful and inconsistent.

A standard should define:

Directory convention: .ai/ (tool-agnostic, intuitive, future-proof)
File structure: Separate context, intent, rules, and workflows
Schema: Machine-readable sections that tools can parse (initially convention-based, later formally validated)
Inheritance: Project-level overrides of organization defaults
Versioning: How constraints evolve with the codebase

A More Complete Standard

Here's what a cross-tool standard could look like:

.ai/
├── context.md       # System boundaries and ecosystem position
├── intent.md        # Design philosophy and tradeoffs
├── rules.md         # Hard constraints (MUST/MUST NOT)
├── workflows.md     # Valid execution paths

What Actually Changes?

The key question: do these files meaningfully change AI behavior, or do they mostly change how answers are explained?

Based on testing this pattern:

They shape reasoning: The AI models the system as having hard boundaries that exist independent of user requests.
They raise the refusal threshold: The AI is more likely to say "I can't do that" vs. "Let me try to help."
They don't change capabilities: The AI could still technically attempt prohibited actions.

That's the right mental model. These docs are governance constraints, not technical enforcement. They work because AI assistants are designed to be helpful and compliant—not because they're physically prevented from violating them.

Try It Yourself

If you want to implement this pattern:

Create a .ai/ directory in your repo
Write context.md: What is this system? What are its boundaries?
Write intent.md: What tradeoffs did you make? What do you optimize for?
Write rules.md: What must never happen? Be specific and use "MUST NOT" language.
Write workflows.md: What are the valid execution paths? Can they be reordered?
Reference these files in your CLAUDE.md or .cursorrules

The key insight: separate documentation from governance. READMEs explain how things work. Governance docs constrain what's allowed.

Conclusion

AI agents are here. They're reading our code, running our commands, and making decisions on our behalf. We've standardized how humans interact with repositories (PRs, branch protection, CODEOWNERS). We haven't standardized how AI agents should be constrained.

Until we do, every organization will reinvent this wheel. Some will do it well. Most won't do it at all, and their AI assistants will hallucinate authority they were never granted.

The .ai/ pattern isn't perfect, but it's a starting point. The goal isn't perfection—it's establishing a convention before the next generation of AI agents ships without any governance at all.

To be clear: this is a proposal, not a proven standard. I haven't validated it at scale across multiple teams or tools. But someone needs to start the conversation, and waiting for perfect evidence means waiting until the problem is already entrenched.

We need this.

Configuring ArgoCD to Access Private GitHub Repositories

Tue, 16 Dec 2025 00:00:00 GMT

This beginner's guide covers how to configure ArgoCD to sync applications from private GitHub repositories and pull private container images from GitHub Container Registry (GHCR). For production environments, consider using External Secrets Operator or Sealed Secrets to manage credentials declaratively rather than imperatively

Prerequisites

ArgoCD installed on your Kubernetes cluster
A GitHub Personal Access Token (PAT) with the following scopes:
- repo - Full control of private repositories
- read:packages - Read packages from GHCR
- write:packages - Push packages to GHCR (for CI/CD)

Part 1: ArgoCD Git Repository Access

ArgoCD needs credentials to clone private repositories. Create a Kubernetes secret with your GitHub username and PAT.

Create the Secret

apiVersion: v1
kind: Secret
metadata:
  name: github-repo-creds
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repo-creds
type: Opaque
stringData:
  url: https://github.com/YOUR_USERNAME/
  type: git
  username: YOUR_USERNAME
  password: YOUR_GITHUB_PAT

Key points:

Use repo-creds as the secret type for prefix matching (applies to all repos under that URL)
The username field should be your actual GitHub username
The password field is your PAT
Include the trailing slash on the URL

Apply and Restart

kubectl apply -f github-repo-creds.yaml
kubectl rollout restart deployment argocd-repo-server -n argocd

Verify

Check that your application syncs:

kubectl get application -n argocd

You should see Synced status instead of Unknown.

Part 2: Private Container Image Pulls

If your container images are also private in GHCR, Kubernetes needs an imagePullSecret to authenticate when pulling images.

Create the Docker Registry Secret

Run this for each namespace where you deploy applications:

kubectl create secret docker-registry ghcr-pull-secret \
  --namespace YOUR_NAMESPACE \
  --docker-server=ghcr.io \
  --docker-username=YOUR_USERNAME \
  --docker-password=YOUR_GITHUB_PAT \
  --docker-email=noreply@github.com

Reference in Your Helm Chart

Add to your values.yaml:

imagePullSecrets:
  - name: ghcr-pull-secret

Update your deployment template to use it:

spec:
  template:
    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

Storing the PAT Securely

Instead of hardcoding the PAT, store it in a secret manager like Azure Key Vault and retrieve it when creating secrets:

PAT=$(az keyvault secret show --vault-name YOUR_VAULT --name github-pat --query value -o tsv)

kubectl create secret docker-registry ghcr-pull-secret \
  --namespace dev \
  --docker-server=ghcr.io \
  --docker-username=YOUR_USERNAME \
  --docker-password="$PAT" \
  --docker-email=noreply@github.com

Summary

Secret	Purpose	Namespace
`github-repo-creds`	ArgoCD clones private repos	`argocd`
`ghcr-pull-secret`	Kubernetes pulls private images	Each deployment namespace

Both secrets use the same PAT with your GitHub username. The repo-creds type enables prefix matching, so one secret covers all repositories under your GitHub account.

Production Approach: External Secrets Operator

The imperative approach above works, but for production you want secrets managed declaratively through GitOps. External Secrets Operator automatically syncs secrets from Azure Key Vault (or AWS Secrets Manager, HashiCorp Vault, etc.) into Kubernetes secrets.

Install External Secrets Operator

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace

Configure Azure Key Vault Access

First, create a SecretStore that connects to your Key Vault. This example uses Azure Workload Identity:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: azure-keyvault
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: "https://YOUR_VAULT.vault.azure.net"
      serviceAccountRef:
        name: external-secrets-sa
        namespace: external-secrets

Declarative GitHub Credentials for ArgoCD

Now define an ExternalSecret that pulls the PAT from Key Vault and creates the ArgoCD secret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: github-repo-creds
  namespace: argocd
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: azure-keyvault
    kind: ClusterSecretStore
  target:
    name: github-repo-creds
    template:
      metadata:
        labels:
          argocd.argoproj.io/secret-type: repo-creds
      data:
        type: git
        url: https://github.com/YOUR_USERNAME/
        username: YOUR_USERNAME
        password: "{{ .github_pat }}"
  data:
    - secretKey: github_pat
      remoteRef:
        key: github-pat

Declarative GHCR Pull Secret

Similarly for the image pull secret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: ghcr-pull-secret
  namespace: dev
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: azure-keyvault
    kind: ClusterSecretStore
  target:
    name: ghcr-pull-secret
    template:
      type: kubernetes.io/dockerconfigjson
      data:
        .dockerconfigjson: |
          {
            "auths": {
              "ghcr.io": {
                "username": "YOUR_USERNAME",
                "password": "{{ .github_pat }}",
                "auth": "{{ printf "%s:%s" "YOUR_USERNAME" .github_pat | b64enc }}"
              }
            }
          }
  data:
    - secretKey: github_pat
      remoteRef:
        key: github-pat

Benefits of This Approach

GitOps-friendly - Secret definitions live in git, actual values stay in Key Vault
Automatic rotation - Change the PAT in Key Vault, secrets update automatically
Audit trail - Key Vault logs all secret access
No imperative commands - Everything is declarative YAML
Single source of truth - One PAT in Key Vault, referenced everywhere

Alternative: Sealed Secrets

If you don't have a secret manager, Sealed Secrets lets you encrypt secrets so they can be committed to git:

# Install kubeseal CLI and controller
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system

# Encrypt a secret
kubectl create secret generic github-repo-creds \
  --namespace argocd \
  --from-literal=password=YOUR_PAT \
  --dry-run=client -o yaml | kubeseal -o yaml > sealed-secret.yaml

The sealed secret can be safely committed to git - only your cluster can decrypt it.

Which Approach Should You Use?

Approach	Best For
Manual `kubectl create`	Learning, development, quick setup
External Secrets Operator	Production with existing secret manager
Sealed Secrets	Production without external secret manager

For most production Kubernetes environments, External Secrets Operator with your cloud provider's secret manager is the recommended approach.

Golden Paths, Day-2 Operations, and Where the Abstractions Crack

Sat, 13 Dec 2025 00:00:00 GMT

Disclaimer: I am a few weeks into building an internal developer platform for myself. Here is what I have learned so far.

Golden paths get a lot of attention in platform engineering discussions. Most of that attention stops at creation—scaffolding repos, setting up CI, and deploying to Kubernetes.

That part is relatively solved.

The harder problem is day-2 operations. That is where golden paths get more complicated.

I have been experimenting for a month with golden paths built on Backstage, Argo CD, and Kubernetes, and then layering Port.io on top to handle the parts that do not fit cleanly into GitOps alone.

Day-1 Is Easy

A typical golden path stack could look like this:

Backstage for service creation and discoverability
GitHub as the source of truth
Argo CD for GitOps reconciliation
Kubernetes as the runtime

Backstage software templates do a solid job at day-1:

Repos are scaffolded consistently
CI/CD pipelines are pre-wired
Deployment manifests exist from the start
Services show up in a catalog instead of disappearing into someone's personal GitHub

That initial experience matters. Early decisions tend to stick, for better or worse.

But once the service exists, for me, Backstage mostly steps aside.

Where Things Get Messy: Day-2

Day-2 is where reality shows up.

Examples:

Restarting or scaling a misbehaving service
Promoting config between environments
Triggering a redeploy without a code change
Inspecting ownership, dependencies, or runtime state
Performing safe operational actions without giving everyone kubectl access

These actions do not always belong in Git. They also do not belong in sending messages to the platform team.

This is where I started looking at Port.io.

How Port.io Fits

I found Port.io most useful for day-2 operations, with some ability to trigger day-0 creation workflows when needed.

Port does not natively scaffold code.

It triggers things that do.

So in practice:

Port can be the button a developer clicks on day-0
Backstage (or a repo template) can still do the scaffolding
Git remains the source of truth
Argo reconciles state

In that sense, Port is not a "day-0 engine" so much as a day-0 orchestrator.

In my lab setup:

Backstage is the entry point for creation, metadata, and discovery
Git and Argo CD remain the source of truth
Port.io becomes a controlled interaction layer on top

Using Port, I modeled:

Services, environments, and clusters as entities
Relationships between repos, workloads, and infrastructure
Actions that map to real operational workflows like restart, sync, and promote

Crucially, those actions do not mutate the cluster directly.

They trigger:

GitHub Actions workflows
Argo CD syncs
Or other well-defined automation steps

In my setup, provisioning remained outside of Port. That constraint mattered. Without it, the platform quickly started to resemble a prettier kubectl wrapper.

Why This Separation Actually Works

Each tool is good at a different thing:

Tool	Question It Answers
Backstage	"What is this?"
Port.io	"What can I safely do to it?"
Argo CD	"What should the cluster look like?"

Backstage is strong at creation but limited once a service is running. Port works well for interaction and visibility but benefits from clear boundaries. Argo enforces desired state but does not provide much guidance at the human level.

Once I stopped trying to make one tool do everything, the system made more sense.

What Success Might Look Like

I am still early in building this, so I do not know yet what success looks like in practice.

What I am looking for instead are signals that things are moving in the right direction:

New services start to look boring in similar ways
Day-2 actions become visible and repeatable
The platform starts answering questions before people ask them (with AI)

If those things start to happen over time, that will feel like progress.

Final Thought

Backstage, Argo CD, Kubernetes, and Port.io are all powerful tools.

None of them are the golden path by themselves.

Golden paths emerge when:

Defaults are intentional
Day-2 operations are treated as first-class concerns
The platform allows teams to step off the path without friction

There are tradeoffs no matter how you design this. That is usually where the real work begins.

Elastic Cloud Fleet Agent + Kubernetes Container Logs

Wed, 10 Dec 2025 00:00:00 GMT

Over the past few weeks, we have been researching how to use Elastic Cloud's Fleet Agent across Kubernetes clusters to monitor containerized workloads. This post covers how we made container logs visible in Kibana, why some setups silently fail, and what it takes to fix them.

The Goal

We wanted to ship logs from every Kubernetes container into Elastic Cloud using the Fleet-managed Elastic Agent. The goal was simple:

Deploy a DaemonSet that runs one Elastic Agent per node
Mount /var/log/pods and /var/log/containers
Point it to our Fleet Server with an enrollment token
Watch logs roll into Kibana

Symptom: No Container Logs

Our deployment looked good on paper: the agent was enrolled, metrics were visible, and /var/log/pods was populated inside the container. But no container logs appeared in Kibana.

The Investigation

1. Reviewed the DaemonSet YAML

Our DaemonSet had mounts for /var/log/pods and /var/lib/containerd, but was missing /var/log/containers — which turned out to be the key input path for the logs.

# Before the fix
volumeMounts:
  - name: pods
    mountPath: /var/log/pods
    readOnly: true
  - name: containerd
    mountPath: /var/lib/containerd
    readOnly: true

volumes:
  - name: pods
    hostPath:
      path: /var/log/pods
  - name: containerd
    hostPath:
      path: /var/lib/containerd

2. Inspected Runtime Mounts

We used this command to check which paths were actually mounted inside the Elastic Agent pod:

kubectl -n kube-system exec -it elastic-agent-XXXX -- mount | grep -E '/var/log|/var/lib/containerd'

The expected /var/log/containers path was not present — meaning the agent had no access to the symlinked container logs.

3. Checked Fleet Integration Configuration

In the Fleet UI (under the "Kubernetes Logs" integration), the log path was set to:

/var/log/containers/*.log

If this path doesn't exist inside the container, logs won't flow. Fleet doesn't surface errors for missing log paths — it just silently fails.

The Fix

We added a hostPath volume for /var/log/containers to the DaemonSet and mounted it inside the container.

# After the fix
volumes:
  - name: containers
    hostPath:
      path: /var/log/containers

volumeMounts:
  - name: containers
    mountPath: /var/log/containers
    readOnly: true

After applying this change, the agent immediately started reading symlinked container logs, and the data appeared in Kibana.

Lessons Learned

Issue	Resolution
Fleet Agent requires `/var/log/containers`, not just `/var/log/pods`	Add the missing hostPath mount
DaemonSet volume mounts must match the Fleet integration's expectations	Verify paths in Fleet UI
The filestream input fails silently when paths are missing	No errors, no logs — just silence

You can inspect what the agent is trying to harvest using:

./elastic-agent inspect components --show-config

Bonus: Persisting Agent Identity Across Restarts

We noticed that after restarting Elastic Agent pods, the old agent would show up as "offline" in Fleet, while a new one appeared as "online." This happened because each pod restart generated a new agent.id, leaving duplicates in Fleet.

To solve this, we mounted the Elastic Agent's internal state directory to the node's filesystem using a hostPath. This ensures that when a pod restarts on the same node, it reuses the same identity.

volumes:
  - name: agent-state
    hostPath:
      path: /var/lib/elastic-agent
      type: DirectoryOrCreate

volumeMounts:
  - name: agent-state
    mountPath: /usr/share/elastic-agent/state

This change stopped the agent duplication and made our Fleet inventory much cleaner.

Gotcha: x509 Certificate Errors from Zscaler

One critical issue we hit during enrollment was related to TLS inspection by Zscaler. Elastic Agent failed to enroll with this error:

x509: certificate signed by unknown authority

Although the connection and token looked correct, the underlying HTTPS call to the Fleet Server was being intercepted by a Zscaler proxy. To resolve this, we needed to install the Zscaler root certificate inside the agent pods.

Solution

1. Create a ConfigMap containing the Zscaler certificate:

apiVersion: v1
kind: ConfigMap
metadata:
  name: zscaler-ca-cert
  namespace: kube-system
data:
  zscaler.crt: |
    -----BEGIN CERTIFICATE-----
    [REDACTED CERT CONTENT]
    -----END CERTIFICATE-----

2. Mount the certificate path in the pod:

volumeMounts:
  - name: zscaler-ca
    mountPath: /etc/ssl/certs

volumes:
  - name: zscaler-ca
    configMap:
      name: zscaler-ca-cert

Once this was in place, the agent enrolled successfully. TLS errors went away immediately.

Debugging Tips

# Check for logs inside container
find /var/log/pods -type f -name "*.log"

# Validate filebeat component
ps aux | grep filebeat
./elastic-agent inspect components --show-config | grep -A 20 filestream

# Verify in Kibana
data_stream.dataset : *container*

Final Thoughts

Elastic's Fleet integration is powerful, but sensitive to configuration. To avoid surprises:

Triple-check your volume mounts — missing paths fail silently
Match the Fleet integration's expected log paths — /var/log/containers, not just /var/log/pods
Persist the agent's state so you don't get stale "offline" entries
If you're behind Zscaler or a TLS proxy, mount your root cert chain

This setup is now running smoothly in the test lab, and container logs across all nodes are flowing into Kibana as expected.

Reference: Sample Helm Chart (Sanitized)

values.yaml (excerpt)

image:
  repository: docker.elastic.co/beats/elastic-agent
  tag: 8.16.6

zscalerCA:
  enabled: true
  fileName: zscaler.crt
  name: zscaler-ca

daemonset.yaml (excerpt)

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: elastic-agent
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: elastic-agent
  template:
    metadata:
      labels:
        app: elastic-agent
    spec:
      serviceAccountName: elastic-agent
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: elastic-agent
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          env:
            - name: FLEET_ENROLL
              value: "1"
            - name: FLEET_URL
              value: "<redacted>"
            - name: FLEET_ENROLLMENT_TOKEN
              value: "<redacted>"
          volumeMounts:
            - name: pods
              mountPath: /var/log/pods
              readOnly: true
            - name: containerd
              mountPath: /var/lib/containerd
              readOnly: true
            - name: containers
              mountPath: /var/log/containers
              readOnly: true
            - name: agent-data
              mountPath: /usr/share/elastic-agent/state
            - name: zscaler-ca
              mountPath: /etc/ssl/certs
      volumes:
        - name: pods
          hostPath:
            path: /var/log/pods
        - name: containerd
          hostPath:
            path: /var/lib/containerd
        - name: containers
          hostPath:
            path: /var/log/containers
        - name: agent-data
          hostPath:
            path: /var/lib/elastic-agent
            type: DirectoryOrCreate
        - name: zscaler-ca
          configMap:
            name: zscaler-ca-cert

Building the ARM Service Portal: A Self-Service Platform for Azure Infrastructure

Mon, 01 Dec 2025 00:00:00 GMT

Teams either wait days (or months) for infra tickets to be provisioned, or let developers create Azure resources directly—which leads to inconsistent, insecure, or untracked infrastructure. Clouds fill with resources created outside Terraform, making governance and cost control harder.

I built a self-service portal to solve this: developers provision Azure resources through a simple UI, while Terraform and GitOps handle everything behind the scenes.

Live: https://portal.chrishouse.io

What It Does

Blueprint-based forms that generate Terraform PRs
GitHub Actions performs plan, apply, and state handling
Real-time cost, health, and deployment status
Shows Azure resources not managed by Terraform
Resource graph to visualize dependencies and ownership
Backstage integration for service catalog and scaffolding

The entire platform bootstraps itself—the Container App, Storage, and Front Door were all deployed through the portal using the same blueprints it provides.

The Architecture

Frontend

Component	Technology
Framework	React SPA
Hosting	Azure Storage Static Websites
CDN	Azure Front Door
Auth	GitHub OAuth
Forms	Dynamic generation from blueprint metadata
Visualization	Azure Resource Graph for dependency mapping

The frontend renders forms dynamically based on blueprint definitions. Each blueprint defines:

Required and optional parameters
Validation rules
Cost estimation hooks
Dependencies on other blueprints

// Example blueprint metadata
{
  "name": "azure-container-app",
  "displayName": "Container App",
  "description": "Deploy a containerized application to Azure Container Apps",
  "parameters": [
    {
      "name": "app_name",
      "type": "string",
      "required": true,
      "validation": "^[a-z0-9-]+$"
    },
    {
      "name": "container_image",
      "type": "string",
      "required": true
    },
    {
      "name": "cpu",
      "type": "select",
      "options": ["0.25", "0.5", "1.0", "2.0"],
      "default": "0.5"
    },
    {
      "name": "memory",
      "type": "select",
      "options": ["0.5Gi", "1Gi", "2Gi", "4Gi"],
      "default": "1Gi"
    }
  ],
  "costEstimate": {
    "type": "azure-container-apps",
    "factors": ["cpu", "memory", "replicas"]
  }
}

Backend

Component	Technology
Runtime	Node.js + Express
Hosting	Azure Container Apps
Integration	GitHub App for PR automation
State	Terraform remote state in Azure Storage
Webhooks	GitHub webhook handling for deployment events

The backend handles:

Terraform code generation from blueprint parameters
Variable validation before PR creation
Azure pricing API integration for cost estimates
GitHub App actions (create PRs, post comments, trigger workflows)
Resource discovery for unmanaged Azure resources

// Simplified PR creation flow
async function createDeploymentPR(blueprint, parameters, user) {
  // 1. Generate Terraform code from template
  const terraformCode = await generateTerraform(blueprint, parameters);

  // 2. Create branch
  const branchName = `deploy/${blueprint.name}/${Date.now()}`;
  await github.createBranch(branchName);

  // 3. Commit generated files
  await github.commitFiles(branchName, [
    { path: `deployments/${parameters.app_name}/main.tf`, content: terraformCode },
    { path: `deployments/${parameters.app_name}/variables.tf`, content: variablesFile },
    { path: `deployments/${parameters.app_name}/terraform.tfvars`, content: tfvarsFile }
  ]);

  // 4. Create PR with cost estimate in description
  const costEstimate = await estimateCost(blueprint, parameters);
  const pr = await github.createPR({
    title: `[Portal] Deploy ${blueprint.displayName}: ${parameters.app_name}`,
    body: generatePRDescription(parameters, costEstimate),
    head: branchName,
    base: 'main'
  });

  // 5. Trigger plan workflow
  await github.triggerWorkflow('terraform-plan.yml', { pr_number: pr.number });

  return pr;
}

GitOps Pipeline

The real magic happens in GitHub Actions. When a PR is created:

# .github/workflows/terraform-plan.yml
name: Terraform Plan

on:
  pull_request:
    paths:
      - 'deployments/**'

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3

      - name: Configure Azure credentials
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}

      - name: Find changed deployments
        id: changes
        run: |
          CHANGED=$(git diff --name-only ${{ github.event.pull_request.base.sha }} | grep '^deployments/' | cut -d'/' -f2 | sort -u)
          echo "deployments=$CHANGED" >> $GITHUB_OUTPUT

      - name: Terraform Init & Plan
        run: |
          for deployment in ${{ steps.changes.outputs.deployments }}; do
            cd deployments/$deployment
            terraform init -backend-config="key=$deployment.tfstate"
            terraform plan -out=plan.tfplan
            terraform show -json plan.tfplan > plan.json
          done

      - name: Post plan to PR
        uses: actions/github-script@v7
        with:
          script: |
            const plan = require('./deployments/${{ steps.changes.outputs.deployments }}/plan.json');
            const summary = formatPlanSummary(plan);

            github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: `## Terraform Plan\n\n${summary}\n\n**Resources:**\n- To create: ${plan.resource_changes.filter(r => r.change.actions.includes('create')).length}\n- To update: ${plan.resource_changes.filter(r => r.change.actions.includes('update')).length}\n- To destroy: ${plan.resource_changes.filter(r => r.change.actions.includes('delete')).length}`
            });

When the PR is merged:

# .github/workflows/terraform-apply.yml
name: Terraform Apply

on:
  push:
    branches: [main]
    paths:
      - 'deployments/**'

jobs:
  apply:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3

      - name: Configure Azure credentials
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}

      - name: Find changed deployments
        id: changes
        run: |
          CHANGED=$(git diff --name-only HEAD~1 | grep '^deployments/' | cut -d'/' -f2 | sort -u)
          echo "deployments=$CHANGED" >> $GITHUB_OUTPUT

      - name: Terraform Apply
        run: |
          for deployment in ${{ steps.changes.outputs.deployments }}; do
            cd deployments/$deployment
            terraform init -backend-config="key=$deployment.tfstate"
            terraform apply -auto-approve
          done

      - name: Update portal status
        run: |
          curl -X POST "${{ secrets.PORTAL_WEBHOOK_URL }}/deployment-complete" \
            -H "Content-Type: application/json" \
            -d '{"deployment": "${{ steps.changes.outputs.deployments }}", "status": "success"}'

Backstage Integration

The portal integrates with Backstage for service catalog and scaffolding. This creates a unified developer experience:

Service Catalog Sync

Every resource deployed through the portal automatically registers in Backstage:

# Generated catalog-info.yaml for each deployment
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: ${app_name}
  description: ${description}
  annotations:
    github.com/project-slug: ${github_repo}
    backstage.io/techdocs-ref: dir:.
    azure.com/resource-group: ${resource_group}
    azure.com/subscription: ${subscription_id}
  links:
    - url: https://${app_name}.azurecontainerapps.io
      title: Live Site
      icon: dashboard
    - url: https://portal.azure.com/#resource${resource_id}
      title: Azure Portal
      icon: cloud
  tags:
    - azure
    - container-app
    - terraform-managed
spec:
  type: service
  lifecycle: production
  owner: ${owner}
  dependsOn:
    - resource:${resource_group}

Software Templates

Backstage software templates use the same blueprints as the portal. When a developer scaffolds a new service:

Backstage creates the GitHub repo with CI/CD workflows
The portal's blueprint form is embedded for infrastructure provisioning
ArgoCD application is generated for Kubernetes deployments
Everything links back to the service catalog

# backstage/templates/azure-fullstack/template.yaml
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: azure-fullstack-template
  title: Azure Full-Stack Application
  description: Create a full-stack app with React frontend and Node.js backend on Azure
spec:
  owner: platform-team
  type: service

  parameters:
    - title: Application Details
      required:
        - name
        - description
      properties:
        name:
          title: Name
          type: string
          pattern: '^[a-z0-9-]+$'
        description:
          title: Description
          type: string

    - title: Infrastructure
      required:
        - environment
        - region
      properties:
        environment:
          title: Environment
          type: string
          enum: ['dev', 'staging', 'production']
        region:
          title: Azure Region
          type: string
          enum: ['eastus', 'westus2', 'westeurope']
        enableDatabase:
          title: Include PostgreSQL Database
          type: boolean
          default: false
        enableRedis:
          title: Include Redis Cache
          type: boolean
          default: false

  steps:
    - id: fetch-base
      name: Fetch Base Template
      action: fetch:template
      input:
        url: ./skeleton
        values:
          name: ${{ parameters.name }}
          description: ${{ parameters.description }}

    - id: publish-github
      name: Publish to GitHub
      action: publish:github
      input:
        repoUrl: github.com?owner=crh225&repo=${{ parameters.name }}
        description: ${{ parameters.description }}

    - id: create-infrastructure
      name: Create Azure Infrastructure
      action: http:backstage:request
      input:
        method: POST
        path: /api/portal/deployments
        body:
          blueprint: azure-fullstack
          parameters:
            app_name: ${{ parameters.name }}
            environment: ${{ parameters.environment }}
            region: ${{ parameters.region }}
            enable_database: ${{ parameters.enableDatabase }}
            enable_redis: ${{ parameters.enableRedis }}

    - id: register-catalog
      name: Register in Catalog
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps['publish-github'].output.repoContentsUrl }}
        catalogInfoPath: /catalog-info.yaml

ArgoCD Integration

For Kubernetes workloads, the portal generates ArgoCD Application manifests:

# Generated ArgoCD Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ${app_name}
  namespace: argocd
  labels:
    app.kubernetes.io/part-of: portal-deployments
  annotations:
    portal.chrishouse.io/blueprint: ${blueprint_name}
    portal.chrishouse.io/deployed-by: ${user}
    portal.chrishouse.io/deployed-at: ${timestamp}
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/crh225/${app_name}.git
    targetRevision: main
    path: helm
    helm:
      releaseName: ${app_name}
      valueFiles:
        - values.yaml
      parameters:
        - name: image.repository
          value: ghcr.io/crh225/${app_name}
        - name: image.tag
          value: ${image_tag}
  destination:
    server: https://kubernetes.default.svc
    namespace: ${namespace}
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Resource Discovery: Finding the Unmanaged

One of the most valuable features is discovering Azure resources that exist outside Terraform. The portal queries Azure Resource Graph and compares against known Terraform state:

async function discoverUnmanagedResources(subscriptionId) {
  // 1. Get all resources from Azure Resource Graph
  const azureResources = await resourceGraph.query(`
    Resources
    | where subscriptionId == '${subscriptionId}'
    | project id, name, type, resourceGroup, location, tags
  `);

  // 2. Get all resources from Terraform state files
  const terraformResources = new Set();
  const stateFiles = await listStateFiles();

  for (const stateFile of stateFiles) {
    const state = await downloadState(stateFile);
    state.resources.forEach(r => {
      terraformResources.add(r.instances[0]?.attributes?.id);
    });
  }

  // 3. Find resources not in Terraform
  const unmanaged = azureResources.filter(r => !terraformResources.has(r.id));

  // 4. Categorize by risk level
  return unmanaged.map(r => ({
    ...r,
    riskLevel: calculateRiskLevel(r),
    estimatedCost: await estimateResourceCost(r),
    recommendation: generateRecommendation(r)
  }));
}

function calculateRiskLevel(resource) {
  // High risk: databases, key vaults, networking
  const highRiskTypes = [
    'Microsoft.Sql/servers',
    'Microsoft.KeyVault/vaults',
    'Microsoft.Network/virtualNetworks',
    'Microsoft.Network/publicIPAddresses'
  ];

  // Medium risk: compute, storage
  const mediumRiskTypes = [
    'Microsoft.Compute/virtualMachines',
    'Microsoft.Storage/storageAccounts',
    'Microsoft.ContainerRegistry/registries'
  ];

  if (highRiskTypes.includes(resource.type)) return 'high';
  if (mediumRiskTypes.includes(resource.type)) return 'medium';
  return 'low';
}

The UI displays unmanaged resources with:

Risk level badges (high/medium/low)
Estimated monthly cost
One-click "Import to Terraform" action
Ownership lookup from tags or activity logs

Cost Estimation

Before any deployment, developers see estimated costs:

async function estimateCost(blueprint, parameters) {
  const pricingClient = new AzureRetailPricesClient();

  // Map blueprint resources to Azure pricing SKUs
  const resourceCosts = await Promise.all(
    blueprint.resources.map(async (resource) => {
      const sku = resolveSkuFromParameters(resource, parameters);
      const prices = await pricingClient.query({
        armRegionName: parameters.region,
        serviceFamily: resource.serviceFamily,
        skuName: sku
      });

      return {
        resource: resource.name,
        sku,
        hourlyRate: prices[0]?.retailPrice || 0,
        monthlyEstimate: (prices[0]?.retailPrice || 0) * 730
      };
    })
  );

  const totalMonthly = resourceCosts.reduce((sum, r) => sum + r.monthlyEstimate, 0);

  return {
    resources: resourceCosts,
    totalMonthly,
    totalYearly: totalMonthly * 12,
    currency: 'USD'
  };
}

The PR description includes a cost breakdown:

## 💰 Cost Estimate

| Resource | SKU | Monthly |
|----------|-----|---------|
| Container App | 0.5 vCPU, 1Gi | $36.50 |
| PostgreSQL Flexible | B1ms | $12.41 |
| Storage Account | Standard_LRS | $2.30 |

**Total: ~$51.21/month**

_Estimates based on Azure retail pricing. Actual costs may vary._

The Resource Graph

Visualizing dependencies helps developers understand what they're deploying:

// Build dependency graph from Terraform state and Azure Resource Graph
async function buildResourceGraph(deploymentName) {
  const state = await getTerraformState(deploymentName);
  const nodes = [];
  const edges = [];

  for (const resource of state.resources) {
    nodes.push({
      id: resource.instances[0].attributes.id,
      label: resource.name,
      type: resource.type,
      status: await getResourceHealth(resource.instances[0].attributes.id)
    });

    // Find dependencies from Terraform
    if (resource.instances[0].dependencies) {
      for (const dep of resource.instances[0].dependencies) {
        edges.push({
          source: resource.instances[0].attributes.id,
          target: dep
        });
      }
    }
  }

  // Enrich with Azure Resource Graph relationships
  const azureGraph = await resourceGraph.query(`
    ResourceContainers
    | where id == '${deploymentResourceGroup}'
    | project-away tenantId
    | join kind=leftouter (
        Resources | project id, name, type, resourceGroup
    ) on resourceGroup
  `);

  return { nodes, edges, azureGraph };
}

The frontend renders this as an interactive graph where developers can:

Click nodes to see resource details
See health status (healthy/degraded/unhealthy)
Trace dependencies upstream and downstream
Filter by resource type or status

Self-Hosting: The Portal Deploys Itself

The ultimate test: can the portal deploy itself?

Yes. The Container App, Storage Account, and Front Door that host the portal were all created using portal blueprints. The bootstrap process:

Initial manual setup: Create resource group, storage for Terraform state, GitHub App
Deploy backend: Use azure-container-app blueprint
Deploy frontend: Use azure-static-website blueprint
Deploy CDN: Use azure-front-door blueprint
Configure DNS: Point portal.chrishouse.io to Front Door

From that point, all updates go through the portal itself.

What I Learned

Blueprint design matters more than UI polish. Well-designed blueprints with sensible defaults reduce form complexity dramatically.
GitOps is the right abstraction for infrastructure. PRs provide review, rollback, and audit trails automatically.
Cost visibility changes behavior. When developers see "$500/month" before clicking deploy, they ask questions.
Unmanaged resource discovery is surprisingly valuable. Every organization has shadow IT. Making it visible is the first step.
Backstage and custom portals can coexist. Backstage handles catalog and scaffolding well; custom UIs handle specialized workflows better.

What's Next

Policy-as-code: Integrate OPA/Gatekeeper to enforce standards before PRs are created
Drift detection: Alert when deployed resources diverge from Terraform state
Cost anomaly alerts: Notify when spending exceeds estimates
Multi-cloud: Extend blueprints to AWS and GCP

The portal is live at https://portal.chrishouse.io—built and hosted entirely in my personal Azure lab.

How I Built Ephemeral PR Environments on AKS

Fri, 28 Nov 2025 00:00:00 GMT

"Can you deploy this to staging so I can test it?"

If you've heard this a hundred times, you know the pain. Shared staging environments become bottlenecks. Developers step on each other's changes. QA can't reproduce bugs because someone else deployed over the fix.

I wanted something better for my Cloud Self-Service Portal project: every pull request gets its own isolated environment, automatically, with zero manual intervention.

Here's how I built it.

The Architecture

When a PR is opened against my repo, GitHub Actions:

Builds a Docker image tagged with the PR number
Creates a dedicated Kubernetes namespace (armportal-pr-{PR_NUMBER})
Deploys the app with its own ingress to AKS
Posts the live URL back to the PR as a comment

When the PR is closed or merged? Everything gets cleaned up automatically.

PR #42 gets:

Namespace: armportal-pr-42
URL: https://portal-api-pr-42.pr.chrishouse.io
Its own secrets, configs, and TLS certificate

The Wildcard Certificate Trick

Here's where it gets interesting. You might think each PR environment needs its own TLS certificate. That would mean:

Waiting for Let's Encrypt rate limits
Managing dozens of certificates
Slower deployments

Instead, I use a single wildcard certificate for *.pr.chrishouse.io.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: pr-wildcard-tls
  namespace: cert-manager
spec:
  secretName: pr-wildcard-tls
  dnsNames:
    - "*.pr.chrishouse.io"
  issuerRef:
    name: letsencrypt-dns01
    kind: ClusterIssuer

One certificate, unlimited PR environments. PR #1 through PR #9999 all just work.

The catch? Wildcard certificates require DNS-01 validation (HTTP-01 can't verify wildcards). That's where Cloudflare comes in.

Cloudflare DNS-01 Challenge

Let's Encrypt needs to verify I own *.pr.chrishouse.io. With DNS-01, cert-manager automatically:

Creates a TXT record in Cloudflare: _acme-challenge.pr.chrishouse.io
Let's Encrypt verifies the record exists
Certificate is issued
TXT record is cleaned up

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns01
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token

The Cloudflare API token lives in Azure Key Vault and gets synced to Kubernetes via a daily CronJob using the Secrets Store CSI Driver. If I rotate the token in Azure, Kubernetes picks it up automatically.

The Deployment Flow

When a PR is opened, here's what happens:

1. Build & Push

Docker image gets built and pushed to Azure Container Registry with a PR-specific tag:

- name: Build and push Docker image
  run: |
    docker build -t $ACR_REGISTRY/armportal-backend:pr-${{ github.event.pull_request.number }} .
    docker push $ACR_REGISTRY/armportal-backend:pr-${{ github.event.pull_request.number }}

2. Create Namespace & Copy Secrets

The clever bit: I copy production secrets to the PR namespace, stripping Kubernetes metadata with jq so the apply doesn't fail. Same pattern for the wildcard TLS certificate—copy it from cert-manager namespace to the PR namespace.

kubectl get secret backend-secrets -n armportal-backend -o json | \
  jq 'del(.metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp)' | \
  jq '.metadata.namespace = "armportal-pr-'$PR_NUMBER'"' | \
  kubectl apply -f -

3. Deploy with Templating

I use envsubst to inject PR-specific values into a deployment template:

export PR_NUMBER=${{ github.event.pull_request.number }}
envsubst < pr-deployment-template.yaml | kubectl apply -f -

4. Post the URL

GitHub Actions comments on the PR with the live environment URL and health check endpoint:

- name: Comment on PR
  uses: actions/github-script@v7
  with:
    script: |
      github.rest.issues.createComment({
        owner: context.repo.owner,
        repo: context.repo.repo,
        issue_number: context.issue.number,
        body: `## PR Environment Deployed!

        **API URL:** https://portal-api-pr-${prNumber}.pr.chrishouse.io
        **Health Check:** https://portal-api-pr-${prNumber}.pr.chrishouse.io/api/health`
      })

Auto-Cleanup: The Unsung Hero

The deployment is cool. The cleanup is what makes it sustainable.

When a PR is closed (merged or abandoned), GitHub Actions:

on:
  pull_request:
    types: [closed]

jobs:
  cleanup:
    steps:
      - name: Delete namespace
        run: kubectl delete namespace armportal-pr-${{ github.event.pull_request.number }}

      - name: Delete Docker image
        run: |
          az acr repository delete \
            --name $ACR_NAME \
            --image armportal-backend:pr-${{ github.event.pull_request.number }} \
            --yes
        continue-on-error: true

Deletes the Kubernetes namespace — cascades to everything inside (pods, services, ingress, secrets, configmaps)
Deletes the Docker image from ACR — with continue-on-error so failed builds don't break cleanup

One command, complete cleanup.

Cost Optimization

PR environments are intentionally lightweight:

Setting	Value
Replicas	1 (no HA needed for testing)
CPU Request	100m
Memory Request	128Mi
Storage	None (stateless by design)

The AKS cluster runs on a single B2s node (2 vCPU, 4GB RAM). With proper resource limits, I can run 10+ PR environments simultaneously without issues.

What I Learned

Wildcard certs scale infinitely. One certificate handles unlimited PR environments without rate limits or per-deployment delays.
DNS-01 > HTTP-01 for automation. No need to expose port 80 or deal with ingress routing during cert validation.
Namespace-per-PR is the right abstraction. Kubernetes namespaces provide perfect isolation with cascading deletes built in.
Copy secrets, don't recreate them. The jq metadata stripping pattern is simple but powerful.
Post URLs to PRs. Discoverability matters. If developers can't find the environment, they won't use it.

The Result

Every PR now gets:

A live, isolated environment in ~2 minutes
HTTPS with valid certificates
Automatic cleanup when merged/closed
Zero manual intervention required

Reviewers can test changes in production-like conditions. QA can reproduce bugs on the exact commit. No more "works on my machine."

Cyber Mario — Public Testing Beta

Sun, 14 Sep 2025 00:00:00 GMT

Welcome to the Cyber Mario testing beta — a neon-soaked platformer engineered for the open web.
It’s fast, punchy, and built to feel great on both keyboard and gamepad, desktop or phone.

🚀 Play the Beta

The game runs right here, in the blog — no install, no sign-in.
Launch it here → Play Cyber Mario
Grab a controller if you’ve got one. Press ~ to open the map.
See how far you can push your chain, and tag your best flag-pole bonus.

Neon on. Coins up. See you in World 1-1.

🎮 How to Play

Move: Arrow keys / A-D or left stick
Jump: Z / Space / A (Xbox)
Fire (when powered): X (Xbox) / K (keyboard)
Slam: Press Down while in the air
World Map: Press ~ (Backquote) or click World Map ▾
Reserve power-up (SMW-style): Press R to drop from the top-center box

The HUD keeps things clean: hearts, time, coins, and your running score. A lightweight profiler HUD (Ctrl+Shift+D) is available for devs who want frame timing and draw-call stats.

✨ What’s New in the Beta

A SMW-inspired Reserve Box

Pick up a mushroom/wing/fire while you’re already powered? It gets tucked into a top-center reserve box. Tap/click it or press R to deploy — perfect for clutch saves or boss runs.

World Map with True Unlocks

The World Map auto-filters and shows lock icons on levels you haven’t earned yet. Beat 1-2 to unlock 1-3, and so on — first level in each world is always open. Locks match the castle lock styling for a consistent vibe.

Finish a level and see a single, consolidated summary: coins, enemy KOs, chain max, flag bonus, time bonus, clear bonus, and the total. From there, confirm and roll into the next course.

Controller UX on Desktop

A small controller indicator sits at the top-right: grey when disconnected, cyber-lime when a pad is live. You’ll also get toasts on connect/disconnect and a one-click controls sheet tailored for Xbox mapping.

🧠 Design Notes

Controls That Forgive (and Reward)

The jump logic includes coyote time and a jump buffer, so late jumps still register, and you can chain stomps or slam attacks without fighting your inputs.

Scoring That Feels Right

Like SMW, score is lifetime-style across a run and tallies only when you clear a course. Sub-levels contribute to their parent; dying mid-course won’t corrupt your saved totals.

Sub-Levels & Checkpoints

Ducts, pipes, and side rooms act as segments; their loot and KOs feed the parent level’s totals. Checkpoint flags provide a one-time respawn with remaining time preserved.

📟 Tech Under the Hood (for the curious)

Pure Canvas + React single page with careful draw batching
PWA niceties: manifest, theme color, iOS icons, and full-screen helpers
Mobile polish: orientation nudge, input-zoom prevention, touch-gesture hard locks, and dynamic viewport sizing
Local storage: world-clear state and scoring durability live client-side now, with a clean path to a server later
Performance tools: an optional on-screen frame budget profiler for FPS/ms and tile/draw counts

(If you like peeking behind the curtain, many of these features are wired directly in cyber-mario.js — reserve box, world map, controller UI, summary modal, mobile UX helpers, and more.)

🗺 Progression & Unlocks

First level of each world is always playable.
Subsequent levels unlock once you’ve cleared the previous main level.
Sub-levels never appear in the map and always roll up to their parent.
Your cleared worlds persist in cm_worlds_cleared_v1 (browser storage).

🔊 Audio & Feel

A chill chiptune loop sets the mood. Jumps, wins, and the occasional Boo cackle round out the soundscape. Toggle BGM on/off from the HUD anytime.

📱 Devices & Performance

Desktop: Chrome or Edge recommended; gamepad strongly supported.
iOS/Android: Mobile-first tweaks reduce accidental pinch-zoom and keep the canvas full-bleed.
Debug HUD: Enable when needed; it’s hidden by default so players stay immersed.

🧪 What to Test Right Now

Map unlocks after clearing a level (1-2 → 1-3, etc.)
Reserve box behavior when picking up a second power-up
Course-clear modal totals vs HUD totals
Gamepad mapping: A = Jump, X = Fire; slam + world-map shortcuts
Sub-level coin/score roll-up into the parent course

If you spot glitches, capture a short clip or screenshot plus the level ID and what you were doing just before it happened.

🛣 Roadmap

Boss pass with telegraphed patterns and arena locks
Particle polish on stomps, bricks, and flag bonuses
Optional leaderboards (opt-in, seed-based submission)
Achievements and time-attack variants

Terminal Hacker — Season 1: Access Granted

Sat, 30 Aug 2025 00:00:00 GMT

Welcome to Terminal Hacker, a retro-inspired browser game hidden right inside this blog.

It looks like a terminal.
It feels like a terminal.
But behind the prompt lies a season-based hacking challenge with secrets, puzzles, and cheeky Easter eggs.

🎮 How it Works

Terminal Hacker simulates a Unix-like shell environment directly in your browser.

Type commands at the prompt, just like you would on Linux.
Explore a file system with /etc, /var/log, /home/guest, and systemd unit files.
Investigate logs in /var/log/journal and /var/log/pods.
Summon raptors (yes, really).
Track your progress with built-in stats and sharing.

Your progress is stored in browser local storage — no backend, no signup.
When you run shutdown or reboot, your progress resets (but your seed identity persists).

📟 The Core Experience

Every command counts toward progress:

Commands run
Discoveries made
Uptime kept

At any time you can generate a shareable ASCII card of your stats.
Copy, paste, brag.

✨ Season 1: Access Granted

This is the inaugural season.
Your mission: Access Granted, explore, and uncover as much as possible before the season ends.

There are commands to master, logs to read, and secrets hidden in unexpected places.
No walkthroughs. No spoilers. Just your wits and curiosity.

At the end of your run, you’ll have a seed like CH-AB12CD —
your hacker ID for Season 1.

🏅 Badges (Season 1)

Season 1 includes five badges to discover:

Explorer
Archivist
Summoner
Bootloader Clear
Chronos

Collect them all to maximize your progress.

📚 Command Reference (Highlights)

ls, cd, pwd, cat, touch, echo, tee, mkdir, rm, mv, cp

Search & Logs

grep, grep -R, find, locate, journalctl, systemctl, klogs

Fun & Flavor

fortune, raptor, bootloader, qte, hint, share, sudo, uptime, uuid, cal, random

🔄 Resetting the Box

Sometimes you dig too deep, delete the wrong thing, or just want to try a different path.
That’s what the reset commands are for:

reboot — restart the system, keep your seed.
restart — alias for reboot.
shutdown — power off and clear progress.

Your seed identity always remains. Everything else — files, notes, progress — resets.
Think of it as wiping your VM clean and starting fresh.

Hints aren’t free — you have to earn them.

Type hint and you’ll get a random clue.
When you run out, type share to generate an ASCII stats card.
Sharing it earns you one more hint for your run.

The best players use hints sparingly — because the real fun is discovery.

🕹 Why It’s Fun

Feels real. Boot messages, system logs, and file structures make it feel alive.
Playful surprises. Raptors, boss fights, hidden commands.
Hacky nostalgia. If you’ve ever SSH’d into a Raspberry Pi, it feels familiar.
Social bragging. Share your ASCII stats card — your hacker cred in monospace glory.

🗓 Seasons Roadmap

Terminal Hacker isn’t just one box — it’s a saga.

🔥 Season 2 — Daemon Dance

Services come alive. Pods get noisier, and logs whisper secrets.

🌐 Season 3 — Network Intrigue

Networking commands appear. Hidden hosts respond.

📦 Season 4 — Container Chaos

Containers and pods expand. Debugging becomes the game.

👑 Season 5 — Final Root

Root secrets, privilege escalation, and a final boss.

🚀 Coming Soon Features

Beyond the seasons, bigger features are in the works:

Leaderboards — global ranking
Score submission — secure seed + stats upload
Tamper-proofing — scores can’t be faked

🎯 What’s Next?

Your seed will carry forward.
Progress stacks season over season, building your hacker legacy.

🔮 Final Teaser

The raptor is just the beginning.
The real fun starts when the box fights back.

Access Granted.
Crack the system.
And get ready for Daemon Dance in Season 2.

From Raspberry Pi to the Cloud: Publishing Kubernetes Dashboard Securely with Cloudflare Tunnel

Mon, 25 Aug 2025 00:00:00 GMT

Over the weekend I decided to take my Raspberry Pi 5 a step further and turn it into a Kubernetes playground. My goal was to get the Kubernetes Dashboard up and running and then make it securely accessible from the internet, without relying on port forwarding or static IPs. The tools I chose were K3s for the cluster, Cloudflare DNS for my domain (chrishouse.io), and Cloudflare Tunnel to publish the service safely.

Installing K3s

The first step was the cluster itself. I initially tried MicroK8s but hit snapd version issues, so I pivoted to K3s which installed quickly. On the Pi I ran:

curl -sfL https://get.k3s.io | sh -

A quick check confirmed my node was live:

sudo k3s kubectl get node

Deploying the Kubernetes Dashboard

Once the cluster was healthy, I deployed the Kubernetes Dashboard. The manifests are published by the project, so installation was straightforward. The service exposed itself via a NodePort on 30001, and I verified it was working locally:

sudo k3s kubectl -n kubernetes-dashboard get svc

Which returned:

kubernetes-dashboard   NodePort   10.43.6.40   <none>   443:30001/TCP

From there I tested directly on the Pi:

curl -vk https://localhost:30001

The output confirmed a TLS handshake with a self-signed certificate and an HTTP 200 response containing the HTML of the dashboard login page.

Adding Cloudflare DNS and Tunnel

The challenge came when I wanted to expose this beyond my LAN. My ISP provides a dynamic IP, my Orbi router doesn’t like giving up ports 80 and 443, and I didn’t want to maintain fragile port forwarding rules anyway. That’s when I turned to Cloudflare.

I had already registered a domain (chrishouse.io) and moved DNS into Cloudflare. From there I installed cloudflared on the Pi:

curl -fsSL https://pkg.cloudflare.com/cloudflare-main.gpg | sudo tee /usr/share/keyrings/cloudflare-main.gpg >/dev/null
echo 'deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared jammy main' | sudo tee /etc/apt/sources.list.d/cloudflared.list
sudo apt update
sudo apt install cloudflared -y

After installation, I authenticated it against my Cloudflare account:

cloudflared tunnel login

That opened a browser window where I logged in and selected my domain. Credentials were saved to ~/.cloudflared/cert.pem.

I created a named tunnel for the dashboard:

cloudflared tunnel create k3s-dashboard

And mapped it to my subdomain:

cloudflared tunnel route dns k3s-dashboard dashboard.chrishouse.io

This created a CNAME record in Cloudflare pointing dashboard.chrishouse.io at the tunnel rather than my home IP.

Configuring the Tunnel

The last piece was configuring the tunnel itself. I created a config file:

sudo nano /etc/cloudflared/config.yml

With the contents:

tunnel: k3s-dashboard
credentials-file: /home/chris/.cloudflared/<tunnel-id>.json

ingress:
  - hostname: dashboard.chrishouse.io
    service: https://localhost:30001
    originRequest:
      noTLSVerify: true
  - service: http_status:404

The critical setting here was noTLSVerify: true because the Dashboard serves HTTPS with a self-signed certificate. Without that, Cloudflared would reject the connection.

Finally, I installed it as a service:

sudo cloudflared service install
sudo systemctl enable cloudflared
sudo systemctl start cloudflared

And checked the logs with:

sudo journalctl -u cloudflared -f

The logs showed multiple connections registered to Cloudflare edge locations. At that point, opening https://dashboard.chrishouse.io from my browser worked flawlessly.

Instead of a self-signed cert, the connection terminated at Cloudflare’s edge with a valid certificate, and the dashboard login page loaded securely.

Closing Thoughts

What started as an experiment in running Kubernetes on a Raspberry Pi ended up being a deep dive into modern networking. By leveraging Cloudflare Tunnel, I didn’t have to expose my home IP, configure NAT rules on my router, or worry about whether my ISP changed my IP overnight. My Raspberry Pi simply maintained an outbound connection to Cloudflare’s global edge, and Cloudflare handled TLS termination and routing.

It’s remarkable that with a $60 Raspberry Pi, an open-source Kubernetes distribution, and a free Cloudflare account, I was able to publish a production-grade web application securely to the internet.

This approach doesn’t just apply to Kubernetes — the same pattern works for any internal service you want to access remotely without punching holes in your firewall.

As a related project, I’ve also been experimenting with Meshtastic and building a live mesh radio map for the Memphis area. That project, called Memphis Meshview, takes real-time MQTT packet data from local Meshtastic nodes and renders them on an interactive map. If you’d like to see a working example of how these lightweight IoT networks can be visualized in real time, you can check it out at https://meshview.chrishouse.io/map.

Beyond maps, it also provides a persistent conversation log where messages exchanged across the mesh network are stored and browsable at https://meshview.chrishouse.io/chat. It’s a live example of how these techniques can be used to make distributed, community-driven networks visible and interactive on the web.

Memphis Meshview Project: Visualizing Meshtastic with Maps and Chat Logs

Sat, 23 Aug 2025 00:00:00 GMT

Introduction

Meshtastic makes off-grid communication possible by linking LoRa radios into a self-healing mesh. But once you connect nodes to MQTT, the game changes — your local mesh can bridge into a global one, with all messages and positions flowing through a central broker.

This inspired the Memphis Meshview project, where I set up a dashboard to visualize Meshtastic activity around Memphis. The goal was twofold:

Map the nodes reporting in via MQTT.
Persist chat logs for historical reference.

Together, this creates a living snapshot of the mesh — a community map where you can see who’s online, where they are, and what messages are flowing.

MQTT as the Backbone

Meshtastic nodes normally just talk directly over LoRa. But when a node has internet access, it can publish messages to an MQTT broker (and subscribe back).

In our case:

The broker connects to the Memphis namespace (msh/US/memphisme.sh).
Every node that has MQTT enabled forwards its messages there.
Other MQTT-connected nodes receive the same traffic, even if they’re physically too far apart to hear each other over radio.

This creates a virtual backbone that ties together distant or isolated clusters.

Building the Live Map

The map piece uses:

Node coordinates: Many Meshtastic devices can report GPS location.
Frontend visualization: A Leaflet.js-based map hosted at meshview.chrishouse.io/map.
Backend logic: MQTT subscriptions feed updates into a datastore, which then updates the map markers.

Each node shows up as a point, with metadata like:

Callsign or device name
Last seen time
Altitude (when reported)
Whether it came in via radio or via MQTT

This lets the community quickly see coverage holes and hotspots.

Persistent Chat Logs

Beyond mapping, I wanted conversation history. Normally, Meshtastic messages are ephemeral — if you miss them live, they’re gone.

To fix that:

I subscribed to the MQTT chat topics.
Messages were written into a database with timestamps, user IDs, and message content.
A simple web front-end at meshview.chrishouse.io/chat lets you scroll back through the conversation log.

This made the mesh feel more like a living community, not just random bursts of radio packets.

Lessons Learned

MQTT is a Force Multiplier
Even if your local radio range is small, MQTT bridges make the mesh global. One solar node in Memphis can trade messages with a handheld in Colorado.
Data Normalization Matters
Different nodes send different fields. Some send GPS reliably, others don’t. Building a clean map means handling messy input gracefully.
Persistence Changes Behavior
Once people knew there was a persistent chat log, conversations became more thoughtful — it shifted from ephemeral pings to community discussions.
Visualization Inspires Growth
The map encouraged others to light up their nodes, just to “see themselves” show up in Memphis Meshview. A visible network drives participation.

Closing Thoughts

Meshtastic’s beauty lies in its simplicity: radios talking to each other with no infrastructure. But once you add MQTT, you unlock a shared digital layer that lets local meshes become part of a bigger story.

The Memphis Meshview project is one example — a mix of mapping, chat logging, and visualization that makes the invisible mesh visible. And in a world where off-grid communication matters more every year, seeing the network grow in real time is as inspiring as it is useful.

Building a Personal Mesh Network with Meshtastic

Sat, 23 Aug 2025 00:00:00 GMT

Introduction

Mesh networking has always fascinated me. The idea that devices can form their own network without relying on cell towers, Wi-Fi, or even the internet is powerful — especially in rural or off-grid scenarios.

Over the last few weeks, I set out to build a personal 3-node Meshtastic network to learn more about LoRa radios, MQTT integrations, and practical deployment considerations. My setup included:

A mobile node (car / handheld).
A solar-powered outdoor node.
A desk node that always listens for messages.

Here’s what I learned about node roles, placement, and how to avoid common mistakes.

Node Setup

1. Mobile Node (Car / Portable)

Hardware: Heltec LoRa V3 (with onboard battery charging via USB-C).
Role: Client.
Why: Mobile nodes move frequently and don’t provide stable routing. They’re best kept as clients that connect to stronger repeater or router nodes.

2. Solar-Powered Outdoor Node

Hardware: Heltec LoRa board + small LiPo battery + 10W solar panel with charge controller.
Placement: Mounted ~30 ft on my roofline.
Role: Initially tested as Router, but later kept as Client.
Lesson: Unless you can mount an antenna very high (100 ft above ground level or better), routers often just add network noise. A client role avoids unnecessary re-broadcast loops while still being available when powered.

3. Desk Node (Always-On Receiver)

Hardware: Heltec LoRa board powered by USB-C on my desk.
Role: Client.
Why: It doesn’t need to forward messages — it simply provides a stable “listener” for monitoring the network.

Understanding Node Roles

Meshtastic supports different roles:

Client → Best for most setups. Receives and transmits messages but doesn’t rebroadcast everything.
Router → For very high, clear-line-of-sight placements (e.g., mountaintops, tall towers). Always rebroadcasts, ensuring wide coverage.
Repeater → Similar to router, but specifically designed to extend coverage in areas where clients cannot directly connect. Needs very good placement.

In practice:
👉 Unless you can get your antenna very high with clear tree lines, stick to Client mode. Misconfigured routers at low height can actually hurt the mesh by creating redundant traffic.

Lessons Learned

Antenna Height Matters More Than Power
My 30 ft rooftop node wasn’t high enough to justify being a router. At lower heights, the Fresnel zone gets blocked by trees, houses, and terrain.
Battery Management Is Built In
Boards like the Heltec LoRa V3 include a TP4056-style charging circuit. Plug in USB (or solar with charge controller), and the board manages the LiPo charging automatically.
Don’t Overcomplicate Roles
Many people think they need a router/repeater to make their mesh “better.” In reality, most personal networks run best with all nodes as clients.
Long Range Settings (LoRa Profiles)
Meshtastic includes “LongFast” and “LongSlow” profiles.
- LongFast → Good balance of speed and range.
- LongSlow → Extreme range, but very slow.
“VeryLong” was removed; “LongSlow” is now the equivalent option.
MQTT Adds Reach Beyond Radio Range
While not needed for my personal 3-node test, Meshtastic can bridge over MQTT to let remote nodes join the mesh. Useful if your physical range is limited.

Next Steps

Test larger solar setups to keep outdoor nodes alive through winter.
Add GPS tracking to the mobile node.
Experiment with MQTT to bridge my desk node into the wider Memphis Meshview community project.

Closing Thoughts

Building a personal mesh network taught me that simplicity wins: keep most nodes as clients, get antennas as high as you can, and don’t overcomplicate the setup.

Even a small 3-node network is surprisingly capable — messages reliably hop between my car, roof, and desk. Combined with solar power, this little experiment has shown me the potential of LoRa-based mesh systems for both hobbyist fun and serious off-grid communication.

How I Built and Deployed My Gatsby Blog on Raspberry Pi with Docker, Helm, and Kubernetes

Wed, 20 Aug 2025 00:00:00 GMT

This blog is both a playground and a living documentation platform for my experiments.
It’s running directly on a Raspberry Pi 5 sitting on my desk. I use Gatsby.js as the static site generator, Docker for packaging, K3s (a lightweight Kubernetes distribution) for orchestration, and Helm to make deployments repeatable. Everything is exposed securely via Cloudflare Tunnel.

This post is a deep dive into how I put this together — not just the commands, but the reasoning behind each step, the pitfalls I hit, and the lessons learned along the way.

1. Why Raspberry Pi and Kubernetes?

Before diving into commands, let’s talk about why I even did this.

Raspberry Pi 5: It’s cheap ($60), ARM64-based, and powerful enough to run real workloads. With 8GB RAM, it can host Kubernetes pods and serve real traffic.
K3s: Normal Kubernetes is overkill for a Pi. K3s is trimmed down, lightweight, and optimized for edge devices and IoT. Perfect for homelabs.
Gatsby.js: Static site generators make blogs blazing fast, secure, and easy to deploy as containers.
Docker + Helm: Docker lets me package my site so it runs the same everywhere. Helm makes redeployment a one-liner instead of managing multiple kubectl configs.

So this stack is about learning and also proving that a $60 Pi can host a “production-grade” site.

2. Setting up K3s on the Pi

I originally tried MicroK8s, but ran into snapd version issues on Raspberry Pi OS. K3s was dead simple:

curl -sfL https://get.k3s.io | sh -

Check the node:

sudo k3s kubectl get nodes -o wide

This gives you the single-node cluster ready to schedule workloads.

Key notes:

K3s bundles containerd by default (no need for Docker runtime).
NodePorts and LoadBalancers just work.
Systemd integration makes K3s restart automatically on reboot.

3. Building the Blog with Gatsby

Start with Gatsby’s starter template:

npm init gatsby
cd blog
npm install
npm run develop

This gives you hot reloading and a skeleton site. I then customized it:

Added a terminal-inspired theme (CRT effects, scanlines, boot logs).
Created Markdown-based blog posts inside /content/posts.
Added tags to frontmatter so I could filter posts later.

Once the site worked locally, I ran:

gatsby build

This outputs static HTML into /public — ready to be served.

4. Dockerizing Gatsby Output

The goal: run Gatsby’s static HTML in Nginx inside a container.

Here’s the Dockerfile:

FROM nginx:alpine
COPY public /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf

The nginx.conf ensures single-page app routing works correctly (important for 404 pages and client-side navigation):

server {
  listen 80;
  server_name _;

  root /usr/share/nginx/html;
  index index.html;

  location / {
    try_files $uri /index.html;
  }
}

Then build for ARM64:

docker buildx build --platform linux/arm64 -t 192.168.1.66:32358/gatsby-blog:latest . --push

Verify:

docker buildx imagetools inspect 192.168.1.66:32358/gatsby-blog:latest

This shows manifest lists (good for ARM64 compatibility).

5. Local Container Registry on the Pi

I didn’t want to push to Docker Hub, so I use the Pi as its own registry (via containerd + NodePort).

After pushing, confirm the image is there:

sudo ctr -n k8s.io images ls | grep gatsby-blog

If it shows up, the cluster can pull it.

Pro tip: Add insecure-registries to your Docker config if you’re not using TLS on the Pi registry:

{
  "insecure-registries": ["192.168.1.66:32358"]
}

6. Deploying with Helm

Instead of manually writing kubectl apply -f, I use Helm. My chart:

# values.yaml
image:
  repository: 192.168.1.66:32358/gatsby-blog
  tag: latest
  pullPolicy: Always

service:
  type: NodePort
  port: 80
  targetPort: 80
  nodePort: 32360

ingress:
  enabled: true
  className: "nginx"
  hosts:
    - host: blog.chrishouse.io
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: blog-tls
      hosts:
        - blog.chrishouse.io

Deploy:

helm upgrade --install gatsby ./helm -n blog --create-namespace   --set image.repository=192.168.1.66:32358/gatsby-blog   --set image.tag=latest

Kubernetes pulls from the Pi registry, starts a pod, and exposes it at NodePort 32360.

Check it:

sudo kubectl -n blog get pods -o wide
sudo kubectl -n blog get svc gatsby-gatsby-blog

7. Exposing with Cloudflare Tunnel

I didn’t want to port forward. Enter Cloudflare Tunnel:

tunnel: 4eab82fd-9172-4c0f-aaeb-237c72452dbe
credentials-file: /home/chris/.cloudflared/4eab82fd-9172-4c0f-aaeb-237c72452dbe.json

ingress:
  - hostname: blog.chrishouse.io
    service: http://localhost:32360
  - service: http_status:404

Install and run:

cloudflared tunnel create blog
cloudflared tunnel route dns blog blog.chrishouse.io
sudo cloudflared service install
sudo systemctl start cloudflared

Now requests to https://blog.chrishouse.io are routed securely through Cloudflare.

Benefit: No exposed home IP, no router config, free TLS.

8. Automating the Workflow

I don’t want to manually run 10 commands. So in package.json I added:

"scripts": {
  "build": "gatsby build",
  "docker:build": "docker buildx build --platform linux/arm64 -t 192.168.1.66:32358/gatsby-blog:latest . --push",
  "helm:deploy": "helm upgrade --install gatsby ./helm -n blog --create-namespace --set image.repository=192.168.1.66:32358/gatsby-blog --set image.tag=latest",
  "deploy": "npm run build && npm run docker:build && npm run helm:deploy"
}

Now it’s literally one command:

npm run deploy

9. Lessons Learned Along the Way

Caching issues: Sometimes pods don’t pull the latest image. kubectl rollout restart deploy gatsby-gatsby-blog fixes it.
Resource limits: The Pi can handle ~10 pods, but keep an eye on memory.
Cloudflare TLS: Don’t fight self-signed certs. Terminate TLS at Cloudflare edge.
Helm DRY: Extract common charts into a shared repo (my_helm).
Logging: Container logs are visible in kubectl logs and shipped to Elastic.

10. Future Plans

Add CI/CD via GitHub Actions to trigger npm run deploy.
Add Grafana dashboards on the same Pi (already running Prometheus).
Expand posts and make /blog/:slug the default for navigation.

Final Thoughts

With Gatsby, Docker, Helm, and Cloudflare Tunnel, I now have a full production-like pipeline running on a Raspberry Pi.
This setup is overkill for a personal blog — but that’s the point. It’s a playground to learn Kubernetes, CI/CD, and security practices in a safe, low-cost way.

If a Pi can do it, so can your enterprise cluster.

Inside a Dev Container: How to Standardize .NET Development

Tue, 29 Jul 2025 00:00:00 GMT

Spinning up a .NET application locally in a corporate environment often turns into an exercise in frustration. Developers deal with inconsistent SDK versions, corporate proxy certificates, authentication errors against private feeds, and missing tools. Instead of writing code, they spend hours configuring machines.

Dev Containers fix that.

Dev Containers are Docker-based development environments defined using .devcontainer config files and launched via Visual Studio Code. By defining your development environment as code, Dev Containers let you version, automate, and standardize everything needed to get up and running — regardless of which machine or developer is involved.

Why This Matters

Problem                          →  Solved by
---------------------------------------------------------------
Proxy certificate errors          Trusting Zscaler root certs
HTTPS dev environment             dotnet dev-certs https
NuGet feed authentication         dotnet nuget add source with a secure PAT
"It works on my machine" bugs     Identical container builds
Inconsistent SDKs and tools       Pinned versions in Docker image

Whether onboarding new developers or debugging issues across machines, Dev Containers give us a consistent, secure, and versioned workspace — every time.

The Dockerfile: Building a Reliable .NET Environment

We use Microsoft’s official .NET SDK 8.0 image as our base:

FROM mcr.microsoft.com/dotnet/sdk:8.0

Key Features

Private NuGet feed authentication
We use a Personal Access Token (PAT) to connect to Azure DevOps feeds securely. This is injected as a build argument and configured at build time:

RUN dotnet nuget add source "${ADO_FEED_URL}"     --name "AzureDevOps"     --username "${ADO_FEED_USER}"     --password "${ADO_FEED_PAT}"     --store-password-in-clear-text

Is --store-password-in-clear-text secure?
This option is required when using dotnet nuget add source with credentials. While storing the password in plain text is not ideal for production containers, it is acceptable in temporary dev containers that are not distributed or reused. To enhance security, avoid checking in any secrets and limit access to the container environment.

Zscaler root certificate installation
In a corporate environment with deep packet inspection (like Zscaler), HTTPS traffic can be intercepted unless root certificates are trusted. We add the cert manually:

COPY ZscalerRootCerts/ZscalerRootCertificate.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

Dev HTTPS certificate setup
We generate and trust the HTTPS developer certificate so ASP.NET Core can serve over HTTPS locally:

RUN dotnet dev-certs https --clean &&     dotnet dev-certs https --trust || true

Standard tooling
We install common tools (curl, git, vim, etc.) so the container can support debugging, scripting, and customization.

Complete Dockerfile

FROM mcr.microsoft.com/dotnet/sdk:8.0

# Accept build args
ARG ADO_FEED_URL
ARG ADO_FEED_USER
ARG ADO_FEED_PAT

# Required ENV vars
ENV DOTNET_USE_POLLING_FILE_WATCHER=1     ASPNETCORE_URLS=http://+:8080     DOTNET_RUNNING_IN_CONTAINER=true     NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED=true

# Install tools
RUN apt-get update &&     apt-get install -y --no-install-recommends     ca-certificates     curl     wget     gnupg     git     unzip     bash     vim &&     apt-get clean &&     rm -rf /var/lib/apt/lists/*

# Add Zscaler certs
COPY ["ZscalerRootCerts/ZscalerRootCertificate.crt", "/usr/local/share/ca-certificates/ZscalerRootCertificate.crt"]
COPY ["ZscalerRootCerts/ZscalerRootCertificate.crt", "/etc/ssl/certs/ZscalerRootCertificate.crt"]
RUN update-ca-certificates

# Set up NuGet source securely
RUN dotnet nuget add source "${ADO_FEED_URL}"     --name "AzureDevOps"     --username "${ADO_FEED_USER}"     --password "${ADO_FEED_PAT}"     --store-password-in-clear-text

# Install ASP.NET dev certs
RUN dotnet dev-certs https --clean &&     dotnet dev-certs https --trust || true

# Final devcontainer setup
WORKDIR /workspace
EXPOSE 80
EXPOSE 443

devcontainer.json: Wiring VS Code into the Container

The devcontainer.json file tells VS Code how to build and configure the container:

"dockerFile": "../Dockerfile",
"build": {
  "args": {
    "ADO_FEED_URL": "...",
    "ADO_FEED_USER": "...",
    "ADO_FEED_PAT": "..."
  }
},
"workspaceFolder": "/workspace",
"remoteUser": "root",
"customizations": {
  "vscode": {
    "extensions": ["ms-dotnettools.csharp"]
  }
}

This defines the build args, mounts your local code folder into /workspace, sets up common VS Code extensions, and runs the container as root for ease of configuration.

End Result

Every developer on the team — regardless of their host OS or permissions — can run:

dotnet restore
dotnet run

…and everything works.

There are no surprises around certificate trust, no more configuring private feeds manually, and no wasted time figuring out what’s missing from a machine. It just works, every time.

📁 Sample Folder Structure

MyDotNetApp/
├── .devcontainer/
│   ├── devcontainer.json
│   └── Dockerfile
│
├── ZscalerRootCerts/
│   └── ZscalerRootCertificate.crt
│
├── src/
│   └── MyDotNetApp/
│       ├── MyDotNetApp.csproj
│       ├── Program.cs
│       └── Startup.cs
│
├── MyDotNetApp.sln
└── README.md

How to Use

If you’re new to Dev Containers, getting started is simple — especially in Visual Studio Code.

Prerequisites

Install Docker Desktop (or Podman + Docker CLI on Linux)
Install Visual Studio Code
Install the Dev Containers extension

Steps to Use

Clone your repository and make sure the .devcontainer folder exists.
Open the repo in VS Code.
Press F1 (or Ctrl+Shift+P) and run:
Dev Containers: Reopen in Container
VS Code will build and launch the container, automatically installing dependencies, mounting your source code, and setting up extensions.
Once the container opens, use the terminal or debugging features as you normally would. You’re now working inside the container.

Add a launch.json to launch the app

For .NET apps, add a .vscode/launch.json file to let VS Code run and debug the app inside the container:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": ".NET Core Launch (web)",
      "type": "coreclr",
      "request": "launch",
      "preLaunchTask": "build",
      "program": "${workspaceFolder}/bin/Debug/net8.0/MyDotNetApp.dll",
      "args": [],
      "cwd": "${workspaceFolder}",
      "stopAtEntry": false,
      "serverReadyAction": {
        "action": "openExternally",
        "pattern": "Now listening on: (https?://\\S+)"
      },
      "env": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      },
      "sourceFileMap": {
        "/Views": "${workspaceFolder}/Views"
      }
    }
  ]
}

Final Thoughts

Dev Containers have turned complex setup into a repeatable, version-controlled process. In enterprise environments, where security, repeatability, and private package feeds add complexity, Dev Containers let developers focus on building software instead of fixing environments.

Learn more: https://containers.dev

Converting MicroK8s Kube Prometheus Stack into a Helm Chart

Thu, 12 Jun 2025 00:00:00 GMT

This post shows how to export the Prometheus/Grafana stack that MicroK8s deploys, then restructure it into a Helm chart you can version, reuse, and customize. We’ll dump the manifests with kubectl, split them by resource kind, add Helm templating, and ship the chart to an OCI registry.

Prerequisites

MicroK8s installed and working (microk8s status shows running).
You previously enabled the Observability addon (Prometheus/Grafana/etc.).
Helm 3+ installed.
A shell with kubectl (use microk8s kubectl if you don’t have a separate kubeconfig).
Optional: yq (v4+) for YAML processing (makes splitting easier).

Confirm cluster access and the label we’ll filter on:

microk8s status --wait-ready

# See the resources and confirm the instance label.
microk8s kubectl get all -A -l app.kubernetes.io/instance=kube-prom-stack

If your addon uses a different label, adjust commands accordingly. Common variations include app=prometheus, app.kubernetes.io/part-of=kube-prometheus-stack, or a custom release label. The examples below use app.kubernetes.io/instance=kube-prom-stack throughout.

Step 1 — Dump everything to a single YAML

We’ll export all core workload types plus config and storage objects so nothing is missed.

# One-shot dump (workloads + config + storage), across all namespaces
microk8s kubectl get   deploy,sts,ds,job,cronjob,svc,ep,ingress,cm,secret,sa,role,rolebinding,clusterrole,clusterrolebinding,pvc   -A -l app.kubernetes.io/instance=kube-prom-stack -o yaml   > kube-prom-stack.dump.yaml

Some stacks also install CRDs and CRD-backed resources such as ServiceMonitor, PodMonitor, and PrometheusRule. Dump them too if present:

# Optional but recommended (won't fail if kinds don't exist)
for kind in servicemonitor.monitoring.coreos.com podmonitor.monitoring.coreos.com prometheusrule.monitoring.coreos.com; do
  microk8s kubectl get "$kind" -A -l app.kubernetes.io/instance=kube-prom-stack -o yaml >> kube-prom-stack.dump.yaml 2>/dev/null || true
done

Why a single file first? It’s easier to archive and review. We’ll split it in the next step.

Step 2 — Split the dump into logical files

You can do this by hand, but yq + a few lines of shell makes it painless.

# Make a staging folder
mkdir -p export-split

# Split the multi-doc YAML into numbered chunks
csplit -z -f export-split/chunk- kube-prom-stack.dump.yaml '/^---$/' '{*}'

# For each chunk, detect kind/name/ns and write a smart filename
for f in export-split/chunk-*; do
  kind=$(yq -r '.kind // ""' "$f")
  name=$(yq -r '.metadata.name // ""' "$f")
  ns=$(yq -r '.metadata.namespace // "default"' "$f")
  [ -z "$kind" ] && continue
  out="export-split/${ns}_${kind}_${name}.yaml"
  mv "$f" "$out"
done

Now you have a pile of files like monitoring_Deployment_prometheus.yaml, default_Service_grafana.yaml, etc. This is our source material for Helm templating.

Step 3 — Create a Helm chart skeleton

helm create kube-prom-stack
cd kube-prom-stack

# Remove the example templates Helm generated; we’ll add our own
rm -f templates/*

# Add helpers for names/labels
cat > templates/_helpers.tpl <<'EOF'
{{/*
Expand the chart name.
*/}}
{{- define "kube-prom-stack.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}

{{/*
Create a default fully qualified app name.
*/}}
{{- define "kube-prom-stack.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}

{{/*
Common labels
*/}}
{{- define "kube-prom-stack.labels" -}}
app.kubernetes.io/name: {{ include "kube-prom-stack.name" . }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end -}}
EOF

The helpers consolidate name/label logic across templates so you don’t repeat yourself.

Step 4 — Scaffold `values.yaml` with tunables

We’ll define images, persistence, service types, and Grafana credentials in values.yaml.

# values.yaml
global:
  namespace: monitoring

prometheus:
  image: prom/prometheus
  tag: v2.54.0
  replicas: 1
  service:
    type: ClusterIP
    port: 9090
  persistence:
    enabled: true
    storageClass: microk8s-hostpath
    size: 20Gi

grafana:
  image: grafana/grafana
  tag: "10.4.1"
  replicas: 1
  service:
    type: ClusterIP
    port: 3000
  adminUser: admin
  adminPassword: admin123 # consider overriding via values or an existingSecret
  ingress:
    enabled: false
    className: ""
    hosts: []
    tls: []

alertmanager:
  image: prom/alertmanager
  tag: v0.27.0
  replicas: 1
  service:
    type: ClusterIP
    port: 9093

rbac:
  create: true

serviceMonitors:
  enabled: true

prometheusRules:
  enabled: true

Keep credentials out of Git by using --set-file or a private values-prod.yaml. You can also support existingSecret patterns if you prefer K8s-managed secrets.

Step 5 — Start templating the core Deployments

Prometheus (Deployment)

# templates/prometheus-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "kube-prom-stack.fullname" . }}-prometheus
  namespace: {{ .Values.global.namespace | default .Release.Namespace }}
  labels:
    {{- include "kube-prom-stack.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.prometheus.replicas }}
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/instance: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app.kubernetes.io/name: prometheus
        app.kubernetes.io/instance: {{ .Release.Name }}
    spec:
      serviceAccountName: {{ include "kube-prom-stack.fullname" . }}-prometheus
      containers:
      - name: prometheus
        image: "{{ .Values.prometheus.image }}:{{ .Values.prometheus.tag }}"
        args:
          - "--config.file=/etc/prometheus/prometheus.yml"
          - "--storage.tsdb.path=/prometheus"
        ports:
          - name: http
            containerPort: 9090
        volumeMounts:
          - name: data
            mountPath: /prometheus
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: {{ include "kube-prom-stack.fullname" . }}-prometheus-pvc

We match on labels we control and keep PVC names deterministic via fullname + suffix.

Prometheus Service

# templates/prometheus-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: {{ include "kube-prom-stack.fullname" . }}-prometheus
  namespace: {{ .Values.global.namespace | default .Release.Namespace }}
  labels:
    {{- include "kube-prom-stack.labels" . | nindent 4 }}
spec:
  type: {{ .Values.prometheus.service.type }}
  ports:
    - name: http
      port: {{ .Values.prometheus.service.port }}
      targetPort: http
  selector:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: {{ .Release.Name }}

Services should select by pod template labels, not metadata labels, so scaling/restarts don’t break routing.

Prometheus PVC

# templates/prometheus-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: {{ include "kube-prom-stack.fullname" . }}-prometheus-pvc
  namespace: {{ .Values.global.namespace | default .Release.Namespace }}
  labels:
    {{- include "kube-prom-stack.labels" . | nindent 4 }}
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: {{ .Values.prometheus.persistence.size }}
  storageClassName: {{ .Values.prometheus.persistence.storageClass }}

Step 6 — Grafana (Deployment, Service, Secret, optional Ingress)

Secret (admin creds)

# templates/grafana-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: {{ include "kube-prom-stack.fullname" . }}-grafana-auth
  namespace: {{ .Values.global.namespace | default .Release.Namespace }}
type: Opaque
stringData:
  admin-user: {{ .Values.grafana.adminUser | quote }}
  admin-password: {{ .Values.grafana.adminPassword | quote }}

In production, handle secrets via existingSecret or external secret managers.

Deployment

# templates/grafana-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "kube-prom-stack.fullname" . }}-grafana
  namespace: {{ .Values.global.namespace | default .Release.Namespace }}
  labels:
    {{- include "kube-prom-stack.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.grafana.replicas }}
  selector:
    matchLabels:
      app.kubernetes.io/name: grafana
      app.kubernetes.io/instance: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app.kubernetes.io/name: grafana
        app.kubernetes.io/instance: {{ .Release.Name }}
    spec:
      containers:
      - name: grafana
        image: "{{ .Values.grafana.image }}:{{ .Values.grafana.tag }}"
        env:
          - name: GF_SECURITY_ADMIN_USER
            valueFrom:
              secretKeyRef:
                name: {{ include "kube-prom-stack.fullname" . }}-grafana-auth
                key: admin-user
          - name: GF_SECURITY_ADMIN_PASSWORD
            valueFrom:
              secretKeyRef:
                name: {{ include "kube-prom-stack.fullname" . }}-grafana-auth
                key: admin-password
        ports:
          - name: http
            containerPort: 3000

Service

# templates/grafana-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: {{ include "kube-prom-stack.fullname" . }}-grafana
  namespace: {{ .Values.global.namespace | default .Release.Namespace }}
  labels:
    {{- include "kube-prom-stack.labels" . | nindent 4 }}
spec:
  type: {{ .Values.grafana.service.type }}
  ports:
    - name: http
      port: {{ .Values.grafana.service.port }}
      targetPort: http
  selector:
    app.kubernetes.io/name: grafana
    app.kubernetes.io/instance: {{ .Release.Name }}

Optional Ingress

# templates/grafana-ingress.yaml
{{- if .Values.grafana.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ include "kube-prom-stack.fullname" . }}-grafana
  namespace: {{ .Values.global.namespace | default .Release.Namespace }}
  annotations:
    {{- if .Values.grafana.ingress.className }}
    kubernetes.io/ingress.class: {{ .Values.grafana.ingress.className | quote }}
    {{- end }}
spec:
  ingressClassName: {{ .Values.grafana.ingress.className | default nil }}
  rules:
    {{- range .Values.grafana.ingress.hosts }}
    - host: {{ . | quote }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ include "kube-prom-stack.fullname" $ }}-grafana
                port:
                  number: {{ $.Values.grafana.service.port }}
    {{- end }}
  tls:
    {{- toYaml .Values.grafana.ingress.tls | nindent 4 }}
{{- end }}

Step 7 — Alertmanager, kube-state-metrics, node-exporter

These follow the same pattern: Deployment/DaemonSet + Service + (optional) PVC/ConfigMaps. For node-exporter, you’ll likely have a DaemonSet with host mounts and privileged mode.

# templates/node-exporter-ds.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: {{ include "kube-prom-stack.fullname" . }}-node-exporter
  namespace: {{ .Values.global.namespace | default .Release.Namespace }}
  labels:
    {{- include "kube-prom-stack.labels" . | nindent 4 }}
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: node-exporter
      app.kubernetes.io/instance: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app.kubernetes.io/name: node-exporter
        app.kubernetes.io/instance: {{ .Release.Name }}
    spec:
      hostPID: true
      hostNetwork: true
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.8.2
        args: ["--path.rootfs=/host"]
        volumeMounts:
          - name: host
            mountPath: /host
            readOnly: true
      volumes:
        - name: host
          hostPath:
            path: /
            type: Directory

Validate security context and host mounts against your environment and security policies.

Step 8 — CRD-backed resources (ServiceMonitor, PodMonitor, PrometheusRule)

If MicroK8s deployed the Prometheus Operator CRDs, you’ll see ServiceMonitor/PodMonitor/PrometheusRule objects. Keep them templated and togglable.

# templates/servicemonitors.yaml
{{- if .Values.serviceMonitors.enabled }}
{{- range $i, $sm := .Values.serviceMonitors.items | default list }}
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: {{ include "kube-prom-stack.fullname" $ }}-{{ $sm.name }}
  namespace: {{ $.Values.global.namespace | default $.Release.Namespace }}
spec:
  {{- toYaml $sm.spec | nindent 2 }}
{{- end }}
{{- end }}

Then in values.yaml you can define serviceMonitors.items as raw snippets you copy from your dump.

Same for rules:

# templates/prometheusrules.yaml
{{- if .Values.prometheusRules.enabled }}
{{- range $i, $rule := .Values.prometheusRules.items | default list }}
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: {{ include "kube-prom-stack.fullname" $ }}-{{ $rule.name }}
  namespace: {{ $.Values.global.namespace | default $.Release.Namespace }}
spec:
  {{- toYaml $rule.spec | nindent 2 }}
{{- end }}
{{- end }}

This approach avoids hardcoding dozens of CRD resources in templates; you paste them into values.yaml and keep control with feature flags.

Step 9 — Install safely (two migration options)

Option A — New namespace (safest)

# create a fresh namespace
microk8s kubectl create ns monitoring

# install your chart there
helm install kube-prom --namespace monitoring ./kube-prom-stack

# verify
microk8s kubectl get pods -n monitoring

Option B — Replace the addon in-place

Scale down or disable the addon first to avoid name collisions:

microk8s disable observability
# or delete only the labeled resources
microk8s kubectl delete all,cm,secret,sa,role,rolebinding,clusterrole,clusterrolebinding,pvc      -A -l app.kubernetes.io/instance=kube-prom-stack

Install your Helm chart using the same names (via fullnameOverride if needed) so dashboards and PVCs align.

Helm cannot “adopt” existing resources; either install into a clean namespace or delete the old ones first.

Step 10 — Test & validate

# Port-forward Prometheus
microk8s kubectl -n monitoring port-forward svc/kube-prom-kube-prom-stack-prometheus 9090:9090

# Port-forward Grafana
microk8s kubectl -n monitoring port-forward svc/kube-prom-kube-prom-stack-grafana 3000:3000

Check targets, alert rules, and dashboards. Confirm PVCs are bound against your storageClass (MicroK8s typically uses microk8s-hostpath).

Step 11 — Package and publish the chart

# from the chart root
helm lint
helm package .

# push to an OCI registry (example: Azure Container Registry)
export HELM_EXPERIMENTAL_OCI=1
helm push ./kube-prom-stack-0.1.0.tgz oci://myacr.azurecr.io/helm

For GitOps, push the chart and a values-prod.yaml to your repo; let Argo CD or Flux manage the release lifecycle.

Troubleshooting

Pods won’t start / CrashLoopBackOff — Check mismatched selectors (Service selector must match pod template labels). Verify volumes and security contexts.
No dashboards/targets — If the addon used CRD-backed resources, make sure you imported ServiceMonitor/PodMonitor/PrometheusRule definitions into values.yaml or templates.
PVC Pending — StorageClass name typo or missing MicroK8s hostpath storage: microk8s enable storage.
403 listing targets — Grafana/Prometheus RBAC scoped too narrowly; enable rbac.create or add specific ClusterRoles.
Name collisions — You didn’t uninstall the addon before installing the chart with same names. Use a clean namespace or delete originals.

Appendix A — Quick splitter with `yq` only

#!/usr/bin/env bash
set -euo pipefail
SRC="kube-prom-stack.dump.yaml"
OUT="export-split"
mkdir -p "$OUT"
count=$(yq eval 'select(documentIndex >= 0) | length' "$SRC" >/dev/null 2>&1 || true)
awk 'BEGIN{n=0}/^---$/{n++}{print > sprintf("%s/chunk-%04d.yaml","'"$OUT"'",n)}' "$SRC"
for f in "$OUT"/chunk-*.yaml; do
  kind=$(yq -r '.kind // empty' "$f"); [ -z "$kind" ] && { rm -f "$f"; continue; }
  name=$(yq -r '.metadata.name // "noname"' "$f")
  ns=$(yq -r '.metadata.namespace // "default"' "$f")
  mv "$f" "$OUT/${ns}_${kind}_${name}.yaml"
done

Appendix B — Example tree after templating

kube-prom-stack/
├── Chart.yaml
├── values.yaml
└── templates/
    ├── _helpers.tpl
    ├── alertmanager-deploy.yaml
    ├── grafana-deploy.yaml
    ├── grafana-ingress.yaml
    ├── grafana-secret.yaml
    ├── grafana-svc.yaml
    ├── node-exporter-ds.yaml
    ├── prometheus-deploy.yaml
    ├── prometheus-pvc.yaml
    ├── prometheus-svc.yaml
    ├── prometheusrules.yaml
    └── servicemonitors.yaml

Final thoughts

The MicroK8s addon is fantastic for quick starts, but moving the stack into Helm gives you repeatability and control. The recipe above keeps your exported resources intact while layering on the Helm features you actually need: sane naming, configurable values, and optional CRD resources controlled in one values.yaml.

From Docker Desktop to Podman Desktop — and Why Quadlets Led Me to Kubernetes

Tue, 15 Apr 2025 00:00:00 GMT

Like many developers, I started with Docker Desktop because it “just worked.” But over time, the licensing changes, resource usage, and the need for better security led me to explore Podman Desktop.

This article is both a migration guide and a story of discovery: how I replaced Docker Desktop with Podman, experimented with Quadlets for container management, and ultimately landed on Kubernetes (MicroK8s) for orchestration.

Part 1 — Migrating from Docker Desktop to Podman Desktop

Why leave Docker Desktop?

Licensing & cost: Docker Desktop moved to a paid licensing model for enterprises.
Resource usage: On Windows/macOS, Docker runs a hidden VM; Podman Desktop is lighter.
Security: Podman supports rootless containers, running as your own user.
OCI-first: Podman works with OCI registries like ACR out of the box.

For my daily dev flow, Podman Desktop became a drop-in replacement.

Installing Podman Desktop

On macOS or Windows:

brew install podman-desktop
# or download from https://podman-desktop.io/

Initialize the Podman VM (“machine”):

podman machine init
podman machine start

Check status:

podman info

Building and Running Containers

All your familiar Docker commands work, but with podman:

podman build -t myapp:dev .
podman run -p 8080:80 myapp:dev

Pushing to Azure Container Registry (ACR):

podman login myazurecontainergistry.azurecr.io
podman tag myapp:dev myazurecontainergistry.azurecr.io/myapp:dev
podman push myazurecontainergistry.azurecr.io/myapp:dev

Podman even includes a Docker socket compatibility layer, so tools expecting docker.sock still work.

Podman Compose

For local dev stacks:

podman-compose up

This reads docker-compose.yml the same way as Docker. Perfect for multi-service Angular/.NET stacks.

Part 2 — Discovering Podman Quadlets

At some point, I wanted containers to auto-start at boot and be managed like system services. That’s where Podman Quadlets come in.

Quadlets let you describe containers in .container files that systemd understands.

Example: myapp.container

[Unit]
Description=My App Container

[Container]
Image=myazurecontainergistry.azurecr.io/myapp:latest
Network=host
Volume=/data:/app/data
AutoUpdate=registry

[Install]
WantedBy=default.target

Deploy it:

sudo cp myapp.container /etc/containers/systemd/
systemctl --user daemon-reload
systemctl --user enable --now myapp.service

Now the container runs at boot, restarts on failure, and updates when you push new tags.

Pros of Quadlets:

Great for single-node setups (edge servers, dev machines).
Integrates tightly with systemd.
Simple config, no extra orchestrator needed.

Cons:

No scheduling or scaling beyond one node.
Updating multiple apps at once gets messy.
Secrets/config management is manual.

Quadlets gave me a taste of container-as-service, but also highlighted the limits.

Part 3 — Why Quadlets Pushed Me to Kubernetes (MicroK8s)

Running one or two containers with Quadlets is fine. But as soon as I had:

Multiple APIs (an-application API, Identity, etc.)
Redis/RabbitMQ backing services
Ingress/TLS requirements
Persistent Volumes

…I realized I was re-implementing Kubernetes-lite.

Enter MicroK8s — Canonical’s lightweight Kubernetes.

Installing MicroK8s

sudo snap install microk8s --classic
microk8s status --wait-ready

Enable common addons:

microk8s enable dns storage ingress observability

Deploying with Helm

Instead of writing individual manifests by hand, I packaged my services with Helm charts.

Example deployment (Helm values-driven):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: an-application-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: an-application-api
  template:
    metadata:
      labels:
        app: an-application-api
    spec:
      containers:
      - name: an-application-api
        image: myazurecontainergistry.azurecr.io/an-application-api:1.0.0
        ports:
        - containerPort: 5000
        volumeMounts:
        - mountPath: /config
          name: config
      volumes:
      - name: config
        configMap:
          name: an-application-api-config

Now I had scaling, rolling updates, TLS ingress, and persistent volumes handled by Kubernetes instead of by hand.

Part 4 — Comparing the Approaches

Feature	Docker Desktop	Podman Desktop	Podman Quadlets	MicroK8s (Kubernetes)
Rootless security	❌	✅	✅	✅
Resource usage	Heavy	Light	Light	Moderate
Multi-service dev stack	✅ (Compose)	✅ (Compose)	⚠️ Manual	✅ (Helm/Manifests)
Auto-start at boot	❌	❌	✅	✅ (Deployments)
Scaling/HA	❌	❌	❌	✅
Secrets/Config mgmt	Basic env vars	Basic env vars	Manual	✅ (ConfigMaps/Secrets)
Multi-node cluster	❌	❌	❌	✅

Final Thoughts

Podman Desktop is a great daily driver and replacement for Docker Desktop.
Quadlets are perfect for edge devices or simple single-node setups.
But as soon as you need scaling, persistence, or multiple microservices, Kubernetes is the natural next step.

For me, the progression was:

Docker Desktop → 2. Podman Desktop → 3. Podman Quadlets → 4. MicroK8s with Helm.

Each step taught me something new about containers, orchestration, and the trade-offs between simplicity and power.

Written April 2025 — a reflection on my container journey and why Kubernetes ultimately won out, but not without Podman teaching me valuable lessons along the way.

Converting from npm to Bun

Mon, 13 Jan 2025 00:00:00 GMT

Complete guide to converting an npm-based project to Bun, including Docker, CI/CD, and deployment configurations.

Why Bun?

10-100x faster package installation than npm
Drop-in replacement for Node.js - no code changes needed
Built-in bundler and test runner
Smaller Docker images when using Bun base image

Prerequisites

Existing project using npm
Admin/sudo access (for Bun installation)

Step 1: Install Bun

Windows (PowerShell)

irm bun.sh/install.ps1 | iex

macOS/Linux

curl -fsSL https://bun.sh/install | bash

Verify installation:

bun --version

Step 2: Install Dependencies with Bun

In your project directory:

# Remove npm lock file (optional but recommended)
rm package-lock.json

# Install dependencies with Bun
bun install

This creates a bun.lock file (Bun's lockfile) and installs all packages from package.json.

Note: package.json stays the same - no changes needed!

Step 3: Test Your Build

Verify everything works:

bun run build
bun run dev
bun run test

All npm scripts work as-is with bun run.

Step 4: Update Dockerfile

If you use Docker, update your Dockerfile to use Bun:

Before (npm):

FROM node:20-slim AS builder
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build

After (Bun):

FROM oven/bun:1 AS builder
WORKDIR /app
COPY package.json bun.lock ./
RUN bun install --frozen-lockfile
COPY . .
RUN bun run build

Key changes:

Base image: oven/bun:1
Copy bun.lock instead of package-lock.json
Use bun install --frozen-lockfile (like npm ci)

Step 5: Update CI/CD Pipeline

GitHub Actions

Before:

- name: Set up Node.js
  uses: actions/setup-node@v4
  with:
    node-version: '20'

- name: Install dependencies
  run: npm install

- name: Build
  run: npm run build

After:

- name: Set up Bun
  uses: oven-sh/setup-bun@v2
  with:
    bun-version: latest

- name: Install dependencies
  run: bun install --frozen-lockfile

- name: Build
  run: bun run build

Cloudflare Workers/Pages

No changes needed! Wrangler works with Bun out of the box.

Just update your npm scripts:

{
  "scripts": {
    "deploy": "bun run build && wrangler pages deploy public"
  }
}

Step 6: Update .gitignore

Add Bun-specific ignores:

# Dependencies
node_modules/

# Lock files (choose one approach)
# Option 1: Keep both for compatibility
package-lock.json
yarn.lock

# Option 2: Bun only (remove npm lock from git)
# package-lock.json committed previously will be ignored

Note: bun.lock should be committed to git (just like package-lock.json).

Common Commands

npm	Bun	Description
`npm install`	`bun install`	Install dependencies
`npm install <pkg>`	`bun add <pkg>`	Add package
`npm install -D <pkg>`	`bun add -d <pkg>`	Add dev dependency
`npm uninstall <pkg>`	`bun remove <pkg>`	Remove package
`npm run <script>`	`bun run <script>`	Run package.json script
`npx <command>`	`bunx <command>`	Execute package binary
`npm ci`	`bun install --frozen-lockfile`	Clean install

Verification Checklist

After conversion, verify:

bun install completes successfully
bun run build works
bun run dev starts dev server
All tests pass with bun run test
Docker build succeeds (if applicable)
CI/CD pipeline passes
Production deployment works

Troubleshooting

Peer Dependency Warnings

Bun may show peer dependency warnings. These are usually safe to ignore:

warn: incorrect peer dependency "react@18.3.1"

Native Modules

Most native modules work. If you encounter issues:

bun install --force

Postinstall Scripts

Some packages have postinstall scripts. Check with:

bun pm untrusted

Run blocked postinstall scripts:

bun pm trust <package-name>

Performance Comparison

Real-world example (this blog):

Operation	npm	Bun	Speedup
Fresh install	~60s	~5s	12x faster
Install with cache	~15s	~2s	7.5x faster
Gatsby build	~210s	~207s	Similar*
Dev server start	~15s	~12s	~20% faster

*Gatsby uses its own build system (Webpack), so build time is similar. Projects using Bun's native bundler see much bigger speedups (2-10x faster).

Rollback Plan

If you need to rollback:

Restore package-lock.json from git
Delete bun.lock
Run npm install
Revert Dockerfile and CI/CD changes

Additional Resources

Notes

Bun is API-compatible with Node.js - no code changes needed
Works with all major frameworks (Next.js, Gatsby, Vite, etc.)
Can run .js, .ts, .jsx, .tsx files directly
Not all npm packages are tested with Bun - check compatibility for critical dependencies

AKS MCP Server Setup

Fri, 10 Jan 2025 00:00:00 GMT

Step-by-step setup for the AKS MCP server with Claude Code in VS Code.

Prerequisites

VS Code with Claude Code extension
Azure CLI logged in (az login)
Kubernetes extension for VS Code (installs the AKS MCP binary)

1. Install the Kubernetes Extension

Install the Kubernetes extension from the VS Code marketplace. This bundles the AKS MCP server binary.

The binary gets installed to:

C:\Users\<user>\.vs-kubernetes\tools\aks-mcp\v0.0.11\aks-mcp.exe

2. Create `.mcp.json` in Your Repo

Create a .mcp.json file in your project root:

{
  "mcpServers": {
    "aks-mcp": {
      "command": "C:\\Users\\<user>\\.vs-kubernetes\\tools\\aks-mcp\\v0.0.11\\aks-mcp.exe",
      "args": ["--transport", "stdio", "--access-level", "readwrite"],
      "env": {}
    }
  }
}

Update the path to match your username.

3. Verify Connection

Restart Claude Code. You should see the MCP server connected in the status.

Test with:

"list my aks clusters"
"show cluster status"
"get pods in default namespace"

What You Can Do

The AKS MCP gives Claude access to:

Category	Operations
Clusters	list, show, start, stop, scale, upgrade
Node pools	list, add, delete, scale
kubectl	get, describe, logs, exec, apply
Monitoring	metrics, events, diagnostics
Networking	vnet, nsg, load balancer info

Access Levels

readonly - Safe for production, only read operations
readwrite - Full access including mutations

Notes

MCP servers run locally - no cloud-hosted option yet
The binary is bundled with the K8s extension, not a separate install
Works with any kubeconfig context you have configured

VS Code Agent Skills Setup

Fri, 10 Jan 2025 00:00:00 GMT

Agent Skills is an open standard from Anthropic for defining reusable instructions that AI coding assistants can load automatically.

The Standard

Skills live in .github/skills/<skill-name>/SKILL.md with YAML frontmatter:

---
name: my-skill
description: When to use this skill
license: MIT
metadata:
  author: username
  version: "1.0"
---

# Skill Title

Instructions, rules, and context for the AI...

Skills Created

1. aks-cluster-management

Location: .github/skills/aks-cluster-management/SKILL.md

Purpose: Start, stop, and manage AKS clusters for cost optimization.

Contains:

Cluster inventory (names, resource groups, VM sizes)
Start/stop commands for all clusters
Crossplane webhook cleanup before stopping hub
Cost information

2. arm-portal-governance

Location: .github/skills/arm-portal-governance/SKILL.md

Purpose: Rules and constraints for GitOps infrastructure changes.

Contains:

Critical rules (never/always do)
Architecture overview
Provisioning flow diagram
Environment approval requirements
Repository structure

Creating New Skills

Create .github/skills/<name>/SKILL.md with frontmatter
Add the skill content
Copy the content into .claude/CLAUDE.md for Claude Code compatibility

References

agentskills.io - The spec
Anthropic announcement - Agent Skills in VS Code

Configuring CI/CD with Azure Pipelines and Nx Monorepos

Tue, 14 Dec 2021 00:00:00 GMT

When managing 15+ Angular applications in one product, the overhead of separate package.json files, builds, and pipelines quickly becomes painful. To address this, we turned to Nx, which provides tooling for managing monorepos with shared node_modules, optimized dependency graphs, and powerful affected commands.

This post details how I set up CI/CD pipelines in Azure DevOps for our Nx monorepo — including how to detect changed apps, run only the relevant builds/tests, and trigger releases per micro-application.

Why Nx?

Nx lets you manage a large Angular monorepo with:

A single package.json for dependencies.
Dependency-aware commands like nx affected:build, which only build/test/lint the projects impacted by a given change.
Consistency across builds: every app is built with the same toolchain and caching strategy.

The challenge was integrating this into Azure Pipelines for both CI (build/test) and CD (deployments).

Two Pipeline YAMLs: CI and Reusable Template

We ended up with two main YAML files:

CI wrapper pipeline
Detects what apps were changed in a commit and sets pipeline variables.
Reusable template
Executes npm install, nx lint, nx build, and nx test for each app in a consistent, parameterized way.

Step 1 — Detect Changed Applications

The first job, Get_Affected_App, scans the changeset paths and sets pipeline variables accordingly.

foreach ($path in $changesFolder) {
  if ($path -match '/apps/app1-dashboard') {
    echo "##vso[task.setvariable variable=App1_Dashboard;isOutput=true]$True"
    echo "##vso[task.setvariable variable=NPM_Install;isOutput=true]$True"
    break
  }
}

Explanation:

If any file under apps/app1-dashboard is modified, the variable App1_Dashboard=true is set.

The variable NPM_Install=true ensures npm install runs for the entire repo.

This is repeated for each app in the monorepo (app2-dashboard, app3-dashboard, etc.).

This mechanism ensures that we only queue CI jobs for the applications that actually changed.

Step 2 — Run NPM Install Once

Because Nx shares a single node_modules, we only need to run npm install once per pipeline run, not per app.

- job: NPM_Install_For_All
  pool:
    vmImage: 'ubuntu-latest'
  condition: eq(dependencies.Get_Affected_App.outputs['NX_PROJECT_VARIABLE.NPM_Install'], 'true')
  steps:
    - checkout: self
    - task: NodeTool@0
      inputs:
        versionSpec: '18.x'
    - script: npm ci
      displayName: 'Install Node modules'

Tip: Use npm ci instead of npm install for reproducible builds, since it respects package-lock.json.

Step 3 — Build & Test Affected Applications

Now we spin up one job per app, gated by the variables set earlier.

- job: App1_Dashboard_CI
  pool:
    name: Agent-Pool
    demands: Agent.OS -equals Windows_NT
  dependsOn: NPM_Install_For_All
  condition: eq(dependencies.Get_Affected_App.outputs['NX_PROJECT_VARIABLE.App1_Dashboard'], 'true')
  steps:
    - checkout: self
      persistCredentials: true
    - template: setup-and-build.yml
      parameters:
        app_name: 'app1-dashboard'

This job:

Runs only if App1_Dashboard=true.
Calls the reusable template with the app_name parameter.

Step 4 — Reusable Build Template

setup-and-build.yml is where the Nx magic lives:

parameters:
  app_name: ""

steps:
  - script: |
      echo "Linting ${{ parameters.app_name }}"
      npx nx lint ${{ parameters.app_name }}
    displayName: "Lint"

  - script: |
      echo "Building ${{ parameters.app_name }}"
      npx nx build ${{ parameters.app_name }} --configuration=production
    displayName: "Build"

  - script: |
      echo "Testing ${{ parameters.app_name }}"
      npx nx test ${{ parameters.app_name }} --ci --code-coverage
    displayName: "Test"

Explanation:

Each app is linted, built, and tested consistently.

Using a template reduces duplication across 15+ apps.

Adding new apps is as simple as copying a job block in the CI YAML and pointing it at this template.

Step 5 — Continuous Delivery (CD)

After a successful CI build, the last few steps handle deployment prep:

Copy build artifacts into a staging folder.
Tag the build in Azure DevOps with the app name (not a git tag).
Trigger the release pipeline for that app.

Example tagging step:

- task: PowerShell@2
  displayName: "Tag Build"
  inputs:
    targetType: 'inline'
    script: |
      Write-Host "##vso[build.addbuildtag]${{ parameters.app_name }}"

The release pipeline listens for builds tagged with app1-dashboard, app2-dashboard, etc., and deploys only that micro-application.

Step 6 — Why This Works for Nx Monorepos

Scalable: Adding new apps is trivial — copy a job, change the app_name.
Efficient: Only changed apps are built/tested, saving agent time and compute cost.
Consistent: Every build runs the same lint/build/test flow via the shared template.
Releasable: Build tags cleanly map to release triggers per micro-app.

Lessons Learned

Nx’s affected commands are powerful, but Azure DevOps variable passing required a little brute force scripting.
CI/CD pipelines become far easier to maintain once you standardize on templates.
Using build tags instead of git tags gave us precise control over release triggers.
There’s still room to improve: caching node_modules, leveraging nx cloud, and consolidating release templates.

Conclusion

Migrating to Nx with Azure Pipelines gave us a repeatable, optimized, and modular CI/CD pipeline for a monorepo with over 15 Angular applications.

While this setup may look verbose at first, it has paid off in speed, consistency, and maintainability.
Future improvements will likely include Nx Cloud caching and deployment previews per pull request.

Originally written Dec 14, 2021 — updated with more detail and commentary.

When to Use Angular's forRoot() Method — A Deep Dive

Mon, 06 Nov 2017 00:00:00 GMT

The forRoot() pattern in Angular is one of those things you see in documentation for years before realizing its real power.
In my case, it solved a subtle but frustrating problem: ensuring singleton services work correctly across both eager-loaded and lazy-loaded modules.

The Problem

Angular’s dependency injection system creates new instances of providers at the module level.
If you provide a service inside a shared module, then lazy-loaded modules that import that shared module will each get their own instance of the service.

This breaks the expectation that a service should be a singleton across the entire app.

For example, imagine a simple counter service:

import { Injectable } from '@angular/core';

@Injectable()
export class CounterService {
  private count = 0;

  increment() { this.count++; }
  get value() { return this.count; }
}

Now consider a shared module that provides this service:

import { NgModule } from '@angular/core';
import { CounterService } from './counter.service';

@NgModule({
  providers: [CounterService]
})
export class SharedModule {}

If you import SharedModule directly into both eager and lazy modules, each lazy module gets a new CounterService.
Your counter resets per module — not what you want.

Demo of the Problem

Example without forRoot:
StackBlitz Demo

Eager components share the counter.
Lazy-loaded components do not — they get isolated instances.

The forRoot() Solution

The forRoot() convention is Angular’s way of saying:

This module is providing global, singleton services.

Instead of providing services directly, the module exposes a static method forRoot() that returns a ModuleWithProviders.

shared.module.ts

import { NgModule, ModuleWithProviders } from '@angular/core';
import { CounterService } from './counter.service';

@NgModule({})
export class SharedModule {
  static forRoot(): ModuleWithProviders<SharedModule> {
    return {
      ngModule: SharedModule,
      providers: [CounterService]
    };
  }
}

app.module.ts

import { NgModule } from '@angular/core';
import { BrowserModule } from '@angular/platform-browser';
import { SharedModule } from './shared/shared.module';
import { AppComponent } from './app.component';
import { EagerComponent } from './eager.component';
import { routing } from './app.routing';

@NgModule({
  imports: [
    BrowserModule,
    SharedModule.forRoot(), // ✅ registers singletons once
    routing
  ],
  declarations: [AppComponent, EagerComponent],
  bootstrap: [AppComponent]
})
export class AppModule {}

👉 Example with forRoot:
StackBlitz Demo

Now both eager and lazy-loaded modules share the same CounterService instance.

Best Practices

Use forRoot() when your module provides services that should be singletons across the entire app.
Use a separate SharedModule (without providers) for declarations like directives, pipes, and components.
Don’t mix global services and UI declarations in the same module.
If you also need per-feature service instances, consider adding a forChild() method (used in Angular Router and NgRx).

Why Does This Work?

Angular only calls forRoot() once — in the root module.
Lazy-loaded modules import SharedModule without calling forRoot().
This ensures the service is provided once at the root injector, not per module.

Angular official guide: NgModule FAQ
AngularFirst: The NgModule forRoot convention
NgRx docs (they use forRoot() and forFeature() extensively).

Conclusion

The forRoot() convention is simple but powerful.
It prevents the frustrating “multiple instance” bug that occurs with lazy-loaded modules, and ensures that services like authentication, configuration, or in this case a counter service, remain singletons across your entire app.

First published in 2017. Still one of my most popular posts. Updated with richer explanations, StackBlitz demos, and best practices.