The building blocks​

• Ingress controller: Traefik, installed as the cluster's ingress, routing by hostname to Services.
• LoadBalancer implementation: Traefik is exposed through a Service of type LoadBalancer. With no cloud provider, something in the cluster has to implement that — either a LoadBalancer controller or a shared virtual IP.
• API VIP: kube-vip provides a virtual IP for the Kubernetes API server so the control plane has a stable address independent of which node is master.
• External entry: a tunnel daemon runs as a container and forwards public traffic to the ingress origin on the local network. Internal entry uses local DNS.

What "LoadBalancer" actually means here​

On a cloud provider, a LoadBalancer Service provisions a real balancer. On bare metal it is whatever the implementation provides. In this cluster, Traefik's LoadBalancer status lists all node addresses — the controller lets any node accept and forward traffic for the service. That is convenient and it is also easy to misread: the external address field is not a single stable VIP you can point DNS at.

The API VIP is separate and intentionally so. If the application VIP and the API VIP shared an address, a routing problem in one would take out the other. An early attempt to give applications a managed VIP failed and was reverted (see the post-mortem on the ARP fallout). The design keeps them apart while the application path is reworked.

The two entry paths​

Internet → tunnel daemon → ingress origin → Traefik → Service → Pod
LAN      → local DNS     → node address   → Traefik → Service → Pod

The external path has one manual seam: the tunnel's origin address is configured in the tunnel dashboard, not in Git. The internal path has a different problem.

The DNS single point of failure​

Internal hostnames resolved through the local DNS server, and the wildcard record pointed at one node's address. Every internal request that entered the cluster went through that node first.

This is important to name correctly: the bottleneck is not Traefik. Traefik can be reached on any node. The single point of failure is the DNS record. If that node is down, internal clients cannot reach the cluster even though the ingress path itself is healthy.

The intended topology is:

local DNS → application VIP → Traefik → any node

Once an application VIP exists, the DNS record stops being a SPOF; the VIP takes over that role, and the VIP is a single address by design — which means it needs its own answer for what happens when its holder fails.

Layers and what manages them​

The pattern worth noticing: the pieces closest to the network edge are the least declarative. The tunnel origin and the DNS records are infrastructure state living outside Git, and they are exactly the pieces that are hardest to reconstruct after an outage.

Takeaways​

• A LoadBalancer Service on bare metal is an implementation, not a cloud guarantee. Understand what backs it.
• Keep the API VIP and any application VIP separate.
• A DNS record pointing at a single node is a single point of failure even when the ingress controller is healthy.
• Inventory the manual seams — tunnel configuration, DNS records — before an incident forces you to.

The symptom​

The cluster has three control-plane nodes running embedded etcd for high availability. Over time the logs filled with etcd peer timeouts, the API server slowed down, and leader elections started flapping. Under load the whole control plane became briefly unavailable.

The first reaction was tuning. Raising the etcd heartbeat interval and election timeout (heartbeat-interval=500, election-timeout=5000) made the errors less frequent, which was enough to call the problem solved for a while. It was not a fix: it only made etcd more forgiving of a network that should never have been carrying peer traffic in the first place.

etcd is unusually sensitive to latency and jitter between members. Wireless is exactly that: retries, roaming, and interference that show up as milliseconds of variance at the worst possible moment.

The fix: wire the control planes​

All three control-plane nodes were moved to wired Ethernet (eth0). The worker nodes did not need to change — etcd peers are the strict requirement, not every workload.

The wireless interfaces were kept as backups with a higher route metric, so a cable failure does not take a node off the network.

Migrating one etcd member​

The migration is a per-node procedure, and the ordering matters:

1. Confirm the node's new wired address.

2. Install etcdctl on every control-plane node. K3s does not bundle it, and it has to be reinstalled after a node is reprovisioned.

3. From a healthy member, list the cluster and update the peer URL:

   etcdctl member list
   etcdctl member update <member-id> \
     --peer-urls=https://<new-address>:2380
   

   The certificates live under /var/lib/rancher/k3s/server/tls/etcd/ (server-ca.crt, client.crt, client.key).

4. Update /etc/rancher/k3s/config.yaml on that node with the new node-ip and advertise-address.

5. Restart K3s on the node.

6. Verify: the node is Ready, etcdctl member list shows the new peer URL, and the node annotations point at the wired address.

The caveat that almost broke quorum​

If the node being migrated is the only healthy member, it cannot rejoin on a new peer URL that the other members do not know about yet. The safe sequence is to boot it on the old address first, let quorum recover, then update the peer URL and restart.

Losing quorum mid-migration is the failure mode to design around: on a three-member cluster, one member can be offline at a time, and no more.

kube-vip​

The virtual IP for the API server is managed by kube-vip, which had been configured against the wireless interface. That was switched to eth0 as part of the migration. The gratuitous-ARP workarounds that had accumulated while the control planes were on Wi-Fi — needed because wireless clients can miss ARP updates — are no longer required.

What is not managed declaratively​

Node-level changes — /etc/rancher/k3s/config.yaml, the etcdctl binary, etcd member peer URLs — are outside GitOps. The cluster's workloads are reconciled from Git, but the datastore identity of each control-plane node is manual state. After this migration those details are worth recording somewhere, because the next person to touch the cluster will need them.

Takeaways​

• etcd peer traffic belongs on a wired link. Tuning timeouts buys time, not stability.
• Install etcdctl on every control-plane node before you need it.
• On a three-node control plane, migrate one member at a time and never lose two.
• Keep the wireless interface as a backup with a higher metric instead of deleting it.

A skill is a workflow, not a README​

The first instinct is to hand the agent the documentation. That fails for a predictable reason: documentation describes the API, but integration is a sequence of decisions.

The skill we wrote encodes that sequence:

• Choose the path. Drop-in card, custom UI through context hooks, or an action-link flow.
• Choose the authentication mode. On-chain wallet signing, off-chain sign-in, or a special case.
• Confirm the inputs. Tenant id, card id, environment, wallet chain, keys, signing method. If something is missing, ask — do not guess.
• Integrate minimally. Fit the existing app structure; do not invent a new architecture around it.

It opens with a short decision tree, then goes straight to implementation. Every section exists to remove a decision the agent would otherwise make wrongly by default.

The failure modes are the content​

Half of the skill is a list of ways integrations go wrong: provider mounted in the wrong place, auth mode mismatched with the wallet connection, subscription values parsed without checking the shape they depend on. These are not API errors — they are judgment errors. A skill is valuable exactly to the extent that it prevents them.

You cannot eyeball this​

A skill that reads well can still make an agent worse. The only way to know is to run it against a real task and compare.

The evaluation setup has four parts:

• Evals — the test cases: a prompt, the expected behavior, and what a reviewer should look at.
• Fixtures — an immutable baseline codebase to run against.
• Workspaces — the artifacts of each run: final response, transcript, diff, timing.
• A runner — the orchestration that clones the fixture, creates a workspace, launches the agent, and saves everything for review.

Choices for the first iteration​

A real codebase, pinned. The fixture is a real downstream React application cloned at a fixed commit — not a toy example and not a vendored snapshot. The agent has to work inside an app with its own wallet setup, routing and conventions.

With and without. Every eval runs twice: once with the skill available, once without. The difference between the runs is the signal.

Human review first. Grading is done by a person against a rubric, not by assertions. The rubric scores seven things: path selection, auth mode, provider placement, required parameters, preservation of the existing app, code correctness, and minimality of the diff.

Assets in Git, artifacts out. The skill, eval definitions, rubric and fixture metadata are committed. Run workspaces are gitignored — they are evidence, not source.

What we learned​

• Run the eval before trusting the skill. The differences were subtle: the same integration wired two different ways, one of which would rot.
• A pinned real codebase finds problems a synthetic one cannot. It also forces the runner to be reproducible.
• Human grading is the right MVP. Automating a rubric before you know which dimensions matter freezes the wrong criteria.
• Minimality is a scored dimension. An agent that completes the task by rewriting half the app has failed.

The next steps are the obvious ones: more fixtures (an off-chain auth app), automated graders once the rubric stabilizes, and benchmark aggregation across iterations. But the order matters — skills, real fixtures, human review, and only then automation.

The experiment​

A three-node K3s control plane, all nodes on Wi-Fi at the time, already had a virtual IP for the API server managed by kube-vip. The next step was to give application ingress a virtual IP as well, using kube-vip's services mode. The manifest was committed as a DaemonSet that would announce per-service /32 addresses.

It did not work as intended, and the change was reverted.

The failure chain​

1. The revert commit was pushed, but Flux could not delete the DaemonSet from the cluster: etcd quorum was already broken, so the GitOps reconciliation loop had nothing to read state from and nothing to write state to.
2. Because the object still existed in etcd, the kubelet on the node recreated the pod — as kubelets do — even though the DaemonSet had been removed from Git.
3. The recreated pod announced its /32 service addresses on the wireless interface.
4. Wi-Fi is a shared medium. The access point's ARP table learned one MAC address for many IP addresses, a poisoned entry that then spread.
5. etcd peer traffic on port 2380 could no longer resolve its peers: ARP for the node addresses failed or returned the wrong destination.
6. Quorum collapsed, which kept Flux unable to remove the object, which kept the pod alive. A loop.

Diagnosis​

The clues were unusually confusing:

• The wrong primary address was showing on the wireless interface.
• An SSH session to one node would sometimes land on a different one.
• mDNS names still resolved, which made the network look healthy.
• The one reliable vantage point was a machine outside the affected segment; from there the pattern was obvious.

Recovery​

Order is everything here. The goal is to stop the thing that keeps making things worse before repairing the datastore.

1. Stop K3s on all nodes and remove the stray /32 addresses from the interfaces they were announced on.

2. On one control-plane node, run a single-node cluster reset (k3s server --cluster-reset) so the datastore can be opened again.

3. Start K3s on that node and delete the offending DaemonSet.

4. On each of the other control-plane nodes, wipe both:

   • /var/lib/rancher/k3s/server/db/etcd
   • /var/lib/rancher/k3s/server/tls/etcd

   Deleting only the datastore is not enough — the etcd certificates have to go too, or the peer handshake fails against the reset member. This detail cost the most time.

5. Restart K3s on each node and let them rejoin, then clean up the leftover static pods with crictl stop / crictl rm.

Prevention​

• Do not run kube-vip in services mode on a Wi-Fi network. Announcements that rely on gratuitous ARP only update the router; Wi-Fi peers can miss them.
• Use MetalLB or a static LoadBalancer address instead.
• Take an etcd snapshot before any kube-vip change.
• Understand that a GitOps revert is not an escape hatch: if the datastore is down, the revert cannot be applied. Recovery has to be manual first, declarative second.
• On a three-member etcd cluster, protect quorum above all else. One node can fail; the second failure is the outage.

Takeaways​

The experiment was reverted. The pod was not. The datastore held the state that kept the pod alive, and the pod was the reason the datastore could not recover. Breaking that loop meant going to the nodes and fixing etcd by hand, in the right order, before Git could take over again.

The goal​

The platform already had everything a notification product needs: tenant configuration, alert subscriptions, message publishing, and SDKs for web and server environments. What it did not have was an interface an agent could operate.

The target workflows were concrete:

• an agent detects a large on-chain transaction and notifies subscribed wallets
• a liquidation warning goes out before a position is at risk
• a community manager drafts and broadcasts an announcement
• a developer asks an AI IDE to publish a test notification

Non-agent workflows still had the raw GraphQL and REST APIs. The new layer was for autonomy.

Decisions before code​

Nine decisions shaped the implementation. The important ones:

MCP and a companion skill. MCP is the runtime integration: it gives the agent executable tools. A companion skill is the guidance layer: when to use which tool, how to reason about payloads, and when to ask the user instead of guessing. Tools alone are not enough for a domain with tenant-specific data shapes.

Local-first, stdio only. The server is distributed as an npm package the user runs themselves. Credentials live in environment variables on the user's machine and never leave it. No hosted infrastructure, no SSE transport, no OAuth flow to build.

Exactly three tools. publish_message, get_active_alerts and get_tenant_config. A small surface is a feature: agents select tools more reliably from a short list, and every tool is a commitment to maintain.

Raw payloads, no universal schema. The payload for a message is defined by tenant, topic and template, so the server passes it through as an object rather than inventing an abstraction that would be wrong half the time.

No package installation. The agent may only use the predefined tools or documented direct API calls. It must not install or execute arbitrary packages — a whitelist keeps the blast radius small.

The architecture​

AI agent
   │  stdio
   ▼
local MCP server  ──HTTPS──▶  platform GraphQL + REST APIs
   │
   └── reuses the existing server-side SDK

The server is a thin wrapper over the platform's Node SDK: it reuses the GraphQL and REST clients instead of reimplementing them. Configuration comes from environment variables, the client is initialized lazily on the first tool call, and token refresh is handled transparently.

What the tools look like​

get_tenant_config returns the tenant's configuration and its events with metadata — the information an agent needs to reason about everything else.

get_active_alerts returns the subscribers currently subscribed to an event, with cursor pagination normalized into a simple page object.

publish_message takes an event id, the raw payload object and an optional wallet target list. It maps the target list onto the API's wallet-specific send path and passes the payload through unchanged.

The part that is not code​

The hardest part is payload reasoning. A tenant's topic might require a field that exists nowhere in the metadata. There is no universal function from "event" to "valid payload" — the knowledge lives in the tenant's configuration and sometimes only in the head of the person asking.

That is what the companion skill is for. It teaches the agent to inspect the configuration first, to prefer the MCP path, to fall back to documented direct API calls when MCP is unavailable, and — most importantly — to ask the user when the required shape is ambiguous.

Validation in a real agent​

The server passed its unit-level checks, but the interesting validation was end to end, inside an actual agent runtime:

• the server boots, and a missing credential produces an actionable error instead of a crash
• the agent discovers the tools and the companion skill
• configuration lookup, alert pagination and a real broadcast publish all succeed
• given an ambiguous payload, the agent inspects the configuration before assuming a shape
• when MCP is unavailable, the agent can still explain the direct API path

That last set of scenarios is where the design is really tested. A tool server is easy to demo and hard to make dependable.

What I would keep​

• Small tool surfaces. Three well-named tools beat fifteen convenient ones.
• Pass-through payloads. Abstractions over data shapes you do not control become translation layers you cannot maintain.
• Local-first credentials. It removes an entire class of security review.
• A companion skill. The agent needs judgment about the domain, not just a list of functions.

The symptom​

Customers reported the same thing: install the new version of a wallet extension, open the dapp, click the wallet tile in the connect modal — and get redirected to the wallet vendor's homepage instead of connecting.

The modal was behaving exactly as written. It could not find the wallet, so it assumed the extension was missing and offered to install it.

The root cause​

The SDK had a dedicated integration for this wallet, built around a custom global that old versions injected into the page. Every detection path led back to that global:

const getWalletFromWindow = async () => {
  if (typeof window === 'undefined' || !window.WalletGlobal) {
    throw new Error("wallet is not installed");
  }
  // ...
};

The new wallet release moved to MPC-based key management and stopped injecting the legacy global entirely. By then the old extension was already on its way out: the vendor had stopped shipping updates for it, with store removal announced for the following month.

What the new wallet does instead is announce itself through EIP-6963, the discovery standard for injected wallets: every provider dispatches an announcement event carrying metadata, including an rdns identifier and a display name. Wallets that never touch a custom global are still discoverable — if you listen for announcements.

Wallet detection was vendor-shaped. The standard existed precisely to avoid that.

The fix​

The SDK already had a generic injected-wallet path that listens for EIP-6963 announcements and matches a wallet by substring on rdns or name:

providers.find(
  (p) =>
    p.info?.rdns?.toLowerCase().includes(walletName.toLowerCase()) ||
    p.info?.name?.toLowerCase().includes(walletName.toLowerCase()),
);

The fix was two lines of substance: move the wallet from the dedicated legacy hook to the generic injected hook, and update the install URL to point at the current wallet. The public interface did not change at all.

The cleanup that followed​

The incident exposed how much dedicated machinery existed for a single wallet:

• a legacy hook of a few hundred lines, no longer imported by anything
• a dedicated wallet class separate from the generic EVM wallet
• registry entries: a standalone category, and the wallet listed as a native integration instead of an injected one
• a special case in the wallet instance factory
• tests for all of the above

With the wallet flowing through standard discovery, none of it was necessary. The registry now treats it like any other injected EVM wallet: no special category, no special hook, no special factory branch.

While in there, the type layer was renamed to describe the chain family instead of a wallet implementation — EVM keys, Cosmos keys, Solana keys, Cardano keys. The rename has no runtime impact, but it stops the public types from baking vendor names into anything that imports them. Removing the dedicated class from the public package is a breaking change for the small set of consumers importing it directly, so it ships with the next major version and a short migration note.

What this taught us​

• Prefer standards-based discovery. EIP-6963 exists so integration code does not depend on whichever global a vendor injected this year.
• One integration per wallet does not scale. A registry entry plus a generic path covers wallets that follow the standard; dedicated code is for genuinely special cases.
• Fix the incident and the cleanup together. The "small fix" turned out to be the front door to removing hundreds of lines of abstraction. Left separate, the cleanup would probably never have happened.
• Name types after concepts, not vendors. Chain families are stable; wallet branding is not.

The kernel already provides it. Thermal zones are exposed under /sys:

cat /sys/class/thermal/thermal_zone0/temp
# 47200

The value is in millidegrees Celsius, and type reports cpu-thermal.

The options​

• command_line sensor — reads the file directly. Native, no add-on, no extra service. Chosen here.
• Glances add-on — much broader system stats, at the cost of a running service and more resource use.
• System Monitor — built in, but on Home Assistant OS it does not offer CPU temperature.

Keeping configuration.yaml readable​

The larger principle: configuration.yaml should only include, not define. The real definitions live in config/integrations/, split by domain:

# configuration.yaml
command_line: !include integrations/command_line.yaml

# config/integrations/command_line.yaml
command_line:
  - sensor:
      name: CPU Temperature
      unique_id: cpu_temperature_thermal_zone0
      command: "cat /sys/class/thermal/thermal_zone0/temp"
      unit_of_measurement: "°C"
      value_template: "{{ value | float / 1000 | round(1) }}"
      scan_interval: 30
      device_class: temperature
      state_class: measurement

device_class: temperature gives the entity proper units and graph support, and state_class: measurement makes statistics and long-term history work.

Result​

sensor.cpu_temperature reports values like 47.2 °C at idle on a Raspberry Pi 4, and 30 seconds between samples is more than enough for a slow-moving thermal value.

From there it can go on a dashboard, into a ventilation automation, or into a warning when the board approaches its throttling range.

The symptom​

Three proxies cover the house. One of them stopped discovering devices. From Home Assistant's Bluetooth diagnostics:

• scanning: false
• discovered devices: 0
• no advertisement seen for about 11.5 hours

The Wi-Fi connection was up, the ESPHome API was responsive, and the other two proxies were fine. Nothing in the normal device state suggested a problem — which is exactly why it went unnoticed for half a day.

What did not work​

Home Assistant-side remedies only act on the integration, not on the radio:

• reloading the integration
• disabling and re-enabling the proxy
• talking to it through aioesphomeapi
• an OTA update from the command line, which timed out

The scanner only came back after a compile-and-flash from the ESPHome Dashboard over its WebSocket OTA. That pointed at the device firmware as the layer that had to recover itself.

Why a watchdog on the device​

An automation in Home Assistant could notice a dead proxy and restart it, but it depends on Home Assistant, on the network path to the proxy, and on a polling interval. It has no direct view of the scanner state, and the health logic ends up spread across YAML files.

The firmware already knows the one thing that matters: when it last saw a BLE advertisement. That makes a watchdog trivial and local.

The watchdog​

Track the last advertisement time in a global:

globals:
  - id: last_ble_adv_time
    type: uint32_t
    restore_value: no
    initial_value: "0"

esp32_ble_tracker:
  on_ble_advertise:
    - lambda: |-
        id(last_ble_adv_time) = millis();

Then check it on an interval, with a grace period so a slow boot is not mistaken for a hang:

interval:
  - interval: 2min
    then:
      - lambda: |-
          // Do not reboot while the scanner is still coming up.
          if (millis() < 3 * 60 * 1000) {
            return;
          }
          const uint32_t last = id(last_ble_adv_time);
          if (last == 0 || (millis() - last) > 10 * 60 * 1000) {
            App.safe_reboot();
          }

Ten minutes without a single advertisement is far beyond any normal quiet period, so the reboot is safe. The first occurrence happened weeks before the watchdog existed; since then the proxy recovers on its own.

Two small helpers make the behavior visible:

button:
  - platform: restart
    name: Restart

sensor:
  - platform: uptime
    name: Uptime
    update_interval: 60s

The uptime sensor shows whether the watchdog is firing at all, and the restart button allows a manual kick without opening the dashboard.

Takeaway​

Fix device health on the device. A firmware watchdog is autonomous, local, it needs no network, it lives in one YAML file, and it reacts immediately. Home Assistant automations are for logic that spans devices, not for keeping a single device alive.

The same pattern applies to other silent firmware failures: Wi-Fi that drops while the API socket stays open, sensors that start returning NaN, or a stuck output that never changes state.

The starting point​

The first version of the SDK was a single React hooks package. One hook owned almost everything:

• API communication over REST, with hand-written request and response types
• multi-chain wallet signing adapters
• auth and token lifecycle
• loading and error state
• business logic

For a small startup this is a reasonable design. It maximizes time to market and minimizes integration friction: a partner installs one package, mounts a provider, and the hook does the rest.

Where it started to hurt​

Three problems appeared as the product grew.

Framework lock-in. All logic was bound to the React lifecycle. Anything that was not React — a Vue or Svelte integration, a script, a background worker — could not reuse the SDK at all.

Maintenance burden. Ten-plus supported chains turned the hook into a giant switch-case abstraction. Responsibilities blurred: API concerns, signing concerns and UI state lived in the same file.

Testability. Testing business logic required mounting React components. That made tests slow and brittle, and it quietly discouraged coverage.

The constraints​

The migration had to be done by one engineer, alongside normal feature delivery. Live customers were running the old architecture in production, so downtime was not an option. There was no hard deadline, but there was also no freeze: the new architecture had to coexist with the old one, and customers could adopt it at their own pace.

The options​

Split the hook into smaller hooks — useAuth, useSigning, useAlerts. Rejected: the logic stays React-bound, so the framework lock-in and testability problems remain. It only delays them.

A framework adapter over a monolithic core — invert the dependency but keep the core shaped by one framework's needs. Rejected: every supported framework becomes another maintenance surface.

A pure TypeScript client with thin framework wrappers. Chosen. The core owns the logic; React, Vue or anything else becomes a small adapter.

The extraction​

The core became a standalone TypeScript package: a client class owning the auth state machine, wallet signing adapters, the API layer and the token lifecycle. It has no React dependency and runs anywhere JavaScript runs.

Around the same time, the communication layer moved from REST to GraphQL. Hand-maintained types were replaced by code generation against the schema. That eliminated a class of schema drift bugs and reduced cross-team coordination cost — the types could no longer disagree with the API.

The migration itself ran in three phases:

1. Coexistence. The old hooks and the new client lived side by side inside the existing React package. Customers were unaffected.
2. New surface. A new React package was built entirely on the new core, replacing the old one.
3. Removal. Once the last customers had migrated, the legacy packages were removed in a major version.

The outcome​

The SDK became a platform core: one place for auth, signing and API logic, usable from any framework or no framework at all. The full migration took about three quarters with zero downtime. Adding a chain no longer means touching React code, which cuts regression risk.

What I would do differently​

• Decouple earlier. The pain was predictable; waiting for it to become acute cost more than starting the extraction would have.
• Treat a framework-agnostic core as a day-one principle, not a later refactor.
• Define a formal deprecation policy before the first breaking change.
• Run integration tests that exercise the old and new paths side by side during coexistence. Behavioural drift between them is the biggest risk in this kind of migration, and it stays invisible until a customer hits it.

Migrating a live SDK is mostly a communication problem wearing an architecture costume. Phased, independently shippable steps are what make it survivable — especially when there is only one engineer.

Home Assistant → mqtt.publish → Zigbee2MQTT → IR blaster → appliance

The interesting part is the protocol. IR has no feedback and no handshake, and air conditioner remotes do not send commands — they send full state.

Codes are states, not buttons​

A TV remote sends "volume up". An air conditioner remote sends the complete state every time: power, mode, target temperature and fan speed together. There is no "make it one degree warmer"; there is only "Cool, 26 °C, fan auto".

That means one learned code per state you want to use, and automation logic that picks a state instead of pressing buttons.

Learning codes​

In the Zigbee2MQTT device page, expose and enable Learn IR code, point the physical remote at the blaster, and press the button you want to capture. The code lands in a sensor like sensor.<device>_learned_ir_code as a long base64 string — often over a thousand characters.

A practical set for an air conditioner:

• Auto
• Cool 25
• Cool 26
• Off

Sending codes​

From Home Assistant the reliable path is mqtt.publish:

action: mqtt.publish
data:
  topic: zigbee2mqtt/<friendly_name>/set
  payload: '{"ir_code_to_send": "<BASE64>"}'

Three things that cost time before they are understood:

1. The topic needs the Zigbee2MQTT friendly name, which may be different from the name Home Assistant shows for the device.
2. base_topic is needed when publishing from Home Assistant or mosquitto_pub, but not in the Zigbee2MQTT Dev Console — the console is already inside that topic namespace. Adding the prefix there produces "Entity 'zigbee2mqtt' unknown".
3. Do not use text.set_value on the exposed IR-code text entity. It is limited to 255 characters while the codes are much longer, and the send fails with a generic "Unknown error".

Automation design​

Because every code is a complete state, keep the state set small and let the automation choose:

• room temperature above 28 °C → send Cool 26
• nobody home for 30 minutes → send Off

Store the long base64 strings in secrets.yaml and reference them, so the automations stay readable and the codes can be updated in one place.

Debug checklist​

• Entity unknown → the topic or the friendly name is wrong.
• "Entity 'zigbee2mqtt' unknown" → remove the base_topic prefix in the Dev Console.
• "Unknown error" → the code is too long for the text entity; use mqtt.publish instead.

IR stays a one-way protocol, so Home Assistant never knows whether the appliance received the command. Design for the states you can send, and keep the automation logic simple enough to reason about from the codes alone.

The feature lets dApp developers configure interactive on-chain actions through a backend UI and render them as components through the SDK. When a user clicks an action, the SDK asks the backend for a serialized transaction, hands it to the host application, and lets the host sign and submit it. The feature went from a spike in late March 2025 to production in May 2025.

The constraints​

• Time to market. The product team wanted a fast delivery path, which initially pushed the design toward a brand-new package.
• No breaking changes. Existing consumers of the client and React packages could not be affected.
• Blockchain-agnostic. The action-execution layer could not be tied to EVM, Solana, SUI or anything else; the SDK should return a transaction payload and let the host handle signing.
• A different auth model. The feature does not require a logged-in user session. It only needs authParams — a wallet public key and a blockchain type — to sign the activation request.

The options​

A new package. The original proposal was a separate package containing the UI components and a new client. On paper this avoids touching existing packages and gives clean separation.

The problems only show up when you draw the dependency graph:

• The UI needs types and the client from the existing frontend package, plus React context patterns — theme support, error views, CSS variables — that already live in the React package. It would either duplicate them or depend on the package it was trying to stay separate from.
• Cross-package type imports were already fragile. One shared types package was importing AuthParams from the client package and had to use import type to avoid a circular dependency. Another package in the graph would have turned a line into a mesh.
• The maintenance overhead would be real, and the isolation would be an illusion.

Integrate into the existing packages. Add a GraphQL query to the API package, a REST call to the dataplane package, a sibling client to the frontend package, and a context plus components to the React package.

This was the proposal I wrote up and brought to the team. The key realization: the feature is purely additive. New exports cannot break existing consumers, so the "no breaking changes" argument for a separate package was a false constraint.

Decision: integrate. The no breaking changes risk was not real, and the code would immediately want to cross the package boundary anyway. The accepted trade-off is a slightly larger React package, mitigated by tree-shaking — consumers who do not import the feature don't pay for it.

Design decisions​

A sibling client, not a subclass. The feature has a fundamentally different auth model: no persistent session, no storage. Making it a subclass of the main client would have forced it to inherit behaviour it must not have. It became a sibling with a much lighter configuration:

type LinkClientConfig = {
  env?: Environment;
  authParams: AuthParams;
};

The SDK never touches the wallet. The backend returns a serialized transaction; the SDK passes it to the host through an actionHandler callback. The host signs and submits. This is what keeps the feature blockchain-agnostic at the API boundary.

Reuse the existing UI infrastructure. Theme support, error views, CSS variables and the established classNames override pattern all came from the React package instead of being reimplemented.

How it works​

The feature is split across four layers:

api package      → a GraphQL query for the link configuration
dataplane package → a REST call to activate an action
frontend package  → models + a client + a factory function
react package     → a context provider + components + input widgets

The data flow:

User visits the link URL
  → provider initializes the link client
  → component mounts and fetches the config
      → GraphQL returns the raw config as a JSON string
      → JSON.parse + a type guard validate it
  → context stores the config per link id
  → action state is initialized with default inputs per action

User fills in inputs and clicks an action
  → the action validates that the blockchain type matches the config
  → the client POSTs { actionId, authParams, inputs } to the dataplane
  → the response carries transactions plus success/failure messages
  → actionHandler(payload) — the host signs and submits

Three details worth copying:

Dictionaries keyed by id. The context holds configs keyed by link id and action state keyed by ${linkId}:${actionId}. A single provider can serve multiple components on the same page without refetching anything.

Dual service injection. Config fetch is a GraphQL read (cacheable, tenant-level). Action execution is a REST write (user-specific, requires auth params). The client takes both services explicitly rather than hiding the difference.

A preAction prop. The component accepts an optional pre-action with disabled, label and onClick. This lets the host gate execution behind a wallet-connection step without the SDK knowing anything about wallet state. If it is omitted, the action button executes directly.

Outcome​

The feature shipped to production in May 2025 with zero breaking changes. The cross-package circular dependency was avoided with type-only imports, and a component test covered the new UI. One post-launch fix was needed: the dataplane endpoint path was case-sensitive and the deployment had it capitalized; the client was corrected to match.

What I took away​

• For an additive feature, "we might break something" is rarely a reason to create a new package. New exports are safe by construction.
• Draw the dependency graph before choosing the boundaries. A package that must depend on the thing it is supposed to be isolated from is just indirection.
• Two different auth models deserve two sibling clients, not inheritance.
• Letting the host sign the transaction is what makes a feature blockchain-agnostic. That is an API-boundary decision, not a refactor.

Ours was not honest. The backend had two ways of reporting failure, and the client only handled one of them.

Two error channels​

GraphQL gives an API two distinct places to put an error.

Protocol errors live in the top-level errors array:

{
  "errors": [
    {
      "message": "The current user is not authorized to access this resource.",
      "extensions": { "code": "AUTH_NOT_AUTHENTICATED" }
    }
  ],
  "data": { "alert": null }
}

These are system-level failures: missing authentication, broken requests, legacy operations. Our HTTP client already converted them into a generic ClientError, so this channel worked.

Payload errors live inside the response data:

{
  "errors": null,
  "data": {
    "createSlackChannelTarget": {
      "slackChannelTarget": null,
      "errors": [
        {
          "__typename": "TargetLimitExceededError",
          "message": "You have reached the maximum number of targets."
        }
      ]
    }
  }
}

This channel carries business-rule failures: quota limits, missing targets, invalid arguments. It is type-safe by design — the __typename tells you exactly which error you got.

And it was largely ignored. Mutation methods returned the payload to the caller and left the errors array for them to notice. Some did not even use the pattern yet. The result was an API surface where the same failure could be a rejection, a silent no-op, or a null field depending on which mutation you called.

What good looks like​

We wanted three properties:

• One type to check. Callers should be able to ask "is this an authentication error?" without string matching a message.
• No lost context. The original __typename, the message and the underlying cause should all survive the conversion.
• No big-bang break. Existing throws had to keep working while new code adopted the new type.

The design​

We added an errors/ module to the client package rather than replacing the existing error handling. At its centre is a base class and a factory:

export class SdkError extends Error {
  readonly errorType: string;
  readonly code: string;
  readonly cause?: unknown;

  static from(e: unknown): SdkError {
    if (e instanceof SdkError) return e;
    if (isPayloadError(e)) return fromPayloadError(e);
    if (e instanceof Error) return new SdkUnknownError(e.message, 'UNKNOWN', e);
    return new SdkUnknownError(String(e), 'UNKNOWN', e);
  }
}

The factory is the key detail. Instead of asking every call site to know which class to construct, they all funnel through SdkError.from(...). It is idempotent, it accepts anything, and it never throws while trying to describe a throw.

A type guard identifies payload errors by their shape:

function isPayloadError(
  e: unknown,
): e is { __typename: string; message: string } {
  return (
    typeof e === 'object' &&
    e !== null &&
    '__typename' in e &&
    'message' in e
  );
}

Then a switch maps error families onto subclasses:

TargetLimitExceededError, TargetDoesNotExistError, Web3TargetNotFoundError, ...
  → SdkTargetError (TARGET)
UnauthorizedAccessError
  → SdkAuthenticationError (AUTHENTICATION)
ArgumentError, ArgumentOutOfRangeError, ArgumentNullError
  → SdkValidationError (VALIDATION)
everything else
  → SdkUnknownError (UNKNOWN)

Each subclass carries an errorType category, the original backend code, a timestamp and the cause. Consumers can now branch on category instead of probing messages.

At the call sites, mutations validate their payload before returning:

const mutation = await this.service.createWebPushTarget(input);
const errors = mutation.createWebPushTarget.errors;
if (errors && errors.length > 0) {
  throw SdkError.from(errors[0]);
}
return mutation;

In the React layer, unsafe casts disappeared:

// before
.catch((e) => setError(e as Error))

// after
.catch((e) => setError(SdkError.from(e)))

The unglamorous part​

The interesting engineering was not the class hierarchy. It was the schema archaeology needed to make the hierarchy complete.

Payload errors only exist if the schema declares them, and it did not declare them consistently. We catalogued every mutation the SDK consumed, recorded which ones implemented the pattern, and then added the missing error fragments to the schema and type generation. Only after the types were honest could the runtime be.

A few operations also turned out to return protocol errors where the schema promised payload errors. The abstraction had to tolerate both — which the from() factory does by construction.

The deliberate breaking change​

The migration shipped in a major release, because it changed observable behaviour: mutations that previously swallowed payload errors now throw them. The release notes called it out explicitly, with a migration example:

try {
  await client.deleteAlerts({ ids: alertIds });
} catch (error) {
  if (error instanceof SdkValidationError) {
    // handle validation error
  }
}

That is the honest version. The previous behaviour — accepting an empty ID list and returning as if it did something — was the actual bug.

Why categories matter​

Unified errors are not just nicer to catch. They make automations possible.

We wanted to wire an on-call paging tool into the SDK's CI/CD pipeline, but paging a human on every expected failure is worse than no paging at all. Categorised errors let the pipeline ignore known conditions such as rate limits while still escalating genuine faults. Without the abstraction, that classification would have been string matching on messages — which breaks the first time a message is reworded.

If an SDK reports failures in two ways, callers will handle one of them and ignore the other. Collapsing both into a typed, categorised error is the smallest change that makes the API tell the truth.

The goal​

After the migration: the cluster is the output of a repository. A change is a commit, reviewable, revertible, and reconciled automatically. Manual kubectl apply is the exception, not the workflow.

Tooling​

• Flux CD as the GitOps controller, bootstrapped into the cluster.
• SOPS with age for encrypting secrets in the repository.
• flux check --pre before bootstrapping, to validate the cluster meets the prerequisites.

The bootstrap installs the Flux controllers and wires them to the repository, after which everything else arrives through reconciliation.

Repository layout​

clusters/production/     # flux-system + one Kustomization per app
apps/<app>/base/         # manifests for a single app
apps/<app>/kustomization.yaml
apps/<app>/secrets.enc.yaml
infrastructure/namespaces/
scripts/
docs/

The clusters/ directory describes what should exist in the cluster; apps/ describes what each thing is; infrastructure/ holds shared prerequisites like namespaces. Splitting them matters because an app's manifests should be movable without changing how the cluster consumes them.

How reconciliation is wired​

Each app gets its own Flux Kustomization resource that points at its directory:

• interval: 10m — how often the repository is compared to the cluster.
• retryInterval: 2m, timeout: 5m — bounded retries for a broken apply.
• prune: true — resources removed from Git are removed from the cluster.
• wait: true plus health checks — reconciliation is not "done" until the resources are actually healthy.
• decryption.provider: sops — secrets are decrypted in-cluster at apply time.

Ten minutes sounds slow when iterating, and it is. For a home cluster the trade is fine: drift gets corrected without anyone watching, and a bad commit is undone with git revert instead of a manual cleanup.

Secrets​

Secrets are committed encrypted. SOPS is configured with an age key pair: the public key is used to encrypt, the private key never enters Git. The private key is backed up offline, and the cluster receives it once as a sops-age secret in the flux-system namespace so the controllers can decrypt at apply time.

The practical benefit is reviewability: an encrypted diff still shows which keys changed, so a secret rotation is visible in a pull request without ever exposing the value.

Proving it with a pilot app​

The migration was validated with a single representative app — a web application with a database, an encrypted secret, persistent storage and an ingress. Once that app reconciled end to end, the pattern was repeated for the rest.

Verification steps that were worth formalizing:

• flux get kustomizations shows Ready and Applied revision.
• kubectl get all -n <app> matches the repository.
• An intentional annotation change in Git shows up in the cluster.
• git revert of that change rolls it back without manual intervention.

What stayed manual​

Node-level configuration (the kubelet and datastore settings on each control-plane node) is not part of this repository. GitOps covers workloads and their configuration, not the machines running them. That boundary is worth writing down so it is not mistaken for drift.

Takeaways​

• The win is not automation for its own sake; it is that the intended state is written down once and reviewed like code.
• Encrypted secrets in Git are workable when the key management is explicit and the private key is backed up somewhere that is not the repository.
• A pilot app is enough to prove the layout before migrating everything.
• Keep a rollback path (git revert) and test it early, while the change is still small.

A host on the home LAN became unreachable from other LAN devices. Every ping and connection attempt failed, while the same host answered normally over its Tailscale address. The router logged ICMP redirects toward the affected host during the failures, which pointed at a routing disagreement rather than a broken interface.

How Tailscale subnet routers work​

A subnet router advertises routes for a physical network into the tailnet, so remote clients can reach that network without running Tailscale on every device. Nodes that accept those routes install them in a separate routing table and add a policy rule that sends matching traffic through the Tailscale interface instead of the default route.

That design is what makes subnet routing convenient — and what makes overlaps dangerous.

Root cause​

One host on the LAN was running Tailscale as a subnet router for the same subnet it was connected to, with route acceptance enabled. Two facts combined:

• Inbound traffic reached the host over the LAN, as expected.
• Replies to that traffic matched the accepted-route rule and left through the Tailscale interface.

The return path no longer matched the request path. Requests arrived over Ethernet, replies departed over the VPN, and the peer discarded the replies because they never came back the way they went out. This is asymmetric routing: every interface is up, every route looks plausible in isolation, and traffic still disappears.

The detail that makes this nasty is that the host is the only one affected. Other LAN devices route to it normally; the problem lives entirely in its policy routing rules.

A useful contrast is how different systems ship: some appliance operating systems install a high-priority rule that keeps local-network destinations in the main routing table by default, while a plain general-purpose Linux install does not. The same network, the same Tailscale settings, different failure behavior — the protection rule is the difference.

The fix​

Keep traffic destined for the local subnet in the main routing table, with a priority higher than the accepted-routes rule:

ip rule add from all to <lan-subnet> table main priority 5000

To survive reboots, persist it as a small systemd unit:

[Unit]
Description=Keep local subnet traffic in the main routing table
After=network-online.target tailscaled.service

[Service]
Type=oneshot
ExecStart=/usr/sbin/ip rule add from all to <lan-subnet> table main priority 5000
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Verify by checking which table wins for a lookup toward another LAN host:

ip rule show
ip route get <lan-host> from <affected-host>

Before the fix, the lookup resolves through the Tailscale table; after it, the main table matches first.

Policy routing priorities​

Linux evaluates policy routing rules in ascending priority order, and the first match wins. The relevant neighborhood on a Tailscale node looks like this:

The protection rule has to sit between the local rules and the accepted-routes rule. Priority 5000 is the conventional choice because it is high enough to beat 5270 but low enough to leave Tailscale's own marked traffic alone.

Best practices​

• Do not advertise a subnet from a subnet router that lives inside that same subnet.
• If the overlap is unavoidable, add the protection rule to every node on the subnet, not only to the one that failed.
• Treat ICMP redirects from the router as a routing smell worth investigating.
• Infrastructure devices that do not need remote subnet access should not accept routes at all.

Appendix: what disabling route acceptance does and does not do​

Turning off route acceptance stops a node from installing routes advertised by other subnet routers. It does not block inbound connections to that node.

The direction matters. If a device does not need to reach remote networks through the tailnet, disabling route acceptance removes the outbound table-52 path and is the simplest prevention. If the device must accept routes, the protection rule above keeps its replies on the LAN without giving up access to remote subnets. The two scenarios:

• No route acceptance: no table-52 entries, no asymmetric path, no protection rule needed.
• Route acceptance: table-52 entries exist; add the protection rule so LAN traffic still prefers the main table.

Takeaway​

When a host is unreachable from its own network but fine over Tailscale, suspect policy routing asymmetry before replacing hardware: an overlap between an advertised subnet and a local interface is enough to black-hole traffic while every status indicator stays green.

It kept going: the sync container's health check had failed tens of thousands of times in a row, and the webhook that was supposed to report sync failures had been returning 404 for longer still. Three independent problems, stacked so neatly that each one hid the next.

The layers that failed​

replica node down      → resolution still works (primary answers)
sync job failing       → no visible symptom (replica not serving)
notification broken    → failure report goes nowhere (webhook 404)

Any one of these being healthy would have surfaced the others. The redundancy worked so well that the failure was invisible.

What actual HA requires​

• Monitor the replica, not the service. "DNS resolves" only proves that a resolver is up. A replica check has to ask the replica directly — a query against its own address, not the shared name.
• Check the alert delivery path. A notification channel is a dependency like any other. Stale webhooks, expired tokens and changed URLs all fail silently unless something tests them. A periodic test alert or a dead man's switch turns "nobody was told" into a detectable condition.
• A failing health check must reach a human. Repeatedly failing health checks that only appear in container logs are decoration.
• Test failover. Until the replica has actually served queries while the primary was down, "HA" is a hope. A planned failover test is the only proof.
• Make rejoining automatic. Firewall rules and reconnect configuration have to survive a reboot, or a restarted replica stays offline — which is exactly how a one-day outage becomes a month.

The uncomfortable takeaway​

The system did not fail because DNS stopped working. It failed because the redundancy was never exercised, and the reporting chain had the same blind spot as the thing it reported on. Redundancy without a test is just a second copy of the same assumption.

Two systems, two jobs​

• Prometheus scrapes metrics, evaluates rules, and decides when an alert is firing.
• Alertmanager receives firing alerts and decides how to group, route, inhibit and deliver them.

Keeping the split in mind prevents a common confusion: a rule that never fires and a notification that never arrives are different problems in different systems.

The life of an alert​

Inactive → Pending → Firing

A rule's expression becomes true, the alert enters Pending, and it stays there until it has been continuously true for the rule's for duration. Only then does it become Firing and get sent to Alertmanager. The for window is the difference between "this spiked for one scrape" and "this is broken".

Where the rules come from​

The kube-prometheus-stack chart ships rule packs that can be toggled on and off: application-level rules, node rules, and so on. One toggle is worth calling out — datastore rules are typically disabled by default because managed Kubernetes distributions own the datastore, and enabling the rules without the matching metrics only produces confusion.

To see what is actually installed:

kubectl get prometheusrules
kubectl get prometheusrule <name> -o yaml

The object YAML is the ground truth: the expression, the threshold and the for duration. Reading the live state is the Prometheus UI's Alerts page, which shows each rule as inactive, pending or firing with the current value.

A representative rule looks like this — a pod stuck in a crash loop:

alert: KubePodCrashLooping
expr: max_over_time(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}[5m]) >= 1
for: 15m
labels:
  severity: warning

Know which metrics you actually have​

Two exporters carry very different information:

• kube-state-metrics reports the state of API objects: how many replicas a deployment wants, whether a pod is waiting, and why.
• cAdvisor reports container resource usage: CPU, memory, filesystem.

If cAdvisor is dropped to save resources — a reasonable choice on weak hardware — the object-level rules keep working, but every rule that depends on container memory or CPU silently has no data. Nothing breaks; the alert just never fires. Worth writing down at install time.

Grouping, timing and silencing​

Alertmanager's routing tree decides where alerts go; grouping decides how many messages a person receives. A crash-looping app with a bad replica count can produce several alerts at once, and group_by collapses them:

group_by: [namespace, alertname, severity]

Group too little and the phone buzzes per pod. Group too much and unrelated problems arrive glued together. The timing knobs are:

• group_wait — how long to collect alerts before sending the first message.
• group_interval — how often to send updates about an existing group.
• repeat_interval — how often a still-firing alert is repeated.

Silences are the maintenance tool: a time-boxed mute that expires on its own. They are better than editing rules for planned work, but a silence that is too broad hides new problems inside its scope.

The blind spot: whitebox without blackbox​

Most of this stack is whitebox monitoring — it reports from inside the system. Pods say they are running, services say they exist. What it cannot tell you is whether a user can reach anything.

The classic gap: the ingress is broken, every pod is healthy, and Prometheus is happy. The cluster is "green" and the site is down.

Blackbox probing closes this: probe the important endpoints from outside on a schedule and alert when the response is wrong. If only one thing is added after the initial setup, this is the one with the highest return.

Recording rules​

On a small cluster, dashboards that recompute expensive expressions on every load can cost more than the monitoring is worth. Recording rules precompute an expression into a new metric:

record: job:request_rate:5m
expr: sum(rate(http_requests_total[5m])) by (job)

The dashboard then reads a cheap series. This matters most on low-power hardware, where a heavy query is competing with the workload it observes.

Takeaways​

• A firing rule and a delivered notification are separate systems; design and test both.
• Know which metrics are absent before trusting an alert to cover something.
• Use for durations to filter noise, and group intentionally.
• Whitebox monitoring cannot see broken entry paths. Add a blackbox probe.
• On weak hardware, recording rules are a performance feature, not a luxury.