Most SDK features are designed in code. This one was designed so that partners could define it in an admin panel.

The feature lets dApp developers configure interactive on-chain actions through a backend UI and render them as components through the SDK. When a user clicks an action, the SDK asks the backend for a serialized transaction, hands it to the host application, and lets the host sign and submit it. The feature went from a spike in late March 2025 to production in May 2025.

The constraints​

• Time to market. The product team wanted a fast delivery path, which initially pushed the design toward a brand-new package.
• No breaking changes. Existing consumers of the client and React packages could not be affected.
• Blockchain-agnostic. The action-execution layer could not be tied to EVM, Solana, SUI or anything else; the SDK should return a transaction payload and let the host handle signing.
• A different auth model. The feature does not require a logged-in user session. It only needs authParams — a wallet public key and a blockchain type — to sign the activation request.

The options​

A new package. The original proposal was a separate package containing the UI components and a new client. On paper this avoids touching existing packages and gives clean separation.

The problems only show up when you draw the dependency graph:

• The UI needs types and the client from the existing frontend package, plus React context patterns — theme support, error views, CSS variables — that already live in the React package. It would either duplicate them or depend on the package it was trying to stay separate from.
• Cross-package type imports were already fragile. One shared types package was importing AuthParams from the client package and had to use import type to avoid a circular dependency. Another package in the graph would have turned a line into a mesh.
• The maintenance overhead would be real, and the isolation would be an illusion.

Integrate into the existing packages. Add a GraphQL query to the API package, a REST call to the dataplane package, a sibling client to the frontend package, and a context plus components to the React package.

This was the proposal I wrote up and brought to the team. The key realization: the feature is purely additive. New exports cannot break existing consumers, so the "no breaking changes" argument for a separate package was a false constraint.

Decision: integrate. The no breaking changes risk was not real, and the code would immediately want to cross the package boundary anyway. The accepted trade-off is a slightly larger React package, mitigated by tree-shaking — consumers who do not import the feature don't pay for it.

Design decisions​

A sibling client, not a subclass. The feature has a fundamentally different auth model: no persistent session, no storage. Making it a subclass of the main client would have forced it to inherit behaviour it must not have. It became a sibling with a much lighter configuration:

type LinkClientConfig = {
  env?: Environment;
  authParams: AuthParams;
};

The SDK never touches the wallet. The backend returns a serialized transaction; the SDK passes it to the host through an actionHandler callback. The host signs and submits. This is what keeps the feature blockchain-agnostic at the API boundary.

Reuse the existing UI infrastructure. Theme support, error views, CSS variables and the established classNames override pattern all came from the React package instead of being reimplemented.

How it works​

The feature is split across four layers:

api package      → a GraphQL query for the link configuration
dataplane package → a REST call to activate an action
frontend package  → models + a client + a factory function
react package     → a context provider + components + input widgets

The data flow:

User visits the link URL
  → provider initializes the link client
  → component mounts and fetches the config
      → GraphQL returns the raw config as a JSON string
      → JSON.parse + a type guard validate it
  → context stores the config per link id
  → action state is initialized with default inputs per action

User fills in inputs and clicks an action
  → the action validates that the blockchain type matches the config
  → the client POSTs { actionId, authParams, inputs } to the dataplane
  → the response carries transactions plus success/failure messages
  → actionHandler(payload) — the host signs and submits

Three details worth copying:

Dictionaries keyed by id. The context holds configs keyed by link id and action state keyed by ${linkId}:${actionId}. A single provider can serve multiple components on the same page without refetching anything.

Dual service injection. Config fetch is a GraphQL read (cacheable, tenant-level). Action execution is a REST write (user-specific, requires auth params). The client takes both services explicitly rather than hiding the difference.

A preAction prop. The component accepts an optional pre-action with disabled, label and onClick. This lets the host gate execution behind a wallet-connection step without the SDK knowing anything about wallet state. If it is omitted, the action button executes directly.

Outcome​

The feature shipped to production in May 2025 with zero breaking changes. The cross-package circular dependency was avoided with type-only imports, and a component test covered the new UI. One post-launch fix was needed: the dataplane endpoint path was case-sensitive and the deployment had it capitalized; the client was corrected to match.

What I took away​

• For an additive feature, "we might break something" is rarely a reason to create a new package. New exports are safe by construction.
• Draw the dependency graph before choosing the boundaries. A package that must depend on the thing it is supposed to be isolated from is just indirection.
• Two different auth models deserve two sibling clients, not inheritance.
• Letting the host sign the transaction is what makes a feature blockchain-agnostic. That is an API-boundary decision, not a refactor.

An SDK's error behaviour is part of its public API. Every caller has to make a decision about errors — handle them, retry them, or ignore them — and they can only make that decision if the error they receive is honest about what happened.

Ours was not honest. The backend had two ways of reporting failure, and the client only handled one of them.

Two error channels​

GraphQL gives an API two distinct places to put an error.

Protocol errors live in the top-level errors array:

{
  "errors": [
    {
      "message": "The current user is not authorized to access this resource.",
      "extensions": { "code": "AUTH_NOT_AUTHENTICATED" }
    }
  ],
  "data": { "alert": null }
}

These are system-level failures: missing authentication, broken requests, legacy operations. Our HTTP client already converted them into a generic ClientError, so this channel worked.

Payload errors live inside the response data:

{
  "errors": null,
  "data": {
    "createSlackChannelTarget": {
      "slackChannelTarget": null,
      "errors": [
        {
          "__typename": "TargetLimitExceededError",
          "message": "You have reached the maximum number of targets."
        }
      ]
    }
  }
}

This channel carries business-rule failures: quota limits, missing targets, invalid arguments. It is type-safe by design — the __typename tells you exactly which error you got.

And it was largely ignored. Mutation methods returned the payload to the caller and left the errors array for them to notice. Some did not even use the pattern yet. The result was an API surface where the same failure could be a rejection, a silent no-op, or a null field depending on which mutation you called.

What good looks like​

We wanted three properties:

• One type to check. Callers should be able to ask "is this an authentication error?" without string matching a message.
• No lost context. The original __typename, the message and the underlying cause should all survive the conversion.
• No big-bang break. Existing throws had to keep working while new code adopted the new type.

The design​

We added an errors/ module to the client package rather than replacing the existing error handling. At its centre is a base class and a factory:

export class SdkError extends Error {
  readonly errorType: string;
  readonly code: string;
  readonly cause?: unknown;

  static from(e: unknown): SdkError {
    if (e instanceof SdkError) return e;
    if (isPayloadError(e)) return fromPayloadError(e);
    if (e instanceof Error) return new SdkUnknownError(e.message, 'UNKNOWN', e);
    return new SdkUnknownError(String(e), 'UNKNOWN', e);
  }
}

The factory is the key detail. Instead of asking every call site to know which class to construct, they all funnel through SdkError.from(...). It is idempotent, it accepts anything, and it never throws while trying to describe a throw.

A type guard identifies payload errors by their shape:

function isPayloadError(
  e: unknown,
): e is { __typename: string; message: string } {
  return (
    typeof e === 'object' &&
    e !== null &&
    '__typename' in e &&
    'message' in e
  );
}

Then a switch maps error families onto subclasses:

TargetLimitExceededError, TargetDoesNotExistError, Web3TargetNotFoundError, ...
  → SdkTargetError (TARGET)
UnauthorizedAccessError
  → SdkAuthenticationError (AUTHENTICATION)
ArgumentError, ArgumentOutOfRangeError, ArgumentNullError
  → SdkValidationError (VALIDATION)
everything else
  → SdkUnknownError (UNKNOWN)

Each subclass carries an errorType category, the original backend code, a timestamp and the cause. Consumers can now branch on category instead of probing messages.

At the call sites, mutations validate their payload before returning:

const mutation = await this.service.createWebPushTarget(input);
const errors = mutation.createWebPushTarget.errors;
if (errors && errors.length > 0) {
  throw SdkError.from(errors[0]);
}
return mutation;

In the React layer, unsafe casts disappeared:

// before
.catch((e) => setError(e as Error))

// after
.catch((e) => setError(SdkError.from(e)))

The unglamorous part​

The interesting engineering was not the class hierarchy. It was the schema archaeology needed to make the hierarchy complete.

Payload errors only exist if the schema declares them, and it did not declare them consistently. We catalogued every mutation the SDK consumed, recorded which ones implemented the pattern, and then added the missing error fragments to the schema and type generation. Only after the types were honest could the runtime be.

A few operations also turned out to return protocol errors where the schema promised payload errors. The abstraction had to tolerate both — which the from() factory does by construction.

The deliberate breaking change​

The migration shipped in a major release, because it changed observable behaviour: mutations that previously swallowed payload errors now throw them. The release notes called it out explicitly, with a migration example:

try {
  await client.deleteAlerts({ ids: alertIds });
} catch (error) {
  if (error instanceof SdkValidationError) {
    // handle validation error
  }
}

That is the honest version. The previous behaviour — accepting an empty ID list and returning as if it did something — was the actual bug.

Why categories matter​

Unified errors are not just nicer to catch. They make automations possible.

We wanted to wire an on-call paging tool into the SDK's CI/CD pipeline, but paging a human on every expected failure is worse than no paging at all. Categorised errors let the pipeline ignore known conditions such as rate limits while still escalating genuine faults. Without the abstraction, that classification would have been string matching on messages — which breaks the first time a message is reworded.

If an SDK reports failures in two ways, callers will handle one of them and ignore the other. Collapsing both into a typed, categorised error is the smallest change that makes the API tell the truth.

The cluster ran on kubectl apply, one-off Helm commands and a handful of scripts for long enough that nobody could say what the actual state was. The manifests existed somewhere, secrets lived in shell history and password managers, and rolling something back meant remembering what it looked like before. This is how the cluster moved to a Git repository as its source of truth.

The goal​

After the migration: the cluster is the output of a repository. A change is a commit, reviewable, revertible, and reconciled automatically. Manual kubectl apply is the exception, not the workflow.

Tooling​

• Flux CD as the GitOps controller, bootstrapped into the cluster.
• SOPS with age for encrypting secrets in the repository.
• flux check --pre before bootstrapping, to validate the cluster meets the prerequisites.

The bootstrap installs the Flux controllers and wires them to the repository, after which everything else arrives through reconciliation.

Repository layout​

clusters/production/     # flux-system + one Kustomization per app
apps/<app>/base/         # manifests for a single app
apps/<app>/kustomization.yaml
apps/<app>/secrets.enc.yaml
infrastructure/namespaces/
scripts/
docs/

The clusters/ directory describes what should exist in the cluster; apps/ describes what each thing is; infrastructure/ holds shared prerequisites like namespaces. Splitting them matters because an app's manifests should be movable without changing how the cluster consumes them.

How reconciliation is wired​

Each app gets its own Flux Kustomization resource that points at its directory:

• interval: 10m — how often the repository is compared to the cluster.
• retryInterval: 2m, timeout: 5m — bounded retries for a broken apply.
• prune: true — resources removed from Git are removed from the cluster.
• wait: true plus health checks — reconciliation is not "done" until the resources are actually healthy.
• decryption.provider: sops — secrets are decrypted in-cluster at apply time.

Ten minutes sounds slow when iterating, and it is. For a home cluster the trade is fine: drift gets corrected without anyone watching, and a bad commit is undone with git revert instead of a manual cleanup.

Secrets​

Secrets are committed encrypted. SOPS is configured with an age key pair: the public key is used to encrypt, the private key never enters Git. The private key is backed up offline, and the cluster receives it once as a sops-age secret in the flux-system namespace so the controllers can decrypt at apply time.

The practical benefit is reviewability: an encrypted diff still shows which keys changed, so a secret rotation is visible in a pull request without ever exposing the value.

Proving it with a pilot app​

The migration was validated with a single representative app — a web application with a database, an encrypted secret, persistent storage and an ingress. Once that app reconciled end to end, the pattern was repeated for the rest.

Verification steps that were worth formalizing:

• flux get kustomizations shows Ready and Applied revision.
• kubectl get all -n <app> matches the repository.
• An intentional annotation change in Git shows up in the cluster.
• git revert of that change rolls it back without manual intervention.

What stayed manual​

Node-level configuration (the kubelet and datastore settings on each control-plane node) is not part of this repository. GitOps covers workloads and their configuration, not the machines running them. That boundary is worth writing down so it is not mistaken for drift.

Takeaways​

• The win is not automation for its own sake; it is that the intended state is written down once and reviewed like code.
• Encrypted secrets in Git are workable when the key management is explicit and the private key is backed up somewhere that is not the repository.
• A pilot app is enough to prove the layout before migrating everything.
• Keep a rollback path (git revert) and test it early, while the change is still small.

The symptom​

A host on the home LAN became unreachable from other LAN devices. Every ping and connection attempt failed, while the same host answered normally over its Tailscale address. The router logged ICMP redirects toward the affected host during the failures, which pointed at a routing disagreement rather than a broken interface.

How Tailscale subnet routers work​

A subnet router advertises routes for a physical network into the tailnet, so remote clients can reach that network without running Tailscale on every device. Nodes that accept those routes install them in a separate routing table and add a policy rule that sends matching traffic through the Tailscale interface instead of the default route.

That design is what makes subnet routing convenient — and what makes overlaps dangerous.

Root cause​

One host on the LAN was running Tailscale as a subnet router for the same subnet it was connected to, with route acceptance enabled. Two facts combined:

• Inbound traffic reached the host over the LAN, as expected.
• Replies to that traffic matched the accepted-route rule and left through the Tailscale interface.

The return path no longer matched the request path. Requests arrived over Ethernet, replies departed over the VPN, and the peer discarded the replies because they never came back the way they went out. This is asymmetric routing: every interface is up, every route looks plausible in isolation, and traffic still disappears.

The detail that makes this nasty is that the host is the only one affected. Other LAN devices route to it normally; the problem lives entirely in its policy routing rules.

A useful contrast is how different systems ship: some appliance operating systems install a high-priority rule that keeps local-network destinations in the main routing table by default, while a plain general-purpose Linux install does not. The same network, the same Tailscale settings, different failure behavior — the protection rule is the difference.

The fix​

Keep traffic destined for the local subnet in the main routing table, with a priority higher than the accepted-routes rule:

ip rule add from all to <lan-subnet> table main priority 5000

To survive reboots, persist it as a small systemd unit:

[Unit]
Description=Keep local subnet traffic in the main routing table
After=network-online.target tailscaled.service

[Service]
Type=oneshot
ExecStart=/usr/sbin/ip rule add from all to <lan-subnet> table main priority 5000
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Verify by checking which table wins for a lookup toward another LAN host:

ip rule show
ip route get <lan-host> from <affected-host>

Before the fix, the lookup resolves through the Tailscale table; after it, the main table matches first.

Policy routing priorities​

Linux evaluates policy routing rules in ascending priority order, and the first match wins. The relevant neighborhood on a Tailscale node looks like this:

The protection rule has to sit between the local rules and the accepted-routes rule. Priority 5000 is the conventional choice because it is high enough to beat 5270 but low enough to leave Tailscale's own marked traffic alone.

Best practices​

• Do not advertise a subnet from a subnet router that lives inside that same subnet.
• If the overlap is unavoidable, add the protection rule to every node on the subnet, not only to the one that failed.
• Treat ICMP redirects from the router as a routing smell worth investigating.
• Infrastructure devices that do not need remote subnet access should not accept routes at all.

Appendix: what disabling route acceptance does and does not do​

Turning off route acceptance stops a node from installing routes advertised by other subnet routers. It does not block inbound connections to that node.

The direction matters. If a device does not need to reach remote networks through the tailnet, disabling route acceptance removes the outbound table-52 path and is the simplest prevention. If the device must accept routes, the protection rule above keeps its replies on the LAN without giving up access to remote subnets. The two scenarios:

• No route acceptance: no table-52 entries, no asymmetric path, no protection rule needed.
• Route acceptance: table-52 entries exist; add the protection rule so LAN traffic still prefers the main table.

Takeaway​

When a host is unreachable from its own network but fine over Tailscale, suspect policy routing asymmetry before replacing hardware: an overlap between an advertised subnet and a local interface is enough to black-hole traffic while every status indicator stays green.

The local network runs two DNS resolvers — a primary and a replica — kept in sync so that either can answer queries. That is the shape of high availability. What an audit found was that the replica had been offline for about a month, the primary had been answering everything alone, and nothing had said a word.

It kept going: the sync container's health check had failed tens of thousands of times in a row, and the webhook that was supposed to report sync failures had been returning 404 for longer still. Three independent problems, stacked so neatly that each one hid the next.

The layers that failed​

replica node down      → resolution still works (primary answers)
sync job failing       → no visible symptom (replica not serving)
notification broken    → failure report goes nowhere (webhook 404)

Any one of these being healthy would have surfaced the others. The redundancy worked so well that the failure was invisible.

What actual HA requires​

• Monitor the replica, not the service. "DNS resolves" only proves that a resolver is up. A replica check has to ask the replica directly — a query against its own address, not the shared name.
• Check the alert delivery path. A notification channel is a dependency like any other. Stale webhooks, expired tokens and changed URLs all fail silently unless something tests them. A periodic test alert or a dead man's switch turns "nobody was told" into a detectable condition.
• A failing health check must reach a human. Repeatedly failing health checks that only appear in container logs are decoration.
• Test failover. Until the replica has actually served queries while the primary was down, "HA" is a hope. A planned failover test is the only proof.
• Make rejoining automatic. Firewall rules and reconnect configuration have to survive a reboot, or a restarted replica stays offline — which is exactly how a one-day outage becomes a month.

The uncomfortable takeaway​

The system did not fail because DNS stopped working. It failed because the redundancy was never exercised, and the reporting chain had the same blind spot as the thing it reported on. Redundancy without a test is just a second copy of the same assumption.

A monitoring stack usually gets installed for its dashboards and then trusted for its alerts. But an alert is a pipeline: a rule evaluates, a state changes, a notification is routed, a person is interrupted. Every stage can fail quietly. This is what that pipeline looks like in a small cluster and where the sharp edges are.

Two systems, two jobs​

• Prometheus scrapes metrics, evaluates rules, and decides when an alert is firing.
• Alertmanager receives firing alerts and decides how to group, route, inhibit and deliver them.

Keeping the split in mind prevents a common confusion: a rule that never fires and a notification that never arrives are different problems in different systems.

The life of an alert​

Inactive → Pending → Firing

A rule's expression becomes true, the alert enters Pending, and it stays there until it has been continuously true for the rule's for duration. Only then does it become Firing and get sent to Alertmanager. The for window is the difference between "this spiked for one scrape" and "this is broken".

Where the rules come from​

The kube-prometheus-stack chart ships rule packs that can be toggled on and off: application-level rules, node rules, and so on. One toggle is worth calling out — datastore rules are typically disabled by default because managed Kubernetes distributions own the datastore, and enabling the rules without the matching metrics only produces confusion.

To see what is actually installed:

kubectl get prometheusrules
kubectl get prometheusrule <name> -o yaml

The object YAML is the ground truth: the expression, the threshold and the for duration. Reading the live state is the Prometheus UI's Alerts page, which shows each rule as inactive, pending or firing with the current value.

A representative rule looks like this — a pod stuck in a crash loop:

alert: KubePodCrashLooping
expr: max_over_time(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}[5m]) >= 1
for: 15m
labels:
  severity: warning

Know which metrics you actually have​

Two exporters carry very different information:

• kube-state-metrics reports the state of API objects: how many replicas a deployment wants, whether a pod is waiting, and why.
• cAdvisor reports container resource usage: CPU, memory, filesystem.

If cAdvisor is dropped to save resources — a reasonable choice on weak hardware — the object-level rules keep working, but every rule that depends on container memory or CPU silently has no data. Nothing breaks; the alert just never fires. Worth writing down at install time.

Grouping, timing and silencing​

Alertmanager's routing tree decides where alerts go; grouping decides how many messages a person receives. A crash-looping app with a bad replica count can produce several alerts at once, and group_by collapses them:

group_by: [namespace, alertname, severity]

Group too little and the phone buzzes per pod. Group too much and unrelated problems arrive glued together. The timing knobs are:

• group_wait — how long to collect alerts before sending the first message.
• group_interval — how often to send updates about an existing group.
• repeat_interval — how often a still-firing alert is repeated.

Silences are the maintenance tool: a time-boxed mute that expires on its own. They are better than editing rules for planned work, but a silence that is too broad hides new problems inside its scope.

The blind spot: whitebox without blackbox​

Most of this stack is whitebox monitoring — it reports from inside the system. Pods say they are running, services say they exist. What it cannot tell you is whether a user can reach anything.

The classic gap: the ingress is broken, every pod is healthy, and Prometheus is happy. The cluster is "green" and the site is down.

Blackbox probing closes this: probe the important endpoints from outside on a schedule and alert when the response is wrong. If only one thing is added after the initial setup, this is the one with the highest return.

Recording rules​

On a small cluster, dashboards that recompute expensive expressions on every load can cost more than the monitoring is worth. Recording rules precompute an expression into a new metric:

record: job:request_rate:5m
expr: sum(rate(http_requests_total[5m])) by (job)

The dashboard then reads a cheap series. This matters most on low-power hardware, where a heavy query is competing with the workload it observes.

Takeaways​

• A firing rule and a delivered notification are separate systems; design and test both.
• Know which metrics are absent before trusting an alert to cover something.
• Use for durations to filter noise, and group intentionally.
• Whitebox monitoring cannot see broken entry paths. Add a blackbox probe.
• On weak hardware, recording rules are a performance feature, not a luxury.