A home cluster serves two audiences: the public internet and devices on the local network. Both paths eventually need to land on the ingress controller, and both have their own failure modes. This is the design the cluster settled on, and the part of it that is still weaker than it looks.

The building blocks​

• Ingress controller: Traefik, installed as the cluster's ingress, routing by hostname to Services.
• LoadBalancer implementation: Traefik is exposed through a Service of type LoadBalancer. With no cloud provider, something in the cluster has to implement that — either a LoadBalancer controller or a shared virtual IP.
• API VIP: kube-vip provides a virtual IP for the Kubernetes API server so the control plane has a stable address independent of which node is master.
• External entry: a tunnel daemon runs as a container and forwards public traffic to the ingress origin on the local network. Internal entry uses local DNS.

What "LoadBalancer" actually means here​

On a cloud provider, a LoadBalancer Service provisions a real balancer. On bare metal it is whatever the implementation provides. In this cluster, Traefik's LoadBalancer status lists all node addresses — the controller lets any node accept and forward traffic for the service. That is convenient and it is also easy to misread: the external address field is not a single stable VIP you can point DNS at.

The API VIP is separate and intentionally so. If the application VIP and the API VIP shared an address, a routing problem in one would take out the other. An early attempt to give applications a managed VIP failed and was reverted (see the post-mortem on the ARP fallout). The design keeps them apart while the application path is reworked.

The two entry paths​

Internet → tunnel daemon → ingress origin → Traefik → Service → Pod
LAN      → local DNS     → node address   → Traefik → Service → Pod

The external path has one manual seam: the tunnel's origin address is configured in the tunnel dashboard, not in Git. The internal path has a different problem.

The DNS single point of failure​

Internal hostnames resolved through the local DNS server, and the wildcard record pointed at one node's address. Every internal request that entered the cluster went through that node first.

This is important to name correctly: the bottleneck is not Traefik. Traefik can be reached on any node. The single point of failure is the DNS record. If that node is down, internal clients cannot reach the cluster even though the ingress path itself is healthy.

The intended topology is:

local DNS → application VIP → Traefik → any node

Once an application VIP exists, the DNS record stops being a SPOF; the VIP takes over that role, and the VIP is a single address by design — which means it needs its own answer for what happens when its holder fails.

Layers and what manages them​

The pattern worth noticing: the pieces closest to the network edge are the least declarative. The tunnel origin and the DNS records are infrastructure state living outside Git, and they are exactly the pieces that are hardest to reconstruct after an outage.

Takeaways​

• A LoadBalancer Service on bare metal is an implementation, not a cloud guarantee. Understand what backs it.
• Keep the API VIP and any application VIP separate.
• A DNS record pointing at a single node is a single point of failure even when the ingress controller is healthy.
• Inventory the manual seams — tunnel configuration, DNS records — before an incident forces you to.

The small Kubernetes cluster at home started life on Wi-Fi. The nodes were spread across rooms, wireless was good enough for light workloads, and running a cable to every control plane felt like unnecessary work. That held up for a while — until the etcd members started complaining.

The symptom​

The cluster has three control-plane nodes running embedded etcd for high availability. Over time the logs filled with etcd peer timeouts, the API server slowed down, and leader elections started flapping. Under load the whole control plane became briefly unavailable.

The first reaction was tuning. Raising the etcd heartbeat interval and election timeout (heartbeat-interval=500, election-timeout=5000) made the errors less frequent, which was enough to call the problem solved for a while. It was not a fix: it only made etcd more forgiving of a network that should never have been carrying peer traffic in the first place.

etcd is unusually sensitive to latency and jitter between members. Wireless is exactly that: retries, roaming, and interference that show up as milliseconds of variance at the worst possible moment.

The fix: wire the control planes​

All three control-plane nodes were moved to wired Ethernet (eth0). The worker nodes did not need to change — etcd peers are the strict requirement, not every workload.

The wireless interfaces were kept as backups with a higher route metric, so a cable failure does not take a node off the network.

Migrating one etcd member​

The migration is a per-node procedure, and the ordering matters:

1. Confirm the node's new wired address.

2. Install etcdctl on every control-plane node. K3s does not bundle it, and it has to be reinstalled after a node is reprovisioned.

3. From a healthy member, list the cluster and update the peer URL:

   etcdctl member list
   etcdctl member update <member-id> \
     --peer-urls=https://<new-address>:2380
   

   The certificates live under /var/lib/rancher/k3s/server/tls/etcd/ (server-ca.crt, client.crt, client.key).

4. Update /etc/rancher/k3s/config.yaml on that node with the new node-ip and advertise-address.

5. Restart K3s on the node.

6. Verify: the node is Ready, etcdctl member list shows the new peer URL, and the node annotations point at the wired address.

The caveat that almost broke quorum​

If the node being migrated is the only healthy member, it cannot rejoin on a new peer URL that the other members do not know about yet. The safe sequence is to boot it on the old address first, let quorum recover, then update the peer URL and restart.

Losing quorum mid-migration is the failure mode to design around: on a three-member cluster, one member can be offline at a time, and no more.

kube-vip​

The virtual IP for the API server is managed by kube-vip, which had been configured against the wireless interface. That was switched to eth0 as part of the migration. The gratuitous-ARP workarounds that had accumulated while the control planes were on Wi-Fi — needed because wireless clients can miss ARP updates — are no longer required.

What is not managed declaratively​

Node-level changes — /etc/rancher/k3s/config.yaml, the etcdctl binary, etcd member peer URLs — are outside GitOps. The cluster's workloads are reconciled from Git, but the datastore identity of each control-plane node is manual state. After this migration those details are worth recording somewhere, because the next person to touch the cluster will need them.

Takeaways​

• etcd peer traffic belongs on a wired link. Tuning timeouts buys time, not stability.
• Install etcdctl on every control-plane node before you need it.
• On a three-node control plane, migrate one member at a time and never lose two.
• Keep the wireless interface as a backup with a higher metric instead of deleting it.

Giving an agent tools is the easy part. Making it follow your platform's integration workflow — correctly, on a real codebase, without rewriting the app around it — is where the work is.

A skill is a workflow, not a README​

The first instinct is to hand the agent the documentation. That fails for a predictable reason: documentation describes the API, but integration is a sequence of decisions.

The skill we wrote encodes that sequence:

• Choose the path. Drop-in card, custom UI through context hooks, or an action-link flow.
• Choose the authentication mode. On-chain wallet signing, off-chain sign-in, or a special case.
• Confirm the inputs. Tenant id, card id, environment, wallet chain, keys, signing method. If something is missing, ask — do not guess.
• Integrate minimally. Fit the existing app structure; do not invent a new architecture around it.

It opens with a short decision tree, then goes straight to implementation. Every section exists to remove a decision the agent would otherwise make wrongly by default.

The failure modes are the content​

Half of the skill is a list of ways integrations go wrong: provider mounted in the wrong place, auth mode mismatched with the wallet connection, subscription values parsed without checking the shape they depend on. These are not API errors — they are judgment errors. A skill is valuable exactly to the extent that it prevents them.

You cannot eyeball this​

A skill that reads well can still make an agent worse. The only way to know is to run it against a real task and compare.

The evaluation setup has four parts:

• Evals — the test cases: a prompt, the expected behavior, and what a reviewer should look at.
• Fixtures — an immutable baseline codebase to run against.
• Workspaces — the artifacts of each run: final response, transcript, diff, timing.
• A runner — the orchestration that clones the fixture, creates a workspace, launches the agent, and saves everything for review.

Choices for the first iteration​

A real codebase, pinned. The fixture is a real downstream React application cloned at a fixed commit — not a toy example and not a vendored snapshot. The agent has to work inside an app with its own wallet setup, routing and conventions.

With and without. Every eval runs twice: once with the skill available, once without. The difference between the runs is the signal.

Human review first. Grading is done by a person against a rubric, not by assertions. The rubric scores seven things: path selection, auth mode, provider placement, required parameters, preservation of the existing app, code correctness, and minimality of the diff.

Assets in Git, artifacts out. The skill, eval definitions, rubric and fixture metadata are committed. Run workspaces are gitignored — they are evidence, not source.

What we learned​

• Run the eval before trusting the skill. The differences were subtle: the same integration wired two different ways, one of which would rot.
• A pinned real codebase finds problems a synthetic one cannot. It also forces the runner to be reproducible.
• Human grading is the right MVP. Automating a rubric before you know which dimensions matter freezes the wrong criteria.
• Minimality is a scored dimension. An agent that completes the task by rewriting half the app has failed.

The next steps are the obvious ones: more fixtures (an off-chain auth app), automated graders once the rubric stabilizes, and benchmark aggregation across iterations. But the order matters — skills, real fixtures, human review, and only then automation.

This is the story of an outage that started as a small networking experiment and ended with a control plane that could not maintain quorum. The lesson is about ordering: reverting a Git change does not help if the component that reads Git depends on the datastore that is already broken.

The experiment​

A three-node K3s control plane, all nodes on Wi-Fi at the time, already had a virtual IP for the API server managed by kube-vip. The next step was to give application ingress a virtual IP as well, using kube-vip's services mode. The manifest was committed as a DaemonSet that would announce per-service /32 addresses.

It did not work as intended, and the change was reverted.

The failure chain​

1. The revert commit was pushed, but Flux could not delete the DaemonSet from the cluster: etcd quorum was already broken, so the GitOps reconciliation loop had nothing to read state from and nothing to write state to.
2. Because the object still existed in etcd, the kubelet on the node recreated the pod — as kubelets do — even though the DaemonSet had been removed from Git.
3. The recreated pod announced its /32 service addresses on the wireless interface.
4. Wi-Fi is a shared medium. The access point's ARP table learned one MAC address for many IP addresses, a poisoned entry that then spread.
5. etcd peer traffic on port 2380 could no longer resolve its peers: ARP for the node addresses failed or returned the wrong destination.
6. Quorum collapsed, which kept Flux unable to remove the object, which kept the pod alive. A loop.

Diagnosis​

The clues were unusually confusing:

• The wrong primary address was showing on the wireless interface.
• An SSH session to one node would sometimes land on a different one.
• mDNS names still resolved, which made the network look healthy.
• The one reliable vantage point was a machine outside the affected segment; from there the pattern was obvious.

Recovery​

Order is everything here. The goal is to stop the thing that keeps making things worse before repairing the datastore.

1. Stop K3s on all nodes and remove the stray /32 addresses from the interfaces they were announced on.

2. On one control-plane node, run a single-node cluster reset (k3s server --cluster-reset) so the datastore can be opened again.

3. Start K3s on that node and delete the offending DaemonSet.

4. On each of the other control-plane nodes, wipe both:

   • /var/lib/rancher/k3s/server/db/etcd
   • /var/lib/rancher/k3s/server/tls/etcd

   Deleting only the datastore is not enough — the etcd certificates have to go too, or the peer handshake fails against the reset member. This detail cost the most time.

5. Restart K3s on each node and let them rejoin, then clean up the leftover static pods with crictl stop / crictl rm.

Prevention​

• Do not run kube-vip in services mode on a Wi-Fi network. Announcements that rely on gratuitous ARP only update the router; Wi-Fi peers can miss them.
• Use MetalLB or a static LoadBalancer address instead.
• Take an etcd snapshot before any kube-vip change.
• Understand that a GitOps revert is not an escape hatch: if the datastore is down, the revert cannot be applied. Recovery has to be manual first, declarative second.
• On a three-member etcd cluster, protect quorum above all else. One node can fail; the second failure is the outage.

Takeaways​

The experiment was reverted. The pod was not. The datastore held the state that kept the pod alive, and the pod was the reason the datastore could not recover. Breaking that loop meant going to the nodes and fixing etcd by hand, in the right order, before Git could take over again.

A platform with a mature API is not automatically usable by an AI agent. Agents need tools with names, schemas and boundaries they can reason about — and a way to run them without handing credentials to a stranger. This is the story of adding that layer to an existing notification platform.

The goal​

The platform already had everything a notification product needs: tenant configuration, alert subscriptions, message publishing, and SDKs for web and server environments. What it did not have was an interface an agent could operate.

The target workflows were concrete:

• an agent detects a large on-chain transaction and notifies subscribed wallets
• a liquidation warning goes out before a position is at risk
• a community manager drafts and broadcasts an announcement
• a developer asks an AI IDE to publish a test notification

Non-agent workflows still had the raw GraphQL and REST APIs. The new layer was for autonomy.

Decisions before code​

Nine decisions shaped the implementation. The important ones:

MCP and a companion skill. MCP is the runtime integration: it gives the agent executable tools. A companion skill is the guidance layer: when to use which tool, how to reason about payloads, and when to ask the user instead of guessing. Tools alone are not enough for a domain with tenant-specific data shapes.

Local-first, stdio only. The server is distributed as an npm package the user runs themselves. Credentials live in environment variables on the user's machine and never leave it. No hosted infrastructure, no SSE transport, no OAuth flow to build.

Exactly three tools. publish_message, get_active_alerts and get_tenant_config. A small surface is a feature: agents select tools more reliably from a short list, and every tool is a commitment to maintain.

Raw payloads, no universal schema. The payload for a message is defined by tenant, topic and template, so the server passes it through as an object rather than inventing an abstraction that would be wrong half the time.

No package installation. The agent may only use the predefined tools or documented direct API calls. It must not install or execute arbitrary packages — a whitelist keeps the blast radius small.

The architecture​

AI agent
   │  stdio
   ▼
local MCP server  ──HTTPS──▶  platform GraphQL + REST APIs
   │
   └── reuses the existing server-side SDK

The server is a thin wrapper over the platform's Node SDK: it reuses the GraphQL and REST clients instead of reimplementing them. Configuration comes from environment variables, the client is initialized lazily on the first tool call, and token refresh is handled transparently.

What the tools look like​

get_tenant_config returns the tenant's configuration and its events with metadata — the information an agent needs to reason about everything else.

get_active_alerts returns the subscribers currently subscribed to an event, with cursor pagination normalized into a simple page object.

publish_message takes an event id, the raw payload object and an optional wallet target list. It maps the target list onto the API's wallet-specific send path and passes the payload through unchanged.

The part that is not code​

The hardest part is payload reasoning. A tenant's topic might require a field that exists nowhere in the metadata. There is no universal function from "event" to "valid payload" — the knowledge lives in the tenant's configuration and sometimes only in the head of the person asking.

That is what the companion skill is for. It teaches the agent to inspect the configuration first, to prefer the MCP path, to fall back to documented direct API calls when MCP is unavailable, and — most importantly — to ask the user when the required shape is ambiguous.

Validation in a real agent​

The server passed its unit-level checks, but the interesting validation was end to end, inside an actual agent runtime:

• the server boots, and a missing credential produces an actionable error instead of a crash
• the agent discovers the tools and the companion skill
• configuration lookup, alert pagination and a real broadcast publish all succeed
• given an ambiguous payload, the agent inspects the configuration before assuming a shape
• when MCP is unavailable, the agent can still explain the direct API path

That last set of scenarios is where the design is really tested. A tool server is easy to demo and hard to make dependable.

What I would keep​

• Small tool surfaces. Three well-named tools beat fifteen convenient ones.
• Pass-through payloads. Abstractions over data shapes you do not control become translation layers you cannot maintain.
• Local-first credentials. It removes an entire class of security review.
• A companion skill. The agent needs judgment about the domain, not just a list of functions.

Wallet integrations age badly. One vendor-specific global can disappear in a single extension update — and when it does, the failure looks like a bug in your product.

The symptom​

Customers reported the same thing: install the new version of a wallet extension, open the dapp, click the wallet tile in the connect modal — and get redirected to the wallet vendor's homepage instead of connecting.

The modal was behaving exactly as written. It could not find the wallet, so it assumed the extension was missing and offered to install it.

The root cause​

The SDK had a dedicated integration for this wallet, built around a custom global that old versions injected into the page. Every detection path led back to that global:

const getWalletFromWindow = async () => {
  if (typeof window === 'undefined' || !window.WalletGlobal) {
    throw new Error("wallet is not installed");
  }
  // ...
};

The new wallet release moved to MPC-based key management and stopped injecting the legacy global entirely. By then the old extension was already on its way out: the vendor had stopped shipping updates for it, with store removal announced for the following month.

What the new wallet does instead is announce itself through EIP-6963, the discovery standard for injected wallets: every provider dispatches an announcement event carrying metadata, including an rdns identifier and a display name. Wallets that never touch a custom global are still discoverable — if you listen for announcements.

Wallet detection was vendor-shaped. The standard existed precisely to avoid that.

The fix​

The SDK already had a generic injected-wallet path that listens for EIP-6963 announcements and matches a wallet by substring on rdns or name:

providers.find(
  (p) =>
    p.info?.rdns?.toLowerCase().includes(walletName.toLowerCase()) ||
    p.info?.name?.toLowerCase().includes(walletName.toLowerCase()),
);

The fix was two lines of substance: move the wallet from the dedicated legacy hook to the generic injected hook, and update the install URL to point at the current wallet. The public interface did not change at all.

The cleanup that followed​

The incident exposed how much dedicated machinery existed for a single wallet:

• a legacy hook of a few hundred lines, no longer imported by anything
• a dedicated wallet class separate from the generic EVM wallet
• registry entries: a standalone category, and the wallet listed as a native integration instead of an injected one
• a special case in the wallet instance factory
• tests for all of the above

With the wallet flowing through standard discovery, none of it was necessary. The registry now treats it like any other injected EVM wallet: no special category, no special hook, no special factory branch.

While in there, the type layer was renamed to describe the chain family instead of a wallet implementation — EVM keys, Cosmos keys, Solana keys, Cardano keys. The rename has no runtime impact, but it stops the public types from baking vendor names into anything that imports them. Removing the dedicated class from the public package is a breaking change for the small set of consumers importing it directly, so it ships with the next major version and a short migration note.

What this taught us​

• Prefer standards-based discovery. EIP-6963 exists so integration code does not depend on whichever global a vendor injected this year.
• One integration per wallet does not scale. A registry entry plus a generic path covers wallets that follow the standard; dedicated code is for genuinely special cases.
• Fix the incident and the cleanup together. The "small fix" turned out to be the front door to removing hundreds of lines of abstraction. Left separate, the cleanup would probably never have happened.
• Name types after concepts, not vendors. Chain families are stable; wallet branding is not.

Home Assistant does not expose the host's CPU temperature as an entity. On a Raspberry Pi that is a useful number to have — for dashboards, but also for automations that react to a hot board.

The kernel already provides it. Thermal zones are exposed under /sys:

cat /sys/class/thermal/thermal_zone0/temp
# 47200

The value is in millidegrees Celsius, and type reports cpu-thermal.

The options​

• command_line sensor — reads the file directly. Native, no add-on, no extra service. Chosen here.
• Glances add-on — much broader system stats, at the cost of a running service and more resource use.
• System Monitor — built in, but on Home Assistant OS it does not offer CPU temperature.

Keeping configuration.yaml readable​

The larger principle: configuration.yaml should only include, not define. The real definitions live in config/integrations/, split by domain:

# configuration.yaml
command_line: !include integrations/command_line.yaml

# config/integrations/command_line.yaml
command_line:
  - sensor:
      name: CPU Temperature
      unique_id: cpu_temperature_thermal_zone0
      command: "cat /sys/class/thermal/thermal_zone0/temp"
      unit_of_measurement: "°C"
      value_template: "{{ value | float / 1000 | round(1) }}"
      scan_interval: 30
      device_class: temperature
      state_class: measurement

device_class: temperature gives the entity proper units and graph support, and state_class: measurement makes statistics and long-term history work.

Result​

sensor.cpu_temperature reports values like 47.2 °C at idle on a Raspberry Pi 4, and 30 seconds between samples is more than enough for a slow-moving thermal value.

From there it can go on a dashboard, into a ventilation automation, or into a warning when the board approaches its throttling range.

Bluetooth proxies are how Home Assistant sees BLE devices that are out of range of the host. They are small ESP32 boards running ESPHome, and most of the time they are invisible infrastructure. When one stops scanning, nothing crashes: the device stays on Wi-Fi, the API answers, the logs look calm — only the Bluetooth side goes quiet.

The symptom​

Three proxies cover the house. One of them stopped discovering devices. From Home Assistant's Bluetooth diagnostics:

• scanning: false
• discovered devices: 0
• no advertisement seen for about 11.5 hours

The Wi-Fi connection was up, the ESPHome API was responsive, and the other two proxies were fine. Nothing in the normal device state suggested a problem — which is exactly why it went unnoticed for half a day.

What did not work​

Home Assistant-side remedies only act on the integration, not on the radio:

• reloading the integration
• disabling and re-enabling the proxy
• talking to it through aioesphomeapi
• an OTA update from the command line, which timed out

The scanner only came back after a compile-and-flash from the ESPHome Dashboard over its WebSocket OTA. That pointed at the device firmware as the layer that had to recover itself.

Why a watchdog on the device​

An automation in Home Assistant could notice a dead proxy and restart it, but it depends on Home Assistant, on the network path to the proxy, and on a polling interval. It has no direct view of the scanner state, and the health logic ends up spread across YAML files.

The firmware already knows the one thing that matters: when it last saw a BLE advertisement. That makes a watchdog trivial and local.

The watchdog​

Track the last advertisement time in a global:

globals:
  - id: last_ble_adv_time
    type: uint32_t
    restore_value: no
    initial_value: "0"

esp32_ble_tracker:
  on_ble_advertise:
    - lambda: |-
        id(last_ble_adv_time) = millis();

Then check it on an interval, with a grace period so a slow boot is not mistaken for a hang:

interval:
  - interval: 2min
    then:
      - lambda: |-
          // Do not reboot while the scanner is still coming up.
          if (millis() < 3 * 60 * 1000) {
            return;
          }
          const uint32_t last = id(last_ble_adv_time);
          if (last == 0 || (millis() - last) > 10 * 60 * 1000) {
            App.safe_reboot();
          }

Ten minutes without a single advertisement is far beyond any normal quiet period, so the reboot is safe. The first occurrence happened weeks before the watchdog existed; since then the proxy recovers on its own.

Two small helpers make the behavior visible:

button:
  - platform: restart
    name: Restart

sensor:
  - platform: uptime
    name: Uptime
    update_interval: 60s

The uptime sensor shows whether the watchdog is firing at all, and the restart button allows a manual kick without opening the dashboard.

Takeaway​

Fix device health on the device. A firmware watchdog is autonomous, local, it needs no network, it lives in one YAML file, and it reacts immediately. Home Assistant automations are for logic that spans devices, not for keeping a single device alive.

The same pattern applies to other silent firmware failures: Wi-Fi that drops while the API socket stays open, sensors that start returning NaN, or a stuck output that never changes state.

Every SDK starts with a trade-off. Ours was explicit: ship the fastest possible integration path, and defer the architecture until the product had customers.

The starting point​

The first version of the SDK was a single React hooks package. One hook owned almost everything:

• API communication over REST, with hand-written request and response types
• multi-chain wallet signing adapters
• auth and token lifecycle
• loading and error state
• business logic

For a small startup this is a reasonable design. It maximizes time to market and minimizes integration friction: a partner installs one package, mounts a provider, and the hook does the rest.

Where it started to hurt​

Three problems appeared as the product grew.

Framework lock-in. All logic was bound to the React lifecycle. Anything that was not React — a Vue or Svelte integration, a script, a background worker — could not reuse the SDK at all.

Maintenance burden. Ten-plus supported chains turned the hook into a giant switch-case abstraction. Responsibilities blurred: API concerns, signing concerns and UI state lived in the same file.

Testability. Testing business logic required mounting React components. That made tests slow and brittle, and it quietly discouraged coverage.

The constraints​

The migration had to be done by one engineer, alongside normal feature delivery. Live customers were running the old architecture in production, so downtime was not an option. There was no hard deadline, but there was also no freeze: the new architecture had to coexist with the old one, and customers could adopt it at their own pace.

The options​

Split the hook into smaller hooks — useAuth, useSigning, useAlerts. Rejected: the logic stays React-bound, so the framework lock-in and testability problems remain. It only delays them.

A framework adapter over a monolithic core — invert the dependency but keep the core shaped by one framework's needs. Rejected: every supported framework becomes another maintenance surface.

A pure TypeScript client with thin framework wrappers. Chosen. The core owns the logic; React, Vue or anything else becomes a small adapter.

The extraction​

The core became a standalone TypeScript package: a client class owning the auth state machine, wallet signing adapters, the API layer and the token lifecycle. It has no React dependency and runs anywhere JavaScript runs.

Around the same time, the communication layer moved from REST to GraphQL. Hand-maintained types were replaced by code generation against the schema. That eliminated a class of schema drift bugs and reduced cross-team coordination cost — the types could no longer disagree with the API.

The migration itself ran in three phases:

1. Coexistence. The old hooks and the new client lived side by side inside the existing React package. Customers were unaffected.
2. New surface. A new React package was built entirely on the new core, replacing the old one.
3. Removal. Once the last customers had migrated, the legacy packages were removed in a major version.

The outcome​

The SDK became a platform core: one place for auth, signing and API logic, usable from any framework or no framework at all. The full migration took about three quarters with zero downtime. Adding a chain no longer means touching React code, which cuts regression risk.

What I would do differently​

• Decouple earlier. The pain was predictable; waiting for it to become acute cost more than starting the extraction would have.
• Treat a framework-agnostic core as a day-one principle, not a later refactor.
• Define a formal deprecation policy before the first breaking change.
• Run integration tests that exercise the old and new paths side by side during coexistence. Behavioural drift between them is the biggest risk in this kind of migration, and it stays invisible until a customer hits it.

Migrating a live SDK is mostly a communication problem wearing an architecture costume. Phased, independently shippable steps are what make it survivable — especially when there is only one engineer.

A small Zigbee IR blaster turns dumb appliances into things Home Assistant can control — TVs, fans, air conditioners. The integration is simple on paper:

Home Assistant → mqtt.publish → Zigbee2MQTT → IR blaster → appliance

The interesting part is the protocol. IR has no feedback and no handshake, and air conditioner remotes do not send commands — they send full state.

Codes are states, not buttons​

A TV remote sends "volume up". An air conditioner remote sends the complete state every time: power, mode, target temperature and fan speed together. There is no "make it one degree warmer"; there is only "Cool, 26 °C, fan auto".

That means one learned code per state you want to use, and automation logic that picks a state instead of pressing buttons.

Learning codes​

In the Zigbee2MQTT device page, expose and enable Learn IR code, point the physical remote at the blaster, and press the button you want to capture. The code lands in a sensor like sensor.<device>_learned_ir_code as a long base64 string — often over a thousand characters.

A practical set for an air conditioner:

• Auto
• Cool 25
• Cool 26
• Off

Sending codes​

From Home Assistant the reliable path is mqtt.publish:

action: mqtt.publish
data:
  topic: zigbee2mqtt/<friendly_name>/set
  payload: '{"ir_code_to_send": "<BASE64>"}'

Three things that cost time before they are understood:

1. The topic needs the Zigbee2MQTT friendly name, which may be different from the name Home Assistant shows for the device.
2. base_topic is needed when publishing from Home Assistant or mosquitto_pub, but not in the Zigbee2MQTT Dev Console — the console is already inside that topic namespace. Adding the prefix there produces "Entity 'zigbee2mqtt' unknown".
3. Do not use text.set_value on the exposed IR-code text entity. It is limited to 255 characters while the codes are much longer, and the send fails with a generic "Unknown error".

Automation design​

Because every code is a complete state, keep the state set small and let the automation choose:

• room temperature above 28 °C → send Cool 26
• nobody home for 30 minutes → send Off

Store the long base64 strings in secrets.yaml and reference them, so the automations stay readable and the codes can be updated in one place.

Debug checklist​

• Entity unknown → the topic or the friendly name is wrong.
• "Entity 'zigbee2mqtt' unknown" → remove the base_topic prefix in the Dev Console.
• "Unknown error" → the code is too long for the text entity; use mqtt.publish instead.

IR stays a one-way protocol, so Home Assistant never knows whether the appliance received the command. Design for the states you can send, and keep the automation logic simple enough to reason about from the codes alone.