HotLoop core release notes

HotLoop stops being one image. IoT and Edge ship for the first time, beside the Gateway and the Edge Relay, all four from one tag with their own images and charts.

• Four products, four images, four charts, one tag. hotloop-iot and hotloop-edge are new on ghcr.io and in the chart repository at https://hotloop.io/hotloop. The Gateway keeps ghcr.io/hotloop-io/hotloop and the chart hotloop, so nobody running it changes anything. A product is a build, not a setting: the OPC UA server isn't switched off in the IoT image, it isn't in it.
• MQTT discovery: point ESPHome, Zigbee2MQTT, Tasmota or a Shelly at the broker and it shows up, reading. It can't command anything until an admin accepts it and arms each point by hand. Network discovery listens and sends nothing unless you list a network to probe.
• Historian targets: every reading HotLoop keeps also goes to the plant's own PostgreSQL, TimescaleDB, SQL Server, InfluxDB or MySQL/MariaDB, the same rows with the same quality.
• The database is a setting: SQLite, PostgreSQL or TimescaleDB per product, from the same binary, so a Pi doesn't need a 300 MB database image.
• Edge keeps the plant running with the WAN dead, and its store and forward queue survives a boot during an outage, a link that dies without a reset, and a replay that has to outrun the plant.
• Every box follows :stable and updates itself, and a release that can't start is rolled back. Migrations only add, and an app refuses to start on a schema two releases ahead of it, so that rollback is always safe.
• UniFi on a wall board and a phone: a zooming network map, traces for WAN latency and port traffic, one button that makes the usual alarms, and UniFi events in the logbook under the switch they are about. An agent can ask over MCP for the topology and the console's latest events.
• The Edge Relay takes a pushed config live with no restart and no lost Sparkplug session, and discovers devices on a broker like the Gateway does.
• The license changed: all four products are under the HotLoop Business Source License 1.1. Home and lab use is free, so is use by schools, nonprofits and government, and business use goes through EmberNET.
• Fixed on the way: a discovered device's value could be lost behind its own nothing-yet reading, the embedded MQTT broker mishandled packet ids in both directions, and a dashboard that was still loading could put a person back on the one they had just left.

IoT, Edge, the Gateway and the Edge Relay all ship from this tag, in lockstep. 4.18.0 through 4.18.7 and 4.18.9 died in the Test job on real bugs, and 4.18.8 passed Test and was cancelled by mistake in the middle of publishing, so its four images were pushed as 4.18.8 and no chart and no release ever went out. 4.18.10 supersedes all of them. What isn't proven yet: the IoT and Edge Quadlet units under systemd, except IoT's SQLite unit, which was run, moving a site up a product as a helm upgrade on a real cluster, discovery against real devices on a bench, and the 32-bit arm images on real hardware.

Recipes, scripts, blueprints, helpers, the logbook and a real automation language. UniFi, read-only. Device passwords encrypted at rest.

• Recipes: grade A's setpoints go out together or not at all, checked whole first, written in order through the gate, and read back from the device. The laminated sheet of numbers can retire.
• Scripts: write the CIP cycle once and run it from a screen, a rule, MCP or its own entity, instead of pasting it into four rules and fixing a step in three of them.
• Blueprints: a rule written once with typed blanks, filled in per pump. Each rule keeps the version it was made from, so editing the blueprint on Wednesday doesn't change what pump 7 does on Thursday.
• Helpers for the values people own (batch target, shift, purge timer), a logbook with every state change, write, alarm and rule run on one timeline, and condition, wait and stop steps plus forSec, so "the press has run for ten minutes" is a trigger instead of a delay and a prayer.
• UniFi Network, read-only: switches, ports, PoE, WAN and clients as tags, and a quiet PLC says which switch port it's on. The first real read called a dead 5G backup up, because it trusted the same stale field the console's summary did. It reads the health checks now. A real cable pull and a phone joining SCADA haven't been done on real gear yet.
• Every stored device password is encrypted with a key the chart generates and keeps outside the database, so a dump or a backup holds ciphertext.
• MCP goes from 16 tools to 25. None of the new ones lets an agent make a rule or apply a recipe that names its own permission.
• Fixed: an Edge Relay ignored every config push, persistence off didn't boot, saving a rule in the editor reset its run budget, the command prompt could swallow a Set, a late update could put the Entities screen back in time, the rule editor could leave "name is required" under a rule with a perfectly good name, and App Store deploys lost their tenant labels.

Coming from 4.16.0 it's a plain helm upgrade, and the notes above are the whole list of what bites. Migrations 0009 to 0015 run on the first start, all additive. Operators get helpers.write, scripts.run, recipes.apply and blueprints.apply the moment you upgrade; writing scripts, recipes and blueprints stays admin.

The first published release since 4.3.1. Everything from 4.4.0 to 4.15.3 ships at once, under the HotLoop name, with the Edge Relay beside it.

• Entities. Every tag is an entity with an id that never changes, placed on one ISA-95 equipment tree. A motor, a valve or a PID loop is one entity built from its tags, every alarm is an entity, and every entity command goes through the same write gate as a tag write. MCP, automations and the new Entities and Plant screens all use them.
• Ask before you send. A service call takes ?dry_run=true and answers with the refusal the real call would give, word for word, and the command prompt says "Would be refused: ..." before anybody confirms.
• Paging that reaches a phone: ntfy and a Discord bot with an Acknowledge button, Gotify, and Discord webhooks, on rate limits that fold a flood into one summary instead of dropping alarms.
• Fixes that matter on a running plant. Unshelve could wipe an unacknowledged alarm. An OPC UA subscription could go silent forever while the session said connected. A silent Modbus device took up to 15 minutes to go down, and takes about 11 seconds now. A failed webhook logged its URL, which for Discord and ntfy is the password.
• Ignition 8.3.9 reads the OPC UA server, verified end to end, and the certificate survives a redeploy now, so Ignition stops distrusting it every time the pod moves.
• The Edge Relay ships as its own image and chart, released in lockstep with the Gateway. TimescaleDB 2.30.1, and the extension updates itself on start.
• Images: ghcr.io/hotloop-io/hotloop:4.16.0 and ghcr.io/hotloop-io/hotloop-edge-relay:4.16.0, for amd64 and arm64, pulled with no login. Charts hotloop and hotloop-edge-relay 4.16.0 from https://hotloop.io/hotloop, and a Quadlet unit for the Edge Relay.

Versions 4.4.0 through 4.15.3 below were merged and never published on their own. All of them are in this release. The step by step upgrade from 4.3.1 is at https://docs.hotloop.io/gateway/upgrading/.

The OPC UA server stops opening holes in its address space every time somebody edits the configuration.

• Editing a device, even just its description, pulled it and every tag under it out of the address space and put them back. A SCADA client reading an untouched tag in that moment got BadNodeIdUnknown. An edit now swaps the node in one step, so a client finds the old one or the new one and never nothing.
• A tag removed while a client was browsing its device failed the whole browse. A removed tag now answers BadNodeIdUnknown and leaves browse at once, and the node itself is deleted a few seconds later.
• One test tripped over this about once in sixty runs, and that was the only hint. A new test hammers the server with two clients while devices and tags come and go, and the old code failed it every run.
• The chart gets an edgePublish block for Sparkplug edge publish, so it no longer has to be set through extraEnv. The broker password goes through a Secret and the forward queue gets its own volume.
• TimescaleDB 2.30.1, and the Gateway now runs the extension update on start when the database is behind the image, instead of leaving an existing database on its old version for good.

Every release from this one on publishes the Edge Relay's image and chart next to the Gateway's. Before, the release only built the Gateway.

Pointed the OPC UA driver at Ignition 8.3.9, and Ignition found a bug. It's fixed, and the driver is verified against it.

• An OPC UA device with no tags sent the server nothing, so the server closed the idle session while the device kept saying connected. Every browse then failed with BadSessionIDInvalid until the process restarted.
• No tags is every device in the moment between adding it and clicking Discover tags, so this hit anyone who took longer than a session timeout to get there. A device with no tags now reads the server's state on every poll, and a browse that finds the session gone reconnects.
• Verified against Ignition 8.3.9: Basic256Sha256, SignAndEncrypt, signed in with a user name, browsed, imported, and read live by subscription.

A stock Ignition 8.3 gateway needs four changes before the Gateway can talk to it, the worst being that its OPC UA server listens on localhost only. They are written down in the Gateway's protocol docs. Still not verified against KEPServerEX, WinCC, Siemens, or Beckhoff.

Device credentials are hidden from every role, for every protocol, and scrubbed out of every log and error that could repeat them.

• Before this, any logged-in user, viewers included, could read a device's MQTT broker password, its HTTP and MTConnect credentials, and anything else saved in it through the devices API. Every credential a driver reads now comes back as [redacted], to a viewer, an operator, and an admin alike.
• That covers passwords, header values, a body that mentions a credential, and the user info or token inside an address like mqtt://user:pass@broker or ?api_key=. Any key named like a secret is hidden by default, even one no driver reads yet.
• Saving [redacted] back, or leaving the key out, keeps the stored value. A device cannot be pointed at a new host while keeping its stored credentials, because the Gateway would hand them to the new host.
• A device's last error, the system log, the write audit table, the process log, the MCP tools, the diagnostics endpoint, and site errors are all scrubbed of credentials by value and by shape.

If untrusted users could reach a Gateway running an unreleased build before this fix, rotate the device credentials it holds. There has been no public release since 4.3.1. Backups, the Edge Relay's device file, and fleet config push still carry credentials as stored, because the drivers need them.

The OPC UA driver can log in to a secured server, which is to say most of the ones in production.

• The driver used to send an empty identity token policy, which a compliant server refuses in every mode, including the Gateway's own OPC UA server. It now asks the server for its endpoints, picks the one matching the device's security mode and policy, and uses the policy id that endpoint lists.
• Basic256Sha256, Aes128_Sha256_RsaOaep, and Aes256_Sha256_RsaPss, in Sign or SignAndEncrypt. The deprecated Basic128Rsa15 and Basic256 are never picked, even when a server offers nothing else.
• The server's certificate has to be approved by an operator before anything is sent to it. An unknown certificate is saved for approval and the connection fails with a message saying exactly how.
• A password is refused before it is sent unless the channel is secured and something encrypts it. That includes a user name over an unsecured channel whose token policy claims to encrypt the password, because the key comes from a reply nothing authenticates.
• A refused login waits ten minutes before trying again, so a wrong password cannot lock an account or eat the server's sessions.

Tested against the Gateway's own OPC UA server and Microsoft's opc-plc. Aes256_Sha256_RsaPss failed against opc-plc, so the driver only uses it when a device asks for it by name. In this version the API still returned the MQTT, HTTP, and MTConnect passwords in the clear. 4.15.1 fixed that.

Scheduled reports: historian numbers, crunched on a schedule and delivered as CSV or HTML, by email if you want.

• A report names its tags, a window (the previous 24 hours, the previous calendar day, or the previous 7 days), a bucket (the whole window, per hour, or per day), and the aggregations: min, max, avg, first, last, and count. It runs on an interval, daily at a time, or weekly on a weekday at a time.
• Schedules are wall-clock times in the plant timezone and are computed as calendar dates, never by adding 24 hours, so a 03:00 report still runs at 03:00 the day after the clocks change.
• Only good-quality readings are aggregated, and every row says how many it left out. A tag with no readings in a period says no data. It never says zero.
• A run missed while the Gateway was down runs once on start-up, not once per missed slot. A report with recipients and no mail relay records a failed run instead of a quiet success.
• Viewers read reports, operators run them, and only admins define them, because a definition decides who receives plant data.

API only, and CSV and HTML only. There is no screen and no PDF in this version. Report definitions and run records are in the database backup, and the output files are not.

The Gateway can be an OPC UA server, read-only, so a third-party client can browse and read its tags.

• A read-only OPC UA server exposing the Gateway’s tags as an address space. Nodes are read-only, and there is no write handler at all, so this version has no setting that enables writes.
• Secure channels use Basic256Sha256 in Sign and SignAndEncrypt. SecurityPolicy None is an explicit opt-in.
• Users log in with their HotLoop user name and password, the same user database and bcrypt hashes as the web login, and must hold tags.read and values.read. Permissions are re-checked on live sessions, so revoking one takes effect without waiting for a reconnect.
• Anonymous sessions are an explicit opt-in, and a client’s application certificate has to be approved by an operator before it is trusted.

Writes are deliberately absent. The runtime has one write gate, and an OPC UA write would have to go through it. That deserves its own review, so it did not ride along in this release.

Database backups, taken in pure Go over the existing connection, with a restore that refuses to touch a database that already has data.

• A logical snapshot of every configuration and history table, taken inside one REPEATABLE READ transaction so it is consistent. It is table data only, since migrations rebuild the schema from nothing on every start.
• No pg_dump. The runtime image is distroless with no shell and no external binaries, and backups do not get an exception to that.
• Retention defaults to 7 backups, and 0 keeps everything. An optional interval scheduler runs backups on a plain interval, not cron.
• Restore is deliberately not an API endpoint. It is a flag an operator runs by hand against an empty, freshly migrated database, and it refuses a target that already has data.

Not verified, and stated in the backup docs: very large history volumes, where long-transaction and connection-pool behavior have not been measured, and restoring across Gateway versions that changed the schema.

An optional embedded MQTT broker, so a single-box install can be its own Sparkplug hub.

• An in-process broker with a plain TCP listener on 1883, plus a TLS listener on 8883 whenever a certificate and key are configured.
• Credential authentication, checked in constant time. Anonymous access is an explicit, off-by-default opt-in.
• Fleet management, edge publishing, and an edge relay can all point at localhost with nothing else to stand up.

There is no per-topic access control, by design. The broker is the authorization boundary, and for a production deployment at scale you should still run a dedicated broker such as Mosquitto or EMQX.

Alarm notifications by webhook and email, so a 03:00 alarm can reach a phone.

• Two provider-agnostic channels. A webhook posts JSON to any URL, which composes with PagerDuty, Opsgenie, ntfy, Pushover, Slack, Discord, or a script of your own. Email goes out over SMTP.
• Notifications fire when an alarm is raised, and only then. Cleared, acknowledged, and shelved alarms are deliberately not page-worthy.
• A priority floor per channel, so a channel set to high and above stays quiet for a low alarm.
• SMTP upgrades to STARTTLS automatically when the server offers it, which authenticated relays generally require. It is tested against a local server configured for STARTTLS and authentication, and not against a hosted provider.

Off by default. A test endpoint, restricted to administrators, lets you verify a channel without waiting for a real alarm.

The rename to HotLoop: module path, container images, charts, and environment variables.

• The Go module is now github.com/HotLoop-io/hotloop, and the images are ghcr.io/hotloop-io/hotloop and ghcr.io/hotloop-io/hotloop-edge-relay.
• Both Helm charts are renamed to hotloop and hotloop-edge-relay.
• Every IIOT_ environment variable is now HOTLOOP_.

The App Store labels and annotations under the embernet.ai namespace are unchanged on purpose. A namespaced key names the system that reads it, and those labels are how the EmberNET App Store finds a chart.

Config push: stage a new device file on a fleet relay without touching it by hand.

• The Gateway publishes a device file to a retained MQTT topic per relay, validated before it is published with the same validation the relay uses, so a malformed file never reaches a relay.
• A push and a restart are two separate operator actions on purpose. Pushing to a running relay only stages the file for the next restart, and a relay that is offline picks it up automatically when it reconnects.
• Pushing a config is administrator-only.

Fleet management: see registered edge relays, and command them.

• Membership is declared, not auto-trusted. A birth message from an unregistered relay is logged and ignored.
• Presence comes from Sparkplug birth and death messages, which are outbound from the relay and work through NAT with no inbound connection to a relay ever required.
• Rebirth is open to operators. Restart is administrator-only, since it takes a relay down.

The application cannot enforce this part, so the docs say it plainly: the broker’s own access controls should restrict publishing of node commands to the Gateway’s credentials.

A second binary, the Edge Relay, with no database at all, and two ways to deploy it.

• The Edge Relay is a separate, lighter binary that polls equipment and forwards it, with no PostgreSQL dependency. About 16.9 MB against the Gateway’s 33.4 MB.
• It exposes only /healthz and /status, read-only and unauthenticated, since there is no identity system to guard them and nothing sensitive in the response.
• Deployable on k3s with its own Helm chart, or on a box with no cluster as a Podman Quadlet unit that runs as a real systemd service.

The Gateway can be a Sparkplug B edge node, with a queue that survives a restart.

• A disk-backed queue with at-least-once delivery, oldest-first eviction when it fills, and no loss across a restart.
• Full Sparkplug B edge node behavior: birth, data, and death messages, sequence numbers, and rebirth on request.
• Buffered readings replay as historical data, one message per reading, in order.

Off by default.

A login, and writes off until someone turns them on.

• Local users with bcrypt-hashed passwords, sessions by cookie or bearer token, and three roles: admin, operator, and viewer.
• Every route checks a permission before it does anything.
• Writes are off by default. They used to be on, and a tag is now read-only until somebody marks it writable, one at a time.
• The Helm chart generates an admin account into the release Secret.