ESP32 uplink watchdog reset: recovering the lost sample window

A condition-monitoring node on a SAG mill pinion gearbox samples vibration and temperature at 1 Hz, buffers locally, and publishes to an MQTT broker over Wi-Fi. The publish path is asynchronous: the application task hands a payload to the network stack, then blocks on a queue. If the broker, access point, or DHCP lease stalls, the network task stops draining that queue. The application task keeps sampling. Eventually the queue fills, the application task blocks, and the idle task — which feeds the hardware watchdog — is starved. The watchdog fires, the ESP32 resets, and every sample held in RAM since the last successful publish is gone. That is the failure mode under examination: an uplink-induced watchdog reset that destroys the last sample window.

The mechanism, step by step

The ESP32 task watchdog timer (TWDT) is fed by tasks that must yield. A common configuration subscribes the idle task and the main application task to the TWDT with a 5-second timeout. Under normal operation the application task blocks on `xQueueSend` with a finite timeout, returns to the scheduler, and the idle task runs. When the uplink degrades, three things compound:

1. The transport task blocks in a socket write or TLS handshake with no application-level timeout. 2. The outbound queue fills; `xQueueSend` with `portMAX_DELAY` never returns. 3. The idle task never runs, the TWDT expires, and the reset is logged as `Task watchdog got triggered`.

The reset itself is not the data-loss event. The data loss is that the ring buffer and the unsent queue live in volatile memory. A reset clears them. On a gearbox trending toward a bearing defect, the samples in that window may be the ones that show the incipient sideband. The node reboots in 300–800 ms, reconnects, and resumes publishing — with a hole where the event was.

Why conventional alarms miss it

A broker-side gap detector sees a timestamp discontinuity and raises “no data.” That alarm is indistinguishable from a planned outage, a Wi-Fi roam, or a power cycle. It tells the reliability engineer that data is missing; it does not tell them the node reset because the uplink stalled, nor how many samples were lost, nor whether the loss coincided with a process transient.

The diagnostic distinction matters. A node that drops a publish because the broker was briefly unavailable is a communications problem. A node that resets because a blocked publish starved the watchdog is a firmware architecture problem, and it will recur under the same uplink conditions. Without a reset-reason register read at boot, the two are conflated.

IEC 61511 framing: diagnostic coverage and proof-test

Treat the node as a non-safety-related data source that feeds a safety-related decision, and apply IEC 61511-1 clause 11 diagnostic-coverage discipline to the *data path*, not to the sensor. The sensor is not a safety device and carries no SIL rating. What the standard’s methodology gives us is a structured way to ask: what fraction of dangerous failures does our diagnostic detect, and how do we prove it still works?

For this failure mode, define the dangerous failure as “loss of the last sample window without an attributable, distinguishable record.” Two diagnostics address it:

• **Reset-reason capture.** On boot, read `esp_reset_reason()`. `ESP_RST_TASK_WDT` and `ESP_RST_INT_WDT` are distinguishable from `ESP_RST_POWERON` and `ESP_RST_SW`. Persist the reason, the TWDT task name, and a monotonic boot counter to NVS before the first publish. This converts an anonymous gap into a classified event.
• **Pre-reset breadcrumb.** Write a lightweight heartbeat — last sequence number published, queue depth, uptime — to NVS or RTC memory on a fixed cadence, independent of the uplink. After a reset, the difference between the last breadcrumb and the last broker-received sequence bounds the lost window.

Diagnostic coverage is then an estimate, not a claim: if the breadcrumb cadence is 1 s and the sampling rate is 1 Hz, the unaccounted window is bounded to roughly one sample plus reset latency. That is a design target you can state and test, not a certification.

Proof-testing follows IEC 61511-1 clause 16 logic. At a defined interval — monthly is defensible for a monitored node — inject the failure deliberately. Block the broker at the firewall, or point the node at a black-hole address, and confirm that (a) the TWDT fires, (b) the reset reason is logged as a task watchdog, and (c) the breadcrumb bounds the loss to the design target. Record the result. A proof-test that only confirms the node reboots is not a proof-test; it must confirm the *diagnostic* fired and the *loss bound* held.

Design changes that reduce the window

The recovery path is architectural, not a patch:

• Replace `portMAX_DELAY` sends with a bounded timeout and an explicit drop-and-count policy, so a stalled uplink degrades to sample loss rather than a reset.
• Move the outbound queue to a fixed-size ring buffer with overwrite-oldest semantics, and size it to the longest credible uplink outage.
• Feed the TWDT from the sampling task only if that task genuinely yields; otherwise subscribe a dedicated low-priority task that cannot be blocked by the transport.
• Persist the ring buffer to a wear-levelled partition or FRAM if the asset class justifies the cost. For a SAG mill gearbox, it usually does; for a non-critical conveyor idler, it usually does not.

Getting started

Instrument the reset reason first — it is the cheapest diagnostic and the one that turns an unexplained gap into a classified event. Then set a breadcrumb cadence and a bounded send timeout, and write a proof-test procedure that blocks the uplink on purpose and checks the recorded loss against the design target. The live-verified context for this work sits alongside broader Australian processing developments, such as Renascor hitting battery-grade purity at its Adelaide plant (Australian Mining, 2026-10-09, source: live-verified), where the same discipline — attributable records and tested diagnostics — governs how condition data is trusted.

Grounded in a verified live source: Renascor hits battery-grade purity at Adelaide plant (2026-10-09) — source: live-verified.

Talk to S5 System about your site

S5 System designs and manufactures ultra-low-power industrial IoT sensors for mining, agriculture, infrastructure and utilities.

Discuss your IoT monitoring project — book a call.