How can I implement over-the-air (OTA) firmware updates for a fleet of embedded devices?

Implementing over-the-air (OTA) firmware updates for a fleet of embedded devices requires a robust, secure, and fault-tolerant architecture spanning the device, the communication pipeline, and the cloud backend.

Here is a practical, step-by-step blueprint to build an end-to-end OTA update system.

1. The Device Architecture: Dual-Bank Bootloading

To prevent "brick-bricked" devices during a failed or interrupted update, your memory layout must support safe rollbacks.

• • A/B Partitioning (Dual-Bank Flash): Divide your non-volatile storage (flash) into two application partitions: Bank A (Active) and Bank B (Inactive/Staging).
• • The Bootloader: A small, immutable, secure bootloader runs first on power-up. It verifies the integrity and signature of the active firmware. If an update is triggered and verified, the bootloader flips the active pointer to the new bank.
• • Automatic Rollback: If the new firmware crashes repeatedly or fails to signal a "healthy boot" back to the bootloader within a watchdog timeout window, the bootloader automatically reverts to the previous working bank.

2. Packaging and Cryptographic Signing

Never push raw binary files (.bin) over the air. You need a secure payload format.

• • Delta Updates: For resource-constrained or cellular-connected devices, use tools like bsdiff to generate binary diffs, minimizing payload size and bandwidth costs.
• • Cryptographic Signatures: Package your firmware binary into a container (e.g., JSON manifest + binary, or signed image format) signed with a private key (using algorithms like ECDSA or Ed25519).
• • Root of Trust: Store the corresponding public key securely in hardware-backed storage (such as a secure element, or locked flash/OTP memory) on the microcontroller so the bootloader can verify authenticity before flashing.

3. Transport and Communication Protocols

Choose a lightweight, resilient protocol for device-to-cloud connectivity.

• • MQTT: Ideal for low-bandwidth, high-latency, or intermittent cellular/Wi-Fi links. It supports Quality of Service (QoS 1) to ensure chunk delivery.
• • HTTPS / HTTP Range Requests: Great for reliable downloading of larger chunks where you can resume interrupted downloads cleanly.
• • Chunking and Resumption: Break large firmware files into smaller blocks (e.g., 256 or 1024 bytes) with sequence numbers and checksums (like SHA-256) per block to detect corruption mid-transfer.

4. Fleet Management and Rollout Strategy

Pushing updates to 100% of your fleet simultaneously is a high-risk gamble.

• • Canary Deployments: Push the update to a 1% internal or test fleet first. Monitor error rates, telemetry, and crash logs for 24–48 hours.
• • Staged Rollouts: Automatically scale up to 10%, 50%, and then 100% of the fleet over several days.
• • Conditional Triggers: Allow devices to report their current environmental state (e.g., battery level > 50%, device idle, not actively performing a critical actuation task) before accepting or applying the downloaded update.

5. Leveraging Existing Platforms vs. Building Custom

Building an OTA pipeline from scratch is time-consuming and prone to security oversights. Evaluate whether to build or adopt an existing framework:

• • Managed Cloud Services: Use AWS IoT Jobs / FreeRTOS OTA or Azure IoT Hub Device Management for robust, scalable fleet orchestration out of the box.
• • Open-Source Frameworks: Look into enterprise-grade open-source solutions like Mender or Balena , which provide built-in A/B partition management, update servers, and robust client agents.

If you'd like, let me know:

• What microcontroller or operating system (e.g., Bare-metal, FreeRTOS, Linux) are you running?
• What is your connectivity type (Wi-Fi, Cellular NB-IoT/LTE-M, LoRaWAN)?

I can tailor the exact bootloader or cloud strategy to your stack.

Implementing OTA (over-the-air) firmware updates for an embedded fleet is less about the file transfer and more about building a safe update lifecycle: release management, device authentication, secure delivery, verification, installation, rollback, and fleet monitoring. A production OTA system should assume that devices can lose power, networks can fail, and attackers may try to install unauthorized firmware.

A typical architecture looks like this:

          Firmware CI/CD Pipeline
                    |
                    v
          Signed Firmware Artifact
                    |
                    v
          OTA Update Server / Cloud
                    |
        +-----------+-----------+
        |                       |
     Device A               Device B
        |                       |
  OTA Agent + Bootloader + Secure Storage
        |
  Verify -> Install -> Reboot -> Confirm

1. Design the device update mechanism

Your device needs three main components:

Bootloader

The bootloader is the trusted component that decides whether new firmware is allowed to run.

Responsibilities:

• Verify firmware signatures
• Select which firmware image to boot
• Recover from failed updates
• Enforce rollback rules

For many MCUs, use a proven bootloader rather than writing your own. Examples include MCUboot-based designs, which support signed images and rollback workflows.

OTA update agent

A small application task/service that:

• Checks for available updates
• Downloads firmware
• Validates metadata
• Stores the new image
• Requests a reboot into the bootloader

Persistent update metadata

Store things like:

current_version = 1.4.2
pending_version = 1.5.0
boot_attempts = 2
update_state = testing

Use nonvolatile storage with wear considerations.

---

2. Use a safe firmware layout

For reliability, avoid overwriting the currently running firmware.

Recommended: A/B partition scheme

Example:

Flash:

+----------------+
| Bootloader     |
+----------------+
| Firmware A     |  <-- running
+----------------+
| Firmware B     |  <-- update target
+----------------+
| Settings       |
+----------------+

Update flow:

1. Download firmware B
2. Verify firmware B
3. Mark B as pending
4. Reboot
5. Bootloader starts B
6. Device performs health checks
7. Mark B as confirmed

If anything fails, boot back into A.

This prevents a power loss during flashing from permanently bricking the device.

---

3. Secure the firmware supply chain

Never allow a device to install an arbitrary binary downloaded from a server.

A secure update package usually contains:

firmware.bin

manifest.json
{
  "device_family": "sensor-v2",
  "version": "1.5.0",
  "size": 842193,
  "sha256": "...",
  "minimum_bootloader": "2.0",
  "signature": "..."
}

The device should verify:

• Firmware hash matches
• Signature is valid
• Firmware is intended for this device model
• Version is allowed
• Update is authorized

Digital signatures are preferred because the device only needs the vendor public key; the signing private key remains in your release infrastructure.

Typical choices:

• ECDSA
• Ed25519
• RSA (larger footprint)

---

4. Build an OTA backend

Your server should manage:

Firmware registry

Example:

Firmware
---------
ID: fw-150
Version: 1.5.0
Hardware: sensor-v2
Status: staged
SHA256: abc123
Signature: xyz789

Device inventory

Track:

Device ID
Hardware revision
Current firmware
Last check-in
Battery level
Update status

Deployment policies

Examples:

• Update 1% of devices first
• Wait 24 hours
• Expand to 10%
• Roll out globally

This is called a staged rollout or canary deployment.

---

5. Define the update protocol

A common sequence:

Device check-in

Device -> Server

{
  id: "device123",
  firmware: "1.4.2",
  battery: 82%
}

Server response

Server -> Device

{
  update_available: true,
  url: "...",
  version: "1.5.0",
  checksum: "...",
  signature: "..."
}

Download

Use:

• HTTPS
• MQTT over TLS
• CoAP/DTLS for constrained devices

The transport protects the communication channel, but firmware signing protects the firmware itself.

---

6. Add rollback and recovery

Your device should answer:

"What happens if the update fails?" Common rollback triggers:

• Boot failure count exceeded
• Watchdog reset during startup
• Application health check fails
• Device cannot connect after update

Example:

Boot new firmware

       |
       v

Health check OK?
   /       \
 Yes        No
 |           |
Confirm     Rollback

Also consider anti-rollback protection:

Installed version: 5

Reject:
version 4
version 3

This prevents attackers from reinstalling vulnerable old firmware.

---

7. Protect signing keys

Your firmware signing key is one of your most important assets.

Good practice:

• Keep private keys off developer laptops
• Use an HSM or cloud key management service
• Require approval before production signing
• Rotate keys with a recovery plan

A compromised signing key can become a fleet-wide compromise.

---

8. Add observability

At fleet scale, you need visibility:

Track:

• Update success rate
• Failure reasons
• Download failures
• Boot failures
• Devices stuck on old versions
• Regional/network patterns

Example dashboard:

Firmware 1.5.0 rollout

Target devices: 100,000

Installed:       93,200
Pending:          5,400
Failed:           1,100
Offline:            300

---

9. Test failure scenarios

Before deployment, test:

• Power loss during download
• Power loss during flash
• Corrupt firmware image
• Invalid signature
• Wrong hardware image
• Network interruption
• Rollback behavior
• Storage exhaustion

A good OTA design treats failure recovery as a primary feature, not an exception.

---

10. Consider existing OTA frameworks

Depending on your platform:

MCU-based devices

• MCUboot
• Zephyr OTA mechanisms
• Vendor OTA frameworks

Embedded Linux devices

• RAUC
• SWUpdate
• Mender
• OSTree-based systems

Cloud-managed fleets

• AWS IoT Device Management Jobs
• Azure Device Update
• Eclipse hawkBit

---

A practical minimum production design

For a new connected product, a solid baseline would be:

• Signed firmware images
• Secure boot
• A/B firmware partitions
• OTA agent
• HTTPS or MQTT/TLS transport
• Version manifest
• Anti-rollback counter
• Automatic rollback
• Staged rollout
• Fleet telemetry

That architecture scales from hundreds to millions of devices and is aligned with common secure firmware update guidance.

If you share your hardware platform (e.g., ESP32, STM32, nRF52, embedded Linux, RTOS) and connectivity type (Wi-Fi, cellular, BLE gateway, LoRaWAN), the implementation details can be narrowed down substantially.

A robust OTA system for an embedded fleet is essentially a secure software-delivery pipeline + a fault-tolerant bootloader + fleet orchestration.

A good architecture looks like this:

Developer
   │
   ▼
CI/CD ──► Build firmware ──► Test ──► Sign artifact
                                      │
                                      ▼
                              Firmware repository
                                      │
                                      ▼
                             OTA control plane
                         ┌────────────┴────────────┐
                         │                         │
                    Device groups            Rollout policy
                         │                         │
                         ▼                         ▼
                    MQTT/HTTPS ◄────────────── Job
                         │
                         ▼
                 ┌─────────────────┐
                 │ Embedded device │
                 │ OTA agent       │
                 │ Bootloader      │
                 └────────┬────────┘
                          │
                    verify + install
                          │
                          ▼
                     Reboot safely
                          │
                  health check / commit
                          │
                          ▼
                    Report result

1. Start with a dual-slot firmware design

For most production devices, I'd strongly recommend A/B firmware partitions:

Flash
┌──────────────────────────────┐
│ Bootloader                   │  ← rarely changed
├──────────────────────────────┤
│ Slot A: firmware v1.4.2      │
├──────────────────────────────┤
│ Slot B: firmware v1.5.0      │  ← downloaded here
├──────────────────────────────┤
│ Device configuration/data    │
└──────────────────────────────┘

The device continues running from A while downloading v1.5.0 into B.

Then:

1. Download the image.
2. Verify its signature.
3. Verify its hash/size/version.
4. Mark B as the candidate image.
5. Reboot.
6. Bootloader starts B.
7. Firmware performs a health check.
8. Firmware marks B as confirmed.
9. If anything goes wrong, the bootloader automatically returns to A.

This is much safer than overwriting the currently running firmware.

For devices with constrained flash, you can use other schemes, such as streaming updates or delta updates, but A/B is generally the easiest architecture to make reliable.

2. Cryptographically sign every firmware image

Never rely on HTTPS alone.

TLS protects the download channel, but you also need to protect against someone obtaining credentials or modifying artifacts in your infrastructure.

Your release pipeline should look roughly like:

source
  ↓
compile
  ↓
unit/integration tests
  ↓
firmware.bin
  ↓
SHA-256
  ↓
digital signature
  ↓
artifact repository

The device contains the public verification key, ideally protected by hardware-backed secure storage.

The firmware manifest might contain:

{
  "product": "sensor-x",
  "version": "1.5.0",
  "hardware": "rev-b",
  "size": 1834528,
  "sha256": "...",
  "signature": "...",
  "min_bootloader": "2.1.0"
}

The device should reject the update if the signature is invalid, the hash doesn't match, the hardware isn't compatible, or the version violates your update policy.

NIST specifically recommends mechanisms that restrict updates to authorized entities and authenticate/verify update software, including mechanisms such as digital signatures and checksums.

3. Put a real OTA agent on the device

Your embedded application shouldn't have to understand the entire deployment system.

Create a small OTA agent responsible for:

• receiving update notifications
• downloading firmware
• resuming interrupted downloads
• validating the artifact
• writing to the inactive partition
• communicating with the bootloader
• rebooting
• reporting success/failure
• handling retries
• enforcing version/hardware constraints

Conceptually:

ota_check_for_update();

if (update_available()) {
    download_to_inactive_slot();

    if (!verify_signature())
        reject_update();

    if (!verify_hash())
        reject_update();

    if (!compatible_with_hardware())
        reject_update();

    mark_slot_pending();
    reboot();
}

After reboot:

if (firmware_health_check_passes()) {
    bootloader_confirm_current_image();
} else {
    bootloader_rollback();
}

AWS's current OTA documentation follows essentially this model: the device-side OTA agent receives the job, downloads the image, verifies its digital signature, installs it, and reports success/failure.

4. Separate "update available" from "firmware delivery"

Don't send a 2 MB firmware binary through your MQTT command channel unless there's a compelling reason.

A common design is:

MQTT
  │
  └── "Firmware 1.5.0 is available"
                  │
                  ▼
              HTTPS
                  │
                  ▼
             firmware.bin

MQTT is good for:

• update notifications
• job control
• status
• progress
• cancellation

HTTPS/object storage is generally better for:

• large firmware images
• resumable downloads
• CDN distribution

AWS IoT Jobs, for example, can push OTA jobs using MQTT/HTTP, and its device-management system supports targeting and monitoring fleets.

5. Build fleet-level rollout controls

This is where a prototype OTA system becomes a production OTA system.

Don't do:

Release v1.5.0
       ↓
ALL 500,000 devices

Instead:

v1.5.0
  │
  ├── internal/test devices: 20
  │
  ├── canary: 100
  │
  ├── 1% of fleet
  │
  ├── 10%
  │
  ├── 25%
  │
  ├── 50%
  │
  └── 100%

Define an automatic abort threshold.

For example:

Rollout:
  1% → 10% → 25% → 50% → 100%

Abort if:
  install_failure > 2%
  boot_failure    > 1%
  crash_rate      > 1%
  connectivity    drops significantly

AWS IoT Device Management Jobs supports rollout controls, deployment velocity, failure thresholds, scheduling, and fleet targeting.

6. Maintain device metadata

Your server should know at least:

device_id
hardware_revision
firmware_version
bootloader_version
region
customer/tenant
deployment_group
last_seen
update_status
last_update_failure

Then you can target things like:

hardware_revision == "B"
AND firmware_version < "1.5.0"
AND region == "US"

rather than blindly updating the entire fleet.

Continuous deployment systems can also automatically target devices as they enter a particular group/version state.

7. Design for unreliable networks

Assume the device will lose power or connectivity at the worst possible moment.

Your OTA protocol should support:

• interrupted downloads
• download resumption
• chunked transfers
• checksums per chunk or complete image
• retries with exponential backoff
• power loss during download
• power loss during installation
• duplicate update notifications
• device going offline for weeks
• server/device clock discrepancies

Most importantly, never put the device into an unrecoverable state because an update was interrupted.

The old firmware should remain bootable until the new firmware has passed its health check.

8. Make rollback automatic

A particularly useful state machine is:

IDLE
 │
 ▼
DOWNLOADING
 │
 ▼
DOWNLOADED
 │
 ▼
VERIFIED
 │
 ▼
PENDING_REBOOT
 │
 ▼
BOOTING_NEW
 │
 ├── health check OK ──► CONFIRMED
 │
 └── health check FAIL ─► ROLLBACK

The bootloader should have a boot-attempt counter:

pending image
boot attempt #1
boot attempt #2
boot attempt #3
        │
        └── still unhealthy
                 ↓
             rollback

This protects you from firmware that successfully downloads and authenticates but crashes immediately after boot.

9. Protect the bootloader too

The OTA security boundary should ideally be:

Root of trust
     │
     ▼
Secure bootloader
     │
     ├── verifies firmware signature
     │
     ▼
Application firmware

Otherwise an attacker who can replace the bootloader can potentially bypass your firmware verification.

Also consider:

• secure boot
• anti-rollback/version counters
• hardware-backed keys
• encrypted storage where appropriate
• debug-port locking in production
• certificate/device identity provisioning

Keep the signing private key out of the devices and preferably out of ordinary developer machines.

10. Build observability into the protocol

Every OTA attempt should generate telemetry such as:

{
  "device_id": "abc123",
  "from_version": "1.4.2",
  "to_version": "1.5.0",
  "state": "confirmed",
  "download_bytes": 1834528,
  "download_duration": 42,
  "boot_duration": 8,
  "timestamp": "..."
}

Useful fleet metrics:

• update success rate
• download failure rate
• verification failures
• boot failures
• rollback rate
• average update duration
• devices stuck in DOWNLOADING
• devices stuck on old firmware
• failure rate by hardware revision
• failure rate by geography/network
• firmware crash rate before vs. after update

That lets you detect problems such as:

"v1.5.0 works perfectly on hardware rev C but causes boot failures on rev B."

11. Treat firmware compatibility as a first-class concept

Don't assume:

v1.0 → v2.0

is always valid.

Your manifest can specify:

{
  "version": "2.0.0",
  "hardware": ["rev-b", "rev-c"],
  "min_bootloader": "1.7.0",
  "requires": {
    "config_schema": "4"
  }
}

For incompatible releases, support migration paths:

v1 → v2
v2 → v3

rather than allowing:

v1 → v3

if v3 cannot understand v1's persistent data.

12. A practical cloud architecture

For example, on AWS:

                  CI/CD
                    │
                    ▼
              Firmware signing
                    │
                    ▼
               S3 / artifact store
                    │
                    ▼
             AWS IoT Jobs / OTA
                    │
             ┌──────┴──────┐
             │             │
           MQTT          HTTPS
             │             │
             └──────┬──────┘
                    ▼
                OTA Agent
                    │
                    ▼
             Secure Bootloader
                    │
               ┌────┴────┐
               │         │
             Slot A     Slot B

AWS IoT Device Management Jobs is specifically designed for fleet-wide remote operations and tracks individual job executions, while AWS's OTA tooling supports signed firmware and HTTP/MQTT delivery.

You could implement the same architecture independently with:

• MQTT broker
• object storage + CDN
• REST API
• PostgreSQL/device registry
• message queue
• deployment service
• firmware signing service
• device-side OTA agent
• secure bootloader

The cloud vendor isn't the important part—the security and failure semantics are.

Recommended implementation order

If you're building this from scratch, I'd do it in this order:

1. A/B bootloader with rollback
2. Signed firmware + secure boot
3. Local OTA agent
4. Reliable/resumable firmware download
5. Device update-status reporting
6. Firmware artifact repository
7. OTA job/control plane
8. Canary deployments
9. Automatic rollout/abort policies
10. Fleet monitoring and alerting
11. Anti-rollback and compatibility rules
12. Production key-management/HSM integration

The key principle is: an OTA update should be allowed to fail at every stage without permanently bricking the device. Authentication prevents malicious firmware; A/B + rollback prevents bad firmware; staged deployment prevents one bad release from taking down the whole fleet.

The OTA security boundary should ideally be:

Root of trust
     │
     ▼
Secure bootloader
     │
     ├── verifies firmware signature
     │
     ▼
Application firmware

Otherwise an attacker who can replace the bootloader can potentially bypass your firmware verification.

Also consider:

• secure boot
• anti-rollback/version counters
• hardware-backed keys
• encrypted storage where appropriate
• debug-port locking in production
• certificate/device identity provisioning

Keep the signing private key out of the devices and preferably out of ordinary developer machines.

10. Build observability into the protocol

Every OTA attempt should generate telemetry such as:

{
  "device_id": "abc123",
  "from_version": "1.4.2",
  "to_version": "1.5.0",
  "state": "confirmed",
  "download_bytes": 1834528,
  "download_duration": 42,
  "boot_duration": 8,
  "timestamp": "..."
}

Useful fleet metrics:

• update success rate
• download failure rate
• verification failures
• boot failures
• rollback rate
• average update duration
• devices stuck in DOWNLOADING
• devices stuck on old firmware
• failure rate by hardware revision
• failure rate by geography/network
• firmware crash rate before vs. after update

That lets you detect problems such as:

"v1.5.0 works perfectly on hardware rev C but causes boot failures on rev B."

11. Treat firmware compatibility as a first-class concept

Don't assume:

v1.0 → v2.0

is always valid.

Your manifest can specify:

{
  "version": "2.0.0",
  "hardware": ["rev-b", "rev-c"],
  "min_bootloader": "1.7.0",
  "requires": {
    "config_schema": "4"
  }
}

For incompatible releases, support migration paths:

v1 → v2
v2 → v3

rather than allowing:

v1 → v3

if v3 cannot understand v1's persistent data.

12. A practical cloud architecture

For example, on AWS:

                  CI/CD
                    │
                    ▼
              Firmware signing
                    │
                    ▼
               S3 / artifact store
                    │
                    ▼
             AWS IoT Jobs / OTA
                    │
             ┌──────┴──────┐
             │             │
           MQTT          HTTPS
             │             │
             └──────┬──────┘
                    ▼
                OTA Agent
                    │
                    ▼
             Secure Bootloader
                    │
               ┌────┴────┐
               │         │
             Slot A     Slot B

AWS IoT Device Management Jobs is specifically designed for fleet-wide remote operations and tracks individual job executions, while AWS's OTA tooling supports signed firmware and HTTP/MQTT delivery.

You could implement the same architecture independently with:

Implementing over-the-air (OTA) firmware updates for embedded devices requires a robust, secure, and fault-tolerant architecture. A failure mid-update can "brick" a device, meaning your system must handle power losses, corrupted data, and rollbacks gracefully.

Core Architecture Components

• A Backend Server (Cloud/Local) manages firmware binaries, tracks device versions via a database, and orchestrates rollouts (canary deployments, targeted updates).
• A Communication Protocol (such as MQTT, HTTPS, or CoAP) safely transfers the payload from the server to the device.
• A Bootloader (Primary or Secondary) verifies the cryptographic signature of the downloaded image and handles flashing and switching execution to the new firmware.
• A Partition Scheme divides non-volatile storage (Flash/EEPROM) to support safe writing without overwriting the running system until verification is complete.

Step-by-Step Implementation Roadmap

1. Design the Partition Scheme (Dual-Bank A/Bing)
   • Allocate distinct storage regions: App A and App B . While App A is actively running, the bootloader downloads the new firmware image directly into App B.
   • Include a dedicated non-volatile metadata block (stored in internal flash or an external EEPROM) that tells the bootloader which partition is active, its validity status, and the boot counter/rollback flag.
2. Build a Robust Secondary Bootloader
   • Keep the bootloader small, static, and thoroughly tested because it is difficult to update remotely if it fails.
   • Upon boot, the bootloader reads the metadata block, verifies the cryptographic signature (e.g., using ECDSA or RSA-2048/4096 ) of the target application image, and switches execution if valid.
   • If the new application crashes or fails to report a "healthy" status back to the bootloader within a designated watchdog timeout window, trigger an automatic rollback to the previous working partition.
3. Implement Secure Transport & Delivery
   • Encrypt the update payload in transit using TLS/HTTPS or encrypted MQTT packets.
   • Perform an end-to-end checksum verification (such as SHA-256) on the device immediately after download completes, prior to rebooting into the update routine.
4. Establish Device Fleet Management
   • Assign unique cryptographic keys or device certificates (using a hardware secure element or MCU-based secure storage like ARM TrustZone) to authenticate each device during the handshake.
   • Implement staged rollout mechanisms so updates reach a small percentage of devices (canary phase) before broadcasting to the entire fleet.

Recommended Open-Source Tools & Frameworks

• MCUboot : A secure bootloader for 32-bit MCUs that natively supports dual-image swappable boot strategies and cryptographic signature verification.
• AWS IoT Jobs / Azure Device Update : Cloud-managed services that handle large-scale fleet orchestration, job scheduling, and secure binary delivery for connected hardware.
• BalenaOS : A container-based approach ideal for Linux-capable embedded gateways (like Raspberry Pi or Yocto-based boards) wanting robust transactional updates.

If you'd like to dive deeper, let me know:

• What microcontroller or microprocessor architecture are you using (e.g., STM32, ESP32, Linux SoC)?
• How do the devices connect to the internet (Wi-Fi, Cellular, Ethernet, BLE)?

Yes. For a fleet, I’d design OTA as a secure, transactional software-delivery system, not simply “download a .bin file and flash it.”

A robust architecture looks like:

Developer
   │
   ├─ build firmware
   ├─ test
   └─ sign release
        │
        ▼
 ┌─────────────────┐
 │ Update backend  │
 │                 │
 │ • manifest      │
 │ • artifact CDN  │
 │ • device groups │
 │ • rollout state │
 │ • telemetry     │
 └────────┬────────┘
          │ HTTPS/MQTT
          ▼
 ┌─────────────────┐
 │ Device OTA      │
 │ agent           │
 │                 │
 │ check → download│
 │ → verify → stage│
 │ → reboot        │
 └────────┬────────┘
          ▼
 ┌─────────────────┐
 │ Secure bootloader│
 │                 │
 │ verify → boot   │
 │ → health check  │
 │ → commit/rollback
 └─────────────────┘

1. Make the bootloader the final security boundary

The device should never execute an image merely because the OTA application downloaded it.

Use a hardware/root-of-trust-backed bootloader that verifies a cryptographic signature before booting the image. For example, MCUboot supports signed images and keeps the trusted public key in the bootloader.

The basic chain should be:

Firmware image
      │
      ├── SHA-256 / equivalent hash
      │
      ├── digital signature
      │
      ▼
OTA agent
      │
      ├── verify signature
      ├── verify hash
      ├── verify device/model compatibility
      └── stage image
              │
              ▼
         bootloader
              │
              ├── verify signature AGAIN
              ├── check anti-rollback counter
              └── boot

The second verification is important: the bootloader, not your application, must ultimately decide what executable code is trusted.

2. Use A/B or primary/secondary firmware slots

For most products, this is the single biggest reliability improvement.

For example:

Flash
┌──────────────────────┐
│ Bootloader           │
├──────────────────────┤
│ Slot A: running      │ ← current firmware
├──────────────────────┤
│ Slot B: downloaded   │ ← new firmware
├──────────────────────┤
│ OTA metadata         │
└──────────────────────┘

The device downloads the new firmware into the inactive slot. It doesn't overwrite the currently working firmware.

Then:

1. Verify the complete image.
2. Mark Slot B as a candidate.
3. Reboot.
4. Bootloader starts Slot B.
5. Firmware performs a health check.
6. Firmware explicitly commits itself as good.
7. If it crashes or fails to commit, bootloader returns to Slot A.

MCUboot, for example, implements test swaps specifically to allow this kind of rollback after a bad boot.

This protects against:

• power loss during update
• corrupted downloads
• interrupted network connections
• defective firmware
• boot crashes
• watchdog resets
• accidentally deploying the wrong binary

AWS similarly recommends inactive partitions/atomic updates and rollback for resilient OTA systems.

3. Separate authentication from transport security

Use TLS for the connection, but don't rely on TLS as your firmware authenticity mechanism.

TLS answers:

“Am I communicating securely with the server?”

Firmware signatures answer:

“Is this firmware authorized to execute?”

You want both.

A typical release might contain:

{
  "product": "sensor-v2",
  "hardware": "rev3",
  "version": "4.7.2",
  "security_counter": 109,
  "size": 524288,
  "sha256": "...",
  "download": "...",
  "signature": "..."
}

The device verifies that the manifest and firmware correspond to a trusted release.

For larger fleets, I'd seriously consider adopting the security model of The Update Framework (TUF) rather than inventing your own update-signing protocol. TUF is specifically designed to protect software update systems against repository/signing-key compromise and supports metadata, hashes, signatures, key delegation and expiration.

4. Implement anti-rollback protection

Rollback is useful for recovering from a bad update, but you don't want an attacker deliberately installing an old vulnerable version.

So distinguish:

Automatic rollback:
    4.7.1 → 4.7.2
              ↓ bad
    4.7.1       ← legitimate recovery

Malicious downgrade:
    4.7.2 → 3.2.0
              ↓
           REJECT

Use a monotonically increasing security counter or equivalent trusted version number.

MCUboot supports downgrade prevention using image versions/security counters.

A good rule is:

new_security_counter >= current_security_counter

with the counter stored somewhere the application cannot simply rewrite.

5. Build a real deployment service

Don't have every device blindly ask:

“What's the newest firmware?”

Instead, model deployments explicitly:

Release 4.7.2

     ↓

10-device canary
     ↓
monitor
     ↓
100 devices
     ↓
1,000 devices
     ↓
10,000 devices
     ↓
100% fleet

Your backend should maintain things like:

• device ID
• hardware revision
• current firmware
• target firmware
• update state
• last contact
• update attempt count
• failure reason
• boot/health status

For example:

AVAILABLE
   ↓
DOWNLOADING
   ↓
VERIFIED
   ↓
STAGED
   ↓
REBOOTING
   ↓
HEALTH_CHECK
   ├── success → COMMITTED
   └── failure → ROLLED_BACK

This gives you fleet-wide observability instead of discovering weeks later that 8% of devices are running broken firmware.

AWS's IoT OTA architecture is one example of this model: jobs can target individual devices or groups and deployment progress can be monitored.

6. Make updates resumable

Assume networks will disappear halfway through a 2 MB download.

Don't do:

GET firmware.bin
   ↓
download 1.8 MB
   ↓
Wi-Fi dies
   ↓
start over

Instead:

GET firmware.bin
Range: bytes=...
       ↓
chunk 1
chunk 2
chunk 3
...

Store download progress and resume after reconnecting.

For very constrained devices, consider delta updates so you transfer only the changed portions of the firmware. AWS notes this approach can substantially reduce bandwidth and energy consumption for constrained networks.

7. Design the device-side state machine carefully

I'd make the OTA agent roughly:

while (1) {
    if (!update_available())
        sleep_or_continue_normal_operation();

    manifest = fetch_manifest();

    if (!verify_manifest(manifest))
        reject_update();

    if (!compatible_with_hardware(manifest))
        reject_update();

    if (manifest.security_counter < current_counter)
        reject_update();

    download_to_inactive_slot();

    if (!verify_image_hash())
        discard_image();

    if (!verify_image_signature())
        discard_image();

    mark_image_as_pending();

    reboot();
}

Then the bootloader handles:

if (pending_image &&
    signature_valid(pending_image) &&
    version_allowed(pending_image)) {

    boot_pending_image();
}
else {
    boot_known_good_image();
}

And the new firmware must eventually do something equivalent to:

perform_startup_self_tests();

if (self_tests_pass())
    mark_update_confirmed();

If the device reboots repeatedly before confirmation, the bootloader rolls back.

8. Treat power failure as a normal event

Test this aggressively.

During every stage, randomly cut power:

download       → power loss
verification   → power loss
flash write    → power loss
metadata write → power loss
reboot         → power loss
first boot     → power loss
health check   → power loss

After every test, the device should either:

old firmware works
        OR
new firmware works

—not:

device is permanently bricked

Atomic state transitions and redundant metadata are particularly important here. AWS explicitly calls out network errors, corruption and power/reliability failures as OTA concerns.

9. Protect your signing keys

This is arguably more important than the download infrastructure.

Don't put the production firmware private key:

❌ in Git
❌ in CI environment variables
❌ on developer laptops
❌ inside the device

Prefer:

CI build
   │
   ▼
artifact
   │
   ▼
controlled signing service / HSM
   │
   ▼
signed release

And consider separate keys for:

development
staging
production

TUF is particularly useful if you need sophisticated key delegation and resilience to individual key compromise.

10. Choose your implementation based on the device

MCU + RTOS

A combination such as:

• MCUboot
• secure boot
• A/B slots
• HTTPS/MQTT
• signed firmware
• device certificate/authentication

is a good starting point.

Embedded Linux

Use a purpose-built system such as:

• RAUC
• Mender
• SWUpdate
• OSTree-based systems

rather than writing your own boot/update infrastructure from scratch.

AWS ecosystem

If your fleet already uses AWS IoT, AWS IoT Jobs + OTA facilities can provide much of the fleet deployment layer. AWS's FreeRTOS OTA workflow, for example, handles signed-image verification, version checking and deployment orchestration.

Large/high-security automotive fleet

Look at TUF/Uptane-style architecture, particularly when you have multiple ECUs, delegated authorities, strict rollback protection and a substantial supply-chain threat model. Uptane is specifically an OTA security architecture derived from TUF.

---

A practical production architecture

If I were starting a new connected MCU product today, I'd aim for:

                  ┌───────────────┐
                  │ CI/CD         │
                  │ build + test  │
                  └───────┬───────┘
                          │
                          ▼
                  ┌───────────────┐
                  │ Signing/HSM   │
                  └───────┬───────┘
                          │
                          ▼
              ┌──────────────────────┐
              │ OTA repository       │
              │                      │
              │ signed manifest      │
              │ firmware artifacts   │
              │ rollout configuration│
              └──────────┬───────────┘
                         │
                  HTTPS / MQTT
                         │
          ┌──────────────┴──────────────┐
          ▼                             ▼
     ┌─────────┐                   ┌─────────┐
     │ Device  │       ...         │ Device  │
     │ OTA     │                   │ OTA     │
     │ agent   │                   │ agent   │
     └────┬────┘                   └────┬────┘
          │                             │
     ┌────▼─────┐                  ┌────▼─────┐
     │ Slot A   │                  │ Slot A   │
     │ Slot B   │                  │ Slot B   │
     └────┬─────┘                  └────┬─────┘
          │                             │
     ┌────▼─────┐                  ┌────▼─────┐
     │ Secure   │                  │ Secure   │
     │ bootloader│                 │ bootloader│
     └──────────┘                  └──────────┘

The key design principle: make an OTA failure equivalent to “the device continues running the previous known-good firmware.” If you achieve that, then networking, servers, power interruptions and even bad releases become recoverable operational problems rather than truck-rolls.

For further implementation details, the TUF documentation and MCUboot documentation are good starting points. 

Implementing an over-the-air (OTA) firmware update system for a fleet of embedded devices requires a robust architecture divided into device-side components, backend infrastructure, and secure transport protocols.

Here is a structured breakdown of how to build and implement an OTA pipeline:

• Architecture Overview - The system consists of three main pillars: the cloud backend (manages firmware binaries, device registry, and rollout campaigns), the transport layer (securely moves data from cloud to device), and the bootloader/application architecture on the device itself.
• Dual-Bank (A/B) Memory Partitioning - Partition your non-volatile storage (Flash/EEPROM) into two application slots: Bank A (currently running firmware) and Bank B (stating area for the new firmware). This ensures that if an update fails or crashes on boot, the device can instantly roll back to Bank A.
• Secure Bootloader - Implement a secondary or primary bootloader that verifies the cryptographic signature (using algorithms like Ed25519 or ECDSA) and integrity (using SHA-256) of the new image in Bank B before jumping to it. The bootloader itself should be locked and ideally immutable.
• Delta Updates (Optional Optimization) - For constrained cellular or low-bandwidth connections, use binary differential tools (like bsdiff) to transmit only the differences between the current firmware and the target firmware, significantly reducing data consumption and transfer time.
• Transport and Communication Protocols - Use lightweight, reliable protocols such as MQTT (via TLS) or HTTPS for device communication. Implement chunking/block-wise transfers (e.g., CoAP blockwise or custom offset requests) so that interrupted downloads can resume where they left off rather than restarting from zero.
• Staged Rollout and Telemetry - Never push an update to the entire fleet at once. Use a phased rollout strategy (e.g., 1% → 10% → 100%) coupled with automated device health telemetry to catch regressions or bricking bugs early.

If you'd like, tell me:

• What microcontroller or processor architecture (e.g., ESP32, Cortex-M, Linux-based SoC) are you using?
• What is your connectivity method (Wi-Fi, Cellular, BLE, LoRaWAN)?

I can provide a more tailored architecture or code pattern for your specific hardware stack.

Implementing over-the-air (OTA) firmware updates for a fleet of embedded devices requires a robust architecture spanning the device, communication protocol, and backend cloud infrastructure.

1. Device Architecture (The Bootloader)

At the core of a reliable OTA system is a secure, fault-tolerant bootloader.

• Dual-Bank (A/B) Partitioning : Split your flash memory into two application partitions (A and B). While the device runs on bank A, the new firmware is downloaded into bank B. Once verified, the bootloader flips a flag to boot from B. If B fails to boot or crash-loops, the system rolls back to A.
• Single-Bank Partitioning : If flash space is tight, use a single application partition with an external storage chip (like an external SPI NOR flash or SD card) to stage the binary before overwriting the internal flash.
• Cryptographic Verification : Never execute unverified code. Before switching to a new image, the bootloader must verify a digital signature (using algorithms like ECDSA or RSA-256) attached to the firmware binary using a public key stored in secure hardware (like a secure element or OTP fuses).

2. Communication and Transport Protocol

Moving data efficiently and securely from the cloud to the edge.

• Transport Layer Security (TLS 1.3) : Encrypt all transit data using secure certificates, pinning them if possible to prevent man-in-the-middle attacks.
• Lightweight Protocols : Use MQTT or HTTP(S) depending on your connectivity. MQTT is ideal for low-bandwidth, high-latency cellular or Wi-Fi networks, allowing binary chunks to be sent over specific topics.
• Resumable Downloads : Network drops happen. Implement chunked or block-wise transfers (e.g., using CoAP blockwise transfers or HTTP range requests) so interrupted updates resume where they left off rather than restarting from 0%.

3. Backend and Fleet Management Infrastructure

Managing the lifecycle, targeting, and monitoring of thousands of devices.

• Device Registry & Metadata : Maintain a database tracking each device's hardware version, current firmware version, and geographical or logical group tags.
• Staged Rollouts (Canary Deployments) : Push updates to a small subset of devices first (1% → 10% → 100%) to catch unforeseen bugs before hitting the entire fleet.
• Delta Updates : For constrained cellular links, use binary diff tools (like bsdiff) to send only the bytes that changed between version N and version N+1, significantly reducing data payload sizes.

4. Safety, Telemetry, and Rollback

Ensuring devices don't become "bricks" in the field.

• Watchdog Timers : Configure a hardware watchdog timer (WDT) during the first boot of a new firmware image. If the application fails to kick or reset the watchdog within a set timeframe (e.g., 60 seconds), the system reboots into the safe fallback partition.
• Post-Update Telemetry : Force the newly updated device to report its successful boot status and operational health metrics back to the cloud within a specific window.

If you'd like to dive deeper, let me know:

• What microcontroller or operating system (FreeRTOS, Linux, bare-metal) are you using?
• What is your primary connectivity method (Wi-Fi, Cellular/LTE-M, LoRaWAN)?

I can tailor a more specific architecture or recommend tools like Mender or AWS IoT Device Management.

A robust OTA system is best designed as four cooperating layers:

1. Firmware image + bootloader — verifies and safely activates firmware.
2. Device OTA agent — downloads updates and reports status.
3. Fleet/update service — decides which devices receive which release.
4. Release pipeline — builds, signs, tests, stages, and promotes firmware.

A good reference architecture looks like:

Developer
   │
   ▼
CI/CD ──► Build ──► Test ──► Sign ──► Firmware repository
                                      │
                                      ▼
                              OTA orchestrator
                              /     |      \
                             ▼      ▼       ▼
                          Device  Device   Device
                             │
                    download firmware
                             │
                             ▼
                    ┌─────────────────┐
                    │ Bootloader      │
                    │ verify signature │
                    │ install image   │
                    │ rollback if bad │
                    └────────┬────────┘
                             ▼
                       New firmware

1. Start with a fail-safe boot architecture

For an MCU, use a secure bootloader such as MCUboot. The typical layout is:

Flash
┌─────────────────────────┐
│ Bootloader              │
├─────────────────────────┤
│ Application Slot A      │  ← currently running
├─────────────────────────┤
│ Application Slot B      │  ← downloaded update
├─────────────────────────┤
│ Persistent/config data  │
└─────────────────────────┘

The device downloads the new image into the inactive slot. The bootloader verifies it, boots it in test mode, and the new application marks itself healthy only after successful initialization. If it crashes or fails to confirm itself, the bootloader reverts to the previous image. MCUboot explicitly supports this test/confirm/revert model.

For Linux-class devices, an A/B root filesystem is the analogous approach; systems such as Mender use this model for fail-safe updates.

2. Cryptographically sign every firmware release

Don't rely on HTTPS/TLS alone.

Your update should contain something conceptually like:

Firmware image
├── magic
├── product/hardware ID
├── firmware version
├── security counter
├── image hash
└── digital signature

The device should verify:

• signature is from your trusted signing key
• image hash matches
• image is intended for this hardware/product
• image version is acceptable
• security counter isn't older than the installed security level

This protects against both tampered firmware and malicious downgrades. MCUboot supports signed images and downgrade/rollback protection mechanisms.

Keep the private signing key out of the firmware repository. Ideally, production signing happens through a protected signing service/HSM with tightly controlled CI permissions.

3. Separate "download" from "install"

The OTA agent should roughly implement:

receive update notification
        │
        ▼
check compatibility
        │
        ▼
download to inactive slot
        │
        ▼
verify hash/signature
        │
        ▼
mark image pending
        │
        ▼
reboot
        │
        ▼
bootloader validates image
        │
        ▼
boot new firmware
        │
        ▼
run health checks
        │
   ┌────┴────┐
   │         │
 healthy    failure
   │         │
confirm    reboot
   │         │
   ▼         ▼
 permanent  rollback

Make downloads resumable if devices have unreliable connectivity. Also use atomic state transitions so losing power at any point doesn't leave the device unbootable.

4. Give every device an identity

Each device should have a unique cryptographic identity, preferably provisioned during manufacturing.

For example:

device_id: 8f31...
hardware:  sensor-v3
current_fw: 2.4.1
bootloader: 1.7.0
update_channel: stable

Use mutually authenticated TLS where practical, and authorize devices individually or through tightly controlled groups.

Don't make the device's authorization depend solely on a shared fleet-wide secret.

5. Build an OTA manifest

Instead of simply telling a device "download this binary," publish a signed manifest:

{
  "product": "sensor-v3",
  "version": "2.5.0",
  "security_counter": 17,
  "size": 524288,
  "sha256": "...",
  "url": "...",
  "mandatory": false
}

The device checks the manifest before downloading and checks the firmware again afterward.

You can additionally specify:

minimum_bootloader
hardware_revision
release_channel
dependencies
release_notes

This becomes particularly important when you have multiple hardware revisions in the field.

6. Use staged fleet deployments

Never push a new firmware version to 100% of the fleet immediately.

A safer progression is:

Internal test devices
        ↓
1% canary
        ↓
5%
        ↓
25%
        ↓
50%
        ↓
100%

At every stage monitor:

• download failures
• installation failures
• boot failures
• rollback rate
• crash/reset rate
• battery impact
• connectivity
• application-specific health metrics

Automatically pause/abort the rollout when metrics exceed thresholds.

Fleet-management systems such as AWS IoT Jobs provide mechanisms for targeting device groups and tracking individual job execution; AWS specifically recommends controlled and reversible OTA deployments.

7. Choose your cloud implementation

You don't necessarily need to build the fleet-management layer yourself.

For example:

• AWS IoT Jobs can distribute and track firmware-update jobs across device groups.
• Mender provides an OTA server/client architecture, deployments, monitoring, and rollback.
• For Zephyr + MCUboot specifically, Mender also provides an MCU-oriented integration.

For example, an AWS-oriented implementation can be:

CI/CD
  │
  ├── build
  ├── unit/integration tests
  ├── sign firmware
  │
  ▼
S3 / firmware storage
  │
  ▼
AWS IoT Jobs
  │
  ├── canary group
  ├── production group
  └── rollback/abort policy
  │
  ▼
Device OTA Agent
  │
  ▼
MCUboot
  │
  ▼
A/B firmware

AWS's OTA tooling supports signed firmware, HTTP/MQTT transfer, and IoT Jobs-based deployment.

8. Design the manufacturing and key-management story early

This is often overlooked.

You need a secure process for:

Manufacturing
   │
   ├── unique device identity
   ├── device credentials
   ├── trusted firmware public key
   └── initial firmware

And separately:

Release engineering
   │
   ├── source control
   ├── reproducible build
   ├── security scanning
   ├── tests
   ├── artifact generation
   └── protected signing operation

Consider how you'll rotate signing keys if a key is compromised. Ideally, the bootloader supports a controlled trust-anchor/key-rotation mechanism rather than permanently embedding one immutable key with no recovery path.

9. Test the failure modes aggressively

OTA isn't production-ready until you've deliberately tested:

• power loss during download
• power loss during flash write
• power loss during image swap
• corrupted firmware
• invalid signature
• wrong hardware revision
• firmware downgrade
• boot loop
• application crash immediately after boot
• network disappearing halfway through download
• server becoming unavailable
• insufficient flash/storage
• insufficient battery
• interrupted rollout
• signing-key compromise/revocation scenario

The most important property is:

A failed OTA operation must leave the device either running the old known-good firmware or the new known-good firmware—not neither.

A practical starting stack

If you're building a new MCU fleet, I'd strongly consider:

MCU + Zephyr → MCUboot → A/B firmware slots → HTTPS/MQTT OTA agent → signed firmware → cloud fleet orchestrator → staged deployments + telemetry.

For a Linux embedded device:

Linux → A/B root filesystem → Mender or RAUC/OSTree-style updater → signed artifacts → cloud deployment service → staged rollout.

The exact choice depends heavily on your hardware, RTOS/OS, flash/RAM constraints, connectivity, and fleet size. If you tell me the MCU/SoC (e.g. ESP32, STM32, nRF52/nRF91), RTOS/Linux, connectivity (Wi-Fi/LTE/BLE), and approximate fleet size, I can lay out a concrete OTA architecture and implementation path for it.

Implementing over-the-air (OTA) firmware updates for a fleet of embedded devices requires a robust, secure, and fault-tolerant architecture divided into device-side and cloud-side components.

1. Architectural Components

• Cloud Backend: Manages firmware binaries, metadata (version, target hardware, checksums), deployment campaigns, and device registry.
• Device Client: Handles connectivity, checking for updates, downloading the binary, verifying integrity, and triggering the update process.
• Update Agent / Bootloader: Safely applies the update to flash memory without bricking the device.

2. Core Implementation Strategies

• Dual-Bank (A/B) Partitioning:
  • The flash memory is split into two application slots: Slot A (active) and Slot B (inactive).
  • The bootloader runs the active slot. The OTA client downloads the new image into the inactive slot.
  • Upon verification, the bootloader flips the active pointer on the next reboot. If the new firmware fails health checks, it rolls back to the previous stable slot.
• Single-Bank / External Flash:
  • Used when memory is constrained. The image is downloaded into an external flash chip or a designated safe region of internal flash, then flashed over the running image (risier, requires a robust recovery bootloader).

3. Step-by-Step Implementation Flow

4. Build and Sign Firmware:

   • Compile the binary and generate a cryptographic signature using a private key (e.g., using ECDSA or RSA).

5. Publish to Cloud:

   • Upload the binary and signature to secure cloud storage (e.g., AWS S3, Azure Blob) and register the version metadata.

6. Trigger Notification / Poll:

   • The device periodically polls the server via MQTT/HTTPS, or the cloud pushes a notification via an IoT broker (e.g., AWS IoT Core, HiveMQ).

7. Secure Download:

   • Download the binary in chunks over a secure transport layer (TLS 1.3) and stream it directly into the inactive flash bank.

8. Cryptographic Verification:

   • Before committing, verify the binary's cryptographic signature and hash (e.g., SHA-256) using the public key hardcoded in the immutable bootloader.

9. Atomic Swap and Reboot:

   • Set the bootloader flag to boot from the new partition and trigger a hardware reset.

10. Post-Update Health Check:

    • The new firmware runs a self-test (checking sensors, cloud connectivity) for a set duration before sending a "commit" signal back to the cloud. Otherwise, it triggers an automatic rollback.

11. Essential Security & Fleet Management Practices

• Cryptographic Signing: Never trust the transport layer alone; always sign the firmware binary itself.
• Rollback Protection: Prevent downgrade attacks by storing the minimum allowed firmware version in One-Time Programmable (OTP) memory or secure storage.
• Staged Rollouts: Deploy updates to a canary group (e.g., 1% of the fleet) before rolling out to the remaining 99% to catch unforeseen bugs.
• Delta Updates: For bandwidth-constrained cellular fleets, use binary diff tools (like bsdiff) to send only the modified byte blocks rather than the entire image.

To help tailor this architecture, tell me:

• What microcontroller or processor family (e.g., ESP32, STM32, Cortex-A Linux) are you using?
• How are the devices connected (Wi-Fi, Cellular/NB-IoT, LoRaWAN, Ethernet)?
• Are you building a custom cloud backend or looking to use an existing IoT platform (like AWS IoT or Azure IoT Hub)?

For a fleet, treat OTA as a secure deployment system, not merely “download a binary and flash it.” A robust design has four layers:

                    Cloud / OTA service
                           │
             ┌─────────────┴─────────────┐
             │                           │
       Firmware registry            Deployment manager
       - signed images              - cohorts
       - versions                   - rollout %
       - metadata                   - health gates
             │                           │
             └─────────────┬─────────────┘
                           │ TLS
                           ▼
                    ┌──────────────┐
                    │ OTA Agent    │
                    │ on device    │
                    └──────┬───────┘
                           │
                 download / verify
                           │
                    ┌──────▼──────┐
                    │ Inactive    │
                    │ firmware    │
                    │ slot (B)    │
                    └──────┬──────┘
                           │
                     bootloader
                  verify → boot B
                           │
                    health check
                    ┌──────┴──────┐
                    │             │
                  success       failure
                    │             │
                 confirm       rollback
                    │             │
                    ▼             ▼
                  slot B       slot A

1. Put a reliable bootloader underneath the application

For an MCU, use an A/B firmware layout:

• Slot A = currently working firmware
• Slot B = candidate firmware
• Download the new image into B
• Cryptographically verify B
• Reboot into B
• Have the new firmware perform a self-test
• Only then mark B as confirmed
• If it fails to boot or doesn't confirm, automatically return to A

This is the most important protection against a device becoming permanently bricked. MCUboot, for example, explicitly supports test boots and reverting to the previous image when the new firmware isn't confirmed.

You'll need enough nonvolatile storage for two images, or an equivalent scheme such as a temporary external-flash staging area.

2. Cryptographically sign every firmware image

Don't rely on TLS alone.

Your release pipeline should look approximately like:

source
  ↓
build reproducibly
  ↓
firmware.bin
  ↓
generate manifest
  ↓
sign image with release/private signing key
  ↓
publish image + metadata

The device bootloader should verify the signature using a public key anchored in trusted storage before allowing the image to execute.

The manifest might contain:

{
  "product": "sensor-v2",
  "version": "4.7.2",
  "hardware_revision": "B",
  "image_size": 524288,
  "sha256": "...",
  "security_counter": 17,
  "min_bootloader": "2.1.0"
}

The hash protects integrity; the digital signature establishes that the firmware came from an authorized signer.

Also implement anti-rollback protection. A compromised server shouldn't be able to convince a device to install an old vulnerable image. MCUboot supports security counters specifically for this purpose.

3. Separate transport security from firmware authenticity

Use TLS for the device ↔ server connection, ideally with per-device credentials/certificates.

But think of the security boundaries separately:

• TLS: protects the download channel.
• Firmware signature: protects the firmware itself.
• Secure boot: prevents execution of unauthorized firmware.
• Anti-rollback counter: prevents reverting to known-vulnerable versions.
• Device identity/authorization: determines which device is allowed to receive which update.

This means an attacker who compromises your CDN or update server still can't simply replace a firmware binary with an unsigned one.

4. Have an OTA agent, but keep the bootloader independent

The application/OTA agent should handle:

1. Check for an update.
2. Determine whether the update applies to this hardware.
3. Download it in chunks.
4. Resume interrupted downloads.
5. Verify size/hash/signature.
6. Write it to the inactive slot.
7. Mark the slot as pending.
8. Reboot.

The bootloader should handle the security-critical decision of whether that image is allowed to execute.

Don't make the application responsible for deciding whether an untrusted image is safe to boot.

5. Design for bad connectivity and power loss

Assume all of these will happen:

• device loses Wi-Fi during download
• cellular connection disappears
• power is removed halfway through a flash write
• device resets during installation
• download is corrupted
• device has insufficient storage
• new firmware crashes immediately
• new firmware boots but can't connect to the network
• fleet update server becomes unavailable

Your update state machine should therefore be something like:

IDLE
 │
 ├── update available
 ▼
DOWNLOADING
 │
 ├── interrupted ──→ resume
 │
 ▼
DOWNLOADED
 │
 ├── hash/signature failure ──→ discard
 │
 ▼
VERIFIED
 │
 ▼
PENDING_BOOT
 │
 ▼
REBOOT
 │
 ▼
SELF_TEST
 │
 ├── failure ──→ ROLLBACK
 │
 ▼
CONFIRM
 │
 ▼
RUNNING

Never overwrite your only known-good firmware image unless you have another reliable recovery mechanism.

6. Make the health check meaningful

“Firmware booted” isn't sufficient.

After booting the candidate image, give it a confirmation window—for example:

boot candidate
    ↓
initialize hardware
    ↓
run integrity/self-tests
    ↓
start essential services
    ↓
establish cloud connection
    ↓
send "healthy" heartbeat
    ↓
confirm image

If any critical condition fails, reboot without confirming the image. The bootloader can then revert to the previous image. This is exactly the sort of rollback capability recommended for production IoT deployments.

7. Build fleet management around cohorts

Don't send version 4.7.2 to 100,000 devices simultaneously.

Use staged deployment:

4.7.2 release
     │
     ├── internal test devices
     │
     ├── 1% canary
     │
     ├── 5%
     │
     ├── 20%
     │
     ├── 50%
     │
     └── 100%

At every stage monitor:

• update success rate
• boot failures
• rollback rate
• connectivity after update
• crashes/reboots
• battery consumption
• application-specific health metrics
• geographic/hardware-version differences

Automatically stop the rollout if those metrics cross predefined thresholds.

Fleet OTA platforms such as AWS IoT Device Management provide concepts such as deployment groups, rollout velocity, failure thresholds, and continuous jobs for this purpose.

8. Version more than the firmware

Keep track of at least:

device_id
hardware_revision
bootloader_version
firmware_version
configuration_version
security_counter
update_status
last_update_attempt
last_successful_update
rollback_count

This becomes extremely useful when, for example, you discover that only hardware revision C devices are failing to update.

9. Decide whether you need delta updates

Initially, full-image updates are much simpler and safer.

If your firmware is 2 MB and you have inexpensive connectivity, start there.

Once bandwidth becomes a significant cost, consider binary deltas:

firmware 4.7.1
       +
    delta
       ↓
firmware 4.7.2

But make sure the resulting image is still cryptographically verified as a complete image before execution. Don't let delta-update complexity undermine your recovery model.

10. Protect the signing infrastructure

This is frequently more important than the OTA protocol itself.

Ideally:

Developer
   │
   ▼
CI build
   │
   ▼
artifact repository
   │
   ▼
controlled release approval
   │
   ▼
hardware-backed signing key
   │
   ▼
signed firmware
   │
   ▼
OTA service

Don't put your production firmware signing private key in a developer laptop or ordinary CI secret.

Consider key rotation, multiple signing keys, revocation/recovery procedures, and a secure bootloader key hierarchy.

A practical stack

For an MCU-based product, a good starting architecture would be:

MCUboot is particularly useful because it is OS-independent and supports several embedded ecosystems.

If you're using embedded Linux rather than an MCU, the same principles apply, but I'd generally use an A/B root filesystem/update framework rather than implementing the flash-management logic yourself.

The key design principle is:

At every point during an OTA operation, the device must have a deterministic path back to known-good firmware.

If you tell me your MCU/SoC (e.g. STM32, ESP32, nRF52/nRF53, NXP, Raspberry Pi), connectivity (Wi-Fi/cellular/BLE), and approximate fleet size, I can map this into a concrete architecture and update state machine for your hardware.