OTA upgrade to Sming 4.3

[mikee47]

I've opened this issue to discuss how OTA (over-the-air) upgrading of existing ESP8266 devices might work when migrating to the new partition-table structure.

With partition tables the ESP8266 layout has necessarily been altered. See https://sming.readthedocs.io/en/latest/upgrading/4.2-4.3.html.

Re-flashing a device to 4.3 should be as simple as:

make flashinit
make flash

I've used the Microsoft (TM) Notepad+Pencil App to illustrate some scenarios. On the left are the flash sector numbers (easier than writing addresses).

(1) This is a typical layout for Sming 4.2 devices:

(2) This is the new default layout for Sming 4.3:

The closest we can to the previous layout is shown in (3). There is a further development that could be made to the framework by allowing the partition table to share the boot sector, shown in (4).

[mikee47]

Continuing from above, here are the require steps to perform an OTA upgrade:

1. Ensure the system is booting and running from the APP 2 partition
2. Flash sectors 2 - 7 (the partition table plus configuration sectors)
3. Flash the new Sming 4.3 application image to sector 8 onwards, APP 1
4. Perform a trial reboot of APP . If this fails, the next reboot would revert to APP 2 which, being Sming 4.2, is ignorant of partition tables so will work as before.
5. When stable, set APP 1 as default boot application. The next OTA update would then flash a Sming 4.3 image to APP1, etc.

If anyone has attempted this, please post and we'll see if there are framework changes that can be made to simplify/ease the process. At the very least, we should ensure the documentation covers this process.

[slaff]

For now:

I would suggest the following layout:

• Sector 1: rBoot bootloader.
• Sector 2: starts with the rBoot config, as before, and adds the Partition Tables (PT) in this same sector.
• Sector 3: Application starts here

Such a setup will make it possible to do remote upgrade OTA with minimal changes and risk.

Old apps will continue to work as expected.
For the new apps to work we will need a component which does the appropriate adjustments. The component itself will do very little. It will check if there is a partition table in sector 2 at offset X, X to be determined. if not - copies the current rBoot config and adds the new partition table in sector 2 with offset X. This is the most critical operation as there can be power failure after the second sector is cleared and before the new content is written. After powering on rBoot will generate a standard rBoot config which should work for most of the applications. So such a risk should be limited to small amount of devices.
This component can be enabled by default for all ESP8266 applications. And it should be possible to disable it, if request.
The execution of the component should happen before init().

For the future

rBoot and PT contain certain information that can be found in both. It will be better:

• To make rBoot to work with the partition table directly Add rBoot partition support #2258.
• Decrease the rboot config by removing information that is already part of the partition table and store everything in the second sector. And the application rom should be starting from the 3rd sector. In such way the application rom can use most of the available memory from the maximum allowed 1MB limit.

[mikee47]

This is the most critical operation as there can be power failure after the second sector is cleared and before the new content is written.

Actually, we wouldn't need to erase the sector first as the area should already be blank - so little risk attached.
BUT subsequent OTA updates wishing to switch ROMS would need to clear the sector so it then becomes an issue.

This component can be enabled by default for all ESP8266 applications. And it should be possible to disable it, if request.
The execution of the component should happen before init().

Keeping the rBoot config and partition table in the same sector is bad as subsequent OTA updates risk losing both rBoot config AND PT.

We could keep the PT in the boot sector. Offset 0x0C00, for example, allows 32 partitions with plenty of space for the boot code.
This would avoid the risk of erasing the partition table sector when performing subsequent OTA updates.

Aside: Generally, the partition table should be fixed at deployment and updating it via OTA should rarely be required. However, a safer way to to this might involve reserving two sectors for the partition table, using the second as a backup. OTA updating the boot sector should be avoided if at all possible!

Proposed solution

Losing 4Kbytes of program space from 1MB is a non-issue. Keeping a compatible layout to support OTA updating is valuable though.

It is currently a requirement that the partition table is located immediately after the boot sector.
Moving all the configuration partitions to the start of flash is, IMHO, preferable for the best user experience,
so is the most sensible default setup.

However, relaxing this restriction for the ESP8266 (and Host) would eliminate the above complications.
Therefore, I propose the following changes:

1. For the ESP8266 (and host), allow the partition table to be freely relocatable (after rBoot config).
2. PT must be in its own sector; do not write anything else in there!
3. PT must start at offset 0 i.e. sector boundary, address 0xnnnnn000. (Keeps things simple, also with a mind to how we might perform OTA PT updates).
4. Add some legacy configurations, e.g. legacy-1m, legacy-4m.
   These can be used to create a 4.2-compatible layout.
   The flash size is important of course because it determines where the configuration sectors are.

The OTA update procedure is:

1. Create a custom hardware configuration for Sming 4.3 application based on a legacy configuration.
   e.g. "base_config": "legacy-4m".
2. Verify that the default location for the partition table is acceptable.
   For example, a 4M device could locate this at offset 0x00100000 - the 'gap' between ROM0 and ROM1.
3. Update the configuration with any changes from default.
4. Add a step for the OTA update to write the sector containing the partition table BEFORE the new
   application image.
5. Test! If the OTA update is successful, great. If not, the previous image will still work.