Create simple ArduinoBleOTA library

[vovagorodok]

Thanks for Your fullstack solution (from esp32 to android app)! I have forked this repository in order to create PlatformIO Arduino library:
https://github.com/vovagorodok/ArduinoBleOTA

At the and will be nice to move it to: https://github.com/arduino-libraries

I'm trying to make it as simple as possible with reusing of NimBLE-Arduino and ArduinoOTA(support only WIFI) libraries. Proposed interface will be similar to ArduinoOTA:

#include <ArduinoBleOTA.h>
void setup() {
  ArduinoBleOTA.begin("ArduinoBleOTA", InternalStorage);  // or SDStorage or SerialFlashStorage
}

Or if bluetooth is used not only for OTA:

#include <ArduinoBleOTA.h>
void setup() {
    BLEDevice::init("MyBleDevice");
    auto* server = BLEDevice::createServer();
    ArduinoBleOTA.begin(server, InternalStorage);
    // add your BLE services here
    server->startAdvertising();
}

Additionally I have plan to add support of Atmel based boards using ArduinoBLE library.

I see that @Vincent-Stragier begin with that. Lets connect our forces and create easy library with simple/lightweight protocol.

Draft protocol proposal will have small number of commands (names and numbers can be another like 0x01 or 0xFE):

SIZE 1
PACKAGE 2
INSTALL 3
OK 4
NOK 5

Designations for examples:
-> - recived from phone
<- - send to phone

Sunny day scenario:

-> SIZE <size>
<- OK % open ota storage
-> PACKAGE <data>
<- OK
...
-> PACKAGE <data>
<- OK
-> INSTALL % full <data> size == <size>, apply binary
<- OK

Size to big:

-> SIZE <size>
<- NOK % do nothing

Connection terminated:

-> SIZE <size>
<- OK
-> PACKAGE <data>
<- OK
...
% termination
-> SIZE <size>
<- OK % reopen ota storage and write from scratch
...

Unexpected package:

...
-> PACKAGE <data> % full <data> size > <size>, close ota storage
<- NOK

Instalation error:

...
-> INSTALL % full <data> size < <size>
<- NOK

I thing that most/all corner cases are covered or something is missed?
@fbiego can you describe scenarios(corner cases) that are currently covered with command names (because from code I see some magic numbers 0xFB, 0xFC, 0xFD, FE ..)?
What do you think generally about this idea?

Thanks!

[Vincent-Stragier]

@vovagorodok,

If you take a look at:

ESP32_BLE_OTA_Arduino/esp32_ble_ota_lib_compact/ota_updater.py

Lines 26 to 175 in fa90638

	async def start_ota(ble_address: str, filename: str):
	device = await BleakScanner.find_device_by_address(ble_address, timeout=20.0)
	disconnected_event = asyncio.Event()
	total = 0
	file_content = None
	client = None
	def handle_disconnect(_: BleakClient):
	print("Device disconnected !")
	disconnected_event.set()
	sleep(1)
	# sys.exit(0)
	async def handle_rx(_: int, data: bytearray):
	# print(f'\nReceived: {data = }\n')
	match data[0]:
	case 0xAA:
	print("Starting transfer, mode:", data[1])
	print_progress_bar(0, total, prefix='Upload progress:',
	suffix='Complete', length=50)
	match data[1]:
	case 0: # Slow mode
	# Send first part
	await send_part(0, file_content, client)
	case 1: # Fast mode
	for index in range(file_parts):
	await send_part(index, file_content, client)
	print_progress_bar(index + 1, total,
	prefix='Upload progress:',
	suffix='Complete', length=50)
	case 0xF1: # Send next part and update progress bar
	next_part_to_send = int.from_bytes(
	data[2:3], byteorder='little')
	# print("Next part:", next_part_to_send, "\n")
	await send_part(next_part_to_send, file_content, client)
	print_progress_bar(next_part_to_send + 1, total,
	prefix='Upload progress:',
	suffix='Complete', length=50)
	case 0xF2: # Install firmware
	# ins = 'Installing firmware'
	# print("Installing firmware")
	pass
	case 0x0F:
	print("OTA result: ", str(data[1:], 'utf-8'))
	global ble_ota_dfu_end
	ble_ota_dfu_end = True
	def print_progress_bar(iteration: int, total: int, prefix: str = '', suffix: str = '', decimals: int = 1, length: int = 100, filler: str = '█', print_end: str = "\r"):
	"""
	Call in a loop to create terminal progress bar
	@params:
	iteration - Required : current iteration (Int)
	total - Required : total iterations (Int)
	prefix - Optional : prefix string (Str)
	suffix - Optional : suffix string (Str)
	decimals - Optional : positive number of decimals in percent complete (Int)
	length - Optional : character length of bar (Int)
	filler - Optional : bar fill character (Str)
	print_end - Optional : end character (e.g. "\r", "\r\n") (Str)
	"""
	percent = ("{0:." + str(decimals) + "f}").format(100 *
	(iteration / float(total)))
	filled_length = (length * iteration) // total
	bar = filler * filled_length + '-' * (length - filled_length)
	print(f'\r{prefix} |{bar}| {percent} % {suffix}', end=print_end)
	# Print new line upon complete
	if iteration == total:
	print()
	async def send_part(position: int, data: bytearray, client: BleakClient):
	start = position * PART
	end = (position + 1) * PART
	# print(locals())
	if len(data) < end:
	end = len(data)
	data_length = end - start
	parts = data_length // MTU
	for part_index in range(parts):
	to_be_sent = bytearray([0xFB, part_index])
	for mtu_index in range(MTU):
	to_be_sent.append(
	data[(position*PART)+(MTU * part_index) + mtu_index])
	await send_data(client, to_be_sent)
	if data_length % MTU:
	remaining = data_length % MTU
	to_be_sent = bytearray([0xFB, parts])
	for index in range(remaining):
	to_be_sent.append(
	data[(position*PART)+(MTU * parts) + index])
	await send_data(client, to_be_sent)
	await send_data(client, bytearray([0xFC, data_length//256, data_length %
	256, position//256, position % 256]), True)
	async def send_data(client: BleakClient, data: bytearray, response: bool = False):
	# print(f'{locals()["data"]}')
	await client.write_gatt_char(UART_RX_CHAR_UUID, data, response)
	if not device:
	print("-----------Failed--------------")
	print(f"Device with address {ble_address} could not be found.")
	return
	#raise BleakError(f"A device with address {ble_address} could not be found.")
	async with BleakClient(device, disconnected_callback=handle_disconnect) as local_client:
	client = local_client
	# Load file
	print("Reading from: ", filename)
	file_content = open(filename, "rb").read()
	file_parts = -(len(file_content) // -PART)
	total = file_parts
	file_length = len(file_content)
	print(f'File size: {len(file_content)}')
	# Set the UUID of the service you want to connect to and the callback
	await client.start_notify(UART_TX_CHAR_UUID, handle_rx)
	await asyncio.sleep(1.0)
	# Send file length
	await send_data(client, bytearray([0xFE,
	file_length >> 24 & 0xFF,
	file_length >> 16 & 0xFF,
	file_length >> 8 & 0xFF,
	file_length & 0xFF]))
	# Send number of parts and MTU value
	await send_data(client, bytearray([0xFF,
	file_parts//256,
	file_parts % 256,
	MTU // 256,
	MTU % 256]))
	# Remove previous update and receive transfer mode (start the update)
	await send_data(client, bytearray([0xFD]))
	# Wait til the update is complete
	while not ble_ota_dfu_end:
	await asyncio.sleep(1.0)
	print("Waiting for disconnect... ", end="")
	await disconnected_event.wait()
	print("-----------Complete--------------")

Reading the above, you will have a rough idea of how the process works (I'm using a slightly different method compared to the main code).

As you can see, no security mechanism has been implemented in the code. No pin code for the pairing, no checksum at the end of the upload, no version number hard-coded—sometimes, I had a version date and git version hash at compilation time, but not here —, etc. That could also be something to implement in your library.

I do not have much time to help you at the moment, but I like your idea, even if there are still some mechanisms to be thought.

Regards,
Vincent

[vovagorodok]

Ok, just let me know if you find corner case or bug in my considerations. Will be nice to implement minimal working version and than extend it. For example checksum can be added in future by optional command:

-> CHECKSUM <checksum>
<- OK

before INSTALL command. And developer of mobile app will decide if send it or not. But lets consider it after basic implementation.

I am also interested in @fbiego's point of view

[vovagorodok]

Hi guys,
New library created:
https://github.com/vovagorodok/ArduinoBleOTA
It has built in checksum integrity protection and software/hardware type/version indication.
Security/Authorization protection decided to not implement here, because it can be implemented as separate library (for whole bluetooth connection, not for one OTA service).

Library can be simply added to any project like this:
https://github.com/vovagorodok/ArduinoBleOTA/blob/main/examples/basic/main.ino
Or if there is any additional service:
https://github.com/vovagorodok/ArduinoBleOTA/blob/main/examples/multiservice/main.ino

Next steps:

1. Add new not ESP32 platforms (will be easy to add samd and other using ArduinoBLE library)
2. Increase upload speed. Currently I have ~3kB/s. According to ~12kB/s FFat does it or we can achieve it with SPIFFS (default i gues)?
3. Mobile application. Currently it updates only by 'updater.py' script. I'm not a mobile developer, some help/integration here will be nice

[vovagorodok]

According to step 2.
Achieved ~12kB/s (usually ~10kB/s) even on default.csv. Now data partition type doesn't meter, because uploading does directly to ota partition. Problem was only on bluetooth side, because responses was sent after each package. I'll update main branch tomorrow

[vovagorodok]

Step 1. Done. Added new (not esp32/NimBLE-Arduino lib) platforms like samd,megaavr,mbed,apollo3,mbed_nano,mbed_portenta,mbed_nicla based on ArduinoBLE library. Plans to add stm32 and other

Step 2. Done. Works fine with BLE_OTA_ATTRIBUTE_SIZE=512 and BLE_OTA_BUFFER_SIZE~=512 * 10. Can be tuned or buffer can be completely disabled in order to save static memory: https://github.com/vovagorodok/ArduinoBleOTA/blob/main/doc/ADVANCED.md

Step3. Still open

[sdetweil]

I was referred to this URL on BLE performance..
https://punchthrough.com/ble-throughput-part-4/
one reported 50k/sec thruput..