Skip already uploaded files and add synchronization functionality

[lukaskremla]

Moved (the code changed a lot and this became outdated)

[Josverl]

FWIW : mpremote v1.24.0+ by default skips uploading unchanged files to a board (if the device has hashlib.sha256 available)
This was introduced in micropython/micropython@6f8157d

mpremote uses chunck_size = 256
the hashing code is pretty similar

This hinges on the fact that reading from flash is often much faster than writing it, therefore the penalty of reading to create a hash is much lower than writing too many files.

If you approach is to reduce file writes to the device - then in my experience in attempting the same is that you also will need to account for:

• Time traveling devices
  I may have multiple devices that I swap in and out while developing - If I press Synch on a device-1that was last connected yesterday - I want the connected device to be the same as the sources on my host, rather than only the files that changed since the last sync to device-2
• Files being created on devices, or transported there via other means ( ftp / web )

Timings for : ESP32_GENERIC_S3

mpremote rm :large.txt
PS D:\> measure-command {mpremote cp large.txt : }
TotalSeconds      : 2,2762847

PS D:\> measure-command {mpremote cp large.txt : }
TotalSeconds      : 0,789586

PS D:\> measure-command {mpremote cp Xlarge.txt : }
8
TotalSeconds      : 9,1412448

PS D:\> measure-command {mpremote cp Xlarge.txt : }
TotalSeconds      : 0,8132274

PS D:\> dir .\large.txt
    Directory: D:\
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          18-12-2024    18:22           7291 large.txt
-a---          18-12-2024    22:32          28967 Xlarge.txt