Performance of USB-Commands

[bratkartoffel]

Performance questions about the various usb commands

Feature description

The currently implemented usb protocol is nto that performant and could be optimized in several ways:

1. ReadFile: The size of the transfered data is sent in a fixed block of 0x1000 bytes (4 KiB), although only the first 8 bytes are used. After that, the real data is transfered without any further overhead.
2. ReadFile: The transfer of large files is splitted into independant reads of 0x400000 bytes (4 MiB, see previous point)
3. Directoy listings: The client first asks for the count of directories (GetDirectoryCount) inside a given path, then retrieves each entry name using an numbered index (GetDirectory). After that, the same is done for all files (GetFileCount and GetFile). Lastly, another "StatPath" is executed for each entry to determine the type (directory or file) and the size of that entry. Each call is another 0x1000 bytes (4KiB).
4. (Goldleaf) "View text": Each scroll event causes a full-read of that file
5. (Goldleaf) "View hex": There are two "StatFile" calls before the initial "ReadFile"; Each scroll event causes a read of 16 bytes (for the newly shown line). This causes the server to open, seek and close the file each time.

Benefits

Reduce the transfered amount of data, make listing large directories faster and make it easier to implement an alternative client

Additional information

Each finding could be improved in various ways:

1. The ReadFile could send the size and the data without padding in a single block; As far as I can see, this is also like Strings are transferred
2. The Streaming of a large file could be enhanced by sending the whole file size at once; single read command, but multiple response blocks
3. A new "datatype" would be required to transfer lists of data, so a directory lising could be transfered in a single block. The metadata and name of the entries could be combined into one string, and the strings combined together, separated by a control character?
4. Goldleaf should cache small files
5. This operation could be optimized to use the "StartFile" and "EndFile" commands. The server could cache the content and would prevent not needed disk spins.

These are some of the findings I had when implementing an alternative Quark. I hope you find these information useful and can implement some of my suggestions.

Nevertheless, thank you so much for this great tool!

[XorTroll]

I haven't touched USB commands in a long time, surely there is room for improvement there.
I am actually interested in some of your proposals, might give this a try when I get some time ;)
Wonder how much efficient I could make USB transfers this way

[bratkartoffel]

I've created and invited you to a separate repository with my new code. I used "java-does-usb" as javax-usb is deprecated and haven't been updated in the last 6 years. I'd appreciate it, if we could work out some enhanced protocol together and make goldleaf even better, e.g. with an enhanced usb protocol

[XorTroll]

Could you re-send the invitation? I can't find it

[bratkartoffel]

Sure, invitation resent

[XorTroll]

Do you have Discord? That way we can talk about this more directly

[bratkartoffel]

Not yet, but I'll contact you on the server mentioned in your profile. As I'm pretty busy right now this could take some time

[XorTroll]

No worries, contact me whenever you can