Re-implements as modular server using Express

[DougReeder]

Features/Fixes

• streaming transfer (documents don't need to fit in memory)
• JSON Web Tokens (JWT) so permissions can be validated in memory, without a round trip to a server or even reading a file
• S3-compatible storage (AWS S3 allows up to 5 TB)
• Bug Fix: correctly handles If-None-Match with ETag
• Bug Fix: returns empty listing for nonexistent folder
• Implements current spec: draft-dejong-remotestorage-22
• based on Express, for much cleaner code & standard extension interface
  • easier to reason about
  • stores are standard Express handlers
  • new kinds of authentication will be straighforward
  • Middleware exists for almost anything that is worth doing in an Express server

Support for other extensions

Allows the extensibility of #88 and #89 and armadietto+lucchetto in a standard way. Bespoke versions can be implemented by copying appFactory.js and adding new middleware. It will be much easier to say "yes" to extensions with a limited audience. Middleware for storage quotas, rate limiting and in-app purchase can be in the archive, but don't have to be in the default configuration.

Notes for evaluating

Store

To run the server or test S3-compatible storage, an S3-compatible instance must be running and environment variables set. See notes/S3-streaming-store.md. To run automated tests:

node node_modules/mocha/bin/mocha.js spec/streaming_handler/S3Handler.spec.js

• AWS fully works
• Min.IO is usable, but has a leaky abstraction — the Min.IO web Console only displays the root folder & deleting a user doesn't fully complete.
• OpenIO fails the simultaneous deletion test — it shouldn't be used now, but perhaps some day it will be useable
• Could use testing with Google Cloud Storage, DigitalOcean Spaces, etc.
• Could use more testing with files larger than 1GB — see the "transfers very large files" test

Automated Tests

The original automated tests are still in spec/armadietto. They validate that the monolithic server still functions the same.

The automated tests have been copied copied and reworked to allow testing the modular server against the same tests as the monolithic server. The tests starting with a_ in spec/armadietto set up the monolithic server and call the tests in spec. The tests starting with m_ in spec/modular set up the modular server and call the tests in spec. Thus, the tests validate that the modular server behaves the same as the monolithic server, aside from a few edge cases.

[raucao]

Just a note for self-hosters: Garage is also a great choice for an S3-compatible object storage that is free software and not geared towards enterprise customers. We're currently beta-testing a new remoteStorage service using that over at Kosmos, and haven't seen any issues so far.

[DougReeder]

Garage does work well; I'm using it for development.

I did find a bug, which will need to be fixed before we can support Range requests, but that's a ways off:
If-Match not Implemented for GET

[AddictedToBattlestar]

I have only really looked over the changes in AppFactory as well as some of the files that are imported there to support how express is setup.

I hope my feedback here is helpful regardless of this partial review for the PR.

[DougReeder]

I have only really looked over the changes in AppFactory as well as some of the files that are imported there to support how express is setup.

I hope my feedback here is helpful regardless of this partial review for the PR.

Thanks for the feedback! Having another set of eyes gives me confidence we're on the right course.

[DougReeder]

I would really like to have more feedback on a major change like this, but as this fixes several bugs and removes the obstacles to desired improvements (and doesn't foreclose development on the monolithic server), I will merge this in a week, unless there are objections.