Replacement for Civetweb

[cfriedt]

Introduction

This (previously an RFC) is intended to be discussed by network subsystem maintainers, the TSC, and existing users of the Civetweb module.

The replacement for Civetweb must be ready for LTSv3 (Civetweb was available in LTSv2, and we should have a suitable replacement for LTSv3). However, ideally, this would be completed as soon as v3.3.

Comments & corrections are welcome.

Problem description

Civetweb has been a bit of a pain point for a while due to lack of regular module maintainership. There is also a minor technical shortcoming in that it does not directly support TLS.

The module has been deprecated (#46061) and likely will be removed shortly (e.g. #46746).

However, HTTP is fairly important - some might even say it should be a first-class citizen - not simply for demo purposes, but also for production systems.

With that in mind, we may want to consider providing a suitable alternative to support the following sample applications:

• http_server
• websocket_server

In order to maintain feature parity with Civetweb (as supported within Zephyr) the requirements for a replacement HTTP server are:

• Permissively licensed (Apache-2.0, BSD*, MIT, ..)
• Optimized for size (both code space and memory usage)
• Support for HTTP/1.1
• Support for operating without a filesystem
• Easy extensible to add custom URIs and handlers (e.g. REST)
• WebSocket server support
• Ability to use e.g. Zephyr's JSON library
• Protection against any exploits / CVEs previously met with Civetweb
• IPv4 / IPv6 Support
• Reporting statistics e.g. vis JSON

CVE details and Additional "nice to have" features are discussed in Detailed RFC.

Proposed change

This RFC proposes to adopt one of the two following options:

1. External Zephyr module: Adopt Merecat (fork of thttpd)
2. Internal Zephyr library: Design an HTTP server

Detailed RFC

Additional Details

Aside from the features required for parity, there are a number of "nice to have" features for a replacement HTTP server.

• Support for TLS
  • Note: Zephyr implementation may be in flux (PSA Crypto API adoption in Zephyr #43712)
• Configurable via Kconfig (filesystem, maximum number of clients, timeouts, buffer sizes, features, ..)
• Built with CMake (enable / disable compiling sources based on feature set)
• Decent performance (scales well with number of clients, requests)
• Secure (should not be a regular source of CVEs)
• Easily maintainable (SW architected well)
• Easily testable (via ztest, if possible)
• Support Compressed Resources in Memory (CRiMe)
• Support filesystem resources
• A web-based dashboard (e.g. via REST / JSON)
• HTTP/3 (eventually..)
• HTTP Client / WebSockets Client - quite useful for testing, but could be made available to the application

Previously discovered exploits in Civetweb that have affected Zephyr are listed below:

• TODO

Comparisons with other Web Servers

There are some readily available comparisons of various HTTP servers. E.g. Wikipedia, thttpd site. Mongoose can probably be swapped with Civetweb for comparison purposes, and the same for thttpd and Merecat.

Tabular comparison

Note: - should be interpreted as "not yet but eventually yes".

Feature	Civetweb	Merecat	Custom
License	MIT	BSD-2-clause	Apache
Size?	Optimized	Optimized	-
HTTP/1.1	Yes	Yes	-
No FS	Yes	Yes	-
REST	Yes	Yes	-
WebSockets	Yes	No	-
JSON	Yes	No	Yes
CVEs	Yes	No	Yes
v4/v6	Yes	Yes	Yes
Stats	Yes	?	Yes
TLS	No?	(OpenSSL)	-
X.509	No?	?	-
Kconfig	No	No	Yes
CMake	No	No	Yes
Perf	No?	Yes	-
Secure	Yes?	Yes	-
Authentication	?	Yes	-
Maintainable	No	No	Yes
Testable	No	No	Yes
gzip / deflate	No	(Zlib)	-
FS	No	Yes	Yes
Dash	Yes	No	Yes (JSON)
OAuth	?	?	-
SASL	?	?	-
HTTP/3	No	No	(Eventually..)
-------	----	----	-------
SCORE	11	11	12-20

TLDR / Summary

• Obviously, no need for CGI at this time
• HTTP Authentication is ~OK, but a TLS / mutual auth / X.509 would be much better
• Making an HTTP server requires some dedication
  • as pointed out, writing a simple HTTP server is easy, but writing a robust one, not quite as easy
  • would need to be put on the roadmap
  • not entirely within scope of responsibilities of Network Maintainer (overloaded already)
  • would need to volunteers for maintainers / collaborators
  • would need to have someone from the Security WG involved regularly
  • can use TDD to get up to speed
  • tests to protect against previously encountered CVEs
• Zlib is probably not suitable to include in Zephyr
  • deflate / gzip is luckily not very difficult to implement (ZlibTransport integration gsoc-2022-thrift#91)
  • some resources can be compressed offline, at build time. stored in ROM, no runtime requirements
  • if supporting FS resources and compression, then possibly deflate / gzip as a runtime requirement
• A web-based dashboard would be nice, although both could be configured
• Part of the issue that made Civetweb difficult to maintain was that it was a module
  • Doing the west.yml / revision pinning dance can slow things down a bit
  • Keeping something in-tree could have faster turnaround for bugfixes & testing
• We likely want to use a typically single-threaded ("select") model to reduce resource utilization
  • It may be possible to have some configurable number of worker threads
• Merecat is attractive, but would still require adding support for a number of features:
  • WebSockets
  • mbedTLS
  • compressed resources in ROM
  • Kconfig
  • Tests
  • ...

Proposed change (Detailed)

It does not make a lot of sense to go into detail about adding another HTTP module to Zephyr that simply tracks a third-party repository. We have done that before. It is an option.

Details of focus on proposal of purpose-built HTTP solution for Zephyr

General Design

• Top-down design with Kconfig
  • maximum timeouts / keep-alive
  • maximum buffer sizes
• Test-driven development with C++, POSIX API
  • Functional first, optimize later
  • POSIX API allows for native development
  • Use of STL for ease of bringup (can be swapped later if necessary)
• HTTP is a string-heavy protocol => make heavy use of std::string_view
  • string_view: (pointer, size) for constant strings
  • C++17
• Make use of literals / constexpr as much as possible
  • HTTP Headers: key-value pairs => unordered_map<string_view, string_view>
  • Request Headers held constant while servicing request
  • Response Headers held constant while servicing response
• Resources via unordered_map<string_view, resource_obj>
• Support FS via CONFIG_POSIX_FS
• Initially, single-threaded design ("select" approach)
  • Consider multi-threaded later, if necessary

Compressed Resources in Memory (CRiMe)

• Memory is ROM in this case
• Quite straightforward at build time with CMake and a python script
• For each resource in CONFIG_HTTP_WEB_ROOT="/path/to/dir"
  • GZip / Zip resource
  • Convert to C array with xxd / python
  • Generate metadata, C/C++ source, including array (python script)
  • Generate CMake target

REST Support

• Endpoints declared in application C/C++ (GET, SET, POST, etc)
• Statically registered via iterable section?
• Endpoints really just filling in callbacks for supported methods

Filesystem Support

• depend on CONFIG_POSIX_FS
• should allow users to drop files into e.g. /www on an SD card and immediately serve them as resources
  • need FS "event listener" API (does that already exist?)
• a concrete use-case for optionally supporting runtime compression of served resources
• If a URI contained in the FS (/www) conflicts with a CRiMe object, which has priority?

URI Priority

1. REST endpoint
2. CRiMe
3. FS (if so configured)

Ordering configurable? It may be desireable to have FS resources override CRiMe 🤷

Dependencies

Both options can be implemented in such a way that they do not have external dependencies.

Concerns and Unresolved Questions

• Complexity w.r.t. building and maintaining a good embedded web server
• Complexity about maintaining essentially a fork of Merecat (if that is something we consider)
• Anxiety about another external HTTP module
• How best to integrate TLS?

Alternatives

See previous comparisons

Action Items

• include: net: http: rework http headers #51141
• net: http: define http service and resource iterable sections #55430

[cfriedt]

Just putting this out there for some early feedback, since we have a PR to remove civetweb already.

@mbolivar, @beriberikix, @rlubos

I'm not really sure which option is better, TBH. An HTTP server as a module has been problematic in the past, but a custom one could be a fair amount of work with all of the "nice to have" features. OTOH, an HTTP server that has been around for a long time has been vetted, but if a custom one were created it could be tuned to work with Zephyr quite well, tests could be added for everything from functionality to CVEs to performance.

If there are alternatives that are better than Merecat, it would be great to have them added to the table.

[cfriedt]

Approximate state machine for single-threaded "select" model (really uses poll)

[beriberikix]

Here's a use case that might help in scoping - browser-based device setup. Common for Wi-Fi consumer IoT product, the typical flow is the product tells the consumer to look for the device's local network, connect to an IP or DNS-SD address and enter their network credentials (loosely described here.)

The requirements are fairly modest as the server tends to be simple, typically with minimal static HTML and CSS embedded into the webpage and some way to interact with the OS (ex. store credentials via the Settings subsystem.) Often times these pages are not encrypted but being able to serve them via HTTPS is recommended.

[beriberikix]

Additional ideas:

Would it make sense to support an optional FS or other way to serve binary assets? For developer convenience I can imagine someone wanting a way to upload assets like HTML, CSS, JS & images into a "directory" and simply point the server to it ala /www.

Adding a shell would be a nice-to-have, like the ability to start/stop/clear cache, etc.

CBOR for the RESTful aspects would be a nice-to-have.

Other typical RESTful API features, like authentication (OAuth, SASL) should be explicitly out of scope but noted somewhere in the RFC.

CivetWeb also includes an HTTP client - is that in scope or needed?

X.509 in scope?

[cfriedt]

@beriberikix - generally agree with every point you have made, and ❤️ the idea of browser-based device setup / provisioning.

Would it make sense to support an optional FS or other way to serve binary assets? For developer convenience I can imagine someone wanting a way to upload assets like HTML, CSS, JS & images into a "directory" and simply point the server to it ala /www.

Absolutely - I'll elaborate on the FS option in the RFC.

Adding a shell would be a nice-to-have, like the ability to start/stop/clear cache, etc.

CBOR for the RESTful aspects would be a nice-to-have.

Shell would be nice, but we could also use the browser for that. Mentioned already, but both of those options could be made available via Kconfig - i.e. either web, shell, or both.

Other typical RESTful API features, like authentication (OAuth, SASL) should be explicitly out of scope but noted somewhere in the RFC.

OAuth for sure - thanks for mentioning that. I think there must be a library available that we could pull in.

CivetWeb also includes an HTTP client - is that in scope or needed?

Mentioned as well - I originally thought that was out of scope, because it isn't part of a sample application. However, an HTTP client would probably be smart even to just make the testsuite efficient, so I think there is a definite argument for it.

X.509 in scope?

For sure - I used the term mutual authentication, but X.509 is better.

Will adjust the RFC - thank you for the input!

[carlescufi]

Test-driven development with C++, POSIX API
C++17

So far we have not included any Zephyr-specific code that is written in C++. I am not convinced this is a good idea and, in any case, it would require acceptance at the TSC level. While I completely understand how C++17 can help develop something like a web server, I'd rather see C++ bindings to a C-written server to avoid imposing C++ on anyone.

[rlubos]

CivetWeb also includes an HTTP client - is that in scope or needed?

Mentioned as well - I originally thought that was out of scope, because it isn't part of a sample application. However, an HTTP client would probably be smart even to just make the testsuite efficient, so I think there is a definite argument for it.

Note that Zephyr already has a simple HTTP client library: https://github.com/zephyrproject-rtos/zephyr/blob/main/subsys/net/lib/http/http_client.c

How best to integrate TLS?

Do you plan to use TLS sockets? Any concerns at this point?

[cfriedt]

Test-driven development with C++, POSIX API
C++17

So far we have not included any Zephyr-specific code that is written in C++. I am not convinced this is a good idea and, in any case, it would require acceptance at the TSC level. While I completely understand how C++17 can help develop something like a web server, I'd rather see C++ bindings to a C-written server to avoid imposing C++ on anyone.

@carlescufi My main motivation for using C++ would just be for quick bring up (natively). Given that the lingua franca of Zephyr is C, the API bindings for a custom HTTP impl would also most certainly have to be in C.

[undisputed-seraphim]

Why not just complete the POSIX networking API and let existing network libraries do the rest of the work? I've been trying to port a Boost.ASIO-based application to Zephyr and it's mostly just missing some POSIX APIs and defines. I wrote a hacky, not-for-production-use shim to paper over the missing APIs and it mostly works. Not saying everyone should do this, I was just trying to demo stuff.

[cfriedt]

Note that Zephyr already has a simple HTTP client library: https://github.com/zephyrproject-rtos/zephyr/blob/main/subsys/net/lib/http/http_client.c

Perfect - would be great to leverage that :-)

How best to integrate TLS?

Do you plan to use TLS sockets? Any concerns at this point?

I think it makes sense to use whatever TLS implementation the project intends to support, and there may have been talk of changing that IIRC.

Edit: I checked and it was specifically talk about adopting the PSA Crypto API