improve webserver, add domain redirects (aliases), add tests and admin page ui to manage the config

- make builtin http handlers serve on specific domains, such as for mta-sts, so
  e.g. /.well-known/mta-sts.txt isn't served on all domains.
- add logging of a few more fields in access logging.
- small tweaks/bug fixes in webserver request handling.
- add config option for redirecting entire domains to another (common enough).
- split httpserver metric into two: one for duration until writing header (i.e.
  performance of server), another for duration until full response is sent to
  client (i.e. performance as perceived by users).
- add admin ui, a new page for managing the configs. after making changes
  and hitting "save", the changes take effect immediately. the page itself
  doesn't look very well-designed (many input fields, makes it look messy). i
  have an idea to improve it (explained in admin.html as todo) by making the
  layout look just like the config file. not urgent though.

i've already changed my websites/webapps over.

the idea of adding a webserver is to take away a (the) reason for folks to want
to complicate their mox setup by running an other webserver on the same machine.
i think the current webserver implementation can already serve most common use
cases. with a few more tweaks (feedback needed!) we should be able to get to 95%
of the use cases. the reverse proxy can take care of the remaining 5%.
nevertheless, a next step is still to change the quickstart to make it easier
for folks to run with an existing webserver, with existing tls certs/keys.
that's how this relates to issue #5.

Author: mjl-

.jshintrc

@@ -3,6 +3,7 @@
 	"asi": true,
 	"strict": "implied",
 	"globals": {
+		"self": true,
 		"window": true,
 		"console": true,
 		"document": true,

README.md

@@ -31,6 +31,8 @@ See Quickstart below to get started.
   accounts/domains, and modifying the configuration file.
 - Autodiscovery (with SRV records, Microsoft-style and Thunderbird-style) for
   easy account setup (though not many clients support it).
+- Webserver with serving static files and forwarding requests (reverse
+  proxy), so port 443 can also be used to serve websites.
 - Prometheus metrics and structured logging for operational insight.
 
 Mox is available under the MIT-license and was created by Mechiel Lukkien,
@@ -54,7 +56,7 @@ Verify you have a working mox binary:
 
 	./mox version
 
-Note: Mox only compiles/works on unix systems, not on Plan 9 or Windows.
+Note: Mox only compiles for/works on unix systems, not on Plan 9 or Windows.
 
 You can also run mox with docker image "docker.io/moxmail/mox", with tags like
 "latest", "0.0.1" and "0.0.1-go1.20.1-alpine3.17.2", etc. See docker-compose.yml
@@ -66,22 +68,20 @@ in this repository for instructions on starting.
 The easiest way to get started with serving email for your domain is to get a
 vm/machine dedicated to serving email, name it [host].[domain] (e.g.
 mail.example.com), login as root, create user "mox" and its homedir by running
-"useradd -d /home/mox mox && mkdir /home/mox", download mox to that directory,
-and generate a configuration for your desired email address at your domain:
+`useradd -d /home/mox mox && mkdir /home/mox` (or pick another directory),
+download mox to that directory, and generate a configuration for your desired
+email address at your domain:
 
 	./mox quickstart [email protected]
 
 This creates an account, generates a password and configuration files, prints
 the DNS records you need to manually create and prints commands to start mox and
 optionally install mox as a service.
 
-If you already have email configured for your domain, or if you are already
-sending email for your domain from other machines/services, you should modify
-the suggested configuration and/or DNS records.
-
 A dedicated machine is highly recommended because modern email requires HTTPS,
-and mox currently needs it for automatic TLS.  You can combine mox with an
-existing webserver, but it requires more configuration.
+and mox currently needs it for automatic TLS.  You could combine mox with an
+existing webserver, but it requires more configuration. If you want to serve
+websites on the same machine, use the webserver built into mox.
 
 After starting, you can access the admin web interface on internal IPs.
 
@@ -109,7 +109,6 @@ The code is heavily cross-referenced with the RFCs for readability/maintainabili
 - DANE and DNSSEC.
 - Sending DMARC and TLS reports (currently only receiving).
 - OAUTH2 support, for single sign on.
-- Basic reverse proxy, so port 443 can be used for regular web serving too.
 - Using mox as backup MX.
 - ACME verification over HTTP (in addition to current tls-alpn01).
 - Add special IMAP mailbox ("Queue?") that contains queued but
@@ -182,7 +181,7 @@ and receive emails through it with your favourite email clients, and file an
 issue if you encounter a problem or would like to see a feature/functionality
 implemented.
 
-Instead of switching your email for your domain over to mox, you could simply
+Instead of switching email for your domain over to mox, you could simply
 configure mox for a subdomain, e.g. [you]@moxtest.[yourdomain].
 
 If you have experience with how the email protocols are used in the wild, e.g.
@@ -212,17 +211,17 @@ The admin password can be changed with "mox setadminpassword".
 Unfortunately, mox does not yet provide an option for that. Mox does spam
 filtering based on reputation of received messages. It will take a good amount
 of work to share that information with a backup MX. Without that information,
-spammer could use a backup MX to get their spam accepted. Until mox has a
+spammers could use a backup MX to get their spam accepted. Until mox has a
 proper solution, you can simply run a single SMTP server.
 
 ## How do I stay up to date?
 
-Please set "CheckUpdates: true" in mox.conf. It will check for a new version
-through a DNS TXT request at `_updates.xmox.nl` once per 24h. Only if a new
-version is published, will the changelog be fetched and delivered to the
+Please set "CheckUpdates: true" in mox.conf. Mox will check for a new version
+through a DNS TXT request for `_updates.xmox.nl` once per 24h. Only if a new
+version is published will the changelog be fetched and delivered to the
 postmaster mailbox.
 
-The changelog is at https://updates.xmox.nl/changelog
+The changelog is at https://updates.xmox.nl/changelog.
 
 You could also monitor newly added tags on this repository, or for the docker
 image, but update instructions are in the changelog.
@@ -241,6 +240,6 @@ to [email protected].
 
 ## I'm now running an email server, but how does email work?
 
-Congrats and welcome to the club! Running an email server brings some
-responsibility so you should understand how it works. See
+Congrats and welcome to the club! Running an email server on the internet comes
+with some responsibilities so you should understand how it works. See
 https://explained-from-first-principles.com/email/ for a thorough explanation.

config/config.go

@@ -70,9 +70,12 @@ type Static struct {
 
 // Dynamic is the parsed form of domains.conf, and is automatically reloaded when changed.
 type Dynamic struct {
-	Domains     map[string]Domain  `sconf-doc:"Domains for which email is accepted. For internationalized domains, use their IDNA names in UTF-8."`
-	Accounts    map[string]Account `sconf-doc:"Accounts to which email can be delivered. An account can accept email for multiple domains, for multiple localparts, and deliver to multiple mailboxes."`
-	WebHandlers []WebHandler       `sconf:"optional" sconf-doc:"Handle webserver requests by serving static files, redirecting or reverse-proxying HTTP(s). The first matching WebHandler will handle the request. Built-in handlers for autoconfig and mta-sts always run first. If no handler matches, the response status code is file not found (404). If functionality you need is missng, simply forward the requests to an application that can provide the needed functionality."`
+	Domains            map[string]Domain  `sconf-doc:"Domains for which email is accepted. For internationalized domains, use their IDNA names in UTF-8."`
+	Accounts           map[string]Account `sconf-doc:"Accounts to which email can be delivered. An account can accept email for multiple domains, for multiple localparts, and deliver to multiple mailboxes."`
+	WebDomainRedirects map[string]string  `sconf:"optional" sconf-doc:"Redirect all requests from domain (key) to domain (value). Always redirects to HTTPS. For plain HTTP redirects, use a WebHandler with a WebRedirect."`
+	WebHandlers        []WebHandler       `sconf:"optional" sconf-doc:"Handle webserver requests by serving static files, redirecting or reverse-proxying HTTP(s). The first matching WebHandler will handle the request. Built-in handlers for autoconfig and mta-sts always run first. If no handler matches, the response status code is file not found (404). If functionality you need is missng, simply forward the requests to an application that can provide the needed functionality."`
+
+	WebDNSDomainRedirects map[dns.Domain]dns.Domain `sconf:"-"`
 }
 
 type ACME struct {
@@ -308,10 +311,9 @@ type TLS struct {
 }
 
 type WebHandler struct {
-	LogName    string `sconf:"optional" sconf-doc:"Name to use in logging and metrics."`
-	Domain     string `sconf-doc:"Both Domain and PathRegexp must match for this WebHandler to match a request. Exactly one of WebStatic, WebRedirect, WebForward must be set."`
-	PathRegexp string `sconf-doc:"Regular expression matched against request path, must always start with ^ to ensure matching from the start of the path. The matching prefix can optionally be stripped by WebForward. The regular expression does not have to end with $."`
-
+	LogName               string       `sconf:"optional" sconf-doc:"Name to use in logging and metrics."`
+	Domain                string       `sconf-doc:"Both Domain and PathRegexp must match for this WebHandler to match a request. Exactly one of WebStatic, WebRedirect, WebForward must be set."`
+	PathRegexp            string       `sconf-doc:"Regular expression matched against request path, must always start with ^ to ensure matching from the start of the path. The matching prefix can optionally be stripped by WebForward. The regular expression does not have to end with $."`
 	DontRedirectPlainHTTP bool         `sconf:"optional" sconf-doc:"If set, plain HTTP requests are not automatically permanently redirected (308) to HTTPS. If you don't have a HTTPS webserver configured, set this to true."`
 	WebStatic             *WebStatic   `sconf:"optional" sconf-doc:"Serve static files."`
 	WebRedirect           *WebRedirect `sconf:"optional" sconf-doc:"Redirect requests to configured URL."`
@@ -322,6 +324,37 @@ type WebHandler struct {
 	Path      *regexp.Regexp `sconf:"-" json:"-"`
 }
 
+// Equal returns if wh and o are equal, only looking at fields in the configuration file, not the derived fields.
+func (wh WebHandler) Equal(o WebHandler) bool {
+	clean := func(x WebHandler) WebHandler {
+		x.Name = ""
+		x.DNSDomain = dns.Domain{}
+		x.Path = nil
+		x.WebStatic = nil
+		x.WebRedirect = nil
+		x.WebForward = nil
+		return x
+	}
+	cwh := clean(wh)
+	co := clean(o)
+	if cwh != co {
+		return false
+	}
+	if (wh.WebStatic == nil) != (o.WebStatic == nil) || (wh.WebRedirect == nil) != (o.WebRedirect == nil) || (wh.WebForward == nil) != (o.WebForward == nil) {
+		return false
+	}
+	if wh.WebStatic != nil {
+		return reflect.DeepEqual(wh.WebStatic, o.WebStatic)
+	}
+	if wh.WebRedirect != nil {
+		return wh.WebRedirect.equal(*o.WebRedirect)
+	}
+	if wh.WebForward != nil {
+		return wh.WebForward.equal(*o.WebForward)
+	}
+	return true
+}
+
 type WebStatic struct {
 	StripPrefix      string            `sconf:"optional" sconf-doc:"Path to strip from the request URL before evaluating to a local path. If the requested URL path does not start with this prefix and ContinueNotFound it is considered non-matching and next WebHandlers are tried. If ContinueNotFound is not set, a file not found (404) is returned in that case."`
 	Root             string            `sconf-doc:"Directory to serve files from for this handler. Keep in mind that relative paths are relative to the working directory of mox."`
@@ -331,7 +364,7 @@ type WebStatic struct {
 }
 
 type WebRedirect struct {
-	BaseURL        string `sconf:"optional" sconf-doc:"Base URL to redirect to. The path must be empty and will be replaced, either by the request URL path, or byOrigPathRegexp/ReplacePath. Scheme, host, port and fragment stay intact, and query strings are combined. If empty, the response redirects to a different path through OrigPathRegexp and ReplacePath, which must then be set. Use a URL without scheme to redirect without changing the protocol, e.g. //newdomain/."`
+	BaseURL        string `sconf:"optional" sconf-doc:"Base URL to redirect to. The path must be empty and will be replaced, either by the request URL path, or by OrigPathRegexp/ReplacePath. Scheme, host, port and fragment stay intact, and query strings are combined. If empty, the response redirects to a different path through OrigPathRegexp and ReplacePath, which must then be set. Use a URL without scheme to redirect without changing the protocol, e.g. //newdomain/."`
 	OrigPathRegexp string `sconf:"optional" sconf-doc:"Regular expression for matching path. If set and path does not match, a 404 is returned. The HTTP path used for matching always starts with a slash."`
 	ReplacePath    string `sconf:"optional" sconf-doc:"Replacement path for destination URL based on OrigPathRegexp. Implemented with Go's Regexp.ReplaceAllString: $1 is replaced with the text of the first submatch, etc. If both OrigPathRegexp and ReplacePath are empty, BaseURL must be set and all paths are redirected unaltered."`
 	StatusCode     int    `sconf:"optional" sconf-doc:"Status code to use in redirect, e.g. 307. By default, a permanent redirect (308) is returned."`
@@ -340,10 +373,24 @@ type WebRedirect struct {
 	OrigPath *regexp.Regexp `sconf:"-" json:"-"`
 }
 
+func (wr WebRedirect) equal(o WebRedirect) bool {
+	wr.URL = nil
+	wr.OrigPath = nil
+	o.URL = nil
+	o.OrigPath = nil
+	return reflect.DeepEqual(wr, o)
+}
+
 type WebForward struct {
 	StripPath       bool              `sconf:"optional" sconf-doc:"Strip the matching WebHandler path from the WebHandler before forwarding the request."`
 	URL             string            `sconf-doc:"URL to forward HTTP requests to, e.g. http://127.0.0.1:8123/base. If StripPath is false the full request path is added to the URL. Host headers are sent unmodified. New X-Forwarded-{For,Host,Proto} headers are set. Any query string in the URL is ignored. Requests are made using Go's net/http.DefaultTransport that takes environment variables HTTP_PROXY and HTTPS_PROXY into account."`
 	ResponseHeaders map[string]string `sconf:"optional" sconf-doc:"Headers to add to the response. Useful for adding security- and cache-related headers."`
 
 	TargetURL *url.URL `sconf:"-" json:"-"`
 }
+
+func (wf WebForward) equal(o WebForward) bool {
+	wf.TargetURL = nil
+	o.TargetURL = nil
+	return reflect.DeepEqual(wf, o)
+}

config/doc.go

@@ -557,6 +557,11 @@ describe-static" and "mox config describe-domains":
 					# in calculating probability reduced. E.g. 1 or 2. (optional)
 					RareWords: 0
 
+	# Redirect all requests from domain (key) to domain (value). Always redirects to
+	# HTTPS. For plain HTTP redirects, use a WebHandler with a WebRedirect. (optional)
+	WebDomainRedirects:
+		x:
+
 	# Handle webserver requests by serving static files, redirecting or
 	# reverse-proxying HTTP(s). The first matching WebHandler will handle the request.
 	# Built-in handlers for autoconfig and mta-sts always run first. If no handler
@@ -622,7 +627,7 @@ describe-static" and "mox config describe-domains":
 			WebRedirect:
 
 				# Base URL to redirect to. The path must be empty and will be replaced, either by
-				# the request URL path, or byOrigPathRegexp/ReplacePath. Scheme, host, port and
+				# the request URL path, or by OrigPathRegexp/ReplacePath. Scheme, host, port and
 				# fragment stay intact, and query strings are combined. If empty, the response
 				# redirects to a different path through OrigPathRegexp and ReplacePath, which must
 				# then be set. Use a URL without scheme to redirect without changing the protocol,
@@ -670,14 +675,20 @@ examples with "mox examples", and print a specific example with "mox examples
 
 # Example webhandlers
 
-	# Snippet of domains.conf to configure WebHandlers.
+	# Snippet of domains.conf to configure WebDomainRedirects and WebHandlers.
+
+	# Redirect all requests for mox.example to https://www.mox.example.
+	WebDomainRedirects:
+		mox.example: www.mox.example
+
+	# Each request is matched against these handlers until one matches and serves it.
 	WebHandlers:
 		-
 			# The name of the handler, used in logging and metrics.
 			LogName: staticmjl
 			# With ACME configured, each configured domain will automatically get a TLS
 			# certificate on first request.
-			Domain: mox.example
+			Domain: www.mox.example
 			PathRegexp: ^/who/mjl/
 			WebStatic:
 				StripPrefix: /who/mjl
@@ -690,7 +701,7 @@ examples with "mox examples", and print a specific example with "mox examples
 					X-Mox: hi
 		-
 			LogName: redir
-			Domain: mox.example
+			Domain: www.mox.example
 			PathRegexp: ^/redir/a/b/c
 			# Don't redirect from plain HTTP to HTTPS.
 			DontRedirectPlainHTTP: true
@@ -703,15 +714,15 @@ examples with "mox examples", and print a specific example with "mox examples
 				StatusCode: 307
 		-
 			LogName: oldnew
-			Domain: mox.example
+			Domain: www.mox.example
 			PathRegexp: ^/old/
 			WebRedirect:
 				# Replace path, leaving rest of URL intact.
 				OrigPathRegexp: ^/old/(.*)
 				ReplacePath: /new/$1
 		-
 			LogName: app
-			Domain: mox.example
+			Domain: www.mox.example
 			PathRegexp: ^/app/
 			WebForward:
 				# Strip the path matched by PathRegexp before forwarding the request. So original

docker-compose.yml

@@ -20,6 +20,9 @@ services:
     volumes:
       - ./config:/mox/config
       - ./data:/mox/data
+      # web is optional but recommended to bind in, useful for serving static files with
+      # the webserver.
+      - ./web:/mox/web
     working_dir: /mox
     restart: on-failure
     healthcheck:

http/account.go

@@ -76,7 +76,7 @@ func checkAccountAuth(ctx context.Context, log *mlog.Log, w http.ResponseWriter,
 	}
 	if addr != nil && !mox.LimiterFailedAuth.Add(addr.IP, start, 1) {
 		metrics.AuthenticationRatelimitedInc("httpaccount")
-		http.Error(w, "http 429 - too many auth attempts", http.StatusTooManyRequests)
+		http.Error(w, "429 - too many auth attempts", http.StatusTooManyRequests)
 		return ""
 	}
 

http/admin.go

@@ -16,6 +16,7 @@ import (
 	"net"
 	"net/http"
 	"os"
+	"reflect"
 	"runtime/debug"
 	"sort"
 	"strings"
@@ -31,6 +32,7 @@ import (
 	"github.com/mjl-/sherpadoc"
 	"github.com/mjl-/sherpaprom"
 
+	"github.com/mjl-/mox/config"
 	"github.com/mjl-/mox/dkim"
 	"github.com/mjl-/mox/dmarc"
 	"github.com/mjl-/mox/dmarcdb"
@@ -136,7 +138,7 @@ func checkAdminAuth(ctx context.Context, passwordfile string, w http.ResponseWri
 	}
 	if addr != nil && !mox.LimiterFailedAuth.Add(addr.IP, start, 1) {
 		metrics.AuthenticationRatelimitedInc("httpadmin")
-		http.Error(w, "http 429 - too many auth attempts", http.StatusTooManyRequests)
+		http.Error(w, "429 - too many auth attempts", http.StatusTooManyRequests)
 		return false
 	}
 
@@ -1525,3 +1527,73 @@ func (Admin) LogLevelRemove(ctx context.Context, pkg string) {
 func (Admin) CheckUpdatesEnabled(ctx context.Context) bool {
 	return mox.Conf.Static.CheckUpdates
 }
+
+// WebserverConfig is the combination of WebDomainRedirects and WebHandlers
+// from the domains.conf configuration file.
+type WebserverConfig struct {
+	WebDNSDomainRedirects [][2]dns.Domain // From server to frontend.
+	WebDomainRedirects    [][2]string     // From frontend to server, it's not convenient to create dns.Domain in the frontend.
+	WebHandlers           []config.WebHandler
+}
+
+// WebserverConfig returns the current webserver config
+func (Admin) WebserverConfig(ctx context.Context) (conf WebserverConfig) {
+	conf = webserverConfig()
+	conf.WebDomainRedirects = nil
+	return conf
+}
+
+func webserverConfig() WebserverConfig {
+	r, l := mox.Conf.WebServer()
+	x := make([][2]dns.Domain, 0, len(r))
+	xs := make([][2]string, 0, len(r))
+	for k, v := range r {
+		x = append(x, [2]dns.Domain{k, v})
+		xs = append(xs, [2]string{k.Name(), v.Name()})
+	}
+	sort.Slice(x, func(i, j int) bool {
+		return x[i][0].ASCII < x[j][0].ASCII
+	})
+	sort.Slice(xs, func(i, j int) bool {
+		return xs[i][0] < xs[j][0]
+	})
+	return WebserverConfig{x, xs, l}
+}
+
+// WebserverConfigSave saves a new webserver config. If oldConf is not equal to
+// the current config, an error is returned.
+func (Admin) WebserverConfigSave(ctx context.Context, oldConf, newConf WebserverConfig) (savedConf WebserverConfig) {
+	current := webserverConfig()
+	webhandlersEqual := func() bool {
+		if len(current.WebHandlers) != len(oldConf.WebHandlers) {
+			return false
+		}
+		for i, wh := range current.WebHandlers {
+			if !wh.Equal(oldConf.WebHandlers[i]) {
+				return false
+			}
+		}
+		return true
+	}
+	if !reflect.DeepEqual(oldConf.WebDNSDomainRedirects, current.WebDNSDomainRedirects) || !webhandlersEqual() {
+		xcheckf(ctx, errors.New("config has changed"), "comparing old/current config")
+	}
+
+	// Convert to map, check that there are no duplicates here. The canonicalized
+	// dns.Domain are checked again for uniqueness when parsing the config before
+	// storing.
+	domainRedirects := map[string]string{}
+	for _, x := range newConf.WebDomainRedirects {
+		if _, ok := domainRedirects[x[0]]; ok {
+			xcheckf(ctx, errors.New("already present"), "checking redirect %s", x[0])
+		}
+		domainRedirects[x[0]] = x[1]
+	}
+
+	err := mox.WebserverConfigSet(ctx, domainRedirects, newConf.WebHandlers)
+	xcheckf(ctx, err, "saving webserver config")
+
+	savedConf = webserverConfig()
+	savedConf.WebDomainRedirects = nil
+	return savedConf
+}