From dea26c64505c2d498000d93778d6051407a8b0d8 Mon Sep 17 00:00:00 2001 From: Pierre Gayvallet Date: Mon, 3 Jun 2024 09:34:13 +0200 Subject: [PATCH] Add http2 support for Kibana server (#183465) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Part of https://github.com/elastic/kibana/issues/7104 Add support for `http2` to the Kibana server. `http2` can be enabled by setting `server.protocol: http2` in the Kibana config file. *Note: by default, enabling `http2` requires a valid `h2c` configuration, meaning that it can only run over HTTPS with TLS1.2+* ```yaml ## kibana.yaml server.protocol: http2 server.ssl.enabled: true server.ssl.key: path/to/key server.ssl.certificate: path/my/cerf ``` ## What is this PR doing ### Add HTTP2 support for the Kibana server #### - Plug http2 to the Kibana server Even if HAPI was never officially updated to really support HTTP2, node's `http`/`https`/`http2` modules are compatible enough to be able to just instantiate an http2 server/listener and provide it to HAPI "as a plain https listener". There were some tweaks to do (mostly silencing a few warnings that HAPI was causing by sending http2-illegal headers such as `Connection`), but overall, it went smoothly. #### - Add config validation By default, Kibana will require a valid `h2c` configuration to accept enabling `http2`. It means that TLS must be enabled and that TLS1.2+ should at least be in the list of supported SSL protocols (`server.ssl.supportedProtocols`). Note that default value of this setting includes TLS1.2 and 1.3. #### - Add escape hatch to run `h2` without `h2c` In some situations, it may be required to enable http2 without a valid `h2c` configuration. Kibana supports it, by setting `server.http2.allowUnsecure` to `true`. (*Note, however, that if http2 is enabled without TLS, ALPN protocol negotiation won't work, meaning that most http2 agents/clients will fail connecting unless they're explictly configured to use http2.*) ### Add documentation about this new feature #### - Update the user-facing doc about this new `server.protocol` setting Update the user-facing Kibana settings documentation to include this `http.protocol` setting (and refer to `server.http2.allowUnsecure`) **Note: this setting, and this feature, are considered as experimental** ### Adapt our dev tooling to support running Kibana with http2 enabled #### - Add a `--http2` flag to the dev CLI Enabling this flag will add the proper configuration settings to run Kibana with `http2` enabled in an (almost) valid `h2c` configutation. *Note: when using this flag, even if listening on the same port, the Kibana server will be accessible over https, meaning that you need to use https in your browser to access it. Aka `http://localhost:5601` won't work, you need to use `https://localhost:5601`. Also, we're using the self-signed dev certificates, meaning that you must go though the scary warning of your browser* #### - Implement an http2-compatible base-path proxy The current base path proxy is based on `hapi` and `hapi/h2o2`. I tried for a bunch hours trying to hack around to make it work with http2 proxying, but ultimately gave up and implemented a new version from scratch. Note that with some additional efforts, this new http2 basepath proxy could probably fully replace the existing one and be used for both http1 and http2 traffic, but it's an optimization / refactoring that did not feel required for this PR. ### Adapt the FTR to run suites against http2 #### - Add support to run FTR test suite against an h2c-enabled Kibana Note that with ALPN, clients using http1 should be (and are) able to communicate with http2 Kibana, given h2c/alpn allows protocol negitiation. So adapting our FTR tooling was not really about making it work with http2 (which worked out of the box), but making it work with **the self signed certifcates we use for https on dev mode** Note that I'm not a big fan of what I had to do, however, realistically this was the only possible approach if we want to run arbitrary test suites with TLS/HTTP2 enabled without massively changing our FTR setup. Operations and QA, feel free to chime in there, as this is your territory. #### - Change some FTR test suites to run against an HTTP2-enabled server I added a quick `configureHTTP2` helper function to take any "final" FTR suite config and mutate it to enable `http2`. I then enabled it on a few suites locally, to make sure the suites were passing correctly. I kept two suites running with http2 enabled: - the `console` oss functional tests - the `home` oss functional tests We could possibly enable it for more, but we need to figure out what kind of strategy we want on that matter (see below) ## What is this pull request NOT doing #### - Making sure everything works when HTTP2 is enabled I navigated the applications quite a bit, and did not see anything broken, however I obviously wasn't able to do a full coverage. Also, the self-signed certificate was a huge pain to detect issues really caused by http2 compared to issues because the local setup isn't valid `h2c`. In theory though (famous last words) anything not doing http/1.1 specific hacks such as bfetch should work fine with http2, given that even if using non-http2 clients, ALPN should just allow to fallback to http/1.x (this part was tested) #### - Enabling HTTP2 by default PR isn't doing it for obvious reasons. #### - Enabling HTTP2 for all FTR suites First of all, it's not that easy, because it requires adapting various parts of the config (and even some var env...), and we don't have any proper way to override config "at the end". For instance, if you add the http2 config on a top level config (e.g. the oss functional one that is reuse by the whole world - learned the hard way), it won't work because higher-level configs redefined (and override) the `browser` part of the config, loosing the settings added to run the browser in insecure mode. Secondly, I'm not sure we really need to run that many suites with http2 enabled. I learned working on that PR that we only have like one suite where https is enabled for the Kibana server, and I feel like it could be fine to have the same for http2. In theory it's just a protocol change, unless parts of our apps (e.g. bfetch) are doing things that are specific to http/1.1, switching to http2 should be an implementation detail. But I'd love to get @elastic/kibana-operations and @elastic/appex-qa opinion on that one, given they have more expertise than I do on that area. - Running performances tests We should absolutely run perf testing between http/1.1 over https and http/2, to make sure that it goes into the right directly (at least in term of user perceived speed), but I did not do it in the scope of this PR (and @dmlemeshko is on PTO so... 😅) ## Release Note Add support for `http2` to the Kibana server. `http2` can be enabled by setting `server.protocol: http2` in the Kibana config file. Note: by default, enabling `http2` requires a valid `h2c` configuration, meaning that it can only run over HTTPS with TLS1.2+ Please refer to the Kibana config documentation for more details. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> --- docs/setup/settings.asciidoc | 8 + package.json | 2 + .../src/request.test.ts | 28 +- .../src/request.ts | 7 +- .../src/router.ts | 9 + .../src/strip_illegal_http2_headers.test.ts | 114 ++++++++ .../src/strip_illegal_http2_headers.ts | 49 ++++ .../tsconfig.json | 3 +- .../__snapshots__/http_config.test.ts.snap | 4 + .../src/http_config.test.ts | 67 +++++ .../src/http_config.ts | 44 ++- .../core-http-server/src/http_contract.ts | 3 +- .../http1.ts} | 20 +- .../src/base_path_proxy/http2.ts | 183 +++++++++++++ .../src/base_path_proxy/index.ts | 33 +++ .../src/base_path_proxy/types.ts | 24 ++ .../src/base_path_proxy/utils.ts | 15 ++ .../kbn-cli-dev-mode/src/cli_dev_mode.test.ts | 20 +- packages/kbn-cli-dev-mode/src/cli_dev_mode.ts | 8 +- .../src/config/http_config.ts | 5 + ...s => http1_base_path_proxy_server.test.ts} | 21 +- .../http2_base_path_proxy_server.test.ts | 254 ++++++++++++++++++ packages/kbn-cli-dev-mode/tsconfig.json | 1 + .../src/server/server_config.ts | 1 + .../journey/journey_ftr_harness.ts | 2 +- packages/kbn-server-http-tools/index.ts | 10 +- .../src/get_listener.test.mocks.ts | 23 ++ .../src/get_listener.test.ts | 211 +++++++++++---- .../kbn-server-http-tools/src/get_listener.ts | 26 +- .../src/get_server_options.test.ts | 1 + .../src/get_server_options.ts | 2 + .../src/get_tls_options.test.ts | 1 + .../src/ssl/constants.ts | 12 + .../kbn-server-http-tools/src/ssl/index.ts | 1 + .../src/ssl/ssl_config.ts | 19 +- packages/kbn-server-http-tools/src/types.ts | 8 +- .../src/kbn_client/kbn_client_requester.ts | 1 + src/cli/serve/serve.js | 69 +++-- src/core/server/index.ts | 1 + .../http/http2_protocol.test.ts | 214 +++++++++++++++ .../http/set_tls_config.test.ts | 1 + .../resources/base/bin/kibana-docker | 2 + src/plugins/bfetch/server/plugin.ts | 21 +- src/setup_node_env/exit_on_warning.js | 37 ++- test/api_integration/services/supertest.ts | 20 +- test/common/configure_http2.ts | 85 ++++++ test/common/services/deployment.ts | 12 + test/functional/apps/console/config.ts | 5 +- test/functional/apps/home/config.ts | 5 +- test/functional/services/supertest.ts | 20 +- x-pack/test/common/services/spaces.ts | 11 + yarn.lock | 38 ++- 52 files changed, 1613 insertions(+), 168 deletions(-) create mode 100644 packages/core/http/core-http-router-server-internal/src/strip_illegal_http2_headers.test.ts create mode 100644 packages/core/http/core-http-router-server-internal/src/strip_illegal_http2_headers.ts rename packages/kbn-cli-dev-mode/src/{base_path_proxy_server.ts => base_path_proxy/http1.ts} (92%) create mode 100644 packages/kbn-cli-dev-mode/src/base_path_proxy/http2.ts create mode 100644 packages/kbn-cli-dev-mode/src/base_path_proxy/index.ts create mode 100644 packages/kbn-cli-dev-mode/src/base_path_proxy/types.ts create mode 100644 packages/kbn-cli-dev-mode/src/base_path_proxy/utils.ts rename packages/kbn-cli-dev-mode/src/integration_tests/{base_path_proxy_server.test.ts => http1_base_path_proxy_server.test.ts} (94%) create mode 100644 packages/kbn-cli-dev-mode/src/integration_tests/http2_base_path_proxy_server.test.ts create mode 100644 packages/kbn-server-http-tools/src/ssl/constants.ts create mode 100644 src/core/server/integration_tests/http/http2_protocol.test.ts create mode 100644 test/common/configure_http2.ts diff --git a/docs/setup/settings.asciidoc b/docs/setup/settings.asciidoc index a1f0a4ebed8a4..4405b746ca34f 100644 --- a/docs/setup/settings.asciidoc +++ b/docs/setup/settings.asciidoc @@ -456,6 +456,14 @@ identifies this {kib} instance. *Default: `"your-hostname"`* {kib} is served by a back end server. This setting specifies the port to use. *Default: `5601`* +`server.protocol`:: +experimental[] The http protocol to use, either `http1` or `http2`. Set to `http2` to enable `HTTP/2` support for the {kib} server. +*Default: `http1`* ++ +NOTE: By default, enabling `http2` requires a valid `h2c` configuration, meaning that TLS must be enabled via <> +and <>, if specified, must contain at least `TLSv1.2` or `TLSv1.3`. Strict validation of +the `h2c` setup can be disabled by adding `server.http2.allowUnsecure: true` to the configuration. + [[server-requestId-allowFromAnyIp]] `server.requestId.allowFromAnyIp`:: Sets whether or not the `X-Opaque-Id` header should be trusted from any IP address for identifying requests in logs and forwarded to Elasticsearch. diff --git a/package.json b/package.json index aebe778468d9f..8015b38f52aa2 100644 --- a/package.json +++ b/package.json @@ -1627,6 +1627,8 @@ "html": "1.0.0", "html-loader": "^1.3.2", "http-proxy": "^1.18.1", + "http2-proxy": "^5.0.53", + "http2-wrapper": "^2.2.1", "ignore": "^5.3.0", "jest": "^29.6.1", "jest-canvas-mock": "^2.5.2", diff --git a/packages/core/http/core-http-router-server-internal/src/request.test.ts b/packages/core/http/core-http-router-server-internal/src/request.test.ts index f895cb0e2fde7..921509adfe589 100644 --- a/packages/core/http/core-http-router-server-internal/src/request.test.ts +++ b/packages/core/http/core-http-router-server-internal/src/request.test.ts @@ -217,11 +217,23 @@ describe('CoreKibanaRequest', () => { }); describe('route.protocol property', () => { - it('return a static value for now as only http1 is supported', () => { + it('return the correct value for http/1.0 requests', () => { const request = hapiMocks.createRequest({ raw: { req: { - httpVersion: '2.0', + httpVersion: '1.0', + }, + }, + }); + const kibanaRequest = CoreKibanaRequest.from(request); + + expect(kibanaRequest.protocol).toEqual('http1'); + }); + it('return the correct value for http/1.1 requests', () => { + const request = hapiMocks.createRequest({ + raw: { + req: { + httpVersion: '1.1', }, }, }); @@ -229,6 +241,18 @@ describe('CoreKibanaRequest', () => { expect(kibanaRequest.protocol).toEqual('http1'); }); + it('return the correct value for http/2 requests', () => { + const request = hapiMocks.createRequest({ + raw: { + req: { + httpVersion: '2.0', + }, + }, + }); + const kibanaRequest = CoreKibanaRequest.from(request); + + expect(kibanaRequest.protocol).toEqual('http2'); + }); }); describe('route.options.authRequired property', () => { diff --git a/packages/core/http/core-http-router-server-internal/src/request.ts b/packages/core/http/core-http-router-server-internal/src/request.ts index d3274b0a2a1fe..76994322352cc 100644 --- a/packages/core/http/core-http-router-server-internal/src/request.ts +++ b/packages/core/http/core-http-router-server-internal/src/request.ts @@ -173,8 +173,7 @@ export class CoreKibanaRequest< }); this.httpVersion = isRealReq ? request.raw.req.httpVersion : '1.0'; - // hardcoded for now as only supporting http1 - this.protocol = 'http1'; + this.protocol = getProtocolFromHttpVersion(this.httpVersion); this.route = deepFreeze(this.getRouteInfo(request)); this.socket = isRealReq @@ -374,3 +373,7 @@ function sanitizeRequest(req: Request): { query: unknown; params: unknown; body: body: req.payload, }; } + +function getProtocolFromHttpVersion(httpVersion: string): HttpProtocol { + return httpVersion.split('.')[0] === '2' ? 'http2' : 'http1'; +} diff --git a/packages/core/http/core-http-router-server-internal/src/router.ts b/packages/core/http/core-http-router-server-internal/src/router.ts index 9db0829355587..cd8fa84f662e3 100644 --- a/packages/core/http/core-http-router-server-internal/src/router.ts +++ b/packages/core/http/core-http-router-server-internal/src/router.ts @@ -35,6 +35,7 @@ import { HapiResponseAdapter } from './response_adapter'; import { wrapErrors } from './error_wrapper'; import { Method } from './versioned_router/types'; import { prepareRouteConfigValidation } from './util'; +import { stripIllegalHttp2Headers } from './strip_illegal_http2_headers'; export type ContextEnhancer< P, @@ -265,6 +266,14 @@ export class Router { + let logger: MockedLogger; + + beforeEach(() => { + logger = loggerMock.create(); + }); + + it('removes illegal http2 headers', () => { + const headers = { + 'x-foo': 'bar', + 'x-hello': 'dolly', + connection: 'keep-alive', + 'proxy-connection': 'keep-alive', + 'keep-alive': 'true', + upgrade: 'probably', + 'transfer-encoding': 'chunked', + 'http2-settings': 'yeah', + }; + const output = stripIllegalHttp2Headers({ + headers, + isDev: false, + logger, + requestContext: 'requestContext', + }); + + expect(output).toEqual({ + 'x-foo': 'bar', + 'x-hello': 'dolly', + }); + }); + + it('ignores case when detecting headers', () => { + const headers = { + 'x-foo': 'bar', + 'x-hello': 'dolly', + Connection: 'keep-alive', + 'Proxy-Connection': 'keep-alive', + 'kEeP-AlIvE': 'true', + }; + const output = stripIllegalHttp2Headers({ + headers, + isDev: false, + logger, + requestContext: 'requestContext', + }); + + expect(output).toEqual({ + 'x-foo': 'bar', + 'x-hello': 'dolly', + }); + }); + + it('logs a warning about the illegal header when in dev mode', () => { + const headers = { + 'x-foo': 'bar', + Connection: 'keep-alive', + }; + stripIllegalHttp2Headers({ + headers, + isDev: true, + logger, + requestContext: 'requestContext', + }); + + expect(logger.warn).toHaveBeenCalledTimes(1); + expect(logger.warn).toHaveBeenCalledWith( + `Handler for "requestContext" returned an illegal http2 header: Connection. Please check "request.protocol" in handlers before assigning connection headers` + ); + }); + + it('does not log a warning about the illegal header when not in dev mode', () => { + const headers = { + 'x-foo': 'bar', + Connection: 'keep-alive', + }; + stripIllegalHttp2Headers({ + headers, + isDev: false, + logger, + requestContext: 'requestContext', + }); + + expect(logger.warn).not.toHaveBeenCalled(); + }); + + it('does not mutate the original headers', () => { + const headers = { + 'x-foo': 'bar', + Connection: 'keep-alive', + }; + stripIllegalHttp2Headers({ + headers, + isDev: true, + logger, + requestContext: 'requestContext', + }); + + expect(headers).toEqual({ + 'x-foo': 'bar', + Connection: 'keep-alive', + }); + }); +}); diff --git a/packages/core/http/core-http-router-server-internal/src/strip_illegal_http2_headers.ts b/packages/core/http/core-http-router-server-internal/src/strip_illegal_http2_headers.ts new file mode 100644 index 0000000000000..75517fc498254 --- /dev/null +++ b/packages/core/http/core-http-router-server-internal/src/strip_illegal_http2_headers.ts @@ -0,0 +1,49 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import type { Logger } from '@kbn/logging'; +import type { ResponseHeaders } from '@kbn/core-http-server'; + +// from https://github.com/nodejs/node/blob/v22.2.0/lib/internal/http2/util.js#L557 +const ILLEGAL_HTTP2_CONNECTION_HEADERS = new Set([ + 'connection', + 'proxy-connection', + 'keep-alive', + 'upgrade', + 'transfer-encoding', + 'http2-settings', +]); + +/** + * Return a new version of the provided headers, with all illegal http2 headers removed. + * If `isDev` is `true`, will also log a warning if such header is encountered. + */ +export const stripIllegalHttp2Headers = ({ + headers, + isDev, + logger, + requestContext, +}: { + headers: ResponseHeaders; + isDev: boolean; + logger: Logger; + requestContext: string; +}): ResponseHeaders => { + return Object.entries(headers).reduce((output, [headerName, headerValue]) => { + if (ILLEGAL_HTTP2_CONNECTION_HEADERS.has(headerName.toLowerCase())) { + if (isDev) { + logger.warn( + `Handler for "${requestContext}" returned an illegal http2 header: ${headerName}. Please check "request.protocol" in handlers before assigning connection headers` + ); + } + } else { + output[headerName as keyof ResponseHeaders] = headerValue; + } + return output; + }, {} as ResponseHeaders); +}; diff --git a/packages/core/http/core-http-router-server-internal/tsconfig.json b/packages/core/http/core-http-router-server-internal/tsconfig.json index e4a70cbcddeec..f14271e7bb53a 100644 --- a/packages/core/http/core-http-router-server-internal/tsconfig.json +++ b/packages/core/http/core-http-router-server-internal/tsconfig.json @@ -17,7 +17,8 @@ "@kbn/hapi-mocks", "@kbn/core-logging-server-mocks", "@kbn/logging", - "@kbn/core-http-common" + "@kbn/core-http-common", + "@kbn/logging-mocks" ], "exclude": [ "target/**/*", diff --git a/packages/core/http/core-http-server-internal/src/__snapshots__/http_config.test.ts.snap b/packages/core/http/core-http-server-internal/src/__snapshots__/http_config.test.ts.snap index 5dbbe1fd9ee2e..34cdcd15db7df 100644 --- a/packages/core/http/core-http-server-internal/src/__snapshots__/http_config.test.ts.snap +++ b/packages/core/http/core-http-server-internal/src/__snapshots__/http_config.test.ts.snap @@ -70,6 +70,9 @@ Object { }, }, "host": "localhost", + "http2": Object { + "allowUnsecure": false, + }, "keepaliveTimeout": 120000, "maxPayload": ByteSizeValue { "valueInBytes": 1048576, @@ -80,6 +83,7 @@ Object { }, "payloadTimeout": 20000, "port": 5601, + "protocol": "http1", "requestId": Object { "allowFromAnyIp": false, "ipAllowlist": Array [], diff --git a/packages/core/http/core-http-server-internal/src/http_config.test.ts b/packages/core/http/core-http-server-internal/src/http_config.test.ts index d2bac7e8cf1c0..97da37fe703b0 100644 --- a/packages/core/http/core-http-server-internal/src/http_config.test.ts +++ b/packages/core/http/core-http-server-internal/src/http_config.test.ts @@ -571,6 +571,73 @@ describe('cdn', () => { }); }); +describe('http2 protocol', () => { + it('throws if http2 is enabled but TLS is not', () => { + expect(() => + config.schema.validate({ + protocol: 'http2', + ssl: { + enabled: false, + }, + }) + ).toThrowErrorMatchingInlineSnapshot( + `"http2 requires TLS to be enabled. Use 'http2.allowUnsecure: true' to allow running http2 without a valid h2c setup"` + ); + }); + it('throws if http2 is enabled but TLS has no suitable versions', () => { + expect(() => + config.schema.validate({ + protocol: 'http2', + ssl: { + enabled: true, + supportedProtocols: ['TLSv1.1'], + certificate: '/path/to/certificate', + key: '/path/to/key', + }, + }) + ).toThrowErrorMatchingInlineSnapshot( + `"http2 requires 'ssl.supportedProtocols' to include TLSv1.2 or TLSv1.3. Use 'http2.allowUnsecure: true' to allow running http2 without a valid h2c setup"` + ); + }); + it('does not throws if http2 is enabled and TLS is not if http2.allowUnsecure is true', () => { + expect( + config.schema.validate({ + protocol: 'http2', + http2: { + allowUnsecure: true, + }, + ssl: { + enabled: false, + }, + }) + ).toEqual( + expect.objectContaining({ + protocol: 'http2', + }) + ); + }); + it('does not throws if supportedProtocols are not valid for h2c if http2.allowUnsecure is true', () => { + expect( + config.schema.validate({ + protocol: 'http2', + http2: { + allowUnsecure: true, + }, + ssl: { + enabled: true, + supportedProtocols: ['TLSv1.1'], + certificate: '/path/to/certificate', + key: '/path/to/key', + }, + }) + ).toEqual( + expect.objectContaining({ + protocol: 'http2', + }) + ); + }); +}); + describe('HttpConfig', () => { it('converts customResponseHeaders to strings or arrays of strings', () => { const httpSchema = config.schema; diff --git a/packages/core/http/core-http-server-internal/src/http_config.ts b/packages/core/http/core-http-server-internal/src/http_config.ts index 06e021c8acdb9..746420fad810a 100644 --- a/packages/core/http/core-http-server-internal/src/http_config.ts +++ b/packages/core/http/core-http-server-internal/src/http_config.ts @@ -6,23 +6,21 @@ * Side Public License, v 1. */ +import { EOL, hostname } from 'node:os'; +import url, { URL } from 'node:url'; +import type { Duration } from 'moment'; import { ByteSizeValue, offeringBasedSchema, schema, TypeOf } from '@kbn/config-schema'; -import { IHttpConfig, SslConfig, sslSchema } from '@kbn/server-http-tools'; +import { IHttpConfig, SslConfig, sslSchema, TLS_V1_2, TLS_V1_3 } from '@kbn/server-http-tools'; import type { ServiceConfigDescriptor } from '@kbn/core-base-server-internal'; import { uuidRegexp } from '@kbn/core-base-server-internal'; -import type { ICspConfig, IExternalUrlConfig } from '@kbn/core-http-server'; - -import { hostname, EOL } from 'node:os'; -import url, { URL } from 'node:url'; - -import type { Duration } from 'moment'; +import type { HttpProtocol, ICspConfig, IExternalUrlConfig } from '@kbn/core-http-server'; import type { IHttpEluMonitorConfig } from '@kbn/core-http-server/src/elu_monitor'; import type { HandlerResolutionStrategy } from '@kbn/core-http-router-server-internal'; -import { CspConfigType, CspConfig } from './csp'; +import { CspConfig, CspConfigType } from './csp'; import { ExternalUrlConfig } from './external_url'; import { - securityResponseHeadersSchema, parseRawSecurityResponseHeadersConfig, + securityResponseHeadersSchema, } from './security_response_headers_config'; import { CdnConfig } from './cdn_config'; @@ -123,6 +121,9 @@ const configSchema = schema.object( } }, }), + protocol: schema.oneOf([schema.literal('http1'), schema.literal('http2')], { + defaultValue: 'http1', + }), host: schema.string({ defaultValue: 'localhost', hostname: true, @@ -144,6 +145,9 @@ const configSchema = schema.object( payloadTimeout: schema.number({ defaultValue: 20 * SECOND, }), + http2: schema.object({ + allowUnsecure: schema.boolean({ defaultValue: false }), + }), compression: schema.object({ enabled: schema.boolean({ defaultValue: true }), brotli: schema.object({ @@ -259,6 +263,13 @@ const configSchema = schema.object( return 'cannot use [compression.referrerWhitelist] when [compression.enabled] is set to false'; } + if (rawConfig.protocol === 'http2' && !rawConfig.http2.allowUnsecure) { + const err = ensureValidTLSConfigForH2C(rawConfig.ssl); + if (err) { + return err; + } + } + if ( rawConfig.ssl.enabled && rawConfig.ssl.redirectHttpFromPort !== undefined && @@ -285,6 +296,7 @@ export const config: ServiceConfigDescriptor = { export class HttpConfig implements IHttpConfig { public name: string; public autoListen: boolean; + public protocol: HttpProtocol; public host: string; public keepaliveTimeout: number; public socketTimeout: number; @@ -352,6 +364,7 @@ export class HttpConfig implements IHttpConfig { ); this.maxPayload = rawHttpConfig.maxPayload; this.name = rawHttpConfig.name; + this.protocol = rawHttpConfig.protocol; this.basePath = rawHttpConfig.basePath; this.publicBaseUrl = rawHttpConfig.publicBaseUrl; this.keepaliveTimeout = rawHttpConfig.keepaliveTimeout; @@ -378,3 +391,16 @@ export class HttpConfig implements IHttpConfig { const convertHeader = (entry: any): string => { return typeof entry === 'object' ? JSON.stringify(entry) : String(entry); }; + +const ensureValidTLSConfigForH2C = (tlsConfig: TypeOf): string | undefined => { + if (!tlsConfig.enabled) { + return `http2 requires TLS to be enabled. Use 'http2.allowUnsecure: true' to allow running http2 without a valid h2c setup`; + } + if ( + !tlsConfig.supportedProtocols.includes(TLS_V1_2) && + !tlsConfig.supportedProtocols.includes(TLS_V1_3) + ) { + return `http2 requires 'ssl.supportedProtocols' to include ${TLS_V1_2} or ${TLS_V1_3}. Use 'http2.allowUnsecure: true' to allow running http2 without a valid h2c setup`; + } + return undefined; +}; diff --git a/packages/core/http/core-http-server/src/http_contract.ts b/packages/core/http/core-http-server/src/http_contract.ts index 308ba2dd48785..09be2d4c2933a 100644 --- a/packages/core/http/core-http-server/src/http_contract.ts +++ b/packages/core/http/core-http-server/src/http_contract.ts @@ -408,5 +408,6 @@ export interface HttpServerInfo { * (Only supporting http1 for now) * * - http1: regroups all http/1.x protocols + * - http2: h2 */ -export type HttpProtocol = 'http1'; +export type HttpProtocol = 'http1' | 'http2'; diff --git a/packages/kbn-cli-dev-mode/src/base_path_proxy_server.ts b/packages/kbn-cli-dev-mode/src/base_path_proxy/http1.ts similarity index 92% rename from packages/kbn-cli-dev-mode/src/base_path_proxy_server.ts rename to packages/kbn-cli-dev-mode/src/base_path_proxy/http1.ts index f9aaad7923152..ffbf451384cbf 100644 --- a/packages/kbn-cli-dev-mode/src/base_path_proxy_server.ts +++ b/packages/kbn-cli-dev-mode/src/base_path_proxy/http1.ts @@ -11,28 +11,18 @@ import { Agent as HttpsAgent, ServerOptions as TlsOptions } from 'https'; import apm from 'elastic-apm-node'; import { Server, Request } from '@hapi/hapi'; import HapiProxy from '@hapi/h2o2'; -import { sampleSize } from 'lodash'; -import * as Rx from 'rxjs'; import { take } from 'rxjs'; import { ByteSizeValue } from '@kbn/config-schema'; import { createServer, getServerOptions } from '@kbn/server-http-tools'; -import { DevConfig, HttpConfig } from './config'; -import { Log } from './log'; +import { DevConfig, HttpConfig } from '../config'; +import { Log } from '../log'; +import { getRandomBasePath } from './utils'; +import type { BasePathProxyServer, BasePathProxyServerOptions } from './types'; const ONE_GIGABYTE = 1024 * 1024 * 1024; -const alphabet = 'abcdefghijklmnopqrztuvwxyz'.split(''); -// Thank you, Spencer! :elasticheart: -const getRandomBasePath = () => - Math.random() * 100 < 1 ? 'spalger' : sampleSize(alphabet, 3).join(''); - -export interface BasePathProxyServerOptions { - shouldRedirectFromOldBasePath: (path: string) => boolean; - delayUntil: () => Rx.Observable; -} - -export class BasePathProxyServer { +export class Http1BasePathProxyServer implements BasePathProxyServer { private readonly httpConfig: HttpConfig; private server?: Server; private httpsAgent?: HttpsAgent; diff --git a/packages/kbn-cli-dev-mode/src/base_path_proxy/http2.ts b/packages/kbn-cli-dev-mode/src/base_path_proxy/http2.ts new file mode 100644 index 0000000000000..77119df9c22f9 --- /dev/null +++ b/packages/kbn-cli-dev-mode/src/base_path_proxy/http2.ts @@ -0,0 +1,183 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import Url from 'url'; +import { Agent as HttpsAgent, ServerOptions as TlsOptions } from 'https'; +import http2, { Agent as Http2Agent, AutoRequestOptions } from 'http2-wrapper'; +import http2Proxy from 'http2-proxy'; +import { take } from 'rxjs'; +import { getServerOptions, getServerTLSOptions } from '@kbn/server-http-tools'; + +import { DevConfig, HttpConfig } from '../config'; +import { Log } from '../log'; +import type { BasePathProxyServer, BasePathProxyServerOptions } from './types'; +import { getRandomBasePath } from './utils'; + +export class Http2BasePathProxyServer implements BasePathProxyServer { + private readonly httpConfig: HttpConfig; + private server?: http2.Http2SecureServer; + private httpsAgent?: HttpsAgent; + + constructor( + private readonly log: Log, + httpConfig: HttpConfig, + private readonly devConfig: DevConfig + ) { + this.httpConfig = { + ...httpConfig, + basePath: httpConfig.basePath ?? `/${getRandomBasePath()}`, + }; + } + + public get basePath() { + return this.httpConfig.basePath; + } + + public get targetPort() { + return this.devConfig.basePathProxyTargetPort; + } + + public get host() { + return this.httpConfig.host; + } + + public get port() { + return this.httpConfig.port; + } + + public async start(options: BasePathProxyServerOptions) { + const serverOptions = getServerOptions(this.httpConfig); + + if (this.httpConfig.ssl.enabled) { + const tlsOptions = serverOptions.tls as TlsOptions; + this.httpsAgent = new HttpsAgent({ + ca: tlsOptions.ca, + cert: tlsOptions.cert, + key: tlsOptions.key, + passphrase: tlsOptions.passphrase, + rejectUnauthorized: false, + }); + } + + await this.setupServer(options); + + this.log.write( + `basepath proxy server running at ${Url.format({ + protocol: this.httpConfig.ssl.enabled ? 'https' : 'http', + host: this.httpConfig.host, + pathname: this.httpConfig.basePath, + })}` + ); + } + + public async stop() { + if (this.server !== undefined) { + await new Promise((resolve, reject) => { + this.server!.close((err) => { + if (err) { + reject(err); + } else { + resolve(); + } + }); + }); + this.server = undefined; + } + + if (this.httpsAgent !== undefined) { + this.httpsAgent.destroy(); + this.httpsAgent = undefined; + } + } + + private async setupServer({ delayUntil }: Readonly) { + const tlsOptions = getServerTLSOptions(this.httpConfig.ssl); + this.server = http2.createSecureServer({ + ...tlsOptions, + rejectUnauthorized: false, + allowHTTP1: true, + }); + + const server = this.server; + + const http2Agent = new Http2Agent(); + + server.on('error', (e) => { + this.log.bad('error', `error initializing the base path server: ${e.message}`); + throw e; + }); + + server.listen(this.httpConfig.port, this.httpConfig.host, () => { + server.on('request', (inboundRequest, inboundResponse) => { + const requestPath = Url.parse(inboundRequest.url).path ?? '/'; + + if (requestPath === '/') { + // Always redirect from root URL to the URL with basepath. + inboundResponse.writeHead(302, { + location: this.httpConfig.basePath, + }); + inboundResponse.end(); + } else if (requestPath.startsWith(this.httpConfig.basePath!)) { + // Perform proxy request if requested path is within base path + http2Proxy.web( + inboundRequest, + inboundResponse, + { + protocol: 'https', + hostname: this.httpConfig.host, + port: this.devConfig.basePathProxyTargetPort, + onReq: async (request, options) => { + // Before we proxy request to a target port we may want to wait until some + // condition is met (e.g. until target listener is ready). + await delayUntil().pipe(take(1)).toPromise(); + + const proxyOptions = { + ...options, + ...tlsOptions, + rejectUnauthorized: false, + path: options.path, + agent: { + https: this.httpsAgent ?? false, + http2: http2Agent, + }, + } as AutoRequestOptions; + + const proxyReq = await http2.auto(proxyOptions, (proxyRes) => { + // `http2-proxy` doesn't automatically remove pseudo-headers + for (const name in proxyRes.headers) { + if (name.startsWith(':')) { + delete proxyRes.headers[name]; + } + } + }); + + // `http2-proxy` waits for the `socket` event before calling `h2request.end()` + proxyReq.flushHeaders(); + return proxyReq; + }, + onRes: async (request, response, _proxyRes) => { + // wrong type - proxyRes is declared as Http.ServerResponse but is Http.IncomingMessage + const proxyRes = _proxyRes as unknown as http2.IncomingMessage; + response.setHeader('x-powered-by', 'kibana-base-path-server'); + response.writeHead(proxyRes.statusCode!, proxyRes.headers); + proxyRes.pipe(response); + }, + }, + (err, req, res) => { + if (err) { + this.log.bad('warning', 'base path proxy: error forwarding request', err); + res.statusCode = (err as any).statusCode || 500; + res.end((err as any).stack ?? err.message); + } + } + ); + } + }); + }); + } +} diff --git a/packages/kbn-cli-dev-mode/src/base_path_proxy/index.ts b/packages/kbn-cli-dev-mode/src/base_path_proxy/index.ts new file mode 100644 index 0000000000000..fe0c36b3579d7 --- /dev/null +++ b/packages/kbn-cli-dev-mode/src/base_path_proxy/index.ts @@ -0,0 +1,33 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import type { Log } from '../log'; +import type { CliDevConfig } from '../config'; +import type { BasePathProxyServer } from './types'; +import { Http1BasePathProxyServer } from './http1'; +import { Http2BasePathProxyServer } from './http2'; + +export type { BasePathProxyServer, BasePathProxyServerOptions } from './types'; +export { Http1BasePathProxyServer } from './http1'; +export { Http2BasePathProxyServer } from './http2'; + +export const getBasePathProxyServer = ({ + log, + httpConfig, + devConfig, +}: { + log: Log; + httpConfig: CliDevConfig['http']; + devConfig: CliDevConfig['dev']; +}): BasePathProxyServer => { + if (httpConfig.protocol === 'http2') { + return new Http2BasePathProxyServer(log, httpConfig, devConfig); + } else { + return new Http1BasePathProxyServer(log, httpConfig, devConfig); + } +}; diff --git a/packages/kbn-cli-dev-mode/src/base_path_proxy/types.ts b/packages/kbn-cli-dev-mode/src/base_path_proxy/types.ts new file mode 100644 index 0000000000000..ca343e2744fb0 --- /dev/null +++ b/packages/kbn-cli-dev-mode/src/base_path_proxy/types.ts @@ -0,0 +1,24 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import type { Observable } from 'rxjs'; + +export interface BasePathProxyServer { + readonly basePath: string | undefined; + readonly targetPort: number; + readonly host: string; + readonly port: number; + + start(options: BasePathProxyServerOptions): Promise; + stop(): Promise; +} + +export interface BasePathProxyServerOptions { + shouldRedirectFromOldBasePath: (path: string) => boolean; + delayUntil: () => Observable; +} diff --git a/packages/kbn-cli-dev-mode/src/base_path_proxy/utils.ts b/packages/kbn-cli-dev-mode/src/base_path_proxy/utils.ts new file mode 100644 index 0000000000000..afd05bb90b4ae --- /dev/null +++ b/packages/kbn-cli-dev-mode/src/base_path_proxy/utils.ts @@ -0,0 +1,15 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import { sampleSize } from 'lodash'; + +const alphabet = 'abcdefghijklmnopqrztuvwxyz'.split(''); + +// Thank you, Spencer! :elasticheart: +export const getRandomBasePath = () => + Math.random() * 100 < 1 ? 'spalger' : sampleSize(alphabet, 3).join(''); diff --git a/packages/kbn-cli-dev-mode/src/cli_dev_mode.test.ts b/packages/kbn-cli-dev-mode/src/cli_dev_mode.test.ts index 769242436a270..a6b73571f05ca 100644 --- a/packages/kbn-cli-dev-mode/src/cli_dev_mode.test.ts +++ b/packages/kbn-cli-dev-mode/src/cli_dev_mode.test.ts @@ -29,8 +29,8 @@ const { Optimizer } = jest.requireMock('./optimizer'); jest.mock('./dev_server'); const { DevServer } = jest.requireMock('./dev_server'); -jest.mock('./base_path_proxy_server'); -const { BasePathProxyServer } = jest.requireMock('./base_path_proxy_server'); +jest.mock('./base_path_proxy'); +const { getBasePathProxyServer } = jest.requireMock('./base_path_proxy'); jest.mock('@kbn/ci-stats-reporter'); const { CiStatsReporter } = jest.requireMock('@kbn/ci-stats-reporter'); @@ -47,7 +47,7 @@ let log: TestLog; beforeEach(() => { process.argv = ['node', './script', 'foo', 'bar', 'baz']; log = new TestLog(); - BasePathProxyServer.mockImplementation(() => mockBasePathProxy); + getBasePathProxyServer.mockImplementation(() => mockBasePathProxy); }); afterEach(() => { @@ -142,7 +142,7 @@ it('passes correct args to sub-classes', () => { ] `); - expect(BasePathProxyServer).not.toHaveBeenCalled(); + expect(getBasePathProxyServer).not.toHaveBeenCalled(); expect(log.messages).toMatchInlineSnapshot(`Array []`); }); @@ -163,13 +163,15 @@ it('disables the watcher', () => { it('enables the basePath proxy', () => { new CliDevMode(createOptions({ cliArgs: { basePath: true } })); - expect(BasePathProxyServer).toHaveBeenCalledTimes(1); - expect(BasePathProxyServer.mock.calls[0]).toMatchInlineSnapshot(` + expect(getBasePathProxyServer).toHaveBeenCalledTimes(1); + expect(getBasePathProxyServer.mock.calls[0]).toMatchInlineSnapshot(` Array [ - , - Object {}, Object { - "basePathProxyTargetPort": 9000, + "devConfig": Object { + "basePathProxyTargetPort": 9000, + }, + "httpConfig": Object {}, + "log": , }, ] `); diff --git a/packages/kbn-cli-dev-mode/src/cli_dev_mode.ts b/packages/kbn-cli-dev-mode/src/cli_dev_mode.ts index 05791256f1ff1..f5e731f6a55af 100644 --- a/packages/kbn-cli-dev-mode/src/cli_dev_mode.ts +++ b/packages/kbn-cli-dev-mode/src/cli_dev_mode.ts @@ -29,7 +29,7 @@ import { Log, CliLog } from './log'; import { Optimizer } from './optimizer'; import { DevServer } from './dev_server'; import { Watcher } from './watcher'; -import { BasePathProxyServer } from './base_path_proxy_server'; +import { getBasePathProxyServer, type BasePathProxyServer } from './base_path_proxy'; import { shouldRedirectFromOldBasePath } from './should_redirect_from_old_base_path'; import { CliDevConfig } from './config'; @@ -110,7 +110,11 @@ export class CliDevMode { this.log = log || new CliLog(!!cliArgs.silent); if (cliArgs.basePath) { - this.basePathProxy = new BasePathProxyServer(this.log, config.http, config.dev); + this.basePathProxy = getBasePathProxyServer({ + log: this.log, + devConfig: config.dev, + httpConfig: config.http, + }); } this.watcher = new Watcher({ diff --git a/packages/kbn-cli-dev-mode/src/config/http_config.ts b/packages/kbn-cli-dev-mode/src/config/http_config.ts index 0c2f86a89b105..3f2ac9542a328 100644 --- a/packages/kbn-cli-dev-mode/src/config/http_config.ts +++ b/packages/kbn-cli-dev-mode/src/config/http_config.ts @@ -12,6 +12,9 @@ import { Duration } from 'moment'; export const httpConfigSchema = schema.object( { + protocol: schema.oneOf([schema.literal('http1'), schema.literal('http2')], { + defaultValue: 'http1', + }), host: schema.string({ defaultValue: 'localhost', hostname: true, @@ -49,6 +52,7 @@ export const httpConfigSchema = schema.object( export type HttpConfigType = TypeOf; export class HttpConfig implements IHttpConfig { + protocol: 'http1' | 'http2'; basePath?: string; host: string; port: number; @@ -62,6 +66,7 @@ export class HttpConfig implements IHttpConfig { restrictInternalApis: boolean; constructor(rawConfig: HttpConfigType) { + this.protocol = rawConfig.protocol; this.basePath = rawConfig.basePath; this.host = rawConfig.host; this.port = rawConfig.port; diff --git a/packages/kbn-cli-dev-mode/src/integration_tests/base_path_proxy_server.test.ts b/packages/kbn-cli-dev-mode/src/integration_tests/http1_base_path_proxy_server.test.ts similarity index 94% rename from packages/kbn-cli-dev-mode/src/integration_tests/base_path_proxy_server.test.ts rename to packages/kbn-cli-dev-mode/src/integration_tests/http1_base_path_proxy_server.test.ts index 432f67a75f1b0..a074395d86554 100644 --- a/packages/kbn-cli-dev-mode/src/integration_tests/base_path_proxy_server.test.ts +++ b/packages/kbn-cli-dev-mode/src/integration_tests/http1_base_path_proxy_server.test.ts @@ -13,13 +13,13 @@ import supertest from 'supertest'; import { getServerOptions, createServer, type IHttpConfig } from '@kbn/server-http-tools'; import { ByteSizeValue } from '@kbn/config-schema'; -import { BasePathProxyServer, BasePathProxyServerOptions } from '../base_path_proxy_server'; +import { Http1BasePathProxyServer, BasePathProxyServerOptions } from '../base_path_proxy'; import { DevConfig } from '../config/dev_config'; import { TestLog } from '../log'; -describe('BasePathProxyServer', () => { +describe('Http1BasePathProxyServer', () => { let server: Server; - let proxyServer: BasePathProxyServer; + let proxyServer: Http1BasePathProxyServer; let logger: TestLog; let config: IHttpConfig; let basePath: string; @@ -29,6 +29,7 @@ describe('BasePathProxyServer', () => { logger = new TestLog(); config = { + protocol: 'http1', host: '127.0.0.1', port: 10012, shutdownTimeout: moment.duration(30, 'seconds'), @@ -51,7 +52,7 @@ describe('BasePathProxyServer', () => { // setup and start the proxy server const proxyConfig: IHttpConfig = { ...config, port: 10013 }; const devConfig = new DevConfig({ basePathProxyTarget: config.port }); - proxyServer = new BasePathProxyServer(logger, proxyConfig, devConfig); + proxyServer = new Http1BasePathProxyServer(logger, proxyConfig, devConfig); const options: BasePathProxyServerOptions = { shouldRedirectFromOldBasePath: () => true, delayUntil: () => EMPTY, @@ -322,14 +323,18 @@ describe('BasePathProxyServer', () => { }); describe('shouldRedirect', () => { - let proxyServerWithoutShouldRedirect: BasePathProxyServer; + let proxyServerWithoutShouldRedirect: Http1BasePathProxyServer; let proxyWithoutShouldRedirectSupertest: supertest.Agent; beforeEach(async () => { // setup and start a proxy server which does not use "shouldRedirectFromOldBasePath" const proxyConfig: IHttpConfig = { ...config, port: 10004 }; const devConfig = new DevConfig({ basePathProxyTarget: config.port }); - proxyServerWithoutShouldRedirect = new BasePathProxyServer(logger, proxyConfig, devConfig); + proxyServerWithoutShouldRedirect = new Http1BasePathProxyServer( + logger, + proxyConfig, + devConfig + ); const options: Readonly = { shouldRedirectFromOldBasePath: () => false, // Return false to not redirect delayUntil: () => EMPTY, @@ -365,14 +370,14 @@ describe('BasePathProxyServer', () => { }); describe('constructor option for sending in a custom basePath', () => { - let proxyServerWithFooBasePath: BasePathProxyServer; + let proxyServerWithFooBasePath: Http1BasePathProxyServer; let proxyWithFooBasePath: supertest.Agent; beforeEach(async () => { // setup and start a proxy server which uses a basePath of "foo" const proxyConfig = { ...config, port: 10004, basePath: '/foo' }; // <-- "foo" here in basePath const devConfig = new DevConfig({ basePathProxyTarget: config.port }); - proxyServerWithFooBasePath = new BasePathProxyServer(logger, proxyConfig, devConfig); + proxyServerWithFooBasePath = new Http1BasePathProxyServer(logger, proxyConfig, devConfig); const options: Readonly = { shouldRedirectFromOldBasePath: () => true, delayUntil: () => EMPTY, diff --git a/packages/kbn-cli-dev-mode/src/integration_tests/http2_base_path_proxy_server.test.ts b/packages/kbn-cli-dev-mode/src/integration_tests/http2_base_path_proxy_server.test.ts new file mode 100644 index 0000000000000..a12bae003baa4 --- /dev/null +++ b/packages/kbn-cli-dev-mode/src/integration_tests/http2_base_path_proxy_server.test.ts @@ -0,0 +1,254 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import { readFileSync } from 'fs'; +import { Server } from '@hapi/hapi'; +import { EMPTY } from 'rxjs'; +import moment from 'moment'; +import supertest from 'supertest'; +import { KBN_CERT_PATH, KBN_KEY_PATH } from '@kbn/dev-utils'; +import { getServerOptions, createServer, type IHttpConfig } from '@kbn/server-http-tools'; +import { ByteSizeValue } from '@kbn/config-schema'; + +import { Http2BasePathProxyServer, BasePathProxyServerOptions } from '../base_path_proxy'; +import { DevConfig } from '../config/dev_config'; +import { TestLog } from '../log'; + +describe('Http2BasePathProxyServer', () => { + let server: Server; + let proxyServer: Http2BasePathProxyServer; + let logger: TestLog; + let config: IHttpConfig; + let basePath: string; + let proxySupertest: supertest.Agent; + + beforeAll(() => { + // required for the self-signed certificates used in testing + process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0'; + }); + + beforeEach(async () => { + logger = new TestLog(); + + config = { + protocol: 'http2', + host: '127.0.0.1', + port: 10012, + shutdownTimeout: moment.duration(30, 'seconds'), + keepaliveTimeout: 1000, + socketTimeout: 1000, + payloadTimeout: 1000, + cors: { + enabled: false, + allowCredentials: false, + allowOrigin: [], + }, + ssl: { + enabled: true, + certificate: await readFileSync(KBN_CERT_PATH, 'utf-8'), + key: await readFileSync(KBN_KEY_PATH, 'utf-8'), + cipherSuites: ['TLS_AES_256_GCM_SHA384'], + }, + maxPayload: new ByteSizeValue(1024), + restrictInternalApis: false, + }; + + const serverOptions = getServerOptions(config); + server = createServer(serverOptions); + + // setup and start the proxy server + const proxyConfig: IHttpConfig = { ...config, port: 10013 }; + const devConfig = new DevConfig({ basePathProxyTarget: config.port }); + proxyServer = new Http2BasePathProxyServer(logger, proxyConfig, devConfig); + const options: BasePathProxyServerOptions = { + shouldRedirectFromOldBasePath: () => true, + delayUntil: () => EMPTY, + }; + await proxyServer.start(options); + + // set the base path or throw if for some unknown reason it is not setup + if (proxyServer.basePath == null) { + throw new Error('Invalid null base path, all tests will fail'); + } else { + basePath = proxyServer.basePath; + } + proxySupertest = supertest(`https://127.0.0.1:${proxyConfig.port}`, { http2: true }); + }); + + afterEach(async () => { + await server.stop(); + await proxyServer.stop(); + jest.clearAllMocks(); + }); + + test('root URL will return a 302 redirect', async () => { + await proxySupertest.get('/').expect(302); + }); + + test('root URL will return a redirect location with exactly 3 characters that are a-z (or spalger)', async () => { + const res = await proxySupertest.get('/'); + const location = res.header.location; + expect(location).toMatch(/^\/(spalger|[a-z]{3})$/); + }); + + test('forwards request with the correct path', async () => { + server.route({ + method: 'GET', + path: `${basePath}/foo/{test}`, + handler: (request, h) => { + return h.response(request.params.test); + }, + }); + await server.start(); + + await proxySupertest + .get(`${basePath}/foo/some-string`) + .expect(200) + .then((res) => { + expect(res.text).toBe('some-string'); + }); + }); + + test('can serve http/1.x requests', async () => { + server.route({ + method: 'GET', + path: `${basePath}/foo/{test}`, + handler: (request, h) => { + return h.response(request.params.test); + }, + }); + await server.start(); + + await proxySupertest + .get(`${basePath}/foo/some-string`) + .http2(false) + .expect(200) + .then((res) => { + expect(res.text).toBe('some-string'); + }); + }); + + test('forwards request with the correct query params', async () => { + server.route({ + method: 'GET', + path: `${basePath}/foo/`, + handler: (request, h) => { + return h.response(request.query); + }, + }); + + await server.start(); + + await proxySupertest + .get(`${basePath}/foo/?bar=test&quux=123`) + .expect(200) + .then((res) => { + expect(res.body).toEqual({ bar: 'test', quux: '123' }); + }); + }); + + test('forwards the request body', async () => { + server.route({ + method: 'POST', + path: `${basePath}/foo/`, + handler: (request, h) => { + return h.response(request.payload); + }, + }); + + await server.start(); + + await proxySupertest + .post(`${basePath}/foo/`) + .send({ + bar: 'test', + baz: 123, + }) + .expect(200) + .then((res) => { + expect(res.body).toEqual({ bar: 'test', baz: 123 }); + }); + }); + + test('returns the correct status code', async () => { + server.route({ + method: 'GET', + path: `${basePath}/foo/`, + handler: (request, h) => { + return h.response({ foo: 'bar' }).code(417); + }, + }); + + await server.start(); + + await proxySupertest + .get(`${basePath}/foo/`) + .expect(417) + .then((res) => { + expect(res.body).toEqual({ foo: 'bar' }); + }); + }); + + test('returns the response headers', async () => { + server.route({ + method: 'GET', + path: `${basePath}/foo/`, + handler: (request, h) => { + return h.response({ foo: 'bar' }).header('foo', 'bar'); + }, + }); + + await server.start(); + + await proxySupertest + .get(`${basePath}/foo/`) + .expect(200) + .then((res) => { + expect(res.get('foo')).toEqual('bar'); + }); + }); + + test('forwards request cancellation', async () => { + let propagated = false; + + let notifyRequestReceived: () => void; + const requestReceived = new Promise((resolve) => { + notifyRequestReceived = resolve; + }); + + let notifyRequestAborted: () => void; + const requestAborted = new Promise((resolve) => { + notifyRequestAborted = resolve; + }); + + server.route({ + method: 'GET', + path: `${basePath}/foo/{test}`, + handler: async (request, h) => { + notifyRequestReceived(); + + request.raw.req.once('aborted', () => { + notifyRequestAborted(); + propagated = true; + }); + return await new Promise((resolve) => undefined); + }, + }); + await server.start(); + + const request = proxySupertest.get(`${basePath}/foo/some-string`).end(); + + await requestReceived; + + request.abort(); + + await requestAborted; + + expect(propagated).toEqual(true); + }); +}); diff --git a/packages/kbn-cli-dev-mode/tsconfig.json b/packages/kbn-cli-dev-mode/tsconfig.json index 86ab4cf2592c4..ac2d5ef7b7892 100644 --- a/packages/kbn-cli-dev-mode/tsconfig.json +++ b/packages/kbn-cli-dev-mode/tsconfig.json @@ -25,6 +25,7 @@ "@kbn/import-resolver", "@kbn/picomatcher", "@kbn/repo-packages", + "@kbn/dev-utils", ], "exclude": [ "target/**/*", diff --git a/packages/kbn-health-gateway-server/src/server/server_config.ts b/packages/kbn-health-gateway-server/src/server/server_config.ts index d887d5421d6a9..a4f996866df9a 100644 --- a/packages/kbn-health-gateway-server/src/server/server_config.ts +++ b/packages/kbn-health-gateway-server/src/server/server_config.ts @@ -70,6 +70,7 @@ export const config: ServiceConfigDescriptor = { }; export class ServerConfig implements IHttpConfig { + readonly protocol = 'http1'; host: string; port: number; maxPayload: ByteSizeValue; diff --git a/packages/kbn-journeys/journey/journey_ftr_harness.ts b/packages/kbn-journeys/journey/journey_ftr_harness.ts index 7ef05c29e9c8e..7339b7ffbca01 100644 --- a/packages/kbn-journeys/journey/journey_ftr_harness.ts +++ b/packages/kbn-journeys/journey/journey_ftr_harness.ts @@ -151,7 +151,7 @@ export class JourneyFtrHarness { private async setupBrowserAndPage() { const browser = await this.getBrowserInstance(); const browserContextArgs = this.auth.isCloud() ? {} : { bypassCSP: true }; - this.context = await browser.newContext(browserContextArgs); + this.context = await browser.newContext({ ...browserContextArgs, ignoreHTTPSErrors: true }); if (this.journeyConfig.shouldAutoLogin()) { const cookie = await this.auth.login(); diff --git a/packages/kbn-server-http-tools/index.ts b/packages/kbn-server-http-tools/index.ts index 7efa00c677015..e471f115215ce 100644 --- a/packages/kbn-server-http-tools/index.ts +++ b/packages/kbn-server-http-tools/index.ts @@ -6,7 +6,13 @@ * Side Public License, v 1. */ -export type { IHttpConfig, ISslConfig, ICorsConfig } from './src/types'; +export type { + IHttpConfig, + ISslConfig, + ICorsConfig, + ServerProtocol, + ServerListener, +} from './src/types'; export { createServer } from './src/create_server'; export { defaultValidationErrorHandler } from './src/default_validation_error_handler'; export { getServerListener } from './src/get_listener'; @@ -14,4 +20,4 @@ export { getServerOptions } from './src/get_server_options'; export { getServerTLSOptions } from './src/get_tls_options'; export { getRequestId } from './src/get_request_id'; export { setTlsConfig } from './src/set_tls_config'; -export { sslSchema, SslConfig } from './src/ssl'; +export { sslSchema, SslConfig, TLS_V1, TLS_V1_1, TLS_V1_2, TLS_V1_3 } from './src/ssl'; diff --git a/packages/kbn-server-http-tools/src/get_listener.test.mocks.ts b/packages/kbn-server-http-tools/src/get_listener.test.mocks.ts index 1fab2d9191367..93266695a18f8 100644 --- a/packages/kbn-server-http-tools/src/get_listener.test.mocks.ts +++ b/packages/kbn-server-http-tools/src/get_listener.test.mocks.ts @@ -45,3 +45,26 @@ jest.doMock('https', () => { createServer: createHttpsServerMock, }; }); + +export const createHttp2SecureServerMock = jest.fn(() => { + return { + on: jest.fn(), + setTimeout: jest.fn(), + }; +}); + +export const createHttp2UnsecureServerMock = jest.fn(() => { + return { + on: jest.fn(), + setTimeout: jest.fn(), + }; +}); + +jest.doMock('http2', () => { + const actual = jest.requireActual('https'); + return { + ...actual, + createServer: createHttp2UnsecureServerMock, + createSecureServer: createHttp2SecureServerMock, + }; +}); diff --git a/packages/kbn-server-http-tools/src/get_listener.test.ts b/packages/kbn-server-http-tools/src/get_listener.test.ts index 21e0a93763490..dd64c2dba82fc 100644 --- a/packages/kbn-server-http-tools/src/get_listener.test.ts +++ b/packages/kbn-server-http-tools/src/get_listener.test.ts @@ -10,6 +10,8 @@ import { getServerTLSOptionsMock, createHttpServerMock, createHttpsServerMock, + createHttp2UnsecureServerMock, + createHttp2SecureServerMock, } from './get_listener.test.mocks'; import moment from 'moment'; import { ByteSizeValue } from '@kbn/config-schema'; @@ -18,6 +20,7 @@ import { getServerListener } from './get_listener'; const createConfig = (parts: Partial): IHttpConfig => ({ host: 'localhost', + protocol: 'http1', port: 5601, socketTimeout: 120000, keepaliveTimeout: 120000, @@ -41,99 +44,195 @@ const createConfig = (parts: Partial): IHttpConfig => ({ describe('getServerListener', () => { beforeEach(() => { getServerTLSOptionsMock.mockReset(); + createHttpServerMock.mockClear(); createHttpsServerMock.mockClear(); + createHttp2UnsecureServerMock.mockClear(); + createHttp2SecureServerMock.mockClear(); }); - describe('when TLS is enabled', () => { - it('calls getServerTLSOptions with the correct parameters', () => { - const config = createConfig({ ssl: { enabled: true } }); + describe('When protocol is `http1`', () => { + describe('when TLS is enabled', () => { + it('calls getServerTLSOptions with the correct parameters', () => { + const config = createConfig({ ssl: { enabled: true } }); - getServerListener(config); + getServerListener(config); - expect(getServerTLSOptionsMock).toHaveBeenCalledTimes(1); - expect(getServerTLSOptionsMock).toHaveBeenCalledWith(config.ssl); - }); + expect(getServerTLSOptionsMock).toHaveBeenCalledTimes(1); + expect(getServerTLSOptionsMock).toHaveBeenCalledWith(config.ssl); + }); - it('calls https.createServer with the correct parameters', () => { - const config = createConfig({ ssl: { enabled: true } }); + it('calls https.createServer with the correct parameters', () => { + const config = createConfig({ ssl: { enabled: true } }); - getServerTLSOptionsMock.mockReturnValue({ stub: true }); + getServerTLSOptionsMock.mockReturnValue({ stub: true }); - getServerListener(config); + getServerListener(config); - expect(createHttpsServerMock).toHaveBeenCalledTimes(1); - expect(createHttpsServerMock).toHaveBeenCalledWith({ - stub: true, - keepAliveTimeout: config.keepaliveTimeout, + expect(createHttpsServerMock).toHaveBeenCalledTimes(1); + expect(createHttpsServerMock).toHaveBeenCalledWith({ + stub: true, + keepAliveTimeout: config.keepaliveTimeout, + }); }); - }); - it('properly configures the listener', () => { - const config = createConfig({ ssl: { enabled: true } }); - const server = getServerListener(config); + it('properly configures the listener', () => { + const config = createConfig({ ssl: { enabled: true } }); + const server = getServerListener(config); + + expect(server.setTimeout).toHaveBeenCalledTimes(1); + expect(server.setTimeout).toHaveBeenCalledWith(config.socketTimeout); + + expect(server.on).toHaveBeenCalledTimes(2); + expect(server.on).toHaveBeenCalledWith('clientError', expect.any(Function)); + expect(server.on).toHaveBeenCalledWith('timeout', expect.any(Function)); + }); + + it('returns the https server', () => { + const config = createConfig({ ssl: { enabled: true } }); + + const server = getServerListener(config); - expect(server.setTimeout).toHaveBeenCalledTimes(1); - expect(server.setTimeout).toHaveBeenCalledWith(config.socketTimeout); + const expectedServer = createHttpsServerMock.mock.results[0].value; - expect(server.on).toHaveBeenCalledTimes(2); - expect(server.on).toHaveBeenCalledWith('clientError', expect.any(Function)); - expect(server.on).toHaveBeenCalledWith('timeout', expect.any(Function)); + expect(server).toBe(expectedServer); + }); }); - it('returns the https server', () => { - const config = createConfig({ ssl: { enabled: true } }); + describe('when TLS is disabled', () => { + it('does not call getServerTLSOptions', () => { + const config = createConfig({ ssl: { enabled: false } }); + + getServerListener(config); + + expect(getServerTLSOptionsMock).not.toHaveBeenCalled(); + }); + + it('calls http.createServer with the correct parameters', () => { + const config = createConfig({ ssl: { enabled: false } }); + + getServerTLSOptionsMock.mockReturnValue({ stub: true }); + + getServerListener(config); + + expect(createHttpServerMock).toHaveBeenCalledTimes(1); + expect(createHttpServerMock).toHaveBeenCalledWith({ + keepAliveTimeout: config.keepaliveTimeout, + }); + }); - const server = getServerListener(config); + it('properly configures the listener', () => { + const config = createConfig({ ssl: { enabled: false } }); + const server = getServerListener(config); - const expectedServer = createHttpsServerMock.mock.results[0].value; + expect(server.setTimeout).toHaveBeenCalledTimes(1); + expect(server.setTimeout).toHaveBeenCalledWith(config.socketTimeout); - expect(server).toBe(expectedServer); + expect(server.on).toHaveBeenCalledTimes(2); + expect(server.on).toHaveBeenCalledWith('clientError', expect.any(Function)); + expect(server.on).toHaveBeenCalledWith('timeout', expect.any(Function)); + }); + + it('returns the http server', () => { + const config = createConfig({ ssl: { enabled: false } }); + + const server = getServerListener(config); + + const expectedServer = createHttpServerMock.mock.results[0].value; + + expect(server).toBe(expectedServer); + }); }); }); - describe('when TLS is disabled', () => { - it('does not call getServerTLSOptions', () => { - const config = createConfig({ ssl: { enabled: false } }); + describe('When protocol is `http2`', () => { + const createHttp2Config = (parts: Partial) => + createConfig({ ...parts, protocol: 'http2' }); - getServerListener(config); + describe('when TLS is enabled', () => { + it('calls getServerTLSOptions with the correct parameters', () => { + const config = createHttp2Config({ ssl: { enabled: true } }); - expect(getServerTLSOptionsMock).not.toHaveBeenCalled(); - }); + getServerListener(config); + + expect(getServerTLSOptionsMock).toHaveBeenCalledTimes(1); + expect(getServerTLSOptionsMock).toHaveBeenCalledWith(config.ssl); + }); - it('calls http.createServer with the correct parameters', () => { - const config = createConfig({ ssl: { enabled: false } }); + it('calls http2.createSecureServer with the correct parameters', () => { + const config = createHttp2Config({ ssl: { enabled: true } }); - getServerTLSOptionsMock.mockReturnValue({ stub: true }); + getServerTLSOptionsMock.mockReturnValue({ stub: true }); - getServerListener(config); + getServerListener(config); - expect(createHttpServerMock).toHaveBeenCalledTimes(1); - expect(createHttpServerMock).toHaveBeenCalledWith({ - keepAliveTimeout: config.keepaliveTimeout, + expect(createHttp2SecureServerMock).toHaveBeenCalledTimes(1); + expect(createHttp2SecureServerMock).toHaveBeenCalledWith({ + stub: true, + allowHTTP1: true, + }); }); - }); - it('properly configures the listener', () => { - const config = createConfig({ ssl: { enabled: false } }); - const server = getServerListener(config); + it('properly configures the listener', () => { + const config = createHttp2Config({ ssl: { enabled: true } }); + const server = getServerListener(config); + + expect(server.setTimeout).toHaveBeenCalledTimes(1); + expect(server.setTimeout).toHaveBeenCalledWith(config.socketTimeout); + + expect(server.on).not.toHaveBeenCalled(); + }); + + it('returns the http2 secure server', () => { + const config = createHttp2Config({ ssl: { enabled: true } }); + + const server = getServerListener(config); - expect(server.setTimeout).toHaveBeenCalledTimes(1); - expect(server.setTimeout).toHaveBeenCalledWith(config.socketTimeout); + const expectedServer = createHttp2SecureServerMock.mock.results[0].value; - expect(server.on).toHaveBeenCalledTimes(2); - expect(server.on).toHaveBeenCalledWith('clientError', expect.any(Function)); - expect(server.on).toHaveBeenCalledWith('timeout', expect.any(Function)); + expect(server).toBe(expectedServer); + }); }); - it('returns the http server', () => { - const config = createConfig({ ssl: { enabled: false } }); + describe('when TLS is disabled', () => { + it('does not call getServerTLSOptions', () => { + const config = createHttp2Config({ ssl: { enabled: false } }); + + getServerListener(config); + + expect(getServerTLSOptionsMock).not.toHaveBeenCalled(); + }); + + it('calls http2.createServer with the correct parameters', () => { + const config = createHttp2Config({ ssl: { enabled: false } }); + + getServerTLSOptionsMock.mockReturnValue({ stub: true }); + + getServerListener(config); + + expect(createHttp2UnsecureServerMock).toHaveBeenCalledTimes(1); + expect(createHttp2UnsecureServerMock).toHaveBeenCalledWith({}); + }); - const server = getServerListener(config); + it('properly configures the listener', () => { + const config = createHttp2Config({ ssl: { enabled: false } }); + const server = getServerListener(config); - const expectedServer = createHttpServerMock.mock.results[0].value; + expect(server.setTimeout).toHaveBeenCalledTimes(1); + expect(server.setTimeout).toHaveBeenCalledWith(config.socketTimeout); - expect(server).toBe(expectedServer); + expect(server.on).not.toHaveBeenCalled(); + }); + + it('returns the http2 unsecure server', () => { + const config = createHttp2Config({ ssl: { enabled: false } }); + + const server = getServerListener(config); + + const expectedServer = createHttp2UnsecureServerMock.mock.results[0].value; + + expect(server).toBe(expectedServer); + }); }); }); }); diff --git a/packages/kbn-server-http-tools/src/get_listener.ts b/packages/kbn-server-http-tools/src/get_listener.ts index f1dbe3de753fa..16e39bc003b00 100644 --- a/packages/kbn-server-http-tools/src/get_listener.ts +++ b/packages/kbn-server-http-tools/src/get_listener.ts @@ -8,6 +8,7 @@ import http from 'http'; import https from 'https'; +import http2 from 'http2'; import { getServerTLSOptions } from './get_tls_options'; import type { IHttpConfig, ServerListener } from './types'; @@ -19,7 +20,10 @@ export function getServerListener( config: IHttpConfig, options: GetServerListenerOptions = {} ): ServerListener { - return configureHttp1Listener(config, options); + const useHTTP2 = config.protocol === 'http2'; + return useHTTP2 + ? configureHttp2Listener(config, options) + : configureHttp1Listener(config, options); } const configureHttp1Listener = ( @@ -52,3 +56,23 @@ const configureHttp1Listener = ( return listener; }; + +const configureHttp2Listener = ( + config: IHttpConfig, + { configureTLS = true }: GetServerListenerOptions = {} +): ServerListener => { + const useTLS = configureTLS && config.ssl.enabled; + const tlsOptions = useTLS ? getServerTLSOptions(config.ssl) : undefined; + + const listener = useTLS + ? http2.createSecureServer({ + ...tlsOptions, + // allow ALPN negotiation fallback to HTTP/1.x + allowHTTP1: true, + }) + : http2.createServer({}); + + listener.setTimeout(config.socketTimeout); + + return listener; +}; diff --git a/packages/kbn-server-http-tools/src/get_server_options.test.ts b/packages/kbn-server-http-tools/src/get_server_options.test.ts index 00c140f46f6c7..3430547d3a2db 100644 --- a/packages/kbn-server-http-tools/src/get_server_options.test.ts +++ b/packages/kbn-server-http-tools/src/get_server_options.test.ts @@ -23,6 +23,7 @@ jest.mock('fs', () => { const createConfig = (parts: Partial): IHttpConfig => ({ host: 'localhost', + protocol: 'http1', port: 5601, socketTimeout: 120000, keepaliveTimeout: 120000, diff --git a/packages/kbn-server-http-tools/src/get_server_options.ts b/packages/kbn-server-http-tools/src/get_server_options.ts index fe0a669fd62f5..bc3033afea373 100644 --- a/packages/kbn-server-http-tools/src/get_server_options.ts +++ b/packages/kbn-server-http-tools/src/get_server_options.ts @@ -24,10 +24,12 @@ export function getServerOptions(config: IHttpConfig, { configureTLS = true } = headers: corsAllowedHeaders, } : false; + const options: ServerOptions = { host: config.host, port: config.port, // manually configuring the listener + // @ts-expect-error HAPI types only define http1/https listener, not http2 listener: getServerListener(config, { configureTLS }), // must set to true when manually passing a TLS listener, false otherwise tls: configureTLS && config.ssl.enabled, diff --git a/packages/kbn-server-http-tools/src/get_tls_options.test.ts b/packages/kbn-server-http-tools/src/get_tls_options.test.ts index 0a50209db50c9..e99a885b9c51e 100644 --- a/packages/kbn-server-http-tools/src/get_tls_options.test.ts +++ b/packages/kbn-server-http-tools/src/get_tls_options.test.ts @@ -22,6 +22,7 @@ jest.mock('fs', () => { const createConfig = (parts: Partial): IHttpConfig => ({ host: 'localhost', + protocol: 'http1', port: 5601, socketTimeout: 120000, keepaliveTimeout: 120000, diff --git a/packages/kbn-server-http-tools/src/ssl/constants.ts b/packages/kbn-server-http-tools/src/ssl/constants.ts new file mode 100644 index 0000000000000..71632d52fe8e5 --- /dev/null +++ b/packages/kbn-server-http-tools/src/ssl/constants.ts @@ -0,0 +1,12 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +export const TLS_V1 = 'TLSv1'; +export const TLS_V1_1 = 'TLSv1.1'; +export const TLS_V1_2 = 'TLSv1.2'; +export const TLS_V1_3 = 'TLSv1.3'; diff --git a/packages/kbn-server-http-tools/src/ssl/index.ts b/packages/kbn-server-http-tools/src/ssl/index.ts index cbc3f17f915ef..2144b003a4830 100644 --- a/packages/kbn-server-http-tools/src/ssl/index.ts +++ b/packages/kbn-server-http-tools/src/ssl/index.ts @@ -7,3 +7,4 @@ */ export { SslConfig, sslSchema } from './ssl_config'; +export { TLS_V1, TLS_V1_1, TLS_V1_2, TLS_V1_3 } from './constants'; diff --git a/packages/kbn-server-http-tools/src/ssl/ssl_config.ts b/packages/kbn-server-http-tools/src/ssl/ssl_config.ts index a5d43064baa6b..feb5ff36b820e 100644 --- a/packages/kbn-server-http-tools/src/ssl/ssl_config.ts +++ b/packages/kbn-server-http-tools/src/ssl/ssl_config.ts @@ -11,13 +11,14 @@ import { schema, TypeOf } from '@kbn/config-schema'; import { readPkcs12Keystore, readPkcs12Truststore } from '@kbn/crypto'; import { constants as cryptoConstants } from 'crypto'; import { readFileSync } from 'fs'; +import { TLS_V1, TLS_V1_1, TLS_V1_2, TLS_V1_3 } from './constants'; const protocolMap = new Map([ - ['TLSv1', cryptoConstants.SSL_OP_NO_TLSv1], - ['TLSv1.1', cryptoConstants.SSL_OP_NO_TLSv1_1], - ['TLSv1.2', cryptoConstants.SSL_OP_NO_TLSv1_2], + [TLS_V1, cryptoConstants.SSL_OP_NO_TLSv1], + [TLS_V1_1, cryptoConstants.SSL_OP_NO_TLSv1_1], + [TLS_V1_2, cryptoConstants.SSL_OP_NO_TLSv1_2], // @ts-expect-error According to the docs SSL_OP_NO_TLSv1_3 should exist (https://nodejs.org/docs/latest-v12.x/api/crypto.html) - ['TLSv1.3', cryptoConstants.SSL_OP_NO_TLSv1_3], + [TLS_V1_3, cryptoConstants.SSL_OP_NO_TLSv1_3], ]); export const sslSchema = schema.object( @@ -45,12 +46,12 @@ export const sslSchema = schema.object( redirectHttpFromPort: schema.maybe(schema.number()), supportedProtocols: schema.arrayOf( schema.oneOf([ - schema.literal('TLSv1'), - schema.literal('TLSv1.1'), - schema.literal('TLSv1.2'), - schema.literal('TLSv1.3'), + schema.literal(TLS_V1), + schema.literal(TLS_V1_1), + schema.literal(TLS_V1_2), + schema.literal(TLS_V1_3), ]), - { defaultValue: ['TLSv1.1', 'TLSv1.2', 'TLSv1.3'], minSize: 1 } + { defaultValue: [TLS_V1_1, TLS_V1_2, TLS_V1_3], minSize: 1 } ), clientAuthentication: schema.oneOf( [schema.literal('none'), schema.literal('optional'), schema.literal('required')], diff --git a/packages/kbn-server-http-tools/src/types.ts b/packages/kbn-server-http-tools/src/types.ts index 88533162b2a32..11284455819c7 100644 --- a/packages/kbn-server-http-tools/src/types.ts +++ b/packages/kbn-server-http-tools/src/types.ts @@ -8,8 +8,9 @@ import type { Server as HttpServer } from 'http'; import type { Server as HttpsServer } from 'https'; -import { ByteSizeValue } from '@kbn/config-schema'; +import type { Http2SecureServer, Http2Server } from 'http2'; import type { Duration } from 'moment'; +import { ByteSizeValue } from '@kbn/config-schema'; /** * Composite type of all possible kind of Listener types. @@ -17,9 +18,12 @@ import type { Duration } from 'moment'; * Unfortunately, there's no real common interface between all those concrete classes, * as `net.Server` and `tls.Server` don't list all the APIs we're using (such as event binding) */ -export type ServerListener = HttpServer | HttpsServer; +export type ServerListener = Http2Server | Http2SecureServer | HttpServer | HttpsServer; + +export type ServerProtocol = 'http1' | 'http2'; export interface IHttpConfig { + protocol: ServerProtocol; host: string; port: number; maxPayload: ByteSizeValue; diff --git a/packages/kbn-test/src/kbn_client/kbn_client_requester.ts b/packages/kbn-test/src/kbn_client/kbn_client_requester.ts index 9481e6481b936..1f9718e06794c 100644 --- a/packages/kbn-test/src/kbn_client/kbn_client_requester.ts +++ b/packages/kbn-test/src/kbn_client/kbn_client_requester.ts @@ -96,6 +96,7 @@ export class KbnClientRequester { Url.parse(options.url).protocol === 'https:' ? new Https.Agent({ ca: options.certificateAuthorities, + rejectUnauthorized: false, }) : null; } diff --git a/src/cli/serve/serve.js b/src/cli/serve/serve.js index d431f6620d7f1..48b94a4fa8c61 100644 --- a/src/cli/serve/serve.js +++ b/src/cli/serve/serve.js @@ -101,6 +101,12 @@ export function applyConfigOverrides(rawConfig, opts, extraCliOptions, keystoreC const get = _.partial(_.get, rawConfig); const has = _.partial(_.has, rawConfig); + function ensureNotDefined(path, command = '--ssl') { + if (has(path)) { + throw new Error(`Can't use ${command} when "${path}" configuration is already defined.`); + } + } + if (opts.oss) { delete rawConfig.xpack; } @@ -152,49 +158,59 @@ export function applyConfigOverrides(rawConfig, opts, extraCliOptions, keystoreC } } - if (opts.ssl) { + if (opts.http2) { + set('server.protocol', 'http2'); + } + + // HTTP TLS configuration + if (opts.ssl || opts.http2) { // @kbn/dev-utils is part of devDependencies // eslint-disable-next-line import/no-extraneous-dependencies const { CA_CERT_PATH, KBN_KEY_PATH, KBN_CERT_PATH } = require('@kbn/dev-utils'); - const customElasticsearchHosts = opts.elasticsearch - ? opts.elasticsearch.split(',') - : [].concat(get('elasticsearch.hosts') || []); - - function ensureNotDefined(path) { - if (has(path)) { - throw new Error(`Can't use --ssl when "${path}" configuration is already defined.`); - } - } ensureNotDefined('server.ssl.certificate'); ensureNotDefined('server.ssl.key'); ensureNotDefined('server.ssl.keystore.path'); ensureNotDefined('server.ssl.truststore.path'); ensureNotDefined('server.ssl.certificateAuthorities'); - ensureNotDefined('elasticsearch.ssl.certificateAuthorities'); - const elasticsearchHosts = ( - (customElasticsearchHosts.length > 0 && customElasticsearchHosts) || [ - 'https://localhost:9200', - ] - ).map((hostUrl) => { - const parsedUrl = url.parse(hostUrl); - if (parsedUrl.hostname !== 'localhost') { - throw new Error( - `Hostname "${parsedUrl.hostname}" can't be used with --ssl. Must be "localhost" to work with certificates.` - ); - } - return `https://localhost:${parsedUrl.port}`; - }); set('server.ssl.enabled', true); set('server.ssl.certificate', KBN_CERT_PATH); set('server.ssl.key', KBN_KEY_PATH); set('server.ssl.certificateAuthorities', CA_CERT_PATH); - set('elasticsearch.hosts', elasticsearchHosts); - set('elasticsearch.ssl.certificateAuthorities', CA_CERT_PATH); } } + // Kib/ES encryption + if (opts.ssl) { + // @kbn/dev-utils is part of devDependencies + // eslint-disable-next-line import/no-extraneous-dependencies + const { CA_CERT_PATH } = require('@kbn/dev-utils'); + + const customElasticsearchHosts = opts.elasticsearch + ? opts.elasticsearch.split(',') + : [].concat(get('elasticsearch.hosts') || []); + + ensureNotDefined('elasticsearch.ssl.certificateAuthorities'); + + const elasticsearchHosts = ( + (customElasticsearchHosts.length > 0 && customElasticsearchHosts) || [ + 'https://localhost:9200', + ] + ).map((hostUrl) => { + const parsedUrl = url.parse(hostUrl); + if (parsedUrl.hostname !== 'localhost') { + throw new Error( + `Hostname "${parsedUrl.hostname}" can't be used with --ssl. Must be "localhost" to work with certificates.` + ); + } + return `https://localhost:${parsedUrl.port}`; + }); + + set('elasticsearch.hosts', elasticsearchHosts); + set('elasticsearch.ssl.certificateAuthorities', CA_CERT_PATH); + } + if (opts.elasticsearch) set('elasticsearch.hosts', opts.elasticsearch.split(',')); if (opts.port) set('server.port', opts.port); if (opts.host) set('server.host', opts.host); @@ -262,6 +278,7 @@ export default function (program) { command .option('--dev', 'Run the server with development mode defaults') .option('--ssl', 'Run the dev server using HTTPS') + .option('--http2', 'Run the dev server using HTTP2 with TLS') .option('--dist', 'Use production assets from kbn/optimizer') .option( '--no-base-path', diff --git a/src/core/server/index.ts b/src/core/server/index.ts index c77f7311100c8..901e0e3153651 100644 --- a/src/core/server/index.ts +++ b/src/core/server/index.ts @@ -232,6 +232,7 @@ export type { HttpServiceStart, RawRequest, FakeRawRequest, + HttpProtocol, } from '@kbn/core-http-server'; export type { IExternalUrlPolicy } from '@kbn/core-http-common'; diff --git a/src/core/server/integration_tests/http/http2_protocol.test.ts b/src/core/server/integration_tests/http/http2_protocol.test.ts new file mode 100644 index 0000000000000..f76076de81d43 --- /dev/null +++ b/src/core/server/integration_tests/http/http2_protocol.test.ts @@ -0,0 +1,214 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import { Server } from 'http'; +import supertest from 'supertest'; +import { of } from 'rxjs'; +import { KBN_CERT_PATH, KBN_KEY_PATH } from '@kbn/dev-utils'; +import { Router } from '@kbn/core-http-router-server-internal'; +import { + HttpServer, + HttpConfig, + config as httpConfig, + cspConfig, + externalUrlConfig, +} from '@kbn/core-http-server-internal'; +import { mockCoreContext } from '@kbn/core-base-server-mocks'; +import type { Logger } from '@kbn/logging'; + +const CSP_CONFIG = cspConfig.schema.validate({}); +const EXTERNAL_URL_CONFIG = externalUrlConfig.schema.validate({}); + +describe('Http2 - Smoke tests', () => { + let server: HttpServer; + let config: HttpConfig; + let logger: Logger; + let coreContext: ReturnType; + let innerServerListener: Server; + + const enhanceWithContext = (fn: (...args: any[]) => any) => fn.bind(null, {}); + + beforeAll(() => { + // required for the self-signed certificates used in testing + process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0'; + }); + + beforeEach(() => { + coreContext = mockCoreContext.create(); + logger = coreContext.logger.get(); + + const rawConfig = httpConfig.schema.validate({ + name: 'kibana', + protocol: 'http2', + host: '127.0.0.1', + port: 10002, + ssl: { + enabled: true, + certificate: KBN_CERT_PATH, + key: KBN_KEY_PATH, + cipherSuites: ['TLS_AES_256_GCM_SHA384'], + redirectHttpFromPort: 10003, + }, + shutdownTimeout: '5s', + }); + config = new HttpConfig(rawConfig, CSP_CONFIG, EXTERNAL_URL_CONFIG); + server = new HttpServer(coreContext, 'tests', of(config.shutdownTimeout)); + }); + + afterEach(async () => { + await server?.stop(); + }); + + describe('Basic tests against all supported methods', () => { + beforeEach(async () => { + const { registerRouter, server: innerServer } = await server.setup({ config$: of(config) }); + innerServerListener = innerServer.listener; + + const router = new Router('', logger, enhanceWithContext, { + isDev: false, + versionedRouterOptions: { + defaultHandlerResolutionStrategy: 'oldest', + }, + }); + + router.post({ path: '/', validate: false }, async (context, req, res) => { + return res.ok({ + body: { protocol: req.protocol, httpVersion: req.httpVersion }, + }); + }); + router.get({ path: '/', validate: false }, async (context, req, res) => { + return res.ok({ + body: { protocol: req.protocol, httpVersion: req.httpVersion }, + }); + }); + router.put({ path: '/', validate: false }, async (context, req, res) => { + return res.ok({ + body: { protocol: req.protocol, httpVersion: req.httpVersion }, + }); + }); + router.delete({ path: '/', validate: false }, async (context, req, res) => { + return res.ok({ + body: { protocol: req.protocol, httpVersion: req.httpVersion }, + }); + }); + + registerRouter(router); + + await server.start(); + }); + + describe('POST', () => { + it('should respond to POST endpoint for an HTTP/2 request', async () => { + const response = await supertest(innerServerListener).post('/').http2(); + + expect(response.status).toBe(200); + expect(response.body).toEqual({ protocol: 'http2', httpVersion: '2.0' }); + }); + + it('should respond to POST endpoint for an HTTP/1.x request', async () => { + const response = await supertest(innerServerListener).post('/'); + + expect(response.status).toBe(200); + expect(response.body).toEqual({ protocol: 'http1', httpVersion: '1.1' }); + }); + }); + + describe('GET', () => { + it('should respond to GET endpoint for an HTTP/2 request', async () => { + const response = await supertest(innerServerListener).get('/').http2(); + + expect(response.status).toBe(200); + expect(response.body).toEqual({ protocol: 'http2', httpVersion: '2.0' }); + }); + + it('should respond to GET endpoint for an HTTP/1.x request', async () => { + const response = await supertest(innerServerListener).get('/'); + + expect(response.status).toBe(200); + expect(response.body).toEqual({ protocol: 'http1', httpVersion: '1.1' }); + }); + }); + + describe('DELETE', () => { + it('should respond to DELETE endpoint for an HTTP/2 request', async () => { + const response = await supertest(innerServerListener).delete('/').http2(); + + expect(response.status).toBe(200); + expect(response.body).toEqual({ protocol: 'http2', httpVersion: '2.0' }); + }); + + it('should respond to DELETE endpoint for an HTTP/1.x request', async () => { + const response = await supertest(innerServerListener).delete('/'); + + expect(response.status).toBe(200); + expect(response.body).toEqual({ protocol: 'http1', httpVersion: '1.1' }); + }); + }); + + describe('PUT', () => { + it('should respond to PUT endpoint for an HTTP/2 request', async () => { + const response = await supertest(innerServerListener).put('/').http2(); + + expect(response.status).toBe(200); + expect(response.body).toEqual({ protocol: 'http2', httpVersion: '2.0' }); + }); + + it('should respond to PUT endpoint for an HTTP/1.x request', async () => { + const response = await supertest(innerServerListener).put('/'); + + expect(response.status).toBe(200); + expect(response.body).toEqual({ protocol: 'http1', httpVersion: '1.1' }); + }); + }); + }); + + describe('HTTP2-specific behaviors', () => { + beforeEach(async () => { + const { registerRouter, server: innerServer } = await server.setup({ config$: of(config) }); + innerServerListener = innerServer.listener; + + const router = new Router('', logger, enhanceWithContext, { + isDev: false, + versionedRouterOptions: { + defaultHandlerResolutionStrategy: 'oldest', + }, + }); + + router.get({ path: '/illegal_headers', validate: false }, async (context, req, res) => { + return res.ok({ + headers: { + connection: 'close', + }, + body: { protocol: req.protocol }, + }); + }); + + registerRouter(router); + + await server.start(); + }); + + describe('illegal http2 headers', () => { + it('should strip illegal http2 headers without causing errors when serving HTTP/2', async () => { + const response = await supertest(innerServerListener).get('/illegal_headers').http2(); + + expect(response.status).toBe(200); + expect(response.body).toEqual({ protocol: 'http2' }); + expect(response.header).toEqual(expect.not.objectContaining({ connection: 'close' })); + }); + + it('should keep illegal http2 headers when serving HTTP/1.x', async () => { + const response = await supertest(innerServerListener).get('/illegal_headers'); + + expect(response.status).toBe(200); + expect(response.body).toEqual({ protocol: 'http1' }); + expect(response.header).toEqual(expect.objectContaining({ connection: 'close' })); + }); + }); + }); +}); diff --git a/src/core/server/integration_tests/http/set_tls_config.test.ts b/src/core/server/integration_tests/http/set_tls_config.test.ts index b809a32075733..4966ecafce411 100644 --- a/src/core/server/integration_tests/http/set_tls_config.test.ts +++ b/src/core/server/integration_tests/http/set_tls_config.test.ts @@ -74,6 +74,7 @@ describe('setTlsConfig', () => { name: 'kibana', host: '127.0.0.1', port: 10002, + protocol: 'http1', ssl: { enabled: true, certificate: ES_CERT_PATH, diff --git a/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker b/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker index 64f94aed60cf6..0ca827fcbcd8e 100755 --- a/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker +++ b/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker @@ -155,11 +155,13 @@ kibana_vars=( server.customResponseHeaders server.defaultRoute server.host + server.http2.allowUnsecure server.keepAliveTimeout server.maxPayload server.maxPayloadBytes server.name server.port + server.protocol server.publicBaseUrl server.requestId.allowFromAnyIp server.requestId.ipAllowlist diff --git a/src/plugins/bfetch/server/plugin.ts b/src/plugins/bfetch/server/plugin.ts index 70f5ff36e6e4a..12b99c855cae8 100644 --- a/src/plugins/bfetch/server/plugin.ts +++ b/src/plugins/bfetch/server/plugin.ts @@ -18,6 +18,7 @@ import { RequestHandler, KibanaResponseFactory, AnalyticsServiceStart, + HttpProtocol, } from '@kbn/core/server'; import { map$ } from '@kbn/std'; @@ -65,11 +66,19 @@ export interface BfetchServerSetup { // eslint-disable-next-line @typescript-eslint/no-empty-interface export interface BfetchServerStart {} -const streamingHeaders = { - 'Content-Type': 'application/x-ndjson', - Connection: 'keep-alive', - 'Transfer-Encoding': 'chunked', - 'X-Accel-Buffering': 'no', +const getStreamingHeaders = (protocol: HttpProtocol): Record => { + if (protocol === 'http2') { + return { + 'Content-Type': 'application/x-ndjson', + 'X-Accel-Buffering': 'no', + }; + } + return { + 'Content-Type': 'application/x-ndjson', + Connection: 'keep-alive', + 'Transfer-Encoding': 'chunked', + 'X-Accel-Buffering': 'no', + }; }; interface Query { @@ -144,7 +153,7 @@ export class BfetchServerPlugin const data = request.body; const compress = request.query.compress; return response.ok({ - headers: streamingHeaders, + headers: getStreamingHeaders(request.protocol), body: createStream( handlerInstance.getResponseStream(data), logger, diff --git a/src/setup_node_env/exit_on_warning.js b/src/setup_node_env/exit_on_warning.js index dc6e321074224..82d173cc5e233 100644 --- a/src/setup_node_env/exit_on_warning.js +++ b/src/setup_node_env/exit_on_warning.js @@ -53,6 +53,28 @@ var IGNORE_WARNINGS = [ message: 'The URL https://github.com:crypto-browserify/browserify-rsa.git is invalid. Future versions of Node.js will throw an error.', }, + // supertest in HTTP2 mode uses 0.0.0.0 as the server's name + { + name: 'DeprecationWarning', + code: 'DEP0123', + message: + 'Setting the TLS ServerName to an IP address is not permitted by RFC 6066. This will be ignored in a future version.', + }, + { + // emitted whenever a header not supported by http2 is set. it's not actionable for the end user. + // HAPI sets a connection: close header - see https://github.com/hapijs/hapi/issues/3830 + name: 'UnsupportedWarning', + messageContains: + 'header is not valid, the value will be dropped from the header and will never be in use.', + }, + // We have to enabled NODE_TLS_REJECT_UNAUTHORIZED for FTR testing + // when http2 is enabled to accept dev self-signed certificates + { + ftrOnly: true, + name: 'Warning', + message: + "Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.", + }, ]; if (process.noProcessWarnings !== true) { @@ -68,7 +90,6 @@ if (process.noProcessWarnings !== true) { console.error(); console.error('Terminating process...'); } - process.exit(1); }); @@ -87,10 +108,22 @@ if (process.noProcessWarnings !== true) { function shouldIgnore(warn) { warn = parseWarn(warn); - return IGNORE_WARNINGS.some(function ({ name, code, message, file, line, col }) { + + return IGNORE_WARNINGS.some(function ({ + name, + code, + message, + messageContains, + file, + line, + col, + ftrOnly, + }) { + if (ftrOnly && !process.env.IS_FTR_RUNNER) return false; if (name && name !== warn.name) return false; if (code && code !== warn.code) return false; if (message && message !== warn.message) return false; + if (messageContains && !warn.message.includes(messageContains)) return false; if (file && !warn.frames[0].file.endsWith(file)) return false; if (line && line !== warn.frames[0].line) return false; if (col && col !== warn.frames[0].col) return false; diff --git a/test/api_integration/services/supertest.ts b/test/api_integration/services/supertest.ts index d8ce0d918c45a..bba3c5d731ca6 100644 --- a/test/api_integration/services/supertest.ts +++ b/test/api_integration/services/supertest.ts @@ -10,13 +10,27 @@ import { systemIndicesSuperuser } from '@kbn/test'; import { format as formatUrl } from 'url'; -import supertest from 'supertest'; +import supertest, { AgentOptions } from 'supertest'; import { FtrProviderContext } from '../../functional/ftr_provider_context'; export function KibanaSupertestProvider({ getService }: FtrProviderContext): supertest.Agent { const config = getService('config'); - const kibanaServerUrl = formatUrl(config.get('servers.kibana')); - return supertest(kibanaServerUrl); + const kibanaServerConfig = config.get('servers.kibana'); + const kibanaServerUrl = formatUrl(kibanaServerConfig); + + const options: AgentOptions = {}; + if (kibanaServerConfig.certificateAuthorities) { + options.ca = kibanaServerConfig.certificateAuthorities; + options.rejectUnauthorized = false; + } + + const serverArgs = config.get('kbnTestServer.serverArgs', []) as string[]; + const http2Enabled = serverArgs.includes('--server.protocol=http2'); + if (http2Enabled) { + options.http2 = true; + } + + return supertest(kibanaServerUrl, options); } export function ElasticsearchSupertestProvider({ diff --git a/test/common/configure_http2.ts b/test/common/configure_http2.ts new file mode 100644 index 0000000000000..7b43650e9b023 --- /dev/null +++ b/test/common/configure_http2.ts @@ -0,0 +1,85 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import { readFileSync } from 'fs'; +import { CA_CERT_PATH, KBN_CERT_PATH, KBN_KEY_PATH } from '@kbn/dev-utils'; + +type ConfigType = Record; + +/** + * Enables HTTP2 by adding/changing the appropriate config settings + * + * Important: this must be used on "final" (non-reused) configs, otherwise + * the overrides from the children configs could remove the overrides + * done in that helper. + */ +export const configureHTTP2 = (config: ConfigType): ConfigType => { + // Add env flag to avoid terminating on NODE_TLS_REJECT_UNAUTHORIZED warning + process.env.IS_FTR_RUNNER = 'true'; + + // tell native node agents to trust unsafe certificates + // this is ugly, but unfortunately required, as some libraries (such as supertest) + // have no real alternatives to accept self-signed certs + process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0'; + + // tell webdriver browser to accept self-signed certificates + config.browser.acceptInsecureCerts = true; + + // change the configured kibana server to run on https with the dev CA + config.servers.kibana = { + ...config.servers.kibana, + protocol: 'https', + certificateAuthorities: [readFileSync(CA_CERT_PATH, 'utf-8')], + }; + + const serverArgs = config.kbnTestServer.serverArgs; + + // enable http2 on the kibana server + addOrReplaceKbnServerArg(serverArgs, 'server.protocol', () => 'http2'); + // enable and configure TLS on the kibana server + addOrReplaceKbnServerArg(serverArgs, 'server.ssl.enabled', () => 'true'); + addOrReplaceKbnServerArg(serverArgs, 'server.ssl.key', () => KBN_KEY_PATH); + addOrReplaceKbnServerArg(serverArgs, 'server.ssl.certificate', () => KBN_CERT_PATH); + addOrReplaceKbnServerArg(serverArgs, 'server.ssl.certificateAuthorities', () => CA_CERT_PATH); + // replace the newsfeed test plugin url to use https + addOrReplaceKbnServerArg(serverArgs, 'newsfeed.service.urlRoot', (oldValue) => { + if (!oldValue || !oldValue.includes(config.servers.kibana.hostname)) { + return undefined; + } + return oldValue.replaceAll('http', 'https'); + }); + + return config; +}; + +/** + * Set or replace given `arg` in the provided serverArgs list, using the provided replacer function + */ +const addOrReplaceKbnServerArg = ( + serverArgs: string[], + argName: string, + replacer: (value: string | undefined) => string | undefined +) => { + const argPrefix = `--${argName}=`; + const argIndex = serverArgs.findIndex((value) => value.startsWith(argPrefix)); + + if (argIndex === -1) { + const newArgValue = replacer(undefined); + if (newArgValue !== undefined) { + serverArgs.push(`${argPrefix}${newArgValue}`); + } + } else { + const currentArgValue = serverArgs[argIndex].substring(argPrefix.length); + const newArgValue = replacer(currentArgValue); + if (newArgValue !== undefined) { + serverArgs[argIndex] = `${argPrefix}${newArgValue}`; + } else { + serverArgs.splice(argIndex, 1); + } + } +}; diff --git a/test/common/services/deployment.ts b/test/common/services/deployment.ts index b250d39ce65d6..e61d6b360da19 100644 --- a/test/common/services/deployment.ts +++ b/test/common/services/deployment.ts @@ -7,6 +7,7 @@ */ import { get } from 'lodash'; +import { Agent } from 'https'; import fetch from 'node-fetch'; import { getUrl } from '@kbn/test'; @@ -33,12 +34,23 @@ export class DeploymentService extends FtrService { const baseUrl = this.getHostPort(); const username = this.config.get('servers.kibana.username'); const password = this.config.get('servers.kibana.password'); + const protocol = this.config.get('servers.kibana.protocol'); + + let agent: Agent | undefined; + if (protocol === 'https') { + agent = new Agent({ + // required for self-signed certificates used for HTTPS FTR testing + rejectUnauthorized: false, + }); + } + const response = await fetch(baseUrl + '/api/stats?extended', { method: 'get', headers: { 'Content-Type': 'application/json', Authorization: 'Basic ' + Buffer.from(username + ':' + password).toString('base64'), }, + agent, }); const data = await response.json(); return get(data, 'usage.cloud.is_cloud_enabled', false); diff --git a/test/functional/apps/console/config.ts b/test/functional/apps/console/config.ts index e487d31dcb657..f295f1b826492 100644 --- a/test/functional/apps/console/config.ts +++ b/test/functional/apps/console/config.ts @@ -7,12 +7,13 @@ */ import { FtrConfigProviderContext } from '@kbn/test'; +import { configureHTTP2 } from '../../../common/configure_http2'; export default async function ({ readConfigFile }: FtrConfigProviderContext) { const functionalConfig = await readConfigFile(require.resolve('../../config.base.js')); - return { + return configureHTTP2({ ...functionalConfig.getAll(), testFiles: [require.resolve('.')], - }; + }); } diff --git a/test/functional/apps/home/config.ts b/test/functional/apps/home/config.ts index e487d31dcb657..f295f1b826492 100644 --- a/test/functional/apps/home/config.ts +++ b/test/functional/apps/home/config.ts @@ -7,12 +7,13 @@ */ import { FtrConfigProviderContext } from '@kbn/test'; +import { configureHTTP2 } from '../../../common/configure_http2'; export default async function ({ readConfigFile }: FtrConfigProviderContext) { const functionalConfig = await readConfigFile(require.resolve('../../config.base.js')); - return { + return configureHTTP2({ ...functionalConfig.getAll(), testFiles: [require.resolve('.')], - }; + }); } diff --git a/test/functional/services/supertest.ts b/test/functional/services/supertest.ts index 32ecc3f51759d..10a9803df263e 100644 --- a/test/functional/services/supertest.ts +++ b/test/functional/services/supertest.ts @@ -8,11 +8,25 @@ import { format as formatUrl } from 'url'; -import supertest from 'supertest'; +import supertest, { AgentOptions } from 'supertest'; import { FtrProviderContext } from '../ftr_provider_context'; export function KibanaSupertestProvider({ getService }: FtrProviderContext) { const config = getService('config'); - const kibanaServerUrl = formatUrl(config.get('servers.kibana')); - return supertest(kibanaServerUrl); + const kibanaServerConfig = config.get('servers.kibana'); + const kibanaServerUrl = formatUrl(kibanaServerConfig); + + const options: AgentOptions = {}; + if (kibanaServerConfig.certificateAuthorities) { + options.ca = kibanaServerConfig.certificateAuthorities; + options.rejectUnauthorized = false; + } + + const serverArgs = config.get('kbnTestServer.serverArgs', []) as string[]; + const http2Enabled = serverArgs.includes('--server.protocol=http2'); + if (http2Enabled) { + options.http2 = true; + } + + return supertest(kibanaServerUrl, options); } diff --git a/x-pack/test/common/services/spaces.ts b/x-pack/test/common/services/spaces.ts index ad829e45fccec..b4e99cacee571 100644 --- a/x-pack/test/common/services/spaces.ts +++ b/x-pack/test/common/services/spaces.ts @@ -7,6 +7,7 @@ import type { Space } from '@kbn/spaces-plugin/common'; import Axios from 'axios'; +import Https from 'https'; import { format as formatUrl } from 'url'; import util from 'util'; import { FtrProviderContext } from '../ftr_provider_context'; @@ -16,11 +17,21 @@ export function SpacesServiceProvider({ getService }: FtrProviderContext) { const config = getService('config'); const url = formatUrl(config.get('servers.kibana')); + const certificateAuthorities = config.get('servers.kibana.certificateAuthorities'); + const httpsAgent: Https.Agent | undefined = certificateAuthorities + ? new Https.Agent({ + ca: certificateAuthorities, + // required for self-signed certificates used for HTTPS FTR testing + rejectUnauthorized: false, + }) + : undefined; + const axios = Axios.create({ headers: { 'kbn-xsrf': 'x-pack/ftr/services/spaces/space' }, baseURL: url, maxRedirects: 0, validateStatus: () => true, // we do our own validation below and throw better error messages + httpsAgent, }); return new (class SpacesService { diff --git a/yarn.lock b/yarn.lock index 0064bfb5da54b..d7c6b766d93d8 100644 --- a/yarn.lock +++ b/yarn.lock @@ -19258,6 +19258,11 @@ http2-client@^1.2.5: resolved "https://registry.yarnpkg.com/http2-client/-/http2-client-1.3.5.tgz#20c9dc909e3cc98284dd20af2432c524086df181" integrity sha512-EC2utToWl4RKfs5zd36Mxq7nzHHBuomZboI0yYL6Y0RmBgT7Sgkq4rQ0ezFTYoIsSs7Tm9SJe+o2FcAg6GBhGA== +http2-proxy@^5.0.53: + version "5.0.53" + resolved "https://registry.yarnpkg.com/http2-proxy/-/http2-proxy-5.0.53.tgz#fc6cb07d2bb977a388ebeec4449557f2011e5a1f" + integrity sha512-k9OUKrPWau/YeViJGv5peEFgSGPE2n8CDyk/G3f+JfaaJzbFMPAK5PJTd99QYSUvgUwVBGNbZJCY/BEb+kUZNQ== + http2-wrapper@^1.0.0-beta.5.2: version "1.0.0-beta.5.2" resolved "https://registry.yarnpkg.com/http2-wrapper/-/http2-wrapper-1.0.0-beta.5.2.tgz#8b923deb90144aea65cf834b016a340fc98556f3" @@ -19266,7 +19271,7 @@ http2-wrapper@^1.0.0-beta.5.2: quick-lru "^5.1.1" resolve-alpn "^1.0.0" -http2-wrapper@^2.1.10: +http2-wrapper@^2.1.10, http2-wrapper@^2.2.1: version "2.2.1" resolved "https://registry.yarnpkg.com/http2-wrapper/-/http2-wrapper-2.2.1.tgz#310968153dcdedb160d8b72114363ef5fce1f64a" integrity sha512-V5nVw1PAOgfI3Lmeaj2Exmeg7fenjhRUgz1lPSezy1CuhPYbgQtbQj4jZfEAEMlaL+vupsvhjqCyjzob0yxsmQ== @@ -28874,7 +28879,7 @@ string-replace-loader@^2.2.0: loader-utils "^1.2.3" schema-utils "^1.0.0" -"string-width-cjs@npm:string-width@^4.2.0", "string-width@^1.0.2 || 2 || 3 || 4", string-width@^4.0.0, string-width@^4.1.0, string-width@^4.2.0, string-width@^4.2.2, string-width@^4.2.3: +"string-width-cjs@npm:string-width@^4.2.0": version "4.2.3" resolved "https://registry.yarnpkg.com/string-width/-/string-width-4.2.3.tgz#269c7117d27b05ad2e536830a8ec895ef9c6d010" integrity sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g== @@ -28892,6 +28897,15 @@ string-width@^1.0.1: is-fullwidth-code-point "^1.0.0" strip-ansi "^3.0.0" +"string-width@^1.0.2 || 2 || 3 || 4", string-width@^4.0.0, string-width@^4.1.0, string-width@^4.2.0, string-width@^4.2.2, string-width@^4.2.3: + version "4.2.3" + resolved "https://registry.yarnpkg.com/string-width/-/string-width-4.2.3.tgz#269c7117d27b05ad2e536830a8ec895ef9c6d010" + integrity sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g== + dependencies: + emoji-regex "^8.0.0" + is-fullwidth-code-point "^3.0.0" + strip-ansi "^6.0.1" + string-width@^5.0.1, string-width@^5.1.2: version "5.1.2" resolved "https://registry.yarnpkg.com/string-width/-/string-width-5.1.2.tgz#14f8daec6d81e7221d2a357e668cab73bdbca794" @@ -29001,7 +29015,7 @@ stringify-object@^3.2.1: is-obj "^1.0.1" is-regexp "^1.0.0" -"strip-ansi-cjs@npm:strip-ansi@^6.0.1", strip-ansi@^6.0.0, strip-ansi@^6.0.1: +"strip-ansi-cjs@npm:strip-ansi@^6.0.1": version "6.0.1" resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-6.0.1.tgz#9e26c63d30f53443e9489495b2105d37b67a85d9" integrity sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A== @@ -29015,6 +29029,13 @@ strip-ansi@^3.0.0, strip-ansi@^3.0.1: dependencies: ansi-regex "^2.0.0" +strip-ansi@^6.0.0, strip-ansi@^6.0.1: + version "6.0.1" + resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-6.0.1.tgz#9e26c63d30f53443e9489495b2105d37b67a85d9" + integrity sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A== + dependencies: + ansi-regex "^5.0.1" + strip-ansi@^7.0.1, strip-ansi@^7.1.0: version "7.1.0" resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-7.1.0.tgz#d5b6568ca689d8561370b0707685d22434faff45" @@ -31882,7 +31903,7 @@ workerpool@6.2.1: resolved "https://registry.yarnpkg.com/workerpool/-/workerpool-6.2.1.tgz#46fc150c17d826b86a008e5a4508656777e9c343" integrity sha512-ILEIE97kDZvF9Wb9f6h5aXK4swSlKGUcOEGiIYb2OOu/IrDU9iwj0fD//SsA6E5ibwJxpEvhullJY4Sl4GcpAw== -"wrap-ansi-cjs@npm:wrap-ansi@^7.0.0", wrap-ansi@^7.0.0: +"wrap-ansi-cjs@npm:wrap-ansi@^7.0.0": version "7.0.0" resolved "https://registry.yarnpkg.com/wrap-ansi/-/wrap-ansi-7.0.0.tgz#67e145cff510a6a6984bdf1152911d69d2eb9e43" integrity sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q== @@ -31908,6 +31929,15 @@ wrap-ansi@^6.2.0: string-width "^4.1.0" strip-ansi "^6.0.0" +wrap-ansi@^7.0.0: + version "7.0.0" + resolved "https://registry.yarnpkg.com/wrap-ansi/-/wrap-ansi-7.0.0.tgz#67e145cff510a6a6984bdf1152911d69d2eb9e43" + integrity sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q== + dependencies: + ansi-styles "^4.0.0" + string-width "^4.1.0" + strip-ansi "^6.0.0" + wrap-ansi@^8.1.0: version "8.1.0" resolved "https://registry.yarnpkg.com/wrap-ansi/-/wrap-ansi-8.1.0.tgz#56dc22368ee570face1b49819975d9b9a5ead214"