Skip to content

Commit

Permalink
Added support for automatic expression optimizations (#3)
Browse files Browse the repository at this point in the history
- Key changes:
  - Added support for [automatic expression optimizations](https://pkg.go.dev/github.com/VictoriaMetrics/metricsql#Optimize) for non-full access requests;
- Minor changes:
  - lfgw:
    - Slight improvements in code style;
    - Migrated to go 1.17;
    - Fully deprecated non-OIDC modes;
    - Bumped go.mod deps;
    - Updated base images;
    - Enabled more linters for .golangci-lint;
  - CI:
    - Simplified Taskfile;
    - Enabled dependabot alerts;
    - Added a workflow to publish docker images.
  • Loading branch information
weisdd authored Feb 21, 2022
1 parent 3c49d45 commit b28ca9b
Show file tree
Hide file tree
Showing 18 changed files with 298 additions and 184 deletions.
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
.git
*.md
*.yaml
*.yml
Expand Down
17 changes: 17 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
version: 2

updates:
- package-ecosystem: "docker"
directory: "/"
schedule:
interval: "daily"

- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"

- package-ecosystem: "gomod"
directory: "/"
schedule:
interval: "daily"
55 changes: 55 additions & 0 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
name: main

on:
push:
# pull_request:

concurrency:
group: ${{ github.ref }}

jobs:
lint-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2

- uses: actions/setup-go@v2
with:
go-version: "1.17"

- name: golangci-lint
uses: golangci/golangci-lint-action@v2
with:
skip-go-installation: true

- name: Run tests
shell: bash
run: |
go test ./...
docker:
runs-on: ubuntu-latest
needs:
- lint-test
steps:
- uses: actions/checkout@v2

- name: Login to Docker registry
uses: docker/login-action@v1
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Docker meta
id: meta
uses: docker/metadata-action@v3
with:
images: ghcr.io/${{ github.repository }}

- name: Build and push
id: docker_build
uses: docker/build-push-action@v2
with:
push: true
tags: ${{ steps.meta.outputs.tags }}
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,6 @@
# Dependency directories (remove the comment below to include it)
# vendor/

temp
temp/
acl.yaml
.vscode
19 changes: 19 additions & 0 deletions .golangci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
linters:
enable:
- gocritic
- gosec
## TODO: clean the list
- gofmt
- goimports
- gocyclo
- goconst
# - prealloc
- revive
- unconvert
# - unparam

linters-settings:
gosimple:
go: "1.17"
gocyclo:
min-complexity: 10
41 changes: 29 additions & 12 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,37 +1,54 @@
# CHANGELOG

## 0.5.0

- Key changes:
- Added support for [automatic expression optimizations](https://pkg.go.dev/github.com/VictoriaMetrics/metricsql#Optimize) for non-full access requests;
- Minor changes:
- lfgw:
- Slight improvements in code style;
- Migrated to go 1.17;
- Fully deprecated non-OIDC modes;
- Bumped go.mod deps;
- Updated base images;
- Enabled more linters for .golangci-lint;
- CI:
- Simplified Taskfile;
- Enabled dependabot alerts;
- Added a workflow to publish docker images.

## 0.4.0

* Added support for multiple roles (previously, only the first one would be picked).
- Added support for multiple roles (previously, only the first one would be picked).

## 0.3.0

* Added support for POST requests;
* Updated metricsql from `v0.10.1` to `v0.14.0`.
- Added support for POST requests;
- Updated metricsql from `v0.10.1` to `v0.14.0`.

## 0.2.3

* Added `/federate` to a list of requests that should be rewritten.
- Added `/federate` to a list of requests that should be rewritten.

## 0.2.2

* Moved to go 1.16;
* Bumped dependencies;
* Improved build caching.
- Moved to go 1.16;
- Bumped dependencies;
- Improved build caching.

## 0.2.1

* Adjusted request rewrite logic, so now all requests containing `/api/` are rewritten, whereas previously only those starting with `/api/`. So, non-standard URIs are taken into account now.
* Explicitly specified flush interval for reverse proxy;
- Adjusted request rewrite logic, so now all requests containing `/api/` are rewritten, whereas previously only those starting with `/api/`. So, non-standard URIs are taken into account now.
- Explicitly specified flush interval for reverse proxy;

## 0.2.0

* Added support for extra authorization headers (X-Forwarded-Access-Token, X-Auth-Request-Access-Token).
- Added support for extra authorization headers (X-Forwarded-Access-Token, X-Auth-Request-Access-Token).

## 0.1.1

* Bugfix for doubling URI-path while proxying in case UPSTREAM_URL has non-empty URI.
- Bugfix for doubling URI-path while proxying in case UPSTREAM_URL has non-empty URI.

## 0.1.0

* Initial release.
- Initial release.
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
ARG ALPINE_VERSION=3.13.1
ARG GOLANG_VERSION=1.16.0-alpine3.13
ARG ALPINE_VERSION=3.15.0
ARG GOLANG_VERSION=1.17.7-alpine3.15

FROM golang:${GOLANG_VERSION} as builder

Expand Down
114 changes: 41 additions & 73 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

LFGW is a trivial reverse proxy based on `httputil` and `VictoriaMetrics/metricsql` with a purpose of dynamically rewriting requests to Prometheus-like backends.

More specifically, it manipulates label filters in metric expressions to reduce the scope of metrics exposed to an end user.
More specifically, it manipulates label filters in metric expressions to reduce the scope of metrics exposed to an end user based on user's OIDC-roles.

## Key features

Expand All @@ -11,92 +11,48 @@ More specifically, it manipulates label filters in metric expressions to reduce
* ACL-based request rewrites with implicit deny;
* supports Victoria Metrics' PromQL extensions;
* since it's based on `VictoriaMetrics/metricsql` library, which has way simpler interface than `prometheus`, there is no need to write a separate implementation for every type of MetricExpr on the planet;
* [automatic expression optimizations](https://pkg.go.dev/github.com/VictoriaMetrics/metricsql#Optimize) for non-full access requests;
* it's based on the middleware pattern, so it's easy to implement other, non-OIDC, modes should the need be;
* support for different headers with access tokens (X-Forwarded-Access-Token, X-Auth-Request-Access-Token, Authorization);
* support for different headers with access tokens (`X-Forwarded-Access-Token`, `X-Auth-Request-Access-Token`, `Authorization`);
* requests to sensitive endpoints are blocked by default;
* requests to both `/api/*` and `/federate` endpoints are protected (=rewritten).

## Current limitations

* there's no OIDC callback, so it can only proxy requests that already have a jwt token;
* other modes (POSITIVE_REGEXP, NEGATIVE_REGEXP, NAMESPACE) are deprecated.

## TODO

* tests for handlers;
* improve naming;
* log slow requests;
* metrics;
* add CLI interface (currently, only environment variables are used);
* configurable JMESPath for the `roles` attribute;
* OIDC callback to support for proxying Prometheus web-interface itself;
* structured logging (it'll require an intermediate interface for logging `httputil`'s error logs);
* simple deduplication if there is any performance issue (another option: use `trickster` for request optimizations).

## Similar projects

### Prometheus filter proxy

Link: [Prometheus filter proxy](https://github.com/hoffie/prometheus-filter-proxy)

Minuses:

* based on Prometheus library, so might not support some of Victoria Metrics' extensions;
* it's an overly simple implementation that relies only on URI paths to define the scope of available metrics, so a user might potentially get access to any metrics should the URL become exposed;
* it's based on http client, thus unlikely to be ready for high volumes of requests;
* does not allow to filter out requests to sensitive endpoints (like `/admin/tsdb`);
* no longer maintained.

### Prometheus ACLs
Similar projects are described [here](docs/similar-projects.md).

Link: [Prometheus ACLs](https://github.com/bitsbeats/prometheus-acls)
## Docker images

Pluses:

* based on `httputil` reverse proxy;
* supports label filter deduplication;
* extensive ACL syntax, which allows to specify any label filters, not necessarily limit to `namespace`;
* has an OIDC callback and `gorilla/sessions`, so it's possible to obtain a token through the application itself;
* validates jwt-token;
* metrics;
* configurable JMESPath.

Minuses:

* a user cannot have multiple roles (`When you have multiple roles, the first one that is mentioned in prometheus-acls will be used.`);
* based on Prometheus library, so might not support some of Victoria Metrics' extensions;
* does not allow to filter out requests to sensitive endpoints (like `/admin/tsdb`);
* does not rewrite requests to `/federate` endpoint (at least, at the time of writing);
* http server time-outs are not configured, thus might retain http sessions for much longer than needed.
Docker images are published on [ghcr.io/weisdd/lfgw](https://github.com/weisdd/lfgw/pkgs/container/lfgw).

## Configuration

OIDC roles are expected to be present in `roles` within a jwt token.

### Environment variables

| Module | Variable | Default Value | Description |
| -------------------- | ------------------- | ------------- | ------------------------------------------------------------ |
| **General settings** | | | |
| | `LFGW_MODE` | `oidc` | Currently, only `oidc` mode is supported. So, never mind. |
| | | | |
| **Logging** | | | |
| | `DEBUG` | `false` | Whether to print out debug log messages. |
| | | | |
| **HTTP Server** | | | |
| | `PORT` | `8080` | Port the web server will listen on. |
| | `READ_TIMEOUT` | `10s` | `ReadTimeout` covers the time from when the connection is accepted to when the request body is fully read (if you do read the body, otherwise to the end of the headers). [More details](https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/) |
| | `WRITE_TIMEOUT` | `10s` | `WriteTimeout` normally covers the time from the end of the request header read to the end of the response write (a.k.a. the lifetime of the ServeHTTP). [More details](https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/) |
| | | | |
| **Proxy** | | | |
| | `UPSTREAM_URL` | | Prometheus URL, e.g. `http://prometheus.microk8s.localhost`. |
| | `SAFE_MODE` | `true` | Whether to block requests to sensitive endpoints like `/api/v1/admin/tsdb`, `/api/v1/insert`. |
| | `SET_PROXY_HEADERS` | `false` | Whether to set proxy headers (`X-Forwarded-For`, `X-Forwarded-Proto`, `X-Forwarded-Host`). |
| | | | |
| **OIDC** | | | |
| | `ACL_PATH` | `./acl.yaml` | Path to a file with ACL definitions (OIDC role to namespace bindings). |
| | `OIDC_REALM_URL` | | OIDC Realm URL, e.g. `https://auth.microk8s.localhost/auth/realms/cicd` |
| | `OIDC_CLIENT_ID` | | OIDC Client ID (1*) |
| Module | Variable | Default Value | Description |
| -------------------- | ---------------------- | ------------- | ------------------------------------------------------------ |
| **General settings** | | | |
| | `OPTIMIZE_EXPRESSIONS` | `true` | Whether to automatically optimize expressions for non-full access requests. [More details](https://pkg.go.dev/github.com/VictoriaMetrics/metricsql#Optimize) |
| | | | |
| **Logging** | | | |
| | `DEBUG` | `false` | Whether to print out debug log messages. |
| | | | |
| **HTTP Server** | | | |
| | `PORT` | `8080` | Port the web server will listen on. |
| | `READ_TIMEOUT` | `10s` | `ReadTimeout` covers the time from when the connection is accepted to when the request body is fully read (if you do read the body, otherwise to the end of the headers). [More details](https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/) |
| | `WRITE_TIMEOUT` | `10s` | `WriteTimeout` normally covers the time from the end of the request header read to the end of the response write (a.k.a. the lifetime of the ServeHTTP). [More details](https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/) |
| | | | |
| **Proxy** | | | |
| | `UPSTREAM_URL` | | Prometheus URL, e.g. `http://prometheus.microk8s.localhost`. |
| | `SAFE_MODE` | `true` | Whether to block requests to sensitive endpoints like `/api/v1/admin/tsdb`, `/api/v1/insert`. |
| | `SET_PROXY_HEADERS` | `false` | Whether to set proxy headers (`X-Forwarded-For`, `X-Forwarded-Proto`, `X-Forwarded-Host`). |
| | | | |
| **OIDC** | | | |
| | `ACL_PATH` | `./acl.yaml` | Path to a file with ACL definitions (OIDC role to namespace bindings). |
| | `OIDC_REALM_URL` | | OIDC Realm URL, e.g. `https://auth.microk8s.localhost/auth/realms/cicd` |
| | `OIDC_CLIENT_ID` | | OIDC Client ID (1*) |

(1*): since it's grafana who obtains jwt-tokens in the first place, the specified client id must also be present in the forwarded token (the `audience` field). To put it simply, better to use the same client id for both Grafana and LFGW.

Expand All @@ -122,7 +78,6 @@ team5: min.*, stolon # only those matching namespace=~"^(min.*|stolon)$"
To summarize, here are the key principles used for rewriting requests:
* `.*` - all requests are simply forwarded to an upstream;

* `minio` - all label filters with the `namespace` label are removed, then `namespace="minio"` is added;
* `min.*` - positive regex-match label filters (`namespace=~"X"`) are removed, then `namespace=~"mi.*"` is added;
* `minio, stolon` - positive regex-match label filters (`namespace=~"X"`) are removed, then `namespace=~"^(minio|stolon)$"` is added;
Expand All @@ -136,3 +91,16 @@ Note: a user is free to have multiple roles matching the contents of `acl.yaml`.
=> a prepopulated LF, corresponding to the full access role, is returned;
* multiple "limited" roles
=> definitions of all those roles are merged together, and then LFGW generates a new LF. The process is the same as if this meta-definition was loaded through `acl.yaml`.

## TODO

* tests for handlers;
* improve naming;
* log slow requests;
* metrics;
* add CLI interface (currently, only environment variables are used);
* configurable JMESPath for the `roles` attribute;
* OIDC callback to support for proxying Prometheus web-interface itself;
* structured logging (it'll require an intermediate interface for logging `httputil`'s error logs);
* simple deduplication if there is any performance issue (another option: use `trickster` for request optimizations);
* add a helm chart.
32 changes: 3 additions & 29 deletions Taskfile.yml
Original file line number Diff line number Diff line change
@@ -1,41 +1,15 @@
version: '3'

env:
TARGET: ./...

tasks:
default:
- task: test
- task: lint

test:
cmds:
- go test $TARGET
# - go test ./... -race
- go test ./...

lint:
deps:
- lint:go-vet
- lint:golint
- lint:golangci-lint
- lint:gosec
- lint:staticcheck

lint:go-vet:
cmds:
- go vet $TARGET

lint:golint:
cmds:
- golint $TARGET

lint:golangci-lint:
cmds:
- golangci-lint run $TARGET

lint:gosec:
cmds:
- gosec -quiet $TARGET

lint:staticcheck:
cmds:
- staticcheck $TARGET
- golangci-lint run
4 changes: 2 additions & 2 deletions cmd/lfgw/acl.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ package main

import (
"fmt"
"io/ioutil"
"os"
"regexp"
"strings"
"unicode"
Expand Down Expand Up @@ -97,7 +97,7 @@ func (a *ACL) PrepareLF(ns string) (metricsql.LabelFilter, error) {
func (app *application) loadACL() (ACLMap, error) {
aclMap := make(ACLMap)

yamlFile, err := ioutil.ReadFile(app.ACLPath)
yamlFile, err := os.ReadFile(app.ACLPath)
if err != nil {
return aclMap, err
}
Expand Down
Loading

0 comments on commit b28ca9b

Please sign in to comment.