Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: use Hurl in CI to test Caddy against spec #6255

Draft
wants to merge 7 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,17 @@ on:
branches:
- master
- 2.*
- hurl-tests
pull_request:
branches:
- master
- 2.*

jobs:
test:
permissions:
checks: write
pull-requests: write
strategy:
# Default is true, cancels jobs for other platforms in the matrix if one fails
fail-fast: false
Expand Down Expand Up @@ -140,6 +144,93 @@ jobs:
# echo "step_test ${{ steps.step_test.outputs.status }}\n"
# exit 1

spec-test:
permissions:
checks: write
pull-requests: write
strategy:
matrix:
os:
- linux
go:
- '1.23'

include:
# Set the minimum Go patch version for the given Go minor
# Usable via ${{ matrix.GO_SEMVER }}
- go: '1.23'
GO_SEMVER: '~1.23.0'

# Set some variables per OS, usable via ${{ matrix.VAR }}
# OS_LABEL: the VM label from GitHub Actions (see https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories)
# CADDY_BIN_PATH: the path to the compiled Caddy binary, for artifact publishing
# SUCCESS: the typical value for $? per OS (Windows/pwsh returns 'True')
- os: linux
OS_LABEL: ubuntu-latest
CADDY_BIN_PATH: ./cmd/caddy/caddy
SUCCESS: 0

runs-on: ${{ matrix.OS_LABEL }}

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Install Go
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.GO_SEMVER }}
check-latest: true

- name: Print Go version and environment
id: vars
shell: bash
run: |
printf "curl version: $(curl --version)\n"
printf "Using go at: $(which go)\n"
printf "Go version: $(go version)\n"
printf "\n\nGo environment:\n\n"
go env
printf "\n\nSystem environment:\n\n"
env
printf "Git version: $(git version)\n\n"
# Calculate the short SHA1 hash of the git commit
echo "short_sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT

- name: Get dependencies
run: |
go get -v -t -d ./...
# mkdir test-results
- name: Build Caddy
working-directory: ./cmd/caddy
env:
CGO_ENABLED: 0
run: |
go build -tags nobadger -trimpath -ldflags="-w -s" -v

- name: Install Hurl
env:
HURL_VERSION: "5.0.1"
run: |
curl --location --remote-name https://github.com/Orange-OpenSource/hurl/releases/download/${HURL_VERSION}/hurl_${HURL_VERSION}_amd64.deb
sudo dpkg -i hurl_${HURL_VERSION}_amd64.deb
hurl --version

- name: Run Caddy
run: |
./cmd/caddy/caddy start

- name: Run tests with Hurl
run: |
mkdir hurl-report
find . -name *.hurl -exec hurl --variables-file caddytest/spec/hurl_vars.properties --very-verbose --verbose --test --report-junit hurl-report/junit.xml --color {} \;

- name: Publish Test Results
uses: EnricoMi/publish-unit-test-result-action@v2
with:
files: |
hurl-report/junit.xml

s390x-test:
name: test (s390x on IBM Z)
runs-on: ubuntu-latest
Expand Down
38 changes: 38 additions & 0 deletions caddytest/spec/http/basicauth/spec.hurl
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Configure Caddy
POST http://localhost:2019/load
Content-Type: text/caddyfile
```
{
skip_install_trust
http_port 9080
https_port 9443
local_certs
debug
}
localhost {
log
basic_auth {
john $2a$14$x4HlYwA9Zeer4RkMEYbUzug9XxWmncneR.dcMs.UjalR95URnHg5.
}
respond "Hello, World!"
}
```

# requests without `Authorization` header are rejected with 401
GET https://localhost:9443
[Options]
insecure: true
HTTP 401
[Asserts]
header "WWW-Authenticate" == "Basic realm=\"restricted\""


# requests with `Authorization` header are accepted with 200
GET https://localhost:9443
[BasicAuth]
john:password
[Options]
insecure: true
HTTP 200
[Asserts]
`Hello, World!`
9 changes: 9 additions & 0 deletions caddytest/spec/http/file_server/assets/indexed/index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
<!DOCTYPE html>
<html>
<head>
<title>Index.html Title</title>
</head>
<body>
Index.html
</body>
</html>
1 change: 1 addition & 0 deletions caddytest/spec/http/file_server/assets/indexed/index.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
index.txt
Empty file.
119 changes: 119 additions & 0 deletions caddytest/spec/http/file_server/spec.hurl
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# Configure Caddy with default configuration
POST http://localhost:2019/load
Content-Type: text/caddyfile
```
{
skip_install_trust
http_port 9080
https_port 9443
local_certs
debug
}
localhost {
root {{indexed_root}}
file_server
}
```

# requests without specific file receive index file per
# the default index list: index.html, index.txt
GET https://localhost:9443
[Options]
insecure: true
HTTP 200
[Asserts]
```
<!DOCTYPE html>
<html>
<head>
<title>Index.html Title</title>
</head>
<body>
Index.html
</body>
</html>```


# if index.txt is specifically requested, we expect index.txt
GET https://localhost:9443/index.txt
[Options]
insecure: true
HTTP 200
[Asserts]
body == "index.txt"

# requests for sub-folder followed by .. result in sanitized path
GET https://localhost:9443/non-existent/../index.txt
[Options]
insecure: true
HTTP 200
[Asserts]
body == "index.txt"

# results out of root folder are sanitized,
# and conform to default index list sequence.
GET https://localhost:9443/../
[Options]
insecure: true
HTTP 200
[Asserts]
```
<!DOCTYPE html>
<html>
<head>
<title>Index.html Title</title>
</head>
<body>
Index.html
</body>
</html>```


# Configure Caddy with custsom index "index.txt"
POST http://localhost:2019/load
Content-Type: text/caddyfile
```
{
skip_install_trust
http_port 9080
https_port 9443
local_certs
debug
}
localhost {
root {{indexed_root}}
file_server {
index index.txt
}
}
```

GET https://localhost:9443
[Options]
insecure: true
HTTP 200
[Asserts]
body == "index.txt"


# Configure with a root not containing index files
POST http://localhost:2019/load
Content-Type: text/caddyfile
```
{
skip_install_trust
http_port 9080
https_port 9443
local_certs
debug
}
localhost {
root {{unindexed_root}}
file_server
}
```

GET https://localhost:9443
[Options]
insecure: true
HTTP 404
22 changes: 22 additions & 0 deletions caddytest/spec/http/headers/spec.hurl
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Configure Caddy
POST http://localhost:2019/load
Content-Type: text/caddyfile
```
{
skip_install_trust
http_port 9080
https_port 9443
local_certs
debug
}
localhost {
header "X-Custom-Header" "Custom-Value"
}
```

GET https://localhost:9443
[Options]
insecure: true
HTTP 200
[Asserts]
header "X-Custom-Header" == "Custom-Value"
66 changes: 66 additions & 0 deletions caddytest/spec/http/rewrite/spec.hurl
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Configure Caddy
POST http://localhost:2019/load
User-Agent: hurl/ci
Content-Type: text/caddyfile
```
{
skip_install_trust
http_port 9080
https_port 9443
local_certs
}
localhost {
rewrite /from /to
respond {uri}
}
```

# simple scenario: rewriting /from to /to produces expected result of seeing /to
GET https://localhost:9443/from
[Options]
insecure: true
HTTP 200
[Asserts]
body == "/to"

# unmatched path is passed through unchanged
GET https://localhost:9443
[Options]
insecure: true
HTTP 200
[Asserts]
body == "/"

# having a query parameter does not trip the rewrite and retains the query
GET https://localhost:9443/from?query_param=value
[Options]
insecure: true
HTTP 200
[Asserts]
body == "/to?query_param=value"


# Configure Caddy
POST http://localhost:2019/load
User-Agent: hurl/ci
Content-Type: text/caddyfile
```
{
skip_install_trust
http_port 9080
https_port 9443
local_certs
}
localhost {
rewrite /from /to?a=b
respond {uri}
}
```

# a rewrite with query parameters affects the parameters
GET https://localhost:9443/from?query_param=value
[Options]
insecure: true
HTTP 200
[Asserts]
body == "/to?a=b"
Loading
Loading