Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Beejones/updating npm packages #38

Merged
merged 45 commits into from
Jul 25, 2024
Merged
Show file tree
Hide file tree
Changes from 40 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
527bfa7
first round of updates
beejones Jul 18, 2024
ca68fd1
update package.json incrementally
beejones Jul 18, 2024
cb842c7
add global rollup
beejones Jul 18, 2024
0267e88
change order in dockerfile
beejones Jul 18, 2024
827f6fb
update docker container
beejones Jul 18, 2024
1a60c53
update docker
beejones Jul 19, 2024
70162fc
update dockerfile
beejones Jul 19, 2024
da78e53
update dockerfile
beejones Jul 19, 2024
9887104
remove devcontainer change
beejones Jul 19, 2024
f81c46c
fix install rollup
beejones Jul 19, 2024
e1d9e35
fix docker creation
beejones Jul 19, 2024
4a0996f
update packages
beejones Jul 19, 2024
1989e23
fix packages
beejones Jul 19, 2024
68f766f
remove echo in dockerfile
beejones Jul 19, 2024
34fd0f0
sync ci and devcontainer
beejones Jul 19, 2024
5d5dce9
fix ci failure
beejones Jul 19, 2024
4ad9d8b
fix ci container
beejones Jul 19, 2024
00be9de
incremental cleanup3 AuthenticationService
beejones Jul 19, 2024
4316745
sync ci and devcontainer
beejones Jul 19, 2024
6123373
incremental cleanup IAuthenticationService
beejones Jul 20, 2024
af7341b
incremental cleanup authorization
beejones Jul 20, 2024
7e6712d
incremental cleanup attestation
beejones Jul 20, 2024
9f856df
missing attestation change
beejones Jul 20, 2024
2a57d67
incremental cleanup policies
beejones Jul 21, 2024
41fe7bb
incremental cleanup REPOSITORIES
beejones Jul 21, 2024
3df654b
incremental cleanup utils
beejones Jul 22, 2024
12923d0
incremental cleanup e2etest
beejones Jul 22, 2024
5eaf0b8
incremental cleanup IKeyItems, KeyGeneration
beejones Jul 22, 2024
0359673
lint
beejones Jul 22, 2024
d21e777
undo changes to KeyGeneration
beejones Jul 22, 2024
cd86a3f
incremental cleanup KeyWrapper
beejones Jul 22, 2024
c1be619
incremental cleanup TinkKey
beejones Jul 22, 2024
4470daa
incremental cleanup keyEndpoint
beejones Jul 22, 2024
e639918
incremental cleanup kms, publickeyendpoint
beejones Jul 22, 2024
54a38bb
undo publicEndpoint
beejones Jul 22, 2024
12ac057
Update to key endpoints
beejones Jul 22, 2024
c530185
fix pip packages
beejones Jul 22, 2024
9e04db8
undo public key change
beejones Jul 22, 2024
4dedc76
Add strict tsconfig
beejones Jul 23, 2024
f577dea
remove strict
beejones Jul 23, 2024
589a497
switch back to strict mode
beejones Jul 23, 2024
7234b5f
Addressing review comments
beejones Jul 24, 2024
32b4ffa
Remove seperate wheel install
beejones Jul 24, 2024
2fe6bdf
Update JWT policy file to support arrays
beejones Jul 24, 2024
58b24da
update jwt demo policy
beejones Jul 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .devcontainer/Dockerfile.devcontainer
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Base container image which is built nightly
# Used as a starting point to make building other containers fast

ARG BASE_CCF_IMAGE=5.0.0-dev10-virtual
ARG BASE_CCF_IMAGE=5.0.0-rc0-virtual
ARG ENVIRONMENT=devcontainer

# ignore this hadolint error as BASE_IMAGE contains an image tag
Expand All @@ -12,6 +12,8 @@ ENV NVM_DIR /root/.nvm
ENV TINKEY_VERSION=tinkey-1.10.1
COPY .devcontainer/install_packages.sh .devcontainer/install_nodejs.sh .devcontainer/setup_tinkey.sh /src/

# "Install necessary packages."
RUN /src/install_packages.sh
RUN /src/install_nodejs.sh
RUN /src/setup_tinkey.sh
# "All necessary packages and tinkey setup completed."
5 changes: 2 additions & 3 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"dockerfile": "Dockerfile.devcontainer",
"context": ".."
},
"postCreateCommand": "cd /workspaces/azure-privacy-sandbox-kms && npm i && make build",
"postCreateCommand": "cd /workspaces/azure-privacy-sandbox-kms && scripts/set_python_env.sh && npm i && make build",
"features": {
"ghcr.io/devcontainers/features/docker-in-docker:2": {
"version": "latest",
Expand All @@ -27,6 +27,5 @@
"settings": {
"editor.defaultFormatter": "ms-python.black-formatter"
}
},
"postStartCommand": "pip install -r .devcontainer/requirements.txt && pre-commit install"
}
}
4 changes: 2 additions & 2 deletions .devcontainer/install_nodejs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,5 @@ curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash
&& nvm install node \
&& nvm use node

. $NVM_DIR/nvm.sh \
&& npm install -g npm@latest
echo "Install rollup"
npm install -g [email protected]
34 changes: 7 additions & 27 deletions .github/Dockerfile.ci
Original file line number Diff line number Diff line change
@@ -1,37 +1,17 @@
# Base container image which is built nightly
# Used as a starting point to make building other containers fast

ARG BASE_CCF_IMAGE=5.0.0-dev10-virtual
ARG BASE_CCF_IMAGE=5.0.0-rc0-virtual
ARG ENVIRONMENT=ci

# ignore this hadolint error as BASE_IMAGE contains an image tag
# hadolint ignore=DL3006
FROM mcr.microsoft.com/ccf/app/dev:${BASE_CCF_IMAGE} as base

# Custom Deps
RUN apt-get update && apt-get install -y \
python3-pip \
openssh-client \
make \
libuv1 \
jq \
lsof \
sudo \
tar \
default-jre
COPY .devcontainer/install_packages.sh .devcontainer/install_nodejs.sh .devcontainer/setup_tinkey.sh /src/

# Install Node.js
RUN curl -fsSL https://deb.nodesource.com/setup_current.x | bash -
RUN apt-get install -y nodejs
RUN pip install --upgrade pip setuptools

RUN apt-get -y autoremove \
&& apt-get -y clean

# Setup tinkey
ENV TINKEY_VERSION=tinkey-1.10.1
RUN curl -O https://storage.googleapis.com/tinkey/$TINKEY_VERSION.tar.gz
RUN tar -xzvf $TINKEY_VERSION.tar.gz
RUN cp tinkey /usr/bin/
RUN cp tinkey_deploy.jar /usr/bin/
RUN rm tinkey tinkey_deploy.jar tinkey.bat $TINKEY_VERSION.tar.gz
# "Install necessary packages."
RUN /src/install_packages.sh
RUN /src/install_nodejs.sh
RUN /src/setup_tinkey.sh
# "All necessary packages and tinkey setup completed."
2 changes: 1 addition & 1 deletion .github/devcontainer.json
Copy link
Contributor

@takuro-sato takuro-sato Jul 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the reason that we have two devcontainer.json?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one for CI and one for the devcontainer. Are you saying we can have one for both?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I just didn't know about devcontainers/[email protected].

Are you saying we can have one for both?

I guess it's preferable, but what's your view on that? At least I don't think we need to address it within this PR.

Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"dockerfile": "Dockerfile.ci",
"context": ".."
},
"postCreateCommand": "cd /workspaces/azure-privacy-sandbox-kms && npm i && make build",
"postCreateCommand": "cd /workspaces/azure-privacy-sandbox-kms && scripts/set_python_env.sh && npm i && make build",
"features": {
"ghcr.io/devcontainers/features/docker-in-docker:2": {
"version": "latest",
Expand Down
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,6 @@ lint: ## 🔍 Lint the code base (but don't fix)

# Keep this at the bottom.
clean: ## 🧹 Clean the working folders created during build/demo
@rm -rf .venv_ccf_sandbox
@rm -rf ${PYTHON_VENV}
@rm -rf ${KMS_WORKSPACE}
@rm -rf dist
@rm -rf dist
71 changes: 30 additions & 41 deletions package.json
takuro-sato marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -11,50 +11,39 @@
"node": ">=16"
},
"dependencies": {
"@bufbuild/buf": "^1.30.0",
"@bufbuild/protobuf": "^1.8.0",
"@bufbuild/protoc-gen-es": "^1.8.0",
"@microsoft/ccf-app": "^5.0.0-dev6",
"express": "^4.19.2",
"i": "^0.3.7",
"js-base64": "^3.5.2",
"jsrsasign": "^11.0.0",
"jsrsasign-util": "^1.0.2",
"jwt-decode": "^3.0.0",
"lodash-es": "^4.17.15",
"node-forge": "^1.3.1",
"npm": "^10.5.0",
"pem-jwk": "^2.0.0"
"@bufbuild/protobuf": "1.10.0",
"@microsoft/ccf-app": "5.0.0",
"js-base64": "3.7.7",
"node-forge": "1.3.1",
"npm": "10.8.2",
"pem-jwk": "2.0.0"
},
"devDependencies": {
"@babel/core": "^7.20.5",
"@babel/preset-env": "^7.20.2",
"@babel/preset-typescript": "^7.22.15",
"@jest/globals": "^29.7.0",
"@rollup/plugin-commonjs": "^17.1.0",
"@rollup/plugin-node-resolve": "^11.2.1",
"@rollup/plugin-typescript": "^8.2.0",
"@types/jasmine": "^4.3.0",
"@types/jest": "^29.5.5",
"@types/jsrsasign": "^8.0.7",
"@types/lodash-es": "^4.17.3",
"@types/node": "^18.18.13",
"axios": "^1.5.0",
"babel-jest": "^29.3.1",
"del-cli": "^5.0.0",
"http-server": "^0.13.0",
"@babel/core": "7.24.9",
"@babel/preset-env": "7.24.8",
"@babel/preset-typescript": "7.24.7",
"@jest/globals": "29.7.0",
"@rollup/plugin-commonjs": "26.0.1",
"@rollup/plugin-node-resolve": "15.2.3",
"@rollup/plugin-typescript": "11.1.6",
"@types/jasmine": "5.1.4",
"@types/jest": "29.5.12",
"@types/node": "20.14.11",
"axios": "1.7.2",
"babel-jest": "29.7.0",
"express": "4.19.2",
"inquirer": "9.1.4",
"jest": "^29.7.0",
"js-crypto-key-utils": "^1.0.7",
"js-crypto-rsa": "^1.0.7",
"jsonwebtoken": "^9.0.0",
"node-fetch": "^3.3.2",
"rollup": "^2.79.1",
"ts-jest": "^29.1.5",
"ts-node": "^10.9.2",
"tslib": "^2.0.1",
"typescript": "^5.4.5",
"uglify-js": "^3.17.4"
"jest": "29.7.0",
"js-crypto-key-utils": "1.0.7",
"js-crypto-rsa": "1.0.7",
"jsonwebtoken": "9.0.2",
"node-fetch": "3.3.2",
"rollup": "4.18.1",
"ts-jest": "29.2.2",
"ts-node": "10.9.2",
"tslib": "2.6.3",
"typescript": "5.5.3",
"uglify-js": "3.19.0"
},
"jest": {
"preset": "ts-jest/presets/default-esm",
Expand Down
164 changes: 134 additions & 30 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,31 +1,135 @@
ccf
adtk==0.6.2
anyio==4.4.0
asn1crypto==1.5.1
attrs==23.2.0
azure-core==1.30.2
azure-storage-blob==12.21.0
bcrypt==4.1.3
better-exceptions==0.3.3
blinker==1.8.2
Brotli==1.1.0
cbor2==5.6.4
ccf==5.0.0
certifi==2024.7.4
certvalidator==0.11.1
cffi==1.16.0
cfgv==3.4.0
charset-normalizer==3.3.2
cimetrics==0.3.15
click==8.1.7
colorama==0.4.6
Columnar==1.4.1
ConfigArgParse==1.7
contourpy==1.1.1
cramjam==2.8.3
cryptography==42.0.8
cycler==0.12.1
detect-secrets==1.5.0
distlib==0.3.8
dnspython==2.6.1
docker==7.1.0
docutils==0.20.1
ecdsa==0.19.0
exceptiongroup==1.2.2
fastparquet==2023.10.1
filelock==3.15.4
flask==3.0.3
Flask-Cors==4.0.1
Flask-Login==0.6.3
fonttools==4.53.1
fsspec==2024.6.1
gevent==24.2.1
geventhttpclient==2.0.12
gitdb==4.0.11
GitPython==3.1.43
greenlet==3.0.3
grpcio==1.65.1
grpcio-tools==1.44.0
h11==0.14.0
h2==4.1.0
hpack==4.0.0
httpcore==0.16.3
httpx==0.23.3
hyperframe==6.0.1
identify==2.6.0
idna==3.7
importlib-metadata==8.0.0
importlib-resources==6.4.0
isodate==0.6.1
itsdangerous==2.2.0
jinja2==3.1.4
joblib==1.4.2
jsonschema==4.23.0
jsonschema-path==0.3.3
jsonschema-specifications==2023.12.1
jwcrypto==1.5.6
kiwisolver==1.4.5
lazy-object-proxy==1.10.0
locust==2.25.0
loguru==0.7.2
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.7.5
mdurl==0.1.2
msgpack==1.0.8
nodeenv==1.9.1
numpy==1.24.4
openapi-schema-validator==0.6.2
openapi-spec-validator==0.7.1
oscrypto==1.3.0
packaging==24.1
pandas==2.0.3
paramiko==3.4.0
pathable==0.4.3
patsy==0.5.6
pillow==10.4.0
pkgutil-resolve-name==1.3.10
platformdirs==4.2.2
plotext==5.2.8
polars==1.2.1
pre-commit==3.5.0
prettytable==3.10.2
protobuf==3.20.3
psutil==6.0.0
py-spy==0.3.14
pyasn1==0.6.0
pycose==1.1.0
pycparser==2.22
pygments==2.18.0
PyJWT==2.8.0
pymongo==4.8.0
PyNaCl==1.5.0
pyOpenSSL==24.2.1
pyparsing==2.4.7
python-dateutil==2.9.0.post0
python-iptables==1.0.1
pytz==2024.1
PyYAML==6.0.1
pyzmq==26.0.3
referencing==0.31.0
wheel
paramiko
loguru
psutil
cimetrics>=0.2.1
openapi-spec-validator
PyJWT
docutils
python-iptables
py-spy
GitPython
docker
better_exceptions
pyasn1
Jinja2
httpx[http2] == 0.23.*
locust
pyOpenSSL
grpcio-tools == 1.44.0 # Pin to a working version for SNP platform
JWCrypto == 1.5.*
pycose
rich
# Piccolo dependencies
fastparquet==2023.*
prettytable==3.*
polars
plotext
pre-commit
detect-secrets
requests==2.32.3
rfc3339-validator==0.1.4
rfc3986==1.5.0
rich==13.7.1
roundrobin==0.0.4
rpds-py==0.19.0
scikit-learn==1.3.2
scipy==1.10.1
six==1.16.0
smmap==5.0.1
sniffio==1.3.1
statsmodels==0.14.1
string-color==1.2.3
tabulate==0.9.0
threadpoolctl==3.5.0
tomli==2.0.1
toolz==0.12.1
typing-extensions==4.12.2
tzdata==2024.1
urllib3==2.2.2
virtualenv==20.26.3
wcwidth==0.2.13
werkzeug==3.0.3
zipp==3.19.2
zope.event==5.0
zope.interface==6.4.post2
1 change: 1 addition & 0 deletions scripts/set_python_env.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ if [ ! -d "$PYTHON_VENV" ]; then
python3.8 -m venv $PYTHON_VENV
echo "Activating virtual environment and installing dependencies..."
source $PYTHON_VENV/bin/activate
pip install wheel
pip install -U -r ./requirements.txt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't we install wheel in requirements.txt?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good question. I used to only have it in requirements and it generated an error. This solves the error.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does it solve the problem?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. After fixing the versions of the packages, the error is gone and there is no more need to install wheel first. This must have been a conflict between installed packages

else
echo "Activating existing virtual environment..."
Expand Down
2 changes: 1 addition & 1 deletion src/attestation/SnpAttestationClaims.ts
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ export class SnpAttestationClaims {
// report_id_ma
// reported_tcb
// chip_id
// committed_tcb
// committed_tcbx`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this change for?

Copy link
Contributor Author

@beejones beejones Jul 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no idea how this happened. Will fix this

// current_minor
// current_build
// committed_minor
Expand Down
9 changes: 6 additions & 3 deletions src/authorization/AuthenticationService.ts
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,11 @@ export class AuthenticationService implements IAuthenticationService {
*/
public isAuthenticated(
request: ccfapp.Request<any>,
): [ccfapp.AuthnIdentityCommon, ServiceResult<string>] {
let caller: ccfapp.AuthnIdentityCommon;
): [
ccfapp.AuthnIdentityCommon | undefined,
ServiceResult<string> | undefined,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we always return ServiceResult<string>? I think we want validator.validate(request) without ? after confirming validator is not null/undefined.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a failed result so undefined is never returned.

] {
let caller: ccfapp.AuthnIdentityCommon | undefined = undefined;
try {
const caller = request.caller as unknown as ccfapp.AuthnIdentityCommon;
if (!caller) {
Expand All @@ -61,7 +64,7 @@ export class AuthenticationService implements IAuthenticationService {
const validator = this.validators.get(
<CcfAuthenticationPolicyEnum>caller.policy,
);
return [caller, validator.validate(request)];
return [caller, validator?.validate(request)];
} catch (ex) {
return [
caller,
Expand Down
Loading
Loading