Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS Marketplace Configuration #403

Open
wants to merge 83 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
817bf27
made lrs_install.yml component
Mar 22, 2024
06bff38
added service definition to lrs_install
Mar 22, 2024
8690e62
made sure each step was annotated with ExecuteBash
Mar 27, 2024
8a65ca0
beginnings of official ami pipeline
Apr 1, 2024
6d5d6c8
Merge branch 'main' into ami-build
Apr 1, 2024
76704c3
added trigger-update for AMI
Apr 1, 2024
65cc77e
added step for starting the pipeline
deathtenk Apr 2, 2024
afad5a5
fixed #!/bin/bash
deathtenk Apr 2, 2024
e643f17
made template for parameterizing SSM
deathtenk Apr 8, 2024
3d8760a
made ssm template generate random password on deploy
deathtenk Apr 9, 2024
fb3a194
changed up db and lrs for marketplace
deathtenk Apr 10, 2024
0abed55
added exports for db subnets
deathtenk Apr 11, 2024
7948c48
made subnets refs
deathtenk Apr 11, 2024
bd8e66c
made template for proxy
deathtenk Apr 11, 2024
04030a7
linted proxy template
deathtenk Apr 11, 2024
f9db370
added subnet IDs reference but not working will fix tonight
deathtenk Apr 11, 2024
0beb69c
fixed db subnet ref
deathtenk Apr 12, 2024
2bf0f9a
various fixes for proxy template
deathtenk Apr 15, 2024
4224dd8
version bump on RDSCreateUser
deathtenk Apr 15, 2024
abacbb1
added sg ingress rules for lambda
deathtenk Apr 16, 2024
0e132a4
corrected createuseringress description
deathtenk Apr 16, 2024
2e65f2d
made LrsqlRdsProxyTargetGroup
deathtenk Apr 16, 2024
0be9f60
added dbclusteridentifier output
deathtenk Apr 16, 2024
756afda
added necessary policies for LrsqlRdsProxy
deathtenk Apr 16, 2024
18d683a
fixed perms and created output endpoint
deathtenk Apr 17, 2024
35d01e0
added ssm instance config and fixed access perms
deathtenk Apr 18, 2024
91854d8
Merge branch 'main' into ami-build
deathtenk Apr 18, 2024
4df21b4
blew out proxy template and split it into two optionals
deathtenk Apr 18, 2024
680d919
made nginx proxy
deathtenk Apr 18, 2024
d823ac7
fixed some refs and sg
deathtenk Apr 18, 2024
2e3caed
fixed proxy not executing
deathtenk Apr 18, 2024
a45b704
condensed userdata script
deathtenk Apr 18, 2024
61409ec
removed yet-dev-access key from Proxy instance
deathtenk Apr 18, 2024
0c40122
wrote lambda that forms the ingress_rules for the sg
deathtenk Apr 19, 2024
f140ab3
working marketplace deploy
deathtenk May 7, 2024
fe6554c
Merge branch 'main' into ami-build
deathtenk May 7, 2024
e700af6
added jwt generation resources to marketplace
deathtenk May 7, 2024
e4eab18
converted launchconfig to launchtemplate
deathtenk May 8, 2024
0a4c79b
changed tunnel to support proxying over port 22
deathtenk May 9, 2024
ccd1dd9
moved trigger-update script to marketplace scripts
deathtenk May 9, 2024
17c99d7
fixed securitygroup ref
deathtenk May 9, 2024
0b248ea
renamed 0_imagepipeline since not part of deployment order
deathtenk May 13, 2024
cab66a0
first pass at integrating ami build into GH actions
deathtenk May 13, 2024
23320bf
added regional mappings for app template
deathtenk May 14, 2024
09a8ba4
updated trigger-update to only take version as param
deathtenk May 14, 2024
f80bae5
cleaned up ami ref
deathtenk May 14, 2024
162bcee
added Distribution settings
deathtenk May 14, 2024
618b798
fixed region ref
deathtenk May 14, 2024
a614480
fixed actions script that builds AMI
deathtenk May 14, 2024
9990f44
made role a string since roles are not sensitive
deathtenk May 14, 2024
83b973c
updated params for ami build script
deathtenk May 14, 2024
41863bc
made gh actions policy+role
deathtenk May 14, 2024
fd522bf
got rid of commented out config in 2_db.yml
deathtenk May 14, 2024
9610bc7
temporarily changed trigger method for action for gh cli
deathtenk May 15, 2024
cb08dbf
temporarily put workflow_dispatch on ami action
deathtenk May 15, 2024
29e3083
added sudo to jq install script
deathtenk May 15, 2024
49fcd79
uncommented region param
deathtenk May 15, 2024
91fd458
made sure checkout is using ami-build branch
deathtenk May 15, 2024
18d4997
updated relative PATH for script call
deathtenk May 15, 2024
abb1b5c
corrected path to trigger script
deathtenk May 15, 2024
e97e029
enforced version parameter in trigger-update.sh
deathtenk May 15, 2024
4b030a4
removed trailing v for SEMANTIC_VERSION
deathtenk May 15, 2024
89c8641
updated gh actions role with correct perms
deathtenk May 15, 2024
b155d6b
updated with GetImage access to gh actions policy
deathtenk May 15, 2024
21857b9
added GetComponent to policy
deathtenk May 15, 2024
a41aa44
added GetImageRecipe to gh actions role
deathtenk May 15, 2024
0e7bfe2
made ghactions imagebuilder role more permissive
deathtenk May 15, 2024
14ff7f1
fixed gha trigger for release cut
deathtenk May 15, 2024
1898943
Merge branch 'main' into ami-build
deathtenk May 28, 2024
b720ff9
added valid db types to template
deathtenk May 30, 2024
55a70c6
made ephemeral mode for LRSQL
deathtenk Jun 3, 2024
be681f2
Merge branch 'main' into ami-build
deathtenk Jun 4, 2024
ebd0865
added some more machines to the template
deathtenk Jun 11, 2024
bbcb0f7
added milts parameters for the ALB to template
deathtenk Jun 13, 2024
1823234
Merge branch 'main' into ami-build
deathtenk Jun 18, 2024
5c730e0
added logLevel to 3_lrs.yml
deathtenk Jun 18, 2024
c9c8218
updated trigger-update to be idempotent
deathtenk Aug 12, 2024
389106b
template defines the bridge between imagebuilder and SSM
deathtenk Aug 12, 2024
62b53c2
Merge branch 'main' into ami-build
deathtenk Aug 12, 2024
4238201
updated marketplace lrs template to use ssm AMI
deathtenk Aug 12, 2024
6bac308
moved pipeline templates into own dir
deathtenk Aug 13, 2024
693ffbe
updated docs for marketplace template
deathtenk Aug 19, 2024
29823e7
Merge branch 'main' into ami-build
deathtenk Oct 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/workflows/ami.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Build AMI

on:
push:
tags:
- 'v*.*.*' # Enforce Semantic Versioning
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have you confirmed what happens if we overwrite a tag? That is something that happens on occasion.


jobs:
build_ami:
runs-on: ubuntu-latest
# These permissions are needed by configure-aws-credentials in order
# to interact with GitHub's OIDC Token endpoint.
permissions:
id-token: write # required to use OIDC authentication
contents: read # required to checkout the code from the repo
steps:
- name: Install JQ
run: sudo apt update && sudo apt install -y jq
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

im not seeing where this is utilized

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nevermind, i see it in the sh. Is this usually how we install shell deps?


- name: Checkout repository
uses: actions/checkout@v4

- name: Extract version
id: version
run: echo version=${GITHUB_REF#refs\/tags\/} >> $GITHUB_OUTPUT

- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1-node16
with:
role-to-assume: arn:aws:iam::376099832799:role/lrsql-imagebuilder-ghacti-BuildAMIGithubActionsRole-IHZUC98qYzfG
role-duration-seconds: 900 # 15 min; minimal duration possible
aws-region: us-east-1

- name: Trigger AMI Build
run: $GITHUB_WORKSPACE/dev-resources/template/marketplace/scripts/trigger-update.sh -v ${{ github.event.inputs.version }}
224 changes: 224 additions & 0 deletions dev-resources/template/marketplace/0_vpc.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,224 @@
AWSTemplateFormatVersion: '2010-09-09'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should i assume this is identical to old vpc? if so do we need the dupe?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is identical I just duped for consistency, but duping here isn't necessary.

Description: SQL LRS VPC With public & private Subnets
Parameters:
VpcCidr:
Description: CIDR block for the vpc itself
Type: String
Default: '173.147.0.0/16'
PublicOneCidr:
Description: CIDR block for public subnet one
Type: String
Default: '173.147.0.0/24'
PublicTwoCidr:
Description: CIDR block for public subnet two
Type: String
Default: '173.147.1.0/24'
PrivateOneCidr:
Description: CIDR block for private subnet one
Type: String
Default: '173.147.2.0/24'
PrivateTwoCidr:
Description: CIDR block for private subnet two
Type: String
Default: '173.147.3.0/24'

Resources:
# VPC
VPC:
Type: AWS::EC2::VPC
Properties:
EnableDnsSupport: true
EnableDnsHostnames: true
CidrBlock: !Ref VpcCidr
Tags:
- Key: Name
Value: !Sub '${AWS::StackName}-vpc'

# Two public subnets
PublicSubnetOne:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone:
Fn::Select:
- 0
- Fn::GetAZs: {Ref: 'AWS::Region'}
VpcId: !Ref 'VPC'
CidrBlock: !Ref PublicOneCidr
MapPublicIpOnLaunch: true
Tags:
- Key: Name
Value: !Sub '${AWS::StackName}-public-1'
PublicSubnetTwo:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone:
Fn::Select:
- 1
- Fn::GetAZs: {Ref: 'AWS::Region'}
VpcId: !Ref 'VPC'
CidrBlock: !Ref PublicTwoCidr
MapPublicIpOnLaunch: true
Tags:
- Key: Name
Value: !Sub '${AWS::StackName}-public-2'

# Two Private subnets
PrivateSubnetOne:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone:
Fn::Select:
- 0
- Fn::GetAZs: {Ref: 'AWS::Region'}
VpcId: !Ref 'VPC'
CidrBlock: !Ref PrivateOneCidr
Tags:
- Key: Name
Value: !Sub '${AWS::StackName}-private-1'
PrivateSubnetTwo:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone:
Fn::Select:
- 1
- Fn::GetAZs: {Ref: 'AWS::Region'}
VpcId: !Ref 'VPC'
CidrBlock: !Ref PrivateTwoCidr
Tags:
- Key: Name
Value: !Sub '${AWS::StackName}-private-2'

# Public Subnet Routing
InternetGateway:
Type: AWS::EC2::InternetGateway
GatewayAttachement:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref 'VPC'
InternetGatewayId: !Ref 'InternetGateway'
PublicRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref 'VPC'
PublicRoute:
Type: AWS::EC2::Route
DependsOn: GatewayAttachement
Properties:
RouteTableId: !Ref 'PublicRouteTable'
DestinationCidrBlock: '0.0.0.0/0'
GatewayId: !Ref 'InternetGateway'
PublicSubnetOneRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnetOne
RouteTableId: !Ref PublicRouteTable
PublicSubnetTwoRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnetTwo
RouteTableId: !Ref PublicRouteTable


# Private Subnet Routing
NatGatewayOneAttachment:
Type: AWS::EC2::EIP
DependsOn: GatewayAttachement
Properties:
Domain: vpc
NatGatewayTwoAttachment:
Type: AWS::EC2::EIP
DependsOn: GatewayAttachement
Properties:
Domain: vpc
NatGatewayOne:
Type: AWS::EC2::NatGateway
Properties:
AllocationId: !GetAtt NatGatewayOneAttachment.AllocationId
SubnetId: !Ref PublicSubnetOne
NatGatewayTwo:
Type: AWS::EC2::NatGateway
Properties:
AllocationId: !GetAtt NatGatewayTwoAttachment.AllocationId
SubnetId: !Ref PublicSubnetTwo
PrivateRouteTableOne:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref 'VPC'
PrivateRouteOne:
Type: AWS::EC2::Route
Properties:
RouteTableId: !Ref PrivateRouteTableOne
DestinationCidrBlock: 0.0.0.0/0
NatGatewayId: !Ref NatGatewayOne
PrivateRouteTableOneAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref PrivateRouteTableOne
SubnetId: !Ref PrivateSubnetOne
PrivateRouteTableTwo:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref 'VPC'
PrivateRouteTwo:
Type: AWS::EC2::Route
Properties:
RouteTableId: !Ref PrivateRouteTableTwo
DestinationCidrBlock: 0.0.0.0/0
NatGatewayId: !Ref NatGatewayTwo
PrivateRouteTableTwoAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref PrivateRouteTableTwo
SubnetId: !Ref PrivateSubnetTwo

Outputs:
VPCId:
Description: The ID of the VPC that this stack is deployed in
Value: !Ref 'VPC'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'VPCId' ] ]
PublicSubnetOne:
Description: Public subnet one
Value: !Ref 'PublicSubnetOne'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PublicSubnetOne' ] ]
PublicSubnetTwo:
Description: Public subnet two
Value: !Ref 'PublicSubnetTwo'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PublicSubnetTwo' ] ]
PrivateSubnetOne:
Description: Private subnet one
Value: !Ref 'PrivateSubnetOne'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PrivateSubnetOne' ] ]
PrivateSubnetOneAZ:
Description: Private subnet one AZ
Value: !GetAtt 'PrivateSubnetOne.AvailabilityZone'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PrivateSubnetOneAZ' ] ]
PrivateSubnetTwo:
Description: Private subnet two
Value: !Ref 'PrivateSubnetTwo'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PrivateSubnetTwo' ] ]
PrivateSubnetTwoAZ:
Description: Private subnet two AZ
Value: !GetAtt 'PrivateSubnetTwo.AvailabilityZone'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PrivateSubnetTwoAZ' ] ]
PrivateRouteTableOneId:
Description: Private route table 1 id
Value: !Ref PrivateRouteTableOne
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PrivateRouteTableOneId' ] ]
PrivateRouteTableTwoId:
Description: Private route table 2 id
Value: !Ref PrivateRouteTableTwo
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PrivateRouteTableTwoId' ] ]
PublicRouteTableId:
Description: Public Route table id
Value: !Ref PublicRouteTable
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PublicRouteTableId' ] ]
Loading
Loading