Skip to content

Commit b60e32c

Browse files
valayDavesavingoyal
valayDave
and
savingoyal
authored
Prep for OSS release! (#1)
OSS release for Metaflowbot Co-authored-by: Savin <[email protected]>
1 parent a59d784 commit b60e32c

25 files changed

+966
-178
lines changed

‎.github/workflows/docker_publish.yml

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
name: Docker Image CI
2+
on:
3+
release:
4+
types : [published]
5+
jobs:
6+
build:
7+
runs-on: ubuntu-latest
8+
steps:
9+
- name: Checkout
10+
uses: actions/checkout@v2
11+
- name: Docker meta
12+
id: meta
13+
uses: docker/metadata-action@v3
14+
with:
15+
images: |
16+
outerbounds/metaflowbot
17+
tags: |
18+
type=ref,event=branch
19+
type=ref,event=pr
20+
type=semver,pattern={{version}}
21+
type=semver,pattern={{major}}.{{minor}}
22+
type=sha
23+
- name: Set up Docker Buildx
24+
uses: docker/setup-buildx-action@v1
25+
- name: Login to DockerHub
26+
if: github.event_name != 'pull_request'
27+
uses: docker/login-action@v1
28+
with:
29+
username: ${{ secrets.DOCKER_USERNAME }}
30+
password: ${{ secrets.DOCKER_ACCESS_TOKEN }}
31+
- name: Build and push
32+
uses: docker/build-push-action@v2
33+
with:
34+
context: .
35+
push: ${{ github.event_name != 'pull_request' }}
36+
tags: ${{ steps.meta.outputs.tags }}
37+
labels: ${{ steps.meta.outputs.labels }}

‎.github/workflows/pypi_publish.yml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
name: Publish PyPi
2+
on:
3+
release:
4+
types: [published]
5+
jobs:
6+
publish:
7+
runs-on: ubuntu-latest
8+
steps:
9+
- uses: actions/checkout@v2
10+
- name: Set up Python 3.7.x
11+
uses: actions/setup-python@v1
12+
with:
13+
python-version: '3.7.x'
14+
- name: Install Python 3.7.x dependencies
15+
run: |
16+
python3 -m pip install --upgrade pip
17+
pip3 install setuptools wheel twine
18+
- name: Build package
19+
run: |
20+
python3 setup.py sdist bdist_wheel --universal
21+
- name: Publish package
22+
uses: pypa/gh-action-pypi-publish@release/v1
23+
with:
24+
user: __token__
25+
password: ${{ secrets.PYPI_PASSWORD }}
26+
verbose: true
27+

‎Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,5 +2,5 @@
22
FROM python:3.7.2
33
ADD . /metaflowbot
44
RUN pip3 install /metaflowbot/.
5-
RUN pip3 install git+https://github.com/outerbounds/metaflowbot-jokes-action
5+
RUN pip3 install metaflowbot-actions-jokes
66
CMD python3 -m metaflowbot --slack-bot-token $(echo $SLACK_BOT_TOKEN) server --admin $(echo $ADMIN_USER_ADDRESS)

‎LICENSE

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,7 @@
186186
same "printed page" as the copyright notice for easier
187187
identification within third-party archives.
188188

189-
Copyright [yyyy] [name of copyright owner]
189+
Copyright 2021 Netflix, Inc., Step Computing, Inc
190190

191191
Licensed under the Apache License, Version 2.0 (the "License");
192192
you may not use this file except in compliance with the License.

‎README.md

Lines changed: 14 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,20 @@
1-
# Metaflow Bot
1+
# Metaflowbot - Slack Bot for your Metaflow flows!
22

3-
## Documentation
4-
Thorough Documentation is present in the [Documentation folder](./docs)
5-
## Bot Commands
3+
Metaflowbot makes it fun and easy to monitor your Metaflow runs, past and present. Imagine starting a training run that lasts for hours - you can now monitor it anywhere using Slack on your mobile device!
64

7-
- `@flowey help` | `@flowey hi` : Help
5+
![2021-08-09 15 57 31](https://user-images.githubusercontent.com/763451/128784858-d9e37401-05de-4d02-82c5-29444ab4e1b3.gif)
6+
7+
The bot is [easy to deploy](./docs/deployment.md): It is just a Python process with few external dependencies - no databases needed. Its [security footprint is small](./docs/slack-scopes.md) as it uses only a tightly scoped set of Slack calls. During development you can run the bot on any workstation, so it is quick to [iterate on custom actions](./docs/creating-custom-actions.md) and extend it to suit your needs. For production deployments the bot ships with a [CloudFormation template](./deployment/mfbot-cfn-template.yml) for automating your deployments to AWS.
8+
9+
## Communicating with the bot
10+
11+
There are two ways interact with the Metaflow bot. You can invite the bot on a `channel` or directly speak to it via `direct message`.
12+
13+
- `@flowey help` : Help
814

915
- `@flowey tell me a joke`
1016

11-
- `@flowey inspect` | `@flowey how to inspect` : How to inspect
17+
- `@flowey how to inspect` : How to inspect
1218

1319
- `@flowey inspect HelloFlow` : Inspect `Run`s of a particular `Flow`
1420

@@ -20,34 +26,7 @@ Thorough Documentation is present in the [Documentation folder](./docs)
2026

2127
- `@flowey inspect the latest run of HelloFlow` : Inspect an individual `Run` instance
2228

23-
- `@flowey inspect dberg's latest run of HelloFlow` : Inspect an individual `Run` instance
24-
25-
26-
## Communicating with the bot
27-
28-
There are two places to interact with Metaflowbot : on a `channel` or via `direct message`. But for either places, the following is the general behavior of the bot:
29-
30-
> *When a user messages the bot, the bot will open a new message thread and will engage with the user on the same thread. The user can open multiple threads with the bot. Each thread is an independent discussion*
31-
32-
The following are interaction/UX restrictions based on where the user is conversing with the Metaflow bot.
33-
### Communicating with the bot on a channel
34-
35-
As the current [manifest.yml](./manifest.yml) only supports `app_mention` and `message.im` events. This means that when users want to talk to the bot on a channel, then they need to specifically need to mention `@flowey` or `@custombotname` to talk to the bot. We don't listen to messages on channels only `app_mentions`.
36-
37-
### Communicating with the bot through direct messages
38-
39-
Users can message the bot without `@` mentions via direct messages. The bot will support the same command list.
40-
41-
## References:
29+
- `@flowey inspect savin's latest run of HelloFlow` : Inspect an individual `Run` instance
4230

43-
- [Slack Permission Scopes](https://api.slack.com/scopes)
44-
- [Slack Events](https://api.slack.com/events)
45-
- [Slack Socket Mode](https://slack.dev/python-slack-sdk/socket-mode/index.html#socketmodeclient)
46-
- [How to make threads in slack via python API](https://slack.dev/python-slack-sdk/web/index.html)
4731

48-
## PRE OSS Release TODOS
49-
- [ ] Create Deployment on cloudformation
50-
- [ ] Check Auth of the deployed template.
51-
- [ ] Deploy on ECS as a Fargate task
52-
- [X] Deploy on Heroku with New OSS Code.
53-
- [ ] Document the Rule data structure
32+
If you require some customization for your deployment or need additional help, please feel free to reach out to us at http://slack.outerbounds.co. We are very happy to help!

‎deployment/mfbot-cfn-template.yml

Lines changed: 246 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
1+
Description: Cloudformation Stack for Deploying Metaflowbot
2+
Parameters:
3+
AdminEmailAddress:
4+
Description: Email address of the admin user in the slack workspace
5+
Type: String
6+
MetadataServiceAuthParameterKey:
7+
Default: METADATASERVICE_AUTH_KEY
8+
Description: Key for Metadata service auth parameter in Secrets Manager.
9+
Type: String
10+
MetadataServiceUrl:
11+
Description: URL of the metadata service
12+
Type: String
13+
MetaflowDatastoreSysrootS3:
14+
Description: 'Amazon S3 URL for Metaflow DataStore '
15+
Type: String
16+
MetaflowbotSecretsManagerARN:
17+
Description: ARN of the secret holding Metaflowbot credentials in Secrets Manager
18+
Type: String
19+
SlackAppTokenParameterKey:
20+
Default: SLACK_APP_TOKEN_KEY
21+
Description: Key for SLACK_APP_TOKEN parameter in Secrets Manager.
22+
Type: String
23+
SlackBotTokenParameterKey:
24+
Default: SLACK_BOT_TOKEN_KEY
25+
Description: Key for SLACK_BOT_TOKEN parameter in Secrets Manager.
26+
Type: String
27+
Resources:
28+
EcsClusterRole:
29+
Properties:
30+
AssumeRolePolicyDocument:
31+
Statement:
32+
- Action: sts:AssumeRole
33+
Effect: Allow
34+
Principal:
35+
Service: ecs-tasks.amazonaws.com
36+
Version: '2012-10-17'
37+
ManagedPolicyArns:
38+
- arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM
39+
Path: /
40+
Type: AWS::IAM::Role
41+
EcsTaskRole:
42+
Properties:
43+
AssumeRolePolicyDocument:
44+
Statement:
45+
- Action: sts:AssumeRole
46+
Effect: Allow
47+
Principal:
48+
Service: ecs-tasks.amazonaws.com
49+
Version: '2012-10-17'
50+
Path: /
51+
Type: AWS::IAM::Role
52+
InternetGatewayAttachment:
53+
Properties:
54+
InternetGatewayId: !Ref 'MetaflowbotInternetGateway'
55+
VpcId: !Ref 'MetaflowbotPublicVpc'
56+
Type: AWS::EC2::VPCGatewayAttachment
57+
MetaflowbotCluster:
58+
Type: AWS::ECS::Cluster
59+
MetaflowbotDeployment:
60+
Properties:
61+
Cluster: !Ref 'MetaflowbotCluster'
62+
DesiredCount: 1
63+
LaunchType: FARGATE
64+
NetworkConfiguration:
65+
AwsvpcConfiguration:
66+
AssignPublicIp: ENABLED
67+
SecurityGroups:
68+
- !Ref 'MetaflowbotSecurityGroup'
69+
Subnets:
70+
- !Ref 'MetaflowbotDeploymentSubnet'
71+
TaskDefinition: !Ref 'MetaflowbotTaskDefinition'
72+
Type: AWS::ECS::Service
73+
MetaflowbotDeploymentSubnet:
74+
Properties:
75+
AvailabilityZone: !Select
76+
- 0
77+
- !GetAZs
78+
Ref: AWS::Region
79+
CidrBlock: 10.0.0.0/24
80+
MapPublicIpOnLaunch: true
81+
VpcId: !Ref 'MetaflowbotPublicVpc'
82+
Type: AWS::EC2::Subnet
83+
MetaflowbotInternetGateway:
84+
Type: AWS::EC2::InternetGateway
85+
MetaflowbotLogGroup:
86+
Properties:
87+
LogGroupName: !Join
88+
- ''
89+
- - /ecs/
90+
- !Ref 'AWS::StackName'
91+
- -metaflowbot
92+
Type: AWS::Logs::LogGroup
93+
MetaflowbotPublicVpc:
94+
Properties:
95+
CidrBlock: 10.0.0.0/16
96+
Type: AWS::EC2::VPC
97+
MetaflowbotSecretAccess:
98+
Properties:
99+
PolicyDocument:
100+
Statement:
101+
- Action:
102+
- secretsmanager:GetSecretValue
103+
Effect: Allow
104+
Resource:
105+
- !Ref 'MetaflowbotSecretsManagerARN'
106+
Sid: S3GetObject
107+
Version: '2012-10-17'
108+
PolicyName: Metaflowbot
109+
Roles:
110+
- !Ref 'EcsClusterRole'
111+
Type: AWS::IAM::Policy
112+
MetaflowbotSecurityGroup:
113+
Properties:
114+
GroupDescription: Allow All In and outbound traffic
115+
SecurityGroupEgress:
116+
- CidrIp: '0.0.0.0/0'
117+
FromPort: 0
118+
IpProtocol: tcp
119+
ToPort: 65534
120+
VpcId: !Ref 'MetaflowbotPublicVpc'
121+
Type: AWS::EC2::SecurityGroup
122+
MetaflowbotTaskDefinition:
123+
Properties:
124+
ContainerDefinitions:
125+
- Environment:
126+
- Name: ADMIN_USER_ADDRESS
127+
Value: !Ref 'AdminEmailAddress'
128+
- Name: USERNAME
129+
Value: slackbot
130+
- Name: METAFLOW_SERVICE_URL
131+
Value: !Ref 'MetadataServiceUrl'
132+
- Name: METAFLOW_DATASTORE_SYSROOT_S3
133+
Value: !Ref 'MetaflowDatastoreSysrootS3'
134+
- Name: METAFLOW_DEFAULT_DATASTORE
135+
Value: s3
136+
- Name: METAFLOW_DEFAULT_METADATA
137+
Value: service
138+
Essential: true
139+
Image: outerbounds/metaflowbot
140+
LogConfiguration:
141+
LogDriver: awslogs
142+
Options:
143+
awslogs-group: !Join
144+
- ''
145+
- - /ecs/
146+
- !Ref 'AWS::StackName'
147+
- -metaflowbot
148+
awslogs-region: !Ref 'AWS::Region'
149+
awslogs-stream-prefix: ecs
150+
Name: metaflowbot
151+
Secrets:
152+
- Name: METAFLOW_SERVICE_AUTH_KEY
153+
ValueFrom: !Join
154+
- ''
155+
- - !Ref 'MetaflowbotSecretsManagerARN'
156+
- ':'
157+
- !Ref 'MetadataServiceAuthParameterKey'
158+
- '::'
159+
- Name: SLACK_APP_TOKEN
160+
ValueFrom: !Join
161+
- ''
162+
- - !Ref 'MetaflowbotSecretsManagerARN'
163+
- ':'
164+
- !Ref 'SlackAppTokenParameterKey'
165+
- '::'
166+
- Name: SLACK_BOT_TOKEN
167+
ValueFrom: !Join
168+
- ''
169+
- - !Ref 'MetaflowbotSecretsManagerARN'
170+
- ':'
171+
- !Ref 'SlackBotTokenParameterKey'
172+
- '::'
173+
Cpu: '4096'
174+
ExecutionRoleArn: !GetAtt 'EcsClusterRole.Arn'
175+
Memory: '8192'
176+
NetworkMode: awsvpc
177+
RequiresCompatibilities:
178+
- FARGATE
179+
TaskRoleArn: !GetAtt 'EcsTaskRole.Arn'
180+
Type: AWS::ECS::TaskDefinition
181+
PolicyEcr:
182+
Properties:
183+
PolicyDocument:
184+
Statement:
185+
- Action:
186+
- ecr:GetAuthorizationToken
187+
Effect: Allow
188+
Resource:
189+
- '*'
190+
- Action:
191+
- ecr:GetDownloadUrlForLayer
192+
- ecr:BatchGetImage
193+
- ecr:BatchCheckLayerAvailability
194+
- logs:CreateLogStream
195+
- logs:PutLogEvents
196+
Effect: Allow
197+
Resource:
198+
- '*'
199+
Sid: AllowPull
200+
Version: '2012-10-17'
201+
PolicyName: MetaflowbotEcrPolicy
202+
Roles:
203+
- !Ref 'EcsClusterRole'
204+
Type: AWS::IAM::Policy
205+
PublicDefaultRoute:
206+
Properties:
207+
DestinationCidrBlock: '0.0.0.0/0'
208+
GatewayId: !Ref 'MetaflowbotInternetGateway'
209+
RouteTableId: !Ref 'PublicRouteTable'
210+
Type: AWS::EC2::Route
211+
PublicRouteAssociation:
212+
Properties:
213+
RouteTableId: !Ref 'PublicRouteTable'
214+
SubnetId: !Ref 'MetaflowbotDeploymentSubnet'
215+
Type: AWS::EC2::SubnetRouteTableAssociation
216+
PublicRouteTable:
217+
Properties:
218+
VpcId: !Ref 'MetaflowbotPublicVpc'
219+
Type: AWS::EC2::RouteTable
220+
S3AccessPolicy:
221+
Properties:
222+
PolicyDocument:
223+
Statement:
224+
- Action:
225+
- s3:GetObject
226+
- s3:ListBucket
227+
Effect: Allow
228+
Resource:
229+
- !Join
230+
- ''
231+
- - !Join
232+
- ''
233+
- - 'arn:aws:s3:::'
234+
- !Select
235+
- 1
236+
- !Split
237+
- s3://
238+
- !Ref 'MetaflowDatastoreSysrootS3'
239+
- /*
240+
Sid: S3GetObject
241+
Version: '2012-10-17'
242+
PolicyName: MetaflowbotS3AccessPolicy
243+
Roles:
244+
- !Ref 'EcsTaskRole'
245+
Type: AWS::IAM::Policy
246+

0 commit comments

Comments
 (0)