[Tooling] Integrate Tilt development environment

README.md

@@ -21,12 +21,13 @@ To learn more about SubQuery, [see their docs](https://academy.subquery.network)
   - [tl;dr local development (if not your first time)](#tldr-local-development-if-not-your-first-time)
   - [1. Install dependencies](#1-install-dependencies)
   - [2. Generate types](#2-generate-types)
-  - [3. Run](#3-run)
+  - [3a. Run via Tilt](#3a-run-via-tilt)
+  - [3b. Run via docker-compose](#3b-run-via-docker-compose) 
     - [Localnet ONLY](#localnet-only)
-    - [3.1 Debugging, errors running \& building](#31-debugging-errors-running--building)
-    - [3.2 Using a pre-built image](#32-using-a-pre-built-image)
-    - [3.3 Available Scripts breakdown](#33-available-scripts-breakdown)
-    - [3.4 Using k8s](#34-using-k8s)
+    - [3b.1 Debugging, errors running \& building](#31-debugging-errors-running--building)
+    - [3b.2 Using a pre-built image](#32-using-a-pre-built-image)
+    - [3b.3 Available Scripts breakdown](#33-available-scripts-breakdown)
+  - [3c. Using k8s](#3c-using-k8s)
 
 ## Usage & Query Docs
 
@@ -39,6 +40,9 @@ Connect to the postgres container, update the schema, and explore!
 
 ```bash
 docker exec -it pocketdex_development-postgres-1 psql -U postgres -d postgres
+SET SCHEMA "testnet";
+# OR, if indexing localnet:
+# SET SCHEMA "localnet";
 SET search_path TO app;
 \dt
 ```
@@ -160,7 +164,38 @@ Types will need to be regenerated any time the `graphql.schema` is changed.
 yarn run codegen
 ```
 
-### 3. Run
+### 3a. Run via [Tilt](https://tilt.dev/)
+
+![Tilt](docs/assets/tilt_screenshot.png)
+
+```bash
+# Start tilt
+tilt up
+
+# Delete tilt resources
+tilt down
+```
+
+#### Running against localnet
+
+_NOTE: 🚨 The [poktroll](https://github.com/pokt-network/poktroll) localnet includes pocketdex in its tilt environment. 🚨_
+
+If you need to run pocketdex against poktroll localnet, but can't use the poktroll repo's tilt environment for whatever reason, update (_but don't commit_) the `indexer_values_path` in the `Tiltfile`:
+
+```diff
+  load("./tiltfiles/pocketdex.tilt", "pocketdex")
+  pocketdex("./",
+            genesis_file_name="testnet.json",
+            postgres_values_path="./tiltfiles/k8s/postgres/postgres-values.yaml",
+            pgadmin_values_path="./tiltfiles/k8s/pgadmin/pgadmin-values.yaml",
+-           indexer_values_path="./tiltfiles/k8s/indexer/dev-testnet-indexer-values.yaml",
++           indexer_values_path="./tiltfiles/k8s/indexer/dev-localnet-indexer-values.yaml",
+            gql_engine_values_path="./tiltfiles/k8s/gql-engine/dev-gql-engine-values.yaml")
+```
+
+Tilt will automatically apply the change on save.
+
+### 3b. Run via docker-compose
 
 Dotenv files will be automatically created after the `yarn install` thanks to the `postinstall` script.
 After that, feel free to modify them as you wish.
@@ -215,7 +250,7 @@ Or Stop & clean up (delete postgres data):
 yarn run docker:clean:development
 ```
 
-#### 3.1 Debugging, errors running & building
+#### 3b.1 Debugging, errors running & building
 
 If you're hitting errors with the above command, do a nuclear clean of all potential issues:
 
@@ -227,7 +262,7 @@ docker context use default
 
 Now pick up from the `yarn run docker:build` step above.
 
-#### 3.2 Using a pre-built image
+#### 3b.2 Using a pre-built image
 
 If you are unable to build locally, a pre-built image is available on Docker Hub: [bryanchriswhite/pocketdex-subquery-node:latest](https://hub.docker.com/r/bryanchriswhite/pocketdex-subquery-node).
 
@@ -252,7 +287,7 @@ services:
     ...
 ```
 
-#### 3.3 Available Scripts breakdown
+#### 3b.3 Available Scripts breakdown
 
 * `preinstall` - Enforces the use of Yarn as the package manager.
 * `postinstall` - Executes the `env:prepare` script after the installation process.
@@ -280,6 +315,6 @@ services:
 * `docker:stop:<environment>` - Stops all active services for the specified environment without removing them.
 * `docker:clean:<environment>` - Stops and removes all services, volumes, and networks for the specified environment.
 
-#### 3.4 Using k8s
+### 3c. Using k8s
 
 See the instructions in [docs/kubernetes.md](./docs/kubernetes.md) for deploying using Kubernetes.

Tiltfile

@@ -0,0 +1,8 @@
+load("./tiltfiles/pocketdex.tilt", "pocketdex")
+pocketdex("./",
+          # TODO(@bryanchriswhite): load genesis file name from env var or config file.
+          genesis_file_name="testnet.json",
+          postgres_values_path="./tiltfiles/k8s/postgres/postgres-values.yaml",
+          pgadmin_values_path="./tiltfiles/k8s/pgadmin/pgadmin-values.yaml",
+          indexer_values_path="./tiltfiles/k8s/indexer/dev-testnet-indexer-values.yaml",
+          gql_engine_values_path="./tiltfiles/k8s/gql-engine/dev-gql-engine-values.yaml")

docker-compose.development.yml

@@ -44,6 +44,7 @@ services:
       context: .
       dockerfile: ./docker/dev-node.dockerfile
       args:
+        GENESIS_FILENAME: ${GENESIS_FILENAME:-testnet.json}
         NODE_ENV: ${ENV}
         CHAIN_ID: ${CHAIN_ID}
         ENDPOINT: ${ENDPOINT}

docker-compose.test.yml

@@ -4,6 +4,7 @@ services:
       context: .
       dockerfile: ./docker/dev-node.dockerfile
       args:
+        GENESIS_FILENAME: ${GENESIS_FILENAME:-testnet.json}
         NODE_ENV: ${ENV:-test}
         CHAIN_ID: ${CHAIN_ID:-poktroll}
         ENDPOINT: ${ENDPOINT}

docker/dev-node.dockerfile

@@ -4,6 +4,7 @@ ARG CI=false
 ARG NODE_ENV=development
 ARG ENDPOINT
 ARG CHAIN_ID=poktroll
+ARG GENESIS_FILENAME
 
 ENV NODE_ENV=$NODE_ENV
 ENV ENDPOINT=$ENDPOINT
@@ -33,6 +34,7 @@ WORKDIR /app
 COPY package.json yarn.lock .yarnrc.yml /app/
 COPY scripts /app/scripts
 COPY .yarn /app/.yarn
+COPY ./genesis/${GENESIS_FILENAME} /app/genesis.json
 
 # Install dev dependencies
 RUN yarn install

tiltfiles/k8s/gql-engine/.helmignore

@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/

tiltfiles/k8s/gql-engine/Chart.yaml

@@ -0,0 +1,24 @@
+apiVersion: v2
+name: gql-engine
+description: GraphQL API Server for Pocketdex
+
+# A chart can be either an 'application' or a 'library' chart.
+#
+# Application charts are a collection of templates that can be packaged into versioned archives
+# to be deployed.
+#
+# Library charts provide useful utilities or functions for the chart developer. They're included as
+# a dependency of application charts to inject those utilities and functions into the rendering
+# pipeline. Library charts do not define any templates and therefore cannot be deployed.
+type: application
+
+# This is the chart version. This version number should be incremented each time you make changes
+# to the chart and its templates, including the app version.
+# Versions are expected to follow Semantic Versioning (https://semver.org/)
+version: 0.1.0
+
+# This is the version number of the application being deployed. This version number should be
+# incremented each time you make changes to the application. Versions are not expected to
+# follow Semantic Versioning. They should reflect the version the application is using.
+# It is recommended to use it with quotes.
+appVersion: "1.16.0"

tiltfiles/k8s/gql-engine/dev-gql-engine-values.yaml

@@ -0,0 +1,24 @@
+enable_graphiql_playground: true
+indexer_endpoint: "http://indexer-service:3000"
+port: 3000
+env:
+  NODE_ENV: development
+  # Testnet
+  ENDPOINT: "https://testnet-validated-validator-rpc.poktroll.com"
+  CHAIN_ID: "poktroll"
+  # db schema name
+  DB_SCHEMA: "testnet"
+
+  START_BLOCK: 1
+  # Check docs to see what else you can set here:
+  # Ref: https://academy.subquery.network/indexer/run_publish/references.html#subql-cli
+  SUBQUERY_NODE_EXTRA_PARAMS: "--unfinalized-blocks=true"
+  # Check docs to see what else you can set here:
+  # Ref: https://academy.subquery.network/indexer/run_publish/references.html#subql-query
+  SUBQUERY_GRAPHQL_EXTRA_PARAMS: ""
+
+  DB_HOST: postgres-service
+  DB_PORT: 5432
+  DB_USER: postgres
+  DB_PASS: postgres
+  DB_DATABASE: postgres

tiltfiles/k8s/gql-engine/templates/gql-engine-depl.yaml

@@ -0,0 +1,50 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: gql-engine-deployment
+  labels:
+    app.kubernetes.io/name: gql-engine
+    app.kubernetes.io/part-of: pocketdex
+    app.kubernetes.io/managed-by: helm
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: gql-engine
+      app.kubernetes.io/part-of: pocketdex
+      app.kubernetes.io/managed-by: helm
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: gql-engine
+        app.kubernetes.io/part-of: pocketdex
+        app.kubernetes.io/managed-by: helm
+    spec:
+      initContainers:
+        - name: wait-for-indexer
+          image: busybox
+          command: [ 'sh', '-c', 'until nc -z indexer-service 3000; do echo waiting for indexer; sleep 2; done;' ]
+      containers:
+        - name: gql-engine
+          image: subquerynetwork/subql-query:latest
+          command:
+            - /sbin/tini
+            - --
+            - /bin/run
+            - -f
+            - /app
+            - --name={{ .Values.env.DB_SCHEMA }}
+            - --indexer={{ .Values.indexer_endpoint }}
+            - --port={{ .Values.port }}
+            {{- if .Values.enable_graphiql_playground }}
+            - --playground
+            {{- end }}
+          ports:
+            - containerPort: {{ .Values.port }}
+          env:
+            {{- if .Values.env }}
+              {{- range $key, $value := .Values.env }}
+              - name: {{ $key | quote }}
+                value: {{ $value | quote }}
+              {{- end }}
+              {{- end }}

tiltfiles/k8s/gql-engine/templates/gql-engine-svc.yaml

@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: gql-engine-service
+spec:
+  selector:
+    app.kubernetes.io/name: gql-engine
+    app.kubernetes.io/part-of: pocketdex
+    app.kubernetes.io/managed-by: helm
+  ports:
+    - name: http
+      protocol: TCP
+      port: {{ .Values.port }}
+      targetPort: {{ .Values.port }}
+  type: ClusterIP

tiltfiles/k8s/indexer/.helmignore

@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/

tiltfiles/k8s/indexer/Chart.yaml

@@ -0,0 +1,24 @@
+apiVersion: v2
+name: indexer
+description: SubQuery Indexer Node for Pocketdex
+
+# A chart can be either an 'application' or a 'library' chart.
+#
+# Application charts are a collection of templates that can be packaged into versioned archives
+# to be deployed.
+#
+# Library charts provide useful utilities or functions for the chart developer. They're included as
+# a dependency of application charts to inject those utilities and functions into the rendering
+# pipeline. Library charts do not define any templates and therefore cannot be deployed.
+type: application
+
+# This is the chart version. This version number should be incremented each time you make changes
+# to the chart and its templates, including the app version.
+# Versions are expected to follow Semantic Versioning (https://semver.org/)
+version: 0.1.0
+
+# This is the version number of the application being deployed. This version number should be
+# incremented each time you make changes to the application. Versions are not expected to
+# follow Semantic Versioning. They should reflect the version the application is using.
+# It is recommended to use it with quotes.
+appVersion: "1.16.0"

tiltfiles/k8s/indexer/dev-localnet-indexer-values.yaml

@@ -0,0 +1,23 @@
+port: 3000
+hostNetwork: true
+env:
+  NODE_ENV: development
+  # LocalNet
+  ENDPOINT: "http://localhost:26657"
+  CHAIN_ID: "poktroll"
+  # db schema name
+  DB_SCHEMA: "localnet"
+
+  START_BLOCK: 1
+  # Check docs to see what else you can set here:
+  # Ref: https://academy.subquery.network/indexer/run_publish/references.html#subql-cli
+  SUBQUERY_NODE_EXTRA_PARAMS: "--unfinalized-blocks=true"
+  # Check docs to see what else you can set here:
+  # Ref: https://academy.subquery.network/indexer/run_publish/references.html#subql-query
+  SUBQUERY_GRAPHQL_EXTRA_PARAMS: ""
+
+  DB_HOST: postgres-service
+  DB_PORT: 5432
+  DB_USER: postgres
+  DB_PASS: postgres
+  DB_DATABASE: postgres

tiltfiles/k8s/indexer/dev-testnet-indexer-values.yaml

@@ -0,0 +1,23 @@
+port: 3000
+hostNetwork: false
+env:
+  NODE_ENV: development
+  # Testnet
+  ENDPOINT: "https://testnet-validated-validator-rpc.poktroll.com"
+  CHAIN_ID: "poktroll"
+  # db schema name
+  DB_SCHEMA: "testnet"
+
+  START_BLOCK: 1
+  # Check docs to see what else you can set here:
+  # Ref: https://academy.subquery.network/indexer/run_publish/references.html#subql-cli
+  SUBQUERY_NODE_EXTRA_PARAMS: "--unfinalized-blocks=true"
+  # Check docs to see what else you can set here:
+  # Ref: https://academy.subquery.network/indexer/run_publish/references.html#subql-query
+  SUBQUERY_GRAPHQL_EXTRA_PARAMS: ""
+
+  DB_HOST: postgres-service
+  DB_PORT: 5432
+  DB_USER: postgres
+  DB_PASS: postgres
+  DB_DATABASE: postgres