-
Notifications
You must be signed in to change notification settings - Fork 67
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #381 from cmu-sei/v8
updated docs from Austin W.
- Loading branch information
Showing
3 changed files
with
158 additions
and
64 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,107 +1,103 @@ | ||
| # GHOSTS Core API Overview | ||
| # Setting Up the GHOSTS API | ||
|
|
||
| ???+ info "GHOSTS Source Code" | ||
| The [GHOSTS Source Code Repository](https://github.com/cmu-sei/GHOSTS) is hosted on GitHub. | ||
|
|
||
| *Updated on July 24, 2024* | ||
|
|
||
| The GHOSTS API enables the control and orchestration of non-player characters (NPCs) within a deployment. It supports logging, reporting, and managing individual, groups of, or entire deployments of client installs. | ||
|
|
||
| ## Installation | ||
| The GHOSTS API consists of three components: the API itself for configuring and managing characters and machines, a Postgres database for managing all the pieces, and Grafana for seeing GHOSTS activities in one convenient dashboard. Each of these three components runs in its own docker container. | ||
|
|
||
| 1. Install 🐳 [Docker](https://docs.docker.com/install/) :material-open-in-new: | ||
| 2. Install [Docker Compose](https://docs.docker.com/compose/install/) :material-open-in-new: | ||
| 3. Run the following commands - we'll use [this docker-compose.yml file](https://raw.githubusercontent.com/cmu-sei/GHOSTS/master/src/Ghosts.Api/docker-compose.yml) | ||
| Steps to set up the GHOSTS API: | ||
|
|
||
| ```cmd | ||
| mkdir ghosts | ||
| cd ghosts | ||
| curl https://raw.githubusercontent.com/cmu-sei/GHOSTS/master/src/Ghosts.Api/docker-compose.yml -o docker-compose.yml | ||
| docker-compose up -d | ||
| ``` | ||
| 1. Choose where to host the API | ||
| 2. Install Docker and Docker Compose | ||
| 3. Build the GHOSTS containers | ||
| 4. Test the API | ||
|
|
||
| The required containers will be downloaded and configured automatically. | ||
| ## Step 1 — Choose Where to Host the API | ||
|
|
||
| Once the last command completes, if you open [http://localhost:5000/api/home](http://localhost:5000/api/home) in your browser, you should see the initial API page outlining the version of the install, and a few test machine entries. If this page renders, your API is up, running, and available. | ||
| Choose the machine you'll be using to host the GHOSTS API. If you're just playing around with the GHOSTS, your local machine is fine. If you're configuring an exercise or simulation that other people might care about, consider creating a dedidicate host (server, image, or virtual server) for the API, or even using a container service such as AWS ECS. | ||
|
|
||
| You will still need to set up Grafana. Beware that you must often `chown` the host location of the container as listed in the docker-compose file or the container will just continually restart in error due to insufficient permissions. | ||
| ## Step 2 — Installing Docker | ||
|
|
||
| ## Configuring the API | ||
| You'll need to install Docker and Docker Compose on your API host. | ||
|
|
||
| The API generally has good defaults to get you up and running quickly, but there are some considerations in the `appconfig.json` file: | ||
| First, **install 🐳 [Docker](https://docs.docker.com/install/)**. | ||
|
|
||
| ```json | ||
| "ClientSettings": { | ||
| "OfflineAfterMinutes": 30, ... | ||
| "MatchMachinesBy": null, | ||
| ``` | ||
| Next, **install [Docker Compose](https://docs.docker.com/compose/install/)** which will assist with starting up multiple containers at the same time. | ||
|
|
||
| Can be fqdn|host|resolvedhost|null - null tells the API to match incoming requests with machine records by the machine name. For installations where multiple domains are reporting into the same API, you probably want to use FQDN in order to avoid machines being duplicated. | ||
| Before continuing, test that you have the command for Docker Compose available. | ||
|
|
||
| Open your system's CMD (Linux), Terminal (Mac), or PowerShell (Windows). | ||
|
|
||
| ```json | ||
| "QueueSyncDelayInSeconds": 10, | ||
| "NotificationsQueueSyncDelayInSeconds": 10, | ||
| ``` | ||
| $ docker-compose --version | ||
| ``` | ||
|
|
||
| This is how often the synch job runs. Incoming machine requests are not real-time in order to best bundle like records together. | ||
| ## Step 3 — Installing the GHOSTS API | ||
|
|
||
| ## Configuring Grafana | ||
| Once you have confirmed that Docker and Docker Compose are installed, you can build the containers required for the GHOSTS API. | ||
|
|
||
| - Grafana will be running (if containerized) on port 3000, and we can access it via the same URL we use for the API. | ||
| - The default login is admin/admin. | ||
| - The first step is to set up a datasource named "ghosts" to the ghosts Postgres database. | ||
| - Now import your choice of the [grafana json files](https://github.com/cmu-sei/GHOSTS/tree/master/configuration/grafana) in this repository. It creates the default GHOSTS dashboard. | ||
| Create a directory where you want to store the build files and containers. | ||
|
|
||
| ## Webhooks | ||
| ``` | ||
| $ mkdir ghosts-project | ||
| $ cd ghosts-project | ||
| ``` | ||
|
|
||
| The GHOSTS API provides webhook callbacks based on the configuration on the endpoint: `/api/webhooks`. The payload for creating a webhook is in the format: | ||
| Download the docker compose file for GHOSTS. | ||
|
|
||
| ```json | ||
| { | ||
| "status": 0, | ||
| "description": "some description", | ||
| "postbackUrl": "http://localhost/endpoint:port", | ||
| "postbackMethod": 0, (0 == get, 1 == post) | ||
| "postbackFormat": "see below" | ||
| } | ||
| ``` | ||
| $ curl https://raw.githubusercontent.com/cmu-sei/GHOSTS/master/src/Ghosts.Api/docker-compose.yml -o docker-compose.yml | ||
| ``` | ||
|
|
||
| Payloads can be any format — here is a sample: | ||
| Build all of the containers at once using docker-compose. | ||
|
|
||
| ```json | ||
| { | ||
| 'machine':'[machinename]', | ||
| 'created':'[datetime.utcnow]', | ||
| 'type':'[messagetype]', | ||
| 'payload':'[messagepayload]' | ||
| } | ||
| ``` | ||
| $ docker-compose up -d | ||
| ``` | ||
|
|
||
| On send, the payload will be converted into the correct JSON format: | ||
| Check for the running containers. | ||
|
|
||
| ```json | ||
| { | ||
| "machine":"some_guid", | ||
| "created":"some_datetime", | ||
| "type":"some_message", | ||
| "payload":"some_payload" | ||
| } | ||
| ``` | ||
| $ docker ps -a | ||
| ``` | ||
|
|
||
| If everything succeeds you should see the three new containers for the API, Grafana, and Postgres. | ||
|
|
||
| If the postback method is POST, the payload will be sent as the message body. If the postback method is GET, the payload will be sent as part of the querystring value ?message=`payload`. | ||
|  | ||
|
|
||
| The following events are reported via webhooks: | ||
| ## Step 4 — Testing the API | ||
|
|
||
| 1. Timeline delivered (with the timeline that was delivered as payload) to a machine via API (original API posting of timeline only holds timeline in wait - the client still must check-in in order for that timeline to be delivered) | ||
| 2. Machine requested updates ("checked in") from API | ||
| 3. Machine posted results to API | ||
| By default, the API is hosted on port 5000. You should be able to reach the API from [http://localhost:5000](http://localhost:5000). If you open this page in your browser, you should see the initial API page outlining the version of the install, and a few test machine entries. If this page renders, your API is up, running, and available. | ||
|
|
||
|  | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| > Is the API up and running? | ||
| ### Problem: The API home page has an error | ||
|
|
||
|  | ||
|
|
||
| Answer: Make sure the docker container for Postgres is running using Docker Desktop or the command `docker ps -a` | ||
|
|
||
|  | ||
|
|
||
| You can check the logs with the command `docker logs ghosts-postgres` to look for container errors. | ||
|
|
||
| ### Problem: The social graph link has an error | ||
|
|
||
|  | ||
|
|
||
| Answer: You haven't created a social network yet, this is normal. | ||
|
|
||
| ### Problem: Is the API up and running? | ||
|
|
||
| - Go to `/api/home` in the browser, it should return the current API version and the number of machines and groups under management. If it says relationship not found, restart the API application and it should create the database automatically. | ||
| - Run `docker ps --all` and see that all containers are running normally. If one or more is not running, look at the logs for that machine via `docker logs [machine name]`. | ||
|
|
||
| > The ClientId, ClientResults, and other Client* endpoints are failing. | ||
| The Client* endpoints are for the Clients to use only. There are specific header values set by the client in the request that is used to authenticate the request. If you are not using the client, you will not have these headers set, and these endpoints will fail. | ||
| The Client* endpoints are for the Clients to use only. There are specific header values set by the client in the request that is used to authenticate the request. If you are not using the client, you will not have these headers set, and these endpoints will fail. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| # Configuring Grafana | ||
|
|
||
| *Updated on July 25, 2024* | ||
|
|
||
| [Grafana](https://grafana.com/) for GHOSTS allows the simulation administrator to visualize all activities carried out by the NPCs across the simulation in dashboard with pretty colors and charts. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Your GHOSTS API should be installed and running (see [Setting Up the GHOSTS API](installing-the-api.md)) | ||
|
|
||
| ## Step 1 — Container Set Up | ||
|
|
||
| The Grafana docker container will be installed during the process of [Setting Up the GHOSTS API](installing-the-api.md). Depending on your system and account, the Grafana container may or may not have the permissions it needs to run. | ||
|
|
||
| You can check its status with the docker command | ||
|
|
||
| ``` | ||
| $ docker ps -a | ||
| ``` | ||
|
|
||
| If the container is continuously restarting, Grafana does not have the permissions it needs. | ||
|
|
||
|  | ||
|
|
||
| You can also check the docker logs: | ||
|
|
||
| ``` | ||
| $ docker logs ghosts-grafana | ||
| ``` | ||
|
|
||
|  | ||
|
|
||
| If you don't see this issue, you can continue to Step 2. If you do see this issue you will need to grant permissions on the `_g` directory of the ghosts-api folder (which stores the Grafana data for the GHOSTS API). | ||
|
|
||
| ``` | ||
| $ cd ghosts-api | ||
| $ chmod 777 _g | ||
| ``` | ||
|
|
||
| Ensure the container is running with `docker ps`. | ||
|
|
||
| ## Step 2 — Configuring Grafana | ||
|
|
||
| Once the container is running you can access its front end by default at [localhost:3000](http://localhost:3000) | ||
|
|
||
|  | ||
|
|
||
| The default login is: | ||
|
|
||
| | Username | Password | | ||
| | -------- | -------- | | ||
| | admin | admin | | ||
|
|
||
| Continue through the setup prompts. | ||
|
|
||
| ### Step 3 — Setting the Datasource | ||
|
|
||
| Now you need to tell Grafana where it will be getting its data. | ||
|
|
||
| 1. Open the "Connections" from the left menu. | ||
| 2. Click on "Add new data source" in the top right corner | ||
| 3. Search for "Postgres" and choose the PostgreSQL option | ||
| 4. Name the datasource "ghosts" and leave it as the default | ||
| 5. Under the "Connection" section of the config, set | ||
| - host url to "ghosts-postgres:5432" | ||
| - database name to "ghosts" | ||
| 6. Under the "Authentication" section of the config, set | ||
| - username to "ghosts" | ||
| - password to "scotty@1" | ||
| - TLS/SSL Mode to "disable" | ||
| 7. Leave everything else at its default and click the "Save and test" button at the bottom of the page | ||
|
|
||
|
|
||
| ### Step 4 — Choosing a Dashboard | ||
|
|
||
| Grafana dashboards are very flexible and can be configured to show any statistics that are important to your simulation or experiment. | ||
|
|
||
| GHOSTS comes with some premade dashboards to get you started. You can download those here: https://github.com/cmu-sei/GHOSTS/tree/master/configuration/grafana/dashboards | ||
|
|
||
| - GHOSTS-5-default Grafana dashboard — shows status across all machines | ||
| - GHOSTS-5-group-default Grafana dashboard — shows status with machines grouped by enclave | ||
|
|
||
| #### Loading an existing dashboard | ||
|
|
||
| Navigate to "Dashboards" in the left menu. There will be a blue "New" button in the top right corner. | ||
|
|
||
|  | ||
|
|
||
| Click "New". Then, "import". | ||
|
|
||
| You can either upload one of the dashboard json files from the GHOSTS repository or simply copy and paste the json into the "import via dashboard json model" panel. | ||
|
|
||
| Choose the ghosts datasource you added earlier from the drop down menu and then click "import". | ||
|
|
||
|  | ||
|
|
||
| You are now set up with Grafana! |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters