Merge pull request #393 from cmu-sei/v8

8.2

README.md

@@ -46,6 +46,10 @@ The API server provides a way for clients to interact with the GHOSTS system and
 - Get/manage information from clients regarding their previous or current activities, etc.
 - Orchestrate new activities for particular clients to perform
 
+### [Ghosts UI](src/ghosts.ui/README.md)
+
+The web server provides a way for administrators to interact with the GHOSTS system and its clients.
+
 ### [Ghosts Lite](src/Ghosts.Client.Lite/)
 
 A resource light version of the Windows GHOSTS client that can be run on minimal hardware.

docs/core/lite.md

@@ -0,0 +1,36 @@
+# GHOSTS LITE
+
+Generate web requests and file creation activity with minimal resources (good for when participants lack host access).
+
+## Overview
+
+### What is GHOSTS LITE?
+GHOSTS LITE is a streamlined version of the GHOSTS framework, designed to simulate realistic network activity without the overhead of running actual applications. It programmatically generates web browsing traffic and file creation, emulating real-user actions without launching resource-intensive applications.
+
+### Purpose
+GHOSTS LITE was created to provide an efficient tool for simulating network activity. If you don't give participants access to the host, there is perhaps no need to launch applications like web browsers or office suites — so, GHOSTS LITE reduces resource consumption, including CPU, storage, and memory, simplifying the simulation process and requiring less resources to run.
+
+### Objectives
+- Efficiency: Minimize the resources required to simulate user activity on a network.
+- Simplicity: Simplify installation and configuration compared to the full GHOSTS framework.
+- Realism: Preserve the ability to generate realistic network activity.
+
+## Installation
+
+### Prerequisites
+- Ensure you have the .NET framework installed on your system.
+
+### Download and install ghosts-lite
+
+1. **Download the latest release**
+2. **Extract the downloaded archive** to a directory of your choice.
+3. **Open terminal** and navigate to the directory where you extracted ghosts-lite
+4. **Run the following command** to start the program:
+```bash
+dotnet Ghosts.Client.Lite.dll
+```
+
+## Troubleshooting
+
+- Ensure the .NET framework is installed with version 8.0 or later.
+- If you're experiencing any problems please submit an issue or start a discussion.

docs/core/ui.md

@@ -0,0 +1,100 @@
+# GHOSTS UI
+
+A UI for the GHOSTS API, providing interfaces for:
+
+- Viewing and managing machines and machine groups controlled by the GHOSTS API
+- Creating and managing timeLines that we can then send to machines or machine groups
+- Viewer for Npcs
+
+For any machine or machine group, we can use the UI to deploy new timelines, or view activities. For machine groups, we can create or manage the machines in the group. For timelines, we can view the activities in the timeline, or deploy new activities. It also provides a management interface for timelines as well.
+
+## Install
+
+The easiest way to run the GHOSTS UI is to use Docker. The following is an example we would add to the ghosts `docker-compose.yml` file that will run the GHOSTS UI and the GHOSTS API together:
+
+```yaml
+ghostsui:
+    image: dustinupdyke/ghosts-ui
+    container_name: ghosts-ui
+    ports:
+      - '8080:8080'
+    networks:
+      - ghosts-network
+    environment:
+      GHOSTS_API_URL: http://ghosts-api:5000
+```
+
+## Functions
+
+This first screen is part of the user interface for managing machines within a network simulation or exercise environment, likely related to the GHOSTS framework or a similar system. 
+
+### Machines
+
+![Machines](../../assets/screens/ui/machines.png)
+
+Here's an explanation of the various elements seen here:
+
+New Machine Button: This button allows you to create or add a new machine to the simulation environment.
+
+Search raw JSON: This input field allows you to perform a search within the machine data using raw JSON queries. It provides a more advanced way to filter or find specific data points.
+
+Machine List:
+
+- Id: This column displays the unique identifier for each machine.
+- Name: This column displays the name of the machine. In this case, it is labeled as string, which might be a placeholder or default value.
+- Status: This column shows the current status of the machine. The status Down indicates that the machine is not currently active or reachable.
+
+There are also functions to delete the machine, run a particular timeline on that machine, view its activity, or view its JSON representation.
+
+### Machine Groups
+
+![Machine Groups](../../assets/screens/ui/machine-groups.png)
+
+New Machine Group Button: This button allows you to create or add a new machine group to the simulation environment. Machine groups can be used to collectively manage multiple machines with similar roles or functions.
+
+Machine Group List:
+
+- Id: This column displays the unique identifier for each machine group.
+
+- Name: This column displays the name of the machine group. In this case, it is labeled as string-*, which might be a placeholder or default value, suggesting it is a pattern-based naming convention.
+
+- View Machines (0): This button opens a view to see all the machines currently in this group. The number in parentheses indicates that there are currently no machines in this group.
+
+- Add Machines (1): This button allows you to add machines to the group. The number in parentheses suggests that there is one machine that could be added or is pending addition to this group.
+
+More (Three-dot Menu): This button likely provides additional options such as deleting a grouping, running a timeline on a group of machines, viewing their activity or JSON representation.
+
+### Timelines
+
+![Timelines](../../assets/screens/ui/timelines.png)
+
+This screen is part of the user interface for managing timelines within a network simulation or exercise environment.
+
+- Name: This column displays the name of each timeline. In the example shown, the timeline is named baseline win workstation timeline, which suggests it is a baseline or standard sequence of events for a Windows workstation.
+
+- Timeline Handlers: This column shows the handler or the specific application/component associated with executing the timeline. In this case, the handler is BrowserFirefox, indicating that this timeline is likely related to actions performed in the Firefox browser.
+
+Actions (Three-dot Menu): This button provides access to deleting or editing a timeline, or viewing its JSON representation.
+
+
+### NPCs
+
+![NPCs](../../assets/screens/ui/npcs.png)
+
+This screen is part of the user interface for managing NPCs (Non-Player Characters) within a network simulation or exercise environment. 
+
+Generate Random NPCs Button: This button allows you to generate NPCs automatically with random attributes. This can be useful for populating the environment quickly with varied NPCs.
+
+NPC List:
+
+- Id: This column displays the unique identifier for each NPC. The IDs are represented as UUIDs, which ensure that each NPC has a distinct identifier within the system.
+
+- First Name: This column shows the first name of the NPC. In this case, the NPCs listed have names like Richard, Collie, Ferdinande, Moyra, and Emylee.
+
+- Machine: This column likely shows the machine or environment to which the NPC is assigned. This association would define where or how the NPC interacts within the simulation.
+
+- Campaign: This column indicates the campaign or scenario in which the NPC is participating. All NPCs listed are part of Exercise Season 2024, suggesting they are involved in the same simulation or training exercise.
+
+- Enclave: This column shows the organization or unit to which the NPC belongs. In this case, all NPCs are part of Brigade Abbott Inc and Sons.
+
+- Team: This column indicates the specific team within the enclave to which the NPC belongs. All NPCs listed are part of the Engineering team, which might define their role or tasks within the simulation.

docs/new.md

@@ -3,7 +3,28 @@
 ???+ info "Welcome to GHOSTS"
     Welcome to what's new in the GHOSTS framework. Use this page to review the latest changes.
 
-## Enhancements:
+## What's new in 8.2
+
+- [GHOSTS now has a UI](core/ui.md)! Manage machines and machine groups, deploy new timelines, and view activities. 😍
+- [GHOSTS Shadows](shadows/index.md) provides integrations with an LLM for the purposes of GHOSTS agents. It provides multiple models for activities, chat, content, social, and whatever else you may need. 👻
+- [GHOSTS LITE](core/lite.md) is a stripped-down version of GHOSTS that is designed to be more lightweight and require less resources to run certain activities in training, exercises, and simulations. 👀
+- Fixes #385 default guid 00000000-0000-0000-0000-000000000000 animator NPC bug. 🐛
+- Fixes #384 client path bug. 🐛
+- Fixes animation cancellation token bugs (chat). 🐛
+- Updated documentation for Animations (Thank you to the SEI's TMR team). 🤙
+
+## What's new in 8.1
+
+- GHOSTS LITE BETA - when you want to conserve resources needed and participants won't access that machine directly.
+- API clean up, particularly machine updates and groups.
+- Made sample json objects easier to understand and submit the swagger generated defaults
+- API robustness
+- MachineGroups cleanup
+- Timeline delivery by machine and by group
+
+## What's new in 8.0
+
+### Enhancements:
 
 - ANIMATOR and SPECTRE functionality merged into the GHOSTS API proper. Those projects are now archived. 📁
 - We have moved off mongo — all that data is now stored in Postgres ❤️.
@@ -16,7 +37,7 @@
 - API endpoints have been re-organized in a more logical fashion. 🗂️
 - Added a favicon. 💅
 
-## Bug Fixes:
+### Bug Fixes:
 
 - Updates Grafana docker compose to not use root. 🚫👤
 - Cleans up containers and ensures all are amd64 (not arm!). 💻

docs/shadows/index.md

@@ -0,0 +1,67 @@
+# GHOSTS SHADOWS
+
+Shadows provides access to a locally-hosted LLM for various GHOSTS agent purposes.
+
+It provides multiple interfaces:
+
+- A REST API for the purposes of ghosts agents
+- A UI web interface for testing and demo purposes
+
+The default API endpoints are:
+
+- **Activities**: Answers the question of "what should an NPC do next?"
+- **Chat**: Provides content for an NPC to chat with a player or other NPC
+- **Content**: Provides a richer array of content for the docuements created by NPCs in a range
+- **Social**: Provides content for an NPC to post on a social media system such as GHOSTS Socializer
+
+We suspect there will be many more in the future.
+
+## Running via Docker
+
+Typically, the easiest way to manage deployment is to pass the env var "GHOSTS_OLLAMA_URL" into the container.
+
+    EXPORT GHOSTS_OLLAMA_URL=http://localhost:11434
+
+(where 5900 is the api port and 7860 is the ui port)
+`docker run -d --name shadows -p 5900:5900 -p 7860:7860 dustinupdyke/ghosts-shadows`
+
+## Running on bare metal
+
+Standing up the Shadows stack is currently (you'll need three terminal windows. We'll clean this up eventually):
+
+Get Ollama up and running:
+
+```bash
+cd content-models/content
+ollama create content
+ollama run content
+```
+Now run the two servers (in separate terminals):
+
+```bash
+python api.py
+python ui.py
+```
+
+Eventually ollama will serve multiple models all the time.
+
+So, this loop for ["content", "social", "chat", "activities"] will be:
+
+```bash
+cd content-models/activities
+ollama create activities
+
+cd content-models/chat
+ollama create chat
+
+cd content-models/social
+ollama create social
+
+ollama serve
+```
+
+If you want this to be available beyond localhost, you'd need to run:
+
+    OLLAMA_HOST=0.0.0.0:11434 ollama serve
+
+Now ollama is running all four models concurrently. The API server provides access into each.

mkdocs.yml

@@ -67,6 +67,7 @@ nav:
   - Quick Start: quickstart.md
   - Client: 
     - core/client.md
+    - core/lite.md
     - Client Handlers:
       - Basic Configuration: core/handlers.md
       - Clicks: core/handlers/clicks.md
@@ -95,6 +96,8 @@ nav:
       - Capabilities:
         - Managing Timelines: core/api/timelines.md
       - Grafana: core/grafana.md
+  - UI: 
+    - core/ui.md
   - Animations:
     - animator/index.md
     - Animation Jobs: animator/jobs.md
@@ -103,6 +106,8 @@ nav:
     - content/index.md
     - Pandora: content/pandora.md
     - Pandora Social: content/social.md
+  - Shadows LLM API: 
+    - shadows/index.md
   - Advanced:
     - advanced/index.md
     - Cyclone: advanced/cyclone.md

src/Ghosts.Api/docker-compose.yml

@@ -6,6 +6,15 @@ networks:
   ghosts-network:
     driver: bridge
 services:
+  ghostsui:
+    image: dustinupdyke/ghosts-ui
+    container_name: ghosts-ui
+    ports:
+      - '8080:8080'
+    networks:
+      - ghosts-network
+    environment:
+      GHOSTS_API_URL: http://ghosts-api:5000
   postgres:
     image: postgres
     container_name: ghosts-postgres

src/Ghosts.Client.Lite/README.md

@@ -1,23 +1,19 @@
 # GHOSTS LITE
 
-## Background
+Generate web requests and file creation activity with minimal resources (good for when participants lack host access).
 
-### What is this?
-GHOSTS LITE is a streamlined version of the GHOSTS framework designed to simulate realistic 
-network without the expense of running actual applications. It generates web browsing traffic 
-and file creation programatically and is configured to be similar to real-user actions. 
+## Overview
 
-### Why did we build this? 
-We built ghosts-lite to provide a more efficient tool for simulating network activity. By 
-avoiding the need to launch applications like web browsers or office suites, ghosts-lite is 
-able to reduce the resource consumption and complexity of the simulation, resulting in saving 
-CPU, storage space, and other memory resources.
+### What is GHOSTS LITE?
+GHOSTS LITE is a streamlined version of the GHOSTS framework, designed to simulate realistic network activity without the overhead of running actual applications. It programmatically generates web browsing traffic and file creation, emulating real-user actions without launching resource-intensive applications.
 
-### What were the goals?
+### Purpose
+GHOSTS LITE was created to provide an efficient tool for simulating network activity. If you don't give participants access to the host, there is perhaps no need to launch applications like web browsers or office suites — so, GHOSTS LITE reduces resource consumption, including CPU, storage, and memory, simplifying the simulation process and requiring less resources to run.
 
-- **Efficiency:** Reduce the resources required to simualte user activity on a network.
-- **Simplicity:** Simplify installation and configuration compared to the full GHOSTS framework
-- **Realism:** Maintain the ability to generate realistic network activity.
+### Objectives
+- Efficiency: Minimize the resources required to simulate user activity on a network.
+- Simplicity: Simplify installation and configuration compared to the full GHOSTS framework.
+- Realism: Preserve the ability to generate realistic network activity.
 
 ## Installation
 
@@ -35,6 +31,16 @@ CPU, storage space, and other memory resources.
 - Ensure the .NET framework is installed with version 8.0 or later.
 - If you're experiencing any problems please submit an issue or start a discussion.
 
+## Contributing
+
+We welcome contributions! Please follow these steps to contribute to GHOSTS LITE:
+
+- Fork the repository.
+- Create a new branch (git checkout -b feature-branch).
+- Make your changes and commit them (git commit -m 'Add feature').
+- Push to the branch (git push origin feature-branch).
+- Open a Pull Request.
+
 ## License
 
 [DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. 

src/Ghosts.Client/Properties/AssemblyInfo.cs

@@ -34,4 +34,4 @@
 // [assembly: AssemblyVersion("1.0.*")]
 [assembly: AssemblyVersion("8.0.0.0")]
 [assembly: AssemblyInformationalVersion("8.0.0.0")]
-[assembly: AssemblyFileVersion("8.1.1.0")]
+[assembly: AssemblyFileVersion("8.2.0.0")]

src/ghosts.shadows/.dockerignore

@@ -0,0 +1,14 @@
+.git
+.github
+.cache
+
+*__pycache__*
+__pycache__/*
+
+.dockerignore
+Dockerfile
+
+*venv*
+
+data/*
+docs/*