Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Temporal Nexus: Deploy to Production - Temporal Cloud & Nexus #3078

Open
wants to merge 25 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion docs/production-deployment/cloud/audit-logging.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ tags:
- troubleshooting
---

Audit Logging is a feature of [Temporal Cloud](/cloud/overview) that provides forensic access information at the account level, the user level, and the [Namespace](/namespaces) level.
Audit Logging is a feature of [Temporal Cloud](/cloud/overview) that provides forensic access information at the account level, the user level, [Namespace](/namespaces) level, and the Nexus Endpoint level.
jsundai marked this conversation as resolved.
Show resolved Hide resolved

Audit Logging answers "who, when, and what" questions about Temporal Cloud resources.
These answers can help you evaluate the security of your organization, and they can provide information that you need to satisfy audit and compliance requirements.
Expand Down Expand Up @@ -63,6 +63,10 @@ The following list specifies both the supported events and the Temporal APIs tha
- Request increase in Retention Period: `UpdateNamespace`
- Multi-region Namespace
- Failover Namespace: `FailoverNamespace`
- Nexus Endpoint
- Create Nexus Endpoint: `CreateNexusEndpoint`
- Update Nexus Endpoint: `UpdateNexusEndpoint`
- Delete Nexus Endpoint: `DeleteNexusEndpoint`

### API Key Operation events

Expand Down
5 changes: 4 additions & 1 deletion docs/production-deployment/cloud/metrics/reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ Use the following labels to filter metrics:
| `le` | Less than or equal to (`le`) is used in histograms to categorize observations into buckets based on their value being less than or equal to a predefined upper limit. |
| `operation` | This includes operations such as:<ul><li>SignalWorkflowExecution</li><li>StartBatchOperation</li><li>StartWorkflowExecution</li><li>TaskQueueMgr</li><li>TerminateWorkflowExecution</li><li>UpdateNamespace</li><li>UpdateSchedule</li></ul> See: [Metric Operations](#metrics-operations) |
| `resource_exhausted_cause` | Cause for resource exhaustion. |
| `task_type` | Activity or Workflow. |
| `task_type` | Activity, Workflow, or Nexus. |
| `temporal_account` | Temporal Account. |
| `temporal_namespace` | Temporal Namespace. |
| `temporal_service_type` | Frontend or Matching or History or Worker. |
Expand Down Expand Up @@ -265,6 +265,7 @@ Temporal Cloud includes the following operations labels:
- OperatorDeleteNamespace
- PatchSchedule
- PollActivityTaskQueue
- PollNexusTaskQueue
- PollWorkflowExecutionHistory
- PollWorkflowExecutionUpdate
- PollWorkflowTaskQueue
Expand All @@ -280,6 +281,8 @@ Temporal Cloud includes the following operations labels:
- RespondActivityTaskCompletedById
- RespondActivityTaskFailed
- RespondActivityTaskFailedById
- RespondNexusTaskCompleted
- RespondNexusTaskFailed
- RespondQueryTaskCompleted
- RespondWorkflowTaskCompleted
- RespondWorkflowTaskFailed
Expand Down
69 changes: 69 additions & 0 deletions docs/production-deployment/cloud/nexus/getting-started.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
---
id: getting-started
slug: /cloud/nexus/getting-started
title: Getting Started with Temporal Nexus
description: Get started with Temporal Nexus, including setting up Nexus Endpoints and integrating Nexus into your Temporal Workflows.
sidebar_label: Getting Started
tags:
- temporal cloud
- nexus setup
keywords:
- Temporal Nexus onboarding
- setting up Nexus Endpoints
- temporal cloud
- nexus setup

---

Temporal Nexus Public Preview is available within your Temporal Cloud account, with support in the [Temporal Go SDK](/develop/go).

Calls across existing Namespaces can be enabled by creating a Nexus Endpoint in the Nexus API Registry, creating a Nexus Service in a Worker in the handler Namespace, and then using the Nexus Service from a caller Workflow in a different Namespace.

Monolithic Namespaces can be decomposed into multiple Namespaces, by hiding service implementations behind a Nexus Endpoint in the monolithic Namespace, pointing all consumers at the new Nexus Endpoint, and then changing the Endpoint’s target Namespace to a different Namespace.
Multiple Nexus Endpoints can target a single monolithic Namespace.

## Cross-Namespace

Follow these steps to enable calls across existing Namespaces:

1. Add Nexus Services to the same Workers as the Temporal primitives being abstracted.
2. Add a Nexus Endpoint that:
1. Targets the handler Namespace.
2. Allows the caller Namespace.
3. Make Nexus calls from a caller Workflow in a different Namespace.
1. Use `workflow.NewNexusClient(endpointName, serviceName)`.
2. Execute a Nexus Operation with `nexusClient.ExecutionOperation(...)`.

## Decompose a Monolithic Namespace

Multiple Nexus Endpoints can target a single monolithic Namespace, and then each Endpoint can be updated, one at a time, to target separate Namespaces, for an incremental migration.

Once Nexus Endpoints are in place, targeting a new Namespace can be done with config changes and zero downtime.
New Nexus requests will be routed to the new target Namespace, and existing Nexus requests will be completed in the previous Namespace.

**Follow these steps to decompose a large monolithic Namespace:**

1. Hide service implementations behind a Nexus Endpoint.
1. Add Nexus Services to the same Workers as the Temporal primitives being abstracted.
2. Add Nexus Endpoints to the Nexus API Registry, with monolithic Namespace as the target.
2. Service consumers use the Nexus Endpoint instead of the underlying implementation.
1. This can be done incrementally, until there are no direct caller dependencies on the underlying service implementations (the underlying Temporal primitives).
3. Move service implementations to a different Namespace.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
3. Move service implementations to a different Namespace.
3. Decouple your Service implementations to new Namespaces, where they are grouped in a more functional manner instead of a monolith.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going to resolve these. As discussed, a lot of this will end up needed rewriting in another iteration to fit our style (i.e - reading level, clarity, etc).

1. Create a new Namespace.
2. Add a Worker deployment with the Nexus Service.
3. Update the Nexus Endpoint target to the new Namespace.
4. Configure the Endpoint allowlist to allow calls from the original monolithic Namespace.
5. New Nexus Operations will be routed from callers in the monolithic Namespace to the new Namespace.
4. Quiesce Nexus Operations on the previous target Namespace.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
4. Quiesce Nexus Operations on the previous target Namespace.
4. Prepare Nexus Operations on the previous target Namespace during the migration.

1. Leave the previous Worker deployment running until all existing Nexus Operations in the previous Namespace have completed (and their underlying Workflows, if any).
2. Previous Workers in the monolithic Namespace can have the service implementation removed, since it is now being served from the new Endpoint target Namespace.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
2. Previous Workers in the monolithic Namespace can have the service implementation removed, since it is now being served from the new Endpoint target Namespace.
2. Remove Workers from the monolithic Namespace, since those Workflows and Activities are now served from the new Endpoint targeted Namespace.


Please note the following points:

- For long-lived entity Workflows, new Nexus requests will be routed to the new target Namespace.
Long-lived entity-Workflow will still be running on the previous Namespace.
- When using Signal-with-Start behind a Nexus Operation, a new entity-Workflow will start in the new target Namespace.
- A cancel Nexus Operation request from a caller Workflow will be routed through the Nexus Endpoint to the new target Namespace.
The new Namespace may not have the underlying Workflow.

In these scenarios, Temporal can provide guidance on different migration strategies.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How should someone seek this guidance? Support? Community?

88 changes: 88 additions & 0 deletions docs/production-deployment/cloud/nexus/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
---
id: index
slug: /cloud/nexus
title: Temporal Nexus
description: Discover Temporal Nexus, a powerful feature for connecting durable executions across team, namespace, region, and cloud boundaries. Nexus also enables each team to have their own namespace for improved security, troubleshooting, and blast radius isolation.
sidebar_label: Temporal Nexus
tags:
- temporal cloud
- temporal nexus
keywords:
- Temporal Nexus
- cross-namespace workflows
- modular workflow design
- workflow orchestration
- Temporal Cloud security
- Temporal Cloud services
- distributed workflows
- workflow API management
---

:::tip SUPPORT, STABILITY, and DEPENDENCY INFO

Temporal Nexus is in [Public Preview](https://docs.temporal.io/evaluate/development-production-features/release-stages#public-preview) for Temporal Cloud.

:::

This Temporal Nexus guide covers the following topics:

- [Introduction to Nexus](#nexus-introduction)
- [Using Nexus](/cloud/nexus/using-nexus)
- [Pricing](/cloud/nexus/pricing)
- [Getting Started](/cloud/nexus/getting-started)
- [Operations](/cloud/nexus/operations)

## Introduction to Nexus {#nexus-introduction}

[Temporal Nexus](/nexus) is designed to connect durable executions across team, Namespace, region, and cloud boundaries.
It promotes a more modular architecture for sharing a subset of your team’s capabilities through well-defined service API contracts for other teams to use, that abstract underlying Temporal primitives, like Workflows, or execute arbitrary code.
jsundai marked this conversation as resolved.
Show resolved Hide resolved

**Why use Temporal Nexus?**

Temporal Nexus provides:

- A fully integrated Temporal SDK experience:
- Run Nexus Services in a Worker, often the same Worker as underlying primitives.
- Use Nexus Services from a Workflow.
- Use the same queue-based Worker architecture, for Worker fleet load balancing.
- Built-in Nexus machinery:
- Connects Nexus callers and handlers through Nexus Endpoints.
- Automatic retries & circuit breaking.
- Multi-region routing across Namespaces:
- Secure mTLS connectivity.
- Endpoint access control.
- Observability:
- Integrated into Workflow Event History.
- Bi-direction linking for end-to-end debugging.
- Metrics, traces, and logs.
- Improved blast radius isolation:
- Each team can have it’s own Namespace.
- Per-Namespace security & rate limits.
- Easier troubleshooting with less noise.

**Should you be using Temporal Nexus?**
jsundai marked this conversation as resolved.
Show resolved Hide resolved

- **Modular design** \- Temporal Nexus enables a modular design that helps structure an application for fast growth and supports teams working on their own parts independently.
If you want to decouple callers from handlers to enable moving a service to a different Namespace or Task Queue later, Nexus can encapsulate those implementation details to avoid breaking changes and cross-team coordination.

- **Teams working together** \- When there is no clear API contract between teams working in a large monolithic Namespace, it takes work to understand what Workflows can be called from other Workflows – or if a team has signed up to maintain that interface. This places a burden on both teams, and makes changes more difficult.
Nexus Services define a clear contract that is suitable for sharing across teams, and cleanly separates the public interface from the internal implementation, providing a more curated experience for service consumers while protecting service authors.

- **Keeping things separate and secure** \- When all teams operate in the same monolithic Namespace, everything is available to everyone, and mis-behaving workers can trigger rate limits that affect all teams operating in that monolithic Namespace.
Temporal Nexus enables service modules to run in arbitrary Namespaces, since all Nexus requests are routed through the Nexus Endpoint to the underlying target Namespace and Task Queue. Running a Nexus Service in its own Namespace allows tighter Namespace permissions and limits the blast radius of misbehaving workers.

- **Managing interactions across boundaries** \- When no clean boundary exists it is hard to manage concerns like security and rate-limiting.
A Nexus Endpoint is an API Proxy or Gateway, that can act as a central point of control for these concerns.
Temporal Cloud supports built-in rate-limiting and an Endpoint allowlist \- with a roadmap to support finer-grained rate-limiting and access-control policies suitable for external callers.

- **Adapting as the application grows** \- Temporal Nexus enables a modular application design that can evolve as you grow.
From refactoring service implementations, changing an endpoint to point at a different target Namespace, or splitting a Worker out to a different Task Queue, Temporal Nexus enables you to hide these implementation details and provide a curated Nexus Service contract for other teams to use, so you can grow your application faster with less breaking changes, downtime, and cross-team coordination.

**Advantages of Temporal Nexus:**

- Connect durable execution within and across Namespaces.
- Service contracts suitable for sharing across teams.
- Abstract underlying Temporal primitives, like Workflows – or execute arbitrary code.
- Modular design that streamlines multi-team development.
- Improved security and blast-radius isolation with isolated Namespaces.
- Same queue-based Worker architecture, no bespoke service deployments needed.
Loading