Skip to content

Commit 16d6bae

Browse files
dmetzgaryiminc
dmetzgar
authored and
yiminc
committed
Add package overview documentation (#586)
* Remove TODOs from documentation * Separate paragraph for RecordHeatbeat details * variadic is redundant with functions * Transferring docs to package comments for GoDoc * Added more to cadence overview
1 parent 8b32465 commit 16d6bae

File tree

5 files changed

+910
-31
lines changed

5 files changed

+910
-31
lines changed

‎activity/activity.go

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,11 @@
1818
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
1919
// THE SOFTWARE.
2020

21-
// Package activity contains functions and types used to implement Cadence activities.
2221
package activity
2322

2423
import (
2524
"context"
25+
2626
"github.com/uber-go/tally"
2727
"go.uber.org/cadence/internal"
2828
"go.uber.org/zap"
@@ -64,7 +64,7 @@ func Register(activityFunc interface{}) {
6464
// func sampleActivity() (result string, err error)
6565
// func sampleActivity(arg1 bool) (result int, err error)
6666
// func sampleActivity(arg1 bool) (err error)
67-
// Serialization of all primitive types, structures is supported ... except channels, functions, variadic, unsafe pointer.
67+
// Serialization of all primitive types, structures is supported ... except channels, functions, unsafe pointer.
6868
// If function implementation returns activity.ErrResultPending then activity is not completed from the
6969
// calling workflow point of view. See documentation of activity.ErrResultPending for more info.
7070
// This method calls panic if activityFunc doesn't comply with the expected format.
@@ -90,9 +90,7 @@ func GetMetricsScope(ctx context.Context) tally.Scope {
9090
// RecordHeartbeat sends heartbeat for the currently executing activity
9191
// If the activity is either cancelled (or) workflow/activity doesn't exist then we would cancel
9292
// the context with error context.Canceled.
93-
// TODO: we don't have a way to distinguish between the two cases when context is cancelled because
94-
// context doesn't support overriding value of ctx.Error.
95-
// TODO: Implement automatic heartbeating with cancellation through ctx.
93+
//
9694
// details - the details that you provided here can be seen in the workflow when it receives TimeoutError, you
9795
// can check error TimeOutType()/Details().
9896
func RecordHeartbeat(ctx context.Context, details ...interface{}) {

‎activity/doc.go

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
// Copyright (c) 2017 Uber Technologies, Inc.
2+
//
3+
// Permission is hereby granted, free of charge, to any person obtaining a copy
4+
// of this software and associated documentation files (the "Software"), to deal
5+
// in the Software without restriction, including without limitation the rights
6+
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
7+
// copies of the Software, and to permit persons to whom the Software is
8+
// furnished to do so, subject to the following conditions:
9+
//
10+
// The above copyright notice and this permission notice shall be included in
11+
// all copies or substantial portions of the Software.
12+
//
13+
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
14+
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
15+
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
16+
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
17+
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
18+
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
19+
// THE SOFTWARE.
20+
21+
/*
22+
Package activity contains functions and types used to implement Cadence activities.
23+
24+
The activity is an implementation of a task to be performed as part of a larger workflow. There is no limitation of
25+
what an activity can do. In the context of a workflow, it is in the activities where all operations that affect the
26+
desired results must be implemented.
27+
28+
Overview
29+
30+
The client library for Cadence does all the heavy lifting of handling the async communication between the Cadence
31+
managed service and the worker running the activity. As such, the implementation of the activity can, for the most
32+
part, focus on the business logic. The sample code below shows the implementation of a simple activity that accepts a
33+
string parameter, appends a word to it and then returns the result.
34+
35+
import (
36+
"context"
37+
38+
"go.uber.org/cadence/activity"
39+
"go.uber.org/zap"
40+
)
41+
42+
func init() {
43+
activity.Register(SimpleActivity)
44+
}
45+
46+
func SimpleActivity(ctx context.Context, value string) (string, error) {
47+
activity.GetLogger(ctx).Info("SimpleActivity called.", zap.String("Value", value))
48+
return "Processed: ” + value, nil
49+
}
50+
51+
The following sections explore the elements of the above code.
52+
53+
Declaration
54+
55+
In the Cadence programing model, an activity is implemented with a function. The function declaration specifies the
56+
parameters the activity accepts as well as any values it might return. An activity function can take zero or many
57+
activity specific parameters and can return one or two values. It must always at least return an error value. The
58+
activity function can accept as parameters and return as results any serializable type.
59+
60+
func SimpleActivity(ctx context.Context, value string) (string, error)
61+
62+
The first parameter to the function is context.Context. This is an optional parameter and can be omitted. This
63+
parameter is the standard Go context.
64+
65+
The second string parameter is a custom activity-specific parameter that can be used to pass in data into the activity
66+
on start. An activity can have one or more such parameters. All parameters to an activity function must be
67+
serializable, which essentially means that params can’t be channels, functions, variadic, or unsafe pointer.
68+
69+
The activity declares two return values: (string, error). The string return value is used to return the result of the
70+
activity. The error return value is used to indicate an error was encountered during execution.
71+
72+
Implementation
73+
74+
There is nothing special about activity code. You can write activity implementation code the same way you would any
75+
other Go service code. You can use the usual loggers and metrics collectors. You can use the standard Go concurrency
76+
constructs.
77+
78+
Failing the activity
79+
80+
To mark an activity as failed, all that needs to happen is for the activity function to return an error via the error
81+
return value.
82+
83+
Activity Heartbeating
84+
85+
For long running activities, Cadence provides an API for the activity code to report both liveness and progress back to
86+
the Cadence managed service.
87+
88+
progress := 0
89+
for hasWork {
90+
// send heartbeat message to the server
91+
activity.RecordActivityHeartbeat(ctx, progress)
92+
// do some work
93+
...
94+
progress++
95+
}
96+
97+
When the activity times out due to a missed heartbeat, the last value of the details (progress in the above sample) is
98+
returned from the workflow.ExecuteActivity function as the details field of TimeoutError with TimeoutType_HEARTBEAT.
99+
100+
It is also possible to heartbeat an activity from an external source:
101+
102+
// instantiate a Cadence service Client
103+
client.Client client = client.NewClient(...)
104+
105+
// record heartbeat
106+
err := client.RecordActivityHeartbeat(taskToken, details)
107+
108+
The parameters of the RecordActivityHeartbeat function are:
109+
110+
- taskToken: This is the value of the binary “TaskToken” field of the
111+
“ActivityInfo” struct retrieved inside the activity
112+
- details: This is the serializable payload containing progress information
113+
114+
Activity Cancellation
115+
116+
When an activity is cancelled (or its workflow execution is completed or failed) the context passed into its function
117+
is cancelled which sets its Done channel’s closed state. So an activity can use that to perform any necessary cleanup
118+
and abort its execution. Currently cancellation is delivered only to activities that call RecordActivityHeartbeat.
119+
120+
Async/“Manual” Activity Completion
121+
122+
In certain scenarios completing an activity upon completion of its function is not possible or desirable.
123+
124+
One example would be the UberEATS order processing workflow that gets kicked off once an eater pushes the “Place Order”
125+
button. Here is how that workflow could be implemented using Cadence and the “async activity completion”:
126+
127+
- Activity 1: send order to restaurant
128+
- Activity 2: wait for restaurant to accept order
129+
- Activity 3: schedule pickup of order
130+
- Activity 4: wait for courier to pick up order
131+
- Activity 5: send driver location updates to eater
132+
- Activity 6: complete order
133+
134+
Activities 2 & 4 in the above flow require someone in the restaurant to push a button in the Uber app to complete the
135+
activity. The activities could be implemented with some sort of polling mechanism. However, they can be implemented
136+
much simpler and much less resource intensive as a Cadence activity that is completed asynchronously.
137+
138+
There are 2 parts to implementing an asynchronously completed activity. The first part is for the activity to provide
139+
the information necessary to be able to be completed from an external system and notify the Cadence service that it is
140+
waiting for that outside callback:
141+
142+
// retrieve activity information needed to complete activity asynchronously
143+
activityInfo := activity.GetActivityInfo(ctx)
144+
taskToken := activityInfo.TaskToken
145+
146+
// send the taskToken to external service that will complete the activity
147+
...
148+
149+
// return from activity function indicating the Cadence should wait for an async completion message
150+
return "", activity.ErrResultPending
151+
152+
The second part is then for the external service to call the Cadence service to complete the activity. To complete the
153+
activity successfully you would do the following:
154+
155+
// instantiate a Cadence service Client
156+
// the same client can be used complete or fail any number of activities
157+
client.Client client = client.NewClient(...)
158+
159+
// complete the activity
160+
client.CompleteActivity(taskToken, result, nil)
161+
162+
And here is how you would fail the activity:
163+
164+
// fail the activity
165+
client.CompleteActivity(taskToken, nil, err)
166+
167+
The parameters of the CompleteActivity function are:
168+
169+
- taskToken: This is the value of the binary “TaskToken” field of the
170+
“ActivityInfo” struct retrieved inside the activity.
171+
- result: This is the return value that should be recorded for the activity.
172+
The type of this value needs to match the type of the return value
173+
declared by the activity function.
174+
- err: The error code to return if the activity should terminate with an
175+
error.
176+
177+
If error is not null the value of the result field is ignored.
178+
179+
For a full example of implementing this pattern see the Expense sample.
180+
181+
Registration
182+
183+
In order to for some workflow execution to be able to invoke an activity type, the worker process needs to be aware of
184+
all the implementations it has access to. An activity is registered with the following call:
185+
186+
activity.RegisterActivity(SimpleActivity)
187+
188+
This call essentially creates an in-memory mapping inside the worker process between the fully qualified function name
189+
and the implementation. Unlike in Amazon SWF, workflow and activity types are not registered with the managed service.
190+
If the worker receives a request to start an activity execution for an activity type it does not know it will fail that
191+
request.
192+
193+
*/
194+
package activity

‎cadence.go

Lines changed: 101 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -18,31 +18,107 @@
1818
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
1919
// THE SOFTWARE.
2020

21-
// Package cadence and its subdirectories contain a Cadence client side framework.
22-
//
23-
// The Cadence service is a task orchestrator for your application’s tasks. Applications using Cadence can execute a logical flow of
24-
// tasks, especially long-running business logic, asynchronously or synchronously. and can scale runtime on distributed
25-
// systems without you, the service owner, worrying about infrastructure needs.
26-
//
27-
//A quick example illustrates its use case. Consider Uber Eats where Cadence manages the entire business flow from
28-
// placing an order, accepting it, handling shopping cart processes (adding, updating, and calculating cart items),
29-
// entering the order in a pipeline (for preparing food and coordinating delivery), to scheduling delivery as well as
30-
// handling payments.
31-
//
32-
//Cadence consists of a programming framework (or client library) and a managed service (or backend). The framework
33-
// enables developers to author and coordinate tasks in Go code.
34-
//
35-
// The root cadence package contains common data structures. The subpackages are:
36-
//
37-
// workflow - functions used to implement workflows
38-
//
39-
// activity - functions used to implement activities
40-
//
41-
// client - functions used to create Cadence service client used to start and monitor workflow executions.
42-
//
43-
// worker - functions used to create worker instance used to host workflow and activity code.
44-
//
45-
// testsuite - unit testing framework for activity and workflow testing.
21+
/*
22+
Package cadence and its subdirectories contain the Cadence client side framework.
23+
24+
The Cadence service is a task orchestrator for your application’s tasks. Applications using Cadence can execute a
25+
logical flow of tasks, especially long-running business logic, asynchronously or synchronously. They can also scale at
26+
runtime on distributed systems.
27+
28+
A quick example illustrates its use case. Consider Uber Eats where Cadence manages the entire business flow from
29+
placing an order, accepting it, handling shopping cart processes (adding, updating, and calculating cart items),
30+
entering the order in a pipeline (for preparing food and coordinating delivery), to scheduling delivery as well as
31+
handling payments.
32+
33+
Cadence consists of a programming framework (or client library) and a managed service (or backend). The framework
34+
enables developers to author and coordinate tasks in Go code.
35+
36+
The root cadence package contains common data structures. The subpackages are:
37+
38+
- workflow - functions used to implement workflows
39+
- activity - functions used to implement activities
40+
- client - functions used to create Cadence service client used to start and
41+
monitor workflow executions.
42+
- worker - functions used to create worker instance used to host workflow and
43+
activity code.
44+
- testsuite - unit testing framework for activity and workflow testing
45+
46+
How Cadence works
47+
48+
The Cadence hosted service brokers and persists events generated during workflow execution. Worker nodes owned and
49+
operated by customers execute the coordination and task logic. To facilitate the implementation of worker nodes Cadence
50+
provides a client-side library for the Go language.
51+
52+
In Cadence, you can code the logical flow of events separately as a workflow and code business logic as activities. The
53+
workflow identifies the activities and sequences them, while an activity executes the logic.
54+
55+
Key Features
56+
57+
Dynamic workflow execution graphs - Determine the workflow execution graphs at runtime based on the data you are
58+
processing. Cadence does not pre-compute the execution graphs at compile time or at workflow start time. Therefore, you
59+
have the ability to write workflows that can dynamically adjust to the amount of data they are processing. If you need
60+
to trigger 10 instances of an activity to efficiently process all the data in one run, but only 3 for a subsequent run,
61+
you can do that.
62+
63+
Child Workflows - Orchestrate the execution of a workflow from within another workflow. Cadence will return the results
64+
of the child workflow execution to the parent workflow upon completion of the child workflow. No polling is required in
65+
the parent workflow to monitor status of the child workflow, making the process efficient and fault tolerant.
66+
67+
Durable Timers - Implement delayed execution of tasks in your workflows that are robust to worker failures. Cadence
68+
provides two easy to use APIs, **workflow.Sleep** and **workflow.Timer**, for implementing time based events in your
69+
workflows. Cadence ensures that the timer settings are persisted and the events are generated even if workers executing
70+
the workflow crash.
71+
72+
Signals - Modify/influence the execution path of a running workflow by pushing additional data directly to the workflow
73+
using a signal. Via the Signal facility, Cadence provides a mechanism to consume external events directly in workflow
74+
code.
75+
76+
Task routing - Efficiently process large amounts of data using a Cadence workflow, by caching the data locally on a
77+
worker and executing all activities meant to process that data on that same worker. Cadence enables you to choose the
78+
worker you want to execute a certain activity by scheduling that activity execution in the worker's specific task-list.
79+
80+
Unique workflow ID enforcement - Use business entity IDs for your workflows and let Cadence ensure that only one
81+
workflow is running for a particular entity at a time. Cadence implements an atomic "uniqueness check" and ensures that
82+
no race conditions are possible that would result in multiple workflow executions for the same workflow ID. Therefore,
83+
you can implement your code to attempt to start a workflow without checking if the ID is already in use, even in the
84+
cases where only one active execution per workflow ID is desired.
85+
86+
Perpetual/ContinueAsNew workflows - Run periodic tasks as a single perpetually running workflow. With the
87+
"ContinueAsNew" facility, Cadence allows you to leverage the "unique workflow ID enforcement" feature for periodic
88+
workflows. Cadence will complete the current execution and start the new execution atomically, ensuring you get to
89+
keep your workflow ID. By starting a new execution Cadence also ensures that workflow execution history does not grow
90+
indefinitely for perpetual workflows.
91+
92+
At-most once activity execution - Execute non-idempotent activities as part of your workflows. Cadence will not
93+
automatically retry activities on failure. For every activity execution Cadence will return a success result, a failure
94+
result, or a timeout to the workflow code and let the workflow code determine how each one of those result types should
95+
be handled.
96+
97+
Asynch Activity Completion - Incorporate human input or thrid-party service asynchronous callbacks into your workflows.
98+
Cadence allows a workflow to pause execution on an activity and wait for an external actor to resume it with a
99+
callback. During this pause the activity does not have any actively executing code, such as a polling loop, and is
100+
merely an entry in the Cadence datastore. Therefore, the workflow is unaffected by any worker failures happening over
101+
the duration of the pause.
102+
103+
Activity Heartbeating - Detect unexpected failures/crashes and track progress in long running activities early. By
104+
configuring your activity to report progress periodically to the Cadence server, you can detect a crash that occurs 10
105+
minutes into an hour-long activity execution much sooner, instead of waiting for the 60-minute execution timeout. The
106+
recorded progress before the crash gives you sufficient information to determine whether to restart the activity from
107+
the beginning or resume it from the point of failure.
108+
109+
Timeouts for activities and workflow executions - Protect against stuck and unresponsive activities and workflows with
110+
appropriate timeout values. Cadence requires that timeout values are provided for every activity or workflow
111+
invocation. There is no upper bound on the timeout values, so you can set timeouts that span days, weeks, or even
112+
months.
113+
114+
Visibility - Get a list of all your active and/or completed workflow. Explore the execution history of a particular
115+
workflow execution. Cadence provides a set of visibility APIs that allow you, the workflow owner, to monitor past and
116+
current workflow executions.
117+
118+
Debuggability - Replay any workflow execution history locally under a debugger. The Cadence client library provides an
119+
API to allow you to capture a stack trace from any failed workflow execution history.
120+
121+
*/
46122
package cadence
47123

48124
import "go.uber.org/cadence/internal"

0 commit comments

Comments
 (0)