Workflow Definition Deserialization Fails on Resuming Suspended Workflow Starting w/ v3.3.2

[cali-llama]

Description

When deserializing an existing workflow definition, the deserialization fails ONLY when resuming the workflow instance. The initial workflow execution runs fine, but when it's suspended, and we return to resume the workflow from a bookmark, the deserialization of the workflow definition fails.

I recently upgraded from v3.0.0 to v3.3.3, and I noticed this behavior. I downgraded to v3.3.0 and iteratively applied each patch until the problem surfaced (once I got to v3.3.2).

Will include the json workflow definition of the workflow I'm using to trigger the error.

The stack trace is as follows...

System.NullReferenceException: Object reference not set to an instance of an object.
   at Elsa.Workflows.ActivityFactory.ReadSyntheticInputs(ActivityDescriptor activityDescriptor, IActivity activity, JsonElement activityRoot, JsonSerializerOptions options)
   at Elsa.Workflows.ActivityFactory.Create(Type type, ActivityConstructorContext context)
   at Elsa.Workflows.Serialization.Converters.ActivityJsonConverter.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, T& value, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.Deserialize(Utf8JsonReader& reader, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo`1 jsonTypeInfo, Nullable`1 actualByteCount)
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 json, JsonTypeInfo`1 jsonTypeInfo)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
   at Elsa.Workflows.Serialization.Serializers.JsonActivitySerializer.Deserialize(String serializedActivity)
   at Elsa.Workflows.Management.Mappers.WorkflowDefinitionMapper.Map(WorkflowDefinition source)
   at Elsa.Workflows.Management.Materializers.JsonWorkflowMaterializer.ToWorkflow(WorkflowDefinition definition)
   at Elsa.Workflows.Management.Materializers.JsonWorkflowMaterializer.MaterializeAsync(WorkflowDefinition definition, CancellationToken cancellationToken)
   at Elsa.Workflows.Management.Services.WorkflowDefinitionService.MaterializeWorkflowAsync(WorkflowDefinition definition, CancellationToken cancellationToken)
   at Elsa.Workflows.Management.Services.WorkflowDefinitionService.FindWorkflowGraphAsync(WorkflowDefinitionFilter filter, CancellationToken cancellationToken)
   at Elsa.Workflows.Management.Services.CachingWorkflowDefinitionService.<>c__DisplayClass12_0`1.<<GetFromCacheAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Elsa.Caching.Services.CacheManager.<>c__DisplayClass8_0`1.<<GetOrCreateAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Extensions.Caching.Memory.CacheExtensions.GetOrCreateAsync[TItem](IMemoryCache cache, Object key, Func`2 factory, MemoryCacheEntryOptions createOptions)
   at Elsa.Caching.Services.CacheManager.GetOrCreateAsync[TItem](Object key, Func`2 factory)
   at Elsa.Workflows.Management.Services.CachingWorkflowDefinitionService.GetFromCacheAsync[T](String cacheKey, Func`1 getObjectFunc, Func`2 getChangeTokenKeyFunc)
   at Elsa.Workflows.Management.Services.CachingWorkflowDefinitionService.FindWorkflowGraphAsync(WorkflowDefinitionFilter filter, CancellationToken cancellationToken)
   at Elsa.Workflows.Runtime.LocalWorkflowClient.GetWorkflowGraphAsync(WorkflowDefinitionHandle definitionHandle, CancellationToken cancellationToken)
   at Elsa.Workflows.Runtime.LocalWorkflowClient.GetWorkflowGraphAsync(WorkflowInstance workflowInstance, CancellationToken cancellationToken)
   at Elsa.Workflows.Runtime.LocalWorkflowClient.RunInstanceAsync(WorkflowInstance workflowInstance, RunWorkflowInstanceRequest request, CancellationToken cancellationToken)
   at Elsa.Workflows.Runtime.LocalWorkflowClient.RunInstanceAsync(RunWorkflowInstanceRequest request, CancellationToken cancellationToken)
   at Elsa.Workflows.Runtime.BookmarkResumer.ResumeAsync(BookmarkFilter filter, ResumeBookmarkOptions options, CancellationToken cancellationToken)
   at Services.Workflow.Engine.Runner.ResumeWorkflowAsync(String subdomain, String instanceId, String bookmarkId, ExternalConditionInput conditionInput, CancellationToken cancellationToken) in /Users/chris.nichols/bitbucket/vnextplatform/Services.Workflow/Engine/Runner.cs:line 154

Steps to Reproduce

I'm not sure if this behavior is specific to resuming all workflows, but by steps to repro (consistently) are...

1. Start a workflow with one or more bookmarking activities.
2. Run the workflow until it creates the bookmark and suspends itself.
3. Resume the workflow...In my case, I'm using BookmarkResumer.ResumeAsync() to accomplish this as seen in the stack trace.
4. The error occurs every time in v3.3.2+ when deserializing the definition via JsonActivitySerializer.

Additional Details

I'm running .NET9 via Mono on a MacOS Apple M2 Max ARM chip, and Rider has disabled external library debugging for this chipset temporarily, so my debugging is severely limited, but from what I can tell, the ActivityDescriptor parameters is null in the first parameter to Elsa.Workflows.ActivityFactory.ReadSyntheticInputs(), which causes the exception.

Persistence provider is Postgres

The only other detail of note to mention is that I am creating my workflows manually based off some custom json from our own custom workflow builder experience. But after building the workflow, I am using Elsa's JsonActivitySerializer to build the json representation and then hooking into the IWorkflowDefinitionPublisher.PublishAsync() to publish it.

Attachments

WorkflowDefinition.Data.json

WorkflowDefinition.StringData.json