Log when sub is force applied and apply create as update

[ktuite]

Closes getodk/central#702 and getodk/central#720

Should also help with getodk/central#699

What has been done to verify that this works as intended?

Why is this the best possible solution? Were any other approaches considered?

How does this change affect users? Describe intentional changes to behavior and behavior that could have accidentally been affected by code changes. In other words, what are the regression risks?

Does this change require updates to the API documentation? If so, please update docs/api.yaml as part of this PR.

Before submitting this PR, please make sure you have:

• run make test and confirmed all checks still pass OR confirm CircleCI build passes
• verified that any code from external sources are properly credited in comments or that everything is internally sourced

[ktuite]

Questions

• What to call extra details about how entity submission was processed and how to organize them? I started by just adding flags for forceProcessed, upsert, createAsUpdate, updateAsCreate to add to the entity def source details. updateAsCreate and forceProcessed were important for deciding whether to apply a create as an update.

• Where to add these extra details? In the entity def source feels right, but some of them could also be added to the audit log details. Would that be useful or redundant? Could be useful if showing audit log events about entities through the main system audit log. Whatever decisions made about this will be relevant to exposing the backlog.

[matthew-white]

One high-level question I have is what information needs to be logged. I see that forceProcessed is used in the catch block, but I don't know about the other information. Are you thinking that the other information will be used as part of getodk/central#699?

Instead of adding to source details, I think it'd be nice to add defined columns to the table. The schema of the audit log details has kind of gotten hard to understand, and I'd be worried about a similar situation with the source details. How about adding these two columns to the table?

• An array of the entity actions listed in the submission. (submissionActions?) With this, I think it's possible to derive a lot of other information:
  • upsert: submissionActions contains both create and update (and the version number of the entity def is 1?).
  • updateAsCreate: submissionActions does not contain create, but the version number of the entity def is 1.
  • createAsUpdate: submissionActions does not contain update, but the version number of the entity def is not 1.
  • I think it probably still makes sense to define these variables in the code like you have them, but in terms of what we store in the database, I think it'd be nice to persist just the minimal information needed.
• A boolean column for forceProcessed. We could probably derive this one too with submissionActions, but the SQL wouldn't be pretty: it'd require more than a simple join with the entity def. I think it makes sense to store this value in the database rather than try to derive it on the fly.

Where to add these extra details? In the entity def source feels right, but some of them could also be added to the audit log details. Would that be useful or redundant? Could be useful if showing audit log events about entities through the main system audit log.

My opinion is that it'd be a good idea to avoid storing this information in two places in the database. I agree that the entity def source feels right. We don't currently show audit log events about entities in the Fronted audit log. I think we can figure out what to do if/when we decide to show them. We could probably join audits and entity_def_sources.

Whatever decisions made about this will be relevant to exposing the backlog.

I think there will probably need to be a new, separate query for querying the backlog that will join the backlog table with other tables. The query could fetch information it needs from the entity_def_sources table.

[matthew-white]

Could we just check forceProcessed and not check updateAsCreate? Since only offline updates are put in the backlog, if the root def was forceProcessed, I think that means that it was also an updateAsCreate.

[matthew-white]

That makes sense to me. Only entity updates are force-processed from the backlog, and entity updates are fully handled above. Explanatory comments might be helpful though.

[ktuite]

could try v.source = { ...v.source, ...event.source } with spread syntax
or Object.assign(v.source, event.source)

Note: at some point will need to update Entity.Def.Source forApi() to merge details and forceProcess

[ktuite]

This PR tried a couple different things: over-logging of interesting flags (was the entity processed? what update vs. create vs. upsert vs. other fallback path of entity processing were we on?) as well as enough logging to support Create-As-Update.

I'm closing it in favor of #1251 which has minimal new capturing of the necessary information to be able to handle the create-as-update case.