Entities from finalized registration and update forms after resetting "saved forms and entities"

[dbemke]

ODK Collect version

the master version ce00913

Android version

10, 14

Device used

Redmi 9T, Pixel 7a

Problem description

I’m not sure what should happen with entities from finalized registration and update forms after resetting "saved forms and entities" in Collect setting.
After resetting saved forms there aren’t any entities/rows in the db files. Afterwards, if a user finalizes an update to an entity (in the update form), the finalized entity doesn’t appear on the list in the update form (I guess it isn’t created locally/offline – no new rows in db files). If a user finalizes the registration form, in the update form other entities are removed form the list and there’s only the new entity (registration form creates a row in db files). So there are 2 different scenarios what appears in the update form depending on what a user finalizes after resetting saved forms.
I guess it means that after resetting saved forms local entities are enabled again only after refreshing the list of blank forms (or downloading them manually).
Is it ok for the time being or finalizing an entity form should trigger local entities (after resetting) ?

Steps to reproduce the problem

1. Download a project with entities registration form, follow-up form and update form with some existing entities (e.g. trees registration, follow- up, update with the new entity spec user "one" in https://staging.getodk.cloud/#/projects/101/app-users).
2. Go to Settings- Project management- Reset, select "Saved forms and entities” and tap "Reset”.
3. Go to Settings- Form management- Auto send and set "off”.
4. Go to the update form, select an entity an finalize the form.
5. Go to the update form and check with version of the entity updated in step 4 is on the list (there is the version without the update + other entities are on the list).
6. Go to the registration form and create an entity.
7. Go to the update form and check which entities are on the list (there’s only the new entity, other entities aren’t on the list).

Expected behavior

Instead of being able to reset Saved Forms and entities, users should only be able to reset Saved Forms (like before). Additionally, forms that create/update entities (determinable via Instance#canDeleteBeforeSend) should not be reset. This makes reset and manual delete consistent - users currently can't delete forms that create/update forms.

[lognaturel]

This is not a good state to be in!

Here's what I would expect "saved forms and entities" to do:

• Remove the existing Entity List tables
• Rebuild those tables from the CSVs that were originally downloaded from the server

This would honor the intent of that reset action which is to remove user/locally-created content but keep server/form-level content.

I'm realizing this may be tricky to do currently: I don't know whether there's an easy way to identify those source CSVs at that point in time since they are stored at the form level. If it's a big lift, I think we should consider removing the ability to clear out filled forms and entities for this first version.

This issue is also related to #6425 and to the fact that when multiple forms use the same Entity List we currently download it multiple times. All of these have to do with how the system identifies what is an Entity List vs a static CSV attachment.

[grzesiek2010]

We store the state of entities in the database, marking them as either online or offline. Can’t we just remove the offline ones when resetting saved forms? As I understand it, the result would be the same as deleting all entities from the database and rebuilding it using the CSVs originally downloaded from the server but it's simpler.

[lognaturel]

I believe we only know whether the Entity has any change that was made offline. For example, if you get an Entity from the server and update it locally, it is marked as offline. We don’t keep the original Entity value in the database currently, we mutate it in place.

[grzesiek2010]

Right, how about creating a copy of the tables containing entity data each time we save entities from the server? Then, during resetting, we could restore the last saved state.
This would take some extra time during downloading entities and increase the size of the database but it could be the easiest solution.

[lognaturel]

I'm worried about adding yet another source of truth! The nice thing about reading from CSV again is that it really is what was received from the server. That matches my idea of a reset -- go back to the truth.

How about partially backing out of this reset for this release? We would go back to the reset option just being for "saved forms", not entities. Now that we have db state about each finalized submission saying whether or not it's entity-creating/updating, we can prevent those from being deleted. In other words, if a user chooses to reset submissions, it will only reset the ones that don't create or update entities. I think for now we can leave it like that with no additional explanation and then we can design a longer term solution.

Could we also add an analytics event for resetting submissions?

[seadowg]

After discussing on a call, we decided to go with @lognaturel's proposal. I've added an expected behaviour section to the issue.