feat: do not store attestation in DB

[migmartri]

Currently, attestations are stored both in DB and in the CAS. This causes the DB usage to grow, especially now that we also store policy evaluations, etc in them.

One solution would be to

a) stop storing the attestation in the DB
b) update the biz layer so nothing changes from the API POV, just an inevitable response slowdown for which we could
c) Add a cache layer, i.e something simple in memory like, Nats KV with TTL

In the future we might want to not to couple the wf run describe API call to retrieving attestation too, and have another API endpoint/flag to do so, that way the API call can be fast regardless of cache MISS

cc/ @jiparis @javirln

[javirln]

I'm happy with the option my only concern is the size of the actual attestations. With the in-memory lru approach I would say the "problem" is that is not shared between pods and that is using resources of the actual pods.

And with NATs, the max size of the messages. By default is 1 MB with a maximum configurable of 64 MB

[migmartri]

Good points @javirln, definitely in-memory is not a good idea.

[jiparis]

Also, we still need to support organizations without a configured CAS and use the DB for those.

[migmartri]

Also, we still need to support organizations without a configured CAS and use the DB for those.

We do, but inline is becoming more and more a fallback/development mode in my mind