fix: wrong terminal counts calculated during migration check

[Sidddddarth]

Description

Wrong count calculated as part of migration check that happens every 30 seconds.
We fetch counts of terminal statuses in the status table, but since at startup we "cleanup" old jobs just by appending another aborted status irrespective of it's state, that above query can count more than one terminal status per job.

Understandably while actually migrating the jobs, we see that more jobs than we expect have been moved because we were expecting a lesser number.
This issue happened now because archival tables have a default retention of 24 hours. so on successive restarts, more and more statuses were being appended for the same job. And we expect the following expression number of jobs to be migrated:

numExpectedNumberOfMigratedJobs(e) = number of jobs(a) - number of terminal statuses in status table(b)

Due to the cleanup at startup even when a remains same, b increases based on the retention duration effectively decreasing e. And server panics when it actually migrated more jobs than e.
Now with this fix: we change b to number of jobIDs with terminal status in the status table and it's bound to remain the same even if we append more statuses for the same job.

Linear Ticket

Resolves PIPE-1827

Security

• The code changed/added as part of this pull request won't create any security issues with how the software is being used.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.72%. Comparing base (3605e90) to head (d40ff5e).

Additional details and impacted files

@@                Coverage Diff                 @@
##           release/1.39.x    #5400      +/-   ##
==================================================
+ Coverage           74.70%   74.72%   +0.01%     
==================================================
  Files                 437      437              
  Lines               61269    61269              
==================================================
+ Hits                45773    45785      +12     
+ Misses              12954    12944      -10     
+ Partials             2542     2540       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ktgowtham]

In the startup, what kind of clean up we do? and if for a job a terminal status already exists, why do we append more?

[Sidddddarth]

In the startup, what kind of clean up we do? and if for a job a terminal status already exists, why do we append more?

Just appends an aborted status without checking for the status already present.
like so:

INSERT INTO
	status_table (job_id, job_state, error_response)
SELECT
	job_id, 'aborted', '{"reason": "job max age exceeded"}'
FROM
	jobs_table
WHERE
	created_at <= $deadline;

[cisse21]

Lets change the base to master and release it in 1.40