Purge worker_running_table for INACTIVE worker

[loicmathieu]

When a worker goes to the INACTIVE state, we should purge the worker_running table to be sure nothing is left for this worker.

Analysis may be done to check if other tables should also be purged.

[shivanshsharma13]

Hi @loicmathieu, After looking at the codebase, here's what I found and what I'm planning:

Root cause: When a worker transitions to INACTIVE, the worker_job_running table is never cleaned up for that worker leaving stale entries behind.

Proposed fix:

1. Add a deleteAllForWorker(String workerUid) method to WorkerJobRunningStateStore and implement it in AbstractJdbcWorkerJobRunningStateStore a simple DELETE by worker_uid
2. Call it in handleAllServiceInNotRunningState() when the instance is a WORKER, so entries are purged as part of the INACTIVE transition
3. Add a test asserting that stale entries are gone once the worker reaches INACTIVE

I also noticed the re-emission path (reEmitWorkerJobsForWorker) has some gaps that when a dead worker's tasks are reassigned to another worker, the original stale entries in worker_job_running are sometimes not deleted either. Should I include a fix for that in this PR as well, or keep it focused on just the INACTIVE purge for now?

Happy to get your thoughts before I open a PR!

[loicmathieu]

@shivanshsharma13 this is either we purge entries when transitionned to INACTIVE or we purge them after we re-emit them.
I didn't do a deep analysis for now, the issue is only to have a look later so I'm not sure what's the best approach here.

[shivanshsharma13]

Ok, got it! I'll treat this as research for now and hold off on opening a PR until I have a fixed approch!