Make task poll interval and timeout configurable in jobs scheduler

[teamurko]

Summary

The previously hardcoded values don't work well for some types of ops, e.g.

1. DLO strategy generation task takes usually ~1min
2. compaction task can take days

Changes

• Client-facing API Changes
• Internal API Changes
• Bug Fixes
• New Features
• Performance Improvements
• Code Style
• Refactoring
• Documentation
• Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

• Manually Tested on local docker setup. Please include commands ran, and their output.
• Added new tests for the changes made.
• Updated existing tests to reflect the changes made.
• No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
• Some other form of testing like staging or soak time in production. Please explain.

25/02/17 3:32:22@oh-hadoop-spark:~$ docker compose --profile with_jobs_scheduler run openhouse-jobs-scheduler - --type SNAPSHOTS_EXPIRATION --cluster local --tablesURL http://openhouse-tables:8080 --jobsURL http://openhouse-jobs:8080 - --tableMinAgeThresholdHours 0 --taskPollIntervalMs 5000 --
taskTimeoutMs 120000

2025-02-17 23:33:07 INFO  JobsScheduler:111 - Starting scheduler
2025-02-17 23:33:08 INFO  JobsScheduler:151 - Fetching task list based on the job type: SNAPSHOTS_EXPIRATION
2025-02-17 23:33:09 INFO  OperationTasksBuilder:47 - Fetched metadata for 2 tables
2025-02-17 23:33:09 INFO  OperationTasksBuilder:134 - Found metadata TableMetadata(super=Metadata(creator=openhouse), dbName=u_openhouse, tableName=t, creationTimeMs=1739831342874, isPrimary=true, isTimePartitioned=false, isClustered=false, jobExecutionProperties={}, retentionConfig=null, historyConfig=null, replicationConfig=null)
2025-02-17 23:33:09 INFO  OperationTasksBuilder:134 - Found metadata TableMetadata(super=Metadata(creator=openhouse), dbName=u_openhouse, tableName=u, creationTimeMs=1739831351304, isPrimary=true, isTimePartitioned=false, isClustered=false, jobExecutionProperties={}, retentionConfig=null, historyConfig=null, replicationConfig=null)
2025-02-17 23:33:09 INFO  JobsScheduler:162 - Submitting and running 2 jobs based on the job type: SNAPSHOTS_EXPIRATION
2025-02-17 23:33:09 INFO  OperationTask:84 - Launching job for TableMetadata(super=Metadata(creator=openhouse), dbName=u_openhouse, tableName=t, creationTimeMs=1739831342874, isPrimary=true, isTimePartitioned=false, isClustered=false, jobExecutionProperties={}, retentionConfig=null, historyConfig=null, replicationConfig=null)
2025-02-17 23:33:09 INFO  OperationTask:84 - Launching job for TableMetadata(super=Metadata(creator=openhouse), dbName=u_openhouse, tableName=u, creationTimeMs=1739831351304, isPrimary=true, isTimePartitioned=false, isClustered=false, jobExecutionProperties={}, retentionConfig=null, historyConfig=null, replicationConfig=null)
2025-02-17 23:33:10 INFO  OperationTask:110 - Launched a job with id SNAPSHOTS_EXPIRATION_u_openhouse_u_61555bb0-d77b-4b10-a26f-e689e77fc6f7 for TableMetadata(super=Metadata(creator=openhouse), dbName=u_openhouse, tableName=u, creationTimeMs=1739831351304, isPrimary=true, isTimePartitioned=false, isClustered=false, jobExecutionProperties={}, retentionConfig=null, historyConfig=null, replicationConfig=null)
2025-02-17 23:33:10 INFO  OperationTask:110 - Launched a job with id SNAPSHOTS_EXPIRATION_u_openhouse_t_0de8fc68-25d7-4dcc-a425-93f5606febc2 for TableMetadata(super=Metadata(creator=openhouse), dbName=u_openhouse, tableName=t, creationTimeMs=1739831342874, isPrimary=true, isTimePartitioned=false, isClustered=false, jobExecutionProperties={}, retentionConfig=null, historyConfig=null, replicationConfig=null)
2025-02-17 23:33:30 INFO  OperationTask:156 - Finished job for entity TableMetadata(super=Metadata(creator=openhouse), dbName=u_openhouse, tableName=t, creationTimeMs=1739831342874, isPrimary=true, isTimePartitioned=false, isClustered=false, jobExecutionProperties={}, retentionConfig=null, historyConfig=null, replicationConfig=null): JobId SNAPSHOTS_EXPIRATION_u_openhouse_t_0de8fc68-25d7-4dcc-a425-93f5606febc2, executionId 2, runTime 17744, queuedTime 13709, state SUCCEEDED
2025-02-17 23:33:30 INFO  OperationTask:156 - Finished job for entity TableMetadata(super=Metadata(creator=openhouse), dbName=u_openhouse, tableName=u, creationTimeMs=1739831351304, isPrimary=true, isTimePartitioned=false, isClustered=false, jobExecutionProperties={}, retentionConfig=null, historyConfig=null, replicationConfig=null): JobId SNAPSHOTS_EXPIRATION_u_openhouse_u_61555bb0-d77b-4b10-a26f-e689e77fc6f7, executionId 3, runTime 17748, queuedTime 13734, state SUCCEEDED
2025-02-17 23:33:30 INFO  JobsScheduler:205 - Finishing scheduler for job type SNAPSHOTS_EXPIRATION, tasks stats: 2 created, 2 succeeded, 0 cancelled (timeout), 0 failed, 0 skipped (no state)

[sumedhsakdeo]

lgtm