distributed scheduling?
any great resources to read up on architectures of distributed task schedulers? tc - meh 250
https://landing.google.com/sre/sre-book/chapters/distributed-periodic-scheduling/ It is a big topic actually. By simple scheduling I understand a task that is scheduled to run at a particular time in the future (e.g. cronjob). First off, try to implement it on a single machine (See how implemented java.util.Timer). Next, try to do the same in a distributed world where network and hardware is unreliable. Think about all bad cases might happen with scheduling and how strong consistency and high availability can solve it. Recall consensus algorithms that help to achieve consensus on state of a task (strong consistency) under split brain in networking. A task has to be idempotent also. If you are aware of what I just mentioned, you can design your scheduler.
Nomad by hashicorp, dkron, runbook too
airflow celery executor setup. make some of the components like metadata db, webserver, scheduler have HA setup.
I haven't looked into it much but at the heart of kubernetes is a job scheduler. You could also look at dask.
Do you have a link of it ??
Whats your yoe op?
Following
You can turn on the bell icon at the top right on the screen to get notified
Netflix genie
Uber eng blog on “cadence”, and Apache yarn