Stator
======

Takahē's background task system is called Stator, and rather than being a
transitional task queue, it is instead a *reconciliation loop* system; the
workers look for objects that could have actions taken, try to take them, and
update them if successful.

As someone running Takahē, the most important aspects of this are:

* You have to run at least one Stator worker to make things like follows,
  posting, and timelines work.

* You can run as many workers as you want; there is a locking system to ensure
  they can coexist.

* You can get away without running any workers for a few minutes; the server
  will continue to accept posts and follows from other servers, and will
  process them when a worker comes back up.

* There is no separate queue to run, flush or replay; it is all stored in the
  main database.

* If all your workers die, just restart them, and within a few minutes the
  existing locks will time out and the system will recover itself and process
  everything that's pending.

You run a worker via the command ``manage.py runstator``. It will run forever
until it is killed; send SIGINT (Ctrl-C) to it once to have it enter graceful
shutdown, and a second time to force exiting immediately.


Technical Details
-----------------

Each object managed by Stator has a set of extra columns:

* ``state``, the name of a state in a state machine
* ``state_ready``, a boolean saying if it's ready to have a transition tried
* ``state_changed``, when it entered into its current state
* ``state_attempted``, when a transition was last attempted
* ``state_locked_until``, when the entry is locked by a worker until

They also have an associated state machine which is a subclass of
``stator.graph.StateGraph``, which will define a series of states, the
possible transitions between them, and handlers that run for each state to see
if a transition is possible.

An object becoming ready for execution happens first:

* If it's just entered into a new state, or just created, it is marked ready.
* If ``state_attempted`` is far enough in the past (based on the ``try_interval``
  of the current state), a small scheduling loop marks it as ready.

Then, in the main fast loop of the worker, it:

* Selects an item with ``state_ready`` that is in a state it can handle (some
  states are "externally progressed" and will not have handlers run)
* Fires up a coroutine for that handler and lets it run
* When that coroutine exits, sees if it returned a new state name and if so,
  transitions the object to that state.
* If that coroutine errors or exits with ``None`` as a return value, it marks
  down the attempt and leaves the object to be rescheduled after its ``try_interval``.