Reliability

Leases, heartbeats and recovering interrupted work

Distributed work is only useful if the coordinator can tell the difference between active, completed and interrupted assignments. OpenPuzzle treats that lifecycle as protocol state rather than guessing from silence.

Why a lease exists

An OpenPuzzle assignment is not just a hexadecimal start and end value. It also has a lifecycle. When a client claims a range, the coordinator records who owns that temporary assignment and how long the assignment can be considered live. The lease gives the system a bounded answer to a difficult distributed-systems question: how long should the server continue trusting a client that has stopped talking?

Without a lease, a machine that loses power could leave work permanently marked as active. With an overly aggressive lease, a slow network or temporary outage could cause valid work to be reclaimed too early. OpenPuzzle therefore treats progress updates as evidence that the assignment is still alive and extends the useful lifetime of that assignment while the client is working.

Heartbeats are operational evidence, not proof of a solution

The client reports non-secret execution information such as assignment identity, checkpoint, keys checked and measured speed. These updates allow the coordinator to maintain recent progress and renew the lease. They are deliberately separate from any potential solution material.

A heartbeat should also reflect local reality. If the stored process identity no longer matches a live process, the client must not keep reporting the assignment as running simply because an old PID exists. This is one reason OpenPuzzle 1.0.17 binds process state to the Linux boot identity as well as the PID.

Completion, failure and interruption are different

Normal completion means the engine reached the end of its assigned interval or produced a terminal result that the client can classify. A failure means execution ended abnormally. An interruption is different again: the previous process disappeared without leaving the normal completion evidence that the client expected.

Keeping these outcomes separate matters to the scheduler. A successfully completed range becomes part of completed coverage. An interrupted or cancelled range can be made eligible for future work instead of being falsely counted as searched. This avoids converting a power cut, reboot or killed process into a permanent coverage gap.

Recovery after restart

On startup the client inspects persisted execution state and reconciles it with the current machine state. If process identity is valid and the process is still alive, the assignment can continue to be treated as active. If identity is stale, missing or cannot be verified safely, recovery fails closed: the client does not trust the PID alone.

For legacy state created before boot-bound identity existed, the safe behavior is also conservative. The client prefers recovery and explicit reconciliation over sending a signal to a process that merely happens to reuse the same numeric PID.

Why this improves coverage quality

Coverage metrics are only meaningful when terminal states are honest. If an interrupted range were marked completed, the public percentage would overstate the amount of keyspace actually searched. If every interrupted range were permanently discarded, the coordinator would accumulate holes. Lease expiry, cancellation and recycling provide a practical middle ground.

Operational rule: silence does not prove completion. OpenPuzzle records interruption explicitly and only counts work as completed when the client has acceptable terminal evidence.
Related reading: Boot-bound process identity · Range allocation · Methodology.

← Back to all research notes