gossip

Gossip membership and failure detection

What to watch

Heartbeats spreading; a crashed node marked down after 4s; generations after restart

Seed: 15. The program asks a question at the interesting moment; with Predict on, the run stops there so you can answer before you see what happens.

Run it

Edit any file and press Run. The tabs hold the program and the packages it uses.

# package main
use github.com/synodlang/examples/gossip
use std/lesson
use std/sim

define main
  sim.Seed 15
  set nodes to (sim.Cluster of 6 running gossip)
  each i, n in nodes
    if (i more than 0)
      gossip.Join n to (item 0 of nodes)
  after 8s
    sim.Crash (item 2 of nodes)
    lesson.Ask "How long until the others mark n3 as down?" choosing {"The moment it stops responding, all at once", "A few seconds, each node noticing at its own time", "Never: gossip cannot detect a crash"}
  after 16s
    sim.Restart (item 2 of nodes)
    lesson.Note "n3 comes back with a new generation, so its lower heartbeat is still believed."
  sim.StopAt 25s
# package gossip
use std/disk
use std/lists
use std/net
use std/random
use std/time

digest has
  set known to {}

Node has
  set Generation to 0
  set Heartbeat to 0
  set Known to {}
  set Down to {}
  set peers to {}
  set seenAt to {}

define Join [n: Node] to [seed: node]
  set n.peers to {seed}

# Versions combine generation and heartbeat, so a restarted node is believed again
define [n: Node] Version
  result ((n.Generation times 1000000) plus n.Heartbeat)

define [n: Node] Learn [k: node] at [v: number]
  set item k of n.Known to v
  set item k of n.seenAt to time.Now
  if ((k is not n) and (k is not in n.peers))
    set n.peers to (lists.Append k to n.peers)

define [n: Node] CheckPeers
  set suspects to {}
  each k, t in n.seenAt
    if ((time.Now minus t) more than 4s)
      set suspects to (lists.Append k to suspects)
  set n.Down to suspects

define [n: Node] start
  set n.Generation to ((disk.Read "generation" on n or 0) plus 1)
  disk.Write n.Generation at "generation" on n
  time.Every 1s
    set n.Heartbeat to (n.Heartbeat plus 1)
    n.Learn n at n.Version
    n.CheckPeers
    if ((lists.Size n.peers) more than 0)
      net.Send (message digest with known n.Known) from n to (random.Pick n.peers)

when [n: Node] (net.Received digest from sender)
  each k, v in digest.known
    if (k is in n.Known)
      if (v more than (item k of n.Known))
        n.Learn k at v
    else
      n.Learn k at v
  if (sender is not in n.peers)
    set n.peers to (lists.Append sender to n.peers)

On your own machine: synod run ./gossip. Source on GitHub · all examples