bingo: add a reachability monitor for the status page
datacenterbingo gets a public page showing whether each datacenter's network answers. The probing lives here, next to the schema and the admin, since the public site only has a read-only role. bin/monitor runs as a second container from the same image. Once a minute it opens one TCP connection to every active target (a handshake or a refusal both count as answered) and folds the results into 5-minute rows per datacenter, plus a per-bucket row for control hosts in Russia, so a lost vantage point isn't mistaken for dark sites. Rows are kept 30 days. Targets are managed at /admin/bingo/monitor; only hosts a public source ties to that one site belong there. Migration 017 also grants the new tables to the bingo role, so the public site can read them without a storage redeploy. Assisted-by: Claude:opus-5-5
This commit is contained in:
parent
e0038ef3a4
commit
0cc3e1dece
10 changed files with 415 additions and 1 deletions
42
bin/monitor
Executable file
42
bin/monitor
Executable file
|
|
@ -0,0 +1,42 @@
|
|||
#!/usr/bin/env ruby
|
||||
# Reachability worker for datacenterbingo's status page (Bingo::Monitor).
|
||||
# Runs as its own container from the asxpio image; the web container owns
|
||||
# migrations, so this waits until the monitor tables exist.
|
||||
require 'logger'
|
||||
require 'fileutils'
|
||||
|
||||
$root = File.expand_path('..', __dir__)
|
||||
$logger = Logger.new($stdout)
|
||||
$env = ENV.fetch('RACK_ENV', 'development')
|
||||
$stdout.sync = true
|
||||
|
||||
require_relative '../lib/db'
|
||||
DB.connect!
|
||||
until DB.connection.table_exists?(:bingo_monitor_buckets)
|
||||
$logger.info('monitor: waiting for migrations')
|
||||
sleep 10
|
||||
end
|
||||
require_relative '../lib/bingo'
|
||||
require_relative '../lib/bingo_monitor'
|
||||
|
||||
HEARTBEAT = ENV.fetch('MONITOR_HEARTBEAT', '/tmp/monitor.heartbeat')
|
||||
pruned_at = nil
|
||||
|
||||
loop do
|
||||
started = Time.now.utc
|
||||
begin
|
||||
targets = Bingo::Monitor::Target.where(active: true).all
|
||||
results = Bingo::Monitor.round(targets)
|
||||
Bingo::Monitor.record!(results, at: started)
|
||||
if pruned_at.nil? || started - pruned_at > 3600
|
||||
Bingo::Monitor.prune!(started)
|
||||
pruned_at = started
|
||||
end
|
||||
FileUtils.touch(HEARTBEAT)
|
||||
up = results.count { |_, ms| ms }
|
||||
$logger.info("monitor: #{up}/#{results.size} answered in #{(Time.now.utc - started).round(1)}s")
|
||||
rescue StandardError => e # a DB blip must not end the worker
|
||||
$logger.error("monitor: #{e.class}: #{e.message}")
|
||||
end
|
||||
sleep [Bingo::Monitor::INTERVAL - (Time.now.utc - started), 1].max
|
||||
end
|
||||
Loading…
Add table
Add a link
Reference in a new issue