Troubleshooting Spacewalk, OSAD, and Jabber

Spacewalk is the open-source ancestor of Red Hat Satellite: patch management and configuration delivery for a fleet of RHEL/CentOS hosts. It works well until the messaging layer underneath it (Jabber, carrying OSAD's push notifications to clients) starts misbehaving, and then debugging it means knowing where the logs actually live.

My running notes on this, as a gist:

Where the logs live

# Server-side OSA dispatcher
/var/log/rhn/osa-dispatcher.log

# Client-side OSA agent
/var/log/osad

Checking the messaging layer is actually up

# Confirm jabberd is running
service jabberd status

# Check the OSA dispatcher port range is listening
nmap -sT -p 5200-5400 localhost

Restarting the stack in order

service jabberd restart
rhn-satellite start

On the client:

service osad restart
rhn_check
rhn-profile-sync

# Verbose foreground mode, for watching it live
osad -N -v -v -v -v

When Jabber's database gets stuck

Sometimes restarting isn't enough and the Jabber database itself needs clearing:

rhn-satellite stop
rm -rf /var/lib/jabberd/db/*
rhn-satellite start

Making OSAD self-heal

osad occasionally stops checking in without dying outright, so it's worth forcing a periodic restart rather than waiting for someone to notice a client's gone stale:

# crontab, client-side: restart osad every ~3 hours with jitter
0 */3 * * *  sleep $(( RANDOM % 7200 / 2 )); service osad restart; rhn_check; rhn-profile-sync

The random sleep matters at fleet scale: without it, every client in the same timezone restarts its OSA agent in the same few seconds, and the dispatcher sees a thundering herd of reconnects instead of a steady trickle.

Two version-specific bugs worth knowing about

A c3p0 connection-pooling bug that showed up after a routine update:

yum downgrade c3p0-0.9.1.2-2.jpp5.noarch

And an HTTPS 400 Bad Request from the Spacewalk web UI after an httpd/mod_ssl update:

yum downgrade httpd httpd-tools mod_ssl

Both were known issues at the time (see the c3p0 bugzilla report and the Spacewalk mailing list thread), and both point at the same lesson: patch management software needs its own patch management discipline. Test the update on a non-critical Spacewalk instance before it touches the one every other host depends on.