Daily monitoring of an OpenSimulator grid
CPU, memory, swap, disks, inodes, logs, services and alerts: an operations plan.
Define monitoring coverage
Monitor every region process, ROBUST, MySQL/MariaDB, FSAssets and additional modules separately. Include host/VM system metrics, backup results and a functional login test. A green service status alone is insufficient.
CPU load, latency and steal time
Compare available CPUs with OpenSim process activity and hypervisor capacity. vmstat shows runnable tasks, idle, iowait and steal time where available. Low average CPU use does not rule out saturation of one core or critical thread.
Example: adapt and verify on a test instance.
vmstat 1 10 top -H -p PID_SIMULATEUR ps -eo pid,cmd,%cpu,%mem --sort=-%cpu | head
RAM, Linux cache and swap
On Linux, low 'free' RAM alone is not proof of memory exhaustion: distinguish available RAM, reclaimable cache, actual swap and page-in/page-out activity. Trend process RSS over time. Persistent reproducible growth is more meaningful than one snapshot.
Example: adapt and verify on a test instance.
free -h vmstat 1 10 ps -eo pid,rss,%mem,cmd --sort=-rss | head
Disks, inodes and I/O latency
Monitor SQL volumes, fsassets/data, fsassets/tmp, region caches and backups. Check both df -h and df -i: inode exhaustion can precede byte exhaustion. Measure latency and disk utilization if textures are slow or spool is stalled.
Example: adapt and verify on a test instance.
df -h df -i lsblk -f # Si sysstat est installé : iostat -xz 1 5
Logs: detect failures before user impact
Watch for new errors in OpenSim.log and Robust.log, identify the first relevant message and timestamp, and correlate with SQL/systemd/disk logs. Avoid alerting on every WARN without context.
Example: adapt and verify on a test instance.
grep -nEi 'ERROR|FATAL|exception|timeout|refused' /chemin/vers/OpenSim.log | tail -n 60 grep -nEi 'ERROR|FATAL|exception|timeout|refused' /chemin/vers/Robust.log | tail -n 60
Service state, restarts and dependencies
Check unexpected restarts of ROBUST, MySQL and region processes, available mounts and readiness of dependencies. Automated restart loops may mask a worsening fault; preserve history and exit codes.
Example: adapt and verify on a test instance.
systemctl --failed journalctl -p err --since '1 hour ago' --no-pager findmnt -T /srv/opensim/fsassets/data
Network, DNS and ports
Monitor public login/Hypergrid reachability, private region-to-ROBUST links and simulator TCP/UDP ports. Distinguish a listening socket from a working login flow; an HTTP probe cannot guarantee asset availability.
Example: adapt and verify on a test instance.
ss -lntup getent hosts grid.example.org # Tester depuis un hôte externe autorisé : curl -v http://grid.example.org:8002/
Define actionable alert thresholds
Configure warning/critical thresholds based on actual growth: inode %, byte %, missing mounts, stopped service, failed backups and sustained errors. Example 70/85/95% thresholds are local operational policy, not OpenSimulator settings.
Dashboard and indicators
A useful dashboard ties each service to CPU/RAM, recent errors, latency, availability and last successful backup. Keep a timeline of configuration changes and upgrades. Protect personal information and admin access.
Daily, weekly and monthly routine
Daily: review errors, services, backups, inodes and free space. Weekly: inspect trends, slowdowns, restarts and capacity forecast. Monthly: test a targeted restore and review alerting, certificates, permissions and versions. These cadences are operational suggestions.
Separate diagnosis from tuning
This chapter focuses on evidence and ongoing monitoring; physics-engine tuning, scripting engines, avatar limits and FrameTime belong in the Performance guide. Measure before changing settings and keep a rollback.