Disaster recovery: restore an OpenSimulator grid
Data inventory, consistent shutdown, backups, dependencies and restoration exercises.
Define RPO, RTO and scope
RPO is the maximum acceptable data-loss window; RTO is the target recovery duration. Define both per service: accounts, regions, inventories, assets, money and voice. An OAR copy alone cannot reconstruct an entire grid.
Inventory all dependencies
Inventory ROBUST and region databases, FSAssets metadata, data/spool, INIs, private keys, certificates, DLLs, service units, DNS, automation and external services. Identify add-ons with their own SQL storage (money and custom features).
Prepare a consistent snapshot
For reliable recovery, stop new writes, gracefully stop simulator processes, then ROBUST and asset-producing modules. Capture SQL and files at one logical recovery point. Hot backups require coordinated, validated snapshots; a noon SQL dump and midnight FSAssets copy can be inconsistent.
SQL databases and integrity
Back up every active database, not just the central one. Check dump tool exit codes and archive content. A database dump does not include FSAssets payload files or storage belonging to separate services.
FSAssets payloads and disk space
Copy data and spool, accounting for writes not yet compressed. Check free bytes and inodes on the destination. Never delete assumed orphan files to speed a restore: content hashes may be shared by many assets.
OAR and IAR: complementary layers
OAR supports targeted region restores and IAR user inventory restores. They complement but do not replace SQL and FSAssets backups. Record --noassets options, permissions and versions; test imports in isolation.
Restoration order
On a test host: prepare OS, dependencies, mounts, FSAssets and service accounts; restore databases, files and configs; validate DNS/private network; start ROBUST then simulators; test login and inventory before reopening public access.
Validate recovered data
Test known asset UUIDs, textures, inventory, objects, local teleports and Hypergrid if enabled. Correlate SQL, Robust.log and OpenSim.log. An 'active' systemd unit does not prove the grid is restored.
Avoid simultaneous writers
If the original infrastructure might return, isolate it before opening the restored grid; two live installations using the same identity and databases can write conflicting data. Protect credentials, identities and certificates at every step.
Recovery drills and incident records
Schedule end-to-end restore drills off production and record versions, commands, duration, missing services and lessons learned. Keep offsite/offline copies according to risk. Define who can authorize permanent failover.