OpenSimulator · EN

Disaster recovery: restore an OpenSimulator grid

Data inventory, consistent shutdown, backups, dependencies and restoration exercises.

These procedures cover common OpenSimulator administration tasks. Verify settings for your version and prepare a backup before making changes.

Define RPO, RTO and scope

RPO is the maximum acceptable data-loss window; RTO is the target recovery duration. Define both per service: accounts, regions, inventories, assets, money and voice. An OAR copy alone cannot reconstruct an entire grid.

Inventory all dependencies

Inventory ROBUST and region databases, FSAssets metadata, data/spool, INIs, private keys, certificates, DLLs, service units, DNS, automation and external services. Identify add-ons with their own SQL storage (money and custom features).

Prepare a consistent snapshot

For reliable recovery, stop new writes, gracefully stop simulator processes, then ROBUST and asset-producing modules. Capture SQL and files at one logical recovery point. Hot backups require coordinated, validated snapshots; a noon SQL dump and midnight FSAssets copy can be inconsistent.

SQL databases and integrity

Back up every active database, not just the central one. Check dump tool exit codes and archive content. A database dump does not include FSAssets payload files or storage belonging to separate services.

FSAssets payloads and disk space

Copy data and spool, accounting for writes not yet compressed. Check free bytes and inodes on the destination. Never delete assumed orphan files to speed a restore: content hashes may be shared by many assets.

OAR and IAR: complementary layers

OAR supports targeted region restores and IAR user inventory restores. They complement but do not replace SQL and FSAssets backups. Record --noassets options, permissions and versions; test imports in isolation.

Restoration order

On a test host: prepare OS, dependencies, mounts, FSAssets and service accounts; restore databases, files and configs; validate DNS/private network; start ROBUST then simulators; test login and inventory before reopening public access.

Validate recovered data

Test known asset UUIDs, textures, inventory, objects, local teleports and Hypergrid if enabled. Correlate SQL, Robust.log and OpenSim.log. An 'active' systemd unit does not prove the grid is restored.

Avoid simultaneous writers

If the original infrastructure might return, isolate it before opening the restored grid; two live installations using the same identity and databases can write conflicting data. Protect credentials, identities and certificates at every step.

Recovery drills and incident records

Schedule end-to-end restore drills off production and record versions, commands, duration, missing services and lessons learned. Keep offsite/offline copies according to risk. Define who can authorize permanent failover.

Further reading

Source code and references