FSAssets: hot backups and online pre-copy
Production backups, online rsync pre-copy, SQL consistency and restore validation.
Define hot backups and their limits
Copying files while services are writing is a useful pre-copy, but does not automatically form a consistent restore point. FSAssets links SQL metadata to disk payloads. Preserve SQL and files as a recoverable pair; include any active temporary spool in the plan.
Inventory storage paths and backup destination
Before copying, identify configured BaseDirectory, SpoolDirectory, the actual SQL table/database, and every writer. Examples assume data and spool are both under /srv/opensim/fsassets/. If not, back up all real paths separately. Ensure enough free bytes and inodes on a distinct volume.
Generic example to adapt and test, not a ready-to-run production path.
findmnt -T /srv/opensim/fsassets df -h /srv/opensim/fsassets /mnt/backup df -i /srv/opensim/fsassets /mnt/backup # Les chemins sont fictifs : vérifier BaseDirectory et SpoolDirectory.
Low-impact pre-copy while online
Start a first rsync pass without --delete, with deliberately reduced CPU and disk priority. It can be restarted but may include files captured at different times. nice and ionice affect scheduling depending on the storage and kernel; they cannot guarantee zero production impact.
Generic example to adapt and test, not a ready-to-run production path.
nice -n 19 ionice -c2 -n7 rsync -a --info=progress2 \ /srv/opensim/fsassets/ \ /mnt/backup/fsassets/
Keep a long copy running across SSH disconnects
For long operations, run rsync inside screen or tmux and detach. Reconnect using screen -r and record the final exit code. If the session disappears, inspect the actual process; a missing screen session does not mean rsync is still working.
Generic example to adapt and test, not a ready-to-run production path.
screen -S fsassets_backup # Dans screen : lancer la commande rsync précédente. # Détacher : Ctrl+A, puis D. screen -ls screen -r fsassets_backup # Après sortie de screen, contrôler les journaux et le code de retour.
Preview differences with --dry-run
After the first pass, run rsync -a --dry-run to see what rsync would transfer without writing. Trailing slashes here mean copying a directory’s contents. With active writers, results may change immediately. An empty difference list does not prove SQL consistency, byte-for-byte equality or successful recovery.
Generic example to adapt and test, not a ready-to-run production path.
rsync -a --dry-run --itemize-changes \ /srv/opensim/fsassets/ \ /mnt/backup/fsassets/ # Ne change rien. À répéter à l'état figé.
Create a consistent restore point
The easiest consistency boundary is a short no-write window: gracefully stop or block every process that writes to FSAssets, including the ROBUST asset service and any other writers. While writes are stopped, finish rsync, take a database-consistent SQL backup appropriate to the tables, and retain files and dump as one recovery set. Measure actual duration instead of promising a sub-minute outage.
Generic example to adapt and test, not a ready-to-run production path.
# PSEUDO-PROCÉDURE : adapter à l'architecture réelle. # 1. Bloquer/arrêter TOUTES les écritures FSAssets et vérifier les logs. rsync -a --info=progress2 \ /srv/opensim/fsassets/ \ /mnt/backup/fsassets/ # 2. Tant que les écritures restent arrêtées : mysqldump --single-transaction --routines --triggers --events \ --databases opensim > /mnt/backup/opensim.sql # 3. Vérifier les codes de sortie et conserver ce lot sous un identifiant commun.
SQL backups without shutting down the grid: conditions
An InnoDB --single-transaction dump provides a transactional SQL view under suitable MySQL conditions but does not synchronize the FSAssets payload bytes. A strictly no-pause backup requires a coordinated, tested snapshot or recovery-point mechanism encompassing files, spool and SQL. Do not label a 02:00 SQL dump and a 03:00 file copy as one atomic backup.
Protect backups and handle retention
Copy to a dedicated destination, distinguishing backups from storage migration. Overwriting a single destination each time offers no history. Use dated backups, snapshots or managed retention; avoid --delete on the only recovery copy. Keep a protected copy on another host or off-site.
Verify copy and payload integrity
Check rsync’s exit status, the SQL dump and permissions, then run a final --dry-run while writes remain stopped. -a preserves symlinks, timestamps and permissions, but not necessarily ACLs, xattrs or hardlink relationships: evaluate -HAX --numeric-ids and required privileges where applicable. Content hashing is more expensive and should be scheduled.
Generic example to adapt and test, not a ready-to-run production path.
rsync -a --dry-run --itemize-changes \ /srv/opensim/fsassets/ \ /mnt/backup/fsassets/ df -h /mnt/backup df -i /mnt/backup ls -lh /mnt/backup/opensim.sql
Prove a restore before trusting the backup
Restore to an isolated environment: SQL and FSAssets tree from the same recovery set, matching path format and service version. Read known assets, validate inventory and textures, and inspect errors. Never run a restore experiment on production data.
Disk migration: one additional final step
For disk relocation (different from a backup), pre-copying can reduce the final stoppage. After halting writes and verifying synchronization, prepare the mount, change the active path or create a symlink, and test before retiring the original store. See the migration chapter for mount and rollback precautions.