FSAssets: troubleshooting and logs
Real incidents: exhausted inodes, stuck spool, missing payloads, SQL metadata and permissions.
First rule: read Robust.log
If FSAssets runs within ROBUST, start with Robust.log and its systemd journal if applicable. Find the first [FSASSETS] error preceding the symptom and correlate it with the region's OpenSim.log. FSASSETS startup messages report spool and base directories.
Adapt this example: never run blindly in production.
grep -nEi '\[FSASSETS\]|exception|IOException|database|missing' /chemin/vers/bin/Robust.log | tail -n 150 tail -F /chemin/vers/bin/Robust.log
'No space left on device'
Logs: Robust.log and kernel journal. Possible causes: exhausted bytes, inodes, quota or temporary directory on another partition. Check df -h, df -i on both data and spool, findmnt and quotas. Fix the actual volume after backup; do not delete .gz or spool files.
Adapt this example: never run blindly in production.
df -h /srv/opensim/fsassets/data /srv/opensim/fsassets/tmp df -i /srv/opensim/fsassets/data /srv/opensim/fsassets/tmp findmnt -T /srv/opensim/fsassets/tmp
SQL metadata exists but payload is missing
Logs: [FSASSETS] missing FS, read or decompression failures; code may ask FallbackService if configured. Check the hash in fsassets and derive the path according to UseOsgridFormat, then look for .gz or spool. Fix the path/mount or restore matching payloads; never invent a file under a hash.
Adapt this example: never run blindly in production.
SELECT id,hash,name,type FROM fsassets WHERE id='UUID_REEL_ICI'; # En console FSAssets : show digest UUID_REEL_ICI
Spool stops draining
Logs: [FSASSETS] Writer started, Write cycle complete and exceptions. Verify writer health, read/write permissions, space, inodes and mounted disk. Never delete spool: it may contain the only copies of recent data. Investigate I/O and process state before restarting.
Adapt this example: never run blindly in production.
du -sh /srv/opensim/fsassets/tmp find /srv/opensim/fsassets/tmp -maxdepth 1 -type f | wc -l ls -ld /srv/opensim/fsassets/tmp/spool journalctl -k --since '2 hours ago' | grep -Ei 'I/O error|read-only|ext4|xfs'
SQL errors or missing fsassets table
Logs: [FSASSETS] Can't connect to database, Database open failed, migration or privilege errors. Check [DatabaseService] and possible [AssetService] override, MySQL availability, Realm and actual table. Do not initialize a new empty database as an attempted repair.
Adapt this example: never run blindly in production.
systemctl status mysql --no-pager journalctl -u mysql --since '1 hour ago' --no-pager # SQL (dans le client mysql) : SHOW TABLES LIKE 'fsassets'; SELECT COUNT(*) FROM fsassets;
Permission denied or read-only mount
Logs: UnauthorizedAccessException, IOException, Read-only file system. Check effective service UID, parent directory permissions, ACL, actual mounts, capacity and SELinux/AppArmor if applicable. Fix only the required ownership/mount; chmod -R 777 is not a solution.
Adapt this example: never run blindly in production.
findmnt -T /srv/opensim/fsassets/data namei -l /srv/opensim/fsassets/data ls -ld /srv/opensim/fsassets/data /srv/opensim/fsassets/tmp systemctl status robust.service --no-pager
Migration seems complete but assets remain missing
Logs: [FSASSETS] errors and fallback requests. Check whether rarely accessed UUIDs were imported, Realm, drained spool, legacy backend reachability and file layout. Finish migration under controlled conditions rather than deleting legacy assets.
FSAssets terminates unexpectedly
The fork code has read-error paths that log an exception before terminating the process. Find the first exception in Robust.log and journalctl. Verify disk errors, corrupt gzip data, permissions and SQL availability before attempting a restart.
Adapt this example: never run blindly in production.
journalctl -u robust.service -n 200 --no-pager grep -nEi '\[FSASSETS\]|exception|fatal' /chemin/vers/bin/Robust.log | tail -n 120
Prepare an incident report
Keep timestamps, service identity, relevant UUID/hash, df -h / df -i, findmnt output, full error and surrounding log lines, OpenSimulator version and recent changes. Redact SQL credentials, keys, sensitive IPs and personal details before sharing.