CHAT_OUTPUT_BEGIN
COMMAND_ID=BACKUP-FAILED-UNITS-ROOTCAUSE-213
STATUS=FAIL
RC=2
HOST=pve01
MODE=read-only
COMPONENT=backup-audit
REFERENCE_REGISTER_CHECK=OK
REFERENCE_SHA256=5763f2f8edc75fcf6f3951f8c95896d1cc112ceccb6cf5b64a383f540a4deb8e
ERROR_REGISTER_CHECK=OK
ERROR_REGISTER_SHA256=08b3a769b8e38eaa1acec915352fb328180d70caa7ff6cd9d1ec0da426262752
COMMAND_SHA256=fb59f580fefab96bf42d4c9f720847f498158079695bebebca527795287c0435
DUPLICATE_FAILED_COMMAND_BLOCKED=false
EXECUTION_STARTED=true
CHANGES_MADE=false
SANITIZED=yes
SECRETS_INCLUDED=no
PRIVATE_ADDRESSES_INCLUDED=no
RAW_EVIDENCE_SHA256=5aa2486758a52989308a01aa788b4cd825645ba54a4ca5bc7ac0d46880d624ac
SANITIZED_OUTPUT_SHA256=eb0649d407eb1c9f6ef0ac82df9b958a46ebf6fd91680b20839817454f33891a
OUTPUT_BEGIN
PREVIOUS_COMMAND_ID=BACKUP-OFFHOST-DELTA-212
× homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state
     Loaded: loaded (/etc/systemd/system/homelab-backup-matrix-refresh.service; static)
     Active: failed (Result: exit-code) since Wed 2026-07-22 10:11:15 MSK; 8h ago
 Invocation: 8a8289d574e745078d5316652c3a62cc
   Main PID: 4192756 (code=exited, status=1/FAILURE)
   Mem peak: 12.9M
        CPU: 582ms

Jul 22 10:11:15 pve01 homelab-backup-matrix-refresh[4192756]: MATRIX_STATE_JOBS_TOTAL=41
Jul 22 10:11:15 pve01 homelab-backup-matrix-refresh[4192756]: MATRIX_STATE_JOBS_OK=30
Jul 22 10:11:15 pve01 homelab-backup-matrix-refresh[4192756]: MATRIX_STATE_JOBS_BAD=6
Jul 22 10:11:15 pve01 homelab-backup-matrix-refresh[4192756]: MATRIX_STATE_JOBS_UNMAPPED=0
Jul 22 10:11:15 pve01 homelab-backup-matrix-refresh[4192756]: MATRIX_STATE_JOBS_DISABLED=5
Jul 22 10:11:15 pve01 homelab-backup-matrix-refresh[4192756]: MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
Jul 22 10:11:15 pve01 homelab-backup-matrix-refresh[4192756]: MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
Jul 22 10:11:15 pve01 systemd[1]: homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
Jul 22 10:11:15 pve01 systemd[1]: homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Jul 22 10:11:15 pve01 systemd[1]: Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.

× homelab-vm160-remote-backup.service - VM160 full-image backup streamed from pve02 to pve03
     Loaded: loaded (/etc/systemd/system/homelab-vm160-remote-backup.service; static)
     Active: failed (Result: exit-code) since Tue 2026-07-21 02:39:14 MSK; 1 day 15h ago
 Invocation: 603319cf8a194ed48ec2cca37b86f473
   Main PID: 2028253 (code=exited, status=1/FAILURE)
   Mem peak: 4.2M
        CPU: 97ms

Jul 21 02:39:12 pve01 systemd[1]: Starting homelab-vm160-remote-backup.service - VM160 full-image backup streamed from pve02 to pve03...
Jul 21 02:39:14 pve01 systemd[1]: homelab-vm160-remote-backup.service: Main process exited, code=exited, status=1/FAILURE
Jul 21 02:39:14 pve01 systemd[1]: homelab-vm160-remote-backup.service: Failed with result 'exit-code'.
Jul 21 02:39:14 pve01 systemd[1]: Failed to start homelab-vm160-remote-backup.service - VM160 full-image backup streamed from pve02 to pve03.

× homelab-cluster-admin-pull.service - Pull cluster-admin observer health into pve01
     Loaded: loaded (/etc/systemd/system/homelab-cluster-admin-pull.service; static)
     Active: failed (Result: exit-code) since Tue 2026-07-21 02:37:04 MSK; 1 day 15h ago
 Invocation: 1f07cb8016434bdb92ac64ed740b7f3e
   Main PID: 2014947 (code=exited, status=1/FAILURE)
   Mem peak: 4.6M
        CPU: 64ms

Jul 21 02:36:57 pve01 systemd[1]: Starting homelab-cluster-admin-pull.service - Pull cluster-admin observer health into pve01...
Jul 21 02:37:04 pve01 homelab-cluster-admin-pull[2015088]: STATUS=WARN TS=2026-07-20T23:36:57Z TYPE=cluster-admin-central-pull BAD=1 VMID=180 IP=[PRIVATE_IP] SSH_RC=0 OBSERVER_STATUS=WARN REPORT=/var/lib/homelab-cluster-admin/observer-remote-latest.txt
Jul 21 02:37:04 pve01 homelab-cluster-admin-pull[2015089]: STATUS=WARN TS=2026-07-20T23:36:57Z TYPE=cluster-admin-full-observer-central BAD=1 WARN=0 VMID=180 PULL_RC=0 FULL_STATUS=WARN PROBE_OK=2 PROBE_BAD=2 REPORT=/var/lib/homelab-cluster-admin/full-observer-central-latest.md
Jul 21 02:37:04 pve01 homelab-cluster-admin-pull[2015090]: STATUS=WARN TS=2026-07-20T23:36:57Z TYPE=cluster-admin-restricted-probes BAD=1 VMID=180 MODE=FORCED_COMMAND_READONLY_CURRENT PROBE_OK=2 PROBE_BAD=2 TARGETS=pve01,pve02,pve03,edge SOURCE=full-observer
Jul 21 02:37:04 pve01 systemd[1]: homelab-cluster-admin-pull.service: Main process exited, code=exited, status=1/FAILURE
Jul 21 02:37:04 pve01 systemd[1]: homelab-cluster-admin-pull.service: Failed with result 'exit-code'.
Jul 21 02:37:04 pve01 systemd[1]: Failed to start homelab-cluster-admin-pull.service - Pull cluster-admin observer health into pve01.
Starting homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state...
MATRIX_STATE_STATUS=FAIL
MATRIX_STATE_JOBS_TOTAL=41
MATRIX_STATE_JOBS_OK=30
MATRIX_STATE_JOBS_BAD=6
MATRIX_STATE_JOBS_UNMAPPED=0
MATRIX_STATE_JOBS_DISABLED=5
MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.
Starting homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state...
MATRIX_STATE_STATUS=FAIL
MATRIX_STATE_JOBS_TOTAL=41
MATRIX_STATE_JOBS_OK=30
MATRIX_STATE_JOBS_BAD=6
MATRIX_STATE_JOBS_UNMAPPED=0
MATRIX_STATE_JOBS_DISABLED=5
MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.
Starting homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state...
MATRIX_STATE_STATUS=FAIL
MATRIX_STATE_JOBS_TOTAL=41
MATRIX_STATE_JOBS_OK=30
MATRIX_STATE_JOBS_BAD=6
MATRIX_STATE_JOBS_UNMAPPED=0
MATRIX_STATE_JOBS_DISABLED=5
MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.
Starting homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state...
MATRIX_STATE_STATUS=FAIL
MATRIX_STATE_JOBS_TOTAL=41
MATRIX_STATE_JOBS_OK=30
MATRIX_STATE_JOBS_BAD=6
MATRIX_STATE_JOBS_UNMAPPED=0
MATRIX_STATE_JOBS_DISABLED=5
MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.
Starting homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state...
MATRIX_STATE_STATUS=FAIL
MATRIX_STATE_JOBS_TOTAL=41
MATRIX_STATE_JOBS_OK=30
MATRIX_STATE_JOBS_BAD=6
MATRIX_STATE_JOBS_UNMAPPED=0
MATRIX_STATE_JOBS_DISABLED=5
MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.
Starting homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state...
MATRIX_STATE_STATUS=FAIL
MATRIX_STATE_JOBS_TOTAL=41
MATRIX_STATE_JOBS_OK=30
MATRIX_STATE_JOBS_BAD=6
MATRIX_STATE_JOBS_UNMAPPED=0
MATRIX_STATE_JOBS_DISABLED=5
MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.
Starting homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state...
MATRIX_STATE_STATUS=FAIL
MATRIX_STATE_JOBS_TOTAL=41
MATRIX_STATE_JOBS_OK=30
MATRIX_STATE_JOBS_BAD=6
MATRIX_STATE_JOBS_UNMAPPED=0
MATRIX_STATE_JOBS_DISABLED=5
MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.
Starting homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state...
MATRIX_STATE_STATUS=FAIL
MATRIX_STATE_JOBS_TOTAL=41
MATRIX_STATE_JOBS_OK=30
MATRIX_STATE_JOBS_BAD=6
MATRIX_STATE_JOBS_UNMAPPED=0
MATRIX_STATE_JOBS_DISABLED=5
MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.
Starting homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state...
MATRIX_STATE_STATUS=FAIL
MATRIX_STATE_JOBS_TOTAL=41
MATRIX_STATE_JOBS_OK=30
MATRIX_STATE_JOBS_BAD=6
MATRIX_STATE_JOBS_UNMAPPED=0
MATRIX_STATE_JOBS_DISABLED=5
MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.
Starting homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state...
MATRIX_STATE_STATUS=FAIL
MATRIX_STATE_JOBS_TOTAL=41
MATRIX_STATE_JOBS_OK=30
MATRIX_STATE_JOBS_BAD=6
MATRIX_STATE_JOBS_UNMAPPED=0
MATRIX_STATE_JOBS_DISABLED=5
MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.
Starting homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state...
MATRIX_STATE_STATUS=FAIL
MATRIX_STATE_JOBS_TOTAL=41
MATRIX_STATE_JOBS_OK=30
MATRIX_STATE_JOBS_BAD=6
MATRIX_STATE_JOBS_UNMAPPED=0
MATRIX_STATE_JOBS_DISABLED=5
MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.
Starting homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state...
MATRIX_STATE_STATUS=FAIL
MATRIX_STATE_JOBS_TOTAL=41
MATRIX_STATE_JOBS_OK=30
MATRIX_STATE_JOBS_BAD=6
MATRIX_STATE_JOBS_UNMAPPED=0
MATRIX_STATE_JOBS_DISABLED=5
MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.
Starting homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state...
MATRIX_STATE_STATUS=FAIL
MATRIX_STATE_JOBS_TOTAL=41
MATRIX_STATE_JOBS_OK=30
MATRIX_STATE_JOBS_BAD=6
MATRIX_STATE_JOBS_UNMAPPED=0
MATRIX_STATE_JOBS_DISABLED=5
MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.
Starting homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state...
MATRIX_STATE_STATUS=FAIL
MATRIX_STATE_JOBS_TOTAL=41
MATRIX_STATE_JOBS_OK=30
MATRIX_STATE_JOBS_BAD=6
MATRIX_STATE_JOBS_UNMAPPED=0
MATRIX_STATE_JOBS_DISABLED=5
MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.
Starting homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state...
MATRIX_STATE_STATUS=FAIL
MATRIX_STATE_JOBS_TOTAL=41
MATRIX_STATE_JOBS_OK=30
MATRIX_STATE_JOBS_BAD=6
MATRIX_STATE_JOBS_UNMAPPED=0
MATRIX_STATE_JOBS_DISABLED=5
MATRIX_STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_RENDERED_DIR=/var/lib/homelab-backup-matrix/rendered
homelab-backup-matrix-refresh.service: Main process exited, code=exited, status=1/FAILURE
homelab-backup-matrix-refresh.service: Failed with result 'exit-code'.
Failed to start homelab-backup-matrix-refresh.service - Generate unified homelab backup matrix state.
MATRIX_SYNTAX_CHECK
/usr/local/sbin/homelab-backup-matrix-refresh: line 5: syntax error near unexpected token `('
/usr/local/sbin/homelab-backup-matrix-refresh: line 5: `M=Path("/etc/homelab-backup-matrix/matrix.json")'

OUTPUT_END
CHAT_OUTPUT_END