CHAT_OUTPUT_BEGIN
COMMAND_ID=CONTEXT-SOURCES-FULL-002
STATUS=OK
RC=0
HOST=pve01
MODE=read-only
COMPONENT=cluster-context
REFERENCE_REGISTER_CHECK=OK
REFERENCE_SHA256=f6f7749fda946015116cdbbd754c7e7f20356b4a6cf4df330a1b0b8fdf59cbb2
ERROR_REGISTER_CHECK=OK
ERROR_REGISTER_SHA256=ab8308549db90d235b591af0581580c3c7ba1b287b129cc6111c2eef0a0f9ecc
COMMAND_SHA256=0588dda7ed150c5565d42a0450e55d250252999c86b8be948b4528dee3ed32df
DUPLICATE_FAILED_COMMAND_BLOCKED=false
EXECUTION_STARTED=true
CHANGES_MADE=false
SANITIZED=yes
SECRETS_INCLUDED=no
PRIVATE_ADDRESSES_INCLUDED=no
RAW_EVIDENCE_SHA256=621830f58b028a0116a42964d02bdca31c280d6ff358f518d0e8f24c2ba01366
SANITIZED_OUTPUT_SHA256=621830f58b028a0116a42964d02bdca31c280d6ff358f518d0e8f24c2ba01366
OUTPUT_BEGIN
SOURCE_COMMAND_ID=CONTEXT-SOURCES-001
SOURCE_FILE=CONTEXT-SOURCES-001-20260721T061108Z.txt
SOURCE_SHA256=157fd33f90b94edbb9a19b4592a93722442574b3e8dbc5c99ece5de734517ae3
===== CONTEXT SOURCES DISCOVERY =====
TIME=2026-07-21T09:11:08+03:00
HOST=pve01.gram1.ru

===== AUTHORITATIVE FILES =====

----- FILE=/etc/pve/31_HOMELAB_REFERENCE.md -----
SIZE=164140 MTIME=2026-07-14 21:33:38.000000000 +0300 MODE=640 OWNER=root:www-data
SHA256=f6f7749fda946015116cdbbd754c7e7f20356b4a6cf4df330a1b0b8fdf59cbb2
LINES=2799
----- CONTENT BEGIN -----
HOMELAB REFERENCE
GENERATED=2026-06-29T21:45:29+03:00
AUDIT_DIR=/root/cluster-audit-20260629T201245

CORE_CLUSTER_CONFIG

keyboard: en-us
migration: secure,network=[PRIVATE_IP]/24

FINAL_CLASSIFICATION
MISSING_EXPECTED_PREFIX_COUNT=0
STRICT_POSSIBLE_SECRET_COUNT=0
STRICT_SECRET_SCAN_OK
HEALTH_WARN_ERROR_COUNT=14
NOTE exclude_from_reference_or_delete_later=05_edge_compose_safe.tgz
NOTE forum-prod VMID160 is not in homelab-nightly-all; it has separate local/cloud backup chain
NOTE edge-vm disk scsi1 backup=0 risk must be documented
NOTE Corosync remains on [PRIVATE_IP]/12/13; migration network is [PRIVATE_IP]/24 secure
NOTE edge health WARN/ERROR items should be documented as known current states

HEALTH_NOT_OK
HEALTH_NOT_OK item=adguard-rewrite-sync.txt line=STATUS=ERROR TS=20260629T152553Z TYPE=adguard-rewrite-sync REASON=URLError SECRET_PRINTED=REDACTED
HEALTH_NOT_OK item=backup-restore-coverage-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=backup-restore-dashboard.txt line=STATUS=WARN TS=2026-06-29T18:10:09Z ITEMS=80 NOT_OK=13
HEALTH_NOT_OK item=container-image-lifecycle-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=cron-job-monitoring-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=direct-heartbeat-pilot-readiness.txt line=STATUS=WARN
HEALTH_NOT_OK item=healthchecks-heartbeat-coverage.txt line=STATUS=WARN
HEALTH_NOT_OK item=healthchecks-job-coverage.txt line=STATUS=WARN
HEALTH_NOT_OK item=healthchecks-job-monitoring-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=homepage-service-catalog-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=ingress-dns-route-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=job-monitoring-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=observability-health-surface-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=security-vulnerability-slo.txt line=STATUS=WARN

EVIDENCE_FILES
00_cluster_overview.txt 3544 bytes
00_HOMELAB_ASSISTANT_ERROR_REGISTER.md 2566 bytes
01_pve01_audit.txt 28554 bytes
02_pve02_audit.txt 24786 bytes
03_pve03_audit.txt 16842 bytes
04_edge_vm_basic.txt 34506 bytes
05_edge_compose_redacted.txt 24228 bytes
05_edge_compose_safe.tgz.QUARANTINED.txt 354 bytes
05_edge_compose_tar_errors.txt 166 bytes
06_ACCESS_AND_ERROR_RULES.md 13742 bytes
06_edge_npmplus_routes_safe.txt 17350 bytes
07_edge_systemd_backup_audit.txt 20694 bytes
08_edge_scripts_redacted.txt 198622 bytes
09_forum_prod_basic.txt 14413 bytes
10_forum_codevipe_xenforo_audit.txt 3861 bytes
11_forum_backup_audit_redacted.txt 5952 bytes
12_pve02_codevipe_cloud_backup_audit.txt 6277 bytes
13_pve01_systemd_backup_audit.txt 60109 bytes
14_pve01_scripts_redacted.txt 246474 bytes
15_pve01_ct_110_audit.txt 5049 bytes
15_pve01_ct_112_audit.txt 4843 bytes
16_pve02_ct_111_audit.txt 4758 bytes
16_pve02_ct_113_audit.txt 4843 bytes
17_nextcloud_vm_audit.txt 13293 bytes
18_nextcloud_backup_restore_pve01_audit.txt 4269 bytes
19_pve02_host_automation_audit.txt 10642 bytes
19_pve03_host_automation_audit.txt 7632 bytes
20_AUDIT_INDEX_AND_SECRET_SCAN.txt 11935 bytes
21_proxmox_cluster_config_audit.txt 10855 bytes
22_internal_2_5g_network_inventory.txt 8036 bytes
23_cluster_internal_network_configured.txt 747 bytes
24_cluster_internal_network_speedtest.txt 54665 bytes
25_cluster_internal_network_migration_enabled.txt 1816 bytes
26_migration_network_pvesh_verify.txt 1210 bytes
27_migration_network_canonical_verify.txt 956 bytes
28_dns_configs_redacted.txt 15559 bytes
29_health_summary_all_nodes.txt 39051 bytes
30_FINAL_AUDIT_GAPS_WARNINGS_SECRET_CLASSIFICATION.txt 4366 bytes
31_HOMELAB_REFERENCE.md 1805 bytes
datacenter.cfg.before-canonical-migration-20260629T211046 63 bytes
datacenter.cfg.before-migration-network-20260629T210710 16 bytes



РУССКОЕ_РЕЗЮМЕ_КЛАСТЕРА
- Кластер: homelab, 3 узла, quorum OK.
- LAN: pve01=[PRIVATE_IP], pve02=[PRIVATE_IP], pve03=[PRIVATE_IP].
- Внутренняя сеть: pve01-int=[PRIVATE_IP], pve02-int=[PRIVATE_IP], pve03-int=[PRIVATE_IP].
- Миграции Proxmox: migration: secure,network=[PRIVATE_IP]/24.
- Corosync остаётся на [PRIVATE_IP]/12/13; не переносить без отдельного rollback-плана.

РИСКИ_И_ДЕЙСТВИЯ
- VM160 forum-prod не входит в ночной Proxmox backup.
- У VM130 edge-vm есть риск: дополнительный диск backup=0.
- Нужно ротировать ранее засвеченный Cloudflare token.
- Желательно ротировать XenForo SMTP пароль, который был выведен до редактирования.
- Проверить adguard-rewrite-sync: STATUS=ERROR, REASON=URLError.
- Проверить backup-restore-dashboard: STATUS=WARN, NOT_OK=13.

ДОСТУПЫ_КРАТКО
- Proxmox: ssh root@pve01, ssh root@pve02, ssh root@pve03.
- Edge VM: ssh debian@[PRIVATE_IP], использовать sudo, root-login не использовать.
- Nextcloud VM: ssh debian@[PRIVATE_IP].
- Forum-prod: сначала ssh root@pve02, затем ssh -i [SENSITIVE_PATH] root@[PRIVATE_IP].

НАГРУЗКИ_И_СЕРВИСЫ
- CT110 dns1 pve01 [PRIVATE_IP] AdGuard Home.
- CT111 dns2 pve02 [PRIVATE_IP] AdGuard Home.
- CT112 unbound1 pve01 [PRIVATE_IP] Unbound.
- CT113 unbound2 pve02 [PRIVATE_IP] Unbound.
- VM130 edge-vm pve03 [PRIVATE_IP] Docker ingress/app host.
- VM150 Nextcloud pve01 [PRIVATE_IP] Nextcloud AIO.
- VM160 forum-prod pve02 [PRIVATE_IP] CodeVipe/XenForo.

BACKUP_КРАТКО
- Ночной Proxmox backup включает VMID 110,111,112,113,130,150.
- VM160 forum-prod исключён из ночного Proxmox backup и держится на app-level local/cloud backup.
- Edge VM имеет vzdump/offhost/restore evidence; риск backup=0 по дополнительному диску остаётся.
- Nextcloud VM150 имеет Proxmox backup и restore-proof evidence.
- CodeVipe forum-prod имеет локальный backup каждые 6 часов и cloud backup с pve02 каждые 6 часов.

БЕЗОПАСНОСТЬ_КРАТКО
- Финальная строгая проверка audit-файлов: STRICT_POSSIBLE_SECRET_COUNT=0.
- Redacted DNS configs сохранены в 28_dns_configs_redacted.txt.
- Справочник 31 проверен: REFERENCE_ASSIGNMENT_SECRET_HITS=0.

## FINAL_CLOSURE_20260630_CRITICAL_TAILS

- VM160 backup: closed, nightly includes VM160, manual backup ZSTD_OK.
- VM130 backup: closed, scsi1 backup=1, manual backup ZSTD_OK, exact offhost ZSTD_OK, old local backup removed.
- AdGuard rewrite sync: closed, STATUS=OK, timer active/enabled.
- Cloudflare token: live NPMplus token rotated, API verify OK, old exposed token externally confirmed revoked.
- XenForo SMTP: password rotated, JSON valid, STARTTLS SMTP AUTH OK.
- Final audit: unredacted secret strict scan OK.
- Proof index: 85_FINAL_CLOSURE_PROOF.txt and 87_FINAL_CLOSURE_PROOF_WITH_EXTERNAL_CONFIRMATION.txt.

## FINAL_DASHBOARD_RUNTIME_OK_20260630

- Runtime dashboard: STATUS=OK, ITEMS=69, NOT_OK=0.
- systemd failed units: 0 loaded units listed.
- SLO/coverage WARN files: moved out of runtime dashboard into slo-coverage-backlog-index.txt.
- Backlog index: STATUS=OK, BACKLOG_WARN_COUNT=12, RUNTIME_BLOCKER=NO.
- Final proof: 103_FINAL_ALL_GREEN_DASHBOARD_PROOF.txt.
## ROUTER_NETCRAZE_ULTRA_NC1812_20260630

Источник: self-test Netcraze Ultra NC-1812 от 2026-06-30. Секреты Wi-Fi, пароли и токены в справочник не вносятся.

### Паспорт
- Model: Netcraze Ultra.
- Device description: Netcraze Ultra (NC-1812).
- Hostname: Netcraze-9202.
- Workgroup/domain: WORKGROUP.
- Timezone: Europe/Moscow.
- NTP: master.
- NDNS/caption: ndns-domain.
- Локальный домен/сертификат в логах: aleisaevn.netcraze.pro.
- Firmware/NDM из self-test: 5.01.C.0.0-1, built 2026-06-17.
- sharing-config version: 2.06.1.
- Components auto-update: disabled.
- Auto-update channel: draft.
- Auto-update schedule0: start 05:00, stop 06:00.
- EasyConfig: disabled.
- zram: enabled.
- IPv4 forwarding: enabled.
- IPv6 forwarding: enabled.
- conntrack max entries: 32768.
- TCP established timeout: 1200.
- TCP fin timeout: 30.
- TCP keepalive: 120.

### WAN и резервный интернет
- Main WAN: GigabitEthernet1, renamed ISP, description Ростел.
- WAN security-level: public.
- WAN addressing: DHCP.
- WAN MTU: 1500.
- WAN global priority: 700.
- WAN ping-check profile: default.
- Ping-check hosts from log/profile: ya.ru, vk.com, dzen.ru over TLS/443.
- WAN DHCP observed in router log: 95.84.154.183/23, gateway 95.84.154.1.
- ISP DNS observed in router log: 77.37.251.33, 77.37.255.30.
- Backup/mobile WAN: CdcEthernet0, description SIM.
- CdcEthernet0 USB device-id: 12d1 14dc.
- CdcEthernet0 security-level: public.
- CdcEthernet0 addressing: DHCP.
- CdcEthernet0 global priority: 350.
- CdcEthernet0 observed IP: [PRIVATE_IP]/24 via [PRIVATE_IP].

### LAN / VLAN / bridge
- Home bridge: Bridge0 renamed Home.
- Home description: Основная.
- Home security-level: private.
- Home IP: [PRIVATE_IP]/24.
- Home includes: GigabitEthernet0/Vlan1, AccessPoint, AccessPoint_5G, XGigabitEthernet0.
- Proxmox bridge: Bridge1.
- Proxmox bridge IP: [PRIVATE_IP]/24.
- Proxmox security-level: protected.
- Proxmox includes: GigabitEthernet0/Vlan50, XGigabitEthernet0/Vlan50.
- Port 1: GigabitEthernet0/0, access VLAN 50.
- Port 2: GigabitEthernet0/1, access VLAN 1 + trunk VLAN 50.
- Port 3: GigabitEthernet0/2, access VLAN 1 + trunk VLAN 50.
- Port 4: GigabitEthernet0/3, access VLAN 1 + trunk VLAN 50.
- Port 5: XGigabitEthernet0/0, XGigabitEthernet0/Vlan50 enabled.

### Wi-Fi
- SSID: N9202.
- 2.4 GHz: WifiMaster0, AccessPoint.
- 2.4 GHz compatibility: BGN+AX+BE.
- 2.4 GHz channel width: 40-below.
- 5 GHz: WifiMaster1, AccessPoint_5G.
- 5 GHz compatibility: AN+AC+AX+BE.
- 5 GHz channel width: 160.
- Auto channel rescan: 00:00 interval 1 hour.
- Encryption: WPA2 + WPA3.
- WMM: enabled.
- Beamforming: enabled.
- TWT: enabled.
- DL/UL MU-MIMO: enabled.
- DL/UL OFDMA: enabled.
- Spatial reuse: enabled.
- Band steering: disabled.
- Extra AP interfaces: present but down.
- MWS WLAN Home: bands 0 and 1, bind Home, SSID N9202, encryption wpa2+3.

### DHCP
- Main DHCP pool: _WEBADMIN.
- DHCP range: [PRIVATE_IP]-[PRIVATE_IP].
- Default router: [PRIVATE_IP].
- DHCP DNS: [PRIVATE_IP], [PRIVATE_IP].
- Lease: 25200 seconds.
- Bound interface: Home.
- Guest AP pool: _WEBADMIN_GUEST_AP enabled.

### Static DHCP reservations
- [PRIVATE_IP]  -> 00:e0:70:62:98:fb comp01.
- [PRIVATE_IP]  -> 60:3d:61:d4:57:ea Yandex-Ambi-Lamp.
- [PRIVATE_IP]  -> cc:4b:73:58:9b:c0 Yandex.Station.
- [PRIVATE_IP]  -> 38:8a:06:5b:eb:40 Мой смарт.
- [PRIVATE_IP]  -> a0:9d:c1:84:7b:98 Приставка комната.
- [PRIVATE_IP]  -> ac:ba:c0:51:d3:ca Алиса наша комната.
- [PRIVATE_IP]  -> 54:f1:5f:f3:cd:14 Приставка кухня.
- [PRIVATE_IP] -> ac:ba:c0:50:71:94 Алиса кухня.
- [PRIVATE_IP] -> 24:4b:fe:d8:8d:2c Коля смарт ROG-Phone3.
- [PRIVATE_IP] -> 00:2b:70:9a:a3:cf Коля смарт S21-FE.
- [PRIVATE_IP] -> 00:08:22:34:26:fc Коля SPIDER-10 первый.
- [PRIVATE_IP] -> 6c:c7:ec:93:be:dc Света смарт.
- [PRIVATE_IP] -> 00:1e:8f:b0:d5:af Принтер Canon ч/б.
- [PRIVATE_IP] -> d8:43:ae:68:a3:a1 Коля комп.
- [PRIVATE_IP] -> 80:1f:12:69:24:e6 Сигнализация.
- [PRIVATE_IP] -> 00:2b:70:9a:9e:99 Мой смарт S21-FE.
- [PRIVATE_IP] -> 74:eb:80:2e:aa:4e Света планшет Galaxy Tab S4.
- [PRIVATE_IP] -> 84:47:09:6a:cc:ca PVE01.
- [PRIVATE_IP] -> 6c:1f:f7:28:72:4f PVE02.
- [PRIVATE_IP] -> 08:26:ae:3b:73:61 PVE03.
- [PRIVATE_IP] -> bc:24:11:30:d4:0f AdGuard 1.
- [PRIVATE_IP] -> bc:24:11:5b:7e:d8 AdGuard 2.
- [PRIVATE_IP] -> bc:24:11:f4:c5:d8 WireGuard.
- [PRIVATE_IP] -> bc:24:11:e1:f3:3c NPMplus / edge-vm.
- [PRIVATE_IP] -> bc:24:11:e1:9a:25 Nextcloud.

### NAT / port forwarding
- ISP tcp/80 -> bc:24:11:e1:f3:3c, NPMplus.
- ISP tcp/443 -> bc:24:11:e1:f3:3c, NPMplus.
- ISP tcpudp/3478 -> bc:24:11:e1:9a:25, Nextcloud Talk TURN.
- Home tcp/51820 -> bc:24:11:f4:c5:d8, WireGuard vpn1.

### ACL / firewall
- isolate-private enabled.
- _WEBADMIN_Bridge0 permits Home-to-Proxmox access for DNS1/DNS2, AdGuard UI, NPMplus HTTP/HTTPS/UI, Nextcloud AIO/Talk TURN, Proxmox SSH/8006 and ICMP from admin PC.
- _WEBADMIN_GigabitEthernet1 permits admin PC [PRIVATE_IP] outbound TCP 80/443.
- Hotspot policy: Home permit, Bridge1 permit; known hosts explicitly permitted.

### Router management
- HTTP port: 5080.
- HTTPS port: 5083.
- HTTP/HTTPS security-level: private.
- SSH port: 2222.
- SSH security-level: private.
- Telnet port configured: 2323, security-level private; router log shows Telnet disabled later on 2026-06-26.
- Lockout policy for HTTP/Telnet/SSH: 5 15 3.
- Cloud control2 security-level: private.
- Admin tags: cli, http, cifs, printers, opt.
- SFTP denied for admin in log; SSH CLI works.

### Router services
- service dhcp enabled.
- service dns-proxy enabled.
- service igmp-proxy enabled.
- service http enabled.
- service cifs enabled.
- service ssh enabled.
- service ntp enabled.
- DNS proxy rebind protection: auto.
- mDNS reflector: disabled.
- UPnP LAN: Home.
- DLNA interface: Home.
- CIFS share: NTFS-edgtobxf, volume 6F98787822C01361:, automount, permissive.
- USB storage seen in log: Huawei modem/storage, sdb1 swap around 2G, sdb2 NTFS volume NTFS-edgtobxf.

### Router automation warning
- Netcraze SSH CLI is not a normal POSIX shell.
- Logs contain failed commands: while, unset, follow.
- Automation must use router CLI syntax, not bash syntax.
## EXTERNAL_SERVICES_UPS_NETBIRD_MAIL_SCRIPTS_20260630

### UPS / NUT
- UPS hardware discovered on pve01 USB: Cyber Power System CP1500 AVR UPS, USB ID 0764:0501.
- pve01 role: NUT server + monitor.
- pve01 enabled NUT units: nut-driver-enumerator, nut-monitor, nut-server, nut-driver.target, nut.target.
- pve02 role: NUT monitor/client.
- pve03 role: NUT monitor/client.
- edge-vm: no UPS/NUT/APCUPSD integration discovered.
- UPS detailed proof: 107_UPS_DISCOVERY_AND_SETTINGS.txt and 109_UPS_NUT_DETAILED_STATUS_AND_CONFIGS.txt.
- Secrets from /etc/nut configs must remain redacted in audit output.

### NetBird
- NetBird service is active on pve01, pve02, pve03 and edge-vm.
- NetBird version observed: daemon 0.73.2, CLI 0.73.2.
- pve01: FQDN pve01.netbird.selfhosted, IPv4 100.100.131.41/16.
- pve02: FQDN pve02.netbird.selfhosted, IPv4 100.100.7.2/16.
- pve03: FQDN pve03.netbird.selfhosted, IPv4 100.100.34.141/16.
- edge-vm: FQDN edge-vm.netbird.selfhosted, IPv4 100.100.60.182/16.
- Interface: wt0, type Kernel.
- WireGuard port: 51820.
- Management/Signal: Connected.
- Relays: 2/2 available.
- SSH Server through NetBird: Disabled.
- Observed peers count on listed hosts: 6/9 Connected.
- Inventory proof: 108_EXTERNAL_SERVICES_NETBIRD_MAIL_SCRIPTS_INVENTORY.txt.

### Forum mail / SMTP
- forum-prod host: forum-prod.gram1.ru, VM IP [PRIVATE_IP].
- Local MTA services were not discovered by the first inventory command; forum mail is treated as XenForo external SMTP, not local Postfix/Dovecot.
- XenForo SMTP rotation proof: 80_xenforo_smtp_rotation_closed.txt.
- SMTP auth proof: 79_forum_xenforo_smtp_auth_openssl_no_crlf_no_secret.txt.
- Safe current SMTP inventory: 110_XENFORO_MAIL_EXTERNAL_SMTP_SAFE_INVENTORY.txt.
- SMTP password must never be printed; store only host, port, SSL mode, auth flag, username length, password length and AUTH OK proof.

### Scripts / automations
- Full script/unit inventory proof: 108_EXTERNAL_SERVICES_NETBIRD_MAIL_SCRIPTS_INVENTORY.txt.
- edge-vm owns most application health, dashboard, ingress, backup, NetBox, NPMplus, certificate, Trivy and vulnerability jobs.
- pve01 owns many backup/offhost/restore/health/security/NetBird VPS/rclone/sops/scrutiny jobs.
- pve02 owns CodeVipe cloud backup, Vaultwarden isolated restore drill, smartctl textfile and staging/rclone helpers.
- pve03 owns smartctl textfile, cluster internal IP, NetBird and staging/rclone helpers.
- All scripts must be treated as managed operational surface; every future edit needs proof file in the active cluster-audit directory.
## UPS_NUT_DETAILED_CONFIG_20260630

- USB discovery on pve01: Cyber Power System, Inc. CP1500 AVR UPS, USB ID 0764:0501.
- NUT logical UPS name: cyberpower.
- NUT-reported device model: BR1000ELCD.
- NUT manufacturer: CPS.
- Driver: usbhid-ups.
- NUT driver version: 2.8.1.
- CyberPower HID data version: 0.8.
- NUT USB vendorid/productid: 0764/0501.
- NUT server node: pve01.
- NUT server mode: MODE=netserver.
- NUT server listeners: 127.0.0.1:3493 and [PRIVATE_IP]:3493.
- NUT clients: pve02 and pve03 in MODE=netclient.
- pve01 monitor role: MONITOR cyberpower@localhost master.
- pve02 monitor role: MONITOR cyberpower@[PRIVATE_IP] slave.
- pve03 monitor role: MONITOR cyberpower@[PRIVATE_IP] slave.
- Current UPS status at inventory time: OL.
- Battery charge: 100%.
- Battery warning threshold: 20%.
- Battery low threshold: 10%.
- Runtime estimate: 2544 seconds.
- Runtime low threshold: 300 seconds.
- Battery type: PbAcid.
- Battery voltage: 12.9V nominal 12V.
- Input voltage: 221.0V nominal 230V.
- Output voltage: 221.0V.
- UPS load: 11%.
- Nominal real power: 600W.
- Beeper: disabled.
- Shutdown delay: 20 seconds.
- Start delay: 30 seconds.
- Shutdown command on monitored nodes: /sbin/shutdown -h +0.
- NUT timing: POLLFREQ=5, POLLFREQALERT=5, HOSTSYNC=15, DEADTIME=30, FINALDELAY=5.
- Powerdown flag: /etc/killpower.
- Secrets in /etc/nut/upsd.users and upsmon.conf are intentionally redacted in proofs.
- Proof file: 109_UPS_NUT_DETAILED_STATUS_AND_CONFIGS.txt.

## XENFORO_EXTERNAL_SMTP_CURRENT_20260630

- Host: forum-prod.gram1.ru, VM IP [PRIVATE_IP].
- Local MTA units: none discovered in inventory output.
- Mail mode: XenForo external SMTP, not local Postfix/Dovecot.
- SMTP_HOST=mail.pvepro.ru
- SMTP_PORT=587
- SMTP_SSL=false
- SMTP_AUTH=login
- USERNAME_LEN=17
- PASSWORD_LEN=30
- SECRET_PRINTED=NO
- SMTP AUTH proof: 79_forum_xenforo_smtp_auth_openssl_no_crlf_no_secret.txt.
- Rotation closure proof: 80_xenforo_smtp_rotation_closed.txt.
## TIMERS_AND_SCRIPT_HASHES_REFERENCE_20260630

- Full timers and script SHA256 inventory: 114_TIMERS_AND_SCRIPT_HASHES_INVENTORY.txt.
- edge-vm timers cover runtime dashboard, Paperless guard, external canary, health metrics, restore drill index, AdGuard rewrite sync, NPMplus cert expiry, NetBox backup/sync, retention, cluster daily status, vulnerability and Trivy scans, ingress hardening and NPMplus admin bind.
- pve01 timers cover VPN/NetBird health, health metrics, smartctl, disk space, MkDocs refresh, VPS identity audit, storage capacity, quality gate, evidence catalog, backup freshness, docker health, Filebrowser backup/offhost/restore, NPMplus/Kuma backup, NetBird VPS backup/offhost, Authentik/Gitea/Vaultwarden backup, SOPS secret coverage, mail cloud upload/restore, Immich/Memos/Paperless backup/offhost/restore, auto backup, edge-vm vzdump, secret sanity.
- pve02 timers cover smartctl, CodeVipe cloud backup, dpkg-db backup and Vaultwarden isolated restore drill.
- pve03 timers cover smartctl and dpkg-db backup.
- Script hashes were captured for /usr/local/sbin and /usr/local/bin on edge-vm, pve01, pve02 and pve03.
- Future script edits must create a new proof file and refresh 114_TIMERS_AND_SCRIPT_HASHES_INVENTORY.txt.
## PROXMOX_CLUSTER_NODE_STORAGE_REFERENCE_20260630

### Cluster
- Cluster name: homelab.
- Nodes: 3.
- Quorum: OK / Quorate Yes.
- Expected votes: 3.
- Quorum threshold: 2.
- Transport: knet.
- Secure auth: on.
- Corosync membership: [PRIVATE_IP] pve01, [PRIVATE_IP] pve02, [PRIVATE_IP] pve03.
- Proxmox VE: pve-manager 9.2.3 on all three nodes.
- Debian: 13 / trixie on all three nodes.
- Kernel: 7.0.12-1-pve on all three nodes.
- Datacenter migration config: secure, network=[PRIVATE_IP]/24.
- Internal migration/cluster helper IPs: pve01 [PRIVATE_IP], pve02 [PRIVATE_IP], pve03 [PRIVATE_IP].

### Storage policy
- Storage local: dir /var/lib/vz, content iso/vztmpl/backup/import.
- Storage local-lvm: lvmthin pool data, content rootdir/images.
- pve01 local usage at inventory: 32.56%, local-lvm 17.50%.
- pve02 local usage at inventory: 12.86%, local-lvm 11.65%.
- pve03 local usage at inventory: 35.79%, local-lvm 64.84%.
- pve01 staging LV: /mnt/staging, 300G.
- pve02 staging LV: /mnt/staging, 150G.
- pve03 staging LV: /mnt/staging, 200G.
- pve03 local-lvm is the most constrained active VM storage because VM130 has 96G OS disk plus 150G media disk.

### Nightly Proxmox backup job
- Job: homelab-nightly-all.
- Schedule: 03:30.
- Mode: snapshot.
- Compression: zstd.
- Storage: local.
- Enabled: yes.
- Mail notification: failure.
- Included VMIDs: 110,111,112,113,130,150,160.
- Retention: keep-daily=7, keep-weekly=4, keep-monthly=3.

### Node pve01
- FQDN: pve01.gram1.ru.
- LAN IP: [PRIVATE_IP]/24.
- Internal migration IP: [PRIVATE_IP]/24.
- NetBird IP: 100.100.131.41/16.
- Bridge: vmbr0 over nic0.
- CPU: Intel Core i9-12950HX, 24 logical CPUs.
- RAM: 31Gi.
- Disk: Lexar SSD NQ7A1 1TB.
- Main active workloads: CT110 dns1, CT112 unbound1, VM150 nextcloud.

### Node pve02
- FQDN: pve02.gram1.ru.
- LAN IP: [PRIVATE_IP]/24.
- Internal migration IP: [PRIVATE_IP]/24.
- NetBird IP: 100.100.7.2/16.
- Bridge: vmbr0 over nic2.
- CPU: Intel N100, 4 logical CPUs.
- RAM: 15Gi.
- Disk: SK800-1TB.
- Main active workloads: CT111 dns2, CT113 unbound2, VM160 forum-prod.

### Node pve03
- FQDN: pve03.gram1.ru.
- LAN IP: [PRIVATE_IP]/24.
- Internal migration IP: [PRIVATE_IP]/24.
- NetBird IP: 100.100.34.141/16.
- Bridge: vmbr0 over nic1.
- CPU: Intel Core i7-4900MQ, 8 logical CPUs.
- RAM: 31Gi.
- Disk: Samsung SSD 870 EVO 500GB.
- Main active workload: VM130 edge-vm.

### Proof
- Full raw inventory: 120_PROXMOX_CLUSTER_NODE_STORAGE_NETWORK_INVENTORY.txt.
## PROXMOX_VM_CT_INVENTORY_REFERENCE_20260630

### CT110 dns1
- Type: LXC.
- Node: pve01.
- Hostname/name: dns1.
- IP: [PRIVATE_IP]/24.
- MAC: BC:24:11:F9:3C:E1.
- Role: AdGuard Home DNS primary.
- Resources: 1 core, 512MiB RAM, 256MiB swap, 8G rootfs.
- Network: vmbr0, gateway [PRIVATE_IP].
- Nameserver: [PRIVATE_IP].
- Onboot: yes.
- Startup order: 30.
- Unprivileged: yes.
- Backup job: included in homelab-nightly-all.

### CT111 dns2
- Type: LXC.
- Node: pve02.
- Hostname/name: dns2.
- IP: [PRIVATE_IP]/24.
- MAC: BC:24:11:CF:3F:66.
- Role: AdGuard Home DNS secondary.
- Resources: 1 core, 512MiB RAM, 256MiB swap, 8G rootfs.
- Network: vmbr0, gateway [PRIVATE_IP].
- Nameserver: [PRIVATE_IP].
- Onboot: yes.
- Startup order: 30.
- Unprivileged: yes.
- Backup job: included in homelab-nightly-all.

### CT112 unbound1
- Type: LXC.
- Node: pve01.
- Hostname/name: unbound1.
- IP: [PRIVATE_IP]/24.
- MAC: BC:24:11:22:A6:0D.
- Role: Unbound recursive resolver primary.
- Resources: 1 core, 512MiB RAM, 256MiB swap, 8G rootfs.
- Network: vmbr0, gateway [PRIVATE_IP].
- Nameserver: [PRIVATE_IP].
- Onboot: yes.
- Startup order: 20.
- Unprivileged: yes.
- Backup job: included in homelab-nightly-all.

### CT113 unbound2
- Type: LXC.
- Node: pve02.
- Hostname/name: unbound2.
- IP: [PRIVATE_IP]/24.
- MAC: BC:24:11:9E:AF:BE.
- Role: Unbound recursive resolver secondary.
- Resources: 1 core, 512MiB RAM, 256MiB swap, 8G rootfs.
- Network: vmbr0, gateway [PRIVATE_IP].
- Nameserver: [PRIVATE_IP].
- Onboot: yes.
- Startup order: 20.
- Unprivileged: yes.
- Backup job: included in homelab-nightly-all.

### VM130 edge-vm
- Type: QEMU VM.
- Node: pve03.
- Name: edge-vm.
- IP: [PRIVATE_IP]/24.
- MAC: BC:24:11:E1:F3:3C.
- User: debian.
- Role: edge application host / reverse proxy / monitoring / backup automation host.
- Resources: 4 cores, 12GiB RAM.
- Disks: scsi0 96G OS, scsi1 150G media/data.
- scsi1 backup flag: backup=1.
- Network: vmbr0, gateway [PRIVATE_IP].
- Nameservers: [PRIVATE_IP], [PRIVATE_IP].
- QEMU guest agent: enabled.
- Onboot: yes.
- Startup order: 40.
- Backup job: included in homelab-nightly-all.
- Important note: VM130 has exact offhost backup proof after scsi1 backup=1.

### VM150 nextcloud
- Type: QEMU VM.
- Node: pve01.
- Name: nextcloud.
- IP: [PRIVATE_IP]/24.
- MAC: BC:24:11:E1:9A:25.
- User: debian.
- Role: Nextcloud AIO.
- Resources: 4 cores, 8GiB RAM.
- Disk: scsi0 64G.
- Network: vmbr0, gateway [PRIVATE_IP].
- Nameservers: [PRIVATE_IP], [PRIVATE_IP].
- QEMU guest agent: enabled.
- Onboot: yes.
- Startup order: 50.
- Backup job: included in homelab-nightly-all.

### VM160 forum-prod
- Type: QEMU VM.
- Node: pve02.
- Name: forum-prod.
- IP: [PRIVATE_IP]/24.
- MAC: BC:24:11:B6:45:ED.
- User: ops.
- Role: CodeVipe / XenForo production forum.
- Resources: 2 cores, 4GiB RAM.
- Disk: scsi0 80G.
- Network: vmbr0, gateway [PRIVATE_IP].
- Nameserver in VM config: 1.1.1.1.
- QEMU guest agent: enabled.
- Onboot: yes.
- Startup order: 30.
- Backup job: included in homelab-nightly-all.
- Important note: VM160 manual backup proof exists and VM160 is included in nightly job.

### Proof
- Redacted config inventory: 121_PROXMOX_VM_CT_CONFIG_INVENTORY_REDACTED.txt.
## DNS_INGRESS_CERTIFICATES_REFERENCE_20260630

### DNS chain
- Home DHCP gives clients DNS servers: [PRIVATE_IP] and [PRIVATE_IP].
- dns1: CT110, AdGuard Home, IP [PRIVATE_IP].
- dns2: CT111, AdGuard Home, IP [PRIVATE_IP].
- unbound1: CT112, IP [PRIVATE_IP], listens on [PRIVATE_IP]:5335 and 127.0.0.1:5335.
- unbound2: CT113, IP [PRIVATE_IP], listens on [PRIVATE_IP]:5335 and 127.0.0.1:5335.
- dns1 upstream: [PRIVATE_IP]:5335.
- dns2 upstream: [PRIVATE_IP]:5335.
- AdGuard bootstrap DNS: 1.1.1.1 and 9.9.9.9.
- AdGuard filtering_enabled=true, rewrites_enabled=true, protection_enabled=true.
- AdGuard ratelimit=20.
- Unbound access model: localhost allowed, matching AdGuard IP allowed, rest of [PRIVATE_IP]/24 refused.
- Unbound do-ip6: no.
- Unbound private-address includes [PRIVATE_IP]/16, [PRIVATE_IP]/12 and [PRIVATE_IP]/8.

### AdGuard rewrites
- turn.gram1.ru -> [PRIVATE_IP].
- git.gram1.ru -> [PRIVATE_IP].
- dozzle.gram1.ru -> [PRIVATE_IP].
- paper.gram1.ru -> [PRIVATE_IP].
- memos.gram1.ru -> [PRIVATE_IP].
- photos.gram1.ru -> [PRIVATE_IP].
- auth.gram1.ru -> [PRIVATE_IP].
- backup.gram1.ru -> [PRIVATE_IP].
- Rewrite sync health proof: adguard-rewrite-sync STATUS=OK, ORIGIN_COUNT=8, REPLICA_COUNT=8, CHANGED=0.
- Proof files: 127_ADGUARD_UNBOUND_DNS_SAFE_INVENTORY.txt and 130_ADGUARD_UPSTREAM_REWRITES_UNBOUND_RULES_SAFE.txt.

### Router ingress
- Public WAN router forwards TCP/80 and TCP/443 to NPMplus on edge-vm, [PRIVATE_IP].
- Router forwards TCP/UDP 3478 to Nextcloud Talk TURN on [PRIVATE_IP].
- Router forwards Home TCP/51820 to WireGuard host [PRIVATE_IP].
- NPMplus admin listener is bound to localhost on edge-vm, 127.0.0.1:81; public disabled legacy host npm.gram1.ru exists but enabled=0.

### NPMplus runtime
- Host: edge-vm, [PRIVATE_IP].
- Container: npmplus, image zoeyvid/npmplus:2026-06-17-b1, healthy at inventory time.
- Socket proxy: lscr.io/linuxserver/socket-proxy:latest, 127.0.0.1:2375.
- Database: /opt/npmplus/npmplus/database.sqlite.
- DB integrity: ok.
- Public listen ports on edge-vm: 0.0.0.0:80 and 0.0.0.0:443 by nginx/NPMplus.
- NPMplus admin: 127.0.0.1:81.
- Secret rule: Cloudflare DNS API token exists only inside NPMplus certificate metadata and must never be printed.

### Public NPMplus proxy hosts
- nc.gram1.ru -> http://[PRIVATE_IP]:11000, cert=17, ssl_forced=1, enabled=1.
- uptime.gram1.ru -> http://127.0.0.1:3001, cert=35, ssl_forced=1, enabled=1.
- gotify.gram1.ru -> http://127.0.0.1:8082, cert=35, ssl_forced=1, enabled=1.
- vault.gram1.ru -> http://127.0.0.1:8083, cert=21, ssl_forced=1, enabled=1.
- dockge.gram1.ru -> http://127.0.0.1:5001, cert=26, ssl_forced=1, enabled=1.
- home.gram1.ru -> http://127.0.0.1:3000, cert=27, ssl_forced=1, enabled=1.
- git.gram1.ru -> http://127.0.0.1:3002, cert=29, ssl_forced=1, enabled=1.
- dozzle.gram1.ru -> http://127.0.0.1:9999, cert=30, ssl_forced=1, enabled=1.
- paper.gram1.ru -> http://127.0.0.1:8010, cert=30, ssl_forced=1, enabled=1.
- memos.gram1.ru -> http://127.0.0.1:5230, cert=30, ssl_forced=1, enabled=1.
- photos.gram1.ru -> http://127.0.0.1:2283, cert=30, ssl_forced=1, enabled=1.
- auth.gram1.ru -> http://127.0.0.1:9000, cert=30, ssl_forced=1, enabled=1.
- backup.gram1.ru -> http://[PRIVATE_IP]:9101, cert=31, ssl_forced=1, enabled=1.

### VPN NPMplus proxy hosts
- kuma.vpn.gram1.ru -> http://127.0.0.1:3001, cert=32.
- homepage.vpn.gram1.ru -> http://127.0.0.1:3000, cert=32.
- gitea.vpn.gram1.ru -> http://127.0.0.1:3002, cert=32.
- filebrowser.vpn.gram1.ru -> http://127.0.0.1:8085, cert=32.
- auth.vpn.gram1.ru -> http://127.0.0.1:9000, cert=32.
- actual-budget.vpn.gram1.ru -> http://127.0.0.1:5006, cert=32.
- grafana.vpn.gram1.ru -> http://127.0.0.1:3003, cert=32.
- homebox.vpn.gram1.ru -> http://127.0.0.1:7745, cert=32.
- mealie.vpn.gram1.ru -> http://127.0.0.1:9001, cert=32.
- n8n.vpn.gram1.ru -> http://127.0.0.1:5678, cert=32.
- netbox.vpn.gram1.ru -> http://127.0.0.1:8000, cert=32.
- node-red.vpn.gram1.ru -> http://127.0.0.1:1880, cert=32.
- beszel.vpn.gram1.ru -> http://127.0.0.1:8090, cert=32.
- it-tools.vpn.gram1.ru -> http://127.0.0.1:8084, cert=32.
- karakeep.vpn.gram1.ru -> http://127.0.0.1:3030, cert=32.
- linkding.vpn.gram1.ru -> http://127.0.0.1:9091, cert=32.
- minio.vpn.gram1.ru -> http://127.0.0.1:9003, cert=32.
- ntfy.vpn.gram1.ru -> http://127.0.0.1:8055, cert=32.
- searxng.vpn.gram1.ru -> http://127.0.0.1:8888, cert=32.
- syncthing.vpn.gram1.ru -> http://127.0.0.1:8384, cert=32.
- alertmanager.vpn.gram1.ru -> http://127.0.0.1:9093, cert=32.
- prometheus.vpn.gram1.ru -> http://127.0.0.1:9090, cert=32.
- homeassistant.vpn.gram1.ru -> http://127.0.0.1:8123, cert=32.
- npmplus.vpn.gram1.ru -> https://[PRIVATE_IP]:81, cert=32.
- proxmox.vpn.gram1.ru -> https://[PRIVATE_IP]:8006, cert=32.
- dns1.vpn.gram1.ru -> http://[PRIVATE_IP]:8080, cert=32.
- dns2.vpn.gram1.ru -> http://[PRIVATE_IP]:8080, cert=32.
- checks.vpn.gram1.ru -> http://127.0.0.1:8015, cert=32.
- vikunja.vpn.gram1.ru -> http://127.0.0.1:8016, cert=32.
- bookstack.vpn.gram1.ru -> http://127.0.0.1:8017, cert=32.
- pdf.vpn.gram1.ru -> http://127.0.0.1:8018, cert=32.
- jellyfin.vpn.gram1.ru -> http://127.0.0.1:8019, cert=32.
- audiobooks.vpn.gram1.ru -> http://127.0.0.1:8020, cert=32.
- books.vpn.gram1.ru -> http://127.0.0.1:8021, cert=32.

### NPMplus certificates
- cert=17: nc.gram1.ru, expires 2026-09-12 21:58:35.
- cert=21: vault.gram1.ru, expires 2026-09-13 13:43:23.
- cert=26: dockge.gram1.ru, expires 2026-09-13 14:51:59.
- cert=27: home.gram1.ru, expires 2026-09-13 16:43:44.
- cert=29: git.gram1.ru, expires 2026-09-13 17:36:55.
- cert=30: dozzle.gram1.ru, paper.gram1.ru, memos.gram1.ru, photos.gram1.ru, auth.gram1.ru, expires 2026-09-13 23:12:00.
- cert=31: backup.gram1.ru, expires 2026-09-14 05:04:53.
- cert=32: *.vpn.gram1.ru and vpn.gram1.ru, expires 2026-09-15 23:18:29.
- cert=35: uptime.gram1.ru and gotify.gram1.ru, expires 2026-09-19 21:21:59.
- Deleted/old cert rows observed: cert=1 nc.gram1.ru, cert=18 vault.gram1.ru.

### Proof
- NPMplus schema/listen inventory: 126_DNS_INGRESS_CERT_NPMPLUS_SAFE_INVENTORY.txt.
- NPMplus exact proxy/cert rows: 129_NPMPLUS_PROXY_CERT_STREAM_ROWS_SAFE.txt.
- AdGuard/Unbound inventory: 127_ADGUARD_UNBOUND_DNS_SAFE_INVENTORY.txt.
- AdGuard upstream/rewrites and Unbound rules: 130_ADGUARD_UPSTREAM_REWRITES_UNBOUND_RULES_SAFE.txt.
## EDGE_DOCKER_STACKS_REFERENCE_20260630

### Runtime
- Host: edge-vm, IP [PRIVATE_IP].
- Docker Server version: 29.6.0.
- Docker Compose version: v5.1.4.
- Primary stack root: /opt/stacks.
- Additional compose roots: /opt/npmplus-compose, /opt/gotify-compose, /opt/uptime-kuma-compose, /opt/vaultwarden-compose, /opt/dockge-compose.
- Public ingress terminates through NPMplus on ports 80/443.
- Most app containers expose only 127.0.0.1 ports and are published through NPMplus.
- NPMplus uses host networking.
- socket-proxy exposes Docker API read-oriented surface on 127.0.0.1:2375 and socket-proxy_net.
- Secret values are not stored in the reference. Only .env/secret file paths are inventoried.

### Compose projects observed
- actual-budget.
- audiobookshelf.
- authentik.
- beszel.
- blackbox-exporter.
- bookstack.
- cadvisor.
- calibre-web.
- crowdsec.
- diun.
- dockge-compose.
- dozzle.
- filebrowser.
- gitea.
- gotify-compose.
- healthchecks.
- homeassistant.
- homebox.
- homepage.
- immich.
- it-tools.
- jellyfin.
- karakeep.
- kopia.
- linkding.
- loki-alloy.
- mealie.
- memos.
- minio.
- n8n.
- netbox.
- node-red.
- npmplus.
- ntfy.
- observability-lite.
- paperless.
- searxng.
- socket-proxy.
- stirling-pdf.
- syncthing.
- uptime-kuma-compose.
- vaultwarden-compose.
- vikunja.

### Critical app groups
- Ingress/security: npmplus, socket-proxy, crowdsec.
- Identity/secrets: authentik, vaultwarden.
- Monitoring/alerting: uptime-kuma, gotify, healthchecks, observability-lite, loki-alloy, blackbox-exporter, cadvisor, node-exporter, dozzle, diun.
- Data apps: immich, paperless, memos, gitea, netbox, actual-budget, homebox, mealie, n8n, node-red, vikunja, bookstack.
- Files/media: nextcloud external VM, immich, filebrowser, syncthing, jellyfin, audiobookshelf, calibre-web, stirling-pdf, minio.
- Admin/dashboard: homepage, dockge, netbox, dozzle, NPMplus VPN admin.

### Notable local ports
- Homepage: 127.0.0.1:3000.
- Uptime Kuma: 127.0.0.1:3001.
- Gitea: 127.0.0.1:3002.
- Grafana: 127.0.0.1:3003.
- Actual Budget: 127.0.0.1:5006.
- n8n: 127.0.0.1:5678.
- Paperless: 127.0.0.1:8010.
- Healthchecks: 127.0.0.1:8015.
- Vikunja: 127.0.0.1:8016.
- BookStack: 127.0.0.1:8017.
- Stirling PDF: 127.0.0.1:8018.
- Jellyfin: 127.0.0.1:8019.
- Audiobookshelf: 127.0.0.1:8020.
- Calibre-Web: 127.0.0.1:8021.
- ntfy: 127.0.0.1:8055.
- Gotify: 127.0.0.1:8082.
- Vaultwarden: 127.0.0.1:8083.
- IT-Tools: 127.0.0.1:8084.
- Filebrowser: 127.0.0.1:8085.
- Beszel: 127.0.0.1:8090.
- Prometheus: 127.0.0.1:9090.
- Linkding: 127.0.0.1:9091.
- Alertmanager: 127.0.0.1:9093.
- Node exporter: 127.0.0.1:9100.
- Blackbox exporter: 127.0.0.1:9115.
- Syncthing UI: 127.0.0.1:8384.
- Loki: 127.0.0.1:3100.
- Alloy UI: 127.0.0.1:12345.
- Home Assistant: host network, 0.0.0.0:8123.
- NPMplus admin: 127.0.0.1:81.
- NPMplus public ingress: 0.0.0.0:80 and 0.0.0.0:443.

### Secret/env inventory policy
- Full values from .env, secrets.yaml, secretkey.txt and token files must never be copied into reference.
- Reference may list paths only.
- Full path inventory is in 133_EDGE_DOCKER_STACKS_SAFE_INVENTORY.txt.
- Normalized runtime map is in 134_EDGE_DOCKER_CONTAINER_NORMALIZED_MAP.txt.

### Proof
- Raw Docker/stacks inventory: 133_EDGE_DOCKER_STACKS_SAFE_INVENTORY.txt.
- Normalized container map: 134_EDGE_DOCKER_CONTAINER_NORMALIZED_MAP.txt.
## BACKUP_RESTORE_OFFHOST_CATALOG_REFERENCE_20260630

### Global backup model
- Primary Proxmox backup job: homelab-nightly-all.
- Schedule: daily 03:30.
- Mode: snapshot.
- Compression: zstd.
- Storage: local.
- Retention: keep-daily=7, keep-weekly=4, keep-monthly=3.
- Included VMIDs: 110,111,112,113,130,150,160.
- Backup health dashboard: STATUS=OK, ITEMS=69, NOT_OK=0 at 2026-06-30T10:10:06Z.
- Restore drill index: STATUS=OK, RESTORE_PROOFS=16/16.
- SLO coverage is tracked separately from runtime dashboard. It had WARN backlog in backup-restore-coverage-slo with APP_TOTAL=43, GREEN=3, YELLOW=21, RED=19, MISSING_BACKUP_SIGNAL=31, MISSING_RESTORE_SIGNAL=28, MISSING_TIMER=31.
- Runtime OK means operational backup/restore checks are green; it does not mean every P1/P2 application has full backup+restore automation coverage.

### Proxmox vzdump catalog
- CT110 dns1: included in homelab-nightly-all, local backup on pve01.
- CT111 dns2: included in homelab-nightly-all, local backup on pve02.
- CT112 unbound1: included in homelab-nightly-all, local backup on pve01.
- CT113 unbound2: included in homelab-nightly-all, local backup on pve02.
- VM130 edge-vm: included in homelab-nightly-all, local backup on pve03.
- VM150 nextcloud: included in homelab-nightly-all, local backup on pve01.
- VM160 forum-prod: included in homelab-nightly-all, local backup on pve02.
- VM130 also has dedicated edge-vm-vzdump backup/offhost/restore health proofs.
- VM160 has manual backup proof and nightly job inclusion proof.
- VM130 scsi1 backup flag is enabled after correction: scsi1 backup=1.

### Edge VM / VM130 full-image protection
- edge-vm-vzdump-backup: STATUS=OK, archive size about 28.2G, SHA256 recorded.
- edge-vm-vzdump-offhost: STATUS=OK, destination pve02 /mnt/staging/offhost/edge-vm-vzdump-from-pve03.
- edge-vm-vzdump-restore: STATUS=OK, zstd and vma verification OK.
- mail-cloud-edge-vm: STATUS=OK, recurring chunked upload, 53 parts, download verification enabled.
- Retention for edge-vm cloud upload: RETENTION_KEEP=4.
- Edge restore proof offhost: STATUS=OK, ITEMS=14, OK=14.

### Mail/cloud critical backups
- mail-cloud-critical: STATUS=OK, recurring P0 critical baseline, remote OK 2/2.
- Latest observed critical bundle: p0-critical-baseline-20260630T034644Z.tar.gz.
- Critical bundle size at inventory: 1087208837 bytes.
- Critical retention: RETENTION_KEEP=14.
- mail-cloud-restore-drill: STATUS=OK, CRITICAL_OK=2, EDGE_OK=1.
- vaultwarden isolated restore drill on pve02: STATUS=OK, uses cloud critical baseline plus pve02 offhost component and vaultwarden archive.

### NPMplus / Kuma / ingress config backups
- npmplus-kuma-config-backup: STATUS=OK.
- Archive: /mnt/staging/npmplus-kuma-config-backups/snapshots/npmplus-kuma-config-*.tar.gz.
- NPMplus DB integrity: OK.
- Kuma DB integrity: OK.
- NPM proxy count in health proof: 47.
- Required VPN routes in health proof: 18.
- Wildcard VPN cert check: NPM_CERT32_WILDCARD=1.
- npmplus-kuma-config-offhost: STATUS=OK to pve02.
- npmplus-kuma-config-restore: STATUS=OK with DB integrity checks.
- npmplus restore proof also exists from app backup quality checks.

### DNS / AdGuard / Unbound protection
- AdGuard rewrite sync: STATUS=OK, ORIGIN_COUNT=8, REPLICA_COUNT=8, CHANGED=0.
- dns1/dns2/unbound1/unbound2 are protected by Proxmox vzdump.
- AdGuard rewrite sync plan stored under /var/lib/homelab-private/adguard-rewrite-sync.
- Router self-test config is stored as uploaded/generated reference evidence, not yet automated as recurring router backup.

### Auth / secrets / identity apps
- Vaultwarden backup: STATUS=OK, archive under /mnt/staging/vaultwarden-backups/snapshots.
- Vaultwarden offhost: STATUS=OK to pve02.
- Vaultwarden restore: STATUS=OK, DB integrity OK, tables counted.
- Vaultwarden isolated restore drill: STATUS=OK on pve02.
- Authentik backup: STATUS=OK, archive under /mnt/staging/authentik-backups/snapshots.
- Authentik offhost: STATUS=OK to pve02.
- Authentik restore: STATUS=OK, tables/users/applications/proxy providers checked.
- Secret values must remain out of all reference files; only archive paths, sizes and SHA256 are recorded.

### Git / documentation / inventory apps
- Gitea backup: STATUS=OK, archive under /mnt/staging/gitea-backups/snapshots.
- Gitea offhost: STATUS=OK to pve02.
- Gitea restore: STATUS=OK, DB integrity OK, tables counted.
- NetBox backup: STATUS=OK, PostgreSQL dump under /var/lib/homelab-private/pre-update-backups/netbox-*.
- NetBox offhost: STATUS=OK to pve02.
- NetBox restore dry-run: STATUS=OK, restore container checked.
- Latest NetBox health proof observed: devices=7, sites=1, prefixes=3, IP addresses=12, virtual machines=49.

### Document / notes / personal data apps
- Paperless backup: STATUS=OK, latest snapshot under /mnt/staging/paperless-backups/snapshots.
- Paperless offhost: STATUS=OK to pve02.
- Paperless restore dry-run: STATUS=OK, pg_restore list and restore tables checked.
- Memos backup: STATUS=OK, latest snapshot under /mnt/staging/memos-backups/snapshots.
- Memos offhost: STATUS=OK to pve02.
- Memos restore proof exists from app restore quality checks.
- Actual Budget backup: STATUS=OK, offhost SHA256/listing OK, restore not attempted in observed latest proof.
- Linkding backup: STATUS=OK, offhost SHA256/listing OK, restore not attempted in observed latest proof.
- Karakeep backup: STATUS=OK, offhost SHA256/listing OK, restore not attempted in observed latest proof.
- Mealie backup: STATUS=OK, offhost SHA256/listing OK, restore not attempted in observed latest proof.
- n8n backup: STATUS=OK, offhost SHA256/listing OK, restore not attempted in observed latest proof.

### Files / media / sync apps
- Immich media backup: STATUS=OK, latest snapshot under /mnt/staging/immich-media-backups/snapshots.
- Immich media offhost: STATUS=OK to pve02.
- Immich media restore: STATUS=OK, local/offhost manifest match.
- Immich full consistency backup: STATUS=OK, includes PostgreSQL dump and media archive, offhost path present, restore dry-run OK.
- Nextcloud restore proof: STATUS=OK, OCC OK, DB_DUMP OK, DATA_MANIFEST OK, restore to production NO.
- Nextcloud restore proof is copied offhost.
- Filebrowser backup: STATUS=OK.
- Filebrowser offhost: STATUS=OK.
- Filebrowser restore validation: STATUS=OK.
- Jellyfin restore: STATUS=OK from app backup quality checks.
- Audiobookshelf restore: STATUS=OK from app backup quality checks.
- Calibre-Web restore: STATUS=OK from app backup quality checks.
- Syncthing is inventoried in Docker/stacks; dedicated backup signal not confirmed in this catalog.

### Home/admin/utility apps
- Home Assistant backup: STATUS=OK, archive includes configuration.yaml, storage dir and DB file, offhost SHA256/listing OK, restore not attempted in observed proof.
- Healthchecks restore: STATUS=OK, SQLite integrity OK and temp restore verify OK.
- BookStack restore: STATUS=OK, DB dump OK.
- Vikunja restore: STATUS=OK, SQLite integrity OK and temp restore verify OK.
- Kopia restore: STATUS=OK, synthetic snapshot and restore compare OK.
- Observability config backup: STATUS=OK, includes prometheus.yml, alertmanager.yml and health rules, offhost SHA256/listing OK.
- Ntfy is covered as part of observability/alerting stack but dedicated service backup signal was not confirmed in this catalog.
- MinIO is included in app-stacks-batch backup from 20260617; current dedicated timer was not confirmed.
- Homebox, Node-RED, MinIO, Mealie, Linkding, n8n and Actual Budget were part of app-stacks batch backup proof from 20260617.

### External service backups
- NetBird VPS backup: STATUS=OK, snapshot under /mnt/staging/netbird-vps-backups/snapshots.
- NetBird VPS offhost: STATUS=OK to pve02.
- NetBird VPS restore validation: STATUS=OK, archive SHA256 recorded.
- CodeVipe/forum-prod: VM160 Proxmox backup is present; pve02 also has codevipe-cloud-backup.timer.
- XenForo SMTP rotation has DB backup proof from earlier rotation work; production forum application-level backup catalog should be expanded separately with CodeVipe-specific DB/files/cron/public cutover details.

### Retention and cleanup
- Retention dry-run: STATUS=OK, production changed NO, destructive actions NO.
- Disk retention policy: STATUS=OK, root used pct observed 70 on edge-vm, removed dirs 0, Docker volume prune NO.
- App backup retention dry-run timer exists on edge-vm.
- homelab-backup-freshness timer exists on pve01.
- rollback-image-retention health file exists but had permission issue during one raw read; it must be read with sudo in future collection commands.

### Known coverage gaps / backlog
- backup-restore-coverage-slo remains a coverage backlog, not a runtime blocker.
- Actual Budget latest proof says RESTORE_ATTEMPTED=NO.
- Home Assistant latest proof says RESTORE_ATTEMPTED=NO.
- Linkding latest proof says RESTORE_ATTEMPTED=NO.
- Karakeep latest proof says RESTORE_ATTEMPTED=NO.
- Mealie latest proof says RESTORE_ATTEMPTED=NO.
- n8n latest proof says RESTORE_ATTEMPTED=NO.
- Observability config latest proof says RESTORE_ATTEMPTED=NO.
- Router config backup is currently reference/self-test based, not confirmed as recurring automated backup.
- Syncthing, ntfy, MinIO and several P2 services need either dedicated backup/restore proof or explicit classification as disposable/rebuildable.

### Proof
- Raw backup/offhost/restore inventory: 138_BACKUP_RESTORE_OFFHOST_CATALOG_RAW_INVENTORY.txt.
- Backup timer summary draft: 139_BACKUP_CATALOG_SERVICE_SUMMARY_DRAFT.txt.
- Normalized backup health index: 140_BACKUP_HEALTH_FILES_NORMALIZED_INDEX.txt.
## MONITORING_ALERTING_HEALTH_REFERENCE_20260630

### Monitoring stack
- Primary monitoring host: edge-vm, [PRIVATE_IP].
- Prometheus container: prometheus, local port 127.0.0.1:9090.
- Grafana container: grafana, local port 127.0.0.1:3003, VPN route grafana.vpn.gram1.ru.
- Alertmanager container: alertmanager, local port 127.0.0.1:9093.
- Loki container: loki, local port 127.0.0.1:3100.
- Alloy container: alloy, local port 127.0.0.1:12345.
- Uptime Kuma container: uptime-kuma, local port 127.0.0.1:3001, public route uptime.gram1.ru and VPN route kuma.vpn.gram1.ru.
- Gotify container: gotify, local port 127.0.0.1:8082, public route gotify.gram1.ru.
- ntfy container: ntfy, local port 127.0.0.1:8055, VPN route ntfy.vpn.gram1.ru.
- Healthchecks container: healthchecks, local port 127.0.0.1:8015, VPN route checks.vpn.gram1.ru.
- Blackbox exporter: blackbox-exporter, local port 127.0.0.1:9115.
- cAdvisor: cadvisor, local port 127.0.0.1:8081.
- Node exporter on edge-vm: node-exporter, local port 127.0.0.1:9100.
- Beszel: beszel, local port 127.0.0.1:8090.
- Dozzle: dozzle, local port 127.0.0.1:9999.

### PVE monitoring endpoints
- pve01: prometheus-node-exporter active on :9100, homelab-health-http active on :9101, Proxmox pveproxy on :8006.
- pve02: prometheus-node-exporter active on :9100, homelab-health-http active on :9101, Proxmox pveproxy on :8006.
- pve03: prometheus-node-exporter active on :9100, Proxmox pveproxy on :8006.
- pve03 has node-exporter/SMART timers but no homelab-health-http :9101 endpoint in observed output.
- PVE smartctl textfile timers exist on pve01/pve02/pve03.
- pve01 also runs private VPN host health, NetBird peer health, disk space health, MkDocs refresh, evidence catalog and quality gate timers.

### Prometheus config
- Prometheus scrape interval: 30s.
- Prometheus evaluation interval: 30s.
- Rule files path: /etc/prometheus/rules/*.yml.
- Alertmanager target: alertmanager:9093.
- Blackbox HTTP job targets:
  - https://nc.gram1.ru
  - https://git.gram1.ru
  - https://auth.gram1.ru
  - https://paper.gram1.ru
  - https://backup.gram1.ru
- Edge node exporter scrape target: node-exporter:9100.
- PVE node exporter targets: [PRIVATE_IP]:9100, [PRIVATE_IP]:9100, [PRIVATE_IP]:9100.
- Proxmox exporter job targets pve01/pve02/pve03 through exporter endpoint [PRIVATE_IP]:9221.
- cAdvisor scrape target: cadvisor:8080.

### Alertmanager / notification routing
- Alertmanager route receiver: ntfy.
- Alertmanager webhook target: http://ntfy/homelab-alerts.
- Alert group_by: alertname, instance, severity.
- group_wait: 10s.
- group_interval: 5m.
- repeat_interval: 4h.
- Alert routing proof: alert-routing.txt STATUS=OK, RULES_ADDED=6, PROMETHEUS_RELOAD=OK, TEST_NOTIFICATION_SENT=YES, NTFY_MESSAGES_PUBLISHED_DELTA=1.
- alert-routing-health.txt is the standardized status alias and is STATUS=OK.

### Core Prometheus alert rules
- HomelabPrometheusTargetDown: up == 0 for 5m, severity warning.
- HomelabSmartDiskHealthFailed: homelab_smartctl_health_passed == 0 for 10m, severity critical.
- HomelabRootFilesystemLowSpace: root filesystem available below 15% for 15m, severity warning.
- HomelabPrometheusConfigReloadFailed: prometheus_config_last_reload_successful == 0 for 5m, severity critical.
- HomelabP0HealthStatusNotOk: homelab_health_status tier P0 equals 0 for 15m.
- HomelabP0BackupRestoreHealthStale: P0 backup/offhost/restore health older than 36h.
- HomelabP0WeeklyEdgeVmVzdumpHealthStale: weekly edge-vm vzdump health older than 8d.
- HomelabP1BackupHealthStale: P1 backup health older than 7d.
- HomelabMailCloudRestoreDrillStale: mail-cloud restore drill older than 8d.
- HomelabMkdocsRuntimePublishStale: MkDocs runtime publish older than 30d.
- HomelabHealthMetricsCollectorMissing: homelab health metrics absent from Prometheus.
- HomelabExternalCanaryHealthNotOkOrStale: external canary not OK or older than 30m.
- HomelabBlackboxProbeFailed / Missing: blackbox probe failed or absent.
- HomelabVulnScanHealthNotOkOrStale: vulnerability scan failed or older than 8d.
- HomelabNpmplusCertExpiryHealthNotOkOrStale: NPMplus cert expiry health failed or older than 48h.
- HomelabBlackboxTlsCertificateExpiringSoon: TLS certificate expiry within 14 days.

### Loki / Alloy log pipeline
- Loki version observed: grafana/loki:3.7.2.
- Alloy version observed: grafana/alloy:v1.17.0.
- Loki auth_enabled: false.
- Loki storage: local filesystem under /loki.
- Loki schema: tsdb v13.
- Loki retention_period: 14d.
- Alloy discovers Docker through socket-proxy at tcp://socket-proxy:2375.
- Alloy relabels container, compose_project, compose_service and host=edge-vm.
- Alloy forwards Docker logs to http://loki:3100/loki/api/v1/push.
- Loki readiness observed as "ready".

### Runtime dashboard semantics
- backup-restore-dashboard.txt is an authoritative runtime dashboard file.
- Current observed runtime dashboard: STATUS=OK, ITEMS=69, NOT_OK=0.
- backup-restore-dashboard.json confirmed not_ok=[].
- restore-drill-index.txt observed: STATUS=OK, RESTORE_PROOFS=16/16, MISSING empty.
- cluster-daily-status.txt observed: STATUS=OK, PROBLEM_COUNT=0.
- external-canary.txt observed: STATUS=OK, BAD=0.
- external canary checks include nc.gram1.ru, git.gram1.ru, auth.gram1.ru, backup.gram1.ru.
- Alertmanager local API is reachable.
- Gotify local /health returns green.
- ntfy local /v1/health returns healthy true.
- Uptime Kuma local endpoint returns HTTP 302 to /dashboard, which is expected for UI auth/redirect.
- Healthchecks local status endpoint returns HTTP 301, which is treated as reachable.

### Certificate / vulnerability health
- npmplus-cert-expiry-health.txt is the standardized health alias for certificate expiry.
- npmplus-certificate-expiry.txt is the detailed cert expiry file.
- Detailed cert expiry file observed cert_count=9 and min_cert_days_left=75 with warn threshold 30.
- cluster-daily-status local extra check reports npmplus-cert-expiry status OK, problems=0, min_days=75.
- trivy-scan.txt observed: STATUS=OK, IMAGE_TOTAL=58, IMAGE_SCAN_OK=58, IMAGE_SCAN_ERROR=0, CRITICAL_TOTAL=112, HIGH_TOTAL=1650.
- vuln-scan.txt observed: STATUS=OK, MODE=full, IMAGES=58, TRIVY_OK=49, CRITICAL_MATCHES=203.

### SLO backlog semantics
- Runtime dashboard OK does not mean SLO backlog is closed.
- slo-coverage-backlog-index.txt observed: STATUS=OK, BACKLOG_WARN_COUNT=12, RUNTIME_BLOCKER=NO.
- backup-restore-coverage-slo.txt observed: STATUS=WARN, APP_TOTAL=43, GREEN_COUNT=3, YELLOW_COUNT=21, RED_COUNT=19, MISSING_BACKUP_SIGNAL=31, MISSING_RESTORE_SIGNAL=28, MISSING_TIMER=31, MISSING_EXTERNAL_VALIDATION=43.
- Other observed WARN backlog files include container-image-lifecycle-slo, cron-job-monitoring-slo, direct-heartbeat-pilot-readiness, healthchecks-heartbeat-coverage, healthchecks-job-coverage, healthchecks-job-monitoring-slo, homepage-service-catalog-slo, ingress-dns-route-slo, observability-health-surface-slo and security-vulnerability-slo.
- SLO WARN files are tracked separately and are not runtime blockers unless explicitly promoted to P0 runtime checks.

### Important file locations
- Main health dir on edge-vm: /var/lib/homelab-health.
- Prometheus config: /opt/stacks/observability-lite/prometheus/prometheus.yml.
- Prometheus rules: /opt/stacks/observability-lite/prometheus/rules/.
- Alertmanager config: /opt/stacks/observability-lite/alertmanager/alertmanager.yml.
- Loki config: /opt/stacks/loki-alloy/loki-config.yaml.
- Alloy config: /opt/stacks/loki-alloy/config.alloy.
- Blackbox exporter config: /opt/stacks/blackbox-exporter/blackbox.yml.
- Uptime Kuma DB: /opt/uptime-kuma-compose/data/kuma.db.
- Gotify DB: /opt/gotify-compose/data/gotify.db.
- ntfy DB/cache: /opt/stacks/ntfy/data/user.db and /opt/stacks/ntfy/cache/cache.db.
- Healthchecks settings/data: /opt/stacks/healthchecks/local_settings.py and /var/lib/healthchecks.

### Known caveats
- A previous compact check looked for npmplus-cert-expiry.txt; use npmplus-certificate-expiry.txt for detailed cert expiry and npmplus-cert-expiry-health.txt for standardized health.
- health-metrics.txt is not required to exist as a health file; health metrics may be emitted via node-exporter textfile collector.
- Prometheus targets must be checked with explicit HTTP code and parsed JSON output, not a silent curl pipeline.

### Proof
- Raw monitoring inventory: 143_MONITORING_ALERTING_HEALTH_RAW_INVENTORY.txt.
- Runtime/SLO/alerting status: 144_MONITORING_RUNTIME_SLO_ALERTING_STATUS.txt.
- Kuma/Gotify/ntfy/Healthchecks status: 145_KUMA_GOTIFY_NTFY_HEALTHCHECKS_SAFE_STATUS.txt.
- Corrected monitoring compact status: 146_MONITORING_COMPACT_STATUS_CORRECTED.txt.

## PROMETHEUS_STATUS_CORRECTION_20260630

- Initial monitoring compact check after reference creation reported Prometheus API HTTP_CODE=000 on 127.0.0.1:9090.
- This was recorded as error-register item 30 and checked with proof files 149_PROMETHEUS_DIAG_AFTER_MONITORING_REFERENCE.txt, 150_PROMETHEUS_CONDITIONAL_RECOVERY.txt and 151_PROMETHEUS_REFERENCE_CORRECTION_PROOF.txt.
- The Monitoring/Alerting section must be interpreted together with the latest Prometheus correction proof, not only with the earlier secret-scan proof.

## COMMAND_PREFLIGHT_RULE_20260630

- Strict operator rule: before every command, check both the error register and this reference file.
- Required visible markers before main action: ERROR_REGISTER_CHECK=OK and REFERENCE_CHECK=OK.
- If either check fails, do not run the main action.
- Do not use exit 1 in interactive SSH sessions.
- Proof: 152_COMMAND_PREFLIGHT_RULE_RECORDED.txt.

## PROMETHEUS_TARGETS_SETTLED_20260630

- Prometheus was absent after the initial monitoring reference proof and was restarted in 150_PROMETHEUS_CONDITIONAL_RECOVERY.txt.
- Prometheus readiness proof exists in 151_PROMETHEUS_REFERENCE_CORRECTION_PROOF.txt.
- Target health was rechecked with an additional settled scrape proof in 153_PROMETHEUS_TARGETS_SETTLED_PROOF.txt.
- Interpret Prometheus status from the latest settled proof, not from the earlier immediate-after-start unknown target state.
## PROMETHEUS_TARGETS_SETTLED_PROOF_CORRECTION_20260630

- 153_PROMETHEUS_TARGETS_SETTLED_PROOF.txt is invalid and must not be used as evidence.
- Reason: nested Python inside SSH lost quoting and produced SyntaxError.
- Error-register item 32 records this mistake.
- Valid replacement proof: 154_PROMETHEUS_TARGETS_SETTLED_PROOF_NO_PYTHON.txt.
- Prometheus status must be interpreted from 154 and later proofs, not from 153.

## MONITORING_PROMETHEUS_FINAL_CLOSED_20260630

- Prometheus was initially absent after monitoring reference creation.
- It was restarted and then verified after scrape settling.
- Valid settled proof: 154_PROMETHEUS_TARGETS_SETTLED_PROOF_NO_PYTHON.txt.
- Final observed state: HTTP_CODE=200, PROM_TARGETS_TOTAL=14, PROM_UP_COUNT=14, PROM_NON_UP_COUNT=0.
- Correction/secret scan proof: 156_PROMETHEUS_CORRECTION_PROOF_AND_SECRET_SCAN.txt.
- Invalid proof 153 must not be used.
## SECURITY_ACCESS_SECRETS_OPERATING_MODEL_20260630

### Command safety
- Before every command, check /etc/pve/HOMELAB_ASSISTANT_ERROR_REGISTER.md and current 31_HOMELAB_REFERENCE.md.
- Print ERROR_REGISTER_CHECK=OK and REFERENCE_CHECK=OK before the main action.
- Do not run the main action if either check fails.
- Do not use exit 1 in interactive SSH sessions.
- Avoid long nested SSH/Python/PHP/SQL quoting chains.
- Do not print secrets.

### Access model
- Primary operator entrypoint: root@pve01 / [PRIVATE_IP].
- pve02 and pve03 are reached as root from pve01.
- edge-vm is reached as debian@[PRIVATE_IP] with sudo.
- forum-prod is reached through pve02 using [SENSITIVE_PATH] to root@[PRIVATE_IP].
- PVE users observed: root@pam and prometheus@pve.
- prometheus@pve has PVEAuditor on / for Proxmox exporter access.

### SSH and permissions
- pve01 root keys observed: id_rsa, id_ed25519 and public keys.
- pve02 root keys observed: id_rsa and forum-prod-ci-key.
- pve03 root key observed: id_rsa.
- edge-vm root key observed: id_ed25519; debian authorized_keys observed.
- Security finding: pve01/pve02/pve03 root authorized_keys files were observed as mode 777.
- Correction proof: 160_PVE_ROOT_AUTHORIZED_KEYS_PERMISSIONS_FIXED.txt.
- Private key material must never be copied into the reference.

### Secret storage
- Primary private storage root: /var/lib/homelab-private.
- Edge private secrets root: /var/lib/homelab-private/secrets.
- NPMplus DB: /opt/npmplus/npmplus/database.sqlite, root-only mode observed.
- Gotify DB: /opt/gotify-compose/data/gotify.db.
- Uptime Kuma DB: /opt/uptime-kuma-compose/data/kuma.db.
- Vaultwarden DB: /opt/vaultwarden-compose/data/db.sqlite3.
- Rclone configs exist under root config paths and must not be printed.
- Env/secret inventory is path-only: owner, mode, size and path only.

### Network exposure
- PVE nodes expose SSH :22, Proxmox :8006 and node-exporter :9100.
- pve01/pve02 expose homelab-health-http :9101.
- pve01 exposes NUT :3493 on localhost and [PRIVATE_IP].
- edge-vm exposes public :80/:443 through NPMplus.
- edge-vm NPMplus admin :81 and socket-proxy :2375 are bound to 127.0.0.1.
- edge-vm registry cache :5000 is bound to [PRIVATE_IP].
- edge-vm Home Assistant :8123 is intentionally LAN-exposed.
- Edge ingress hardening proof reports STATUS=OK.

### Rotation and scan proofs
- Cloudflare token rotation completed without printing token values.
- Old Cloudflare token revocation was externally confirmed.
- XenForo SMTP password rotation completed without printing password values.
- XenForo SMTP auth was verified with STARTTLS-safe method.
- Current reference strict scan shows zero strict secret hits.
- Selected generated reference blocks show zero strict secret hits.

### Proof
- Raw security/access inventory: 158_SECURITY_ACCESS_SECRETS_RAW_INVENTORY.txt.
- Secret scan and rotation proof index: 159_SECURITY_SECRET_SCAN_AND_ROTATION_PROOF_INDEX.txt.
- SSH permission correction proof: 160_PVE_ROOT_AUTHORIZED_KEYS_PERMISSIONS_FIXED.txt.
## SECURITY_SSH_PERMISSION_CORRECTION_20260630

- Initial proof 160_PVE_ROOT_AUTHORIZED_KEYS_PERMISSIONS_FIXED.txt is not sufficient as closure evidence.
- Reason: stat without -L still showed 777 after chmod, which may be symlink mode rather than target file mode.
- Error-register item 35 records this proof-quality issue.
- Authoritative replacement proof: 163_PVE_ROOT_AUTHORIZED_KEYS_SYMLINK_AWARE_FIX.txt.
- Valid closure condition: target permissions checked with stat -L are 600 for authorized_keys files and [SENSITIVE_PATH] is 700.
- Final observed status in replacement proof: STATUS=SSH_AUTHORIZED_KEYS_TARGET_PERMISSIONS_OK.
## STORAGE_DISK_SMART_CAPACITY_REFERENCE_20260630

### Proxmox storage model
- Cluster storage is defined in /etc/pve/storage.cfg.
- local storage: dir /var/lib/vz, content iso,vztmpl,backup,import.
- local-lvm storage: lvmthin data, content rootdir,images.
- Nightly vzdump job writes to local storage.
- VM/CT images are primarily stored on local-lvm.
- Staging/offhost paths are under /mnt/staging where present.

### Node disks and capacity
- pve01: primary disk previously inventoried as Lexar SSD NQ7A1 1TB.
- pve02: primary disk previously inventoried as SK800-1TB.
- pve03: primary disk previously inventoried as Samsung 870 EVO 500GB.
- pve01 storage usage was previously observed around local 32.56% and local-lvm 17.50%.
- pve02 storage usage was previously observed around local 12.86% and local-lvm 11.65%.
- pve03 storage usage was previously observed around local 35.79% and local-lvm 64.84%.
- Raw current df/lsblk/LVM/SMART evidence is in 166_STORAGE_DISK_SMART_CAPACITY_RAW_INVENTORY.txt.

### Edge VM storage model
- edge-vm runs Docker application stacks.
- Docker app data lives mainly under /opt/stacks, /opt/npmplus, /opt/gotify-compose, /opt/uptime-kuma-compose, /opt/vaultwarden-compose and /var/lib application paths.
- Immich media is mounted separately at /mnt/immich-media in the container mapping.
- Docker volume and image usage is captured in docker system df output.
- Disk retention health is tracked by disk-retention-policy and disk-retention-cleanup files.

### SMART and health monitoring
- PVE nodes run smartctl textfile timers.
- pve01/pve02/pve03 expose node-exporter :9100 for Prometheus.
- pve01 and pve02 expose homelab-health-http :9101.
- SMART/disk/capacity health evidence is collected from /var/lib/homelab-health where present.
- Prometheus alert rule HomelabSmartDiskHealthFailed watches homelab_smartctl_health_passed == 0.
- Prometheus alert rule HomelabRootFilesystemLowSpace watches root filesystem available below 15%.
- pve03 did not have /var/lib/homelab-health in previous backup-health collection, but it has node-exporter and smartctl timers.

### Retention and cleanup
- Edge disk retention policy previously observed STATUS=OK.
- Edge disk retention policy observed ROOT_USED_PCT=70, REMOVED_DIRS=0, DOCKER_VOLUME_PRUNE=NO.
- Retention dry-run policy is designed to avoid destructive production changes.
- Broad Docker prune is not used as a default cleanup mechanism.
- Cleanup and retention actions must have explicit proof files.

### Operational rules
- Before deleting or pruning storage, create or identify rollback/backup proof.
- Do not delete vzdump/offhost/cloud backup artifacts unless retention policy and proof explicitly allow it.
- Do not infer offhost success from a failed rsync.
- For backup artifacts, verify exact file, size, checksum or zstd/vma integrity where applicable.
- For SMART, use health output plus selected attributes, not full raw dumps with unnecessary noise.

### Proof
- Raw storage/disk/SMART/capacity inventory: 166_STORAGE_DISK_SMART_CAPACITY_RAW_INVENTORY.txt.
- Earlier Proxmox node/storage inventory: 120_PROXMOX_CLUSTER_NODE_STORAGE_NETWORK_INVENTORY.txt.
- Backup/restore catalog: 141_REFERENCE_BACKUP_RESTORE_OFFHOST_CATALOG_BLOCK.txt.
- Monitoring rules and SMART alert model: 147_REFERENCE_MONITORING_ALERTING_HEALTH_BLOCK.txt.
## STORAGE_CAPACITY_CORRECTION_20260630

- Storage block integrity review proof: 169_STORAGE_BLOCK_INTEGRITY_AND_PVE03_CAPACITY_REVIEW.txt.
- 167_REFERENCE_STORAGE_DISK_SMART_CAPACITY_BLOCK.txt integrity status: OK.
- pve03 /mnt/staging current observed use percent: 77.
- pve03 staging is close to the existing WARN threshold of 80% if observed use is 75% or higher.
- pve01 disk-space.txt coverage visibly listed pve01_staging, pve02_staging and edge_immich_media; pve03_staging coverage must be added or explicitly documented elsewhere.
- Storage layer is operationally OK only if SMART remains PASSED and pve03 staging is tracked before it crosses WARN/CRIT thresholds.
- This is a capacity coverage note, not a secret or runtime outage.
## PVE03_STAGING_CAPACITY_MONITOR_20260630

- pve03 /mnt/staging was observed at 77% usage.
- Existing pve01 disk-space.txt did not visibly include pve03_staging coverage.
- A dedicated pve03 staging capacity health check was added on pve01.
- Health file: /var/lib/homelab-health/pve03-staging-capacity.txt.
- Timer: homelab-pve03-staging-capacity-health.timer.
- Service: homelab-pve03-staging-capacity-health.service.
- Current observed status: OK.
- WARN threshold: 80%.
- CRIT threshold: 90%.
- Proof: 173_PVE03_STAGING_CAPACITY_HEALTH_INSTALLED.txt.

## STORAGE_LAYER_FINAL_CLOSED_WITH_PVE03_MONITOR_20260630

- Storage/disk/SMART/capacity layer is closed with an explicit pve03 staging capacity monitor.
- pve03 /mnt/staging was near WARN threshold at 77%, so a dedicated health file and timer were added.
- Health file: /var/lib/homelab-health/pve03-staging-capacity.txt.
- Timer: homelab-pve03-staging-capacity-health.timer.
- Final closure proof: 176_STORAGE_FINAL_CLOSURE_WITH_PVE03_MONITOR.txt.
## SERVICE_DEPENDENCY_MAP_20260630

### Ingress and DNS chain
- Home DHCP DNS: [PRIVATE_IP] and [PRIVATE_IP].
- dns1: CT110 / [PRIVATE_IP] / AdGuard Home.
- dns2: CT111 / [PRIVATE_IP] / AdGuard Home.
- unbound1: CT112 / [PRIVATE_IP]:5335.
- unbound2: CT113 / [PRIVATE_IP]:5335.
- dns1 upstream: [PRIVATE_IP]:5335.
- dns2 upstream: [PRIVATE_IP]:5335.
- Public ingress HTTP/HTTPS: router forwards TCP 80/443 to edge-vm [PRIVATE_IP].
- Reverse proxy: NPMplus on edge-vm, container npmplus, host networking, admin bound to 127.0.0.1:81.
- Edge runtime host: VM130 edge-vm, [PRIVATE_IP], Docker Compose projects count 43.
- NPMplus proxy routes count: 47.
- NPMplus certificates and proxy route source proof: 129_NPMPLUS_PROXY_CERT_STREAM_ROWS_SAFE.txt.

### Core public routes
| Route | Upstream | Runtime owner | Backup/health evidence |
|---|---|---|---|
| nc.gram1.ru | http://[PRIVATE_IP]:11000 | VM150 Nextcloud AIO | VM150 vzdump, nextcloud restore proof, external canary |
| uptime.gram1.ru | http://127.0.0.1:3001 | edge-vm / uptime-kuma | npmplus-kuma backup/restore, Kuma health |
| gotify.gram1.ru | http://127.0.0.1:8082 | edge-vm / gotify | gotify health, alert-routing proof |
| vault.gram1.ru | http://127.0.0.1:8083 | edge-vm / vaultwarden | vaultwarden backup/offhost/restore |
| dockge.gram1.ru | http://127.0.0.1:5001 | edge-vm / dockge | stack inventory |
| home.gram1.ru | http://127.0.0.1:3000 | edge-vm / homepage | homepage container, dashboard route |
| git.gram1.ru | http://127.0.0.1:3002 | edge-vm / gitea | gitea backup/offhost/restore |
| dozzle.gram1.ru | http://127.0.0.1:9999 | edge-vm / dozzle | docker stack inventory |
| paper.gram1.ru | http://127.0.0.1:8010 | edge-vm / paperless | paperless backup/offhost/restore |
| memos.gram1.ru | http://127.0.0.1:5230 | edge-vm / memos | memos backup/offhost/restore |
| photos.gram1.ru | http://127.0.0.1:2283 | edge-vm / immich | immich media/full consistency proof |
| auth.gram1.ru | http://127.0.0.1:9000 | edge-vm / authentik | authentik backup/offhost/restore |
| backup.gram1.ru | http://[PRIVATE_IP]:9101 | pve01 / homelab-health-http | backup dashboard and health proofs |

### VPN routes on wildcard certificate
| Route | Upstream | Runtime owner |
|---|---|---|
| kuma.vpn.gram1.ru | http://127.0.0.1:3001 | uptime-kuma |
| homepage.vpn.gram1.ru | http://127.0.0.1:3000 | homepage |
| gitea.vpn.gram1.ru | http://127.0.0.1:3002 | gitea |
| filebrowser.vpn.gram1.ru | http://127.0.0.1:8085 | filebrowser |
| auth.vpn.gram1.ru | http://127.0.0.1:9000 | authentik |
| actual-budget.vpn.gram1.ru | http://127.0.0.1:5006 | actual-budget |
| grafana.vpn.gram1.ru | http://127.0.0.1:3003 | grafana |
| homebox.vpn.gram1.ru | http://127.0.0.1:7745 | homebox |
| mealie.vpn.gram1.ru | http://127.0.0.1:9001 | mealie |
| n8n.vpn.gram1.ru | http://127.0.0.1:5678 | n8n |
| netbox.vpn.gram1.ru | http://127.0.0.1:8000 | netbox |
| node-red.vpn.gram1.ru | http://127.0.0.1:1880 | node-red |
| beszel.vpn.gram1.ru | http://127.0.0.1:8090 | beszel |
| it-tools.vpn.gram1.ru | http://127.0.0.1:8084 | it-tools |
| karakeep.vpn.gram1.ru | http://127.0.0.1:3030 | karakeep |
| linkding.vpn.gram1.ru | http://127.0.0.1:9091 | linkding |
| minio.vpn.gram1.ru | http://127.0.0.1:9003 | minio |
| ntfy.vpn.gram1.ru | http://127.0.0.1:8055 | ntfy |
| searxng.vpn.gram1.ru | http://127.0.0.1:8888 | searxng |
| syncthing.vpn.gram1.ru | http://127.0.0.1:8384 | syncthing |
| alertmanager.vpn.gram1.ru | http://127.0.0.1:9093 | alertmanager |
| prometheus.vpn.gram1.ru | http://127.0.0.1:9090 | prometheus |
| homeassistant.vpn.gram1.ru | http://127.0.0.1:8123 | homeassistant |
| npmplus.vpn.gram1.ru | https://[PRIVATE_IP]:81 | NPMplus admin, localhost-bound on edge |
| proxmox.vpn.gram1.ru | https://[PRIVATE_IP]:8006 | pve01 Proxmox UI |
| dns1.vpn.gram1.ru | http://[PRIVATE_IP]:8080 | dns1 AdGuard |
| dns2.vpn.gram1.ru | http://[PRIVATE_IP]:8080 | dns2 AdGuard |
| checks.vpn.gram1.ru | http://127.0.0.1:8015 | healthchecks |
| vikunja.vpn.gram1.ru | http://127.0.0.1:8016 | vikunja |
| bookstack.vpn.gram1.ru | http://127.0.0.1:8017 | bookstack |
| pdf.vpn.gram1.ru | http://127.0.0.1:8018 | stirling-pdf |
| jellyfin.vpn.gram1.ru | http://127.0.0.1:8019 | jellyfin |
| audiobooks.vpn.gram1.ru | http://127.0.0.1:8020 | audiobookshelf |
| books.vpn.gram1.ru | http://127.0.0.1:8021 | calibre-web |

### Disabled / special routes
- npm.gram1.ru exists in NPMplus but was observed disabled.
- turn.gram1.ru is an AdGuard rewrite to [PRIVATE_IP] for Nextcloud Talk/TURN.
- Router forwards TCP/UDP 3478 to Nextcloud [PRIVATE_IP].
- Router forwards TCP/UDP 51820 for WireGuard on the Home bridge, not to edge-vm.

### Critical dependency rules
- If a public app is down, check in this order: DNS rewrite/upstream, NPMplus route/cert, upstream container/VM, app health file, backup/restore proof.
- If a VPN route is down, check NetBird first, then NPMplus wildcard cert, then local upstream.
- If DNS is broken, check dns1/dns2 AdGuard and unbound1/unbound2 before app containers.
- If NPMplus is broken, use backup/restore evidence from npmplus-kuma-config and NPMplus DB backup.
- If edge-vm is broken, VM130 full-image backup/offhost/restore proofs are authoritative.
- If pve03 storage approaches WARN/CRIT, use pve03-staging-capacity health file.

### Proof
- Raw service dependency evidence index: 177_SERVICE_DEPENDENCY_MAP_RAW_EVIDENCE_INDEX.txt.
- NPMplus route/cert rows: 129_NPMPLUS_PROXY_CERT_STREAM_ROWS_SAFE.txt.
- AdGuard/Unbound rewrites/upstreams: 130_ADGUARD_UPSTREAM_REWRITES_UNBOUND_RULES_SAFE.txt.
- Edge Docker container normalized map: 134_EDGE_DOCKER_CONTAINER_NORMALIZED_MAP.txt.
- Proxmox VM/CT inventory: 121_PROXMOX_VM_CT_CONFIG_INVENTORY_REDACTED.txt.
- Backup catalog: 141_REFERENCE_BACKUP_RESTORE_OFFHOST_CATALOG_BLOCK.txt.
- Monitoring model: 147_REFERENCE_MONITORING_ALERTING_HEALTH_BLOCK.txt.

## SERVICE_DEPENDENCY_MAP_FINAL_CLOSED_20260630

- Service Dependency Map layer is closed.
- Integrity proof: 180_SERVICE_DEPENDENCY_MAP_INTEGRITY_REVIEW.txt.
- Secret scan proof: 179_SERVICE_DEPENDENCY_MAP_REFERENCE_PROOF_AND_SECRET_SCAN.txt.
- Final closure proof: 181_SERVICE_DEPENDENCY_MAP_FINAL_CLOSURE.txt.
## RUNBOOKS_RECOVERY_PROCEDURES_20260630

### Universal operator preflight
- Before every command, check /etc/pve/HOMELAB_ASSISTANT_ERROR_REGISTER.md and the current 31_HOMELAB_REFERENCE.md.
- Required visible markers: ERROR_REGISTER_CHECK=OK and REFERENCE_CHECK=OK.
- If preflight fails, do not run the main action.
- Do not use exit 1 in interactive SSH sessions.
- Do not paste full terminal transcripts back into shell.
- Do not print secrets.
- Prefer short proof-producing commands over long nested quoting chains.

### First triage order
- Check current reference layer first.
- Check latest proof file named by the relevant reference layer.
- Check health files under /var/lib/homelab-health where applicable.
- Check runtime state only after understanding the owning node, VM, container and proxy route.
- Record every failed command or misleading proof in the error register before moving on.

### Public application down
- Start with Service Dependency Map.
- Check DNS path: dns1/dns2 AdGuard, then unbound1/unbound2.
- Check NPMplus route and certificate using NPMplus DB/proxy proof.
- Check upstream app container or VM.
- Check app-specific health, backup and restore proof.
- Check external-canary and blackbox/Prometheus if route is public.
- Do not change DNS, certificates or proxy routes without DB backup and proof.

### DNS failure
- dns1 is CT110 at [PRIVATE_IP].
- dns2 is CT111 at [PRIVATE_IP].
- unbound1 is CT112 at [PRIVATE_IP]:5335.
- unbound2 is CT113 at [PRIVATE_IP]:5335.
- AdGuard upstreams must point to local Unbound pair.
- Rewrites include public routes for git, dozzle, paper, memos, photos, auth and backup to edge-vm.
- turn.gram1.ru points to Nextcloud [PRIVATE_IP].
- Use DNS/Ingress reference and proof files before editing configs.

### Ingress / NPMplus failure
- Edge VM is VM130 at [PRIVATE_IP].
- NPMplus listens publicly on 80/443 and admin is bound to 127.0.0.1:81.
- NPMplus DB is /opt/npmplus/npmplus/database.sqlite.
- Before modifying certificates or proxy rows, create a DB backup.
- Cloudflare token values must never be printed.
- Use npmplus-kuma-config backup/offhost/restore proofs for recovery.

### Edge VM failure
- VM130 is the Docker runtime host.
- First check Proxmox VM state and pve03 storage.
- Then use VM130 full-image vzdump/offhost/restore proofs.
- Current authoritative edge-vm backup/offhost/restore evidence is in backup catalog and VM130 proofs.
- If restoring, verify exact archive, checksum and VMA/ZSTD integrity before booting replacement.

### Proxmox / VM / CT restore
- Primary nightly job: homelab-nightly-all at 03:30, zstd, snapshot.
- Included VMIDs: 110,111,112,113,130,150,160.
- Use backup catalog for latest known backup/offhost/restore evidence.
- Do not infer offhost success from failed rsync.
- Verify exact artifact, size and checksum where available.
- For VM130 and VM160, use their dedicated closure proofs.

### Monitoring / alerting failure
- Prometheus is on edge-vm at 127.0.0.1:9090.
- Alertmanager is on 127.0.0.1:9093.
- Loki is on 127.0.0.1:3100.
- Uptime Kuma is on 127.0.0.1:3001.
- Gotify is on 127.0.0.1:8082.
- ntfy is on 127.0.0.1:8055.
- Healthchecks is on 127.0.0.1:8015.
- Prometheus proof 153 is invalid and must not be used.
- Use proof 154 and final closure 157 for Prometheus settled target state.

### Backup dashboard / SLO interpretation
- Runtime dashboard OK means current operational checks are green.
- SLO backlog WARN means coverage improvement backlog, not necessarily runtime outage.
- Do not merge SLO backlog WARN into runtime blockers unless explicitly promoted.
- Use slo-coverage-backlog-index for backlog status.

### Storage / capacity event
- pve03 staging is monitored separately because it was observed at 77%.
- Health file: /var/lib/homelab-health/pve03-staging-capacity.txt.
- WARN threshold: 80%.
- CRIT threshold: 90%.
- Timer: homelab-pve03-staging-capacity-health.timer.
- Before cleanup, check retention policy and backup/offhost proof.
- Do not use broad Docker prune by default.

### Security / secret incident
- Stop printing values immediately.
- Identify whether the leak is value, file path, or false-positive text.
- Rotate affected token/password if a value was exposed.
- Create backup before DB/config mutation.
- Re-run strict secret scan after rotation.
- Record proof files and external revocation confirmation where applicable.
- Cloudflare token and XenForo SMTP rotations already have closure proofs.

### Forum / CodeVipe incident
- forum-prod is VM160 at [PRIVATE_IP].
- Access path is through pve02 using [SENSITIVE_PATH]
- XenForo path: /var/www/codevipe/public.
- SMTP host: mail.pvepro.ru on port 587 with STARTTLS semantics.
- Do not enable smtpSsl=true blindly for port 587.
- Do not use fragile nested PHP/base64 SMTP checkers.
- Use XenForo SMTP rotation and auth proof files for current mail state.

### Final evidence rules
- Every remediation gets a numbered proof file.
- Every reference block gets a proof and secret scan file.
- Invalid proof files must be explicitly superseded, not silently ignored.
- Reference closure requires integrity scan, secret scan and required-heading checks.
## HOMELAB_REFERENCE_FINAL_OPERATOR_STATUS_20260630

### Status
- Current homelab reference is complete for the audited local infrastructure scope.
- Final quality precheck passed.
- Required headings are present.
- Bad terminal/log marker scan is clean.
- Strict secret scan is clean.
- This is a reference/operator-status closure, not an archive/export.

### Closed layers
- Critical runtime tails closure.
- Proxmox cluster, node, storage and VM/CT inventory.
- DNS, ingress, NPMplus certificates and AdGuard/Unbound model.
- Edge Docker stacks and container map.
- Backup, restore, offhost and cloud backup catalog.
- Monitoring, alerting, health dashboard and Prometheus correction.
- Security, access and secrets operating model.
- Storage, disk, SMART and capacity model with pve03 staging monitor.
- Service Dependency Map.
- Runbooks and recovery procedures.

### Important superseded/invalid proofs
- 153_PROMETHEUS_TARGETS_SETTLED_PROOF.txt is invalid and must not be used.
- 160_PVE_ROOT_AUTHORIZED_KEYS_PERMISSIONS_FIXED.txt is not sufficient by itself because symlink mode showed 777.
- Use 154 for Prometheus settled targets and 163 for symlink-aware SSH target permissions.

### Current authoritative final proof set
- 185_HOMELAB_REFERENCE_FINAL_QUALITY_SCAN_PRECHECK.txt.
- 184_RUNBOOKS_RECOVERY_REFERENCE_PROOF_AND_SECRET_SCAN.txt.
- 181_SERVICE_DEPENDENCY_MAP_FINAL_CLOSURE.txt.
- 176_STORAGE_FINAL_CLOSURE_WITH_PVE03_MONITOR.txt.
- 165_SECURITY_ACCESS_SECRETS_FINAL_PROOF_AND_SECRET_SCAN.txt.
- 157_MONITORING_PROMETHEUS_FINAL_CLOSURE.txt.
- 142_BACKUP_RESTORE_OFFHOST_REFERENCE_PROOF_AND_SECRET_SCAN.txt.
- 136_EDGE_DOCKER_STACKS_REFERENCE_PROOF_AND_SECRET_SCAN.txt.
- 132_DNS_INGRESS_REFERENCE_PROOF_AND_SECRET_SCAN.txt.

### Operator rule
- Before every future command, continue checking both the error register and this reference file.
- Required visible markers remain ERROR_REGISTER_CHECK=OK and REFERENCE_CHECK=OK.
## HOMELAB_REFERENCE_DEEP_AUDIT_CLOSURE_20260630

### Deep audit result
- Error register and reference consistency audit passed.
- Final proof files unresolved-marker audit passed.
- Reference proof-link audit passed.
- All referenced proof files exist.
- No unresolved final REVIEW_REQUIRED, PROOF_MISSING, HEADING_MISSING, BAD_MARKER, STRICT_SECRET, RULE_CHECK_FAIL, SyntaxError or PERM_BAD markers remained in final proof set.
- Invalid/superseded proofs are explicitly documented and must not be used as authoritative closure evidence.

### Authoritative deep audit proofs
- 189_DEEP_ERROR_REGISTER_REFERENCE_CONSISTENCY_AUDIT.txt.
- 190_DEEP_PROOF_FILES_UNRESOLVED_MARKER_AUDIT.txt.
- 191_DEEP_REFERENCE_PROOF_LINK_AUDIT.txt.

### Scope statement
- This closes the current audited local homelab reference scope.
- External systems can still receive separate dedicated passports if the scope is expanded later.

## NEXTCLOUD_VM150_MAIL_CLOUD_BACKUP_20260630
- VM150 Nextcloud Mail-cloud backup is installed on pve01.
- Timer: homelab-mail-cloud-nextcloud-vm-upload.timer.
- Health file: /var/lib/homelab-health/mail-cloud-nextcloud-vm.txt.
- Current proof health: STATUS=OK, MODE=recurring-chunked, PARTS=14, DOWNLOAD_VERIFY=yes, SECRET_PRINTED=no.
- Proof: 192_NEXTCLOUD_VM150_MAIL_CLOUD_CHUNKED_PROOF.txt.

## ROUTER_RECURRING_BACKUP_BLOCKER_20260630
- Router recurring backup from pve01 is not installed yet.
- Reason: pve01 cannot reach router management ports 2222, 5080, 5083 or 2323 on [PRIVATE_IP].
- Current router backup state remains reference/self-test based until router management access is allowed or startup-config is exported by another approved path.
- Do not run bash/POSIX automation against Netcraze CLI; use router CLI syntax only.
- Proof: 195_ROUTER_RECURRING_BACKUP_BLOCKED_BY_MGMT_ACL_PROOF.txt.

## EDGE_GAP_RESTORE_DRYRUN_CLOSURE_20260630
- Restore dry-run gaps closed for Actual Budget, Home Assistant, Karakeep, Linkding, Mealie, n8n and Observability config.
- Production data was not restored or modified; checks used temporary extraction, SHA verification, tar listing/extraction and SQLite integrity where applicable.
- Current observed restore health: all 7 STATUS=OK with RESTORE_ATTEMPTED=YES and SECRET_PRINTED=no.
- Proof: 197_EDGE_GAP_RESTORE_DRYRUN_CLOSURE_PROOF.txt.

## P2_SMALL_STACKS_BACKUP_RESTORE_CLOSURE_20260630
- P2 small-stack backup and restore dry-run is installed on edge-vm.
- Covered paths: Syncthing config, ntfy cache/data, MinIO data, Homebox data and Node-RED data.
- Timer: homelab-p2-small-stacks-backup-restore.timer.
- Current observed health: STATUS=OK, BACKUP_PATH_COUNT=6, RESTORE_ATTEMPTED=YES, SQLITE_INTEGRITY_OK=yes, SECRET_PRINTED=no.
- Proof: 199_P2_SMALL_STACKS_BACKUP_RESTORE_CLOSURE_PROOF.txt.

## BACKUP_CLOUD_AND_RESTORE_FINAL_CLOSURE_20260630
- Backup/cloud/restore improvement pass is closed with runtime dashboard STATUS=OK and NOT_OK=0.
- VM150 Nextcloud now has Mail-cloud chunked backup with download verification.
- Restore dry-run gaps closed for Actual Budget, Home Assistant, Karakeep, Linkding, Mealie, n8n and Observability config.
- P2 small stacks now have backup and restore dry-run coverage for Syncthing config, ntfy, MinIO, Homebox and Node-RED.
- Router recurring backup remains blocked by management ACL and is documented as blocker, not silently closed.
- Proof: 201_BACKUP_CLOUD_AND_RESTORE_FINAL_CLOSURE_PROOF.txt.

## ROUTER_STARTUP_CONFIG_MAIL_CLOUD_MANUAL_20260630
- Router startup-config manual backup is stored in Mail-cloud crypt remote.
- Source file content is not printed in proofs or reference.
- Health file: /var/lib/homelab-health/router-startup-config-mail-cloud.txt.
- Current observed health: STATUS=OK, UPLOAD_OK=yes, DOWNLOAD_VERIFY=yes, SECRET_PRINTED=no.
- Recurring router backup is still blocked until pve01 can reach router management ports.
- Proof: 203_ROUTER_STARTUP_CONFIG_MAIL_CLOUD_MANUAL_PROOF.txt.

## ROUTER_RUNNING_CONFIG_MAIL_CLOUD_RECURRING_20260630
- Router running-config recurring Mail-cloud backup is installed on pve01.
- Timer: homelab-router-running-config-mail-cloud.timer.
- Health file: /var/lib/homelab-health/router-running-config-mail-cloud.txt.
- Current observed health: STATUS=OK, SSH_RC=0, LOOKS_CONFIG=yes, UPLOAD_OK=yes, DOWNLOAD_VERIFY=yes, SECRET_PRINTED=no.
- Manual startup-config Mail-cloud backup also exists as separate proof.
- Proof: 205_ROUTER_RUNNING_CONFIG_MAIL_CLOUD_RECURRING_PROOF.txt.

## POST_BACKUP_PASS_SLO_RECONCILIATION_20260630
- Post backup/cloud/restore pass SLO reconciliation is closed.
- Runtime backup dashboard remains STATUS=OK with NOT_OK=0.
- Restore index remains STATUS=OK with RESTORE_PROOFS=16/16.
- VM150 Mail-cloud backup, router manual/recurring Mail-cloud backups, P2 small-stacks and 7 edge restore dry-runs are all confirmed OK.
- Existing backup-restore-coverage-slo.txt is historical model output from 20260625 and does not represent current runtime blocker state.
- Current reconciliation health: STATUS=OK, RUNTIME_BLOCKER=NO, SECRET_PRINTED=no.
- Proof: 208_POST_BACKUP_PASS_SLO_RECONCILIATION_PROOF.txt.

## POST_BACKUP_PASS_ALERTING_20260630
- Prometheus alerting for post-backup-pass health files is installed and loaded.
- Rule file: /opt/stacks/observability-lite/prometheus/rules/homelab-post-backup-pass-alerts.yml.
- Alerts: HomelabPostBackupPassHealthNotOk and HomelabPostBackupPassHealthStale.
- Covered checks include VM150 Mail-cloud, router startup/running config Mail-cloud, P2 small-stacks and seven restore dry-run health files.
- Current proof confirms rule validation, Prometheus API visibility and up targets.
- Proof: 210_POST_BACKUP_PASS_ALERTING_PROOF.txt.

## POST_BACKUP_PASS_ALERTS_NOT_FIRING_20260630
- Post-backup-pass Prometheus alert rules are loaded and currently not firing.
- Corrected proof uses direct PromQL ALERTS query for firing state, not grep over rule names.
- Alerts checked: HomelabPostBackupPassHealthNotOk and HomelabPostBackupPassHealthStale.
- Proof: 213_POST_BACKUP_PASS_ALERTS_NOT_FIRING_CORRECTED_PROOF.txt.

## MAIL_CLOUD_CAPACITY_RETENTION_20260630
- Mail-cloud capacity and backup directory inventory was checked after post-backup-pass closure.
- VM150 Nextcloud, router startup/running config, P0 critical and edge-vm backup directories are present on crypt remotes.
- Observed capacity is sufficient: largest used remote was pve03-mail-03-crypt around 65 GiB of 1 TiB.
- Retention shape observed: P0 critical has 10 visible dirs with keep 14, edge-vm has 3 visible dirs with keep 4, new VM150/router jobs have initial dirs.
- Proof: 216_MAIL_CLOUD_CAPACITY_RETENTION_PROOF.txt.

## ROUTER_BACKUP_SECRET_PERMISSION_20260630
- Router recurring backup uses a dedicated routerbackup credential file on pve01.
- Secret file path: /var/lib/homelab-private/secrets/routerbackup.pass.
- File content must never be printed; only mode/owner/size may be checked.
- Current observed permission target: root-owned mode 600.
- Proof: 218_ROUTER_BACKUP_SECRET_PERMISSION_PROOF.txt.

## NEW_BACKUP_UNITS_SYSTEMD_INVENTORY_20260630
- New backup/restore systemd units and timers were inventoried after post-backup-pass closure.
- pve01 units checked: VM150 Mail-cloud upload and router running-config Mail-cloud backup.
- edge-vm units checked: P2 small-stacks backup/restore, backup dashboard and restore drill index.
- Proof records LoadState, ActiveState, UnitFileState and failed-unit counts for pve01 and edge-vm.
- Proof: 222_NEW_BACKUP_UNITS_SYSTEMD_INVENTORY_PROOF.txt.

## POST_BACKUP_PASS_PROM_HEALTH_COVERAGE_20260630
- Prometheus health coverage was checked for the post-backup-pass checks.
- Covered checks: VM150 Mail-cloud, router startup/running config, P2 small-stacks and seven restore dry-runs.
- Proof records health status and age from Prometheus without printing credential values.
- Proof: 224_POST_BACKUP_PASS_PROM_HEALTH_COVERAGE_PROOF.txt.

## POST_BACKUP_PASS_AUDIT_INDEX_20260630
- Audit proof manifest was generated for post-backup-pass evidence files 192-225.
- Manifest records file names, sha256 hashes, count and missing-number check.
- Proof: 226_POST_BACKUP_PASS_AUDIT_INDEX_PROOF.txt.

## POST_BACKUP_PASS_EXTENDED_FINAL_SNAPSHOT_20260630
- Extended post-backup-pass final snapshot was created after capacity, retention, unit inventory, Prometheus coverage and audit index checks.
- Snapshot copies current reference and assistant error register into the audit directory and records sha256 hashes.
- Covered proof range: 192-227.
- Proof: 228_POST_BACKUP_PASS_EXTENDED_FINAL_SNAPSHOT_PROOF.txt.

## POST_BACKUP_PASS_FINAL_AUDIT_INDEX_192_229_20260630
- Final audit manifest was generated after the extended final snapshot.
- Manifest covers proof numbers 192-229, including snapshot and reference-scan proofs.
- Manifest records file names, sha256 hashes, count and missing-number check.
- Proof: 230_POST_BACKUP_PASS_FINAL_AUDIT_INDEX_192_229_PROOF.txt.

## NEW_BACKUP_UNITS_AND_SCRIPTS_INTEGRITY_20260630
- New backup/restore unit files and executable script paths were inventoried and hashed.
- pve01 services checked: VM150 Mail-cloud upload and router running-config Mail-cloud backup.
- edge-vm services checked: P2 small-stacks backup/restore, backup dashboard and restore drill index.
- Proof records FragmentPath, ExecStart path, mode/owner/size and sha256 hashes without printing script contents.
- Proof: 232_NEW_BACKUP_UNITS_AND_SCRIPTS_INTEGRITY_PROOF.txt.

## NEW_BACKUP_UNITS_AND_SCRIPTS_INTEGRITY_CORRECTED_20260630
- Corrected integrity proof was generated because one edge-vm executable required sudo for sha256sum.
- Corrected proof records hashes for pve01 and edge-vm unit fragments and executable scripts without printing script contents.
- Proof: 234_NEW_BACKUP_UNITS_AND_SCRIPTS_INTEGRITY_CORRECTED_PROOF.txt.

## POST_BACKUP_PASS_FINAL_AUDIT_INDEX_192_235_20260630
- Final audit manifest was regenerated after corrected unit/script integrity proof.
- Manifest covers proof numbers 192-235, including corrected integrity proof 234/235.
- Manifest records file names, sha256 hashes, count and missing-number check.
- Proof: 236_POST_BACKUP_PASS_FINAL_AUDIT_INDEX_192_235_PROOF.txt.

## NEW_BACKUP_TIMERS_INTEGRITY_20260630
- New backup/restore timer unit files were inventoried and hashed.
- pve01 timers checked: VM150 Mail-cloud upload and router running-config Mail-cloud backup.
- edge-vm timers checked: P2 small-stacks backup/restore, backup dashboard and restore drill index.
- Proof records FragmentPath, mode/owner/size, sha256 hashes and next-elapse metadata without printing sensitive values.
- Proof: 238_NEW_BACKUP_TIMERS_INTEGRITY_PROOF.txt.

## POST_BACKUP_PASS_FINAL_AUDIT_INDEX_192_239_20260630
- Final audit manifest was regenerated after timer integrity proof.
- Manifest covers proof numbers 192-239 and records file names, sha256 hashes, count and missing-number check.
- Proof: 240_POST_BACKUP_PASS_FINAL_AUDIT_INDEX_192_239_PROOF.txt.

## POST_BACKUP_PASS_FINAL_SNAPSHOT_AFTER_TIMERS_20260630
- Final snapshot was created after timer integrity and final audit manifest 192-239.
- Snapshot copies current reference and assistant error register into the audit directory and records sha256 hashes.
- Covered proof range: 192-241.
- Proof: 242_POST_BACKUP_PASS_FINAL_SNAPSHOT_AFTER_TIMERS_PROOF.txt.

## ASSISTANT_COMMAND_BATCHING_RULE_20260630
- Operator preference: combine related homelab commands into fewer package-style runs to reduce slow repetitive manual work.
- Applies to proof generation, reference updates, validation scans, hash/index refreshes and snapshot steps when safe.
- Mandatory preflight and sensitive-output hygiene still take priority.

## POST_BACKUP_PASS_AUDIT_INDEX_AFTER_BATCHING_RULE_192_245_20260630
- Audit manifest was regenerated after assistant command batching rule was recorded.
- Manifest covers proof numbers 192-245 and records file names, sha256 hashes, count and missing-number check.
- Proof: 246_POST_BACKUP_PASS_AUDIT_INDEX_AFTER_BATCHING_RULE_192_245_PROOF.txt.

## POST_BACKUP_PASS_SNAPSHOT_AFTER_BATCHING_RULE_20260630
- Snapshot was created after assistant command batching rule and audit manifest 192-245.
- Snapshot copies current reference and assistant error register into the audit directory and records sha256 hashes.
- Covered proof range: 192-247.
- Proof: 248_POST_BACKUP_PASS_SNAPSHOT_AFTER_BATCHING_RULE_PROOF.txt.

## ALERTMANAGER_ROUTING_AND_CONFIG_20260630
- Alertmanager routing/config health was checked after post-backup-pass alert rules were installed.
- Proof records container presence, config validation, readiness endpoint and sanitized route/receiver metadata.
- Sensitive receiver values and URLs are intentionally not printed.
- Proof: 250_ALERTMANAGER_ROUTING_AND_CONFIG_PROOF.txt.

## AUDIT_INDEX_192_251_20260630
- Audit manifest was regenerated after Alertmanager routing/config proof.
- Manifest covers proof numbers 192-251 with count and missing-number check.
- Proof: 252_AUDIT_INDEX_192_251_PROOF.txt.

## RCLONE_CONFIG_PERMISSION_AND_REMOTE_INVENTORY_20260630
- rclone config permissions and crypt remote inventory were checked without printing config contents.
- Proof records config owner/mode/size, remote names and capacity summaries for active Mail-cloud crypt remotes.
- Proof: 254_RCLONE_CONFIG_PERMISSION_AND_REMOTE_INVENTORY_PROOF.txt.

## AUDIT_INDEX_192_255_20260630
- Audit manifest was regenerated after rclone config/remote inventory proof.
- Manifest covers proof numbers 192-255 with count and missing-number check.
- Proof: 256_AUDIT_INDEX_192_255_PROOF.txt.

## CURRENT_OPERATIONAL_ROLLUP_20260630
- Current operational rollup was captured after rclone config/remote inventory proof.
- Rollup records failed-unit counts, next timers, Mail-cloud capacity, Prometheus target count, post-backup health status/age, post-backup firing alerts and Alertmanager readiness.
- Proof: 258_CURRENT_OPERATIONAL_ROLLUP_PROOF.txt.

## AUDIT_INDEX_192_259_20260630
- Audit manifest was regenerated after current operational rollup.
- Manifest covers proof numbers 192-259 with count and missing-number check.
- Proof: 260_AUDIT_INDEX_192_259_PROOF.txt.

## SNAPSHOT_AFTER_CURRENT_OPERATIONAL_ROLLUP_20260630
- Snapshot was created after current operational rollup and audit index 192-259.
- Snapshot copies current reference and assistant error register into the audit directory and records sha256 hashes.
- Covered proof range: 192-261.
- Proof: 262_SNAPSHOT_AFTER_CURRENT_OPERATIONAL_ROLLUP_PROOF.txt.

## AUDIT_INDEX_192_263_20260630
- Audit manifest was regenerated after snapshot following current operational rollup.
- Manifest covers proof numbers 192-263 with count and missing-number check.
- Proof: 264_AUDIT_INDEX_192_263_PROOF.txt.

## NEW_BACKUP_SERVICE_RUNTIME_RESULT_20260630
- Runtime result metadata was captured for new backup/restore service units.
- pve01 services checked: VM150 Mail-cloud upload and router running-config Mail-cloud backup.
- edge-vm services checked: P2 small-stacks backup/restore, backup dashboard and restore drill index.
- Proof records LoadState, ActiveState, Result, ExecMainStatus and restart counters without printing sensitive values.
- Proof: 266_NEW_BACKUP_SERVICE_RUNTIME_RESULT_PROOF.txt.

## AUDIT_INDEX_192_267_20260630
- Audit manifest was regenerated after new backup service runtime-result proof.
- Manifest covers proof numbers 192-267 with count and missing-number check.
- Proof: 268_AUDIT_INDEX_192_267_PROOF.txt.

## NEW_BACKUP_UNITS_JOURNAL_ERROR_SCAN_20260630
- Journal warning/error counters were captured for new backup/restore units and timers over the last 24 hours.
- Proof records only counts, not journal message bodies, to avoid sensitive-output risk.
- Proof: 270_NEW_BACKUP_UNITS_JOURNAL_ERROR_SCAN_PROOF.txt.

## AUDIT_INDEX_192_271_20260630
- Audit manifest was regenerated after journal error scan proof.
- Manifest covers proof numbers 192-271 with count and missing-number check.
- Proof: 272_AUDIT_INDEX_192_271_PROOF.txt.

## NEW_BACKUP_UNITS_JOURNAL_ERROR_TRIAGE_20260630
- Journal warning/error triage was captured after warning/error counters were non-zero.
- Proof records per-unit warning/error counts and redacted journal fragments.
- Proof: 274_NEW_BACKUP_UNITS_JOURNAL_ERROR_TRIAGE_PROOF.txt.

## AUDIT_INDEX_192_275_20260630
- Audit manifest was regenerated after journal error triage proof.
- Manifest covers proof numbers 192-275 with count and missing-number check.
- Proof: 276_AUDIT_INDEX_192_275_PROOF.txt.

## VM150_MAIL_CLOUD_JOURNAL_NOISE_CLASSIFICATION_20260630
- VM150 Mail-cloud upload journal warnings/errors were classified after journal triage found non-zero counters.
- Current service result, Prometheus health and firing-alert status are used to decide whether journal noise is blocking.
- Proof includes redacted journal indicators only, not secrets.
- Proof: 278_VM150_MAIL_CLOUD_JOURNAL_NOISE_CLASSIFICATION_PROOF.txt.

## AUDIT_INDEX_192_279_20260630
- Audit manifest was regenerated after VM150 Mail-cloud journal-noise classification.
- Manifest covers proof numbers 192-279 with count and missing-number check.
- Proof: 280_AUDIT_INDEX_192_279_PROOF.txt.

## SNAPSHOT_AFTER_VM150_JOURNAL_NOISE_CLASSIFICATION_20260630
- Snapshot was created after VM150 Mail-cloud journal noise was classified as non-blocking historical noise.
- Snapshot copies current reference and assistant error register into the audit directory and records sha256 hashes.
- Covered proof range: 192-281.
- Proof: 282_SNAPSHOT_AFTER_VM150_JOURNAL_NOISE_CLASSIFICATION_PROOF.txt.

## AUDIT_INDEX_192_283_20260630
- Audit manifest was regenerated after snapshot following VM150 journal-noise classification.
- Manifest covers proof numbers 192-283 with count and missing-number check.
- Proof: 284_AUDIT_INDEX_192_283_PROOF.txt.

## POST_BACKUP_PASS_CLOSURE_SUMMARY_20260630
- Post-backup-pass closure summary was generated after VM150 journal-noise classification.
- Summary checks backup SLO closure, alert rules, firing-alert status, Prometheus health coverage, failed-unit state, Alertmanager readiness, rclone config hygiene, journal-noise classification and audit index completeness.
- Proof: 286_POST_BACKUP_PASS_CLOSURE_SUMMARY_PROOF.txt.

## SNAPSHOT_AFTER_POST_BACKUP_PASS_CLOSURE_SUMMARY_20260630
- Snapshot was created after post-backup-pass closure summary.
- Snapshot copies current reference and assistant error register into the audit directory and records sha256 hashes.
- Covered proof range: 192-287.
- Proof: 288_SNAPSHOT_AFTER_POST_BACKUP_PASS_CLOSURE_SUMMARY_PROOF.txt.

## AUDIT_INDEX_192_289_20260630
- Audit manifest was regenerated after post-backup-pass closure summary snapshot.
- Manifest covers proof numbers 192-289 with count and missing-number check.
- Proof: 290_AUDIT_INDEX_192_289_PROOF.txt.

## TOMORROW_FIRST_RUN_VERIFICATION_PLAN_20260630
- First scheduled-run verification should be done after 2026-07-01 08:15 MSK.
- Verify P2 small-stacks, router running-config Mail-cloud and VM150 Nextcloud Mail-cloud first automatic runs.
- Also verify service results, health status/age, Mail-cloud directories, dashboard, restore index, Prometheus firing alerts and Alertmanager readiness.
- Proof: 292_TOMORROW_FIRST_RUN_VERIFICATION_PLAN_PROOF.txt.

## AUDIT_INDEX_192_293_20260630
- Audit manifest was regenerated after tomorrow first-run verification plan.
- Manifest covers proof numbers 192-293 with count and missing-number check.
- Proof: 294_AUDIT_INDEX_192_293_PROOF.txt.

## END_OF_DAY_POST_BACKUP_PASS_SNAPSHOT_20260630
- End-of-day snapshot was created after post-backup-pass closure summary and tomorrow first-run verification plan.
- Snapshot copies current reference and assistant error register into the audit directory and records sha256 hashes.
- Covered proof range before this snapshot: 192-295.
- Proof: 296_END_OF_DAY_POST_BACKUP_PASS_SNAPSHOT_PROOF.txt.

## AUDIT_INDEX_192_297_20260630
- Final end-of-day audit manifest was generated after post-backup-pass snapshot.
- Manifest covers proof numbers 192-297 with count and missing-number check.
- Proof: 298_AUDIT_INDEX_192_297_PROOF.txt.

## HOME_PORTAL_DISCOVERY_20260630
- Home portal discovery started for https://home.gram1.ru/.
- Goal: make the portal show and open all web services, including external service cards such as NetBird and mail.
- Discovery proof records DNS, HTTPS/TLS headers, reverse-proxy candidates, compose files, domain references and homepage config candidates without printing secrets.
- Proof: 300_HOME_PORTAL_DISCOVERY_PROOF.txt.

## HOME_PORTAL_CONFIG_AND_SERVICE_INVENTORY_20260630
- Home portal config and service inventory was collected for https://home.gram1.ru/.
- Inventory records portal containers, config files, redacted card lines, compose web hints, domain references and common subdomain probes.
- Proof: 302_HOME_PORTAL_CONFIG_AND_SERVICE_INVENTORY_PROOF.txt.

## HOME_PORTAL_GAP_ANALYSIS_20260630
- Home portal gap analysis was collected for gethomepage/homepage behind npmplus.
- Proof records Homepage config hashes, redacted current cards/bookmarks, current URLs, npmplus route lines, local web-port probes and known service container candidates.
- Proof: 304_HOME_PORTAL_GAP_ANALYSIS_PROOF.txt.

## HOME_PORTAL_AUDIT_INDEX_300_305_20260630
- Home portal audit manifest was generated for proof numbers 300-305.
- Proof: 306_HOME_PORTAL_AUDIT_INDEX_300_305_PROOF.txt.

## HOME_PORTAL_CARD_AND_API_ERROR_ANALYSIS_20260630
- Home portal card/API-error analysis was collected before editing Homepage.
- Proof records current card URLs/titles, redacted widget config, redacted Homepage log errors, NPMPlus internal route files/routes and candidate public/VPN cards.
- Proof: 308_HOME_PORTAL_CARD_AND_API_ERROR_ANALYSIS_PROOF.txt.

## HOME_PORTAL_AUDIT_INDEX_300_309_20260630
- Home portal audit manifest was regenerated after card/API-error analysis.
- Manifest covers proof numbers 300-309 with count and missing-number check.
- Proof: 310_HOME_PORTAL_AUDIT_INDEX_300_309_PROOF.txt.

## HOME_PORTAL_DEDUP_AND_EXTERNAL_CARDS_APPLY_20260630
- Homepage services.yaml was backed up, duplicate VPN cards were removed where a non-VPN card existed, and missing web cards for NetBird, Mail and Webmail were added without VPN suffix in the card names.
- SMTP/IMAP were intentionally not added as cards because they are protocol endpoints, not web UI cards.
- Homepage container was restarted and portal/card URLs were checked.
- Proof: 312_HOME_PORTAL_DEDUP_AND_EXTERNAL_CARDS_APPLY_PROOF.txt.

## HOME_PORTAL_AUDIT_INDEX_300_313_20260630
- Home portal audit manifest was regenerated after duplicate-card cleanup and external-card addition.
- Manifest covers proof numbers 300-313 with count and missing-number check.
- Proof: 314_HOME_PORTAL_AUDIT_INDEX_300_313_PROOF.txt.

## HOME_PORTAL_DEDUP_AND_EXTERNAL_CARDS_CORRECTED_APPLY_20260630
- Corrected Homepage apply removed duplicate VPN cards where a non-VPN card existed and added NetBird, Mail and Webmail cards with non-VPN names.
- The added card URLs use currently reachable vpn.gram1.ru web endpoints because public netbird/mail/webmail hosts did not resolve/open during analysis.
- SMTP/IMAP were not added as cards because they are protocol endpoints, not web UIs.
- Proof: 316_HOME_PORTAL_DEDUP_AND_EXTERNAL_CARDS_CORRECTED_APPLY_PROOF.txt.

## HOME_PORTAL_AUDIT_INDEX_300_317_20260630
- Home portal audit manifest was regenerated after corrected duplicate-card cleanup and external-card addition.
- Manifest covers proof numbers 300-317 with count and missing-number check.
- Proof: 318_HOME_PORTAL_AUDIT_INDEX_300_317_PROOF.txt.

## HOMELAB_COMMAND_SAFETY_HARDENING_20260630
- Command safety rule strengthened after the home portal base64 apply failure.
- Opaque generated payloads are not acceptable for homelab changes; commands must be transparent, bounded, and validated before modification.
- Success requires content-specific checks in addition to service/runtime checks.
- Proof: 320_HOMELAB_COMMAND_SAFETY_HARDENING_RULE_PROOF.txt.

## HOME_PORTAL_AUDIT_INDEX_300_321_20260630
- Home portal audit manifest was regenerated after command-safety hardening rule.
- Manifest covers proof numbers 300-321 with count and missing-number check.
- Proof: 322_HOME_PORTAL_AUDIT_INDEX_300_321_PROOF.txt.

## HOME_PORTAL_POST_APPLY_VALIDATION_AND_API_TRIAGE_20260630
- Home portal was validated after duplicate cleanup and external-card addition.
- Proof checks portal HTTP status, Homepage container state, required cards, duplicate VPN card pairs, current card URLs, redacted API-error logs and widget configuration.
- Proof: 324_HOME_PORTAL_POST_APPLY_VALIDATION_AND_API_TRIAGE_PROOF.txt.

## HOME_PORTAL_AUDIT_INDEX_300_325_20260630
- Home portal audit manifest was regenerated after post-apply validation and API triage.
- Manifest covers proof numbers 300-325 with count and missing-number check.
- Proof: 326_HOME_PORTAL_AUDIT_INDEX_300_325_PROOF.txt.

## HOME_PORTAL_API_ERROR_ROOT_CAUSE_20260630
- Homepage API error root-cause analysis was collected after the UI showed API errors.
- Current evidence points to the Open-Meteo/weather widget timing out from Homepage, separate from service cards.
- HTTP 200 for the portal is not sufficient for UI closure when Homepage logs still contain API errors.
- Proof: 328_HOME_PORTAL_API_ERROR_ROOT_CAUSE_PROOF.txt.

## HOME_PORTAL_AUDIT_INDEX_300_329_20260630
- Home portal audit manifest was regenerated after API-error root-cause analysis.
- Manifest covers proof numbers 300-329 with count and missing-number check.
- Proof: 330_HOME_PORTAL_AUDIT_INDEX_300_329_PROOF.txt.

## HOME_PORTAL_DISABLE_OPENMETEO_WIDGET_20260630
- Open-Meteo/weather widget was disabled after root-cause analysis showed Homepage API errors from api.open-meteo.com timeouts.
- This keeps the portal focused on service-launcher cards and prevents external weather API failures from causing UI API errors.
- Portal closure now requires current Homepage API-error logs to be zero after restart/reload.
- Proof: 332_HOME_PORTAL_DISABLE_OPENMETEO_WIDGET_PROOF.txt.

## HOME_PORTAL_AUDIT_INDEX_300_333_20260630
- Home portal audit manifest was regenerated after disabling Open-Meteo/weather widget.
- Manifest covers proof numbers 300-333 with count and missing-number check.
- Proof: 334_HOME_PORTAL_AUDIT_INDEX_300_333_PROOF.txt.

## HOME_PORTAL_LINK_OPEN_AUDIT_20260630
- Home portal card links were audited after Open-Meteo/weather widget was disabled.
- Audit extracts current Homepage href URLs and checks whether each opens with an acceptable HTTP status.
- Acceptable statuses include success, redirects and auth-gated 401/403 pages because those still indicate a reachable web service.
- Proof: 336_HOME_PORTAL_LINK_OPEN_AUDIT_PROOF.txt.

## HOME_PORTAL_AUDIT_INDEX_300_337_20260630
- Home portal audit manifest was regenerated after link-open audit.
- Manifest covers proof numbers 300-337 with count and missing-number check.
- Proof: 338_HOME_PORTAL_AUDIT_INDEX_300_337_PROOF.txt.

## HOME_PORTAL_EXPLICIT_VPN_DUPLICATE_CLEANUP_CORRECTED_20260630
- Explicit VPN duplicate cards were removed from Homepage with a short transparent perl edit: Homepage VPN, Authentik VPN, Gitea VPN and Uptime Kuma VPN.
- Old router URL [PRIVATE_IP] was replaced with [PRIVATE_IP] where present.
- Proof: 342_HOME_PORTAL_EXPLICIT_VPN_DUPLICATE_CLEANUP_CORRECTED_PROOF.txt.

## HOME_PORTAL_REMAINING_DUPLICATE_OCCURRENCE_ANALYSIS_20260630
- Remaining Homepage duplicate/VPN/router occurrences were inspected after corrected cleanup still reported REVIEW.
- Proof records exact redacted file/line occurrences before another edit.
- Proof: 344_HOME_PORTAL_REMAINING_DUPLICATE_OCCURRENCE_ANALYSIS_PROOF.txt.

## HOME_PORTAL_ACTIVE_CONFIG_SCOPE_RECHECK_20260630
- Homepage duplicate cleanup was rechecked against active config files only, excluding logs and backup files.
- Previous broad grep included historical logs/backups and therefore overstated remaining active duplicates.
- Proof: 346_HOME_PORTAL_ACTIVE_CONFIG_SCOPE_RECHECK_PROOF.txt.

## HOME_PORTAL_ACTIVE_VPN_DUPLICATES_FINAL_CLEANUP_20260630
- Active Homepage services/bookmarks files were cleaned from semantic VPN duplicate cards and old router URL references.
- Removed targets: Homepage VPN, Authentik VPN, Gitea VPN and Uptime Kuma VPN, plus matching vpn.gram1.ru duplicate URLs.
- Proof: 350_HOME_PORTAL_ACTIVE_VPN_DUPLICATES_FINAL_CLEANUP_PROOF.txt.

## HOME_PORTAL_NPMPLUS_DEFAULT_ROUTE_ANALYSIS_20260630
- Home portal NetBird/Mail/Webmail cards were investigated after user saw the NPMPlus default page from client [PRIVATE_IP].
- Analysis checks URL body fingerprints, NPMPlus route config matches, service container candidates and Homepage card lines.
- Proof: 356_HOME_PORTAL_NPMPLUS_DEFAULT_ROUTE_ANALYSIS_PROOF.txt.

## HOME_PORTAL_CURRENT_STATE_20260630
- Active Homepage URL: https://home.gram1.ru.
- Homepage container is running and portal HTTP check returned 200 after latest changes.
- NetBird card points to https://nb.pvepro.ru.
- Mail card points to https://mail.pvepro.ru.
- Webmail card was removed; no separate Webmail card is currently configured.
- Router bookmark points to http://[PRIVATE_IP]:5080 for operator client LAN [PRIVATE_IP].
- Homepage services.yaml was updated via YAML-aware edit for siteMonitor fields.
- Current active services.yaml has SITEMONITOR_COUNT=43.
- Latest validation before this reference update: YAML_ERRORS_LAST_5M=0, HOME_HTTP_CODE=200, HOMEPAGE_RUNNING=true.
- Do not use sed-only insertion for siteMonitor; use YAML-aware editing and validate YAML before restart.
- Related active-state checks: NB_LINK_ACTIVE=yes, MAIL_LINK_ACTIVE=yes, WEBMAIL_ACTIVE=no, ROUTER_192_168_1_1_ACTIVE=yes.

## HOME_PORTAL_DIRECT_LINK_FIX_20260630
- Direct Homepage link correction requested by operator.
- NetBird: https://nb.pvepro.ru.
- Mail: https://mail.pvepro.ru.
- Webmail card removed.
- Router restored to http://[PRIVATE_IP]:5080.

## HOME_PORTAL_YAML_AWARE_SITEMONITOR_20260630
- siteMonitor fields were added using YAML-aware PyYAML editing, not line-based sed insertion.
- Added siteMonitor to 43 service cards.
- Excluded cards: Homepage, NPMplus, Router and Public Domain.
- YAML validation passed before restart: YAML_VALIDATE_RC=0.
- Homepage restarted successfully and reported YAML_ERRORS=0.

## HOMELAB_CLUSTER_BACKLOG_20260630
- Cluster backlog was created at /etc/pve/HOMELAB_CLUSTER_BACKLOG.md.
- PBS is intentionally out of scope; backup strategy remains Mail-cloud based.
- First next action: finish Homepage final validation.

## HOMEPAGE_BACKUP_GAP_CLASSIFICATION_20260630
- Homepage backup coverage gap classification file created at /etc/pve/HOMEPAGE_BACKUP_GAP_CLASSIFICATION.md.
- It defines closure requirements for the 13 remaining Homepage backup coverage gap/special-review rows.
- PBS is intentionally out of scope; closure must use Mail-cloud backups or explicit stateless/config-only classification.

## HOMEPAGE_EXISTING_BACKUP_EVIDENCE_CLOSURE_20260630
- Homepage backup coverage matrix started closing existing-evidence rows.
- Router maps to existing Mail-cloud startup/running-config backup proofs when ROUTER_EXISTING_EVIDENCE=yes.
- MinIO Console maps to existing P2 small-stacks backup/restore proof when MINIO_EXISTING_EVIDENCE=yes.
- Proof: 364_HOMEPAGE_EXISTING_BACKUP_EVIDENCE_CLOSURE_PROOF.txt.

## HOME_EXTERNAL_CLOUDFLARE_ALEXHOST_20260630
- Homepage External group includes Cloudflare card: https://dash.cloudflare.com.
- Homepage External group includes AlexHost Billing card: https://bill.alexhost.com/.
- Both cards have siteMonitor enabled for green status dots.
- YAML validation passed, Homepage returned HTTP 200, container running, and no Homepage YAML errors were seen after restart.
- Proof: 368_HOME_EXTERNAL_CLOUDFLARE_ALEXHOST_PROOF.txt.

## HOME_EXTERNAL_RELAY_REMOVED_20260630
- Homepage External card/link for relay.pvepro.ru was removed because relay.pvepro.ru had DNS but no reachable HTTP/HTTPS endpoint from edge-vm.
- Proof: 382_HOME_EXTERNAL_RELAY_REMOVE_PROOF.txt.

## ROUTER_CLI_PERMISSION_LIMIT_20260630
- Netcraze routerbackup SSH access is read-only for backup: show running-config works, but ACL/config commands are denied.
- Homepage Moscow Router red-dot fix requires router ACL/Web UI/admin-level change, not routerbackup.
- Relevant proofs: 391, 393, 394, 395.

## ROUTER_CLI_PROOF_REPAIR_20260630
- Error register updated for blank proof summary extraction in 395.
- Corrected derived counts are recorded in 396_ROUTER_CLI_ERROR_REGISTER_REPAIR_PROOF.txt.

## HOME_EXTERNAL_NETBIRD_ALEXHOST_NORMALIZE_20260630
- Homepage External NetBird and AlexHost Billing href/siteMonitor URLs normalized with trailing slash.
- Homepage container node checks returned HTTP 200 for both URLs.
- Proof: 397_HOME_EXTERNAL_NETBIRD_ALEXHOST_NORMALIZE_PROOF.txt.

## ROUTER_HTTP_SERVICE_FORBIDDEN_FROM_EDGE_VM_20260701
- Moscow Router ACL is restored and correct after manual Web UI edits.
- Remaining blocker for Homepage green dot is HTTP/HTTPS 403 from router web service to edge-vm/Bridge1, not network reachability.
- Relevant proofs: 399, 400, 401, 402.

## HOME_MOSCOW_ROUTER_HEALTH_ENDPOINT_20260701
- Moscow Router Homepage card href: http://[PRIVATE_IP]:5080.
- Moscow Router Homepage siteMonitor: http://[PRIVATE_IP]:18083/router-moscow.
- Health endpoint service on edge-vm: homelab-router-moscow-health-http.service.
- Endpoint semantics: returns 200 when TCP to router [PRIVATE_IP]:5080 succeeds, else 503.
- Relevant proofs: 405, 406, 407.

## HOME_MOSCOW_ROUTER_HEALTH_ENDPOINT_FIXED_20260701
- Moscow Router Homepage card target after fixed apply: href http://[PRIVATE_IP]:5080, siteMonitor http://[PRIVATE_IP]:18083/router-moscow.
- Fixed updater handles Homepage service-name-key YAML shape and property-style YAML shape.
- Relevant proofs: 405, 406, 407, 408.

## HOME_MOSCOW_ROUTER_HEALTH_ENDPOINT_APPLIED_EXACT_20260701
- Moscow Router Homepage card exact active YAML shape was service-key style.
- Applied href: http://[PRIVATE_IP]:5080.
- Applied siteMonitor: http://[PRIVATE_IP]:18083/router-moscow.
- Health endpoint service: homelab-router-moscow-health-http.service on edge-vm.
- Relevant proofs: 406, 409, 410.

## FORUM_PROD_VM160_REBUILD_BASELINE_20260701
- Old test VM160 was stopped and destroyed cleanly: proof 419, CONFIG_EXISTS=no, VM160_DISKS_LEFT=0.
- New VM160 forum-prod was recreated on pve02 with 4 vCPU, 8 GiB RAM, 200G disk, MAC BC:24:11:B6:45:ED, IP [PRIVATE_IP]/24, gateway [PRIVATE_IP].
- New VM identity confirmed: forum-prod / forum-prod.gram1.ru, Debian 12 bookworm, SSH OK, qemu-agent OK, chrony OK.
- Swapfile 8G enabled with /swapfile mode 600 and fstab entry: proof 430. Proof 429 is invalid due nested command substitution quoting.
- SURY PHP repo added; PHP 8.5.7 available and installed with Nginx 1.22.1, MariaDB 10.11.14 and required XenForo modules.
- Beget-compatible profile applied: memory_limit 256M, post/upload 1024M, max_input_vars 10000, MariaDB utf8mb4/utf8mb4_unicode_ci, innodb_buffer_pool_size 2G.
- Operator reference copies placed in /root/homelab-operator inside VM160 for local VM command preflight.

## FORUM_PROD_CODEVIPE_FIRST_RECIPE_STRATEGY_20260701
- Current VM160 is now laboratory state, not final production closure.
- Bulk import of five forums reached DB/files/nginx/php-fpm ready state, but frontend body stayed empty under PHP 8.5.7.
- Strategy changed: prove CodeVipe first on controlled runtime, likely PHP 8.3 or PHP 8.2, then destroy VM160 and rebuild all five forums from scratch using the proven recipe.
- Do not publish Cloudflare/NPMplus routes for these forums until CodeVipe recipe is proven and final rebuild is complete.
- Proofs: failed/diagnostic chain 451-480; next snapshot proof 481 before further lab work.

## FORUM_PROD_FRESH5_DEPLOYED_OK_20260701
- VM160 forum-prod on pve02, IP [PRIVATE_IP].
- Runtime: Debian 12, Nginx, MariaDB 10.11, PHP 8.3-FPM.
- Five XenForo forums deployed from fresh ZIP backups: codevipe.ru, gamevipe.ru, hkmods.ru, zakrutim.ru, dsmods.ru.
- Webroots: /var/www/forums/{codevipe,gamevipe,hkmods,zakrutim,dsmods}/public.
- DBs: xf_codevipe, xf_gamevipe, xf_hkmods, xf_zakrutim, xf_dsmods.
- Final health proof inside VM: /root/evidence/513_FORUM_PROD_FRESH5_FINAL_HEALTH_PROOF.txt.
- Final result: all five frontend/admin HTTP 200, www redirects 301, internal_data 403, no new XenForo errors.
- Lesson: fresh file ZIPs contain populated internal_data/code_cache; broken branch lacked compiled template cache.

## OPERATOR_RULE_CONTINUE_WITH_NEXT_COMMAND_20260701
- Rule: when an infrastructure step ends at an expected prompt or safe checkpoint, do not stop with only a status summary.
- Rule: immediately provide the next executable command in the same response.
- Applies to: SSH enter/exit points, VM prompt confirmations, snapshot confirmations, file copy confirmations, successful health checks, and other obvious next-step transitions.
- Exception: stop only if the previous output contains an error, ambiguity, dangerous state, missing file, secret exposure risk, or a required user decision.

## OPERATOR_RULE_BATCH_2_3_COMMANDS_20260701
- Rule: when safe, provide 2-3 sequential executable commands in one response instead of stopping after each expected checkpoint.
- Applies after: successful prompt transitions, clean health checks, completed copies, completed snapshots, and non-destructive inventory.
- Do not batch commands across: destructive operations, public exposure, DNS/SSL changes, secret handling, unclear errors, missing files, or required user decisions.
- Each command must still print ERROR_REGISTER_CHECK=OK and REFERENCE_CHECK=OK before work.

## FORUM_PROD_FRESH5_PUBLIC_CUTOVER_OK_20260701
- Public cutover completed for codevipe.ru, gamevipe.ru, hkmods.ru, zakrutim.ru, dsmods.ru.
- Cloudflare A records for main and www names point to edge public IP 95.84.154.183.
- Edge NPMplus manual routes terminate TLS with real Let’s Encrypt certificates and proxy to forum-prod VM160 at [PRIVATE_IP]:80.
- Final public proof: /root/evidence/546_PUBLIC_FORUMS_FINAL_DNS_HTTPS_PROOF.txt on pve01.
- Expected public state: main domains HTTPS 200 with ssl_verify_result 0; HTTP redirects to HTTPS; www redirects to main domain through forum-prod canonical vhosts.

## EDGE_FORUM_PUBLICATION_BACKUP_20260701
- Edge NPMplus forum publication backup created after public cutover.
- Backup location: /opt/npmplus/manual-backups/forum-public-ok-*.tar.gz on edge-vm [PRIVATE_IP].
- Includes manual proxy_host configs 200-204 and Let’s Encrypt forum certificates under /opt/npmplus/tls/forum-certs.

## FORUM_CERT_RENEWAL_CONFIGURED_20260701
- Edge certificate renewal configured on edge-vm [PRIVATE_IP].
- Token stored at /opt/npmplus/secure/forum_cf_token.env mode 600 root-only.
- Renewal script: /opt/npmplus/scripts/renew-forum-certs.sh.
- Cron: /etc/cron.d/forum-cert-renew, daily 03:17.
- Proof: /root/evidence/550_EDGE_FORUM_CERT_RENEWAL_SETUP_PROOF.txt on pve01.

## CLOUDFLARE_FORUM_DNS_AUDIT_OK_20260701
- Full Cloudflare DNS audit completed for codevipe.ru, gamevipe.ru, hkmods.ru, zakrutim.ru, dsmods.ru.
- Proof: /root/evidence/554_CLOUDFLARE_FULL_DNS_AUDIT_PROOF.txt on pve01.
- Result: root and www A records for all five zones point to 95.84.154.183.
- Result: no root/www AAAA records, no root/www CNAME conflicts, no records with old IP 87.236.18.45.
- Remaining cdn.* CNAME and TXT/DMARC/DKIM records were observed and not changed.

## FORUMS_FINAL_DNS_AUDIT_SNAPSHOT_OK_20260701
- Final post-DNS-audit snapshot completed after shortening Proxmox snapshot name.
- Snapshot name: forum-dns-ok-065203Z
- Proof: /root/evidence/557_FINAL_SHORT_SNAPSHOT_AFTER_DNS_AUDIT_PROOF.txt on pve01.
- Current public state: all five main domains resolve to 95.84.154.183 and return HTTPS 200 with valid certificates.

## FORUM_LOCAL_BACKUP_CONFIGURED_20260701
- Local forum backup configured inside VM160 forum-prod.
- Script: /root/scripts/forum-backup.sh.
- Cron: /etc/cron.d/forum-local-backup, daily 02:42.
- Retention: removes per-run directories older than 14 days; compressed archives remain under /var/backups/forums.
- Proof: /root/evidence/560_FORUM_BACKUP_SCRIPT_SETUP_PROOF.txt on pve01.

## FORUM_MAIL_NOREPLY_AND_SPF_CLEANUP_20260701
- XenForo email fields defaultEmailAddress/contactEmailAddress/bounceEmailAddress/emailSenderName set to noreply@pvepro.ru for all five forums.
- SPF TXT on codevipe.ru, gamevipe.ru, hkmods.ru, zakrutim.ru, dsmods.ru changed from stale 87.236.18.45 to v=spf1 -all.
- DKIM, DMARC and cdn.* CNAME records were not changed.
- Proofs: 564_UPDATE_XENFORO_EMAIL_NOREPLY_PROOF.txt, 565_CF_UPDATE_FORUM_SPF_NOREPLY_PROOF.txt, 566_FORUM_MAIL_NOREPLY_FINAL_AUDIT_PROOF.txt.

## FORUM_CDN_RECORDS_DELETED_20260701
- Deleted cdn.* CNAME records for codevipe.ru, gamevipe.ru, hkmods.ru, zakrutim.ru, dsmods.ru.
- Reason: old cdn.* records pointed to previous provider/trbcdn and no current public/local forum references were observed.
- Root and www A records remain on 95.84.154.183.
- Proofs: /root/evidence/567_CF_DELETE_FORUM_CDN_RECORDS_PROOF.txt and /root/evidence/568_CDN_DELETE_FINAL_AUDIT_REFERENCE_PROOF.txt on pve01.

## FORUM_SPF_DEDUPED_AND_CDN_DELETED_20260701
- Fixed duplicate SPF state caused by earlier failed cleanup attempt.
- Each of codevipe.ru, gamevipe.ru, hkmods.ru, zakrutim.ru, dsmods.ru now has exactly one SPF TXT: v=spf1 -all.
- Old SPF references to 87.236.18.* are absent.
- cdn.* CNAME records are absent.
- Public HTTPS remained healthy for all five forums.
- Proofs: 569_CF_FIX_DUPLICATE_SPF_RECORDS_PROOF.txt and 570_MAIL_DNS_FINAL_CLEAN_AUDIT_PROOF.txt.

## FORUM_PUBLIC_DNS_MAIL_CLEAN_FINAL_20260701
- Public web and mail-related DNS cleanup completed for five forum domains.
- @ and www A records point to 95.84.154.183.
- cdn.* CNAME records removed.
- SPF deduped to exactly one TXT: v=spf1 -all on each forum domain.
- Old provider IP 87.236.18.* absent from forum domain TXT records.
- XenForo visible email identity changed to noreply@pvepro.ru on all forums.
- Proofs: 570_MAIL_DNS_FINAL_CLEAN_AUDIT_PROOF.txt, 571_FORUM_MAIL_TRANSPORT_READINESS_PROOF.txt, 572_FORUM_PUBLIC_AND_MAIL_DNS_FINAL_SUMMARY_PROOF.txt.

## FORUM_SMTP_TRANSPORT_PAUSED_20260701
- Forum XenForo email identity is noreply@pvepro.ru, but actual SMTP transport is paused.
- Reason: failed SMTP authentication triggered Mailcow netfilter ban for 95.84.154.183.
- Broken msmtp secret/config files were removed from forum-prod.
- Do not re-enable SMTP until the mailbox/app password is rotated and tested with a single controlled attempt.
- Proof: /root/evidence/590_FORUM_SMTP_FAILED_CONFIG_DISABLED_PROOF.txt.

## POST_INCIDENT_STABLE_STATE_20260701
- Incident after failed forum SMTP testing resolved.
- Mailcow and NetBird are reachable again: mail.pvepro.ru/admin and nb.pvepro.ru return HTTP 200.
- Mailcow netfilter ban for 95.84.154.183 was removed on VPS mail.pvepro.ru.
- Forum sites remain healthy over HTTPS.
- Forum SMTP transport remains paused; broken msmtp configs/secrets removed from forum-prod.
- Do not retry SMTP until the mailbox/app password is rotated.
- Proofs: 589_POST_UNBAN_MAILCOW_NETBIRD_VERIFY.txt on VPS, 590_FORUM_SMTP_FAILED_CONFIG_DISABLED_PROOF.txt, 591_POST_INCIDENT_PUBLIC_STATUS_PROOF.txt, 592_POST_INCIDENT_FINAL_STABLE_STATE_PROOF.txt.

## FORUM_SMTP_ONE_SHOT_TEST_OK_20260701
- One controlled SMTP test from forum-prod to mail.pvepro.ru:587 succeeded using rotated credentials.
- No persistent forum SMTP config was enabled.
- One-shot secret/config files were removed after the test.
- Mailcow and NetBird remained reachable after the test.
- Proofs: /root/evidence/594_FORUM_SMTP_ONE_SHOT_TEST_PROOF.txt and /root/evidence/595_SMTP_TEST_AFTERCHECK_PROOF.txt.

## SMTP_ONESHOT_OK_STABLE_FINAL_20260701
- Rotated SMTP credential was tested once from forum-prod to mail.pvepro.ru:587 successfully.
- No persistent SMTP config or secret was left on forum-prod after the one-shot test.
- Mailcow and NetBird remained reachable after the test.
- Forum websites remained healthy.
- Persistent forum SMTP is still not enabled; enable it only with an explicit command and hidden password input.
- Proofs: 594_FORUM_SMTP_ONE_SHOT_TEST_PROOF.txt, 595_SMTP_TEST_AFTERCHECK_PROOF.txt, 596_SMTP_ONESHOT_OK_FINAL_STABLE_PROOF.txt.

## FORUM_PERSISTENT_MSMTP_ENABLED_20260701
- Persistent msmtp-mta transport enabled on forum-prod as /usr/sbin/sendmail.
- SMTP relay: mail.pvepro.ru:587 STARTTLS, sender noreply@pvepro.ru.
- Secret stored only as root:www-data 0640 base64 file under /etc/msmtp/pvepro.env; no raw SMTP_PASS file remains.
- PHP mail() as www-data succeeded after persistent config installation.
- Mailcow and NetBird remained reachable after enabling persistent forum SMTP.
- Proofs: /root/evidence/597_ENABLE_FORUM_PERSISTENT_MSMTP_PROOF.txt and /root/evidence/598_PERSISTENT_MSMTP_FINAL_VERIFY_PROOF.txt.

## FORUM_PERSISTENT_SMTP_FINAL_STABLE_20260701
- Persistent forum SMTP via msmtp is enabled on forum-prod.
- PHP mail() as www-data succeeded after installation.
- Forum domains remain healthy over HTTPS.
- Mailcow and NetBird remain reachable after enabling persistent forum SMTP.
- Secret storage is root:www-data 0640 under /etc/msmtp/pvepro.env; raw [SENSITIVE_PATH] SMTP_PASS file is absent.
- Snapshot after enable: forum-smtp-on-080840Z.
- Proofs: 597_ENABLE_FORUM_PERSISTENT_MSMTP_PROOF.txt, 598_PERSISTENT_MSMTP_FINAL_VERIFY_PROOF.txt, 600_FORUM_PERSISTENT_SMTP_FINAL_STABLE_PROOF.txt.

## FORUM_PROD_FRESH5_MAILRU_CLOUD_BACKUP_OK_20260701
- Existing Mail.ru Cloud rclone crypt remotes on pve02 were reused; no new cloud account was configured.
- Old codevipe-cloud-backup.timer was stale after fresh five-forum rebuild and is disabled.
- New pve02 timer forum-prod-fresh5-cloud-backup.timer uploads current forum-prod local archives from VM160 /var/backups/forums.
- Current backup contains all five forums: codevipe, gamevipe, hkmods, zakrutim, dsmods.
- Upload target: pve02-mail-01/02/03/04-crypt under app/forums/forum-prod/fresh5/<timestamp>.
- Archive is split into 1500M parts before upload, with SHA256SUMS and MANIFEST.txt.
- First fresh5 cloud upload proof: /root/evidence/608_FORUM_PROD_FRESH5_CLOUD_BACKUP_FIRST_RUN_PROOF.txt.

## FORUM_PROD_FRESH5_MAILRU_CLOUD_RESTORE_DRILL_OK_20260701
- Restore drill from Mail.ru Cloud crypt remote completed for forum-prod fresh5 backup.
- Remote source used: pve02-mail-01-crypt app/forums/forum-prod/fresh5 latest timestamp.
- Downloaded split parts, MANIFEST.txt, SHA256SUMS and SHA256SUMS.local were verified.
- Part SHA256 verification passed.
- Split archive was reconstructed and full archive hash matched SHA256SUMS.local.
- tar listing confirmed DB dumps and file archives for codevipe, gamevipe, hkmods, zakrutim and dsmods.
- Proof: /root/evidence/612_FORUM_FRESH5_CLOUD_RESTORE_DRILL_CORRECTED_PROOF.txt.

## FORUM_POSTLAUNCH_BACKUP_MAIL_FINAL_ROLLUP_OK_20260701
- Five public XenForo forums are healthy over HTTPS.
- Mailcow admin and NetBird UI are reachable after persistent forum SMTP enablement.
- Persistent forum SMTP via msmtp is enabled and tested.
- Local backup exists on VM160 under /var/backups/forums.
- Mail.ru Cloud fresh5 backup is enabled on pve02 via forum-prod-fresh5-cloud-backup.timer.
- Old CodeVipe-only cloud timer is disabled.
- Cloud restore drill passed: split parts verified, archive reconstructed, SHA256 matched, tar contained DB/file archives for all five forums.
- Final rollup proof: /root/evidence/614_FORUM_POSTLAUNCH_BACKUP_MAIL_FINAL_ROLLUP_PROOF.txt.

## FORUM_XENFORO_JOB_RUNNER_CONFIGURED_20260701
- XenForo CLI job runner configured inside VM160 forum-prod.
- Script: /root/scripts/forum-xenforo-run-jobs.sh.
- Cron: /etc/cron.d/forum-xenforo-run-jobs every 5 minutes.
- Purpose: process XenForo job queues and cron tasks independent of forum traffic.
- First run proof: /root/evidence/617_XENFORO_RUN_JOBS_ONCE_PROOF.txt.
- Cron verification proof: /root/evidence/618_XENFORO_JOB_RUNNER_CRON_CONFIG_PROOF.txt.

## FORUM_XENFORO_CRON_DAEMON_AND_JOBRUNNER_OK_20260701
- cron daemon inside VM160 forum-prod is active/enabled and was proven by temporary /etc/cron.d execution proof.
- XenForo job runner script executed successfully after cron daemon verification.
- Due XenForo cron count returned to zero after run.
- Proof: /root/evidence/621_FORUM_CRON_DAEMON_JOBRUNNER_VERIFY_PROOF.txt.

## FORUM_DBTECH_TOR_TIMEOUT_CLASSIFIED_20260701
- Each forum had one historical XenForo error_log row for cURL timeout to check.torproject.org/torbulkexitlist.
- Source addon: DBTech/Security DragonByte Security 5.0.0.
- dbtech_security_tornodes option is 0 on all five forums.
- Classified as non-blocking historical external network timeout/noise; not a forum runtime failure.
- Error rows were not deleted.
- Proof: /root/evidence/622_TOR_TIMEOUT_CLASSIFICATION_FINAL_PROOF.txt.

## FORUM_XENFORO_BUILTIN_MAIL_DELIVERED_BUT_SPAM_20260701
- Built-in XenForo outgoing email test from site "Моды для Hollow Knight" was delivered to aleisaev@yandex.ru.
- Sender was noreply@pvepro.ru.
- Yandex placed the message in Spam with warning that links/images were disabled.
- This proves forum SMTP transport works, but deliverability/reputation needs tuning.
- Custom proof 623 is invalid due /root path permission issue and is superseded by this user-observed built-in test.
- Next evidence needed: Yandex raw message headers, especially Authentication-Results and DKIM-Signature.
- Proof: /root/evidence/627_XENFORO_BUILTIN_MAIL_DELIVERED_SPAM_CLASSIFICATION_PROOF.txt.

## FORUM_YANDEX_DELIVERABILITY_SPAM_WITH_SPF_DKIM_PASS_20260701
- Built-in XenForo outgoing email test was delivered to aleisaev@yandex.ru but placed in Spam.
- User-provided Yandex headers showed SPF pass and DKIM pass for pvepro.ru.
- Return-Path, From and DKIM domain aligned on pvepro.ru.
- Yandex spam score observed: X-Yandex-Spam: 4.
- Classification: SMTP/Mailcow/XenForo transport is OK; remaining issue is deliverability/reputation/content filtering.
- DNS change is not required based on this header proof.
- Next actions: mark as Not spam in Yandex, add/check pvepro.ru in Yandex Postmaster, warm up sender reputation, optionally plan per-forum sender domains later.

## FORUM_PROJECT_FINAL_ACCEPTANCE_OK_20260701
- Five XenForo forums are published and healthy over HTTPS: codevipe.ru, gamevipe.ru, hkmods.ru, zakrutim.ru, dsmods.ru.
- VM160 forum-prod is the active production forum VM on pve02 at [PRIVATE_IP].
- Runtime services are configured: nginx, MariaDB, PHP 8.3 FPM, qemu-agent, cron, msmtp/sendmail transport.
- NPMplus on edge terminates TLS and proxies all five forums to VM160.
- Let’s Encrypt certificates are active for all five domains.
- Forum SMTP transport works through noreply@pvepro.ru; Yandex headers proved SPF pass and DKIM pass. Spam placement is classified as reputation/content deliverability, not server failure.
- XenForo job runner cron is configured and cron daemon execution was proven.
- Local backup is configured on VM160 and Mail.ru Cloud fresh5/all-forums backup is configured on pve02.
- Cloud restore drill passed: split parts verified, archive SHA matched, tar contained DB/file/config archives for all five forums.
- DBTech Tor timeout rows are classified as historical external timeout noise and were not deleted.
- Final proof: /root/evidence/629_FORUM_PROJECT_FINAL_ACCEPTANCE_PROOF.txt.

## FORUM_NEXT_CHAT_HANDOFF_FULL_STATE_20260701

### Start point
- Start shell: root@pve01.
- Canonical truth files:
  - /etc/pve/HOMELAB_ASSISTANT_ERROR_REGISTER.md
  - /etc/pve/31_HOMELAB_REFERENCE.md
- Before any infra command, strictly check both files and relevant context blocks.
- Every infra command must print:
  - ERROR_REGISTER_CHECK=OK
  - REFERENCE_CHECK=OK

### Production forum VM
- VMID: 160.
- Name: forum-prod.
- Node: pve02.
- IP: [PRIVATE_IP].
- OS: Debian 12.
- Runtime: nginx, MariaDB 10.11, PHP 8.3 FPM, qemu-agent, cron, msmtp/sendmail transport.
- Final accepted snapshot: forum-final-accepted-091934Z.
- Mail final snapshot: forum-mail-ok-091815Z.
- Postlaunch snapshot: forum-postlaunch-ok-085501Z.

### Public forums
- codevipe.ru -> Форум CodeVipe.
- gamevipe.ru -> Форум GameVipe.
- hkmods.ru -> Моды для Hollow Knight.
- zakrutim.ru -> Консервирование и закрутки.
- dsmods.ru -> Секреты и моды Doom.
- All five are public HTTPS 200 with valid TLS.
- Root/www DNS points through edge public IP 95.84.154.183.
- NPMplus on edge terminates TLS and proxies to http://[PRIVATE_IP]:80.

### Mail
- Forum sender: noreply@pvepro.ru.
- Transport: XenForo sendmail -> msmtp -> mail.pvepro.ru.
- Built-in XenForo mail test reached Yandex.
- Yandex headers showed SPF pass and DKIM pass for pvepro.ru.
- Yandex placed the test in Spam with X-Yandex-Spam: 4.
- Classification: transport/authentication OK; remaining issue is sender reputation/content filtering.
- DNS change is not required from the captured header proof.
- Manual next action: click “Не спам!” in Yandex and add/check pvepro.ru in Yandex Postmaster.

### Backups and restore
- Local backup configured inside VM160.
- Local archive path pattern: /var/backups/forums/forum-backup-*.tar.gz.
- pve02 Mail.ru Cloud fresh5/all-forums backup configured.
- New timer: forum-prod-fresh5-cloud-backup.timer active/enabled.
- Old codevipe-only timer disabled/inactive.
- Cloud remotes:
  - pve02-mail-01-crypt
  - pve02-mail-02-crypt
  - pve02-mail-03-crypt
  - pve02-mail-04-crypt
- Latest verified cloud stage during final acceptance: 20260701T083354Z.
- Restore drill passed: split parts verified, archive SHA matched, tar contained DB/file/config archives for all five forums.

### XenForo jobs and cron
- cron daemon was missing, then installed and enabled.
- /etc/cron.d execution was proven by temporary cron proof.
- XenForo job runner configured:
  - /root/scripts/forum-xenforo-run-jobs.sh
  - /etc/cron.d/forum-xenforo-run-jobs
- Purpose: process XenForo jobs and cron tasks independent of visitor traffic.

### DBTech/Tor timeout
- Each forum had one historical xf_error_log row for cURL timeout to check.torproject.org.
- Source addon: DBTech/Security DragonByte Security 5.0.0.
- dbtech_security_tornodes=0 on all five forums.
- Classified as non-blocking historical external timeout noise.
- Rows were not deleted.

### Key final proofs
- /root/evidence/629_FORUM_PROJECT_FINAL_ACCEPTANCE_PROOF.txt
- /root/evidence/628_FORUM_MAIL_DELIVERABILITY_FINAL_SNAPSHOT_PROOF.txt
- /root/evidence/627_YANDEX_HEADER_DELIVERABILITY_CLASSIFICATION_PROOF.txt
- /root/evidence/622_TOR_TIMEOUT_CLASSIFICATION_FINAL_PROOF.txt
- /root/evidence/621_FORUM_CRON_DAEMON_JOBRUNNER_VERIFY_PROOF.txt
- /root/evidence/614_FORUM_POSTLAUNCH_BACKUP_MAIL_FINAL_ROLLUP_PROOF.txt
- /root/evidence/613_FORUM_FRESH5_CLOUD_RESTORE_DRILL_FINALIZE_PROOF.txt
- /root/evidence/612_FORUM_FRESH5_CLOUD_RESTORE_DRILL_CORRECTED_PROOF.txt
- /root/evidence/609_FORUM_PROD_FRESH5_CLOUD_BACKUP_FINALIZE_PROOF.txt
----- CONTENT END -----

----- FILE=/etc/pve/HOMELAB_ASSISTANT_ERROR_REGISTER.md -----
SIZE=77979 MTIME=2026-07-15 09:27:06.000000000 +0300 MODE=640 OWNER=root:www-data
SHA256=ab8308549db90d235b591af0581580c3c7ba1b287b129cc6111c2eef0a0f9ecc
LINES=934
----- CONTENT BEGIN -----
# HOMELAB ASSISTANT ERROR REGISTER

Назначение: перед каждой следующей командой сверяться с этим файлом.

## Критические ошибки ассистента
1. Повторно дал слишком большой интерактивный paste в shell.
2. Повторно использовал here-doc/Markdown/backticks там, где нужен короткий безопасный файл или отдельный скрипт.
3. Дал генератор справочника прямо в терминал вместо безопасного маленького шага.
4. Нарушил своё же правило: не давать длинные вложенные команды с кавычками.

## Жёсткие правила перед каждой командой
CHECK-1: команда не должна быть большим paste.
CHECK-2: команда не должна содержать большой here-doc.
CHECK-3: команда не должна смешивать Markdown, backticks и shell-логику.
CHECK-4: команда не должна иметь вложенный ssh с несколькими уровнями кавычек.
CHECK-5: команда не должна печатать секреты.
CHECK-6: если создаётся файл, сначала маленький безопасный шаг, потом проверка.
CHECK-7: если команда длиннее 8 строк, её нельзя давать в интерактивный shell.
CHECK-8: для edge-vm использовать debian@[PRIVATE_IP] и sudo.
CHECK-9: для forum-prod использовать pve02 и ключ [SENSITIVE_PATH]
CHECK-10: Corosync не трогать без отдельного плана и rollback.

## Текущие важные факты
Internal network: [PRIVATE_IP]/24.
Migration config: migration: secure,network=[PRIVATE_IP]/24.
Corosync remains on [PRIVATE_IP]/12/13.
VM160 forum-prod is not in Proxmox nightly backup.
VM130 edge-vm has secondary disk backup=0 risk.
05_edge_compose_safe.tgz quarantined.

## Правило для справочника
Не генерировать большой справочник через интерактивную вставку.
Следующий справочник делать только маленькими append-блоками или через файл, созданный вне интерактивного shell.

11. Ошибка: считать offhost OK после failed rsync.
Если rsync/scp упал, нельзя проверять latest offhost-файл без сверки имени.
Проверка должна подтвердить именно новый архив, например дату 2026_06_30-00_20_59.
Старый OFFHOST_ZSTD_OK не закрывает новый backup.

12. Ошибка: широкий secret-поиск по /opt/stacks дал шум.
Не искать Cloudflare/SMTP сразу по /opt/stacks, MkDocs site, backups и HTML-отчётам.
Для ротации сначала смотреть live-конфиги: /opt/npmplus/tls/certbot/renewal/*.conf и реальные credentials-файлы из них.
Значения секретов не печатать; выводить только пути, ключи и redacted-поля.

13. Ошибка: SQL с одинарными кавычками внутри одинарной SSH-команды ломается.
Нельзя писать sqlite SQL вида j.type in ('object','array') внутри ssh '...'.
Для sqlite через ssh использовать SQL без одинарных кавычек: char(36), length(j.atom), двойные внешние кавычки, либо отдельный файл.
Перед запуском проверять, что команда не содержит конфликтующих уровней кавычек.

14. Ошибка: путать контекст входа и узел выполнения.
Стартовая точка оператора: root@pve01 / [PRIVATE_IP].
edge-vm: ssh debian@[PRIVATE_IP], внутри использовать sudo.
pve02/pve03: ssh root@pve02 или ssh root@pve03.
forum-prod: заходить через pve02, ключ [SENSITIVE_PATH]
Перед каждой командой явно понимать: где выполняется локальная часть, где remote-часть, каким пользователем.

15. Ошибка: повторно нарушено правило №13 после его добавления.
Снова был использован SQL JSON-path в одинарных кавычках внутри ssh '...'.
Запрещено использовать в remote sqlite SQL фрагменты вида '$.key' или '.key'.
Для JSON-path использовать только char(...), без одинарных кавычек внутри SQL.
Команду с ошибкой char(36)||.dns_provider считать битой и не использовать.

16. Основополагающее правило: перед каждой командой ассистент обязан явно написать: Сверка с файлом ошибок: CHECK пройден.
После этой строки можно давать только одну короткую команду или один логический блок команд.
Нельзя выдавать команды без предварительной сверки с этим файлом ошибок.
Нельзя продолжать после собственной ошибки без записи ошибки в этот файл.

19. Уточнение формата: не писать отдельную строку сверки перед каждой командой в пакете.
Сверка должна быть внутри самой команды первым логическим блоком: grep правил из файла ошибок и RULE_CHECK_OK.
Разрешено давать до трёх логически связанных команд сразу, по возможности в одну строку каждая.
Не писать фразу "ждём" после команд; указывать только контрольные строки результата.

20. Ошибка: широкий XenForo SMTP-поиск по src дал шум исходников.
Для XenForo SMTP не grep-ать весь /src; сначала смотреть live src/config.php без значений, затем БД XenForo и xf_option/emailTransport.
Значения DB-паролей и SMTP-паролей не печатать.

21. Ошибка: nested PHP php -r дал Parse error на forum-prod.
Команды 102-104 считать битыми: они не подтвердили и не изменили smtpSsl.
Не продолжать длинные вложенные ssh+php -r с кавычками; для forum-prod лучше войти в VM или использовать короткие mysql/php команды без вложенного кода.
Рабочими считать: DB backup перед SMTP, SMTP password update, JSON postcheck; SMTP auth ещё не закрыт из-за CODE=530 на AUTH LOGIN.

22. Ошибка: самодельный base64 PHP для SMTP auth сломан.
Команда 108 дала PHP Parse error на smtpHost и пустой proof-файл.
Не использовать больше сгенерированные большие base64 PHP для SMTP/XenForo.
Для XenForo SMTP использовать короткие mysql-запросы к xf_option и готовые штатные механизмы XenForo.

23. Ошибка: MariaDB не поддержала cast('true' as json) в JSON_SET.
Команда 111 не изменила smtpSsl: OLD_SMTPSSL=false, SQL ERROR 1064, NEW_SMTPSSL=false.
Перед боевым UPDATE сначала проверять JSON_SET синтаксис на тестовом JSON через SELECT.
Не считать PHP_FPM_RELOAD_OK подтверждением изменения БД.

24. Ошибка: exit 1 в interactive-check закрыл SSH-сессию.
Нельзя использовать { echo RULE_CHECK_FAIL; exit 1; } в командах, выполняемых прямо в интерактивном root@pve01 shell.
При failed-check использовать безопасный шаблон: if grep ...; then echo RULE_CHECK_OK; ...; else echo RULE_CHECK_FAIL; fi.
Не делать fragile grep по точной строке с кавычками, если в файле ошибок строка содержит shell/SQL quoting.

25. Ошибка анализа: не надо слепо включать smtpSsl=true при smtpPort=587.
smtpPort=587 обычно означает STARTTLS, а не implicit SMTPS.
CODE=530 на AUTH LOGIN означает, что ручной checker не сделал STARTTLS; это не доказывает неверный SMTP-пароль.
Сначала проверять AUTH через openssl s_client -starttls smtp, без вывода секрета.

26. Ошибка: openssl -crlf вместе с ручным CRLF ломает SMTP AUTH checker.
Команда 116 получила 535 Invalid base64 data in continued response после 334 Username.
Это указывает на битую base64-строку checker-а, а не на неверный SMTP-пароль.
Не использовать -crlf, если команды уже отправляются с явным \r\n.
Для 587 оставлять smtpSsl=false и проверять STARTTLS корректным checker-ом.

27. Ошибка: Python heredoc внутри ssh сломал not_ok диагностику.
Команда 138 пересобрала dashboard, но диагностический Python-блок упал с NameError из-за кавычек.
Не использовать heredoc Python внутри вложенного ssh; для dashboard verification использовать cat/grep JSON или простые команды.

28. Ошибка: docker compose ls --format json | wc -l дал ложный COMPOSE_PROJECT_COUNT=1.
Причина: JSON-вывод может быть одной строкой/структурой, wc -l не является счётчиком проектов.
Для Docker Compose count использовать обычный docker compose ls --all + awk по строкам таблицы или JSON parser, не wc -l.

29. Ошибка: monitoring compact status искал неверные имена health-файлов.
Факт: npmplus-cert-expiry.txt не является подробным файлом; подробный файл называется npmplus-certificate-expiry.txt, а статусный alias npmplus-cert-expiry-health.txt.
Факт: health-metrics.txt не обязан существовать как health-файл; health metrics могут публиковаться через node-exporter textfile collector.
Для Prometheus targets проверять HTTP_CODE/RC явно, не оставлять пустой блок PROMETHEUS_TARGETS_LOCAL.

30. Ошибка/инцидент: Prometheus API HTTP_CODE=000 после monitoring reference.
Факт: в monitoring compact status curl к 127.0.0.1:9090 вернул HTTP_CODE=000, а raw inventory не показал контейнер prometheus среди monitoring containers.
Нельзя считать monitoring слой runtime-closed только по secret scan/reference proof; нужно отдельно проверить Prometheus container/API и обновить справочник по факту.

31. Строгое правило: перед каждой командой сверяться с файлом ошибок и справочником.
Перед любым действием команда обязана проверять /etc/pve/HOMELAB_ASSISTANT_ERROR_REGISTER.md и текущий 31_HOMELAB_REFERENCE.md.
Команда обязана печатать ERROR_REGISTER_CHECK=OK и REFERENCE_CHECK=OK до основного действия.
Если сверка не пройдена, команда не должна выполнять основное действие. Не использовать exit 1, чтобы не закрывать SSH-сессию.

32. Ошибка: 153_PROMETHEUS_TARGETS_SETTLED_PROOF содержит SyntaxError из-за вложенного Python внутри SSH.
Факт: Python -c внутри ssh потерял кавычки вокруг /tmp/prom-targets-settled.json, data, activeTargets, labels, job, health.
Файл 153_PROMETHEUS_TARGETS_SETTLED_PROOF.txt нельзя считать валидным settled proof.
Для таких проверок не использовать вложенный Python; использовать curl + grep/sed/awk либо отдельный локальный файл-скрипт.

34. Ошибка операционного ввода: полный терминальный лог MobaXterm попал в shell.
Факт: bash пытался выполнить строки Authenticating, MobaXterm banner, RULE_CHECK_OK, PROOF_OK и таблицы вывода как команды.
Такие выводы не являются инфраструктурным состоянием; считать невалидными только соответствующие failed command attempts.
Дальше давать короткие команды и не вставлять обратно полный transcript в shell.

33. Security finding: root authorized_keys на PVE-нодах имел права 777.
Факт: inventory 158 показал [SENSITIVE_PATH] и backup-файлы с режимом 777 на pve01/pve02/pve03.
Нужно исправлять на [SENSITIVE_PATH]=700 и authorized_keys/authorized_keys.bak*=600, затем фиксировать before/after proof.

35. Ошибка проверки: 160_PVE_ROOT_AUTHORIZED_KEYS_PERMISSIONS_FIXED не доказал исправление прав.
Факт: AFTER всё ещё показал 777 для [SENSITIVE_PATH] на pve01/pve02/pve03.
Возможная причина: authorized_keys является symlink, а stat без -L показывает режим ссылки 777.
Нужно проверять stat -L целевого файла и считать 160 недостаточным proof.

36. Quality check: Storage block needs integrity and pve03 capacity coverage review.
Факт: terminal output around storage block creation showed possible paste/truncation artifact near smartctCK=FAIL.
Факт: pve03 /mnt/staging observed at 77%, close to WARN threshold 80%, while disk-space.txt did not visibly list pve03_staging.
Before closing storage layer, verify block integrity and pve03 capacity coverage.

37. Coverage gap: pve03_staging missing from disk-space health coverage.
Факт: 169 showed pve03 /mnt/staging at 77%, close to WARN 80%, while disk-space.txt listed pve01_staging, pve02_staging and edge_immich_media only.
Before closing storage fully, add or document pve03_staging capacity monitoring.

38. Quality check: Service Dependency Map block needs integrity review.
Факт: terminal output around command 222 showed paste artifact near "FAIL; fi".
Before closing Service Dependency Map, verify 178_REFERENCE_SERVICE_DEPENDENCY_MAP_BLOCK.txt and the reference section for shell/log garbage or truncation.

39. Ошибка проверки: post deep audit BAD_MARKER scan caught prose false positive.
Факт: строка справочника с фразой No unresolved final REVIEW_REQUIRED... SyntaxError... была принята за BAD_MARKER.
Это не runtime/log мусор, а документированная фраза deep audit closure. В scan нужно исключать эту prose-строку.

40. Ошибка/вывод: VM150 Mail cloud single-file upload failed with 413.
Факт: Mail/WebDAV rejected one 6.6G vzdump file with 413 Request Entity Too Large.
Решение: use VM150 recurring chunked script with 512M parts and download verification.

41. Ошибка: post-backup-pass-slo-reconciliation checked pve01 router health on edge-vm.
Факт: first reconciliation wrote STATUS=ERROR because router-running-config-mail-cloud.txt is on pve01, not edge-vm.
Fix: build reconciliation from pve01 and edge-vm health files by correct owner.

## ASSISTANT_COMMAND_BATCHING_RULE_20260630
- Rule: combine related infrastructure checks, proof generation, reference updates and validation scans into one safe batch command when practical.
- Do not split work into many small commands when the operations are low-risk, read-only, or naturally part of one evidence block.
- Still keep mandatory preflight checks, short bounded commands, no large paste, no here-docs, no interactive shell traps, and no sensitive values in output.
- Split into smaller commands only when safety, rollback, quoting risk, or troubleshooting clarity requires it.

## HOME_PORTAL_BASE64_APPLY_FAILURE_20260630
- Prior home-portal apply attempt 312 failed because a large base64 Python payload decoded as invalid UTF-8.
- Rule: avoid large opaque base64 script payloads for homelab changes; use transparent bounded shell/perl/python commands instead.

## HOMELAB_COMMAND_SAFETY_HARDENING_20260630
- Rule: do not use large opaque payloads, generated base64 scripts, or hidden multi-line script blobs for homelab changes.
- Use transparent bounded shell/perl/python commands, or create a temporary readable script and validate it before execution.
- Any script used for changes must pass a syntax/parse check before it is allowed to modify files or restart services.
- Any Traceback, UnicodeDecodeError, SyntaxError, Permission denied, command-not-found, nonzero critical RC, or failed write marker must force REVIEW or FAILED, never OK.
- Success requires both runtime health and content-specific post-checks: expected files changed, expected cards exist, duplicates absent, service/container alive, and no new critical log pattern.
- HTTP 200 alone is not a success condition for configuration changes.
- If an apply step fails, the next command must explicitly record the failed attempt and then run a corrected transparent apply command.

## HOME_PORTAL_API_ERROR_CLOSURE_RULE_20260630
- Rule: home portal closure requires current Homepage API-error logs to be zero after restart/reload, not only HTTP 200.
- If Homepage UI shows API error or logs contain current httpProxy/API timeout errors, the portal is REVIEW until the widget/API cause is disabled or fixed.
- External informational widgets such as weather/Open-Meteo must not block the service launcher portal.

## HOME_PORTAL_COMPLEX_QUOTING_FAILURE_20260630
- Previous home portal cleanup command failed in local bash before execution: syntax error near unexpected token `('.
- Cause: command was too complex and fragile due to nested shell/perl/python quoting.
- Rule: do not use deeply nested one-liners for config edits; first run short state checks, then apply small transparent edits.

## HOME_PORTAL_ACTIVE_CONFIG_SCOPE_RULE_20260630
- Rule: Homepage cleanup validation must check active config files only, not logs or backup files.
- Active files are services.yaml, bookmarks.yaml, widgets.yaml, settings.yaml and docker.yaml under /opt/stacks/homepage/config.
- Matches in /logs/ or *.bak* are historical evidence only and must not make cleanup REVIEW unless the active config still contains the target.

## HOME_PORTAL_LINK_VALIDATION_NPMPLUS_DEFAULT_RULE_20260630
- Rule: Homepage link validation must not treat HTTP 200 as success if the body is the NPMPlus default “Congratulations” / host-not-set-up page.
- Green siteMonitor dots can be false positives when NPMPlus returns its default vhost page.
- A portal card is openable only when DNS, reverse-proxy host mapping, and target content are all valid.

## ROUTER_CLI_STDIN_APPLY_FAILURE_20260630
- Context: attempted Netcraze router ACL apply through SSH stdin/multiline for Homepage Moscow Router monitor fix.
- Evidence: 391_MOSCOW_ROUTER_EDGE_VM_WEBADMIN_ACL_APPLY_PROOF.txt.
- Failure: STDIN_SHOW_OK=no, APPLY_SKIPPED=stdin_show_failed, CONFIG_HAS_EDGE_5080=no, CONFIG_HAS_EDGE_5083=no.
- Rule: do not apply Netcraze config through stdin/multiline SSH; prove CLI input mode first.

## ROUTER_CLI_PROBE_STDIN_CONSUMPTION_ERROR_20260630
- Context: ACL syntax read-only probe loop executed only one command because ssh consumed the loop stdin.
- Evidence: 393_ROUTER_CLI_ACL_SYNTAX_READONLY_PROBE_PROOF.txt showed COMMAND_COUNT=1.
- Fix pattern: use ssh -n or redirect SSH stdin away from the command-list loop for all future SSH-in-loop probes.

## ROUTERBACKUP_READONLY_CLI_PERMISSION_LIMIT_20260630
- Context: routerbackup user can run show running-config but cannot execute configuration commands such as access-list.
- Evidence: 394_ROUTER_CLI_ACL_SYNTAX_READONLY_PROBE_FIXED_PROOF.txt.
- Observed: CMD_TEXT=access-list ? returned Core::Configurator error execute denied [cli].
- Rule: do not attempt ACL/config changes as routerbackup; use admin/operator router access or router Web UI for write changes.

## PROOF_SUMMARY_EXTRACTION_BLANK_20260630
- Context: proof 395 tried to read COMMAND_COUNT/ZERO_RC_COUNT from proof files 393/394, but those summary fields were printed to console and were not persisted inside the proof files.
- Evidence: 395_ROUTER_CLI_ERROR_REGISTER_UPDATE_PROOF.txt has blank P393_COMMAND_COUNT/P394_COMMAND_COUNT/P394_ZERO_RC_COUNT fields.
- Rule: when creating follow-up proof summaries, derive counts from persisted proof body lines such as CMD_INDEX and CMD_RC, or write summary fields into the proof file itself.

## ROUTER_ACL_UI_RULE_DELETE_AND_RESTORE_20260701
- Context: while moving Moscow Router monitor rules to _WEBADMIN_Bridge1, two _WEBADMIN_Bridge0 Nextcloud Talk TURN rules were accidentally removed.
- Deleted/restored rules: Home [PRIVATE_IP]/24 to Nextcloud [PRIVATE_IP] TCP/UDP 3478.
- Verification proof: 399_ROUTER_ACL_RESTORE_AFTER_UI_PROOF.txt.
- Rule: after manual router Web UI ACL edits, compare _WEBADMIN_Bridge0, _WEBADMIN_Bridge1 and _WEBADMIN_GigabitEthernet1 before continuing.

## ROUTER_MONITOR_NODE_LOOKUP_PROBE_BUG_20260701
- Context: proof 400 tested Moscow router HTTPS monitor from Homepage container with custom Node lookup callback.
- Failure: NODE_HTTPS_RESOLVE_ERROR=ERR_INVALID_IP_ADDRESS and NODE_HTTPS_RESOLVE_INSECURE_ERROR=ERR_INVALID_IP_ADDRESS.
- Impact: Node HTTPS resolve subtest is invalid; curl --resolve results remain valid.
- Rule: do not rely on custom Node lookup callback until separately tested; prefer curl --resolve or simple direct Node HTTP checks.

## ROUTER_HTTP_SERVICE_FORBIDDEN_FROM_EDGE_VM_20260701
- Context: Moscow Router Homepage monitor after ACL fix.
- Evidence: proofs 399, 400, 401.
- Observed: edge-vm TCP to [PRIVATE_IP]:5080/5083 is OK, but router HTTP/HTTPS returns 403 for all tested paths.
- Impact: Homepage siteMonitor cannot be made green via direct router HTTP URL until router web service allows the Bridge1/Proxmox/protected segment or an explicit safe monitor endpoint is used.
- Rule: do not keep changing ACL for this symptom; ACL is already correct. Fix service access scope or use a dedicated monitor endpoint.

## PY_COMPILE_PYC_PERMISSION_ERROR_20260701
- Context: installing edge-vm Moscow router health endpoint.
- Mistake: assistant used `python3 -m py_compile` on a script installed under /usr/local/sbin as root.
- Actual impact: non-root syntax check attempted to create /usr/local/sbin/__pycache__ and failed with Permission denied.
- Recovery: validate syntax with `ast.parse` instead of py_compile, because it does not write .pyc files.
- Rule: do not use py_compile against root-owned system paths from an unprivileged user.

## ROUTER_HOME_IP_DOCKER_ROUTE_CONFLICT_20260701
- Context: Moscow Router Homepage siteMonitor attempt using http://[PRIVATE_IP]:5080.
- Evidence: proof 405 showed route to [PRIVATE_IP] via Docker bridge and Homepage container ECONNRESET.
- Impact: do not use [PRIVATE_IP] directly as Homepage siteMonitor from edge-vm.
- Mitigation: use dedicated edge-vm health endpoint that checks router TCP [PRIVATE_IP]:5080 and returns HTTP 200/503 for Homepage.

## HOMEPAGE_ROUTER_MOSCOW_YAML_TITLE_SHAPE_MISMATCH_20260701
- Context: applying Moscow Router Homepage siteMonitor health endpoint.
- Mistake: assistant apply script matched only property-style cards with `title`, but active Homepage YAML can use service-name-key style like `- Service Name:`.
- Evidence: proof 407 returned YAML_CHANGE_COUNT=0 and APPLY_SKIPPED=unexpected_match_count while health endpoint proof 406 was OK.
- Recovery: use YAML-aware updater that handles both property-style and service-name-key style cards.

## HOMEPAGE_ROUTER_MOSCOW_APPLY_SCRIPT_SYNTAX_ERROR_20260701
- Context: applying Moscow Router Homepage health endpoint.
- Mistake: assistant generated Python script with invalid f-string escaping.
- Evidence: proof 408 showed SyntaxError in /tmp/homepage-router-moscow-apply-fixed.py.
- Actual impact: YAML was not changed, so Homepage green dot could not appear.
- Recovery: stop applying blind updater; first discover exact active YAML shape, then apply minimal YAML-aware edit.
- Rule: validate generated Python syntax locally before scp/remote execution; avoid f-strings in generated maintenance scripts.

## FORUM_PROD_VM160_FIRST_BOOT_SSH255_20260701
- Context: clean rebuilt VM160 first boot.
- Symptom: PING_OK=yes and TCP22_OK=yes, but SSH_RC=255.
- Rule: do not proceed with OS baseline until SSH failure is diagnosed; likely old known_hosts key or cloud-init/root-key issue.

## FORUM_PROD_VM160_NESTED_HOSTNAME_PROOF_QUOTING_20260701
- Context: VM160 first SSH proof after rebuild.
- Issue: command substitution $(hostname) inside nested ssh was expanded on pve02 before entering VM160.
- Impact: VM_HOSTNAME=pve02 in proof 426 is not valid VM identity evidence.
- Rule: for nested SSH identity checks, run literal hostname commands without local command substitution.

## FORUM_PROD_VM160_SWAP_PROOF_QUOTING_20260701
- Context: VM160 swapfile proof 429.
- Issue: nested command substitution expanded on pve02, producing invalid SWAPFILE/FSTAB proof lines.
- Impact: proof 429 is not valid closure evidence even though swap was active.
- Rule: avoid nested $(...) in VM SSH proofs; use literal remote commands and clean proof.

## FORUM_UPLOAD_ARCHIVES_VM_VERIFY_QUOTING_20260701
- Context: proof 446 copy/check archives inside VM160.
- Issue: nested SSH quoting expanded shell variables incorrectly, producing gzip checks against empty .gz and blank TAR_TOP lines.
- Impact: SCP_TO_VM_RC=0 and file size list are valid, but GZIP/TAR proof lines in 446 are invalid.
- Rule: verify VM archive integrity locally inside forum-prod, not through nested quoted loops.

## FORUM_PROD_BULK_IMPORT_PHP85_EMPTY_FRONTEND_20260701
- Context: rebuilt VM160, five XenForo forums imported in bulk on PHP 8.5.7.
- Symptom: all five forums returned HTTP 200 with empty body; XenForo logged Template public:PAGE_CONTAINER/forum_list is unknown.
- Additional failed approach: proof 480 ran xf-dev:rebuild-caches on CodeVipe and returned RC=1, not a valid fix.
- Decision: stop bulk repair, treat current VM160 as lab, find a working CodeVipe recipe first, then destroy/rebuild cleanly for all five forums.
- Rule: no more mass rebuild/repair commands across all forums until single-forum CodeVipe recipe is proven.

## XENFORO_REQUIREMENTS_DIRECT_DOWNLOAD_403_20260701
- Context: Tried to download XenForo requirements ZIP directly from xenforo.com inside forum-prod.
- Issue: curl returned HTTP 403; unzip then failed because the ZIP was not downloaded.
- Evidence: proof 491 showed DOWNLOAD_RC=22, ZIP_LIST_RC=9, UNZIP_RC=9.
- Impact: proof 491 is not a valid server compatibility test.
- Rule: use operator-uploaded xenforo23-requirements-test.zip from /root/forum-upload instead of relying on direct curl download.

## PVE01_UNZIP_MISSING_FOR_XF_REQUIREMENTS_VERIFY_20260701
- Context: proof 492 tried to inspect uploaded xenforo23-requirements-test.zip on pve01.
- Issue: pve01 does not have unzip installed, so ZIP_TEST_RC=127 was not a ZIP integrity result.
- Impact: proof 492 confirmed file presence only, not archive validity.
- Rule: verify ZIP with Python zipfile or inside forum-prod rather than installing unzip on Proxmox just for inspection.

## FRESH5_DEPLOY_SUCCESS_20260701
- Context: VM160 was destroyed/recreated, then five XenForo forums were deployed from fresh ZIP backups.
- Result: proof 513 confirms all five forums locally healthy.
- Note: proof 511 initially failed dsmods path detection; proof 512 fixed dsmods by locating public_html via src/XF.php marker.
- Rule: future restores should preserve internal_data/code_cache or rebuild compiled template cache before smoke testing.

## NPMPLUS_SQLITE_PASTE_FAILURE_20260701
- Context: NPMplus SQLite inspection command was too complex and a Python fragment was pasted into bash.
- Issue: shell entered multiline prompt and produced syntax errors.
- Impact: do not trust that SQLite inspection attempt.
- Rule: avoid nested SSH heredoc/Python for this task; use simpler commands, generated nginx configs, or uploaded/local script files.

## NPMPLUS_PUBLISH_SCRIPT_WRONG_ENV_SOURCE_525_20260701
- Context: proof 525 tried to create forum proxy hosts in NPMplus.
- Issue: script sourced /opt/npmplus/.env, but INITIAL_ADMIN_EMAIL/PASSWORD were not present there; they are in the npmplus container env.
- Impact: no forum proxy hosts were created by proof 525.
- Rule: read NPMplus API login values from docker inspect env internally, never print them.

## NPMPLUS_API_LOGIN_MISSED_HTTPS81_AND_JSON_ESCAPE_20260701
- Context: NPMplus API login attempts in proofs 526/527 failed.
- Issue: commands missed the likely HTTPS admin endpoint on port 81 and built JSON by string interpolation instead of jq escaping.
- Impact: no proxy hosts were created by 526/527.
- Rule: for NPMplus API use jq-generated JSON and test https://127.0.0.1:81/api/tokens with -k before publish.

## NPMPLUS_API_AUTH_UNAVAILABLE_MANUAL_ROUTES_20260701
- Context: NPMplus API login attempts failed even though the forum VM and edge-to-forum connectivity are healthy.
- Issue: API credentials from container initial env are not accepted by current NPMplus API.
- Impact: do not use NPMplus API for this publish path.
- Rule: publish forum routes through backed-up manual nginx proxy_host configs, then issue proper certificates after DNS points to edge.

## EDGE_CERTBOT_TOKEN_FILE_PERMISSION_535_20260701
- Context: proof 535 attempted DNS-01 certificate issue for five forum domains.
- Issue: /tmp/forum_cf_token.env was root-owned mode 600, so debian user could not source it directly.
- Impact: certbot ran without token and all certificate attempts failed; DNS cutover must not proceed from proof 535.
- Rule: read Cloudflare token via sudo without printing it, then create temporary certbot credentials file and remove it after use.

## EDGE_CERTBOT_FAILED_DNS_CUTOVER_ABORTED_20260701
- Context: edge manual routes are working locally, but certbot DNS-01 certificate issue failed in proof 538.
- Issue: all five certbot attempts returned RC=1; proof 539 correctly aborted DNS cutover because edge cert precheck was not ready.
- Impact: Cloudflare DNS records were not changed by proof 539; public DNS still points to old IP 87.236.18.45.
- Rule: inspect certbot logs and Cloudflare token zone permissions before retrying certificate issuance or DNS cutover.

## CLOUDFLARE_TOKEN_NO_ZONE_ACCESS_20260701
- Context: Cloudflare token discovered on edge verified as a token but could not access forum zones.
- Evidence: proof 542 showed all five zones missing and DNS record create probes failed.
- Impact: certbot DNS-01 failed and DNS cutover was correctly aborted; public DNS still points to old hosting.
- Rule: install a new Cloudflare token with Zone Read and DNS Edit for the five forum zones before retrying certs/DNS.

## FORUM_PUBLICATION_FINAL_SUCCESS_20260701
- Context: five XenForo forums were rebuilt on VM160, routed through edge NPMplus, issued Let’s Encrypt certs using a corrected Cloudflare token, and cut over in DNS.
- Result: final public proof 546 passed.
- Important corrections: initial NPMplus API route failed; manual Nginx routes were used. Initial Cloudflare token had no zone access; corrected token passed zone/DNS probe.

## FINAL_SNAPSHOT_NAME_TOO_LONG_556_20260701
- Context: proof 556 final health gate passed for all five public forums.
- Issue: Proxmox snapshot failed because generated snapshot name exceeded 40 characters.
- Evidence: qm snapshot returned snapname value may only be 40 characters long.
- Impact: forum health was OK, but proof 556 snapshot step was not completed.
- Fix: rerun snapshot with short name.

## CLOUDFLARE_DNS_AUDIT_TXT_SUBSTRING_GAP_20260701
- Context: prior Cloudflare DNS audit checked web A/AAAA/CNAME state for root/www records.
- Issue: it did not deeply inspect TXT/SPF contents for old provider IP substrings.
- Evidence: user screenshot showed SPF TXT containing old 87.236.18.* value.
- Impact: web routing is OK, but mail-related DNS may still contain stale provider data.
- Rule: perform full DNS record audit including TXT/SPF/DKIM/DMARC/MX/CDN and substring search for old IPs before DNS cleanup.

## CLOUDFLARE_DNS_CLEANUP_DRYRUN_QUOTE_563_20260701
- Context: SPF cleanup dry-run command 563 had broken shell quoting and left the terminal at multiline prompt.
- Impact: proof 563 is invalid and no DNS cleanup was performed by it.
- Rule: use copied scripts for Cloudflare DNS mutation instead of fragile inline nested quotes.

## SPF_DUPLICATE_AFTER_565_20260701
- Context: SPF cleanup command 565 attempted to replace stale SPF records.
- Issue: backup write permission problem caused record detection failure, so new v=spf1 -all records were created while old v=spf1 ip4:87.236.18.45 records remained.
- Impact: domains temporarily had duplicate SPF records, which is invalid for mail validation.
- Fix: delete all SPF TXT records for the five forum zones, then create exactly one v=spf1 -all TXT per zone.

## MSMTP_SECRET_SOURCE_PARSE_ERROR_576_20260701
- Context: proof 576 installed msmtp but sendmail auth test failed.
- Issue: [SENSITIVE_PATH] was sourced as shell, but SMTP password contained shell-special characters; source failed and auth used an invalid/truncated secret path.
- Impact: msmtp package installed, but mail sending was not proven working.
- Rule: never source SMTP secret files containing arbitrary passwords; store password base64 and use msmtp passwordeval helper accessible to www-data.

## FORUM_MSMTP_MAIL_TRANSPORT_STILL_FAILING_20260701
- Context: attempted to fix msmtp config with passwordeval helper.
- Issue: one or more sendmail/PHP mail tests still failed.
- Impact: XenForo mail sending is not yet proven.
- Next step: verify SMTP credential/password and provider policy for noreply@pvepro.ru.

## SMTP_PASSWORD_EXPOSED_ROTATE_REQUIRED_20260701
- Context: SMTP password was exposed in terminal output during failed msmtp setup.
- Cause: secret file with raw SMTP_PASS was later parsed/sourced by shell; shell error printed the raw assignment line.
- Impact: treat that SMTP password as compromised.
- Required action: rotate the mailbox/app password in the mail provider panel before any further SMTP testing.
- Rule: never store arbitrary passwords as shell-sourceable KEY=value files; use non-printing secret storage only.

## NETBIRD_MAILCOW_REPORTED_DOWN_AFTER_SMTP_INCIDENT_20260701
- Context: user reported NetBird and Mailcow became unavailable after forum mail-transport work.
- Impact: treat as active incident until service reachability and container/VM state are proven.
- Rule: do not change DNS, firewall, proxy, mail, NetBird, or Mailcow configs before read-only triage proof.

## FORUM_SMTP_FAILED_CONFIG_DISABLED_AFTER_MAILCOW_BAN_20260701
- Context: forum-prod SMTP/msmtp test used invalid or compromised credentials and triggered Mailcow netfilter warnings/ban for 95.84.154.183.
- Impact: forum mail transport is not proven and must remain disabled until SMTP password is rotated and tested once carefully.
- Action: remove forum-prod msmtp secret/config files so XenForo cannot keep retrying broken SMTP auth.
- Rule: do not test SMTP auth again until a fresh rotated mailbox/app password is available.

## FORUM_OLD_CODEVIPE_CLOUD_BACKUP_STALE_AFTER_FRESH5_20260701
- Context: pve02 codevipe-cloud-backup.timer existed before fresh five-forum rebuild.
- Issue: old script points to /root/backups/codevipe and old remote path app/codevipe/forum-prod/local.
- Evidence: latest service run failed with missing /root/backups/codevipe; remote inventory contains old codevipe-public/codevipe.sql files, not current five-forum archive.
- Impact: old timer must not be treated as valid current backup for all five forums.
- Fix: create a new forum-prod fresh5 cloud backup using existing pve02 Mail-cloud rclone crypt remotes.

## FORUM_RESTORE_DRILL_610_STATUS_FLAG_BUG_20260701
- Context: restore drill proof 610 downloaded Mail.ru Cloud backup, verified split part SHA256, reconstructed archive, matched full archive SHA256, and confirmed all five forum DB/file archives in tar.
- Issue: final RESTORE_DRILL_OK was set to no because the script tested an unassigned shell variable RCLONE_REMOTE_PVE02_MAIL_01_RC.
- Impact: proof 610 final status flag is invalid, but its detailed integrity checks are valid.
- Fix: rerun corrected restore validation against the downloaded cloud backup artifacts and produce proof 612.

## FORUM_XENFORO_MAIL_SMOKE_TEST_FAILED_20260701
- XenForo-level mail smoke test did not return success for all five forums.
- Check proof 623 and msmtp log before retrying.

## XENFORO_MAIL_SMOKE_623_INVALID_ROOT_PATH_20260701
- Context: custom XenForo mail smoke test 623 failed with Could not open input file /root/scripts/623_xenforo_mail_smoke.php.
- Cause: the script was run as www-data from /root/scripts; /root is not traversable by www-data.
- Impact: proof 623 is invalid and should not be used to judge mail delivery.
- Superseding evidence: user ran XenForo built-in outgoing email test; message was delivered to Yandex but placed in Spam.
- Next step: investigate deliverability/authentication headers and pvepro.ru SPF/DKIM/DMARC/PTR.

## FORUM_NEXT_CHAT_KNOWN_ERRORS_AND_CLOSED_INCIDENTS_20260701

### Closed / classified incidents
- SMTP_PASSWORD_EXPOSED_ROTATE_REQUIRED_20260701:
  - A previous bad command sourced a raw SMTP secret and printed it.
  - Treat old password as compromised.
  - Later persistent SMTP was rebuilt using safe files and verified.
  - Never print or package secrets.

- XENFORO_MAIL_SMOKE_623_INVALID_ROOT_PATH_20260701:
  - Custom mail proof 623 failed with "Could not open input file".
  - Cause: PHP was run as www-data from /root/scripts, but /root is not traversable by www-data.
  - Impact: proof 623 is invalid and must not be used to judge mail delivery.
  - Superseded by user-observed built-in XenForo test and Yandex header proof.

- FORUM_MSMTP_MAIL_TRANSPORT_STILL_FAILING_20260701 / FORUM_SMTP_FAILED_CONFIG_DISABLED_AFTER_MAILCOW_BAN_20260701:
  - Earlier SMTP attempts failed and triggered mailcow netfilter ban for forum public IP.
  - Ban was removed.
  - Persistent msmtp transport was later enabled and verified.
  - Mailcow and NetBird remained reachable after final tests.

- SPF_DUPLICATE_AFTER_565_20260701:
  - Earlier SPF cleanup created duplicate SPF records.
  - Fixed by deleting duplicates and recreating exactly one SPF per forum domain.
  - Final forum domains use v=spf1 -all because sending uses noreply@pvepro.ru.

- FORUM_RESTORE_DRILL_610_STATUS_FLAG_BUG_20260701:
  - Restore drill 610 had a status flag bug despite successful detailed checks.
  - Corrected restore drill 612 passed.

- FORUM_DBTECH_TOR_TIMEOUT_CLASSIFIED_20260701:
  - One historical timeout row per forum to check.torproject.org.
  - DBTech/Security active, but dbtech_security_tornodes=0.
  - Classified as external timeout noise, not runtime failure.

### Current non-blocking items
- Yandex placed built-in XenForo test mail in Spam despite SPF pass and DKIM pass.
- Classification: deliverability/reputation/content filtering, not server failure.
- Recommended manual action: click “Не спам!” and add/check pvepro.ru in Yandex Postmaster.

### Safety rules for next chat
- Do not print secrets.
- Do not download or upload:
  - [SENSITIVE_PATH]
  - /etc/msmtprc
  - /etc/msmtp/*
  - rclone configs
  - Cloudflare tokens
  - DB dumps
  - VM disks
  - backup archives
- For handoff, only share the two truth files and selected non-secret proof files.

## PARKED_DOMAINS_STAGE4_DNS01_PREFLIGHT_FAILED_20260701
- Context: parked-domain public apply proof 634.
- Issue: DNS-01 based apply was attempted before a clear CAN_APPLY=yes preflight result was visible in terminal output.
- Observed: stage4 stopped safely with preflight_failed; CERTBOT_PRESENT=no, CERTBOT_DNS_CLOUDFLARE_PLUGIN=no, and CF_ZONE_ACCESS=no for newfi.ru, hapusya.ru and kingofwolk.ru.
- Impact: no parked-domain NPMplus route/certificate changes were applied by stage4; public HTTPS validation still failed certificate verification.
- Rule: do not use Cloudflare DNS-01 for these parked domains unless a token with explicit zone access is installed; because DNS already points to edge, prefer a copied-script HTTP-01 flow with temporary ACME challenge routes and content-specific validation.

## PARKED_DOMAINS_STAGE5_BASH_LOCAL_SETU_BUG_20260701
- Context: parked-domain HTTP-01 apply proof 635.
- Issue: edge script used Bash `local id="$1" ... conf="$WORK/.../$id.conf"` and `local host="$1" ... tmp="$WORK/.../$host.html"` under `set -u`; dependent variables are not safe inside the same local assignment command.
- Observed: `id: unbound variable`, `source can not be empty`, then `host: unbound variable`; certbot did not run, final HTTPS routes were not installed.
- Impact: local parked page remained OK; public HTTPS remained not closed.
- Rule: split dependent Bash local assignments into separate statements, syntax-check copied scripts before execution, and keep content-specific validation for parked routes.

## PARKED_DOMAINS_STAGE6_TEMP_HTTP_ROOT_VALIDATION_BUG_20260701
- Context: parked-domain HTTP-01 fixed apply proof 636.
- Issue: temporary HTTP-01 route validation checked HTTP `/` for parked marker, but expected behavior before certificates is root HTTP 301 to HTTPS.
- Observed: LOCAL_HTTP_VALIDATE returned HTTP_CODE=301 MARKER_OK=no, script rolled back its temporary route configs and did not run certbot.
- Impact: local parked page remained OK; public HTTPS remained not closed.
- Rule: validate `/.well-known/acme-challenge/<token>` for HTTP-01 readiness, not HTTP `/`; also scan all active NPMplus nginx config dirs for unmanaged domain conflicts before writing routes.

## PARKED_DOMAINS_STAGE7_NEWFI_ROOT_ACME_REDIRECT_20260701
- Context: parked domains HTTP-01 Stage7 stopped before certbot because ACME validation path returned 301 for root newfi.ru while www.newfi.ru, hapusya.ru, www.hapusya.ru, kingofwolk.ru and www.kingofwolk.ru returned 200.
- Impact: no final parked-domain HTTPS routes/certificates were applied by Stage7; temporary route configs were rolled back.
- Rule: do not run another apply stage until active NPMplus nginx route selection for newfi.ru is diagnosed with nginx -T and Host-header probes.

## PARKED_DOMAINS_STAGE8_DEFAULT_HTTP_TO_HTTPS_REDIRECT_20260701
- Context: parked-domain route autopsy proof 638.
- Finding: no active NPMplus config contains newfi.ru/hapusya.ru/kingofwolk.ru, but unknown HTTP hosts and ACME paths return 301 to HTTPS from NPMplus default routing.
- Impact: HTTP-01 cannot work through the default server. A dedicated temporary ACME server block must be proven before certbot; do not run another certbot apply without a successful ACME-path probe for all six hostnames.
- Rule: after every parked-domain temp route attempt, rollback the route and prove rollback before continuing.

## PARKED_DOMAINS_STAGE9_NEWFI_EXACT_PROBE_NEEDED_20260701
- Context: parked domains HTTP-01 probe stage9 installed one combined temporary server block for all six hostnames.
- Issue: five hostnames returned ACME probe 200, but root newfi.ru still returned 301.
- Impact: do not continue certbot/apply until exact newfi.ru route behavior is isolated.
- Rule: run a temporary exact newfi.ru probe block and roll it back before any HTTP-01 apply.

## PARKED_DOMAINS_STAGE10_EXACT_NEWFI_PROBE_OK_20260701
- Context: parked-domain Stage10 proof 640.
- Result: exact server block for newfi.ru on NPMplus intercepted both root and ACME paths with HTTP 200 and custom X-Parked-Probe markers.
- Stage9 combined server_name block passed 5/6 but failed bare newfi.ru, so final HTTP-01 cutover must use separate exact per-host server blocks.
- Rule: do not use one combined parked-domain server_name block for these domains; use exact host blocks and content-specific validation.

## PARKED_DOMAINS_STAGE11_LOW_ID_EXACT_ROUTE_FAILED_20260701
- Context: parked-domain Stage11 proof 641.
- Result: exact per-host route files 700-705 passed nginx -t but bare newfi.ru still hit HTTP->HTTPS redirect on ACME path.
- Earlier Stage10 proof showed exact newfi.ru route works when placed at high id 998.
- Hypothesis: file/include priority or NPMplus generated ordering makes low id 700 unsuitable for bare newfi.ru in this runtime.
- Rule: do not retry parked-domain HTTP-01 with low id 700-705. First prove fixed high-id exact blocks, then apply with those exact ids.

## PARKED_DOMAINS_SWITCH_TO_DNS01_AFTER_HTTP01_FAILURES_20260701
- Context: parked domains newfi.ru, hapusya.ru and kingofwolk.ru.
- HTTP-01 attempts hit NPMplus default HTTP-to-HTTPS redirect/include-order problems.
- Stage12 cleanup confirmed temp files 980-985 absent and nginx reload OK.
- Rule: do not continue HTTP-01 for these parked domains; use DNS-01 with a dedicated Cloudflare token that has Zone Read and DNS Edit for the three zones.

## PARKED_DOMAINS_STAGE15_FALSE_LOCAL_VALIDATE_ROLLBACK_20260701
- Context: Stage15 DNS-01 succeeded and public validation showed parked page for most hosts, but local edge curl --resolve validation returned CURL_RC=60 and triggered rollback.
- Impact: do not rerun Stage15 as-is.
- Rule: for parked domains finalization, use existing DNS-01 certificates and validate from pve01 public HTTPS with content marker; do not use the failed local --resolve TLS check as the closure gate.

## DOMAIN_ACTIONS_STAGE18_PLACEHOLDER_AND_RENEWAL_PROOF_20260701
- Context: after parked domains and portfolio audit, operator requested removing domain enumeration from placeholder and checking certificate renewal paths.
- Rule: Stage18 must not change DNS records or NPMplus route files. It may update placeholder HTML, run existing forum renewal script once for runtime proof, and run certbot dry-run for parked certificates using existing DNS-01 hooks.

## GRAM1_ROOT_WWW_PLACEHOLDER_CLOSE_STAGE19_20260701
- Context: operator requested gram1.ru root/www to use the existing placeholder page.
- Rule: only root gram1.ru and www.gram1.ru may be routed. Existing gram1.ru subdomain NPMplus routes must not be changed.
- Rule: use DNS-01 with a dedicated gram1 Cloudflare token; do not use HTTP-01.
- Stage18 parked dry-run note: kingofwolk.ru dry-run hit Let's Encrypt rateLimited/service-busy after active certificate issuance and public HTTPS were already OK. Treat as transient external dry-run failure, not route/cert closure blocker.

## PVEPRO_EDGE_LANDING_STAGE21_20260701
- Context: pvepro.ru root/www currently point to the Mailcow VPS and return Mailcow with TLS hostname mismatch.
- Decision: because pve01 has no SSH key access to the external VPS, do not modify the VPS. Instead move only pvepro.ru and www.pvepro.ru A records to edge and serve an edge landing page.
- Safety rule: do not change mail.pvepro.ru, nb.pvepro.ru, MX, SPF, DKIM, DMARC or forum SMTP configuration.
- Use a dedicated Cloudflare token for pvepro.ru. Do not print token values.

## PVEPRO_STAGE21_LANDING_SERVICE_FAILED_20260701
- Context: Stage21 stopped safely before DNS/cert/NPMplus route because local pvepro landing service returned CURL_RC=7 on 127.0.0.1:18089.
- Impact: DNS root/www pvepro.ru remained on 185.139.214.215 and no route/cert changes were applied.
- Rule: Stage22 must fix and prove the local landing service first, then proceed to DNS-01 cert, NPMplus route and root/www DNS only after local marker PVEPRO_LANDING_OK is reachable.

## PVEPRO_STAGE22_VALIDATION_AND_COPY_GUARD_20260701
- Context: Stage22 succeeded on edge-side landing, cert, NPMplus route and Cloudflare A upserts, but wrapper had a non-critical scp wildcard failure due __pycache__ and public validation used cached DNS for root pvepro.ru.
- Mistakes recorded: do not scp scripts/* after local py_compile because __pycache__ may be present; do not continue after critical scp failure in future apply stages; do not classify PVEPro landing as NetBird just because it contains a link text; use Cloudflare authoritative A records and --resolve validation when DNS propagation may lag.
- Stage23 must be read-only for infrastructure and may close only if edge route, Cloudflare A records, HTTPS landing marker, and mail/nb unchanged checks pass.

## TAFTAUTO_CERT_AUTORENEW_BLOCKED_NO_PRIVATE_ACCESS_20260701
- Context: taftauto.ru points to the dacha router public IP. Operator confirmed the router model is Netcraze-like, same family as Moscow router.
- Current limitation: there is no safe private remote access path to the dacha router yet.
- Rule: do not expose router admin publicly and do not attempt certificate deployment to the router over the public internet.
- Closure status: certificate autodeploy is intentionally blocked until VPN/NetBird/WireGuard or another private management path exists.

## TAFTAUTO_CERT_AUTORENEW_BLOCKED_NO_PRIVATE_ACCESS_20260701
- Context: taftauto.ru points to the dacha router public IP. Operator confirmed the router model is Netcraze-like, same family as Moscow router.
- Current limitation: there is no safe private remote access path to the dacha router yet.
- Rule: do not expose router admin publicly and do not attempt certificate deployment to the router over the public internet.
- Closure status: certificate autodeploy is intentionally blocked until VPN/NetBird/WireGuard or another private management path exists.

## TAFTAUTO_WG_STAGE24_SECRET_AND_IMPORT_RULE_20260701
- Context: configuring private management path for taftauto.ru dacha router.
- Rule: do not print WireGuard private keys, PSK, or client config body in proof/chat.
- Rule: generated client config may be stored only as root/debian mode 600 file and must be imported into the Netcraze router UI manually.
- Rule: do not open router public admin or certificate deployment until WireGuard handshake is proven.

## TAFTAUTO_WG_PSK_ROTATION_SCRIPT_BROKE_DATAPLANE_20260702
- Assistant gave unsafe PSK rotation flow; after profile reimport dacha interface changed from Wireguard0 to Wireguard1 and data-plane broke until security-level, ACL and routes were restored on Wireguard1.
- Proof 666 is superseded because it showed PUBLIC_SSH_22_STILL_OPEN.
- Final fixed proof: /root/evidence/667_TAFTAUTO_WG_PSK_ROTATED_PUBLIC_CLOSED_OK_20260702_PROOF.txt

## TAFTAUTO_CERTBOT_CLOUDFLARE_PLUGIN_MISSING_20260702
- Certbot on edge rejected --dns-cloudflare-* arguments: dns-cloudflare plugin missing/not loaded.
- No taftauto certificate issued in failed run; no DNS A records or NPMplus routes changed.

## HOMEPAGE_SERVICES_YAML_BAD_INDENT_20260702
- Assistant inserted Homepage services.yaml entries with wrong indentation; YAMLException bad indentation at line 227.
- Fix: rebuild SmartApe under External Homelabs with 2-space item indentation, remove Router Moscow, rebuild Sites category.

## HOMEPAGE_SERVICES_YAML_BAD_INDENT_REPAIR_20260702
- Previous apply broke services.yaml indentation and did not follow YAML-aware rule.
- Repair uses latest valid services.yaml backup, PyYAML parse/dump, validates before restart.

## TAFTAUTO_CERTBOT_DRYRUN_BAD_FLAG_20260702
- Assistant used unsupported certbot flag --manual-public-ip-logging-ok with certbot 4.0.0 during taftauto renew dry-run.
- No certificate, DNS, NPMplus route, or deploy state changed by the failed dry-run.

## TAFTAUTO_CERT_RENEW_DRYRUN_RATE_LIMITED_20260702
- Proof 672 is superseded/partial: it printed STATUS=OK even though certbot dry-run failed.
- Failure reason: Let's Encrypt staging/service returned rateLimited / Service busy; retry later.
- Cloudflare manual auth and cleanup hooks did run successfully.
- Deploy hook was installed and manually invoked successfully before the dry-run.
- Do not retry immediately.

## HOMEPAGE_XENFORO_INFO_INLINE_PYTHON_SYNTAX_ERROR_20260702
- Assistant used invalid inline Python syntax with def after semicolon while adding XenForo.info to Homepage External.
- Failed before services.yaml write.

## HOMEPAGE_USEFUL_ROUTER_NAME_MISMATCH_20260702
- Previous apply looked for Russian Router card name, but active Homepage Useful group contains Router and Public Domain.
- Failed before write. Fix uses YAML-aware edit with exact names Router/Public Domain.

## HOMEPAGE_USEFUL_ROUTER_CARD_NOT_FOUND_20260702
- Apply expected Useful -> Router in active services.yaml, but YAML parse did not find it. No write was performed.
- Need verify active Homepage config path, docker mount, and any alternate services.yaml before next apply.

## 20260702_CF_TOKEN_AUDIT_BROKEN_COMMAND
- Ошибка: ассистент дал хрупкую команду аудита Cloudflare-токенов с вложенным heredoc/Python через ssh.
- Факт: bash начал интерпретировать Python-код, появились syntax error и вторичный prompt >.
- Правило: не давать nested heredoc/сложные кавычки через ssh; для длинной логики сначала класть скрипт отдельным файлом, затем запускать его.
- Статус: аудит токенов не выполнен, инфраструктурные изменения не должны считаться применёнными.

## LESSON_20260702_CROWDSEC_NETBIRD_EXIT_ROUTE
- Do not treat e3qxxx.netbird.selfhosted / 100.100.125.70 and e3qxxx-183-106.netbird.selfhosted / 100.100.183.106 as VPS egress nodes; they are user mobile peers.
- Correct egress candidates are relay.pvepro.ru / relay.netbird.selfhosted / 100.100.19.1 for Moldova and mail.pvepro.ru / mail.netbird.selfhosted / 100.100.147.204 for USA.
- Access Policy edge-vm -> relay/mail only proves peer reachability. It is not an internet exit route.
- Required NetBird fix for CrowdSec CAPI was Network Routing exit route 0.0.0.0/0 distributed to edge-vm group, routing peer relay, masquerade/NAT enabled.
- Before attempting CrowdSec CAPI, verify on edge-vm: ip route get 1.1.1.1 uses wt0, wg allowed-ips contains 0.0.0.0/0, and Cloudflare trace no longer shows home IP 95.84.154.183.

## LESSON_20260702_HOMEPAGE_MONITORS_REPAIR_NOT_DELETE
- Do not delete or disable Homepage siteMonitor fields to hide red badges.
- Fix the underlying monitor path, DNS, proxy route, or local health-wrapper instead.
- Do not touch Cloudflare Homepage card when operator says it is green and opens correctly.
- Before changing Homepage, identify the live config host; in this state live Homepage is on core-apps [PRIVATE_IP], not edge-vm.

## LESSON_20260702_KUMA_ADD_ONE_MONITOR_ONLY
- When asked to fix one missing Uptime Kuma service, add only one monitor, then generate proposals separately.
- Before direct Kuma DB mutation, stop the container and create a DB backup.
- Verify DB integrity before starting Kuma again.
- Do not touch Cloudflare when operator says it is green and opens correctly.

## LESSON_20260702_DOCKGE_STALE_STACKS_NOT_CONTAINERS
- Dockge inactive items after migration can be stale compose folders, not stopped containers.
- First classify runtime projects across all Docker hosts before deleting or archiving anything.
- For remote Dockge visibility, projects must be under /opt/stacks or bind-mounted there.
- Do not delete containers to fix Dockge inactive UI; connect agents and archive only confirmed stale moved stack definitions.

## STAGE4B_MASKED_SQL_QUERY_FAILURE_20260714
- Previous Stage4B read-only audit returned empty observation-summary, metadata-key, trigger and function blocks because nested SSH/SQL quoting broke queries while stderr was hidden.
- Empty blocks are query failures, not proof that metadata, triggers or functions are absent.
- Future PostgreSQL audits must use ON_ERROR_STOP, visible stderr, independent query RC and quote-safe dollar literals.

## STAGE4C_SCHEMA_MIGRATIONS_ID_ASSUMPTION_20260714
- Stage4C read-only preflight incorrectly assumed schema_migrations has a column named id.
- PostgreSQL returned column id does not exist and ON_ERROR_STOP correctly stopped all following SQL checks.
- Production DB and application were not changed. Future migration audits must inspect information_schema first and query migration rows without assuming column names.

## STAGE4C_PGDUMP_DEV_NULL_FSYNC_INVALID_PROBE_20260714
- Stage4C custom-format pg_dump capability probe incorrectly used /dev/null as the output file.
- pg_dump failed only because fsync on /dev/null returned Invalid argument.
- Production database and application were not changed.
- Future custom dump capability checks must use a real temporary regular file, validate it with pg_restore --list, and remove it afterward.

## STAGE4C_REMOTE_PREP_THIRD_SUBSTEP_FAILED_20260714
- Isolated migration dry-run did not start: REMOTE_PREP_RC=0|0|1.
- Remote directory creation and SCP succeeded; the third ownership/mode preparation substep returned RC=1.
- Production database and application were not changed.
- Exact cause requires read-only residual directory and permission inspection before cleanup or retry.

## STAGE4C_REMOTE_PREP_GLOB_AFTER_CHMOD_ROOT_CAUSE_20260714
- Exact cause of REMOTE_PREP_RC=0|0|1: after the directory became postgres-owned mode 0700, the unprivileged debian shell could not traverse it and could not expand the wildcard used by chmod.
- Candidate files and SHA256 verification were valid; no temporary database was created and production was unchanged.
- Future preparation must use exact privileged paths or a verified shared group, never an unprivileged wildcard after restrictive chmod.

## STAGE4C_MIGRATION003_FINGERPRINT_DOLLAR_QUOTE_FAILURE_20260714
- Migration 003 isolated dry-run failed at fingerprint CHECK because the regex end anchor collided with the SQL dollar-quote delimiter and produced an unterminated dollar-quoted string.
- The migration transaction rolled back, the temporary database was removed, and production remained unchanged.
- Fingerprint validation must use length plus translate without a regex dollar anchor; corrected candidate must pass a fresh isolated dry-run before production consideration.

## STAGE4C_SEAL_OUTER_RC_MASKING_20260714
- Stage4C final seal correctly refused creation because overall health was WARN, returning internal RC=40.
- The former wrapper then printed a second misleading COMMAND_RC=0 because top-level true masked the internal result.
- Future commands must capture the guarded subshell through PIPESTATUS and print exactly one authoritative COMMAND_RC before the mandatory final true.
- Production database, application and services were unchanged.

## STAGE4C_DIAGNOSIS_EXPECTED_DERIVED_FILE_ON_VM180_20260714
- Previous diagnosis incorrectly treated cluster-admin-restricted-probes.txt as a required source file on VM180.
- The file is a central derived health artifact on pve01; VM180 supplies full-observer evidence instead.
- This false expectation caused diagnostic RC=32 without any infrastructure failure or production change.

## STAGE4C_NO_JOBS_RUNNING_TEXT_COUNTED_AS_JOB_20260714
- Stage4C seal preflight incorrectly counted the systemctl informational text No jobs running as one active job.
- The parser counted every nonempty output line instead of accepting only rows beginning with a numeric systemd Job ID.
- No service was started and no production state changed during the blocked attempt.
- Future job counts must match a numeric first field only.

## STAGE4D_ADAPTER_SOURCE_AUDIT_ABORTED_BEFORE_FINAL_MARKERS_20260714
- The evidence-only Stage4D adapter source audit returned RC=1 before emitting its VM180 completion and final audit markers.
- The six wrapper validation failures are cascading missing-marker checks and do not prove six independent infrastructure faults.
- Production, database, application, services, timers, health and desired-state were unchanged.
- Inspect the preserved runner stdout, stderr and result JSON before correcting or rerunning the task.

## STAGE4D_LOCAL_EXPANSION_OF_REMOTE_Q_UNDER_NOUNSET_20260714
- The first Stage4D adapter source audit embedded a large remote program inside a locally double-quoted SSH argument.
- Local shell expansion reached the remote variable Q while nounset was enabled, causing Q: unbound variable before SSH execution.
- VM180 and PostgreSQL audits did not start; production state was unchanged.
- Remote shell and SQL programs must be separate syntax-checked files passed through SSH stdin.

## STAGE4D_REMOTE_PYTHON311_ADAPTER_SELFTEST_RC1_20260714
- Canonical adapter candidate passed local Python 3.13 self-tests with 48 envelopes but returned RC=1 on VM180 Python 3.11.2.
- Remote upload and cleanup succeeded, and production database remained 0|0|OK.
- The exact traceback and failing assertion must be captured by an isolated identical reproduction before changing the candidate.

## STAGE4D_PYTHON311_FSTRING_COMPATIBILITY_AND_REPRO_VALIDATOR_20260714
- Adapter candidate passed Python 3.13 but failed Python 3.11 parsing at Path(row["path"]) inside a double-quoted f-string.
- Exact fix is Python 3.11-compatible quoting: Path(row['path']).
- The reproduction command captured the expected compile failure but incorrectly returned CAPTURE_BAD=1 because its validator required compile RC=0.
- Production database remained 0|0|OK and desired-state remained clean.
- Future adapter candidates must pass syntax and full self-tests on both pve01 and VM180 runtimes before acceptance.

## STAGE4E_DESIGN_VALIDATOR_ABORTED_BEFORE_LOCAL_VALIDATION_MARKER_20260714
- Restricted probe-agent design candidate compiled successfully but its local validator exited before producing LOCAL_VALIDATION_RC.
- The wrapper validation count is cascading missing-marker evidence and does not represent twenty-one independent faults.
- Active external probes were not executed and production state was unchanged.
- Inspect the preserved validator traceback and exact assertion before modifying the candidate.

## STAGE4E_VALIDATOR_SCANNED_PYC_AND_OWN_FORBIDDEN_LITERALS_20260714
- Stage4E design validator recursively read every candidate file as UTF-8 after py_compile had created binary __pycache__ bytecode.
- This caused UnicodeDecodeError before any design assertion failed.
- The phase-wide forbidden-pattern scan also included validate_design.py itself, which necessarily contains the prohibited literals it verifies.
- Validators must use an explicit text-file allowlist and scans must exclude validator implementation and binary artifacts.
- Production, database, services, timers and desired-state were unchanged; active external probes were not executed.

## STAGE4E_V2_VALIDATOR_FAILED_STATIC_COMPILE_20260714
- Stage4E design candidate v2 failed static compilation before local design validation started.
- The reported validation count is cascading missing-marker evidence, not twenty-five independent design defects.
- The generated validator must be inspected at the exact SyntaxError line before another candidate is created.
- Production, database, application, services, timers, health and desired-state were unchanged; active external probes were not executed.

## STAGE4E_PIPESTATUS_LOST_AFTER_FIRST_ASSIGNMENT_20260714
- VM180 validation copy used a pipeline and read PIPESTATUS in two separate assignments.
- The first assignment reset PIPESTATUS, leaving COPY_RC empty and preventing remote validation and cleanup.
- Retry must use explicit SCP operations without pipeline status parsing.
- Production remained unchanged and active external probes were not executed.

## STAGE4F_VALIDATOR_EXPECTED_HEALTH_REFRESH_FROM_NONWRITING_BACKUP_SCRIPT_20260714
- The controlled backup service completed with Result=success and ExecMainStatus=0.
- Validation incorrectly required cluster-admin-incident-engine-backup.txt to receive a fresh timestamp.
- The executed backup script does not write that health file; therefore the stale health timestamp was not evidence of backup failure.
- Stage4F backup proof must use the service result, fresh latest JSON, produced artifacts and off-host/restore evidence instead.
- The backup must not be rerun merely to satisfy the invalid health-file freshness assertion.

## STRICT_RULE_20260714_CLOSE_TAILS_IMMEDIATELY
- Failure class: переход к следующей задаче при наличии незакрытого хвоста.
- Mandatory anti-regression: после ошибки запрещено продолжать другой scope до точной root cause, исправления, повторной проверки, очистки, proof и seal.
- Closure gate: UNRESOLVED_TAIL_COUNT=0; BLOCKER_COUNT=0; TEMPORARY_ARTIFACT_COUNT=0; ROLLBACK_VERIFIED=YES; HEALTH=OK; PROOF_UPDATED=YES; REFERENCE_UPDATED=YES.
- Если закрытие невозможно из-за внешней зависимости, опасной неоднозначности или обязательного решения пользователя, фиксировать BLOCKED/OPEN и не заявлять CLOSED.
- Proof: /var/lib/homelab-change-sets/operator-rule-close-tails-immediately-v1/20260714T171631Z/report.txt

## ERROR_20260714_DEPENDENCY_FACT_QUERY_GLOBAL_LIMIT
- Symptom: dependency audit вернул только latest_collector_status.
- Root cause: ORDER BY 1 LIMIT 1 в конце UNION ALL применился ко всему набору фактов.
- Correction: latest collector status извлекается скалярным подзапросом внутри CTE facts; глобальный LIMIT отсутствует.
- Anti-regression: multi-fact SQL audit обязан проверять точное ожидаемое число строк до извлечения значений.
- Closure proof: /var/lib/homelab-cluster-admin-stage4/stage4h-collector-provenance-readiness-scope-v1/report.txt
- Status: CLOSED.

## ERROR_20260714_COLLECTOR_PATCH_REQUIREMENT_PATTERN_MISMATCH
- Symptom: MIGRATION_PLAN_COLLECTOR_PATCH_REQUIRED_COUNT=1 при ожидаемых двух подтверждениях.
- Root cause: узкий шаблон учитывал collector_patch_required, но не collector_patch_required_after_stage4c.
- Correction: оба поля проверяются отдельно, затем складываются как semantic requirement count.
- Anti-regression: связанные JSON-контракты проверяются по собственным точным именам полей, а не одним унифицированным grep-шаблоном.
- Closure proof: /var/lib/homelab-cluster-admin-stage4/stage4h-collector-provenance-readiness-scope-v1/report.txt
- Status: CLOSED; PRIOR_TAIL_UNRESOLVED_COUNT=0.

## ERROR_20260714_PRIVILEGED_FILE_REDIRECTION_BEFORE_SUDO
- Symptom: bash reported Permission denied while counting collector.py lines.
- Root cause: `< collector.py` was opened by the unprivileged caller shell before sudo executed wc.
- Correction: run sudo wc -l collector.py without caller-side input redirection.
- Anti-regression: never combine sudo with `< protected-file`; pass the protected path as an argument to the privileged command.
- Closure proof: /var/lib/homelab-cluster-admin-stage4/stage4h-collector-baseline-audit-final-v1/report.txt
- Status: CLOSED.

## ERROR_20260714_COLLECTOR_RUN_COLUMN_ASSUMPTION
- Symptom: SQL failed because completed_at did not exist.
- Root cause: collector_runs columns were assumed instead of read from information_schema; actual names are finished_at and error_text.
- Correction: assert required and forbidden column counts before querying recent runs.
- Anti-regression: every schema-sensitive audit must discover and validate exact column names before the main query.
- Closure proof: /var/lib/homelab-cluster-admin-stage4/stage4h-collector-baseline-audit-final-v1/report.txt
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.

## ERROR_20260714_REMOTE_STDIN_ARGUMENT_SHIFT
- Symptom: remote harness выполнил chmod для пути bash.
- Root cause: дополнительный positional token bash был передан после SSH command и стал первым аргументом stdin-скрипта.
- Correction: remote command вызывает bash -s -- с ровно тремя явно встроенными аргументами: candidate path, SHA256 и byte count.
- Anti-regression: stdin-скрипт обязан валидировать $1 по разрешённому path pattern и затем подтверждать точные SHA256 и byte count; запрещено добавлять отдельный bash-token после remote command.
- Closure proof: /var/lib/homelab-change-sets/stage4h-collector-provenance-patch-candidate-v1/20260714T174131Z/report.txt
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.

## ERROR_20260714_OVERSIZED_COMMAND_PARSE_FAILURE
- Symptom: Bash получил одиночный фрагмент и сообщил syntax error near unexpected token `(`.
- Root cause: чрезмерно длинная интерактивная команда была обрезана или повреждена при передаче до полного разбора Bash.
- Impact: side-effect audit подтвердил production state 0|0|OK, неизменный collector, активный timer и отсутствие временных БД, каталогов и процессов.
- Correction: монолитные команды запрещены; сложные действия выполняются через отдельный проверяемый task-скрипт и короткий launcher.
- Anti-regression: interactive payload target <= 8000 bytes; syntax-check перед запуском; remote output сохраняется независимо от RC.
- Closure proof: /var/lib/homelab-change-sets/operator-command-size-guard-v1/20260714T182653Z/report.txt
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.

## ERROR_20260714_HOMELAB_ADMIN_HELP_RC_ASSUMPTION
- Symptom: template audit завершился HOMELAB_ADMIN_HELP_FAILED.
- Root cause: был ошибочно ожидаем RC=0 от неподдерживаемого аргумента --help.
- Actual contract: неизвестный аргумент печатает usage и возвращает RC=64.
- Correction: проверять version либо считать точную usage-строку вместе с RC=64 штатным результатом.
- Anti-regression: перед проверкой CLI сначала фиксировать поддерживаемые команды; не предполагать GNU-style --help.
- Closure proof: /var/lib/homelab-change-sets/homelab-admin-cli-contract-fix-v1/20260714T183338Z/report.txt
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.

## ERROR_20260715_EXPECTED_NEGATIVE_RC_TRAP_CLASSIFICATION
- Symptom: schema-v2 task v3 build stopped while intentionally testing the --invalid argument path.
- Root cause: an expected RC64 was executed while the generic ERR trap remained active.
- Correction: expected failures are evaluated only through an explicit if/else branch with immediate RC capture.
ANTI_REGRESSION_TOKEN=EXPECTED_RC64_INTERCEPTED_ERR_TRAP
- Required pattern: if command; then rc=0; else rc=$?; fi, followed by a separate exact expected-RC assertion.
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.

## ERROR_20260715_DUPLICATE_MACHINE_MARKER_COUNT
- Symptom: error-register candidate construction stopped with RC1 before applying the candidate.
- Root cause: the same marker text appeared in both a heading and a descriptive line, while the guard expected one substring occurrence.
- Correction: headings and prose do not contain machine-token values; validation counts only complete exact token lines.
ANTI_REGRESSION_TOKEN=DUPLICATE_MACHINE_TOKEN_IN_HEADING_AND_BODY
- Required pattern: grep -Fxc against a complete machine line, never grep -Foc against an unrestricted substring.
- Production impact: none.
- Temporary artifacts: removed and verified.
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
- Registered at: 2026-07-15T02:56:19Z

## ERROR_20260715_OUTER_WRAPPER_SHELL_SYNTAX
- Symptom: the operator shell rejected the generated one-line wrapper with syntax error near the final subshell parenthesis.
- Root cause: the outer wrapper contained an unbalanced compound shell construct before the closing subshell delimiter.
- Impact: parsing failed before the start marker, guards and homelab-admin invocation; no task phase or infrastructure action executed.
- Correction: use a linear wrapper with explicitly closed if/case blocks and avoid nested brace groups in conditional expressions.
ANTI_REGRESSION_TOKEN=OUTER_WRAPPER_UNBALANCED_SUBSHELL_SYNTAX
- Anti-regression: after any parser-level failure, treat the attempted phase as not executed and verify package and run-state before retrying.
- Production impact: none.
- Task package impact: none.
- Temporary artifacts: none.
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
- Registered at: 2026-07-15T03:24:05Z

## ERROR_20260715_INLINE_REMOTE_SHELL_IN_APPLY_PHASE
- Symptom: homelab-admin lint rejected Stage4H task v6 with ERR-002-INLINE-REMOTE-SHELL.
- Root cause: apply.sh invoked bash -s through an inline quoted SSH program and streamed the remote script through stdin.
- Correction: copy the declared remote script to an isolated fixed path on VM180 and invoke that file directly with positional arguments.
ANTI_REGRESSION_TOKEN=INLINE_REMOTE_SHELL_REPLACED_BY_STAGED_REMOTE_SCRIPT_PATH
- Anti-regression: phase scripts may perform transport orchestration, but remote shell program bodies must exist as separately declared, syntax-checked remote script files.
- Remote stdout and stderr must be preserved before evaluating the remote return code.
- Task v6 status: REJECTED_BY_LINT; never executed and never mutated.
- Production database impact: none.
- Live collector impact: none.
- Temporary database and remote root count after rejection: zero.
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
- Registered at: 2026-07-15T04:13:46Z

## ERROR_20260715_OPTIONAL_RUN_DIRECTORY_FIND_UNDER_PIPEFAIL
- Symptom: task-v7 builder stopped while counting formal runs for rejected task v6.
- Root cause: find was executed against an optional directory that did not exist; under pipefail the assignment returned RC1.
- Correction: test directory existence first and assign zero without invoking find when it is absent.
ANTI_REGRESSION_TOKEN=MISSING_OPTIONAL_RUN_DIRECTORY_TREATED_AS_ZERO
- Anti-regression: optional paths must have an explicit existence branch before find under pipefail.
- Production impact: none.
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
- Registered at: 2026-07-15T04:44:06Z

## ERROR_20260715_PG_CONSTRAINT_CONTYPE_CHAR_CONCAT
- Symptom: isolated Stage4H acceptance failed during schema baseline capture.
- Root cause: pg_constraint.contype uses the internal PostgreSQL char type and was concatenated without an explicit text cast.
- Correction: use contype::text or CAST(contype AS text).
ANTI_REGRESSION_TOKEN=PG_CATALOG_CHAR_CONCAT_REQUIRES_EXPLICIT_TEXT_CAST
- Machine rule: ERR-014-PG-CATALOG-CHAR-CONCAT.
- Negative self-test: uncast expression rejected with RC64.
- Positive self-test: explicit text cast accepted with RC0.
- Task v7 mutated: no.
- Production impact: none.
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
- Registered at: 2026-07-15T06:27:06Z

## ERROR_20260715_LIVE_COLLECTOR_VERIFIER_WRONG_PATH
- Symptom: verified linter installation was rolled back after the final collector integrity guard returned UNKNOWN.
- Root cause: verifier queried a nonexistent path instead of the canonical live collector path declared by the task acceptance script.
- Canonical path: /opt/cluster-admin-incident-engine/collector.py.
- Incorrect path: /opt/homelab-cluster-admin/cluster-admin-collector.py.
ANTI_REGRESSION_TOKEN=LIVE_COLLECTOR_VERIFIER_PATH_MUST_MATCH_TASK_CANONICAL_PATH
- Correction: derive and guard the live collector path from the immutable task contract before executing the hash query.
- Production impact: none; the canonical collector hash remained unchanged.
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
- Registered at: 2026-07-15T06:27:06Z
----- CONTENT END -----

===== PROXMOX VERSION =====
proxmox-ve: 9.2.0 (running kernel: 7.0.12-1-pve)
pve-manager: 9.2.3 (running version: 9.2.3/d0fde103346cf89a)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0: 7.0.12-1
proxmox-kernel-7.0.12-1-pve-signed: 7.0.12-1
proxmox-kernel-7.0.6-2-pve-signed: 7.0.6-2
proxmox-kernel-7.0.2-6-pve-signed: 7.0.2-6
ceph-fuse: 19.2.3-pve4
corosync: 3.1.10-pve2
criu: 4.1.1-1
frr-pythontools: 10.6.1-1+pve2
ifupdown2: 3.3.0-1+pmx12
intel-microcode: 3.20251111.1~deb13u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.1
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.1
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.1.6
libpve-cluster-perl: 9.1.6
libpve-common-perl: 9.1.16
libpve-guest-common-perl: 6.0.4
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.6
libpve-notify-perl: 9.1.6
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.6
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-1
proxmox-backup-client: 4.2.2-1
proxmox-backup-file-restore: 4.2.2-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.3
proxmox-kernel-helper: 9.2.0
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.5
pve-cluster: 9.1.6
pve-container: 6.1.10
pve-docs: 9.2.2
pve-edk2-firmware: 4.2025.05-2
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.18-4
pve-ha-manager: 5.2.4
pve-i18n: 3.8.0
pve-qemu-kvm: 11.0.0-4
pve-xtermjs: 6.0.0-1
qemu-server: 9.1.17
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.2-pve1

===== CLUSTER STATUS =====
Cluster information
-------------------
Name:             homelab
Config Version:   3
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Tue Jul 21 09:11:09 2026
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000001
Ring ID:          1.9d
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2  
Flags:            Quorate 

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 [PRIVATE_IP] (local)
0x00000002          1 [PRIVATE_IP]
0x00000003          1 [PRIVATE_IP]

===== CLUSTER NODES =====
[
   {
      "id" : "cluster",
      "name" : "homelab",
      "nodes" : 3,
      "quorate" : 1,
      "type" : "cluster",
      "version" : 3
   },
   {
      "id" : "node/pve03",
      "ip" : "[PRIVATE_IP]",
      "level" : "",
      "local" : 0,
      "name" : "pve03",
      "nodeid" : 2,
      "online" : 1,
      "type" : "node"
   },
   {
      "id" : "node/pve01",
      "ip" : "[PRIVATE_IP]",
      "level" : "",
      "local" : 1,
      "name" : "pve01",
      "nodeid" : 1,
      "online" : 1,
      "type" : "node"
   },
   {
      "id" : "node/pve02",
      "ip" : "[PRIVATE_IP]",
      "level" : "",
      "local" : 0,
      "name" : "pve02",
      "nodeid" : 3,
      "online" : 1,
      "type" : "node"
   }
]

===== CLUSTER RESOURCES =====
[
  {
    "maxcpu": 1,
    "maxdisk": 8350298112,
    "maxmem": 536870912,
    "name": "dns1",
    "node": "pve01",
    "plugintype": null,
    "status": "running",
    "storage": null,
    "template": 0,
    "type": "lxc",
    "vmid": 110
  },
  {
    "maxcpu": 1,
    "maxdisk": 8350298112,
    "maxmem": 536870912,
    "name": "dns2",
    "node": "pve02",
    "plugintype": null,
    "status": "running",
    "storage": null,
    "template": 0,
    "type": "lxc",
    "vmid": 111
  },
  {
    "maxcpu": 1,
    "maxdisk": 8350298112,
    "maxmem": 536870912,
    "name": "unbound1",
    "node": "pve01",
    "plugintype": null,
    "status": "running",
    "storage": null,
    "template": 0,
    "type": "lxc",
    "vmid": 112
  },
  {
    "maxcpu": 1,
    "maxdisk": 8350298112,
    "maxmem": 536870912,
    "name": "unbound2",
    "node": "pve02",
    "plugintype": null,
    "status": "running",
    "storage": null,
    "template": 0,
    "type": "lxc",
    "vmid": 113
  },
  {
    "maxcpu": 4,
    "maxdisk": 103079215104,
    "maxmem": 12884901888,
    "name": "edge-vm",
    "node": "pve03",
    "plugintype": null,
    "status": "running",
    "storage": null,
    "template": 0,
    "type": "qemu",
    "vmid": 130
  },
  {
    "maxcpu": 4,
    "maxdisk": 68719476736,
    "maxmem": 8589934592,
    "name": "nextcloud",
    "node": "pve01",
    "plugintype": null,
    "status": "running",
    "storage": null,
    "template": 0,
    "type": "qemu",
    "vmid": 150
  },
  {
    "maxcpu": 4,
    "maxdisk": 214748364800,
    "maxmem": 8589934592,
    "name": "forum-prod",
    "node": "pve02",
    "plugintype": null,
    "status": "running",
    "storage": null,
    "template": 0,
    "type": "qemu",
    "vmid": 160
  },
  {
    "maxcpu": 4,
    "maxdisk": 128849018880,
    "maxmem": 10737418240,
    "name": "core-apps",
    "node": "pve01",
    "plugintype": null,
    "status": "running",
    "storage": null,
    "template": 0,
    "type": "qemu",
    "vmid": 170
  },
  {
    "maxcpu": 4,
    "maxdisk": 128849018880,
    "maxmem": 8589934592,
    "name": "monitoring",
    "node": "pve01",
    "plugintype": null,
    "status": "running",
    "storage": null,
    "template": 0,
    "type": "qemu",
    "vmid": 171
  },
  {
    "maxcpu": 2,
    "maxdisk": 25769803776,
    "maxmem": 2147483648,
    "name": "cluster-admin",
    "node": "pve02",
    "plugintype": null,
    "status": "running",
    "storage": null,
    "template": 0,
    "type": "qemu",
    "vmid": 180
  },
  {
    "maxcpu": 4,
    "maxdisk": 85899345920,
    "maxmem": 8589934592,
    "name": "homelab-ops-worker",
    "node": "pve03",
    "plugintype": null,
    "status": "running",
    "storage": null,
    "template": 0,
    "type": "qemu",
    "vmid": 190
  },
  {
    "maxcpu": 4,
    "maxdisk": 103079215104,
    "maxmem": 12884901888,
    "name": "edge-cold-standby",
    "node": "pve02",
    "plugintype": null,
    "status": "stopped",
    "storage": null,
    "template": 0,
    "type": "qemu",
    "vmid": 9130
  },
  {
    "maxcpu": 4,
    "maxdisk": 105089261568,
    "maxmem": 16536748032,
    "name": null,
    "node": "pve02",
    "plugintype": null,
    "status": "online",
    "storage": null,
    "template": null,
    "type": "node",
    "vmid": null
  },
  {
    "maxcpu": 8,
    "maxdisk": 105089261568,
    "maxmem": 33553518592,
    "name": null,
    "node": "pve03",
    "plugintype": null,
    "status": "online",
    "storage": null,
    "template": null,
    "type": "node",
    "vmid": null
  },
  {
    "maxcpu": 24,
    "maxdisk": 105089261568,
    "maxmem": 33379028992,
    "name": null,
    "node": "pve01",
    "plugintype": null,
    "status": "online",
    "storage": null,
    "template": null,
    "type": "node",
    "vmid": null
  },
  {
    "maxcpu": null,
    "maxdisk": 105089261568,
    "maxmem": null,
    "name": null,
    "node": "pve02",
    "plugintype": "dir",
    "status": "available",
    "storage": "local",
    "template": null,
    "type": "storage",
    "vmid": null
  },
  {
    "maxcpu": null,
    "maxdisk": 105089261568,
    "maxmem": null,
    "name": null,
    "node": "pve03",
    "plugintype": "dir",
    "status": "available",
    "storage": "local",
    "template": null,
    "type": "storage",
    "vmid": null
  },
  {
    "maxcpu": null,
    "maxdisk": 105089261568,
    "maxmem": null,
    "name": null,
    "node": "pve01",
    "plugintype": "dir",
    "status": "available",
    "storage": "local",
    "template": null,
    "type": "storage",
    "vmid": null
  },
  {
    "maxcpu": null,
    "maxdisk": 846016282624,
    "maxmem": null,
    "name": null,
    "node": "pve02",
    "plugintype": "lvmthin",
    "status": "available",
    "storage": "local-lvm",
    "template": null,
    "type": "storage",
    "vmid": null
  },
  {
    "maxcpu": null,
    "maxdisk": 355660201984,
    "maxmem": null,
    "name": null,
    "node": "pve03",
    "plugintype": "lvmthin",
    "status": "available",
    "storage": "local-lvm",
    "template": null,
    "type": "storage",
    "vmid": null
  },
  {
    "maxcpu": null,
    "maxdisk": 869168840704,
    "maxmem": null,
    "name": null,
    "node": "pve01",
    "plugintype": "lvmthin",
    "status": "available",
    "storage": "local-lvm",
    "template": null,
    "type": "storage",
    "vmid": null
  },
  {
    "maxcpu": null,
    "maxdisk": null,
    "maxmem": null,
    "name": null,
    "node": "pve02",
    "plugintype": null,
    "status": "ok",
    "storage": null,
    "template": null,
    "type": "network",
    "vmid": null
  },
  {
    "maxcpu": null,
    "maxdisk": null,
    "maxmem": null,
    "name": null,
    "node": "pve03",
    "plugintype": null,
    "status": "ok",
    "storage": null,
    "template": null,
    "type": "network",
    "vmid": null
  },
  {
    "maxcpu": null,
    "maxdisk": null,
    "maxmem": null,
    "name": null,
    "node": "pve01",
    "plugintype": null,
    "status": "ok",
    "storage": null,
    "template": null,
    "type": "network",
    "vmid": null
  }
]

===== STORAGE STATUS =====
Name             Type     Status     Total (KiB)      Used (KiB) Available (KiB)        %
local             dir     active       102626232        43159780        54207188   42.06%
local-lvm     lvmthin     active       848797696       280103239       568694456   33.00%

===== PVE01 QEMU =====
      VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID       
       150 nextcloud            running    8192              64.00 10349     
       170 core-apps            running    10240            120.00 10425     
       171 monitoring           running    8192             120.00 10529     

===== PVE01 LXC =====
VMID       Status     Lock         Name                
110        running                 dns1                
112        running                 unbound1            

===== HOMELAB SYSTEMD UNITS =====
dpkg-db-backup.service                                   static          -
dpkg-db-backup.timer                                     enabled         enabled
filebrowser-backup.service                               static          -
filebrowser-backup.timer                                 enabled         enabled
homelab-alerting-health.service                          static          -
homelab-alerting-health.timer                            enabled         enabled
homelab-alertmanager-backup.service                      static          -
homelab-alertmanager-backup.timer                        enabled         enabled
homelab-appbackup-dockge-stacks.service                  static          -
homelab-appbackup-dockge-stacks.timer                    enabled         enabled
homelab-appbackup-mariadb-batch.service                  static          -
homelab-appbackup-mariadb-batch.timer                    enabled         enabled
homelab-appbackup-npmplus-edge.service                   static          -
homelab-appbackup-npmplus-edge.timer                     enabled         enabled
homelab-appbackup-postgres-batch.service                 static          -
homelab-appbackup-postgres-batch.timer                   enabled         enabled
homelab-appbackup-sqlite-batch.service                   static          -
homelab-appbackup-sqlite-batch.timer                     enabled         enabled
homelab-authentik-backup.service                         static          -
homelab-authentik-backup.timer                           disabled        enabled
homelab-auto-backup.service                              static          -
homelab-auto-backup.timer                                enabled         enabled
homelab-backup-coverage-matrix.service                   static          -
homelab-backup-coverage-matrix.timer                     enabled         enabled
homelab-backup-freshness.service                         static          -
homelab-backup-freshness.timer                           enabled         enabled
homelab-backup-healthcheck.service                       static          -
homelab-backup-healthcheck.timer                         enabled         enabled
homelab-backup-matrix-refresh.service                    static          -
homelab-backup-matrix-refresh.timer                      disabled        enabled
homelab-backup-sla-health.service                        static          -
homelab-backup-sla-health.timer                          enabled         enabled
homelab-beszel-backup.service                            static          -
homelab-beszel-backup.timer                              enabled         enabled
homelab-blackbox-exporter-backup.service                 static          -
homelab-blackbox-exporter-backup.timer                   enabled         enabled
homelab-capacity-risk.service                            static          -
homelab-capacity-risk.timer                              enabled         enabled
homelab-cluster-admin-incident-engine-backup.service     static          -
homelab-cluster-admin-incident-engine-backup.timer       enabled         enabled
homelab-cluster-admin-inventory-sync.service             static          -
homelab-cluster-admin-inventory-sync.timer               enabled         enabled
homelab-cluster-admin-pull.service                       static          -
homelab-cluster-admin-pull.timer                         disabled        enabled
homelab-cluster-admin-webpanel-sync.service              static          -
homelab-cluster-admin-webpanel-sync.timer                enabled         enabled
homelab-cluster-internal-ip.service                      enabled         enabled
homelab-cluster-passport.service                         static          -
homelab-cluster-passport.timer                           enabled         enabled
homelab-crypto-portfolio-backup.service                  static          -
homelab-crypto-portfolio-backup.timer                    enabled         enabled
homelab-crypto-portfolio-chart-health.service            static          -
homelab-crypto-portfolio-chart-health.timer              enabled         enabled
homelab-dependency-map-check.service                     static          -
homelab-dependency-map-check.timer                       enabled         enabled
homelab-desired-state-sync.service                       static          -
homelab-desired-state-sync.timer                         enabled         enabled
homelab-disk-space-health.service                        static          -
homelab-disk-space-health.timer                          enabled         enabled
homelab-docker-health.service                            static          -
homelab-docker-health.timer                              enabled         enabled
homelab-drift-check.service                              static          -
homelab-drift-check.timer                                enabled         enabled
homelab-duty-admin-report.service                        static          -
homelab-duty-admin-report.timer                          enabled         enabled
homelab-duty-admin-v2.service                            static          -
homelab-duty-admin-v2.timer                              enabled         enabled
homelab-edge-vm-offhost-freshness.service                static          -
homelab-edge-vm-offhost-freshness.timer                  enabled         enabled
homelab-edge-vm-vzdump-backup.service                    static          -
homelab-edge-vm-vzdump-backup.timer                      disabled        enabled
homelab-emergency-backup.service                         static          -
homelab-emergency-backup.timer                           enabled         enabled
homelab-evidence-catalog.service                         static          -
homelab-evidence-catalog.timer                           enabled         enabled
homelab-evidence-seal.service                            static          -
homelab-evidence-seal.timer                              disabled        enabled
homelab-extended-appbackup.service                       static          -
homelab-extended-appbackup.timer                         enabled         enabled
homelab-external-probe-vps-health.service                static          -
homelab-external-probe-vps-health.timer                  enabled         enabled
homelab-final-readiness-gate.service                     static          -
homelab-final-readiness-gate.timer                       enabled         enabled
homelab-forum-snuffleupagus-health.service               static          -
homelab-forum-snuffleupagus-health.timer                 enabled         enabled
homelab-gitea-backup.service                             static          -
homelab-gitea-backup.timer                               disabled        enabled
homelab-gitea-secondary-backup.service                   static          -
homelab-gitea-secondary-backup.timer                     enabled         enabled
homelab-golden-state-index.service                       static          -
homelab-golden-state-index.timer                         enabled         enabled
homelab-health-http-9101.service                         enabled         enabled
homelab-health-http.service                              enabled         enabled
homelab-health-metrics.service                           static          -
homelab-health-metrics.timer                             enabled         enabled
homelab-incident-journal.service                         static          -
homelab-incident-journal.timer                           enabled         enabled
homelab-ingress-standby-refresh.service                  static          -
homelab-ingress-standby-refresh.timer                    disabled        enabled
homelab-kuma-monitor-policy.service                      static          -
homelab-kuma-monitor-policy.timer                        enabled         enabled
homelab-legacy-monitoring-backups.service                static          -
homelab-legacy-monitoring-backups.timer                  enabled         enabled
homelab-mail-cloud-critical-upload.service               static          -
homelab-mail-cloud-critical-upload.timer                 enabled         enabled
homelab-mail-cloud-edge-vm-upload.service                static          -
homelab-mail-cloud-edge-vm-upload.timer                  enabled         enabled
homelab-mail-cloud-nextcloud-vm-upload.service           static          -
homelab-mail-cloud-nextcloud-vm-upload.timer             enabled         enabled
homelab-mail-cloud-restore-drill.service                 static          -
homelab-mail-cloud-restore-drill.timer                   enabled         enabled
homelab-mkdocs-auto-refresh.service                      static          -
homelab-mkdocs-auto-refresh.timer                        enabled         enabled
homelab-npmplus-kuma-config-backup.service               static          -
homelab-npmplus-kuma-config-backup.timer                 disabled        enabled
homelab-overall-health.service                           static          -
homelab-overall-health.timer                             enabled         enabled
homelab-private-vpn-hosts-health.service                 static          -
homelab-private-vpn-hosts-health.timer                   enabled         enabled
homelab-pve03-staging-capacity-health.service            static          -
homelab-pve03-staging-capacity-health.timer              enabled         enabled
homelab-quality-gate.service                             static          -
homelab-quality-gate.timer                               enabled         enabled
homelab-router-running-config-mail-cloud.service         static          -
homelab-router-running-config-mail-cloud.timer           enabled         enabled
homelab-router-watchdog.service                          static          -
homelab-router-watchdog.timer                            enabled         enabled
homelab-runbook-generate.service                         static          -
homelab-runbook-generate.timer                           enabled         enabled
homelab-safe-autoheal.service                            static          -
homelab-safe-autoheal.timer                              enabled         enabled
homelab-safe-exec-health.service                         static          -
homelab-safe-exec-health.timer                           disabled        enabled
homelab-scrutiny.service                                 enabled         enabled
homelab-secret-exposure-guard.service                    static          -
homelab-secret-exposure-guard.timer                      enabled         enabled
homelab-secret-sanity.service                            static          -
homelab-secret-sanity.timer                              enabled         enabled
homelab-service-registry-check.service                   static          -
homelab-service-registry-check.timer                     enabled         enabled
homelab-smartctl-textfile.service                        static          -
homelab-smartctl-textfile.timer                          enabled         enabled
homelab-sops-edge-secret-coverage.service                static          -
homelab-sops-edge-secret-coverage.timer                  enabled         enabled
homelab-storage-capacity.service                         static          -
homelab-storage-capacity.timer                           enabled         enabled
homelab-vaultwarden-backup.service                       static          -
homelab-vaultwarden-backup.timer                         disabled        enabled
homelab-verified-backup-health.service                   static          -
homelab-verified-backup-health.timer                     enabled         enabled
homelab-vm160-remote-backup.service                      static          -
homelab-vm160-remote-backup.timer                        disabled        enabled
homelab-vm-backup-health.service                         static          -
homelab-vm-backup-health.timer                           enabled         enabled
homelab-vm-backup-policy.service                         static          -
homelab-vm-backup-policy.timer                           enabled         enabled
homelab-vm-full-backup.service                           static          -
homelab-vm-full-backup.timer                             enabled         enabled
homelab-vm-full-restore-drill.service                    static          -
homelab-vm-full-restore-drill.timer                      enabled         enabled
homelab-vm-local-dumps-2cloud.service                    static          -
homelab-vm-local-dumps-2cloud.timer                      enabled         enabled
homelab-vm-local-dumps-retention.service                 static          -
homelab-vm-local-dumps-retention.timer                   enabled         enabled
homelab-vps-identity-audit.service                       static          -
homelab-vps-identity-audit.timer                         enabled         enabled
immich-media-backup.service                              static          -
immich-media-backup.timer                                disabled        enabled
memos-backup.service                                     static          -
memos-backup.timer                                       enabled         enabled
netbird-vps-backup.service                               static          -
netbird-vps-backup.timer                                 disabled        enabled
paperless-backup.service                                 static          -
paperless-backup.timer                                   disabled        enabled
skladchik-reports-monitor-backup-restore-check.service   static          -
skladchik-reports-monitor-backup-restore-check.timer     enabled         enabled
skladchik-reports-monitor-backup.service                 static          -
skladchik-reports-monitor-backup.timer                   enabled         enabled

===== HOMELAB TIMERS =====
Tue 2026-07-21 09:11:14 MSK               3s Tue 2026-07-21 09:10:14 MSK       56s ago homelab-cluster-admin-webpanel-sync.timer              homelab-cluster-admin-webpanel-sync.service
Tue 2026-07-21 09:11:14 MSK               3s Tue 2026-07-21 09:10:14 MSK       56s ago homelab-private-vpn-hosts-health.timer                 homelab-private-vpn-hosts-health.service
Tue 2026-07-21 09:11:49 MSK              37s Tue 2026-07-21 09:10:45 MSK       25s ago homelab-router-watchdog.timer                          homelab-router-watchdog.service
Tue 2026-07-21 09:12:16 MSK          1min 5s Tue 2026-07-21 09:10:14 MSK       56s ago homelab-cluster-admin-inventory-sync.timer             homelab-cluster-admin-inventory-sync.service
Tue 2026-07-21 09:14:26 MSK         3min 14s Tue 2026-07-21 09:09:25 MSK  1min 45s ago homelab-health-metrics.timer                           homelab-health-metrics.service
Tue 2026-07-21 09:15:00 MSK         3min 48s Tue 2026-07-21 09:00:14 MSK     10min ago homelab-incident-journal.timer                         homelab-incident-journal.service
Tue 2026-07-21 09:15:00 MSK         3min 48s Tue 2026-07-21 09:00:14 MSK     10min ago homelab-safe-autoheal.timer                            homelab-safe-autoheal.service
Tue 2026-07-21 09:15:26 MSK         4min 14s Tue 2026-07-21 09:01:14 MSK      9min ago homelab-duty-admin-v2.timer                            homelab-duty-admin-v2.service
Tue 2026-07-21 09:15:30 MSK         4min 18s Tue 2026-07-21 09:00:35 MSK     10min ago homelab-external-probe-vps-health.timer                homelab-external-probe-vps-health.service
Tue 2026-07-21 09:15:30 MSK         4min 19s Tue 2026-07-21 09:10:14 MSK       56s ago homelab-crypto-portfolio-chart-health.timer            homelab-crypto-portfolio-chart-health.service
Tue 2026-07-21 09:15:45 MSK         4min 34s Tue 2026-07-21 09:10:45 MSK       25s ago homelab-smartctl-textfile.timer                        homelab-smartctl-textfile.service
Tue 2026-07-21 09:16:28 MSK             5min Tue 2026-07-21 09:00:41 MSK     10min ago homelab-forum-snuffleupagus-health.timer               homelab-forum-snuffleupagus-health.service
Tue 2026-07-21 09:16:30 MSK             5min Tue 2026-07-21 09:01:30 MSK      9min ago homelab-disk-space-health.timer                        homelab-disk-space-health.service
Tue 2026-07-21 09:19:30 MSK             8min Tue 2026-07-21 09:04:30 MSK      6min ago homelab-pve03-staging-capacity-health.timer            homelab-pve03-staging-capacity-health.service
Tue 2026-07-21 09:30:55 MSK            19min Tue 2026-07-21 09:02:14 MSK      8min ago homelab-mkdocs-auto-refresh.timer                      homelab-mkdocs-auto-refresh.service
Tue 2026-07-21 11:42:25 MSK         2h 31min Tue 2026-07-21 05:42:25 MSK  3h 28min ago homelab-vps-identity-audit.timer                       homelab-vps-identity-audit.service
Wed 2026-07-22 00:00:00 MSK              14h Tue 2026-07-21 00:00:14 MSK        9h ago dpkg-db-backup.timer                                   dpkg-db-backup.service
Wed 2026-07-22 00:01:28 MSK              14h Tue 2026-07-21 00:06:03 MSK        9h ago homelab-storage-capacity.timer                         homelab-storage-capacity.service
Wed 2026-07-22 00:06:04 MSK              14h Tue 2026-07-21 00:03:14 MSK        9h ago homelab-quality-gate.timer                             homelab-quality-gate.service
Wed 2026-07-22 00:08:06 MSK              14h Tue 2026-07-21 00:09:32 MSK        9h ago homelab-evidence-catalog.timer                         homelab-evidence-catalog.service
Wed 2026-07-22 00:09:43 MSK              14h Tue 2026-07-21 00:09:32 MSK        9h ago homelab-backup-freshness.timer                         homelab-backup-freshness.service
Wed 2026-07-22 00:10:12 MSK              14h Tue 2026-07-21 00:16:14 MSK        8h ago homelab-docker-health.timer                            homelab-docker-health.service
Wed 2026-07-22 02:15:00 MSK              17h Tue 2026-07-21 02:15:05 MSK        6h ago homelab-vm-full-backup.timer                           homelab-vm-full-backup.service
Wed 2026-07-22 03:10:02 MSK              17h Tue 2026-07-21 03:13:14 MSK  5h 57min ago homelab-emergency-backup.timer                         homelab-emergency-backup.service
Wed 2026-07-22 03:35:00 MSK              18h Tue 2026-07-21 03:35:14 MSK  5h 35min ago filebrowser-backup.timer                               filebrowser-backup.service
Wed 2026-07-22 03:55:02 MSK              18h Tue 2026-07-21 03:51:25 MSK  5h 19min ago homelab-appbackup-dockge-stacks.timer                  homelab-appbackup-dockge-stacks.service
Wed 2026-07-22 04:17:29 MSK              19h Tue 2026-07-21 04:19:17 MSK  4h 51min ago homelab-appbackup-npmplus-edge.timer                   homelab-appbackup-npmplus-edge.service
Wed 2026-07-22 04:26:54 MSK              19h Tue 2026-07-21 04:28:14 MSK  4h 42min ago homelab-gitea-secondary-backup.timer                   homelab-gitea-secondary-backup.service
Wed 2026-07-22 04:38:04 MSK              19h Tue 2026-07-21 04:34:48 MSK  4h 36min ago skladchik-reports-monitor-backup.timer                 skladchik-reports-monitor-backup.service
Wed 2026-07-22 04:39:28 MSK              19h Tue 2026-07-21 04:48:40 MSK  4h 22min ago homelab-edge-vm-offhost-freshness.timer                homelab-edge-vm-offhost-freshness.service
Wed 2026-07-22 05:01:44 MSK              19h Tue 2026-07-21 04:57:14 MSK  4h 13min ago homelab-appbackup-sqlite-batch.timer                   homelab-appbackup-sqlite-batch.service
Wed 2026-07-22 05:13:38 MSK              20h Tue 2026-07-21 04:55:35 MSK  4h 15min ago homelab-sops-edge-secret-coverage.timer                homelab-sops-edge-secret-coverage.service
Wed 2026-07-22 05:23:56 MSK              20h Tue 2026-07-21 05:17:57 MSK  3h 53min ago homelab-router-running-config-mail-cloud.timer         homelab-router-running-config-mail-cloud.service
Wed 2026-07-22 05:27:13 MSK              20h Tue 2026-07-21 05:29:05 MSK  3h 42min ago homelab-appbackup-postgres-batch.timer                 homelab-appbackup-postgres-batch.service
Wed 2026-07-22 05:47:19 MSK              20h Tue 2026-07-21 05:46:14 MSK  3h 24min ago homelab-appbackup-mariadb-batch.timer                  homelab-appbackup-mariadb-batch.service
Wed 2026-07-22 06:01:19 MSK              20h Tue 2026-07-21 05:57:30 MSK  3h 13min ago homelab-extended-appbackup.timer                       homelab-extended-appbackup.service
Wed 2026-07-22 06:27:43 MSK              21h Tue 2026-07-21 06:32:25 MSK  2h 38min ago homelab-crypto-portfolio-backup.timer                  homelab-crypto-portfolio-backup.service
Wed 2026-07-22 06:38:51 MSK              21h Tue 2026-07-21 06:43:54 MSK  2h 27min ago homelab-verified-backup-health.timer                   homelab-verified-backup-health.service
Wed 2026-07-22 06:39:22 MSK              21h Tue 2026-07-21 06:40:25 MSK  2h 30min ago homelab-alertmanager-backup.timer                      homelab-alertmanager-backup.service
Wed 2026-07-22 06:44:16 MSK              21h Tue 2026-07-21 06:40:25 MSK  2h 30min ago homelab-mail-cloud-critical-upload.timer               homelab-mail-cloud-critical-upload.service
Wed 2026-07-22 06:52:06 MSK              21h Tue 2026-07-21 06:47:47 MSK  2h 23min ago homelab-beszel-backup.timer                            homelab-beszel-backup.service
Wed 2026-07-22 06:58:58 MSK              21h Tue 2026-07-21 06:57:30 MSK  2h 13min ago homelab-blackbox-exporter-backup.timer                 homelab-blackbox-exporter-backup.service
Wed 2026-07-22 07:02:27 MSK              21h Tue 2026-07-21 07:05:30 MSK   2h 5min ago homelab-backup-healthcheck.timer                       homelab-backup-healthcheck.service
Wed 2026-07-22 07:07:14 MSK              21h Tue 2026-07-21 07:18:55 MSK  1h 52min ago homelab-legacy-monitoring-backups.timer                homelab-legacy-monitoring-backups.service
Wed 2026-07-22 07:16:17 MSK              22h Tue 2026-07-21 07:15:45 MSK  1h 55min ago homelab-drift-check.timer                              homelab-drift-check.service
Wed 2026-07-22 07:35:06 MSK              22h Tue 2026-07-21 07:36:59 MSK  1h 34min ago homelab-vm-backup-health.timer                         homelab-vm-backup-health.service
Wed 2026-07-22 07:36:00 MSK              22h Tue 2026-07-21 07:21:14 MSK  1h 49min ago homelab-service-registry-check.timer                   homelab-service-registry-check.service
Wed 2026-07-22 07:38:21 MSK              22h Tue 2026-07-21 07:36:14 MSK  1h 34min ago homelab-cluster-admin-incident-engine-backup.timer     homelab-cluster-admin-incident-engine-backup.service
Wed 2026-07-22 07:41:49 MSK              22h Tue 2026-07-21 07:53:41 MSK  1h 17min ago homelab-mail-cloud-nextcloud-vm-upload.timer           homelab-mail-cloud-nextcloud-vm-upload.service
Wed 2026-07-22 07:47:14 MSK              22h Tue 2026-07-21 07:51:41 MSK  1h 19min ago homelab-dependency-map-check.timer                     homelab-dependency-map-check.service
Wed 2026-07-22 07:49:26 MSK              22h Tue 2026-07-21 07:56:14 MSK  1h 14min ago homelab-runbook-generate.timer                         homelab-runbook-generate.service
Wed 2026-07-22 07:53:13 MSK              22h Tue 2026-07-21 07:51:14 MSK  1h 19min ago homelab-alerting-health.timer                          homelab-alerting-health.service
Wed 2026-07-22 07:58:00 MSK              22h Tue 2026-07-21 08:09:34 MSK   1h 1min ago homelab-duty-admin-report.timer                        homelab-duty-admin-report.service
Wed 2026-07-22 08:03:06 MSK              22h Tue 2026-07-21 07:49:14 MSK  1h 21min ago homelab-desired-state-sync.timer                       homelab-desired-state-sync.service
Wed 2026-07-22 08:10:37 MSK              22h Tue 2026-07-21 08:07:25 MSK   1h 3min ago homelab-backup-sla-health.timer                        homelab-backup-sla-health.service
Wed 2026-07-22 08:18:43 MSK              23h Tue 2026-07-21 08:01:14 MSK   1h 9min ago homelab-overall-health.timer                           homelab-overall-health.service
Wed 2026-07-22 08:24:56 MSK              23h Tue 2026-07-21 08:29:14 MSK     41min ago homelab-capacity-risk.timer                            homelab-capacity-risk.service
Wed 2026-07-22 08:24:58 MSK              23h Tue 2026-07-21 08:17:02 MSK     54min ago homelab-kuma-monitor-policy.timer                      homelab-kuma-monitor-policy.service
Wed 2026-07-22 08:28:27 MSK              23h Tue 2026-07-21 08:31:30 MSK     39min ago homelab-final-readiness-gate.timer                     homelab-final-readiness-gate.service
Wed 2026-07-22 08:37:02 MSK              23h Tue 2026-07-21 08:30:07 MSK     41min ago homelab-vm-backup-policy.timer                         homelab-vm-backup-policy.service
Wed 2026-07-22 08:40:13 MSK              23h Tue 2026-07-21 08:50:14 MSK     20min ago homelab-backup-coverage-matrix.timer                   homelab-backup-coverage-matrix.service
Wed 2026-07-22 08:43:29 MSK              23h Tue 2026-07-21 08:54:07 MSK     17min ago homelab-golden-state-index.timer                       homelab-golden-state-index.service
Wed 2026-07-22 08:44:57 MSK              23h Tue 2026-07-21 08:35:41 MSK     35min ago homelab-secret-exposure-guard.timer                    homelab-secret-exposure-guard.service
Wed 2026-07-22 08:53:01 MSK              23h Tue 2026-07-21 08:51:14 MSK     19min ago homelab-cluster-passport.timer                         homelab-cluster-passport.service
Wed 2026-07-22 08:53:36 MSK              23h Tue 2026-07-21 08:41:14 MSK     29min ago memos-backup.timer                                     memos-backup.service
Sun 2026-07-26 03:34:47 MSK           4 days Sun 2026-07-19 03:45:37 MSK    2 days ago homelab-auto-backup.timer                              homelab-auto-backup.service
Sun 2026-07-26 05:17:43 MSK           4 days Sun 2026-07-19 05:11:25 MSK    2 days ago skladchik-reports-monitor-backup-restore-check.timer   skladchik-reports-monitor-backup-restore-check.service
Sun 2026-07-26 09:32:36 MSK           5 days Sun 2026-07-19 09:32:29 MSK 1 day 23h ago homelab-mail-cloud-edge-vm-upload.timer                homelab-mail-cloud-edge-vm-upload.service
Sun 2026-07-26 10:29:15 MSK           5 days Sun 2026-07-19 10:06:07 MSK 1 day 23h ago homelab-vm-local-dumps-2cloud.timer                    homelab-vm-local-dumps-2cloud.service
Sun 2026-07-26 11:13:32 MSK           5 days Sun 2026-07-19 11:08:41 MSK 1 day 22h ago homelab-vm-local-dumps-retention.timer                 homelab-vm-local-dumps-retention.service
Mon 2026-07-27 00:09:19 MSK           5 days Mon 2026-07-20 00:00:29 MSK  1 day 9h ago homelab-secret-sanity.timer                            homelab-secret-sanity.service
Mon 2026-07-27 07:34:51 MSK           5 days Mon 2026-07-20 07:25:14 MSK  1 day 1h ago homelab-mail-cloud-restore-drill.timer                 homelab-mail-cloud-restore-drill.service
Thu 2026-10-15 06:00:00 MSK 2 months 24 days Wed 2026-07-15 06:00:00 MSK             - homelab-vm-full-restore-drill.timer                    homelab-vm-full-restore-drill.service

===== FAILED UNITS =====
  UNIT                                  LOAD   ACTIVE SUB    DESCRIPTION
● homelab-backup-matrix-refresh.service loaded failed failed Generate unified homelab backup matrix state
● homelab-cluster-admin-pull.service    loaded failed failed Pull cluster-admin observer health into pve01
● homelab-vm160-remote-backup.service   loaded failed failed VM160 full-image backup streamed from pve02 to pve03

Legend: LOAD   → Reflects whether the unit definition was properly loaded.
        ACTIVE → The high-level unit activation state, i.e. generalization of SUB.
        SUB    → The low-level unit activation state, values depend on unit type.

3 loaded units listed.

===== HOMELAB UNIT FILE PATHS =====
2026-06-15T13:00:42.7590703540 644 root:root 259 /etc/systemd/system/homelab-auto-backup.service
2026-06-15T13:00:58.1712446040 644 root:root 205 /etc/systemd/system/homelab-auto-backup.timer
2026-06-15T13:04:11.9638937260 644 root:root 124 /etc/systemd/system/homelab-backup-healthcheck.service
2026-06-15T13:04:11.9649165750 644 root:root 204 /etc/systemd/system/homelab-backup-healthcheck.timer
2026-06-16T19:56:57.2280572190 644 root:root 133 /etc/systemd/system/homelab-disk-space-health.service
2026-06-16T19:56:57.2283922310 644 root:root 188 /etc/systemd/system/homelab-disk-space-health.timer
2026-06-18T06:26:08.5462695380 644 root:root 446 /etc/systemd/system/homelab-private-vpn-hosts-health.service
2026-06-18T06:26:08.5473243400 644 root:root 232 /etc/systemd/system/homelab-private-vpn-hosts-health.timer
2026-06-18T07:10:14.4453177530 644 root:root 204 /etc/systemd/system/homelab-npmplus-kuma-config-backup.service
2026-06-18T07:10:14.4468289380 644 root:root 230 /etc/systemd/system/homelab-npmplus-kuma-config-backup.timer
2026-06-18T16:26:18.3740018850 644 root:root 181 /etc/systemd/system/homelab-authentik-backup.service
2026-06-18T16:26:18.3753734820 644 root:root 208 /etc/systemd/system/homelab-authentik-backup.timer
2026-06-18T17:09:28.1614174140 644 root:root 173 /etc/systemd/system/homelab-gitea-backup.service
2026-06-18T17:09:28.1628104380 644 root:root 200 /etc/systemd/system/homelab-gitea-backup.timer
2026-06-18T17:35:10.8723268760 644 root:root 185 /etc/systemd/system/homelab-vaultwarden-backup.service
2026-06-18T17:35:10.8735951370 644 root:root 212 /etc/systemd/system/homelab-vaultwarden-backup.timer
2026-06-18T18:56:51.5685208630 644 root:root 200 /etc/systemd/system/homelab-edge-vm-vzdump-backup.service
2026-06-18T18:56:51.5698180780 644 root:root 225 /etc/systemd/system/homelab-edge-vm-vzdump-backup.timer
2026-06-18T22:13:10.8353373180 644 root:root 271 /etc/systemd/system/homelab-mail-cloud-critical-upload.service
2026-06-18T22:13:10.8363221220 644 root:root 231 /etc/systemd/system/homelab-mail-cloud-critical-upload.timer
2026-06-18T22:23:32.0559055120 644 root:root 268 /etc/systemd/system/homelab-mail-cloud-edge-vm-upload.service
2026-06-18T22:23:32.0560225000 644 root:root 233 /etc/systemd/system/homelab-mail-cloud-edge-vm-upload.timer
2026-06-18T22:41:10.0626047720 644 root:root 252 /etc/systemd/system/homelab-mail-cloud-restore-drill.service
2026-06-18T22:41:10.0636047870 644 root:root 217 /etc/systemd/system/homelab-mail-cloud-restore-drill.timer
2026-06-20T23:10:27.1119168660 644 root:root 134 /etc/systemd/system/homelab-smartctl-textfile.service
2026-06-20T23:10:27.1128425330 644 root:root 208 /etc/systemd/system/homelab-smartctl-textfile.timer
2026-06-21T00:30:59.6377784930 644 root:root 619 /etc/systemd/system/homelab-scrutiny.service
2026-06-21T10:39:21.2127986960 644 root:root 205 /etc/systemd/system/homelab-sops-edge-secret-coverage.service
2026-06-21T10:39:21.2139615830 644 root:root 230 /etc/systemd/system/homelab-sops-edge-secret-coverage.timer
2026-06-21T11:06:12.5516186940 644 root:root 191 /etc/systemd/system/homelab-mkdocs-auto-refresh.timer
2026-06-21T15:54:26.1428955170 644 root:root 295 /etc/systemd/system/homelab-mkdocs-auto-refresh.service
2026-06-22T12:34:40.6028014260 644 root:root 416 /etc/systemd/system/homelab-health-metrics.service
2026-06-22T12:34:40.6037061340 644 root:root 219 /etc/systemd/system/homelab-health-metrics.timer
2026-06-24T16:00:07.1378187060 644 root:root 127 /etc/systemd/system/homelab-vps-identity-audit.service
2026-06-24T16:00:07.1388187210 644 root:root 147 /etc/systemd/system/homelab-vps-identity-audit.timer
2026-06-27T04:30:28.4973721150 644 root:root 169 /etc/systemd/system/homelab-secret-sanity.timer
2026-06-27T04:30:28.4973721150 644 root:root 249 /etc/systemd/system/homelab-secret-sanity.service
2026-06-27T05:04:37.4313772970 644 root:root 201 /etc/systemd/system/homelab-backup-freshness.timer
2026-06-27T05:06:45.3586435000 644 root:root 285 /etc/systemd/system/homelab-docker-health.service
2026-06-27T05:06:45.3594437270 644 root:root 204 /etc/systemd/system/homelab-docker-health.timer
2026-06-27T05:08:30.6395106090 644 root:root 259 /etc/systemd/system/homelab-storage-capacity.service
2026-06-27T05:08:30.6403077050 644 root:root 211 /etc/systemd/system/homelab-storage-capacity.timer
2026-06-27T05:09:24.1641095310 644 root:root 259 /etc/systemd/system/homelab-evidence-catalog.service
2026-06-27T05:09:24.1648443770 644 root:root 201 /etc/systemd/system/homelab-evidence-catalog.timer
2026-06-27T05:16:32.9994853080 644 root:root 189 /etc/systemd/system/homelab-quality-gate.timer
2026-06-27T05:16:32.9994853080 644 root:root 213 /etc/systemd/system/homelab-quality-gate.service
2026-06-29T21:00:02.0934499640 644 root:root 314 /etc/systemd/system/homelab-cluster-internal-ip.service
2026-06-30T13:42:51.4303494920 644 root:root 151 /etc/systemd/system/homelab-pve03-staging-capacity-health.service
2026-06-30T13:42:51.4310788310 644 root:root 180 /etc/systemd/system/homelab-pve03-staging-capacity-health.timer
2026-06-30T14:51:25.1233593980 644 root:root 271 /etc/systemd/system/homelab-mail-cloud-nextcloud-vm-upload.service
2026-06-30T14:51:25.1244045430 644 root:root 176 /etc/systemd/system/homelab-mail-cloud-nextcloud-vm-upload.timer
2026-06-30T16:30:01.3301453960 644 root:root 214 /etc/systemd/system/homelab-router-running-config-mail-cloud.service
2026-06-30T16:30:01.3311218770 644 root:root 185 /etc/systemd/system/homelab-router-running-config-mail-cloud.timer
2026-07-03T11:51:30.8268733400 644 root:root 171 /etc/systemd/system/homelab-emergency-backup.timer
2026-07-03T11:51:30.8268733400 644 root:root 197 /etc/systemd/system/homelab-emergency-backup.service
2026-07-03T12:04:53.4624196890 644 root:root 182 /etc/systemd/system/homelab-appbackup-dockge-stacks.timer
2026-07-03T12:04:53.4624196890 644 root:root 207 /etc/systemd/system/homelab-appbackup-dockge-stacks.service
2026-07-03T12:12:11.4472306810 644 root:root 246 /etc/systemd/system/homelab-backup-freshness.service
2026-07-03T12:29:06.4444996780 644 root:root 181 /etc/systemd/system/homelab-appbackup-npmplus-edge.timer
2026-07-03T12:29:06.4444996780 644 root:root 212 /etc/systemd/system/homelab-appbackup-npmplus-edge.service
2026-07-03T12:37:48.5320271520 644 root:root 140 /etc/systemd/system/homelab-verified-backup-health.service
2026-07-03T12:37:48.5320271520 644 root:root 173 /etc/systemd/system/homelab-verified-backup-health.timer
2026-07-03T12:44:42.9312711690 644 root:root 109 /etc/systemd/system/homelab-drift-check.service
2026-07-03T12:44:42.9312711690 644 root:root 153 /etc/systemd/system/homelab-drift-check.timer
2026-07-03T13:16:53.8209829850 644 root:root 171 /etc/systemd/system/homelab-appbackup-sqlite-batch.timer
2026-07-03T13:16:53.8209829850 644 root:root 417 /etc/systemd/system/homelab-appbackup-sqlite-batch.service
2026-07-03T13:20:57.4660979330 644 root:root 173 /etc/systemd/system/homelab-appbackup-postgres-batch.timer
2026-07-03T13:25:04.7183999550 644 root:root 418 /etc/systemd/system/homelab-appbackup-postgres-batch.service
2026-07-03T13:33:49.3146898050 644 root:root 172 /etc/systemd/system/homelab-appbackup-mariadb-batch.timer
2026-07-03T13:33:49.3146898050 644 root:root 200 /etc/systemd/system/homelab-appbackup-mariadb-batch.service
2026-07-03T13:41:24.9497954780 644 root:root 138 /etc/systemd/system/homelab-service-registry-check.service
2026-07-03T13:41:24.9497954780 644 root:root 171 /etc/systemd/system/homelab-service-registry-check.timer
2026-07-03T13:42:08.5796973730 644 root:root 127 /etc/systemd/system/homelab-dependency-map-check.service
2026-07-03T13:42:08.5796973730 644 root:root 162 /etc/systemd/system/homelab-dependency-map-check.timer
2026-07-03T13:43:01.6263487930 644 root:root 115 /etc/systemd/system/homelab-overall-health.service
2026-07-03T13:43:01.6263487930 644 root:root 156 /etc/systemd/system/homelab-overall-health.timer
2026-07-03T13:44:21.6709762300 644 root:root 268 /etc/systemd/system/homelab-health-http.service
2026-07-03T13:46:25.4886845470 644 root:root 119 /etc/systemd/system/homelab-vm-backup-health.service
2026-07-03T13:46:25.4886845470 644 root:root 158 /etc/systemd/system/homelab-vm-backup-health.timer
2026-07-03T13:51:34.0440941410 644 root:root 273 /etc/systemd/system/homelab-health-http-9101.service
2026-07-03T14:07:11.3293592880 644 root:root 184 /etc/systemd/system/homelab-alerting-health.service
2026-07-03T14:07:11.3303593020 644 root:root 163 /etc/systemd/system/homelab-alerting-health.timer
2026-07-03T14:19:38.2269689900 644 root:root 168 /etc/systemd/system/homelab-runbook-generate.timer
2026-07-03T14:19:38.2269689900 644 root:root 174 /etc/systemd/system/homelab-runbook-generate.service
2026-07-03T14:30:48.2259704210 644 root:root 159 /etc/systemd/system/homelab-desired-state-sync.service
2026-07-03T14:30:48.2259704210 644 root:root 168 /etc/systemd/system/homelab-desired-state-sync.timer
2026-07-03T14:35:20.4059353590 644 root:root 159 /etc/systemd/system/homelab-duty-admin-report.service
2026-07-03T14:35:20.4059353590 644 root:root 169 /etc/systemd/system/homelab-duty-admin-report.timer
2026-07-06T16:38:41.0738565590 644 root:root 127 /etc/systemd/system/homelab-incident-journal.timer
2026-07-06T16:38:41.0738565590 644 root:root 147 /etc/systemd/system/homelab-incident-journal.service
2026-07-06T16:46:56.0853501450 644 root:root 159 /etc/systemd/system/homelab-backup-sla-health.service
2026-07-06T16:46:56.0853501450 644 root:root 169 /etc/systemd/system/homelab-backup-sla-health.timer
2026-07-06T17:07:42.1531722210 644 root:root 160 /etc/systemd/system/homelab-kuma-monitor-policy.service
2026-07-06T17:07:42.1531722210 644 root:root 168 /etc/systemd/system/homelab-kuma-monitor-policy.timer
2026-07-06T17:17:27.4890025440 644 root:root 134 /etc/systemd/system/homelab-safe-autoheal.timer
2026-07-06T17:17:27.4890025440 644 root:root 151 /etc/systemd/system/homelab-safe-autoheal.service
2026-07-06T17:28:46.8602514040 644 root:root 155 /etc/systemd/system/homelab-final-readiness-gate.service
2026-07-06T17:28:46.8602514040 644 root:root 162 /etc/systemd/system/homelab-final-readiness-gate.timer
2026-07-06T17:51:59.0742410960 644 root:root 170 /etc/systemd/system/homelab-vm-backup-policy.service
2026-07-06T17:51:59.0742410960 644 root:root 181 /etc/systemd/system/homelab-vm-backup-policy.timer
2026-07-06T18:27:29.1535313910 644 root:root 181 /etc/systemd/system/homelab-vm-local-dumps-2cloud.service
2026-07-06T18:27:29.1535313910 644 root:root 191 /etc/systemd/system/homelab-vm-local-dumps-2cloud.timer
2026-07-06T18:56:19.5296144310 644 root:root 176 /etc/systemd/system/homelab-vm-local-dumps-retention.timer
2026-07-06T18:56:19.5296144310 644 root:root 177 /etc/systemd/system/homelab-vm-local-dumps-retention.service
2026-07-06T19:08:11.3245947220 644 root:root 147 /etc/systemd/system/homelab-capacity-risk.service
2026-07-06T19:08:11.3245947220 644 root:root 161 /etc/systemd/system/homelab-capacity-risk.timer
2026-07-06T19:27:15.6118940830 644 root:root 157 /etc/systemd/system/homelab-secret-exposure-guard.service
2026-07-06T19:27:15.6118940830 644 root:root 163 /etc/systemd/system/homelab-secret-exposure-guard.timer
2026-07-06T19:44:22.9146001910 644 root:root 147 /etc/systemd/system/homelab-cluster-passport.service
2026-07-06T19:44:22.9146001910 644 root:root 158 /etc/systemd/system/homelab-cluster-passport.timer
2026-07-06T20:01:29.9500539270 644 root:root 151 /etc/systemd/system/homelab-golden-state-index.service
2026-07-06T20:01:29.9500539270 644 root:root 160 /etc/systemd/system/homelab-golden-state-index.timer
2026-07-07T00:22:27.7023075630 644 root:root 131 /etc/systemd/system/homelab-backup-coverage-matrix.service
2026-07-07T00:22:27.7031435610 644 root:root 164 /etc/systemd/system/homelab-backup-coverage-matrix.timer
2026-07-07T04:22:28.9708924180 644 root:root 165 /etc/systemd/system/homelab-extended-appbackup.service
2026-07-07T04:22:28.9708924180 644 root:root 166 /etc/systemd/system/homelab-extended-appbackup.timer
2026-07-07T06:37:16.6351692020 644 root:root 160 /etc/systemd/system/homelab-forum-snuffleupagus-health.timer
2026-07-07T06:37:16.6351692020 644 root:root 167 /etc/systemd/system/homelab-forum-snuffleupagus-health.service
2026-07-07T07:28:55.1971196770 644 root:root 168 /etc/systemd/system/homelab-external-probe-vps-health.timer
2026-07-07T07:28:55.1971196770 644 root:root 174 /etc/systemd/system/homelab-external-probe-vps-health.service
2026-07-07T08:01:44.7118958700 644 root:root 150 /etc/systemd/system/homelab-duty-admin-v2.service
2026-07-07T08:01:44.7118958700 644 root:root 156 /etc/systemd/system/homelab-duty-admin-v2.timer
2026-07-08T15:08:13.7211859980 644 root:root 235 /etc/systemd/system/homelab-edge-vm-offhost-freshness.service
2026-07-08T15:08:13.7216805560 644 root:root 252 /etc/systemd/system/homelab-edge-vm-offhost-freshness.timer
2026-07-09T02:10:26.8244661930 644 root:root 142 /etc/systemd/system/homelab-cluster-admin-pull.service
2026-07-09T02:10:26.8256177450 644 root:root 145 /etc/systemd/system/homelab-cluster-admin-pull.timer
2026-07-09T08:11:07.2434837370 644 root:root 162 /etc/systemd/system/homelab-cluster-admin-webpanel-sync.service
2026-07-09T08:11:07.2446282660 644 root:root 169 /etc/systemd/system/homelab-cluster-admin-webpanel-sync.timer
2026-07-10T02:50:51.8861195400 644 root:root 361 /etc/systemd/system/homelab-vm-full-backup.service
2026-07-10T02:50:51.8871195540 644 root:root 201 /etc/systemd/system/homelab-vm-full-backup.timer
2026-07-10T02:51:05.4785581640 644 root:root 357 /etc/systemd/system/homelab-vm-full-restore-drill.service
2026-07-10T02:51:05.4796181450 644 root:root 234 /etc/systemd/system/homelab-vm-full-restore-drill.timer
2026-07-10T11:28:07.6943454530 644 root:root 187 /etc/systemd/system/homelab-backup-matrix-refresh.service
2026-07-10T11:28:07.6943454530 644 root:root 217 /etc/systemd/system/homelab-backup-matrix-refresh.timer
2026-07-10T20:33:12.8858520910 644 root:root 162 /etc/systemd/system/homelab-crypto-portfolio-backup.timer
2026-07-10T20:33:12.8858520910 644 root:root 164 /etc/systemd/system/homelab-crypto-portfolio-backup.service
2026-07-10T21:46:05.4474358550 644 root:root 204 /etc/systemd/system/homelab-alertmanager-backup.timer
2026-07-10T21:46:05.4474358550 644 root:root 209 /etc/systemd/system/homelab-alertmanager-backup.service
2026-07-10T22:04:53.2805465050 644 root:root 197 /etc/systemd/system/homelab-beszel-backup.service
2026-07-10T22:04:53.2805465050 644 root:root 202 /etc/systemd/system/homelab-beszel-backup.timer
2026-07-10T22:19:57.8193011610 644 root:root 219 /etc/systemd/system/homelab-blackbox-exporter-backup.service
2026-07-10T22:19:57.8193011610 644 root:root 224 /etc/systemd/system/homelab-blackbox-exporter-backup.timer
2026-07-10T22:44:47.4338992100 644 root:root 227 /etc/systemd/system/homelab-legacy-monitoring-backups.service
2026-07-10T22:44:47.4338992100 644 root:root 234 /etc/systemd/system/homelab-legacy-monitoring-backups.timer
2026-07-11T00:36:01.9494545100 644 root:root 235 /etc/systemd/system/homelab-cluster-admin-inventory-sync.service
2026-07-11T00:36:01.9494545100 644 root:root 235 /etc/systemd/system/homelab-cluster-admin-inventory-sync.timer
2026-07-11T01:27:09.7763208170 644 root:root 231 /etc/systemd/system/homelab-cluster-admin-incident-engine-backup.service
2026-07-11T01:27:09.7763208170 644 root:root 238 /etc/systemd/system/homelab-cluster-admin-incident-engine-backup.timer
2026-07-11T22:01:25.6167081820 644 root:root 199 /etc/systemd/system/homelab-router-watchdog.service
2026-07-11T22:01:25.6179399040 644 root:root 236 /etc/systemd/system/homelab-router-watchdog.timer
2026-07-12T21:35:41.6488572630 644 root:root 197 /etc/systemd/system/homelab-crypto-portfolio-chart-health.service
2026-07-12T21:35:41.6488811480 644 root:root 230 /etc/systemd/system/homelab-crypto-portfolio-chart-health.timer
2026-07-18T01:38:18.1727154050 644 root:root 245 /etc/systemd/system/homelab-vm160-remote-backup.service
2026-07-18T01:38:18.1738421500 644 root:root 189 /etc/systemd/system/homelab-vm160-remote-backup.timer
2026-07-20T22:15:38.8869753850 644 root:root 316 /etc/systemd/system/homelab-gitea-secondary-backup.service
2026-07-20T22:15:38.8883231940 644 root:root 208 /etc/systemd/system/homelab-gitea-secondary-backup.timer
2026-07-20T23:59:13.9750909030 644 root:root 361 /etc/systemd/system/homelab-ingress-standby-refresh.service
2026-07-20T23:59:13.9756878160 644 root:root 227 /etc/systemd/system/homelab-ingress-standby-refresh.timer
2026-07-21T00:09:32.1964449440 644 root:root 510 /etc/systemd/system/homelab-safe-exec-health.service
2026-07-21T00:09:32.1984090270 644 root:root 236 /etc/systemd/system/homelab-safe-exec-health.timer
2026-07-21T00:33:12.4631182460 644 root:root 527 /etc/systemd/system/homelab-evidence-seal.service
2026-07-21T00:33:12.4638041710 644 root:root 218 /etc/systemd/system/homelab-evidence-seal.timer

===== PVE01 DOCKER =====
DOCKER_INSTALLED=NO

===== EXISTING HOMELAB STATE DIRECTORIES =====

----- DIR=/var/lib/homelab-health -----
2026-07-09T15:29:41.9851382980 236 /var/lib/homelab-health/proxmox-pve01-offhost.txt
2026-07-09T15:29:41.9851382980 236 /var/lib/homelab-health/pve01-offhost.txt
2026-07-09T15:29:41.9851382980 276 /var/lib/homelab-health/proxmox-pve01-backup.txt
2026-07-09T15:29:41.9851382980 276 /var/lib/homelab-health/pve01-backup.txt
2026-07-09T15:33:52.8128905890 397 /var/lib/homelab-health/service-readiness-proxmox-pve01.txt
2026-07-09T15:42:17.7925620460 203 /var/lib/homelab-health/filebrowser-offhost.txt
2026-07-09T15:47:18.8311350910 276 /var/lib/homelab-health/service-readiness-filebrowser.txt
2026-07-09T15:53:45.2480051850 207 /var/lib/homelab-health/actual-budget-offhost.txt
2026-07-09T15:53:45.2480051850 207 /var/lib/homelab-health/actual-offhost.txt
2026-07-09T15:53:45.2480051850 213 /var/lib/homelab-health/actual-budget-restore.txt
2026-07-09T15:53:45.2480051850 213 /var/lib/homelab-health/actual-restore.txt
2026-07-09T15:53:45.2480051850 269 /var/lib/homelab-health/actual-backup.txt
2026-07-09T15:53:45.2480051850 269 /var/lib/homelab-health/actual-budget-backup.txt
2026-07-09T16:04:50.0111067090 281 /var/lib/homelab-health/service-readiness-actual-budget.txt
2026-07-09T19:01:20.7901933280 195 /var/lib/homelab-health/homebox-offhost.txt
2026-07-09T19:01:20.7901933280 201 /var/lib/homelab-health/homebox-restore.txt
2026-07-09T19:01:20.7901933280 251 /var/lib/homelab-health/homebox-backup.txt
2026-07-09T19:06:43.1941052730 268 /var/lib/homelab-health/service-readiness-homebox.txt
2026-07-09T19:17:35.0170448070 193 /var/lib/homelab-health/mealie-offhost.txt
2026-07-09T19:17:35.0170448070 199 /var/lib/homelab-health/mealie-restore.txt
2026-07-09T19:17:35.0170448070 250 /var/lib/homelab-health/mealie-backup.txt
2026-07-09T19:23:39.3486047240 283 /var/lib/homelab-health/service-readiness-mealie.txt
2026-07-09T19:46:33.3515780290 299 /var/lib/homelab-health/service-readiness-paperless.txt
2026-07-09T19:49:28.4202504550 212 /var/lib/homelab-health/memos-restore.txt
2026-07-09T19:50:22.5980774800 261 /var/lib/homelab-health/service-readiness-memos.txt
2026-07-09T20:00:19.8381943590 214 /var/lib/homelab-health/linkding-backup.txt
2026-07-09T20:00:19.8381943590 227 /var/lib/homelab-health/linkding-offhost.txt
2026-07-09T20:00:19.8381943590 227 /var/lib/homelab-health/linkding-restore.txt
2026-07-09T20:40:58.8843463730 270 /var/lib/homelab-health/service-readiness-linkding.txt
2026-07-09T20:47:41.7774675140 301 /var/lib/homelab-health/healthchecks-canonical-domain-repair.txt
2026-07-09T20:47:41.7774675140 340 /var/lib/homelab-health/service-readiness-healthchecks.txt
2026-07-09T20:51:20.8507958870 210 /var/lib/homelab-health/vikunja-backup.txt
2026-07-09T20:51:20.8507958870 222 /var/lib/homelab-health/vikunja-offhost.txt
2026-07-09T20:51:20.8507958870 222 /var/lib/homelab-health/vikunja-restore.txt
2026-07-09T20:55:48.6988688880 269 /var/lib/homelab-health/service-readiness-vikunja.txt
2026-07-09T20:58:33.5284046710 218 /var/lib/homelab-health/bookstack-backup.txt
2026-07-09T20:58:33.5284046710 232 /var/lib/homelab-health/bookstack-offhost.txt
2026-07-09T20:58:33.5284046710 232 /var/lib/homelab-health/bookstack-restore.txt
2026-07-09T21:01:03.7247153370 297 /var/lib/homelab-health/service-readiness-bookstack.txt
2026-07-09T21:08:55.5329737450 243 /var/lib/homelab-health/stirling-pdf-backup.txt
2026-07-09T21:08:55.5329737450 247 /var/lib/homelab-health/stirling-pdf-offhost.txt
2026-07-09T21:08:55.5329737450 247 /var/lib/homelab-health/stirling-pdf-restore.txt
2026-07-09T21:13:22.9720909840 269 /var/lib/homelab-health/service-readiness-stirling-pdf.txt
2026-07-09T21:16:17.6627935760 214 /var/lib/homelab-health/jellyfin-backup.txt
2026-07-09T21:16:17.6627935760 227 /var/lib/homelab-health/jellyfin-offhost.txt
2026-07-09T21:16:17.6627935760 227 /var/lib/homelab-health/jellyfin-restore.txt
2026-07-09T21:19:11.7814873230 270 /var/lib/homelab-health/service-readiness-jellyfin.txt
2026-07-09T21:22:25.6704869300 238 /var/lib/homelab-health/audiobookshelf-backup.txt
2026-07-09T21:22:25.6704869300 257 /var/lib/homelab-health/audiobookshelf-offhost.txt
2026-07-09T21:22:25.6704869300 257 /var/lib/homelab-health/audiobookshelf-restore.txt
2026-07-09T21:25:22.5602235460 279 /var/lib/homelab-health/service-readiness-audiobookshelf.txt
2026-07-09T21:46:20.9338292150 225 /var/lib/homelab-health/calibre-web-backup.txt
2026-07-09T21:46:20.9338292150 242 /var/lib/homelab-health/calibre-web-offhost.txt
2026-07-09T21:46:20.9338292150 242 /var/lib/homelab-health/calibre-web-restore.txt
2026-07-09T21:57:20.3030989240 271 /var/lib/homelab-health/service-readiness-calibre-web.txt
2026-07-09T22:04:04.0733864250 236 /var/lib/homelab-health/homeassistant-backup.txt
2026-07-09T22:04:04.0733864250 252 /var/lib/homelab-health/homeassistant-offhost.txt
2026-07-09T22:04:04.0733864250 252 /var/lib/homelab-health/homeassistant-restore.txt
2026-07-09T22:07:41.2227931800 281 /var/lib/homelab-health/service-readiness-homeassistant.txt
2026-07-09T22:13:14.6980274280 225 /var/lib/homelab-health/it-tools-backup.txt
2026-07-09T22:13:14.6980274280 227 /var/lib/homelab-health/it-tools-offhost.txt
2026-07-09T22:13:14.6980274280 227 /var/lib/homelab-health/it-tools-restore.txt
2026-07-09T22:23:57.3350983150 271 /var/lib/homelab-health/service-readiness-it-tools.txt
2026-07-09T22:28:35.5724369860 214 /var/lib/homelab-health/karakeep-backup.txt
2026-07-09T22:28:35.5724369860 227 /var/lib/homelab-health/karakeep-offhost.txt
2026-07-09T22:28:35.5724369860 227 /var/lib/homelab-health/karakeep-restore.txt
2026-07-09T22:32:36.7481977520 327 /var/lib/homelab-health/service-readiness-karakeep.txt
2026-07-09T22:39:24.2375519320 194 /var/lib/homelab-health/n8n-backup.txt
2026-07-09T22:39:24.2375519320 202 /var/lib/homelab-health/n8n-offhost.txt
2026-07-09T22:39:24.2375519320 202 /var/lib/homelab-health/n8n-restore.txt
2026-07-09T22:42:27.8124145040 261 /var/lib/homelab-health/service-readiness-n8n.txt
2026-07-09T22:46:45.7324363630 214 /var/lib/homelab-health/node-red-backup.txt
2026-07-09T22:46:45.7324363630 227 /var/lib/homelab-health/node-red-offhost.txt
2026-07-09T22:46:45.7324363630 227 /var/lib/homelab-health/node-red-restore.txt
2026-07-09T22:50:18.7317577520 270 /var/lib/homelab-health/service-readiness-node-red.txt
2026-07-09T22:55:13.5023542440 217 /var/lib/homelab-health/syncthing-backup.txt
2026-07-09T22:55:13.5023542440 232 /var/lib/homelab-health/syncthing-offhost.txt
2026-07-09T22:55:13.5023542440 232 /var/lib/homelab-health/syncthing-restore.txt
2026-07-09T22:59:10.6320324800 273 /var/lib/homelab-health/service-readiness-syncthing.txt
2026-07-09T23:09:25.0745502100 254 /var/lib/homelab-health/immich-restore.txt
2026-07-09T23:09:25.0745502100 278 /var/lib/homelab-health/immich-backup.txt
2026-07-09T23:09:25.0745502100 287 /var/lib/homelab-health/immich-offhost.txt
2026-07-09T23:12:55.3968081120 349 /var/lib/homelab-health/service-readiness-immich.txt
2026-07-09T23:39:37.9114415990 206 /var/lib/homelab-health/gotify-backup.txt
2026-07-09T23:39:37.9124416140 217 /var/lib/homelab-health/gotify-offhost.txt
2026-07-09T23:39:37.9124416140 217 /var/lib/homelab-health/gotify-restore.txt
2026-07-09T23:39:41.3274938750 198 /var/lib/homelab-health/ntfy-backup.txt
2026-07-09T23:39:41.3274938750 207 /var/lib/homelab-health/ntfy-offhost.txt
2026-07-09T23:39:41.3274938750 207 /var/lib/homelab-health/ntfy-restore.txt
2026-07-09T23:39:50.5946356910 229 /var/lib/homelab-health/uptime-kuma-backup.txt
2026-07-09T23:39:50.5946356910 242 /var/lib/homelab-health/uptime-kuma-offhost.txt
2026-07-09T23:39:50.5946356910 242 /var/lib/homelab-health/uptime-kuma-restore.txt
2026-07-09T23:45:51.3911568860 311 /var/lib/homelab-health/service-readiness-nextcloud.txt
2026-07-09T23:55:38.2600376160 277 /var/lib/homelab-health/service-readiness-dns1.txt
2026-07-09T23:55:38.2600376160 277 /var/lib/homelab-health/service-readiness-dns2.txt
2026-07-09T23:55:38.2600376160 284 /var/lib/homelab-health/service-readiness-homepage.txt
2026-07-09T23:58:29.2076174390 405 /var/lib/homelab-health/external-link-aleisaevn-netcraze-5083.txt
2026-07-10T00:12:45.3264837560 292 /var/lib/homelab-health/deep-service-backup-audit.txt
2026-07-10T21:46:36.9609141470 200 /var/lib/homelab-health/alertmanager-two-cloud-backup.txt
2026-07-10T22:05:25.8570406280 189 /var/lib/homelab-health/beszel-two-cloud-backup.txt
2026-07-10T22:20:26.1397323180 209 /var/lib/homelab-health/blackbox-exporter-two-cloud-backup.txt
2026-07-10T23:00:03.1168349530 151 /var/lib/homelab-health/legacy-monitoring-backups-migrated.txt
2026-07-10T23:25:23.2350187140 422 /var/lib/homelab-health/drift-baseline-acceptance.txt
2026-07-11T00:52:36.4697413270 151 /var/lib/homelab-health/cluster-admin-inventory-sync.txt
2026-07-11T02:25:25.1542644770 188 /var/lib/homelab-health/cluster-admin-public-route.txt
2026-07-11T02:26:01.7218294760 176 /var/lib/homelab-health/cluster-admin-prometheus-integration.txt
2026-07-11T02:30:49.6252637080 249 /var/lib/homelab-health/cluster-admin-incident-engine-backup.txt
2026-07-11T04:07:23.3393000650 361 /var/lib/homelab-health/service-readiness-cluster-admin-incident-engine.txt
2026-07-11T04:39:28.9976540970 212 /var/lib/homelab-health/cluster-admin-webpanel.txt
2026-07-11T06:23:40.9114023280 361 /var/lib/homelab-health/cluster-mail-cloud-cleanup.txt
2026-07-13T01:15:40.0109283350 386 /var/lib/homelab-health/cluster-admin-stage0-remediation.txt
2026-07-13T02:08:25.4011133770 436 /var/lib/homelab-health/cluster-admin-stage0-desired-state-sync.txt
2026-07-13T02:14:59.9458929520 400 /var/lib/homelab-health/cluster-admin-stage0-convergence.txt
2026-07-13T02:22:57.9862108430 523 /var/lib/homelab-health/cluster-admin-stage0-readiness-fix.txt
2026-07-13T02:26:02.8616913120 587 /var/lib/homelab-health/cluster-admin-stage0-baseline.txt
2026-07-13T02:32:35.0280552640 515 /var/lib/homelab-health/cluster-admin-stage1-capture.txt
2026-07-13T02:35:09.6424265770 727 /var/lib/homelab-health/cluster-admin-stage1-design-inputs.txt
2026-07-13T02:46:43.8820757430 698 /var/lib/homelab-health/cluster-admin-stage1-data-shape.txt
2026-07-13T03:11:06.6805116280 1065 /var/lib/homelab-health/cluster-admin-stage1-migration-dry-run.txt
2026-07-13T03:11:06.6805116280 406 /var/lib/homelab-health/cluster-admin-stage1-migration-compatibility.txt
2026-07-13T03:11:06.7005119340 926 /var/lib/homelab-health/cluster-admin-stage1-preapply-backup.txt
2026-07-13T03:22:22.4838707540 898 /var/lib/homelab-health/cluster-admin-stage1-foundation-apply.txt
2026-07-13T03:26:26.3016083230 956 /var/lib/homelab-health/cluster-admin-stage1-final-seal.txt
2026-07-13T03:29:40.0025776360 821 /var/lib/homelab-health/appbackupctl-routing.txt
2026-07-13T03:35:17.3427496850 1215 /var/lib/homelab-health/cluster-admin-stage2-design.txt
2026-07-13T03:39:46.1788887390 1361 /var/lib/homelab-health/cluster-admin-stage2-dry-run.txt
2026-07-13T04:05:34.7337002360 854 /var/lib/homelab-health/cluster-admin-stage2-recovery.txt
2026-07-13T04:15:16.8986427760 1035 /var/lib/homelab-health/cluster-admin-stage2-placeholder-diagnosis.txt
2026-07-13T04:20:17.9572677960 1318 /var/lib/homelab-health/cluster-admin-stage2-dynamic-redryrun.txt
2026-07-13T04:24:04.9957556580 1727 /var/lib/homelab-health/cluster-admin-stage2-preapply-backup.txt
2026-07-13T04:39:27.6419256780 1705 /var/lib/homelab-health/cluster-admin-stage2-apply.txt
2026-07-13T04:49:49.9254840660 1338 /var/lib/homelab-health/cluster-admin-stage2-final-seal.txt
2026-07-13T05:04:04.9066160570 1349 /var/lib/homelab-health/cluster-admin-stage3-api-design.txt
2026-07-13T05:36:13.0711962680 854 /var/lib/homelab-health/cluster-admin-stage3-api-auth-diagnosis.txt
2026-07-13T05:45:21.0465781570 1724 /var/lib/homelab-health/cluster-admin-stage3-api-dry-run.txt
2026-07-13T05:53:43.0632562670 1433 /var/lib/homelab-health/cluster-admin-stage3-api-runtime-dry-run.txt
2026-07-13T06:05:42.4882482970 728 /var/lib/homelab-health/cluster-admin-stage3-preapply-checkpoint.txt
2026-07-13T06:15:56.2396234100 1118 /var/lib/homelab-health/cluster-admin-stage3-preapply-backup.txt
2026-07-13T06:20:26.7327543700 1191 /var/lib/homelab-health/cluster-admin-stage3-api-candidate-normalized.txt
2026-07-13T06:22:40.5977987490 1106 /var/lib/homelab-health/cluster-admin-stage3-api-apply.txt
2026-07-13T06:25:40.2405422500 732 /var/lib/homelab-health/cluster-admin-stage3-production-verifier-prepared.txt
2026-07-13T06:28:56.1285342090 1207 /var/lib/homelab-health/cluster-admin-stage3-production-verify.txt
2026-07-13T06:44:42.7609957740 972 /var/lib/homelab-health/cluster-admin-stage3-postapply-backup.txt
2026-07-13T06:47:15.5353299170 1281 /var/lib/homelab-health/cluster-admin-stage3-final-seal.txt
2026-07-13T06:50:50.2356102460 507 /var/lib/homelab-health/crypto-portfolio-remove-grt-discovery.txt
2026-07-13T06:52:07.9807980800 488 /var/lib/homelab-health/crypto-portfolio-remove-grt-target.txt
2026-07-13T07:00:02.5170483350 822 /var/lib/homelab-health/crypto-portfolio-remove-grt-current.txt
2026-07-13T07:05:43.6132589850 879 /var/lib/homelab-health/crypto-portfolio-remove-grt-surface.txt
2026-07-13T07:20:46.9160557760 1741 /var/lib/homelab-health/crypto-portfolio-remove-grt-candidate.txt
2026-07-13T07:35:30.8895464510 1403 /var/lib/homelab-health/crypto-portfolio-remove-grt-runtime-verifier-prepared.txt
2026-07-13T07:41:31.9430520860 1497 /var/lib/homelab-health/crypto-portfolio-remove-grt-runtime-dry-run.txt
2026-07-13T07:52:08.1187504330 1724 /var/lib/homelab-health/crypto-portfolio-remove-grt-browser-smoke.txt
2026-07-13T08:05:15.1507543740 762 /var/lib/homelab-health/crypto-portfolio-remove-grt-desired-state-diagnostic.txt
2026-07-13T08:07:21.2856783820 1184 /var/lib/homelab-health/crypto-portfolio-remove-grt-desired-state-model.txt
2026-07-13T08:10:44.7277816120 1647 /var/lib/homelab-health/crypto-portfolio-remove-grt-prechange-backup.txt
2026-07-13T08:39:10.8547948980 330 /var/lib/homelab-health/crypto-portfolio-remove-grt-recovery-audit.txt
2026-07-13T08:41:01.3664795150 684 /var/lib/homelab-health/crypto-portfolio-remove-grt-failed-units-diagnostic.txt
2026-07-13T08:48:13.2600633440 1640 /var/lib/homelab-health/crypto-portfolio-remove-grt-production-apply.txt
2026-07-13T09:11:24.1542650490 1347 /var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-candidate.txt
2026-07-13T10:04:47.2300377080 1490 /var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-browser-smoke.txt
2026-07-13T10:42:57.1487691660 1851 /var/lib/homelab-health/crypto-portfolio-ui-preapply-backup.txt
2026-07-13T11:12:13.3023976030 192 /var/lib/homelab-health/crypto-portfolio-restore.txt
2026-07-13T11:12:13.3023976030 266 /var/lib/homelab-health/crypto-portfolio-offhost.txt
2026-07-13T11:12:13.3023976030 325 /var/lib/homelab-health/service-readiness-crypto-portfolio.txt
2026-07-13T11:12:13.3023976030 353 /var/lib/homelab-health/crypto-portfolio-backup.txt
2026-07-13T11:12:14.9294222730 757 /var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-production-closure.txt
2026-07-13T11:37:59.5159486840 1407 /var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-final-seal.txt
2026-07-13T11:37:59.5171037700 1407 /var/lib/homelab-health/crypto-portfolio-runtime-final-seal.txt
2026-07-13T14:03:06.0152234300 388 /var/lib/homelab-health/hapusya-site-preapply-backup.txt
2026-07-13T14:34:59.2083653510 533 /var/lib/homelab-health/hapusya-site-v2-preapply-backup.txt
2026-07-13T14:45:27.4649272800 793 /var/lib/homelab-health/hapusya-site-production-apply.txt
2026-07-13T14:48:15.6134864600 694 /var/lib/homelab-health/hapusya-site-production-browser.txt
2026-07-13T15:10:58.1372215280 156 /var/lib/homelab-health/hapusya-site-offhost.txt
2026-07-13T15:10:58.1372215280 185 /var/lib/homelab-health/hapusya-site-retention.txt
2026-07-13T15:10:58.1372215280 194 /var/lib/homelab-health/hapusya-site-restore.txt
2026-07-13T15:10:58.1372215280 499 /var/lib/homelab-health/hapusya-site-backup.txt
2026-07-13T15:10:58.1382215430 424 /var/lib/homelab-health/hapusya-site-backup-path-mapping.txt
2026-07-13T15:26:19.5872390600 651 /var/lib/homelab-health/hapusya-site-desired-state-sync.txt
2026-07-13T15:33:08.6444630100 1520 /var/lib/homelab-health/hapusya-site-final-seal.txt
2026-07-13T20:29:02.8967807310 628 /var/lib/homelab-health/backup-matrix-vm-health-feedback-fix.txt
2026-07-13T20:40:37.0653547720 648 /var/lib/homelab-health/backup-matrix-desired-state-persistence.txt
2026-07-13T20:48:14.6313253050 943 /var/lib/homelab-health/backup-matrix-cluster-preseal-convergence.txt
2026-07-13T20:51:32.4163384480 887 /var/lib/homelab-health/backup-matrix-cluster-desired-state-sync.txt
2026-07-13T20:51:32.4223385400 1445 /var/lib/homelab-health/backup-matrix-cluster-final-seal.txt
2026-07-13T21:13:01.1129724320 556 /var/lib/homelab-health/mail-cloud-edge-vm-runtime-contract-fix.txt
2026-07-13T21:15:43.9664537840 670 /var/lib/homelab-health/mail-cloud-edge-vm-readonly-probe.txt
2026-07-13T21:27:22.2230931620 630 /var/lib/homelab-health/mail-cloud-edge-vm-controlled-no-retention-mode.txt
2026-07-13T21:39:22.4540680390 719 /var/lib/homelab-health/mail-cloud-edge-vm-controlled-upload.txt
2026-07-13T21:40:21.6099694610 408 /var/lib/homelab-health/mail-cloud-edge-vm-service-convergence.txt
2026-07-13T21:56:30.9657631360 881 /var/lib/homelab-health/mail-cloud-edge-vm-matrix-registration.txt
2026-07-14T04:02:58.3995631560 1245 /var/lib/homelab-health/mail-cloud-edge-vm-final-seal.txt
2026-07-14T04:12:54.7747139510 707 /var/lib/homelab-health/remaining-special-families-matrix-registration.txt
2026-07-14T04:15:48.2773761880 1115 /var/lib/homelab-health/special-vm-emergency-non-app-final-seal.txt
2026-07-14T04:47:22.1524259850 902 /var/lib/homelab-health/cluster-admin-node-online-green-dot-fix.txt
2026-07-14T04:54:55.1333752320 1021 /var/lib/homelab-health/cluster-admin-collapsible-health-sections.txt
2026-07-14T08:33:26.7124814180 629 /var/lib/homelab-health/cluster-admin-stage4c-database-migration-dry-run.txt
2026-07-14T08:48:04.5210170900 482 /var/lib/homelab-health/homelab-admin-runner.txt
2026-07-14T09:14:58.6397106130 702 /var/lib/homelab-health/cluster-admin-stage4d-adapter-candidates-in-isolation.txt
2026-07-14T09:39:00.0749801060 610 /var/lib/homelab-health/cluster-admin-stage4e-restricted-probe-agent-design.txt
2026-07-14T10:39:53.2848761800 600 /var/lib/homelab-health/homelab-admin-guardrails.txt
2026-07-14T10:54:50.6815482590 692 /var/lib/homelab-health/cluster-admin-stage4f-preapply-backup.txt
2026-07-14T20:02:48.4191051180 266 /var/lib/homelab-health/cluster-admin-stage4g-production-apply-read-only.txt
2026-07-14T20:16:32.3610333130 213 /var/lib/homelab-health/operator-rule-close-tails-immediately.txt
2026-07-14T20:23:46.5434583710 328 /var/lib/homelab-health/cluster-admin-stage4h-scope-definition.txt
2026-07-14T20:29:41.0475984260 268 /var/lib/homelab-health/cluster-admin-stage4h-collector-baseline-audit.txt
2026-07-14T20:41:34.5353531810 455 /var/lib/homelab-health/cluster-admin-stage4h-collector-patch-candidate.txt
2026-07-14T21:26:54.1724254520 226 /var/lib/homelab-health/operator-command-size-guard.txt
2026-07-14T21:29:30.0818406610 312 /var/lib/homelab-health/cluster-admin-stage4h-final-acceptance-contract.txt
2026-07-14T21:33:40.0697208200 209 /var/lib/homelab-health/homelab-admin-cli-contract.txt
2026-07-14T21:53:16.5230291610 357 /var/lib/homelab-health/cluster-admin-stage4h-task-package-build.txt
2026-07-15T10:10:19.9159742040 1732 /var/lib/homelab-health/cluster-admin-stage4h-final-acceptance.txt
2026-07-15T10:17:53.0215082300 921 /var/lib/homelab-health/cluster-admin-stage4i-scope-definition.txt
2026-07-16T01:32:11.0876121490 237 /var/lib/homelab-health/homelab-ops-bootstrap.txt
2026-07-16T09:20:34.5941094330 75 /var/lib/homelab-health/router-hard-power.txt
2026-07-16T12:14:29.4893285610 202 /var/lib/homelab-health/xenforo-branding-rollout.txt
2026-07-16T12:41:28.8422409190 192 /var/lib/homelab-health/xenforo-branding-header-visibility.txt
2026-07-16T12:48:31.4566023130 185 /var/lib/homelab-health/xenforo-branding-header-order.txt
2026-07-16T13:01:16.2035467580 250 /var/lib/homelab-health/homelab-ops-worker-vm190.txt
2026-07-17T04:59:42.6476420400 242 /var/lib/homelab-health/netbird-vps-offhost.txt
2026-07-17T09:31:15.5873108310 443 /var/lib/homelab-health/paperless-offhost.txt
2026-07-17T14:21:55.3567278930 272 /var/lib/homelab-health/cluster-storage-retention.txt
2026-07-18T01:52:29.8931662090 684 /var/lib/homelab-health/paperless-backup.txt
2026-07-18T01:54:49.4850009180 728 /var/lib/homelab-health/netbird-vps-trust.txt
2026-07-18T02:54:28.2167554790 200 /var/lib/homelab-health/full-handoff-recovery.txt
2026-07-19T09:41:56.0870397860 437 /var/lib/homelab-health/mail-cloud-edge-vm.txt
2026-07-20T09:13:15.5923007820 193 /var/lib/homelab-health/memos-offhost.txt
2026-07-21T00:33:12.9785038850 214 /var/lib/homelab-health/evidence-seal/latest.env
2026-07-21T02:37:04.7969419110 195 /var/lib/homelab-health/cluster-admin-observer.txt
2026-07-21T02:37:04.7969419110 200 /var/lib/homelab-health/cluster-admin-restricted-probes.txt
2026-07-21T02:37:04.7969419110 219 /var/lib/homelab-health/cluster-admin-full-observer.txt
2026-07-21T02:39:14.3120870600 153 /var/lib/homelab-health/vm160-remote-backup.txt
2026-07-21T04:55:35.3129057980 316 /var/lib/homelab-health/edge-vm-vzdump-offhost.txt
2026-07-21T04:55:35.3129057980 334 /var/lib/homelab-health/edge-vm-vzdump-restore.txt
2026-07-21T04:55:35.3129057980 409 /var/lib/homelab-health/edge-vm-offhost-freshness.txt
2026-07-21T04:55:35.3408368720 314 /var/lib/homelab-health/sops-edge-secret-coverage.txt
2026-07-21T05:18:05.9056051500 633 /var/lib/homelab-health/router-running-config-mail-cloud.txt
2026-07-21T05:42:36.0601490370 236 /var/lib/homelab-health/vps-identity-audit.txt
2026-07-21T06:42:44.4264139010 339 /var/lib/homelab-health/mail-cloud-critical.txt
2026-07-21T07:05:43.0783306490 1583 /var/lib/homelab-health/backup-last-run.log
2026-07-21T07:05:43.0870410520 217 /var/lib/homelab-health/backup.txt
2026-07-21T07:05:43.0879592500 274 /var/lib/homelab-health/backup.json
2026-07-21T07:15:46.8155363020 232 /var/lib/homelab-health/drift-check.txt
2026-07-21T07:36:59.5343449800 170 /var/lib/homelab-health/vm-backup.txt
2026-07-21T07:49:15.8201883610 136 /var/lib/homelab-health/desired-state.txt
2026-07-21T07:56:14.9677282530 146 /var/lib/homelab-health/runbooks.txt
2026-07-21T07:56:26.4127609750 475 /var/lib/homelab-health/mail-cloud-nextcloud-vm.txt
2026-07-21T08:01:15.0701844650 1522 /var/lib/homelab-health/overall.txt
2026-07-21T08:10:47.5199010040 117 /var/lib/homelab-health/duty-admin.txt
2026-07-21T08:29:16.9008054160 178 /var/lib/homelab-health/capacity-risk.txt
2026-07-21T08:31:47.8421084810 128 /var/lib/homelab-health/final-readiness.txt
2026-07-21T08:35:41.8646792150 147 /var/lib/homelab-health/secret-exposure.txt
2026-07-21T08:50:15.1546349960 698 /var/lib/homelab-health/filebrowser-backup.txt
2026-07-21T08:50:15.1749916820 796 /var/lib/homelab-health/filebrowser-restore.txt
2026-07-21T08:50:15.2338941910 811 /var/lib/homelab-health/npmplus-kuma-config-backup.txt
2026-07-21T08:50:15.2339779780 812 /var/lib/homelab-health/npmplus-kuma-config-offhost.txt
2026-07-21T08:50:15.2340536730 812 /var/lib/homelab-health/npmplus-kuma-config-restore.txt
2026-07-21T08:50:15.2530255710 754 /var/lib/homelab-health/authentik-backup.txt
2026-07-21T08:50:15.2531096640 755 /var/lib/homelab-health/authentik-offhost.txt
2026-07-21T08:50:15.2531846660 755 /var/lib/homelab-health/authentik-restore.txt
2026-07-21T08:50:15.2737214530 738 /var/lib/homelab-health/gitea-backup.txt
2026-07-21T08:50:15.2738013060 739 /var/lib/homelab-health/gitea-offhost.txt
2026-07-21T08:50:15.2738768280 739 /var/lib/homelab-health/gitea-restore.txt
2026-07-21T08:50:15.2739499850 744 /var/lib/homelab-health/vaultwarden-backup.txt
2026-07-21T08:50:15.2740214440 745 /var/lib/homelab-health/vaultwarden-offhost.txt
2026-07-21T08:50:15.2740905310 745 /var/lib/homelab-health/vaultwarden-restore.txt
2026-07-21T08:50:15.3412420180 805 /var/lib/homelab-health/backup-coverage-matrix.txt
2026-07-21T08:50:15.4230013070 769 /var/lib/homelab-health/edge-vm-vzdump-backup.txt
2026-07-21T08:50:15.4602967240 776 /var/lib/homelab-health/extended-appbackup.txt
2026-07-21T08:50:15.6009660870 829 /var/lib/homelab-health/mail-cloud-restore-drill.txt
2026-07-21T08:50:15.6983060740 802 /var/lib/homelab-health/backup-framework.txt
2026-07-21T08:50:15.7459043920 760 /var/lib/homelab-health/vm-backup-policy.txt
2026-07-21T08:50:15.7701962780 763 /var/lib/homelab-health/vm170-vm171-full-backup.txt
2026-07-21T08:50:15.7943029380 822 /var/lib/homelab-health/vm170-vm171-full-restore-drill.txt
2026-07-21T08:50:15.8187786300 800 /var/lib/homelab-health/vm-local-dumps-cloud.txt
2026-07-21T08:50:15.8439264310 826 /var/lib/homelab-health/vm-local-dumps-retention.txt
2026-07-21T08:50:15.8671718640 718 /var/lib/homelab-health/immich-media-backup.txt
2026-07-21T08:50:15.8912318140 656 /var/lib/homelab-health/memos-backup.txt
2026-07-21T08:50:15.9706145680 773 /var/lib/homelab-health/paperless-restore.txt
2026-07-21T08:51:15.1419524730 115 /var/lib/homelab-health/cluster-passport.txt
2026-07-21T08:51:15.4733058530 323 /var/lib/homelab-health/immich-media-offhost.txt
2026-07-21T08:54:09.3706207380 509 /var/lib/homelab-health/golden-state.txt
2026-07-21T09:00:14.9472448680 142 /var/lib/homelab-health/incident-journal.txt
2026-07-21T09:00:16.1272438700 295 /var/lib/homelab-health/alerting.txt
2026-07-21T09:00:16.5952705670 327 /var/lib/homelab-health/kuma-monitor-policy.txt
2026-07-21T09:00:17.1552793000 154 /var/lib/homelab-health/backup-sla.txt
2026-07-21T09:00:17.1710283480 141 /var/lib/homelab-health/service-registry.txt
2026-07-21T09:00:17.1903657410 128 /var/lib/homelab-health/dependency-map.txt
2026-07-21T09:00:17.1922798770 142 /var/lib/homelab-health/safe-autoheal.txt
2026-07-21T09:00:35.8775712520 539 /var/lib/homelab-health/external-probe-vps.txt
2026-07-21T09:00:42.8766803960 233 /var/lib/homelab-health/forum-snuffleupagus.txt
2026-07-21T09:01:18.5488450380 317 /var/lib/homelab-health/duty-admin-v2.txt
2026-07-21T09:01:30.9278621010 489 /var/lib/homelab-health/disk-space.txt
2026-07-21T09:02:24.8882711660 170 /var/lib/homelab-health/mkdocs-auto-refresh.txt
2026-07-21T09:04:30.1762249020 0 /var/lib/homelab-health/pve03-staging-capacity.txt.err
2026-07-21T09:04:30.5477223940 207 /var/lib/homelab-health/pve03-staging-capacity.txt
2026-07-21T09:10:15.0418809250 139 /var/lib/homelab-health/crypto-portfolio-chart-health.txt
2026-07-21T09:10:15.5101489600 215 /var/lib/homelab-health/cluster-admin-webpanel-sync.txt
2026-07-21T09:10:17.6889902650 641 /var/lib/homelab-health/private-vpn-hosts.txt
2026-07-21T09:10:47.2970978580 227 /var/lib/homelab-health/router-watchdog.txt
2026-07-21T09:10:48.3731143150 477 /var/lib/homelab-health/netbird-peers.txt

----- DIR=/var/lib/homelab-backup-matrix -----
2026-07-01T04:31:05.4336469050 312 /var/lib/homelab-backup-matrix/evidence/netbird_vps_backup/0-netbird-vps-backup.txt
2026-07-07T18:24:59.9494760330 357 /var/lib/homelab-backup-matrix/evidence/netbird_vps_backup/0-netbird-vps-trust.txt
2026-07-09T10:22:35.4762594970 286 /var/lib/homelab-backup-matrix/evidence/immich_media_backup/0-immich-media-backup.txt
2026-07-09T10:30:21.2893338800 393 /var/lib/homelab-backup-matrix/evidence/homelab_npmplus_kuma_config_backup/1-npmplus-kuma-config-offhost.txt
2026-07-09T10:30:21.2893338800 399 /var/lib/homelab-backup-matrix/evidence/homelab_npmplus_kuma_config_backup/2-npmplus-kuma-config-restore.txt
2026-07-09T10:30:21.2893338800 416 /var/lib/homelab-backup-matrix/evidence/homelab_npmplus_kuma_config_backup/0-npmplus-kuma-config-backup.txt
2026-07-09T15:42:17.7925620460 209 /var/lib/homelab-backup-matrix/evidence/filebrowser_restore_validate/0-filebrowser-restore.txt
2026-07-09T15:42:17.7925620460 263 /var/lib/homelab-backup-matrix/evidence/filebrowser_backup/0-filebrowser-backup.txt
2026-07-09T19:42:55.8422576240 221 /var/lib/homelab-backup-matrix/evidence/paperless_backup/0-paperless-backup.txt
2026-07-09T19:42:55.8422576240 232 /var/lib/homelab-backup-matrix/evidence/paperless_restore_dry_run/0-paperless-restore.txt
2026-07-09T19:49:28.4202504550 201 /var/lib/homelab-backup-matrix/evidence/memos_backup/0-memos-backup.txt
2026-07-09T23:31:32.0510186090 353 /var/lib/homelab-backup-matrix/evidence/homelab_mail_cloud_restore_drill/0-mail-cloud-restore-drill.txt
2026-07-09T23:39:31.6643460000 221 /var/lib/homelab-backup-matrix/evidence/homelab_authentik_backup/0-authentik-backup.txt
2026-07-09T23:39:31.6643460000 232 /var/lib/homelab-backup-matrix/evidence/homelab_authentik_backup/1-authentik-offhost.txt
2026-07-09T23:39:31.6643460000 232 /var/lib/homelab-backup-matrix/evidence/homelab_authentik_backup/2-authentik-restore.txt
2026-07-09T23:39:36.8524253930 205 /var/lib/homelab-backup-matrix/evidence/homelab_gitea_backup/0-gitea-backup.txt
2026-07-09T23:39:36.8524253930 212 /var/lib/homelab-backup-matrix/evidence/homelab_gitea_backup/1-gitea-offhost.txt
2026-07-09T23:39:36.8524253930 212 /var/lib/homelab-backup-matrix/evidence/homelab_gitea_backup/2-gitea-restore.txt
2026-07-09T23:39:53.9766874470 226 /var/lib/homelab-backup-matrix/evidence/homelab_vaultwarden_backup/0-vaultwarden-backup.txt
2026-07-09T23:39:53.9766874470 242 /var/lib/homelab-backup-matrix/evidence/homelab_vaultwarden_backup/1-vaultwarden-offhost.txt
2026-07-09T23:39:53.9766874470 242 /var/lib/homelab-backup-matrix/evidence/homelab_vaultwarden_backup/2-vaultwarden-restore.txt
2026-07-10T04:59:04.1128229600 277 /var/lib/homelab-backup-matrix/evidence/homelab_vm_backup_health/1-edge-vm-vzdump-backup.txt
2026-07-10T12:47:24.3262767100 11477 /var/lib/homelab-backup-matrix/evidence-map.json
2026-07-10T12:57:44.4526176160 1657 /var/lib/homelab-backup-matrix/legacy-health-cutover-paths.txt
2026-07-10T13:10:05.0808425340 11888 /var/lib/homelab-backup-matrix/migration-plan.json
2026-07-12T04:56:52.9608056990 277 /var/lib/homelab-backup-matrix/evidence/homelab_edge_vm_vzdump_backup/0-edge-vm-vzdump-backup.txt
2026-07-13T20:26:36.9252845450 748 /var/lib/homelab-backup-matrix/compat/homelab_vm_backup_health--backup.txt--62f47c24aa6b.txt
2026-07-13T20:26:36.9253893680 764 /var/lib/homelab-backup-matrix/compat/homelab_vm_backup_health--edge-vm-vzdump-offhost.txt--6b16ee077767.txt
2026-07-13T20:26:36.9254686050 764 /var/lib/homelab-backup-matrix/compat/homelab_vm_backup_health--edge-vm-vzdump-restore.txt--bfe4c7a45ee2.txt
2026-07-13T20:26:36.9255441420 751 /var/lib/homelab-backup-matrix/compat/homelab_vm_backup_health--vm-backup.txt--11c1b9f3c8ae.txt
2026-07-15T08:44:36.3676757710 515 /var/lib/homelab-backup-matrix/evidence/homelab_vm_full_restore_drill/1-latest-restore-drill.txt
2026-07-15T08:44:36.3692351120 515 /var/lib/homelab-backup-matrix/evidence/homelab_vm_full_restore_drill/0-vm170-vm171-full-restore-drill.txt
2026-07-18T01:52:29.8722644570 731 /var/lib/homelab-backup-matrix/compat/netbird_vps_backup--netbird-vps-trust.txt--44dfdc0d5d0a.txt
2026-07-18T01:52:29.8931212020 684 /var/lib/homelab-backup-matrix/compat/paperless_backup--paperless-backup.txt--b3a9ed8e7905.txt
2026-07-19T05:11:28.9578382090 404 /var/lib/homelab-backup-matrix/evidence/skladchik_reports_monitor_backup_restore_check/0-restore-check-health.txt
2026-07-19T09:41:56.0870397860 437 /var/lib/homelab-backup-matrix/evidence/homelab_mail_cloud_edge_vm_upload/0-mail-cloud-edge-vm.txt
2026-07-19T10:13:40.6950277460 544 /var/lib/homelab-backup-matrix/evidence/homelab_vm_local_dumps_retention/1-latest-vm-local-dumps-cloud.json
2026-07-19T10:13:40.6960277610 445 /var/lib/homelab-backup-matrix/evidence/homelab_vm_local_dumps_2cloud/0-vm-local-dumps-cloud.txt
2026-07-19T11:08:42.8436095010 184 /var/lib/homelab-backup-matrix/evidence/homelab_vm_local_dumps_retention/0-vm-local-dumps-retention.txt
2026-07-20T08:47:17.4079908050 509 /var/lib/homelab-backup-matrix/evidence/homelab_golden_state_index/0-golden-state.txt
2026-07-20T17:21:30.2733358650 641 /var/lib/homelab-backup-matrix/evidence/homelab_npmplus_kuma_config_backup/3-private-vpn-hosts.txt
2026-07-21T02:39:14.3120870600 153 /var/lib/homelab-backup-matrix/evidence/vm160_remote_backup/0-vm160-remote-backup.txt
2026-07-21T03:09:43.9100609310 485 /var/lib/homelab-backup-matrix/evidence/homelab_vm_full_backup/1-latest-full-backup.txt
2026-07-21T03:09:43.9161554950 485 /var/lib/homelab-backup-matrix/evidence/homelab_vm_full_backup/0-vm170-vm171-full-backup.txt
2026-07-21T03:13:42.8950037020 553 /var/lib/homelab-backup-matrix/evidence/homelab_emergency_backup/0-latest-emergency-pack.json
2026-07-21T03:13:42.8950037020 553 /var/lib/homelab-backup-matrix/evidence/homelab_verified_backup_health/1-latest-emergency-pack.json
2026-07-21T03:51:53.5230549890 601 /var/lib/homelab-backup-matrix/evidence/homelab_appbackup_dockge_stacks/0-latest-dockge-stacks.json
2026-07-21T04:19:48.6158169700 880 /var/lib/homelab-backup-matrix/evidence/homelab_appbackup_npmplus_edge/0-latest-npmplus-edge.json
2026-07-21T04:35:22.4622843520 484 /var/lib/homelab-backup-matrix/evidence/skladchik_reports_monitor_backup/0-health.txt
2026-07-21T04:55:35.3129057980 316 /var/lib/homelab-backup-matrix/evidence/homelab_vm_backup_health/2-edge-vm-vzdump-offhost.txt
2026-07-21T04:55:35.3129057980 334 /var/lib/homelab-backup-matrix/evidence/homelab_vm_backup_health/3-edge-vm-vzdump-restore.txt
2026-07-21T05:18:05.9056051500 633 /var/lib/homelab-backup-matrix/evidence/homelab_router_running_config_mail_cloud/0-router-running-config-mail-cloud.txt
2026-07-21T06:06:45.3183873770 213 /var/lib/homelab-backup-matrix/evidence/homelab_extended_appbackup/0-extended-appbackup.txt
2026-07-21T06:42:44.4264139010 339 /var/lib/homelab-backup-matrix/evidence/homelab_mail_cloud_critical_upload/0-mail-cloud-critical.txt
2026-07-21T06:45:16.5316587930 155 /var/lib/homelab-backup-matrix/evidence/homelab_verified_backup_health/2-backup-framework.txt
2026-07-21T06:45:16.5316587930 7256 /var/lib/homelab-backup-matrix/evidence/homelab_verified_backup_health/0-backup-registry-latest.txt
2026-07-21T07:05:43.0870410520 217 /var/lib/homelab-backup-matrix/evidence/homelab_vm_backup_health/0-backup.txt
2026-07-21T07:36:59.5343449800 170 /var/lib/homelab-backup-matrix/evidence/homelab_vm_backup_health/4-vm-backup.txt
2026-07-21T07:56:26.4127609750 475 /var/lib/homelab-backup-matrix/evidence/homelab_mail_cloud_nextcloud_vm_upload/0-mail-cloud-nextcloud-vm.txt
2026-07-21T08:30:13.9066752070 185 /var/lib/homelab-backup-matrix/evidence/homelab_vm_backup_policy/0-vm-backup-policy.txt
2026-07-21T08:45:17.1624701130 154 /var/lib/homelab-backup-matrix/evidence/homelab_backup_sla_health/0-backup-sla.txt
2026-07-21T08:50:15.0600323370 317 /var/lib/homelab-backup-matrix/evidence/homelab_backup_coverage_matrix/0-backup-coverage-matrix.txt
2026-07-21T08:50:15.1342359450 492 /var/lib/homelab-backup-matrix/rendered/dpkg-db-backup.txt
2026-07-21T08:50:15.1540337770 538 /var/lib/homelab-backup-matrix/rendered/filebrowser-backup.txt
2026-07-21T08:50:15.1545768810 698 /var/lib/homelab-backup-matrix/compat/filebrowser_backup--filebrowser-backup.txt--51237e75817c.txt
2026-07-21T08:50:15.1748329110 625 /var/lib/homelab-backup-matrix/rendered/filebrowser-restore-validate.txt
2026-07-21T08:50:15.1749419720 796 /var/lib/homelab-backup-matrix/compat/filebrowser_restore_validate--filebrowser-restore.txt--0340a8b1a5de.txt
2026-07-21T08:50:15.1958070440 637 /var/lib/homelab-backup-matrix/rendered/homelab-appbackup-dockge-stacks.txt
2026-07-21T08:50:15.2143009170 577 /var/lib/homelab-backup-matrix/rendered/homelab-appbackup-mariadb-batch.txt
2026-07-21T08:50:15.2337385240 631 /var/lib/homelab-backup-matrix/rendered/homelab-appbackup-npmplus-edge.txt
2026-07-21T08:50:15.2338453140 811 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_npmplus_edge--npmplus-kuma-config-backup.txt--f5c6697bf2fc.txt
2026-07-21T08:50:15.2339418310 812 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_npmplus_edge--npmplus-kuma-config-offhost.txt--1638681c118d.txt
2026-07-21T08:50:15.2340198050 812 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_npmplus_edge--npmplus-kuma-config-restore.txt--6f07430aa43a.txt
2026-07-21T08:50:15.2340955590 802 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_npmplus_edge--private-vpn-hosts.txt--a14118858bd9.txt
2026-07-21T08:50:15.2528809530 582 /var/lib/homelab-backup-matrix/rendered/homelab-appbackup-postgres-batch.txt
2026-07-21T08:50:15.2529784080 754 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_postgres_batch--authentik-backup.txt--800cae69eb31.txt
2026-07-21T08:50:15.2530726290 755 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_postgres_batch--authentik-offhost.txt--d42912853ad6.txt
2026-07-21T08:50:15.2531503130 755 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_postgres_batch--authentik-restore.txt--b81c6165c2a7.txt
2026-07-21T08:50:15.2735658650 572 /var/lib/homelab-backup-matrix/rendered/homelab-appbackup-sqlite-batch.txt
2026-07-21T08:50:15.2736758420 738 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_sqlite_batch--gitea-backup.txt--4b591c9d4b9b.txt
2026-07-21T08:50:15.2737669460 739 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_sqlite_batch--gitea-offhost.txt--0fe7014b9333.txt
2026-07-21T08:50:15.2738429560 739 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_sqlite_batch--gitea-restore.txt--d0854df47f34.txt
2026-07-21T08:50:15.2739171710 744 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_sqlite_batch--vaultwarden-backup.txt--0c7639e2a8c2.txt
2026-07-21T08:50:15.2739891140 745 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_sqlite_batch--vaultwarden-offhost.txt--41b1d10c7f2a.txt
2026-07-21T08:50:15.2740584210 745 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_sqlite_batch--vaultwarden-restore.txt--2247a2977d58.txt
2026-07-21T08:50:15.3019034840 607 /var/lib/homelab-backup-matrix/rendered/homelab-authentik-backup.txt
2026-07-21T08:50:15.3210502310 518 /var/lib/homelab-backup-matrix/rendered/homelab-auto-backup.txt
2026-07-21T08:50:15.3409347590 629 /var/lib/homelab-backup-matrix/rendered/homelab-backup-coverage-matrix.txt
2026-07-21T08:50:15.3411214110 805 /var/lib/homelab-backup-matrix/compat/homelab_backup_coverage_matrix--backup-coverage-matrix.txt--fcdfdafd14d8.txt
2026-07-21T08:50:15.3609141800 542 /var/lib/homelab-backup-matrix/rendered/homelab-backup-freshness.txt
2026-07-21T08:50:15.3807132660 551 /var/lib/homelab-backup-matrix/rendered/homelab-backup-healthcheck.txt
2026-07-21T08:50:15.3997651780 598 /var/lib/homelab-backup-matrix/rendered/homelab-backup-sla-health.txt
2026-07-21T08:50:15.3998838880 757 /var/lib/homelab-backup-matrix/compat/homelab_backup_sla_health--backup-sla.txt--b7b83ec539a3.txt
2026-07-21T08:50:15.4228298640 595 /var/lib/homelab-backup-matrix/rendered/homelab-edge-vm-vzdump-backup.txt
2026-07-21T08:50:15.4229475270 769 /var/lib/homelab-backup-matrix/compat/homelab_edge_vm_vzdump_backup--edge-vm-vzdump-backup.txt--112363b34713.txt
2026-07-21T08:50:15.4427017570 603 /var/lib/homelab-backup-matrix/rendered/homelab-emergency-backup.txt
2026-07-21T08:50:15.4601250520 608 /var/lib/homelab-backup-matrix/rendered/homelab-extended-appbackup.txt
2026-07-21T08:50:15.4602433910 776 /var/lib/homelab-backup-matrix/compat/homelab_extended_appbackup--extended-appbackup.txt--9e6d2ebe69e3.txt
2026-07-21T08:50:15.4846247920 583 /var/lib/homelab-backup-matrix/rendered/homelab-gitea-backup.txt
2026-07-21T08:50:15.5053264870 603 /var/lib/homelab-backup-matrix/rendered/homelab-golden-state-index.txt
2026-07-21T08:50:15.5261090100 649 /var/lib/homelab-backup-matrix/rendered/homelab-mail-cloud-critical-upload.txt
2026-07-21T08:50:15.5497812040 645 /var/lib/homelab-backup-matrix/rendered/homelab-mail-cloud-edge-vm-upload.txt
2026-07-21T08:50:15.5751798050 673 /var/lib/homelab-backup-matrix/rendered/homelab-mail-cloud-nextcloud-vm-upload.txt
2026-07-21T08:50:15.6005257930 649 /var/lib/homelab-backup-matrix/rendered/homelab-mail-cloud-restore-drill.txt
2026-07-21T08:50:15.6008056360 829 /var/lib/homelab-backup-matrix/compat/homelab_mail_cloud_restore_drill--mail-cloud-restore-drill.txt--aec14f187258.txt
2026-07-21T08:50:15.6302773490 657 /var/lib/homelab-backup-matrix/rendered/homelab-npmplus-kuma-config-backup.txt
2026-07-21T08:50:15.6516842010 693 /var/lib/homelab-backup-matrix/rendered/homelab-router-running-config-mail-cloud.txt
2026-07-21T08:50:15.6783067700 619 /var/lib/homelab-backup-matrix/rendered/homelab-vaultwarden-backup.txt
2026-07-21T08:50:15.6981173180 632 /var/lib/homelab-backup-matrix/rendered/homelab-verified-backup-health.txt
2026-07-21T08:50:15.6982460880 802 /var/lib/homelab-backup-matrix/compat/homelab_verified_backup_health--backup-framework.txt--5cdc2f4dfaac.txt
2026-07-21T08:50:15.7231207570 589 /var/lib/homelab-backup-matrix/rendered/homelab-vm-backup-health.txt
2026-07-21T08:50:15.7457453080 596 /var/lib/homelab-backup-matrix/rendered/homelab-vm-backup-policy.txt
2026-07-21T08:50:15.7458568190 760 /var/lib/homelab-backup-matrix/compat/homelab_vm_backup_policy--vm-backup-policy.txt--b7c9d2d099d3.txt
2026-07-21T08:50:15.7698895230 594 /var/lib/homelab-backup-matrix/rendered/homelab-vm-full-backup.txt
2026-07-21T08:50:15.7700805790 763 /var/lib/homelab-backup-matrix/compat/homelab_vm_full_backup--vm170-vm171-full-backup.txt--84ca9b2d5334.txt
2026-07-21T08:50:15.7941438050 639 /var/lib/homelab-backup-matrix/rendered/homelab-vm-full-restore-drill.txt
2026-07-21T08:50:15.7942564800 822 /var/lib/homelab-backup-matrix/compat/homelab_vm_full_restore_drill--vm170-vm171-full-restore-drill.txt--296c241364b3.txt
2026-07-21T08:50:15.8186148570 627 /var/lib/homelab-backup-matrix/rendered/homelab-vm-local-dumps-2cloud.txt
2026-07-21T08:50:15.8187301490 800 /var/lib/homelab-backup-matrix/compat/homelab_vm_local_dumps_2cloud--vm-local-dumps-cloud.txt--b8453dc56a16.txt
2026-07-21T08:50:15.8437441190 646 /var/lib/homelab-backup-matrix/rendered/homelab-vm-local-dumps-retention.txt
2026-07-21T08:50:15.8438694010 826 /var/lib/homelab-backup-matrix/compat/homelab_vm_local_dumps_retention--vm-local-dumps-retention.txt--bba459c6cd8c.txt
2026-07-21T08:50:15.8667988910 556 /var/lib/homelab-backup-matrix/rendered/immich-media-backup.txt
2026-07-21T08:50:15.8670229770 718 /var/lib/homelab-backup-matrix/compat/immich_media_backup--immich-media-backup.txt--8994915171ad.txt
2026-07-21T08:50:15.8909020950 508 /var/lib/homelab-backup-matrix/rendered/memos-backup.txt
2026-07-21T08:50:15.8911074740 656 /var/lib/homelab-backup-matrix/compat/memos_backup--memos-backup.txt--ef6fd685e9e3.txt
2026-07-21T08:50:15.9190603340 576 /var/lib/homelab-backup-matrix/rendered/netbird-vps-backup.txt
2026-07-21T08:50:15.9488250690 537 /var/lib/homelab-backup-matrix/rendered/paperless-backup.txt
2026-07-21T08:50:15.9702101810 607 /var/lib/homelab-backup-matrix/rendered/paperless-restore-dry-run.txt
2026-07-21T08:50:15.9704189420 773 /var/lib/homelab-backup-matrix/compat/paperless_restore_dry_run--paperless-restore.txt--acfbc4e465fa.txt
2026-07-21T08:50:15.9918669420 627 /var/lib/homelab-backup-matrix/rendered/skladchik-reports-monitor-backup.txt
2026-07-21T08:50:16.0138324250 712 /var/lib/homelab-backup-matrix/rendered/skladchik-reports-monitor-backup-restore-check.txt
2026-07-21T08:50:16.0376266730 600 /var/lib/homelab-backup-matrix/rendered/vm160-remote-backup.txt
2026-07-21T08:50:16.0377791200 7616 /var/lib/homelab-backup-matrix/compat-map.json
2026-07-21T08:50:16.0380568130 36830 /var/lib/homelab-backup-matrix/state.json
2026-07-21T08:50:16.0381265160 357 /var/lib/homelab-backup-matrix/health.txt

----- DIR=/var/lib/homelab-cluster-admin -----
2026-07-09T02:15:16.4348302020 212 /var/lib/homelab-cluster-admin/restricted-probes-latest.md
2026-07-13T01:12:15.2820928160 357 /var/lib/homelab-cluster-admin/central-pull-latest.md
2026-07-21T02:37:04.7963344210 259 /var/lib/homelab-cluster-admin/full-observer-central-latest.md
2026-07-21T02:37:04.7963344210 272 /var/lib/homelab-cluster-admin/observer-remote-latest.txt

----- DIR=/var/lib/homelab-admin -----
2026-07-14T08:48:04.5070168760 2594 /var/lib/homelab-admin/current-state.json
2026-07-16T01:13:21.5726815610 93 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v1/.gitignore
2026-07-16T01:13:21.5727162640 1106 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v1/ACTIVE_CHANGES.md
2026-07-16T01:13:21.5727637410 3683 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v1/AGENTS.md
2026-07-16T01:13:21.5728454810 1206 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v1/README.md
2026-07-16T01:17:33.7922430400 25 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v1/.bootstrap-task-id
2026-07-16T01:17:34.8439897360 1147 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v1/CURRENT_CONTEXT.md
2026-07-16T01:24:23.7545066840 1206 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v2/README.md
2026-07-16T01:24:23.7545631560 3683 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v2/AGENTS.md
2026-07-16T01:24:23.7546125270 1106 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v2/ACTIVE_CHANGES.md
2026-07-16T01:24:23.7546326130 93 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v2/.gitignore
2026-07-16T01:26:14.8577149440 25 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v2/.bootstrap-task-id
2026-07-16T01:26:15.9203183270 1147 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v2/CURRENT_CONTEXT.md
2026-07-16T01:29:17.1899811830 216 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v2/apply-state.json
2026-07-16T03:11:54.7617119420 100 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-v3/bootstrap-key.pub
2026-07-16T03:11:54.7619281900 28 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-v3/.task-id
2026-07-16T03:11:54.7679096450 832 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-v3/plan.json
2026-07-16T12:12:29.3146422310 1352 /var/lib/homelab-admin/staging/xenforo-branding-rollout-v3/forum-map.tsv
2026-07-16T12:12:29.3157094100 92160 /var/lib/homelab-admin/staging/xenforo-branding-rollout-v3/payload.tar
2026-07-16T12:12:33.3952889200 28 /var/lib/homelab-admin/staging/xenforo-branding-rollout-v3/.task-id
2026-07-16T12:12:33.4021048840 371 /var/lib/homelab-admin/staging/xenforo-branding-rollout-v3/plan.json
2026-07-16T12:13:31.2199747110 323 /var/lib/homelab-admin/staging/xenforo-branding-rollout-v3/apply-state.json
2026-07-16T12:14:29.3376202110 242 /var/lib/homelab-admin/staging/xenforo-branding-rollout-v3/verify-state.json
2026-07-16T12:35:08.5536209670 340 /var/lib/homelab-admin/staging/xenforo-branding-header-visibility-v3/forum-map.tsv
2026-07-16T12:35:12.7600519140 38 /var/lib/homelab-admin/staging/xenforo-branding-header-visibility-v3/.task-id
2026-07-16T12:35:12.7617956360 259 /var/lib/homelab-admin/staging/xenforo-branding-header-visibility-v3/prepare.json
2026-07-16T12:35:15.3572982170 985 /var/lib/homelab-admin/staging/xenforo-branding-header-visibility-v3/forum-plan.tsv
2026-07-16T12:36:17.0107678250 292 /var/lib/homelab-admin/staging/xenforo-branding-header-visibility-v3/apply-state.json
2026-07-16T12:41:28.6907635740 282 /var/lib/homelab-admin/staging/xenforo-branding-header-visibility-v3/verify-state.json
2026-07-16T12:47:29.5876473720 340 /var/lib/homelab-admin/staging/xenforo-branding-header-order-v1/forum-map.tsv
2026-07-16T12:47:33.7730329860 33 /var/lib/homelab-admin/staging/xenforo-branding-header-order-v1/.task-id
2026-07-16T12:47:33.7750294120 254 /var/lib/homelab-admin/staging/xenforo-branding-header-order-v1/prepare.json
2026-07-16T12:47:36.3380684650 985 /var/lib/homelab-admin/staging/xenforo-branding-header-order-v1/forum-plan.tsv
2026-07-16T12:48:22.1017657740 287 /var/lib/homelab-admin/staging/xenforo-branding-header-order-v1/apply-state.json
2026-07-16T12:48:31.2981607510 287 /var/lib/homelab-admin/staging/xenforo-branding-header-order-v1/verify-state.json
2026-07-16T12:57:01.2226647630 100 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-v4/bootstrap-key.pub
2026-07-16T12:57:01.2229047360 28 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-v4/.task-id
2026-07-16T12:57:01.2280411540 832 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-v4/plan.json
2026-07-16T12:58:33.9325456800 251 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-v4/apply-state.json
2026-07-16T13:30:34.7942944220 27 /var/lib/homelab-admin/staging/pve03-kpartx-dependency-v1/.task-id
2026-07-16T13:30:35.6961419590 269 /var/lib/homelab-admin/staging/pve03-kpartx-dependency-v1/plan.json
2026-07-16T13:32:19.0726550040 46 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v3/.task-id
2026-07-16T13:32:19.0737338470 311 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v3/plan.json
2026-07-16T13:37:47.0276492630 46 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v4/.task-id
2026-07-16T13:37:47.0287839370 311 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v4/plan.json
2026-07-16T13:45:58.6002136100 46 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v5/.task-id
2026-07-16T13:45:58.6013488190 311 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v5/plan.json
2026-07-16T13:51:23.3271017050 46 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v6/.task-id
2026-07-16T13:51:23.3293401540 311 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v6/plan.json
2026-07-16T13:51:25.8933795560 692 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v6/guest-state-audit.json
2026-07-16T13:51:25.9020688350 95 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v6/known_hosts.ip
2026-07-16T14:08:50.9476829420 42 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-jq-dependency-v1/.task-id
2026-07-16T14:08:50.9485294130 95 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-jq-dependency-v1/known_hosts.ip
2026-07-16T14:08:51.4604803550 287 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-jq-dependency-v1/plan.json
2026-07-16T14:12:22.9698312740 42 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-jq-dependency-v2/.task-id
2026-07-16T14:12:22.9707119200 95 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-jq-dependency-v2/known_hosts.ip
2026-07-16T14:12:23.4917512910 287 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-jq-dependency-v2/plan.json
2026-07-16T14:13:50.1103799100 43 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-base-bootstrap-v1/.task-id
2026-07-16T14:13:50.1111850760 95 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-base-bootstrap-v1/known_hosts.ip
2026-07-16T14:13:50.8632590890 979 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-base-bootstrap-v1/plan.json
2026-07-16T15:56:47.1548416000 46 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-core-registration-v2/.task-id
2026-07-16T16:06:12.1374995170 46 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-core-registration-v3/.task-id
2026-07-16T16:06:31.4132842520 918 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-core-registration-v3/plan.json
2026-07-16T16:09:04.9578565910 40 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-core-registration-v3/netbox-created.json
2026-07-16T22:49:51.9162150280 49 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-gitea-actions-runner-v2/.task-id
2026-07-16T22:49:51.9169336150 95 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-gitea-actions-runner-v2/known_hosts.ip
2026-07-16T22:49:52.9123866110 546 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-gitea-actions-runner-v2/plan.json
2026-07-18T00:45:40.5710369220 37 /var/lib/homelab-admin/repair-stage/homelab-dev-null-repair-v1/before.txt
2026-07-18T00:45:41.1140451260 163 /var/lib/homelab-admin/seals/homelab-dev-null-repair-v1.json
2026-07-18T02:04:55.1701966370 376 /var/lib/homelab-admin/seals/homelab-backup-tail-closure-v1.json
2026-07-18T03:00:03.7692669430 682 /var/lib/homelab-admin/seals/homelab-full-handoff-recovery-v1.json
2026-07-18T03:00:04.0863952050 378 /var/lib/homelab-admin/seals/homelab-full-handoff-seal-normalization-v1.json
2026-07-18T04:06:17.0615961730 373 /var/lib/homelab-admin/seals/homelab-logrotate-namespace-latch-clearance-v1.json
2026-07-18T13:53:02.8576614270 469 /var/lib/homelab-admin/seals/skladchik-reports-monitor-controlled-reauth-v1.json
2026-07-18T13:53:05.3889306280 235 /var/lib/homelab-admin/seals/skladchik-reports-monitor-az52-url-hotfix-v1.json
2026-07-19T18:02:00.5391470400 30 /var/lib/homelab-admin/staging/pve03-r8152-rx-ring-canary-v1/.task-id
2026-07-19T18:02:00.5400167640 89 /var/lib/homelab-admin/staging/pve03-r8152-rx-ring-canary-v1/known-hosts.sha256
2026-07-19T18:02:00.5410167800 41 /var/lib/homelab-admin/staging/pve03-r8152-rx-ring-canary-v1/desired-head
2026-07-19T18:02:02.2810433540 275 /var/lib/homelab-admin/staging/pve03-r8152-rx-ring-canary-v1/remote-baseline.json
2026-07-19T18:02:02.2810433540 65 /var/lib/homelab-admin/staging/pve03-r8152-rx-ring-canary-v1/remote-script.sha256
2026-07-19T18:02:39.2949351360 30 /var/lib/homelab-admin/staging/pve03-r8152-rx-ring-canary-v1/seal-run-id
2026-07-19T18:02:39.3036087780 1000 /var/lib/homelab-admin/seals/pve03-r8152-rx-ring-canary-v1.json
2026-07-19T18:02:39.3076088390 65 /var/lib/homelab-admin/staging/pve03-r8152-rx-ring-canary-v1/canonical-seal.sha256

----- DIR=/var/lib/homelab-chat-run -----
2026-07-21T08:54:39.0070746130 133 /var/lib/homelab-chat-run/raw/TRANSPORT-WRAPPER-001-20260721T055438Z.log
2026-07-21T08:54:39.0280243940 133 /var/lib/homelab-chat-run/public/TRANSPORT-WRAPPER-001-20260721T055438Z.txt
2026-07-21T09:01:42.5216104950 96 /var/lib/homelab-chat-run/raw/TRANSPORT-WRAPPER-004-20260721T060142Z.log
2026-07-21T09:01:42.5447263200 96 /var/lib/homelab-chat-run/public/TRANSPORT-WRAPPER-004-20260721T060142Z.txt
2026-07-21T09:01:42.5660598920 1325 /var/lib/homelab-chat-run/public/TRANSPORT-WRAPPER-004-20260721T060142Z.json
2026-07-21T09:01:42.5676112120 872 /var/lib/homelab-chat-run/public/TRANSPORT-WRAPPER-004-20260721T060142Z-latest.txt
2026-07-21T09:05:16.9049535880 82 /var/lib/homelab-chat-run/raw/TRANSPORT-WRAPPER-INNER-005-20260721T060516Z.log
2026-07-21T09:05:16.9253045330 82 /var/lib/homelab-chat-run/public/TRANSPORT-WRAPPER-INNER-005-20260721T060516Z.txt
2026-07-21T09:05:16.9482306600 1316 /var/lib/homelab-chat-run/public/TRANSPORT-WRAPPER-INNER-005-20260721T060516Z.json
2026-07-21T09:05:16.9509543050 864 /var/lib/homelab-chat-run/public/TRANSPORT-WRAPPER-INNER-005-20260721T060516Z-latest.txt
2026-07-21T09:11:08.4524214090 248 /var/lib/homelab-chat-run/state/command-ledger.tsv
2026-07-21T09:11:08.4544214400 0 /var/lib/homelab-chat-run/state/runner.lock
2026-07-21T09:11:11.5614689600 319582 /var/lib/homelab-chat-run/raw/CONTEXT-SOURCES-001-20260721T061108Z.log

===== DISCOVERY COMPLETE =====
CHANGES_MADE=NO

OUTPUT_END
CHAT_OUTPUT_END
