homelab-admin/homelab-public-context
1
0
Fork 0
Code Activity
Files
e0ea57efc49c818e640c2476f89663dc2aa2e6b4
homelab-public-context/runtime/latest.txt
T
homelab-runtime-publisher e0ea57efc4 runtime: publish CONTEXT-SOURCES-001
2026-07-21 06:11:12 +00:00

4830 lines
314 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
CHAT_OUTPUT_BEGIN
COMMAND_ID=CONTEXT-SOURCES-001
STATUS=OK
RC=0
HOST=pve01
MODE=read-only
COMPONENT=cluster-context
REFERENCE_REGISTER_CHECK=OK
REFERENCE_SHA256=f6f7749fda946015116cdbbd754c7e7f20356b4a6cf4df330a1b0b8fdf59cbb2
ERROR_REGISTER_CHECK=OK
ERROR_REGISTER_SHA256=ab8308549db90d235b591af0581580c3c7ba1b287b129cc6111c2eef0a0f9ecc
COMMAND_SHA256=7ae13694e41dd5b1453b7b375daa2d8092a436d01d281e2fcd23aa0c4fac00ee
DUPLICATE_FAILED_COMMAND_BLOCKED=false
EXECUTION_STARTED=true
CHANGES_MADE=false
SANITIZED=yes
SECRETS_INCLUDED=no
PRIVATE_ADDRESSES_INCLUDED=no
RAW_EVIDENCE_SHA256=b7e598a873b8ddaff198680da6d7d2b50d6ccfbfcc6ed9a5e03141304bcf7d32
SANITIZED_OUTPUT_SHA256=157fd33f90b94edbb9a19b4592a93722442574b3e8dbc5c99ece5de734517ae3
OUTPUT_BEGIN
===== CONTEXT SOURCES DISCOVERY =====
TIME=2026-07-21T09:11:08+03:00
HOST=pve01.gram1.ru
===== AUTHORITATIVE FILES =====
----- FILE=/etc/pve/31_HOMELAB_REFERENCE.md -----
SIZE=164140 MTIME=2026-07-14 21:33:38.000000000 +0300 MODE=640 OWNER=root:www-data
SHA256=f6f7749fda946015116cdbbd754c7e7f20356b4a6cf4df330a1b0b8fdf59cbb2
LINES=2799
----- CONTENT BEGIN -----
HOMELAB REFERENCE
GENERATED=2026-06-29T21:45:29+03:00
AUDIT_DIR=/root/cluster-audit-20260629T201245
CORE_CLUSTER_CONFIG
keyboard: en-us
migration: secure,network=[PRIVATE_IP]/24
FINAL_CLASSIFICATION
MISSING_EXPECTED_PREFIX_COUNT=0
STRICT_POSSIBLE_SECRET_COUNT=0
STRICT_SECRET_SCAN_OK
HEALTH_WARN_ERROR_COUNT=14
NOTE exclude_from_reference_or_delete_later=05_edge_compose_safe.tgz
NOTE forum-prod VMID160 is not in homelab-nightly-all; it has separate local/cloud backup chain
NOTE edge-vm disk scsi1 backup=0 risk must be documented
NOTE Corosync remains on [PRIVATE_IP]/12/13; migration network is [PRIVATE_IP]/24 secure
NOTE edge health WARN/ERROR items should be documented as known current states
HEALTH_NOT_OK
HEALTH_NOT_OK item=adguard-rewrite-sync.txt line=STATUS=ERROR TS=20260629T152553Z TYPE=adguard-rewrite-sync REASON=URLError SECRET_PRINTED=REDACTED
HEALTH_NOT_OK item=backup-restore-coverage-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=backup-restore-dashboard.txt line=STATUS=WARN TS=2026-06-29T18:10:09Z ITEMS=80 NOT_OK=13
HEALTH_NOT_OK item=container-image-lifecycle-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=cron-job-monitoring-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=direct-heartbeat-pilot-readiness.txt line=STATUS=WARN
HEALTH_NOT_OK item=healthchecks-heartbeat-coverage.txt line=STATUS=WARN
HEALTH_NOT_OK item=healthchecks-job-coverage.txt line=STATUS=WARN
HEALTH_NOT_OK item=healthchecks-job-monitoring-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=homepage-service-catalog-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=ingress-dns-route-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=job-monitoring-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=observability-health-surface-slo.txt line=STATUS=WARN
HEALTH_NOT_OK item=security-vulnerability-slo.txt line=STATUS=WARN
EVIDENCE_FILES
00_cluster_overview.txt 3544 bytes
00_HOMELAB_ASSISTANT_ERROR_REGISTER.md 2566 bytes
01_pve01_audit.txt 28554 bytes
02_pve02_audit.txt 24786 bytes
03_pve03_audit.txt 16842 bytes
04_edge_vm_basic.txt 34506 bytes
05_edge_compose_redacted.txt 24228 bytes
05_edge_compose_safe.tgz.QUARANTINED.txt 354 bytes
05_edge_compose_tar_errors.txt 166 bytes
06_ACCESS_AND_ERROR_RULES.md 13742 bytes
06_edge_npmplus_routes_safe.txt 17350 bytes
07_edge_systemd_backup_audit.txt 20694 bytes
08_edge_scripts_redacted.txt 198622 bytes
09_forum_prod_basic.txt 14413 bytes
10_forum_codevipe_xenforo_audit.txt 3861 bytes
11_forum_backup_audit_redacted.txt 5952 bytes
12_pve02_codevipe_cloud_backup_audit.txt 6277 bytes
13_pve01_systemd_backup_audit.txt 60109 bytes
14_pve01_scripts_redacted.txt 246474 bytes
15_pve01_ct_110_audit.txt 5049 bytes
15_pve01_ct_112_audit.txt 4843 bytes
16_pve02_ct_111_audit.txt 4758 bytes
16_pve02_ct_113_audit.txt 4843 bytes
17_nextcloud_vm_audit.txt 13293 bytes
18_nextcloud_backup_restore_pve01_audit.txt 4269 bytes
19_pve02_host_automation_audit.txt 10642 bytes
19_pve03_host_automation_audit.txt 7632 bytes
20_AUDIT_INDEX_AND_SECRET_SCAN.txt 11935 bytes
21_proxmox_cluster_config_audit.txt 10855 bytes
22_internal_2_5g_network_inventory.txt 8036 bytes
23_cluster_internal_network_configured.txt 747 bytes
24_cluster_internal_network_speedtest.txt 54665 bytes
25_cluster_internal_network_migration_enabled.txt 1816 bytes
26_migration_network_pvesh_verify.txt 1210 bytes
27_migration_network_canonical_verify.txt 956 bytes
28_dns_configs_redacted.txt 15559 bytes
29_health_summary_all_nodes.txt 39051 bytes
30_FINAL_AUDIT_GAPS_WARNINGS_SECRET_CLASSIFICATION.txt 4366 bytes
31_HOMELAB_REFERENCE.md 1805 bytes
datacenter.cfg.before-canonical-migration-20260629T211046 63 bytes
datacenter.cfg.before-migration-network-20260629T210710 16 bytes
РУССКОЕ_РЕЗЮМЕ_КЛАСТЕРА
- Кластер: homelab, 3 узла, quorum OK.
- LAN: pve01=[PRIVATE_IP], pve02=[PRIVATE_IP], pve03=[PRIVATE_IP].
- Внутренняя сеть: pve01-int=[PRIVATE_IP], pve02-int=[PRIVATE_IP], pve03-int=[PRIVATE_IP].
- Миграции Proxmox: migration: secure,network=[PRIVATE_IP]/24.
- Corosync остаётся на [PRIVATE_IP]/12/13; не переносить без отдельного rollback-плана.
РИСКИ_И_ДЕЙСТВИЯ
- VM160 forum-prod не входит в ночной Proxmox backup.
- У VM130 edge-vm есть риск: дополнительный диск backup=0.
- Нужно ротировать ранее засвеченный Cloudflare token.
- Желательно ротировать XenForo SMTP пароль, который был выведен до редактирования.
- Проверить adguard-rewrite-sync: STATUS=ERROR, REASON=URLError.
- Проверить backup-restore-dashboard: STATUS=WARN, NOT_OK=13.
ДОСТУПЫ_КРАТКО
- Proxmox: ssh root@pve01, ssh root@pve02, ssh root@pve03.
- Edge VM: ssh debian@[PRIVATE_IP], использовать sudo, root-login не использовать.
- Nextcloud VM: ssh debian@[PRIVATE_IP].
- Forum-prod: сначала ssh root@pve02, затем ssh -i [SENSITIVE_PATH] root@[PRIVATE_IP].
НАГРУЗКИ_И_СЕРВИСЫ
- CT110 dns1 pve01 [PRIVATE_IP] AdGuard Home.
- CT111 dns2 pve02 [PRIVATE_IP] AdGuard Home.
- CT112 unbound1 pve01 [PRIVATE_IP] Unbound.
- CT113 unbound2 pve02 [PRIVATE_IP] Unbound.
- VM130 edge-vm pve03 [PRIVATE_IP] Docker ingress/app host.
- VM150 Nextcloud pve01 [PRIVATE_IP] Nextcloud AIO.
- VM160 forum-prod pve02 [PRIVATE_IP] CodeVipe/XenForo.
BACKUP_КРАТКО
- Ночной Proxmox backup включает VMID 110,111,112,113,130,150.
- VM160 forum-prod исключён из ночного Proxmox backup и держится на app-level local/cloud backup.
- Edge VM имеет vzdump/offhost/restore evidence; риск backup=0 по дополнительному диску остаётся.
- Nextcloud VM150 имеет Proxmox backup и restore-proof evidence.
- CodeVipe forum-prod имеет локальный backup каждые 6 часов и cloud backup с pve02 каждые 6 часов.
БЕЗОПАСНОСТЬ_КРАТКО
- Финальная строгая проверка audit-файлов: STRICT_POSSIBLE_SECRET_COUNT=0.
- Redacted DNS configs сохранены в 28_dns_configs_redacted.txt.
- Справочник 31 проверен: REFERENCE_ASSIGNMENT_SECRET_HITS=0.
## FINAL_CLOSURE_20260630_CRITICAL_TAILS
- VM160 backup: closed, nightly includes VM160, manual backup ZSTD_OK.
- VM130 backup: closed, scsi1 backup=1, manual backup ZSTD_OK, exact offhost ZSTD_OK, old local backup removed.
- AdGuard rewrite sync: closed, STATUS=OK, timer active/enabled.
- Cloudflare token: live NPMplus token rotated, API verify OK, old exposed token externally confirmed revoked.
- XenForo SMTP: password rotated, JSON valid, STARTTLS SMTP AUTH OK.
- Final audit: unredacted secret strict scan OK.
- Proof index: 85_FINAL_CLOSURE_PROOF.txt and 87_FINAL_CLOSURE_PROOF_WITH_EXTERNAL_CONFIRMATION.txt.
## FINAL_DASHBOARD_RUNTIME_OK_20260630
- Runtime dashboard: STATUS=OK, ITEMS=69, NOT_OK=0.
- systemd failed units: 0 loaded units listed.
- SLO/coverage WARN files: moved out of runtime dashboard into slo-coverage-backlog-index.txt.
- Backlog index: STATUS=OK, BACKLOG_WARN_COUNT=12, RUNTIME_BLOCKER=NO.
- Final proof: 103_FINAL_ALL_GREEN_DASHBOARD_PROOF.txt.
## ROUTER_NETCRAZE_ULTRA_NC1812_20260630
Источник: self-test Netcraze Ultra NC-1812 от 2026-06-30. Секреты Wi-Fi, пароли и токены в справочник не вносятся.
### Паспорт
- Model: Netcraze Ultra.
- Device description: Netcraze Ultra (NC-1812).
- Hostname: Netcraze-9202.
- Workgroup/domain: WORKGROUP.
- Timezone: Europe/Moscow.
- NTP: master.
- NDNS/caption: ndns-domain.
- Локальный домен/сертификат в логах: aleisaevn.netcraze.pro.
- Firmware/NDM из self-test: 5.01.C.0.0-1, built 2026-06-17.
- sharing-config version: 2.06.1.
- Components auto-update: disabled.
- Auto-update channel: draft.
- Auto-update schedule0: start 05:00, stop 06:00.
- EasyConfig: disabled.
- zram: enabled.
- IPv4 forwarding: enabled.
- IPv6 forwarding: enabled.
- conntrack max entries: 32768.
- TCP established timeout: 1200.
- TCP fin timeout: 30.
- TCP keepalive: 120.
### WAN и резервный интернет
- Main WAN: GigabitEthernet1, renamed ISP, description Ростел.
- WAN security-level: public.
- WAN addressing: DHCP.
- WAN MTU: 1500.
- WAN global priority: 700.
- WAN ping-check profile: default.
- Ping-check hosts from log/profile: ya.ru, vk.com, dzen.ru over TLS/443.
- WAN DHCP observed in router log: 95.84.154.183/23, gateway 95.84.154.1.
- ISP DNS observed in router log: 77.37.251.33, 77.37.255.30.
- Backup/mobile WAN: CdcEthernet0, description SIM.
- CdcEthernet0 USB device-id: 12d1 14dc.
- CdcEthernet0 security-level: public.
- CdcEthernet0 addressing: DHCP.
- CdcEthernet0 global priority: 350.
- CdcEthernet0 observed IP: [PRIVATE_IP]/24 via [PRIVATE_IP].
### LAN / VLAN / bridge
- Home bridge: Bridge0 renamed Home.
- Home description: Основная.
- Home security-level: private.
- Home IP: [PRIVATE_IP]/24.
- Home includes: GigabitEthernet0/Vlan1, AccessPoint, AccessPoint_5G, XGigabitEthernet0.
- Proxmox bridge: Bridge1.
- Proxmox bridge IP: [PRIVATE_IP]/24.
- Proxmox security-level: protected.
- Proxmox includes: GigabitEthernet0/Vlan50, XGigabitEthernet0/Vlan50.
- Port 1: GigabitEthernet0/0, access VLAN 50.
- Port 2: GigabitEthernet0/1, access VLAN 1 + trunk VLAN 50.
- Port 3: GigabitEthernet0/2, access VLAN 1 + trunk VLAN 50.
- Port 4: GigabitEthernet0/3, access VLAN 1 + trunk VLAN 50.
- Port 5: XGigabitEthernet0/0, XGigabitEthernet0/Vlan50 enabled.
### Wi-Fi
- SSID: N9202.
- 2.4 GHz: WifiMaster0, AccessPoint.
- 2.4 GHz compatibility: BGN+AX+BE.
- 2.4 GHz channel width: 40-below.
- 5 GHz: WifiMaster1, AccessPoint_5G.
- 5 GHz compatibility: AN+AC+AX+BE.
- 5 GHz channel width: 160.
- Auto channel rescan: 00:00 interval 1 hour.
- Encryption: WPA2 + WPA3.
- WMM: enabled.
- Beamforming: enabled.
- TWT: enabled.
- DL/UL MU-MIMO: enabled.
- DL/UL OFDMA: enabled.
- Spatial reuse: enabled.
- Band steering: disabled.
- Extra AP interfaces: present but down.
- MWS WLAN Home: bands 0 and 1, bind Home, SSID N9202, encryption wpa2+3.
### DHCP
- Main DHCP pool: _WEBADMIN.
- DHCP range: [PRIVATE_IP]-[PRIVATE_IP].
- Default router: [PRIVATE_IP].
- DHCP DNS: [PRIVATE_IP], [PRIVATE_IP].
- Lease: 25200 seconds.
- Bound interface: Home.
- Guest AP pool: _WEBADMIN_GUEST_AP enabled.
### Static DHCP reservations
- [PRIVATE_IP] -> 00:e0:70:62:98:fb comp01.
- [PRIVATE_IP] -> 60:3d:61:d4:57:ea Yandex-Ambi-Lamp.
- [PRIVATE_IP] -> cc:4b:73:58:9b:c0 Yandex.Station.
- [PRIVATE_IP] -> 38:8a:06:5b:eb:40 Мой смарт.
- [PRIVATE_IP] -> a0:9d:c1:84:7b:98 Приставка комната.
- [PRIVATE_IP] -> ac:ba:c0:51:d3:ca Алиса наша комната.
- [PRIVATE_IP] -> 54:f1:5f:f3:cd:14 Приставка кухня.
- [PRIVATE_IP] -> ac:ba:c0:50:71:94 Алиса кухня.
- [PRIVATE_IP] -> 24:4b:fe:d8:8d:2c Коля смарт ROG-Phone3.
- [PRIVATE_IP] -> 00:2b:70:9a:a3:cf Коля смарт S21-FE.
- [PRIVATE_IP] -> 00:08:22:34:26:fc Коля SPIDER-10 первый.
- [PRIVATE_IP] -> 6c:c7:ec:93:be:dc Света смарт.
- [PRIVATE_IP] -> 00:1e:8f:b0:d5:af Принтер Canon ч/б.
- [PRIVATE_IP] -> d8:43:ae:68:a3:a1 Коля комп.
- [PRIVATE_IP] -> 80:1f:12:69:24:e6 Сигнализация.
- [PRIVATE_IP] -> 00:2b:70:9a:9e:99 Мой смарт S21-FE.
- [PRIVATE_IP] -> 74:eb:80:2e:aa:4e Света планшет Galaxy Tab S4.
- [PRIVATE_IP] -> 84:47:09:6a:cc:ca PVE01.
- [PRIVATE_IP] -> 6c:1f:f7:28:72:4f PVE02.
- [PRIVATE_IP] -> 08:26:ae:3b:73:61 PVE03.
- [PRIVATE_IP] -> bc:24:11:30:d4:0f AdGuard 1.
- [PRIVATE_IP] -> bc:24:11:5b:7e:d8 AdGuard 2.
- [PRIVATE_IP] -> bc:24:11:f4:c5:d8 WireGuard.
- [PRIVATE_IP] -> bc:24:11:e1:f3:3c NPMplus / edge-vm.
- [PRIVATE_IP] -> bc:24:11:e1:9a:25 Nextcloud.
### NAT / port forwarding
- ISP tcp/80 -> bc:24:11:e1:f3:3c, NPMplus.
- ISP tcp/443 -> bc:24:11:e1:f3:3c, NPMplus.
- ISP tcpudp/3478 -> bc:24:11:e1:9a:25, Nextcloud Talk TURN.
- Home tcp/51820 -> bc:24:11:f4:c5:d8, WireGuard vpn1.
### ACL / firewall
- isolate-private enabled.
- _WEBADMIN_Bridge0 permits Home-to-Proxmox access for DNS1/DNS2, AdGuard UI, NPMplus HTTP/HTTPS/UI, Nextcloud AIO/Talk TURN, Proxmox SSH/8006 and ICMP from admin PC.
- _WEBADMIN_GigabitEthernet1 permits admin PC [PRIVATE_IP] outbound TCP 80/443.
- Hotspot policy: Home permit, Bridge1 permit; known hosts explicitly permitted.
### Router management
- HTTP port: 5080.
- HTTPS port: 5083.
- HTTP/HTTPS security-level: private.
- SSH port: 2222.
- SSH security-level: private.
- Telnet port configured: 2323, security-level private; router log shows Telnet disabled later on 2026-06-26.
- Lockout policy for HTTP/Telnet/SSH: 5 15 3.
- Cloud control2 security-level: private.
- Admin tags: cli, http, cifs, printers, opt.
- SFTP denied for admin in log; SSH CLI works.
### Router services
- service dhcp enabled.
- service dns-proxy enabled.
- service igmp-proxy enabled.
- service http enabled.
- service cifs enabled.
- service ssh enabled.
- service ntp enabled.
- DNS proxy rebind protection: auto.
- mDNS reflector: disabled.
- UPnP LAN: Home.
- DLNA interface: Home.
- CIFS share: NTFS-edgtobxf, volume 6F98787822C01361:, automount, permissive.
- USB storage seen in log: Huawei modem/storage, sdb1 swap around 2G, sdb2 NTFS volume NTFS-edgtobxf.
### Router automation warning
- Netcraze SSH CLI is not a normal POSIX shell.
- Logs contain failed commands: while, unset, follow.
- Automation must use router CLI syntax, not bash syntax.
## EXTERNAL_SERVICES_UPS_NETBIRD_MAIL_SCRIPTS_20260630
### UPS / NUT
- UPS hardware discovered on pve01 USB: Cyber Power System CP1500 AVR UPS, USB ID 0764:0501.
- pve01 role: NUT server + monitor.
- pve01 enabled NUT units: nut-driver-enumerator, nut-monitor, nut-server, nut-driver.target, nut.target.
- pve02 role: NUT monitor/client.
- pve03 role: NUT monitor/client.
- edge-vm: no UPS/NUT/APCUPSD integration discovered.
- UPS detailed proof: 107_UPS_DISCOVERY_AND_SETTINGS.txt and 109_UPS_NUT_DETAILED_STATUS_AND_CONFIGS.txt.
- Secrets from /etc/nut configs must remain redacted in audit output.
### NetBird
- NetBird service is active on pve01, pve02, pve03 and edge-vm.
- NetBird version observed: daemon 0.73.2, CLI 0.73.2.
- pve01: FQDN pve01.netbird.selfhosted, IPv4 100.100.131.41/16.
- pve02: FQDN pve02.netbird.selfhosted, IPv4 100.100.7.2/16.
- pve03: FQDN pve03.netbird.selfhosted, IPv4 100.100.34.141/16.
- edge-vm: FQDN edge-vm.netbird.selfhosted, IPv4 100.100.60.182/16.
- Interface: wt0, type Kernel.
- WireGuard port: 51820.
- Management/Signal: Connected.
- Relays: 2/2 available.
- SSH Server through NetBird: Disabled.
- Observed peers count on listed hosts: 6/9 Connected.
- Inventory proof: 108_EXTERNAL_SERVICES_NETBIRD_MAIL_SCRIPTS_INVENTORY.txt.
### Forum mail / SMTP
- forum-prod host: forum-prod.gram1.ru, VM IP [PRIVATE_IP].
- Local MTA services were not discovered by the first inventory command; forum mail is treated as XenForo external SMTP, not local Postfix/Dovecot.
- XenForo SMTP rotation proof: 80_xenforo_smtp_rotation_closed.txt.
- SMTP auth proof: 79_forum_xenforo_smtp_auth_openssl_no_crlf_no_secret.txt.
- Safe current SMTP inventory: 110_XENFORO_MAIL_EXTERNAL_SMTP_SAFE_INVENTORY.txt.
- SMTP password must never be printed; store only host, port, SSL mode, auth flag, username length, password length and AUTH OK proof.
### Scripts / automations
- Full script/unit inventory proof: 108_EXTERNAL_SERVICES_NETBIRD_MAIL_SCRIPTS_INVENTORY.txt.
- edge-vm owns most application health, dashboard, ingress, backup, NetBox, NPMplus, certificate, Trivy and vulnerability jobs.
- pve01 owns many backup/offhost/restore/health/security/NetBird VPS/rclone/sops/scrutiny jobs.
- pve02 owns CodeVipe cloud backup, Vaultwarden isolated restore drill, smartctl textfile and staging/rclone helpers.
- pve03 owns smartctl textfile, cluster internal IP, NetBird and staging/rclone helpers.
- All scripts must be treated as managed operational surface; every future edit needs proof file in the active cluster-audit directory.
## UPS_NUT_DETAILED_CONFIG_20260630
- USB discovery on pve01: Cyber Power System, Inc. CP1500 AVR UPS, USB ID 0764:0501.
- NUT logical UPS name: cyberpower.
- NUT-reported device model: BR1000ELCD.
- NUT manufacturer: CPS.
- Driver: usbhid-ups.
- NUT driver version: 2.8.1.
- CyberPower HID data version: 0.8.
- NUT USB vendorid/productid: 0764/0501.
- NUT server node: pve01.
- NUT server mode: MODE=netserver.
- NUT server listeners: 127.0.0.1:3493 and [PRIVATE_IP]:3493.
- NUT clients: pve02 and pve03 in MODE=netclient.
- pve01 monitor role: MONITOR cyberpower@localhost master.
- pve02 monitor role: MONITOR cyberpower@[PRIVATE_IP] slave.
- pve03 monitor role: MONITOR cyberpower@[PRIVATE_IP] slave.
- Current UPS status at inventory time: OL.
- Battery charge: 100%.
- Battery warning threshold: 20%.
- Battery low threshold: 10%.
- Runtime estimate: 2544 seconds.
- Runtime low threshold: 300 seconds.
- Battery type: PbAcid.
- Battery voltage: 12.9V nominal 12V.
- Input voltage: 221.0V nominal 230V.
- Output voltage: 221.0V.
- UPS load: 11%.
- Nominal real power: 600W.
- Beeper: disabled.
- Shutdown delay: 20 seconds.
- Start delay: 30 seconds.
- Shutdown command on monitored nodes: /sbin/shutdown -h +0.
- NUT timing: POLLFREQ=5, POLLFREQALERT=5, HOSTSYNC=15, DEADTIME=30, FINALDELAY=5.
- Powerdown flag: /etc/killpower.
- Secrets in /etc/nut/upsd.users and upsmon.conf are intentionally redacted in proofs.
- Proof file: 109_UPS_NUT_DETAILED_STATUS_AND_CONFIGS.txt.
## XENFORO_EXTERNAL_SMTP_CURRENT_20260630
- Host: forum-prod.gram1.ru, VM IP [PRIVATE_IP].
- Local MTA units: none discovered in inventory output.
- Mail mode: XenForo external SMTP, not local Postfix/Dovecot.
- SMTP_HOST=mail.pvepro.ru
- SMTP_PORT=587
- SMTP_SSL=false
- SMTP_AUTH=login
- USERNAME_LEN=17
- PASSWORD_LEN=30
- SECRET_PRINTED=NO
- SMTP AUTH proof: 79_forum_xenforo_smtp_auth_openssl_no_crlf_no_secret.txt.
- Rotation closure proof: 80_xenforo_smtp_rotation_closed.txt.
## TIMERS_AND_SCRIPT_HASHES_REFERENCE_20260630
- Full timers and script SHA256 inventory: 114_TIMERS_AND_SCRIPT_HASHES_INVENTORY.txt.
- edge-vm timers cover runtime dashboard, Paperless guard, external canary, health metrics, restore drill index, AdGuard rewrite sync, NPMplus cert expiry, NetBox backup/sync, retention, cluster daily status, vulnerability and Trivy scans, ingress hardening and NPMplus admin bind.
- pve01 timers cover VPN/NetBird health, health metrics, smartctl, disk space, MkDocs refresh, VPS identity audit, storage capacity, quality gate, evidence catalog, backup freshness, docker health, Filebrowser backup/offhost/restore, NPMplus/Kuma backup, NetBird VPS backup/offhost, Authentik/Gitea/Vaultwarden backup, SOPS secret coverage, mail cloud upload/restore, Immich/Memos/Paperless backup/offhost/restore, auto backup, edge-vm vzdump, secret sanity.
- pve02 timers cover smartctl, CodeVipe cloud backup, dpkg-db backup and Vaultwarden isolated restore drill.
- pve03 timers cover smartctl and dpkg-db backup.
- Script hashes were captured for /usr/local/sbin and /usr/local/bin on edge-vm, pve01, pve02 and pve03.
- Future script edits must create a new proof file and refresh 114_TIMERS_AND_SCRIPT_HASHES_INVENTORY.txt.
## PROXMOX_CLUSTER_NODE_STORAGE_REFERENCE_20260630
### Cluster
- Cluster name: homelab.
- Nodes: 3.
- Quorum: OK / Quorate Yes.
- Expected votes: 3.
- Quorum threshold: 2.
- Transport: knet.
- Secure auth: on.
- Corosync membership: [PRIVATE_IP] pve01, [PRIVATE_IP] pve02, [PRIVATE_IP] pve03.
- Proxmox VE: pve-manager 9.2.3 on all three nodes.
- Debian: 13 / trixie on all three nodes.
- Kernel: 7.0.12-1-pve on all three nodes.
- Datacenter migration config: secure, network=[PRIVATE_IP]/24.
- Internal migration/cluster helper IPs: pve01 [PRIVATE_IP], pve02 [PRIVATE_IP], pve03 [PRIVATE_IP].
### Storage policy
- Storage local: dir /var/lib/vz, content iso/vztmpl/backup/import.
- Storage local-lvm: lvmthin pool data, content rootdir/images.
- pve01 local usage at inventory: 32.56%, local-lvm 17.50%.
- pve02 local usage at inventory: 12.86%, local-lvm 11.65%.
- pve03 local usage at inventory: 35.79%, local-lvm 64.84%.
- pve01 staging LV: /mnt/staging, 300G.
- pve02 staging LV: /mnt/staging, 150G.
- pve03 staging LV: /mnt/staging, 200G.
- pve03 local-lvm is the most constrained active VM storage because VM130 has 96G OS disk plus 150G media disk.
### Nightly Proxmox backup job
- Job: homelab-nightly-all.
- Schedule: 03:30.
- Mode: snapshot.
- Compression: zstd.
- Storage: local.
- Enabled: yes.
- Mail notification: failure.
- Included VMIDs: 110,111,112,113,130,150,160.
- Retention: keep-daily=7, keep-weekly=4, keep-monthly=3.
### Node pve01
- FQDN: pve01.gram1.ru.
- LAN IP: [PRIVATE_IP]/24.
- Internal migration IP: [PRIVATE_IP]/24.
- NetBird IP: 100.100.131.41/16.
- Bridge: vmbr0 over nic0.
- CPU: Intel Core i9-12950HX, 24 logical CPUs.
- RAM: 31Gi.
- Disk: Lexar SSD NQ7A1 1TB.
- Main active workloads: CT110 dns1, CT112 unbound1, VM150 nextcloud.
### Node pve02
- FQDN: pve02.gram1.ru.
- LAN IP: [PRIVATE_IP]/24.
- Internal migration IP: [PRIVATE_IP]/24.
- NetBird IP: 100.100.7.2/16.
- Bridge: vmbr0 over nic2.
- CPU: Intel N100, 4 logical CPUs.
- RAM: 15Gi.
- Disk: SK800-1TB.
- Main active workloads: CT111 dns2, CT113 unbound2, VM160 forum-prod.
### Node pve03
- FQDN: pve03.gram1.ru.
- LAN IP: [PRIVATE_IP]/24.
- Internal migration IP: [PRIVATE_IP]/24.
- NetBird IP: 100.100.34.141/16.
- Bridge: vmbr0 over nic1.
- CPU: Intel Core i7-4900MQ, 8 logical CPUs.
- RAM: 31Gi.
- Disk: Samsung SSD 870 EVO 500GB.
- Main active workload: VM130 edge-vm.
### Proof
- Full raw inventory: 120_PROXMOX_CLUSTER_NODE_STORAGE_NETWORK_INVENTORY.txt.
## PROXMOX_VM_CT_INVENTORY_REFERENCE_20260630
### CT110 dns1
- Type: LXC.
- Node: pve01.
- Hostname/name: dns1.
- IP: [PRIVATE_IP]/24.
- MAC: BC:24:11:F9:3C:E1.
- Role: AdGuard Home DNS primary.
- Resources: 1 core, 512MiB RAM, 256MiB swap, 8G rootfs.
- Network: vmbr0, gateway [PRIVATE_IP].
- Nameserver: [PRIVATE_IP].
- Onboot: yes.
- Startup order: 30.
- Unprivileged: yes.
- Backup job: included in homelab-nightly-all.
### CT111 dns2
- Type: LXC.
- Node: pve02.
- Hostname/name: dns2.
- IP: [PRIVATE_IP]/24.
- MAC: BC:24:11:CF:3F:66.
- Role: AdGuard Home DNS secondary.
- Resources: 1 core, 512MiB RAM, 256MiB swap, 8G rootfs.
- Network: vmbr0, gateway [PRIVATE_IP].
- Nameserver: [PRIVATE_IP].
- Onboot: yes.
- Startup order: 30.
- Unprivileged: yes.
- Backup job: included in homelab-nightly-all.
### CT112 unbound1
- Type: LXC.
- Node: pve01.
- Hostname/name: unbound1.
- IP: [PRIVATE_IP]/24.
- MAC: BC:24:11:22:A6:0D.
- Role: Unbound recursive resolver primary.
- Resources: 1 core, 512MiB RAM, 256MiB swap, 8G rootfs.
- Network: vmbr0, gateway [PRIVATE_IP].
- Nameserver: [PRIVATE_IP].
- Onboot: yes.
- Startup order: 20.
- Unprivileged: yes.
- Backup job: included in homelab-nightly-all.
### CT113 unbound2
- Type: LXC.
- Node: pve02.
- Hostname/name: unbound2.
- IP: [PRIVATE_IP]/24.
- MAC: BC:24:11:9E:AF:BE.
- Role: Unbound recursive resolver secondary.
- Resources: 1 core, 512MiB RAM, 256MiB swap, 8G rootfs.
- Network: vmbr0, gateway [PRIVATE_IP].
- Nameserver: [PRIVATE_IP].
- Onboot: yes.
- Startup order: 20.
- Unprivileged: yes.
- Backup job: included in homelab-nightly-all.
### VM130 edge-vm
- Type: QEMU VM.
- Node: pve03.
- Name: edge-vm.
- IP: [PRIVATE_IP]/24.
- MAC: BC:24:11:E1:F3:3C.
- User: debian.
- Role: edge application host / reverse proxy / monitoring / backup automation host.
- Resources: 4 cores, 12GiB RAM.
- Disks: scsi0 96G OS, scsi1 150G media/data.
- scsi1 backup flag: backup=1.
- Network: vmbr0, gateway [PRIVATE_IP].
- Nameservers: [PRIVATE_IP], [PRIVATE_IP].
- QEMU guest agent: enabled.
- Onboot: yes.
- Startup order: 40.
- Backup job: included in homelab-nightly-all.
- Important note: VM130 has exact offhost backup proof after scsi1 backup=1.
### VM150 nextcloud
- Type: QEMU VM.
- Node: pve01.
- Name: nextcloud.
- IP: [PRIVATE_IP]/24.
- MAC: BC:24:11:E1:9A:25.
- User: debian.
- Role: Nextcloud AIO.
- Resources: 4 cores, 8GiB RAM.
- Disk: scsi0 64G.
- Network: vmbr0, gateway [PRIVATE_IP].
- Nameservers: [PRIVATE_IP], [PRIVATE_IP].
- QEMU guest agent: enabled.
- Onboot: yes.
- Startup order: 50.
- Backup job: included in homelab-nightly-all.
### VM160 forum-prod
- Type: QEMU VM.
- Node: pve02.
- Name: forum-prod.
- IP: [PRIVATE_IP]/24.
- MAC: BC:24:11:B6:45:ED.
- User: ops.
- Role: CodeVipe / XenForo production forum.
- Resources: 2 cores, 4GiB RAM.
- Disk: scsi0 80G.
- Network: vmbr0, gateway [PRIVATE_IP].
- Nameserver in VM config: 1.1.1.1.
- QEMU guest agent: enabled.
- Onboot: yes.
- Startup order: 30.
- Backup job: included in homelab-nightly-all.
- Important note: VM160 manual backup proof exists and VM160 is included in nightly job.
### Proof
- Redacted config inventory: 121_PROXMOX_VM_CT_CONFIG_INVENTORY_REDACTED.txt.
## DNS_INGRESS_CERTIFICATES_REFERENCE_20260630
### DNS chain
- Home DHCP gives clients DNS servers: [PRIVATE_IP] and [PRIVATE_IP].
- dns1: CT110, AdGuard Home, IP [PRIVATE_IP].
- dns2: CT111, AdGuard Home, IP [PRIVATE_IP].
- unbound1: CT112, IP [PRIVATE_IP], listens on [PRIVATE_IP]:5335 and 127.0.0.1:5335.
- unbound2: CT113, IP [PRIVATE_IP], listens on [PRIVATE_IP]:5335 and 127.0.0.1:5335.
- dns1 upstream: [PRIVATE_IP]:5335.
- dns2 upstream: [PRIVATE_IP]:5335.
- AdGuard bootstrap DNS: 1.1.1.1 and 9.9.9.9.
- AdGuard filtering_enabled=true, rewrites_enabled=true, protection_enabled=true.
- AdGuard ratelimit=20.
- Unbound access model: localhost allowed, matching AdGuard IP allowed, rest of [PRIVATE_IP]/24 refused.
- Unbound do-ip6: no.
- Unbound private-address includes [PRIVATE_IP]/16, [PRIVATE_IP]/12 and [PRIVATE_IP]/8.
### AdGuard rewrites
- turn.gram1.ru -> [PRIVATE_IP].
- git.gram1.ru -> [PRIVATE_IP].
- dozzle.gram1.ru -> [PRIVATE_IP].
- paper.gram1.ru -> [PRIVATE_IP].
- memos.gram1.ru -> [PRIVATE_IP].
- photos.gram1.ru -> [PRIVATE_IP].
- auth.gram1.ru -> [PRIVATE_IP].
- backup.gram1.ru -> [PRIVATE_IP].
- Rewrite sync health proof: adguard-rewrite-sync STATUS=OK, ORIGIN_COUNT=8, REPLICA_COUNT=8, CHANGED=0.
- Proof files: 127_ADGUARD_UNBOUND_DNS_SAFE_INVENTORY.txt and 130_ADGUARD_UPSTREAM_REWRITES_UNBOUND_RULES_SAFE.txt.
### Router ingress
- Public WAN router forwards TCP/80 and TCP/443 to NPMplus on edge-vm, [PRIVATE_IP].
- Router forwards TCP/UDP 3478 to Nextcloud Talk TURN on [PRIVATE_IP].
- Router forwards Home TCP/51820 to WireGuard host [PRIVATE_IP].
- NPMplus admin listener is bound to localhost on edge-vm, 127.0.0.1:81; public disabled legacy host npm.gram1.ru exists but enabled=0.
### NPMplus runtime
- Host: edge-vm, [PRIVATE_IP].
- Container: npmplus, image zoeyvid/npmplus:2026-06-17-b1, healthy at inventory time.
- Socket proxy: lscr.io/linuxserver/socket-proxy:latest, 127.0.0.1:2375.
- Database: /opt/npmplus/npmplus/database.sqlite.
- DB integrity: ok.
- Public listen ports on edge-vm: 0.0.0.0:80 and 0.0.0.0:443 by nginx/NPMplus.
- NPMplus admin: 127.0.0.1:81.
- Secret rule: Cloudflare DNS API token exists only inside NPMplus certificate metadata and must never be printed.
### Public NPMplus proxy hosts
- nc.gram1.ru -> http://[PRIVATE_IP]:11000, cert=17, ssl_forced=1, enabled=1.
- uptime.gram1.ru -> http://127.0.0.1:3001, cert=35, ssl_forced=1, enabled=1.
- gotify.gram1.ru -> http://127.0.0.1:8082, cert=35, ssl_forced=1, enabled=1.
- vault.gram1.ru -> http://127.0.0.1:8083, cert=21, ssl_forced=1, enabled=1.
- dockge.gram1.ru -> http://127.0.0.1:5001, cert=26, ssl_forced=1, enabled=1.
- home.gram1.ru -> http://127.0.0.1:3000, cert=27, ssl_forced=1, enabled=1.
- git.gram1.ru -> http://127.0.0.1:3002, cert=29, ssl_forced=1, enabled=1.
- dozzle.gram1.ru -> http://127.0.0.1:9999, cert=30, ssl_forced=1, enabled=1.
- paper.gram1.ru -> http://127.0.0.1:8010, cert=30, ssl_forced=1, enabled=1.
- memos.gram1.ru -> http://127.0.0.1:5230, cert=30, ssl_forced=1, enabled=1.
- photos.gram1.ru -> http://127.0.0.1:2283, cert=30, ssl_forced=1, enabled=1.
- auth.gram1.ru -> http://127.0.0.1:9000, cert=30, ssl_forced=1, enabled=1.
- backup.gram1.ru -> http://[PRIVATE_IP]:9101, cert=31, ssl_forced=1, enabled=1.
### VPN NPMplus proxy hosts
- kuma.vpn.gram1.ru -> http://127.0.0.1:3001, cert=32.
- homepage.vpn.gram1.ru -> http://127.0.0.1:3000, cert=32.
- gitea.vpn.gram1.ru -> http://127.0.0.1:3002, cert=32.
- filebrowser.vpn.gram1.ru -> http://127.0.0.1:8085, cert=32.
- auth.vpn.gram1.ru -> http://127.0.0.1:9000, cert=32.
- actual-budget.vpn.gram1.ru -> http://127.0.0.1:5006, cert=32.
- grafana.vpn.gram1.ru -> http://127.0.0.1:3003, cert=32.
- homebox.vpn.gram1.ru -> http://127.0.0.1:7745, cert=32.
- mealie.vpn.gram1.ru -> http://127.0.0.1:9001, cert=32.
- n8n.vpn.gram1.ru -> http://127.0.0.1:5678, cert=32.
- netbox.vpn.gram1.ru -> http://127.0.0.1:8000, cert=32.
- node-red.vpn.gram1.ru -> http://127.0.0.1:1880, cert=32.
- beszel.vpn.gram1.ru -> http://127.0.0.1:8090, cert=32.
- it-tools.vpn.gram1.ru -> http://127.0.0.1:8084, cert=32.
- karakeep.vpn.gram1.ru -> http://127.0.0.1:3030, cert=32.
- linkding.vpn.gram1.ru -> http://127.0.0.1:9091, cert=32.
- minio.vpn.gram1.ru -> http://127.0.0.1:9003, cert=32.
- ntfy.vpn.gram1.ru -> http://127.0.0.1:8055, cert=32.
- searxng.vpn.gram1.ru -> http://127.0.0.1:8888, cert=32.
- syncthing.vpn.gram1.ru -> http://127.0.0.1:8384, cert=32.
- alertmanager.vpn.gram1.ru -> http://127.0.0.1:9093, cert=32.
- prometheus.vpn.gram1.ru -> http://127.0.0.1:9090, cert=32.
- homeassistant.vpn.gram1.ru -> http://127.0.0.1:8123, cert=32.
- npmplus.vpn.gram1.ru -> https://[PRIVATE_IP]:81, cert=32.
- proxmox.vpn.gram1.ru -> https://[PRIVATE_IP]:8006, cert=32.
- dns1.vpn.gram1.ru -> http://[PRIVATE_IP]:8080, cert=32.
- dns2.vpn.gram1.ru -> http://[PRIVATE_IP]:8080, cert=32.
- checks.vpn.gram1.ru -> http://127.0.0.1:8015, cert=32.
- vikunja.vpn.gram1.ru -> http://127.0.0.1:8016, cert=32.
- bookstack.vpn.gram1.ru -> http://127.0.0.1:8017, cert=32.
- pdf.vpn.gram1.ru -> http://127.0.0.1:8018, cert=32.
- jellyfin.vpn.gram1.ru -> http://127.0.0.1:8019, cert=32.
- audiobooks.vpn.gram1.ru -> http://127.0.0.1:8020, cert=32.
- books.vpn.gram1.ru -> http://127.0.0.1:8021, cert=32.
### NPMplus certificates
- cert=17: nc.gram1.ru, expires 2026-09-12 21:58:35.
- cert=21: vault.gram1.ru, expires 2026-09-13 13:43:23.
- cert=26: dockge.gram1.ru, expires 2026-09-13 14:51:59.
- cert=27: home.gram1.ru, expires 2026-09-13 16:43:44.
- cert=29: git.gram1.ru, expires 2026-09-13 17:36:55.
- cert=30: dozzle.gram1.ru, paper.gram1.ru, memos.gram1.ru, photos.gram1.ru, auth.gram1.ru, expires 2026-09-13 23:12:00.
- cert=31: backup.gram1.ru, expires 2026-09-14 05:04:53.
- cert=32: *.vpn.gram1.ru and vpn.gram1.ru, expires 2026-09-15 23:18:29.
- cert=35: uptime.gram1.ru and gotify.gram1.ru, expires 2026-09-19 21:21:59.
- Deleted/old cert rows observed: cert=1 nc.gram1.ru, cert=18 vault.gram1.ru.
### Proof
- NPMplus schema/listen inventory: 126_DNS_INGRESS_CERT_NPMPLUS_SAFE_INVENTORY.txt.
- NPMplus exact proxy/cert rows: 129_NPMPLUS_PROXY_CERT_STREAM_ROWS_SAFE.txt.
- AdGuard/Unbound inventory: 127_ADGUARD_UNBOUND_DNS_SAFE_INVENTORY.txt.
- AdGuard upstream/rewrites and Unbound rules: 130_ADGUARD_UPSTREAM_REWRITES_UNBOUND_RULES_SAFE.txt.
## EDGE_DOCKER_STACKS_REFERENCE_20260630
### Runtime
- Host: edge-vm, IP [PRIVATE_IP].
- Docker Server version: 29.6.0.
- Docker Compose version: v5.1.4.
- Primary stack root: /opt/stacks.
- Additional compose roots: /opt/npmplus-compose, /opt/gotify-compose, /opt/uptime-kuma-compose, /opt/vaultwarden-compose, /opt/dockge-compose.
- Public ingress terminates through NPMplus on ports 80/443.
- Most app containers expose only 127.0.0.1 ports and are published through NPMplus.
- NPMplus uses host networking.
- socket-proxy exposes Docker API read-oriented surface on 127.0.0.1:2375 and socket-proxy_net.
- Secret values are not stored in the reference. Only .env/secret file paths are inventoried.
### Compose projects observed
- actual-budget.
- audiobookshelf.
- authentik.
- beszel.
- blackbox-exporter.
- bookstack.
- cadvisor.
- calibre-web.
- crowdsec.
- diun.
- dockge-compose.
- dozzle.
- filebrowser.
- gitea.
- gotify-compose.
- healthchecks.
- homeassistant.
- homebox.
- homepage.
- immich.
- it-tools.
- jellyfin.
- karakeep.
- kopia.
- linkding.
- loki-alloy.
- mealie.
- memos.
- minio.
- n8n.
- netbox.
- node-red.
- npmplus.
- ntfy.
- observability-lite.
- paperless.
- searxng.
- socket-proxy.
- stirling-pdf.
- syncthing.
- uptime-kuma-compose.
- vaultwarden-compose.
- vikunja.
### Critical app groups
- Ingress/security: npmplus, socket-proxy, crowdsec.
- Identity/secrets: authentik, vaultwarden.
- Monitoring/alerting: uptime-kuma, gotify, healthchecks, observability-lite, loki-alloy, blackbox-exporter, cadvisor, node-exporter, dozzle, diun.
- Data apps: immich, paperless, memos, gitea, netbox, actual-budget, homebox, mealie, n8n, node-red, vikunja, bookstack.
- Files/media: nextcloud external VM, immich, filebrowser, syncthing, jellyfin, audiobookshelf, calibre-web, stirling-pdf, minio.
- Admin/dashboard: homepage, dockge, netbox, dozzle, NPMplus VPN admin.
### Notable local ports
- Homepage: 127.0.0.1:3000.
- Uptime Kuma: 127.0.0.1:3001.
- Gitea: 127.0.0.1:3002.
- Grafana: 127.0.0.1:3003.
- Actual Budget: 127.0.0.1:5006.
- n8n: 127.0.0.1:5678.
- Paperless: 127.0.0.1:8010.
- Healthchecks: 127.0.0.1:8015.
- Vikunja: 127.0.0.1:8016.
- BookStack: 127.0.0.1:8017.
- Stirling PDF: 127.0.0.1:8018.
- Jellyfin: 127.0.0.1:8019.
- Audiobookshelf: 127.0.0.1:8020.
- Calibre-Web: 127.0.0.1:8021.
- ntfy: 127.0.0.1:8055.
- Gotify: 127.0.0.1:8082.
- Vaultwarden: 127.0.0.1:8083.
- IT-Tools: 127.0.0.1:8084.
- Filebrowser: 127.0.0.1:8085.
- Beszel: 127.0.0.1:8090.
- Prometheus: 127.0.0.1:9090.
- Linkding: 127.0.0.1:9091.
- Alertmanager: 127.0.0.1:9093.
- Node exporter: 127.0.0.1:9100.
- Blackbox exporter: 127.0.0.1:9115.
- Syncthing UI: 127.0.0.1:8384.
- Loki: 127.0.0.1:3100.
- Alloy UI: 127.0.0.1:12345.
- Home Assistant: host network, 0.0.0.0:8123.
- NPMplus admin: 127.0.0.1:81.
- NPMplus public ingress: 0.0.0.0:80 and 0.0.0.0:443.
### Secret/env inventory policy
- Full values from .env, secrets.yaml, secretkey.txt and token files must never be copied into reference.
- Reference may list paths only.
- Full path inventory is in 133_EDGE_DOCKER_STACKS_SAFE_INVENTORY.txt.
- Normalized runtime map is in 134_EDGE_DOCKER_CONTAINER_NORMALIZED_MAP.txt.
### Proof
- Raw Docker/stacks inventory: 133_EDGE_DOCKER_STACKS_SAFE_INVENTORY.txt.
- Normalized container map: 134_EDGE_DOCKER_CONTAINER_NORMALIZED_MAP.txt.
## BACKUP_RESTORE_OFFHOST_CATALOG_REFERENCE_20260630
### Global backup model
- Primary Proxmox backup job: homelab-nightly-all.
- Schedule: daily 03:30.
- Mode: snapshot.
- Compression: zstd.
- Storage: local.
- Retention: keep-daily=7, keep-weekly=4, keep-monthly=3.
- Included VMIDs: 110,111,112,113,130,150,160.
- Backup health dashboard: STATUS=OK, ITEMS=69, NOT_OK=0 at 2026-06-30T10:10:06Z.
- Restore drill index: STATUS=OK, RESTORE_PROOFS=16/16.
- SLO coverage is tracked separately from runtime dashboard. It had WARN backlog in backup-restore-coverage-slo with APP_TOTAL=43, GREEN=3, YELLOW=21, RED=19, MISSING_BACKUP_SIGNAL=31, MISSING_RESTORE_SIGNAL=28, MISSING_TIMER=31.
- Runtime OK means operational backup/restore checks are green; it does not mean every P1/P2 application has full backup+restore automation coverage.
### Proxmox vzdump catalog
- CT110 dns1: included in homelab-nightly-all, local backup on pve01.
- CT111 dns2: included in homelab-nightly-all, local backup on pve02.
- CT112 unbound1: included in homelab-nightly-all, local backup on pve01.
- CT113 unbound2: included in homelab-nightly-all, local backup on pve02.
- VM130 edge-vm: included in homelab-nightly-all, local backup on pve03.
- VM150 nextcloud: included in homelab-nightly-all, local backup on pve01.
- VM160 forum-prod: included in homelab-nightly-all, local backup on pve02.
- VM130 also has dedicated edge-vm-vzdump backup/offhost/restore health proofs.
- VM160 has manual backup proof and nightly job inclusion proof.
- VM130 scsi1 backup flag is enabled after correction: scsi1 backup=1.
### Edge VM / VM130 full-image protection
- edge-vm-vzdump-backup: STATUS=OK, archive size about 28.2G, SHA256 recorded.
- edge-vm-vzdump-offhost: STATUS=OK, destination pve02 /mnt/staging/offhost/edge-vm-vzdump-from-pve03.
- edge-vm-vzdump-restore: STATUS=OK, zstd and vma verification OK.
- mail-cloud-edge-vm: STATUS=OK, recurring chunked upload, 53 parts, download verification enabled.
- Retention for edge-vm cloud upload: RETENTION_KEEP=4.
- Edge restore proof offhost: STATUS=OK, ITEMS=14, OK=14.
### Mail/cloud critical backups
- mail-cloud-critical: STATUS=OK, recurring P0 critical baseline, remote OK 2/2.
- Latest observed critical bundle: p0-critical-baseline-20260630T034644Z.tar.gz.
- Critical bundle size at inventory: 1087208837 bytes.
- Critical retention: RETENTION_KEEP=14.
- mail-cloud-restore-drill: STATUS=OK, CRITICAL_OK=2, EDGE_OK=1.
- vaultwarden isolated restore drill on pve02: STATUS=OK, uses cloud critical baseline plus pve02 offhost component and vaultwarden archive.
### NPMplus / Kuma / ingress config backups
- npmplus-kuma-config-backup: STATUS=OK.
- Archive: /mnt/staging/npmplus-kuma-config-backups/snapshots/npmplus-kuma-config-*.tar.gz.
- NPMplus DB integrity: OK.
- Kuma DB integrity: OK.
- NPM proxy count in health proof: 47.
- Required VPN routes in health proof: 18.
- Wildcard VPN cert check: NPM_CERT32_WILDCARD=1.
- npmplus-kuma-config-offhost: STATUS=OK to pve02.
- npmplus-kuma-config-restore: STATUS=OK with DB integrity checks.
- npmplus restore proof also exists from app backup quality checks.
### DNS / AdGuard / Unbound protection
- AdGuard rewrite sync: STATUS=OK, ORIGIN_COUNT=8, REPLICA_COUNT=8, CHANGED=0.
- dns1/dns2/unbound1/unbound2 are protected by Proxmox vzdump.
- AdGuard rewrite sync plan stored under /var/lib/homelab-private/adguard-rewrite-sync.
- Router self-test config is stored as uploaded/generated reference evidence, not yet automated as recurring router backup.
### Auth / secrets / identity apps
- Vaultwarden backup: STATUS=OK, archive under /mnt/staging/vaultwarden-backups/snapshots.
- Vaultwarden offhost: STATUS=OK to pve02.
- Vaultwarden restore: STATUS=OK, DB integrity OK, tables counted.
- Vaultwarden isolated restore drill: STATUS=OK on pve02.
- Authentik backup: STATUS=OK, archive under /mnt/staging/authentik-backups/snapshots.
- Authentik offhost: STATUS=OK to pve02.
- Authentik restore: STATUS=OK, tables/users/applications/proxy providers checked.
- Secret values must remain out of all reference files; only archive paths, sizes and SHA256 are recorded.
### Git / documentation / inventory apps
- Gitea backup: STATUS=OK, archive under /mnt/staging/gitea-backups/snapshots.
- Gitea offhost: STATUS=OK to pve02.
- Gitea restore: STATUS=OK, DB integrity OK, tables counted.
- NetBox backup: STATUS=OK, PostgreSQL dump under /var/lib/homelab-private/pre-update-backups/netbox-*.
- NetBox offhost: STATUS=OK to pve02.
- NetBox restore dry-run: STATUS=OK, restore container checked.
- Latest NetBox health proof observed: devices=7, sites=1, prefixes=3, IP addresses=12, virtual machines=49.
### Document / notes / personal data apps
- Paperless backup: STATUS=OK, latest snapshot under /mnt/staging/paperless-backups/snapshots.
- Paperless offhost: STATUS=OK to pve02.
- Paperless restore dry-run: STATUS=OK, pg_restore list and restore tables checked.
- Memos backup: STATUS=OK, latest snapshot under /mnt/staging/memos-backups/snapshots.
- Memos offhost: STATUS=OK to pve02.
- Memos restore proof exists from app restore quality checks.
- Actual Budget backup: STATUS=OK, offhost SHA256/listing OK, restore not attempted in observed latest proof.
- Linkding backup: STATUS=OK, offhost SHA256/listing OK, restore not attempted in observed latest proof.
- Karakeep backup: STATUS=OK, offhost SHA256/listing OK, restore not attempted in observed latest proof.
- Mealie backup: STATUS=OK, offhost SHA256/listing OK, restore not attempted in observed latest proof.
- n8n backup: STATUS=OK, offhost SHA256/listing OK, restore not attempted in observed latest proof.
### Files / media / sync apps
- Immich media backup: STATUS=OK, latest snapshot under /mnt/staging/immich-media-backups/snapshots.
- Immich media offhost: STATUS=OK to pve02.
- Immich media restore: STATUS=OK, local/offhost manifest match.
- Immich full consistency backup: STATUS=OK, includes PostgreSQL dump and media archive, offhost path present, restore dry-run OK.
- Nextcloud restore proof: STATUS=OK, OCC OK, DB_DUMP OK, DATA_MANIFEST OK, restore to production NO.
- Nextcloud restore proof is copied offhost.
- Filebrowser backup: STATUS=OK.
- Filebrowser offhost: STATUS=OK.
- Filebrowser restore validation: STATUS=OK.
- Jellyfin restore: STATUS=OK from app backup quality checks.
- Audiobookshelf restore: STATUS=OK from app backup quality checks.
- Calibre-Web restore: STATUS=OK from app backup quality checks.
- Syncthing is inventoried in Docker/stacks; dedicated backup signal not confirmed in this catalog.
### Home/admin/utility apps
- Home Assistant backup: STATUS=OK, archive includes configuration.yaml, storage dir and DB file, offhost SHA256/listing OK, restore not attempted in observed proof.
- Healthchecks restore: STATUS=OK, SQLite integrity OK and temp restore verify OK.
- BookStack restore: STATUS=OK, DB dump OK.
- Vikunja restore: STATUS=OK, SQLite integrity OK and temp restore verify OK.
- Kopia restore: STATUS=OK, synthetic snapshot and restore compare OK.
- Observability config backup: STATUS=OK, includes prometheus.yml, alertmanager.yml and health rules, offhost SHA256/listing OK.
- Ntfy is covered as part of observability/alerting stack but dedicated service backup signal was not confirmed in this catalog.
- MinIO is included in app-stacks-batch backup from 20260617; current dedicated timer was not confirmed.
- Homebox, Node-RED, MinIO, Mealie, Linkding, n8n and Actual Budget were part of app-stacks batch backup proof from 20260617.
### External service backups
- NetBird VPS backup: STATUS=OK, snapshot under /mnt/staging/netbird-vps-backups/snapshots.
- NetBird VPS offhost: STATUS=OK to pve02.
- NetBird VPS restore validation: STATUS=OK, archive SHA256 recorded.
- CodeVipe/forum-prod: VM160 Proxmox backup is present; pve02 also has codevipe-cloud-backup.timer.
- XenForo SMTP rotation has DB backup proof from earlier rotation work; production forum application-level backup catalog should be expanded separately with CodeVipe-specific DB/files/cron/public cutover details.
### Retention and cleanup
- Retention dry-run: STATUS=OK, production changed NO, destructive actions NO.
- Disk retention policy: STATUS=OK, root used pct observed 70 on edge-vm, removed dirs 0, Docker volume prune NO.
- App backup retention dry-run timer exists on edge-vm.
- homelab-backup-freshness timer exists on pve01.
- rollback-image-retention health file exists but had permission issue during one raw read; it must be read with sudo in future collection commands.
### Known coverage gaps / backlog
- backup-restore-coverage-slo remains a coverage backlog, not a runtime blocker.
- Actual Budget latest proof says RESTORE_ATTEMPTED=NO.
- Home Assistant latest proof says RESTORE_ATTEMPTED=NO.
- Linkding latest proof says RESTORE_ATTEMPTED=NO.
- Karakeep latest proof says RESTORE_ATTEMPTED=NO.
- Mealie latest proof says RESTORE_ATTEMPTED=NO.
- n8n latest proof says RESTORE_ATTEMPTED=NO.
- Observability config latest proof says RESTORE_ATTEMPTED=NO.
- Router config backup is currently reference/self-test based, not confirmed as recurring automated backup.
- Syncthing, ntfy, MinIO and several P2 services need either dedicated backup/restore proof or explicit classification as disposable/rebuildable.
### Proof
- Raw backup/offhost/restore inventory: 138_BACKUP_RESTORE_OFFHOST_CATALOG_RAW_INVENTORY.txt.
- Backup timer summary draft: 139_BACKUP_CATALOG_SERVICE_SUMMARY_DRAFT.txt.
- Normalized backup health index: 140_BACKUP_HEALTH_FILES_NORMALIZED_INDEX.txt.
## MONITORING_ALERTING_HEALTH_REFERENCE_20260630
### Monitoring stack
- Primary monitoring host: edge-vm, [PRIVATE_IP].
- Prometheus container: prometheus, local port 127.0.0.1:9090.
- Grafana container: grafana, local port 127.0.0.1:3003, VPN route grafana.vpn.gram1.ru.
- Alertmanager container: alertmanager, local port 127.0.0.1:9093.
- Loki container: loki, local port 127.0.0.1:3100.
- Alloy container: alloy, local port 127.0.0.1:12345.
- Uptime Kuma container: uptime-kuma, local port 127.0.0.1:3001, public route uptime.gram1.ru and VPN route kuma.vpn.gram1.ru.
- Gotify container: gotify, local port 127.0.0.1:8082, public route gotify.gram1.ru.
- ntfy container: ntfy, local port 127.0.0.1:8055, VPN route ntfy.vpn.gram1.ru.
- Healthchecks container: healthchecks, local port 127.0.0.1:8015, VPN route checks.vpn.gram1.ru.
- Blackbox exporter: blackbox-exporter, local port 127.0.0.1:9115.
- cAdvisor: cadvisor, local port 127.0.0.1:8081.
- Node exporter on edge-vm: node-exporter, local port 127.0.0.1:9100.
- Beszel: beszel, local port 127.0.0.1:8090.
- Dozzle: dozzle, local port 127.0.0.1:9999.
### PVE monitoring endpoints
- pve01: prometheus-node-exporter active on :9100, homelab-health-http active on :9101, Proxmox pveproxy on :8006.
- pve02: prometheus-node-exporter active on :9100, homelab-health-http active on :9101, Proxmox pveproxy on :8006.
- pve03: prometheus-node-exporter active on :9100, Proxmox pveproxy on :8006.
- pve03 has node-exporter/SMART timers but no homelab-health-http :9101 endpoint in observed output.
- PVE smartctl textfile timers exist on pve01/pve02/pve03.
- pve01 also runs private VPN host health, NetBird peer health, disk space health, MkDocs refresh, evidence catalog and quality gate timers.
### Prometheus config
- Prometheus scrape interval: 30s.
- Prometheus evaluation interval: 30s.
- Rule files path: /etc/prometheus/rules/*.yml.
- Alertmanager target: alertmanager:9093.
- Blackbox HTTP job targets:
- https://nc.gram1.ru
- https://git.gram1.ru
- https://auth.gram1.ru
- https://paper.gram1.ru
- https://backup.gram1.ru
- Edge node exporter scrape target: node-exporter:9100.
- PVE node exporter targets: [PRIVATE_IP]:9100, [PRIVATE_IP]:9100, [PRIVATE_IP]:9100.
- Proxmox exporter job targets pve01/pve02/pve03 through exporter endpoint [PRIVATE_IP]:9221.
- cAdvisor scrape target: cadvisor:8080.
### Alertmanager / notification routing
- Alertmanager route receiver: ntfy.
- Alertmanager webhook target: http://ntfy/homelab-alerts.
- Alert group_by: alertname, instance, severity.
- group_wait: 10s.
- group_interval: 5m.
- repeat_interval: 4h.
- Alert routing proof: alert-routing.txt STATUS=OK, RULES_ADDED=6, PROMETHEUS_RELOAD=OK, TEST_NOTIFICATION_SENT=YES, NTFY_MESSAGES_PUBLISHED_DELTA=1.
- alert-routing-health.txt is the standardized status alias and is STATUS=OK.
### Core Prometheus alert rules
- HomelabPrometheusTargetDown: up == 0 for 5m, severity warning.
- HomelabSmartDiskHealthFailed: homelab_smartctl_health_passed == 0 for 10m, severity critical.
- HomelabRootFilesystemLowSpace: root filesystem available below 15% for 15m, severity warning.
- HomelabPrometheusConfigReloadFailed: prometheus_config_last_reload_successful == 0 for 5m, severity critical.
- HomelabP0HealthStatusNotOk: homelab_health_status tier P0 equals 0 for 15m.
- HomelabP0BackupRestoreHealthStale: P0 backup/offhost/restore health older than 36h.
- HomelabP0WeeklyEdgeVmVzdumpHealthStale: weekly edge-vm vzdump health older than 8d.
- HomelabP1BackupHealthStale: P1 backup health older than 7d.
- HomelabMailCloudRestoreDrillStale: mail-cloud restore drill older than 8d.
- HomelabMkdocsRuntimePublishStale: MkDocs runtime publish older than 30d.
- HomelabHealthMetricsCollectorMissing: homelab health metrics absent from Prometheus.
- HomelabExternalCanaryHealthNotOkOrStale: external canary not OK or older than 30m.
- HomelabBlackboxProbeFailed / Missing: blackbox probe failed or absent.
- HomelabVulnScanHealthNotOkOrStale: vulnerability scan failed or older than 8d.
- HomelabNpmplusCertExpiryHealthNotOkOrStale: NPMplus cert expiry health failed or older than 48h.
- HomelabBlackboxTlsCertificateExpiringSoon: TLS certificate expiry within 14 days.
### Loki / Alloy log pipeline
- Loki version observed: grafana/loki:3.7.2.
- Alloy version observed: grafana/alloy:v1.17.0.
- Loki auth_enabled: false.
- Loki storage: local filesystem under /loki.
- Loki schema: tsdb v13.
- Loki retention_period: 14d.
- Alloy discovers Docker through socket-proxy at tcp://socket-proxy:2375.
- Alloy relabels container, compose_project, compose_service and host=edge-vm.
- Alloy forwards Docker logs to http://loki:3100/loki/api/v1/push.
- Loki readiness observed as "ready".
### Runtime dashboard semantics
- backup-restore-dashboard.txt is an authoritative runtime dashboard file.
- Current observed runtime dashboard: STATUS=OK, ITEMS=69, NOT_OK=0.
- backup-restore-dashboard.json confirmed not_ok=[].
- restore-drill-index.txt observed: STATUS=OK, RESTORE_PROOFS=16/16, MISSING empty.
- cluster-daily-status.txt observed: STATUS=OK, PROBLEM_COUNT=0.
- external-canary.txt observed: STATUS=OK, BAD=0.
- external canary checks include nc.gram1.ru, git.gram1.ru, auth.gram1.ru, backup.gram1.ru.
- Alertmanager local API is reachable.
- Gotify local /health returns green.
- ntfy local /v1/health returns healthy true.
- Uptime Kuma local endpoint returns HTTP 302 to /dashboard, which is expected for UI auth/redirect.
- Healthchecks local status endpoint returns HTTP 301, which is treated as reachable.
### Certificate / vulnerability health
- npmplus-cert-expiry-health.txt is the standardized health alias for certificate expiry.
- npmplus-certificate-expiry.txt is the detailed cert expiry file.
- Detailed cert expiry file observed cert_count=9 and min_cert_days_left=75 with warn threshold 30.
- cluster-daily-status local extra check reports npmplus-cert-expiry status OK, problems=0, min_days=75.
- trivy-scan.txt observed: STATUS=OK, IMAGE_TOTAL=58, IMAGE_SCAN_OK=58, IMAGE_SCAN_ERROR=0, CRITICAL_TOTAL=112, HIGH_TOTAL=1650.
- vuln-scan.txt observed: STATUS=OK, MODE=full, IMAGES=58, TRIVY_OK=49, CRITICAL_MATCHES=203.
### SLO backlog semantics
- Runtime dashboard OK does not mean SLO backlog is closed.
- slo-coverage-backlog-index.txt observed: STATUS=OK, BACKLOG_WARN_COUNT=12, RUNTIME_BLOCKER=NO.
- backup-restore-coverage-slo.txt observed: STATUS=WARN, APP_TOTAL=43, GREEN_COUNT=3, YELLOW_COUNT=21, RED_COUNT=19, MISSING_BACKUP_SIGNAL=31, MISSING_RESTORE_SIGNAL=28, MISSING_TIMER=31, MISSING_EXTERNAL_VALIDATION=43.
- Other observed WARN backlog files include container-image-lifecycle-slo, cron-job-monitoring-slo, direct-heartbeat-pilot-readiness, healthchecks-heartbeat-coverage, healthchecks-job-coverage, healthchecks-job-monitoring-slo, homepage-service-catalog-slo, ingress-dns-route-slo, observability-health-surface-slo and security-vulnerability-slo.
- SLO WARN files are tracked separately and are not runtime blockers unless explicitly promoted to P0 runtime checks.
### Important file locations
- Main health dir on edge-vm: /var/lib/homelab-health.
- Prometheus config: /opt/stacks/observability-lite/prometheus/prometheus.yml.
- Prometheus rules: /opt/stacks/observability-lite/prometheus/rules/.
- Alertmanager config: /opt/stacks/observability-lite/alertmanager/alertmanager.yml.
- Loki config: /opt/stacks/loki-alloy/loki-config.yaml.
- Alloy config: /opt/stacks/loki-alloy/config.alloy.
- Blackbox exporter config: /opt/stacks/blackbox-exporter/blackbox.yml.
- Uptime Kuma DB: /opt/uptime-kuma-compose/data/kuma.db.
- Gotify DB: /opt/gotify-compose/data/gotify.db.
- ntfy DB/cache: /opt/stacks/ntfy/data/user.db and /opt/stacks/ntfy/cache/cache.db.
- Healthchecks settings/data: /opt/stacks/healthchecks/local_settings.py and /var/lib/healthchecks.
### Known caveats
- A previous compact check looked for npmplus-cert-expiry.txt; use npmplus-certificate-expiry.txt for detailed cert expiry and npmplus-cert-expiry-health.txt for standardized health.
- health-metrics.txt is not required to exist as a health file; health metrics may be emitted via node-exporter textfile collector.
- Prometheus targets must be checked with explicit HTTP code and parsed JSON output, not a silent curl pipeline.
### Proof
- Raw monitoring inventory: 143_MONITORING_ALERTING_HEALTH_RAW_INVENTORY.txt.
- Runtime/SLO/alerting status: 144_MONITORING_RUNTIME_SLO_ALERTING_STATUS.txt.
- Kuma/Gotify/ntfy/Healthchecks status: 145_KUMA_GOTIFY_NTFY_HEALTHCHECKS_SAFE_STATUS.txt.
- Corrected monitoring compact status: 146_MONITORING_COMPACT_STATUS_CORRECTED.txt.
## PROMETHEUS_STATUS_CORRECTION_20260630
- Initial monitoring compact check after reference creation reported Prometheus API HTTP_CODE=000 on 127.0.0.1:9090.
- This was recorded as error-register item 30 and checked with proof files 149_PROMETHEUS_DIAG_AFTER_MONITORING_REFERENCE.txt, 150_PROMETHEUS_CONDITIONAL_RECOVERY.txt and 151_PROMETHEUS_REFERENCE_CORRECTION_PROOF.txt.
- The Monitoring/Alerting section must be interpreted together with the latest Prometheus correction proof, not only with the earlier secret-scan proof.
## COMMAND_PREFLIGHT_RULE_20260630
- Strict operator rule: before every command, check both the error register and this reference file.
- Required visible markers before main action: ERROR_REGISTER_CHECK=OK and REFERENCE_CHECK=OK.
- If either check fails, do not run the main action.
- Do not use exit 1 in interactive SSH sessions.
- Proof: 152_COMMAND_PREFLIGHT_RULE_RECORDED.txt.
## PROMETHEUS_TARGETS_SETTLED_20260630
- Prometheus was absent after the initial monitoring reference proof and was restarted in 150_PROMETHEUS_CONDITIONAL_RECOVERY.txt.
- Prometheus readiness proof exists in 151_PROMETHEUS_REFERENCE_CORRECTION_PROOF.txt.
- Target health was rechecked with an additional settled scrape proof in 153_PROMETHEUS_TARGETS_SETTLED_PROOF.txt.
- Interpret Prometheus status from the latest settled proof, not from the earlier immediate-after-start unknown target state.
## PROMETHEUS_TARGETS_SETTLED_PROOF_CORRECTION_20260630
- 153_PROMETHEUS_TARGETS_SETTLED_PROOF.txt is invalid and must not be used as evidence.
- Reason: nested Python inside SSH lost quoting and produced SyntaxError.
- Error-register item 32 records this mistake.
- Valid replacement proof: 154_PROMETHEUS_TARGETS_SETTLED_PROOF_NO_PYTHON.txt.
- Prometheus status must be interpreted from 154 and later proofs, not from 153.
## MONITORING_PROMETHEUS_FINAL_CLOSED_20260630
- Prometheus was initially absent after monitoring reference creation.
- It was restarted and then verified after scrape settling.
- Valid settled proof: 154_PROMETHEUS_TARGETS_SETTLED_PROOF_NO_PYTHON.txt.
- Final observed state: HTTP_CODE=200, PROM_TARGETS_TOTAL=14, PROM_UP_COUNT=14, PROM_NON_UP_COUNT=0.
- Correction/secret scan proof: 156_PROMETHEUS_CORRECTION_PROOF_AND_SECRET_SCAN.txt.
- Invalid proof 153 must not be used.
## SECURITY_ACCESS_SECRETS_OPERATING_MODEL_20260630
### Command safety
- Before every command, check /etc/pve/HOMELAB_ASSISTANT_ERROR_REGISTER.md and current 31_HOMELAB_REFERENCE.md.
- Print ERROR_REGISTER_CHECK=OK and REFERENCE_CHECK=OK before the main action.
- Do not run the main action if either check fails.
- Do not use exit 1 in interactive SSH sessions.
- Avoid long nested SSH/Python/PHP/SQL quoting chains.
- Do not print secrets.
### Access model
- Primary operator entrypoint: root@pve01 / [PRIVATE_IP].
- pve02 and pve03 are reached as root from pve01.
- edge-vm is reached as debian@[PRIVATE_IP] with sudo.
- forum-prod is reached through pve02 using [SENSITIVE_PATH] to root@[PRIVATE_IP].
- PVE users observed: root@pam and prometheus@pve.
- prometheus@pve has PVEAuditor on / for Proxmox exporter access.
### SSH and permissions
- pve01 root keys observed: id_rsa, id_ed25519 and public keys.
- pve02 root keys observed: id_rsa and forum-prod-ci-key.
- pve03 root key observed: id_rsa.
- edge-vm root key observed: id_ed25519; debian authorized_keys observed.
- Security finding: pve01/pve02/pve03 root authorized_keys files were observed as mode 777.
- Correction proof: 160_PVE_ROOT_AUTHORIZED_KEYS_PERMISSIONS_FIXED.txt.
- Private key material must never be copied into the reference.
### Secret storage
- Primary private storage root: /var/lib/homelab-private.
- Edge private secrets root: /var/lib/homelab-private/secrets.
- NPMplus DB: /opt/npmplus/npmplus/database.sqlite, root-only mode observed.
- Gotify DB: /opt/gotify-compose/data/gotify.db.
- Uptime Kuma DB: /opt/uptime-kuma-compose/data/kuma.db.
- Vaultwarden DB: /opt/vaultwarden-compose/data/db.sqlite3.
- Rclone configs exist under root config paths and must not be printed.
- Env/secret inventory is path-only: owner, mode, size and path only.
### Network exposure
- PVE nodes expose SSH :22, Proxmox :8006 and node-exporter :9100.
- pve01/pve02 expose homelab-health-http :9101.
- pve01 exposes NUT :3493 on localhost and [PRIVATE_IP].
- edge-vm exposes public :80/:443 through NPMplus.
- edge-vm NPMplus admin :81 and socket-proxy :2375 are bound to 127.0.0.1.
- edge-vm registry cache :5000 is bound to [PRIVATE_IP].
- edge-vm Home Assistant :8123 is intentionally LAN-exposed.
- Edge ingress hardening proof reports STATUS=OK.
### Rotation and scan proofs
- Cloudflare token rotation completed without printing token values.
- Old Cloudflare token revocation was externally confirmed.
- XenForo SMTP password rotation completed without printing password values.
- XenForo SMTP auth was verified with STARTTLS-safe method.
- Current reference strict scan shows zero strict secret hits.
- Selected generated reference blocks show zero strict secret hits.
### Proof
- Raw security/access inventory: 158_SECURITY_ACCESS_SECRETS_RAW_INVENTORY.txt.
- Secret scan and rotation proof index: 159_SECURITY_SECRET_SCAN_AND_ROTATION_PROOF_INDEX.txt.
- SSH permission correction proof: 160_PVE_ROOT_AUTHORIZED_KEYS_PERMISSIONS_FIXED.txt.
## SECURITY_SSH_PERMISSION_CORRECTION_20260630
- Initial proof 160_PVE_ROOT_AUTHORIZED_KEYS_PERMISSIONS_FIXED.txt is not sufficient as closure evidence.
- Reason: stat without -L still showed 777 after chmod, which may be symlink mode rather than target file mode.
- Error-register item 35 records this proof-quality issue.
- Authoritative replacement proof: 163_PVE_ROOT_AUTHORIZED_KEYS_SYMLINK_AWARE_FIX.txt.
- Valid closure condition: target permissions checked with stat -L are 600 for authorized_keys files and [SENSITIVE_PATH] is 700.
- Final observed status in replacement proof: STATUS=SSH_AUTHORIZED_KEYS_TARGET_PERMISSIONS_OK.
## STORAGE_DISK_SMART_CAPACITY_REFERENCE_20260630
### Proxmox storage model
- Cluster storage is defined in /etc/pve/storage.cfg.
- local storage: dir /var/lib/vz, content iso,vztmpl,backup,import.
- local-lvm storage: lvmthin data, content rootdir,images.
- Nightly vzdump job writes to local storage.
- VM/CT images are primarily stored on local-lvm.
- Staging/offhost paths are under /mnt/staging where present.
### Node disks and capacity
- pve01: primary disk previously inventoried as Lexar SSD NQ7A1 1TB.
- pve02: primary disk previously inventoried as SK800-1TB.
- pve03: primary disk previously inventoried as Samsung 870 EVO 500GB.
- pve01 storage usage was previously observed around local 32.56% and local-lvm 17.50%.
- pve02 storage usage was previously observed around local 12.86% and local-lvm 11.65%.
- pve03 storage usage was previously observed around local 35.79% and local-lvm 64.84%.
- Raw current df/lsblk/LVM/SMART evidence is in 166_STORAGE_DISK_SMART_CAPACITY_RAW_INVENTORY.txt.
### Edge VM storage model
- edge-vm runs Docker application stacks.
- Docker app data lives mainly under /opt/stacks, /opt/npmplus, /opt/gotify-compose, /opt/uptime-kuma-compose, /opt/vaultwarden-compose and /var/lib application paths.
- Immich media is mounted separately at /mnt/immich-media in the container mapping.
- Docker volume and image usage is captured in docker system df output.
- Disk retention health is tracked by disk-retention-policy and disk-retention-cleanup files.
### SMART and health monitoring
- PVE nodes run smartctl textfile timers.
- pve01/pve02/pve03 expose node-exporter :9100 for Prometheus.
- pve01 and pve02 expose homelab-health-http :9101.
- SMART/disk/capacity health evidence is collected from /var/lib/homelab-health where present.
- Prometheus alert rule HomelabSmartDiskHealthFailed watches homelab_smartctl_health_passed == 0.
- Prometheus alert rule HomelabRootFilesystemLowSpace watches root filesystem available below 15%.
- pve03 did not have /var/lib/homelab-health in previous backup-health collection, but it has node-exporter and smartctl timers.
### Retention and cleanup
- Edge disk retention policy previously observed STATUS=OK.
- Edge disk retention policy observed ROOT_USED_PCT=70, REMOVED_DIRS=0, DOCKER_VOLUME_PRUNE=NO.
- Retention dry-run policy is designed to avoid destructive production changes.
- Broad Docker prune is not used as a default cleanup mechanism.
- Cleanup and retention actions must have explicit proof files.
### Operational rules
- Before deleting or pruning storage, create or identify rollback/backup proof.
- Do not delete vzdump/offhost/cloud backup artifacts unless retention policy and proof explicitly allow it.
- Do not infer offhost success from a failed rsync.
- For backup artifacts, verify exact file, size, checksum or zstd/vma integrity where applicable.
- For SMART, use health output plus selected attributes, not full raw dumps with unnecessary noise.
### Proof
- Raw storage/disk/SMART/capacity inventory: 166_STORAGE_DISK_SMART_CAPACITY_RAW_INVENTORY.txt.
- Earlier Proxmox node/storage inventory: 120_PROXMOX_CLUSTER_NODE_STORAGE_NETWORK_INVENTORY.txt.
- Backup/restore catalog: 141_REFERENCE_BACKUP_RESTORE_OFFHOST_CATALOG_BLOCK.txt.
- Monitoring rules and SMART alert model: 147_REFERENCE_MONITORING_ALERTING_HEALTH_BLOCK.txt.
## STORAGE_CAPACITY_CORRECTION_20260630
- Storage block integrity review proof: 169_STORAGE_BLOCK_INTEGRITY_AND_PVE03_CAPACITY_REVIEW.txt.
- 167_REFERENCE_STORAGE_DISK_SMART_CAPACITY_BLOCK.txt integrity status: OK.
- pve03 /mnt/staging current observed use percent: 77.
- pve03 staging is close to the existing WARN threshold of 80% if observed use is 75% or higher.
- pve01 disk-space.txt coverage visibly listed pve01_staging, pve02_staging and edge_immich_media; pve03_staging coverage must be added or explicitly documented elsewhere.
- Storage layer is operationally OK only if SMART remains PASSED and pve03 staging is tracked before it crosses WARN/CRIT thresholds.
- This is a capacity coverage note, not a secret or runtime outage.
## PVE03_STAGING_CAPACITY_MONITOR_20260630
- pve03 /mnt/staging was observed at 77% usage.
- Existing pve01 disk-space.txt did not visibly include pve03_staging coverage.
- A dedicated pve03 staging capacity health check was added on pve01.
- Health file: /var/lib/homelab-health/pve03-staging-capacity.txt.
- Timer: homelab-pve03-staging-capacity-health.timer.
- Service: homelab-pve03-staging-capacity-health.service.
- Current observed status: OK.
- WARN threshold: 80%.
- CRIT threshold: 90%.
- Proof: 173_PVE03_STAGING_CAPACITY_HEALTH_INSTALLED.txt.
## STORAGE_LAYER_FINAL_CLOSED_WITH_PVE03_MONITOR_20260630
- Storage/disk/SMART/capacity layer is closed with an explicit pve03 staging capacity monitor.
- pve03 /mnt/staging was near WARN threshold at 77%, so a dedicated health file and timer were added.
- Health file: /var/lib/homelab-health/pve03-staging-capacity.txt.
- Timer: homelab-pve03-staging-capacity-health.timer.
- Final closure proof: 176_STORAGE_FINAL_CLOSURE_WITH_PVE03_MONITOR.txt.
## SERVICE_DEPENDENCY_MAP_20260630
### Ingress and DNS chain
- Home DHCP DNS: [PRIVATE_IP] and [PRIVATE_IP].
- dns1: CT110 / [PRIVATE_IP] / AdGuard Home.
- dns2: CT111 / [PRIVATE_IP] / AdGuard Home.
- unbound1: CT112 / [PRIVATE_IP]:5335.
- unbound2: CT113 / [PRIVATE_IP]:5335.
- dns1 upstream: [PRIVATE_IP]:5335.
- dns2 upstream: [PRIVATE_IP]:5335.
- Public ingress HTTP/HTTPS: router forwards TCP 80/443 to edge-vm [PRIVATE_IP].
- Reverse proxy: NPMplus on edge-vm, container npmplus, host networking, admin bound to 127.0.0.1:81.
- Edge runtime host: VM130 edge-vm, [PRIVATE_IP], Docker Compose projects count 43.
- NPMplus proxy routes count: 47.
- NPMplus certificates and proxy route source proof: 129_NPMPLUS_PROXY_CERT_STREAM_ROWS_SAFE.txt.
### Core public routes
| Route | Upstream | Runtime owner | Backup/health evidence |
|---|---|---|---|
| nc.gram1.ru | http://[PRIVATE_IP]:11000 | VM150 Nextcloud AIO | VM150 vzdump, nextcloud restore proof, external canary |
| uptime.gram1.ru | http://127.0.0.1:3001 | edge-vm / uptime-kuma | npmplus-kuma backup/restore, Kuma health |
| gotify.gram1.ru | http://127.0.0.1:8082 | edge-vm / gotify | gotify health, alert-routing proof |
| vault.gram1.ru | http://127.0.0.1:8083 | edge-vm / vaultwarden | vaultwarden backup/offhost/restore |
| dockge.gram1.ru | http://127.0.0.1:5001 | edge-vm / dockge | stack inventory |
| home.gram1.ru | http://127.0.0.1:3000 | edge-vm / homepage | homepage container, dashboard route |
| git.gram1.ru | http://127.0.0.1:3002 | edge-vm / gitea | gitea backup/offhost/restore |
| dozzle.gram1.ru | http://127.0.0.1:9999 | edge-vm / dozzle | docker stack inventory |
| paper.gram1.ru | http://127.0.0.1:8010 | edge-vm / paperless | paperless backup/offhost/restore |
| memos.gram1.ru | http://127.0.0.1:5230 | edge-vm / memos | memos backup/offhost/restore |
| photos.gram1.ru | http://127.0.0.1:2283 | edge-vm / immich | immich media/full consistency proof |
| auth.gram1.ru | http://127.0.0.1:9000 | edge-vm / authentik | authentik backup/offhost/restore |
| backup.gram1.ru | http://[PRIVATE_IP]:9101 | pve01 / homelab-health-http | backup dashboard and health proofs |
### VPN routes on wildcard certificate
| Route | Upstream | Runtime owner |
|---|---|---|
| kuma.vpn.gram1.ru | http://127.0.0.1:3001 | uptime-kuma |
| homepage.vpn.gram1.ru | http://127.0.0.1:3000 | homepage |
| gitea.vpn.gram1.ru | http://127.0.0.1:3002 | gitea |
| filebrowser.vpn.gram1.ru | http://127.0.0.1:8085 | filebrowser |
| auth.vpn.gram1.ru | http://127.0.0.1:9000 | authentik |
| actual-budget.vpn.gram1.ru | http://127.0.0.1:5006 | actual-budget |
| grafana.vpn.gram1.ru | http://127.0.0.1:3003 | grafana |
| homebox.vpn.gram1.ru | http://127.0.0.1:7745 | homebox |
| mealie.vpn.gram1.ru | http://127.0.0.1:9001 | mealie |
| n8n.vpn.gram1.ru | http://127.0.0.1:5678 | n8n |
| netbox.vpn.gram1.ru | http://127.0.0.1:8000 | netbox |
| node-red.vpn.gram1.ru | http://127.0.0.1:1880 | node-red |
| beszel.vpn.gram1.ru | http://127.0.0.1:8090 | beszel |
| it-tools.vpn.gram1.ru | http://127.0.0.1:8084 | it-tools |
| karakeep.vpn.gram1.ru | http://127.0.0.1:3030 | karakeep |
| linkding.vpn.gram1.ru | http://127.0.0.1:9091 | linkding |
| minio.vpn.gram1.ru | http://127.0.0.1:9003 | minio |
| ntfy.vpn.gram1.ru | http://127.0.0.1:8055 | ntfy |
| searxng.vpn.gram1.ru | http://127.0.0.1:8888 | searxng |
| syncthing.vpn.gram1.ru | http://127.0.0.1:8384 | syncthing |
| alertmanager.vpn.gram1.ru | http://127.0.0.1:9093 | alertmanager |
| prometheus.vpn.gram1.ru | http://127.0.0.1:9090 | prometheus |
| homeassistant.vpn.gram1.ru | http://127.0.0.1:8123 | homeassistant |
| npmplus.vpn.gram1.ru | https://[PRIVATE_IP]:81 | NPMplus admin, localhost-bound on edge |
| proxmox.vpn.gram1.ru | https://[PRIVATE_IP]:8006 | pve01 Proxmox UI |
| dns1.vpn.gram1.ru | http://[PRIVATE_IP]:8080 | dns1 AdGuard |
| dns2.vpn.gram1.ru | http://[PRIVATE_IP]:8080 | dns2 AdGuard |
| checks.vpn.gram1.ru | http://127.0.0.1:8015 | healthchecks |
| vikunja.vpn.gram1.ru | http://127.0.0.1:8016 | vikunja |
| bookstack.vpn.gram1.ru | http://127.0.0.1:8017 | bookstack |
| pdf.vpn.gram1.ru | http://127.0.0.1:8018 | stirling-pdf |
| jellyfin.vpn.gram1.ru | http://127.0.0.1:8019 | jellyfin |
| audiobooks.vpn.gram1.ru | http://127.0.0.1:8020 | audiobookshelf |
| books.vpn.gram1.ru | http://127.0.0.1:8021 | calibre-web |
### Disabled / special routes
- npm.gram1.ru exists in NPMplus but was observed disabled.
- turn.gram1.ru is an AdGuard rewrite to [PRIVATE_IP] for Nextcloud Talk/TURN.
- Router forwards TCP/UDP 3478 to Nextcloud [PRIVATE_IP].
- Router forwards TCP/UDP 51820 for WireGuard on the Home bridge, not to edge-vm.
### Critical dependency rules
- If a public app is down, check in this order: DNS rewrite/upstream, NPMplus route/cert, upstream container/VM, app health file, backup/restore proof.
- If a VPN route is down, check NetBird first, then NPMplus wildcard cert, then local upstream.
- If DNS is broken, check dns1/dns2 AdGuard and unbound1/unbound2 before app containers.
- If NPMplus is broken, use backup/restore evidence from npmplus-kuma-config and NPMplus DB backup.
- If edge-vm is broken, VM130 full-image backup/offhost/restore proofs are authoritative.
- If pve03 storage approaches WARN/CRIT, use pve03-staging-capacity health file.
### Proof
- Raw service dependency evidence index: 177_SERVICE_DEPENDENCY_MAP_RAW_EVIDENCE_INDEX.txt.
- NPMplus route/cert rows: 129_NPMPLUS_PROXY_CERT_STREAM_ROWS_SAFE.txt.
- AdGuard/Unbound rewrites/upstreams: 130_ADGUARD_UPSTREAM_REWRITES_UNBOUND_RULES_SAFE.txt.
- Edge Docker container normalized map: 134_EDGE_DOCKER_CONTAINER_NORMALIZED_MAP.txt.
- Proxmox VM/CT inventory: 121_PROXMOX_VM_CT_CONFIG_INVENTORY_REDACTED.txt.
- Backup catalog: 141_REFERENCE_BACKUP_RESTORE_OFFHOST_CATALOG_BLOCK.txt.
- Monitoring model: 147_REFERENCE_MONITORING_ALERTING_HEALTH_BLOCK.txt.
## SERVICE_DEPENDENCY_MAP_FINAL_CLOSED_20260630
- Service Dependency Map layer is closed.
- Integrity proof: 180_SERVICE_DEPENDENCY_MAP_INTEGRITY_REVIEW.txt.
- Secret scan proof: 179_SERVICE_DEPENDENCY_MAP_REFERENCE_PROOF_AND_SECRET_SCAN.txt.
- Final closure proof: 181_SERVICE_DEPENDENCY_MAP_FINAL_CLOSURE.txt.
## RUNBOOKS_RECOVERY_PROCEDURES_20260630
### Universal operator preflight
- Before every command, check /etc/pve/HOMELAB_ASSISTANT_ERROR_REGISTER.md and the current 31_HOMELAB_REFERENCE.md.
- Required visible markers: ERROR_REGISTER_CHECK=OK and REFERENCE_CHECK=OK.
- If preflight fails, do not run the main action.
- Do not use exit 1 in interactive SSH sessions.
- Do not paste full terminal transcripts back into shell.
- Do not print secrets.
- Prefer short proof-producing commands over long nested quoting chains.
### First triage order
- Check current reference layer first.
- Check latest proof file named by the relevant reference layer.
- Check health files under /var/lib/homelab-health where applicable.
- Check runtime state only after understanding the owning node, VM, container and proxy route.
- Record every failed command or misleading proof in the error register before moving on.
### Public application down
- Start with Service Dependency Map.
- Check DNS path: dns1/dns2 AdGuard, then unbound1/unbound2.
- Check NPMplus route and certificate using NPMplus DB/proxy proof.
- Check upstream app container or VM.
- Check app-specific health, backup and restore proof.
- Check external-canary and blackbox/Prometheus if route is public.
- Do not change DNS, certificates or proxy routes without DB backup and proof.
### DNS failure
- dns1 is CT110 at [PRIVATE_IP].
- dns2 is CT111 at [PRIVATE_IP].
- unbound1 is CT112 at [PRIVATE_IP]:5335.
- unbound2 is CT113 at [PRIVATE_IP]:5335.
- AdGuard upstreams must point to local Unbound pair.
- Rewrites include public routes for git, dozzle, paper, memos, photos, auth and backup to edge-vm.
- turn.gram1.ru points to Nextcloud [PRIVATE_IP].
- Use DNS/Ingress reference and proof files before editing configs.
### Ingress / NPMplus failure
- Edge VM is VM130 at [PRIVATE_IP].
- NPMplus listens publicly on 80/443 and admin is bound to 127.0.0.1:81.
- NPMplus DB is /opt/npmplus/npmplus/database.sqlite.
- Before modifying certificates or proxy rows, create a DB backup.
- Cloudflare token values must never be printed.
- Use npmplus-kuma-config backup/offhost/restore proofs for recovery.
### Edge VM failure
- VM130 is the Docker runtime host.
- First check Proxmox VM state and pve03 storage.
- Then use VM130 full-image vzdump/offhost/restore proofs.
- Current authoritative edge-vm backup/offhost/restore evidence is in backup catalog and VM130 proofs.
- If restoring, verify exact archive, checksum and VMA/ZSTD integrity before booting replacement.
### Proxmox / VM / CT restore
- Primary nightly job: homelab-nightly-all at 03:30, zstd, snapshot.
- Included VMIDs: 110,111,112,113,130,150,160.
- Use backup catalog for latest known backup/offhost/restore evidence.
- Do not infer offhost success from failed rsync.
- Verify exact artifact, size and checksum where available.
- For VM130 and VM160, use their dedicated closure proofs.
### Monitoring / alerting failure
- Prometheus is on edge-vm at 127.0.0.1:9090.
- Alertmanager is on 127.0.0.1:9093.
- Loki is on 127.0.0.1:3100.
- Uptime Kuma is on 127.0.0.1:3001.
- Gotify is on 127.0.0.1:8082.
- ntfy is on 127.0.0.1:8055.
- Healthchecks is on 127.0.0.1:8015.
- Prometheus proof 153 is invalid and must not be used.
- Use proof 154 and final closure 157 for Prometheus settled target state.
### Backup dashboard / SLO interpretation
- Runtime dashboard OK means current operational checks are green.
- SLO backlog WARN means coverage improvement backlog, not necessarily runtime outage.
- Do not merge SLO backlog WARN into runtime blockers unless explicitly promoted.
- Use slo-coverage-backlog-index for backlog status.
### Storage / capacity event
- pve03 staging is monitored separately because it was observed at 77%.
- Health file: /var/lib/homelab-health/pve03-staging-capacity.txt.
- WARN threshold: 80%.
- CRIT threshold: 90%.
- Timer: homelab-pve03-staging-capacity-health.timer.
- Before cleanup, check retention policy and backup/offhost proof.
- Do not use broad Docker prune by default.
### Security / secret incident
- Stop printing values immediately.
- Identify whether the leak is value, file path, or false-positive text.
- Rotate affected token/password if a value was exposed.
- Create backup before DB/config mutation.
- Re-run strict secret scan after rotation.
- Record proof files and external revocation confirmation where applicable.
- Cloudflare token and XenForo SMTP rotations already have closure proofs.
### Forum / CodeVipe incident
- forum-prod is VM160 at [PRIVATE_IP].
- Access path is through pve02 using [SENSITIVE_PATH]
- XenForo path: /var/www/codevipe/public.
- SMTP host: mail.pvepro.ru on port 587 with STARTTLS semantics.
- Do not enable smtpSsl=true blindly for port 587.
- Do not use fragile nested PHP/base64 SMTP checkers.
- Use XenForo SMTP rotation and auth proof files for current mail state.
### Final evidence rules
- Every remediation gets a numbered proof file.
- Every reference block gets a proof and secret scan file.
- Invalid proof files must be explicitly superseded, not silently ignored.
- Reference closure requires integrity scan, secret scan and required-heading checks.
## HOMELAB_REFERENCE_FINAL_OPERATOR_STATUS_20260630
### Status
- Current homelab reference is complete for the audited local infrastructure scope.
- Final quality precheck passed.
- Required headings are present.
- Bad terminal/log marker scan is clean.
- Strict secret scan is clean.
- This is a reference/operator-status closure, not an archive/export.
### Closed layers
- Critical runtime tails closure.
- Proxmox cluster, node, storage and VM/CT inventory.
- DNS, ingress, NPMplus certificates and AdGuard/Unbound model.
- Edge Docker stacks and container map.
- Backup, restore, offhost and cloud backup catalog.
- Monitoring, alerting, health dashboard and Prometheus correction.
- Security, access and secrets operating model.
- Storage, disk, SMART and capacity model with pve03 staging monitor.
- Service Dependency Map.
- Runbooks and recovery procedures.
### Important superseded/invalid proofs
- 153_PROMETHEUS_TARGETS_SETTLED_PROOF.txt is invalid and must not be used.
- 160_PVE_ROOT_AUTHORIZED_KEYS_PERMISSIONS_FIXED.txt is not sufficient by itself because symlink mode showed 777.
- Use 154 for Prometheus settled targets and 163 for symlink-aware SSH target permissions.
### Current authoritative final proof set
- 185_HOMELAB_REFERENCE_FINAL_QUALITY_SCAN_PRECHECK.txt.
- 184_RUNBOOKS_RECOVERY_REFERENCE_PROOF_AND_SECRET_SCAN.txt.
- 181_SERVICE_DEPENDENCY_MAP_FINAL_CLOSURE.txt.
- 176_STORAGE_FINAL_CLOSURE_WITH_PVE03_MONITOR.txt.
- 165_SECURITY_ACCESS_SECRETS_FINAL_PROOF_AND_SECRET_SCAN.txt.
- 157_MONITORING_PROMETHEUS_FINAL_CLOSURE.txt.
- 142_BACKUP_RESTORE_OFFHOST_REFERENCE_PROOF_AND_SECRET_SCAN.txt.
- 136_EDGE_DOCKER_STACKS_REFERENCE_PROOF_AND_SECRET_SCAN.txt.
- 132_DNS_INGRESS_REFERENCE_PROOF_AND_SECRET_SCAN.txt.
### Operator rule
- Before every future command, continue checking both the error register and this reference file.
- Required visible markers remain ERROR_REGISTER_CHECK=OK and REFERENCE_CHECK=OK.
## HOMELAB_REFERENCE_DEEP_AUDIT_CLOSURE_20260630
### Deep audit result
- Error register and reference consistency audit passed.
- Final proof files unresolved-marker audit passed.
- Reference proof-link audit passed.
- All referenced proof files exist.
- No unresolved final REVIEW_REQUIRED, PROOF_MISSING, HEADING_MISSING, BAD_MARKER, STRICT_SECRET, RULE_CHECK_FAIL, SyntaxError or PERM_BAD markers remained in final proof set.
- Invalid/superseded proofs are explicitly documented and must not be used as authoritative closure evidence.
### Authoritative deep audit proofs
- 189_DEEP_ERROR_REGISTER_REFERENCE_CONSISTENCY_AUDIT.txt.
- 190_DEEP_PROOF_FILES_UNRESOLVED_MARKER_AUDIT.txt.
- 191_DEEP_REFERENCE_PROOF_LINK_AUDIT.txt.
### Scope statement
- This closes the current audited local homelab reference scope.
- External systems can still receive separate dedicated passports if the scope is expanded later.
## NEXTCLOUD_VM150_MAIL_CLOUD_BACKUP_20260630
- VM150 Nextcloud Mail-cloud backup is installed on pve01.
- Timer: homelab-mail-cloud-nextcloud-vm-upload.timer.
- Health file: /var/lib/homelab-health/mail-cloud-nextcloud-vm.txt.
- Current proof health: STATUS=OK, MODE=recurring-chunked, PARTS=14, DOWNLOAD_VERIFY=yes, SECRET_PRINTED=no.
- Proof: 192_NEXTCLOUD_VM150_MAIL_CLOUD_CHUNKED_PROOF.txt.
## ROUTER_RECURRING_BACKUP_BLOCKER_20260630
- Router recurring backup from pve01 is not installed yet.
- Reason: pve01 cannot reach router management ports 2222, 5080, 5083 or 2323 on [PRIVATE_IP].
- Current router backup state remains reference/self-test based until router management access is allowed or startup-config is exported by another approved path.
- Do not run bash/POSIX automation against Netcraze CLI; use router CLI syntax only.
- Proof: 195_ROUTER_RECURRING_BACKUP_BLOCKED_BY_MGMT_ACL_PROOF.txt.
## EDGE_GAP_RESTORE_DRYRUN_CLOSURE_20260630
- Restore dry-run gaps closed for Actual Budget, Home Assistant, Karakeep, Linkding, Mealie, n8n and Observability config.
- Production data was not restored or modified; checks used temporary extraction, SHA verification, tar listing/extraction and SQLite integrity where applicable.
- Current observed restore health: all 7 STATUS=OK with RESTORE_ATTEMPTED=YES and SECRET_PRINTED=no.
- Proof: 197_EDGE_GAP_RESTORE_DRYRUN_CLOSURE_PROOF.txt.
## P2_SMALL_STACKS_BACKUP_RESTORE_CLOSURE_20260630
- P2 small-stack backup and restore dry-run is installed on edge-vm.
- Covered paths: Syncthing config, ntfy cache/data, MinIO data, Homebox data and Node-RED data.
- Timer: homelab-p2-small-stacks-backup-restore.timer.
- Current observed health: STATUS=OK, BACKUP_PATH_COUNT=6, RESTORE_ATTEMPTED=YES, SQLITE_INTEGRITY_OK=yes, SECRET_PRINTED=no.
- Proof: 199_P2_SMALL_STACKS_BACKUP_RESTORE_CLOSURE_PROOF.txt.
## BACKUP_CLOUD_AND_RESTORE_FINAL_CLOSURE_20260630
- Backup/cloud/restore improvement pass is closed with runtime dashboard STATUS=OK and NOT_OK=0.
- VM150 Nextcloud now has Mail-cloud chunked backup with download verification.
- Restore dry-run gaps closed for Actual Budget, Home Assistant, Karakeep, Linkding, Mealie, n8n and Observability config.
- P2 small stacks now have backup and restore dry-run coverage for Syncthing config, ntfy, MinIO, Homebox and Node-RED.
- Router recurring backup remains blocked by management ACL and is documented as blocker, not silently closed.
- Proof: 201_BACKUP_CLOUD_AND_RESTORE_FINAL_CLOSURE_PROOF.txt.
## ROUTER_STARTUP_CONFIG_MAIL_CLOUD_MANUAL_20260630
- Router startup-config manual backup is stored in Mail-cloud crypt remote.
- Source file content is not printed in proofs or reference.
- Health file: /var/lib/homelab-health/router-startup-config-mail-cloud.txt.
- Current observed health: STATUS=OK, UPLOAD_OK=yes, DOWNLOAD_VERIFY=yes, SECRET_PRINTED=no.
- Recurring router backup is still blocked until pve01 can reach router management ports.
- Proof: 203_ROUTER_STARTUP_CONFIG_MAIL_CLOUD_MANUAL_PROOF.txt.
## ROUTER_RUNNING_CONFIG_MAIL_CLOUD_RECURRING_20260630
- Router running-config recurring Mail-cloud backup is installed on pve01.
- Timer: homelab-router-running-config-mail-cloud.timer.
- Health file: /var/lib/homelab-health/router-running-config-mail-cloud.txt.
- Current observed health: STATUS=OK, SSH_RC=0, LOOKS_CONFIG=yes, UPLOAD_OK=yes, DOWNLOAD_VERIFY=yes, SECRET_PRINTED=no.
- Manual startup-config Mail-cloud backup also exists as separate proof.
- Proof: 205_ROUTER_RUNNING_CONFIG_MAIL_CLOUD_RECURRING_PROOF.txt.
## POST_BACKUP_PASS_SLO_RECONCILIATION_20260630
- Post backup/cloud/restore pass SLO reconciliation is closed.
- Runtime backup dashboard remains STATUS=OK with NOT_OK=0.
- Restore index remains STATUS=OK with RESTORE_PROOFS=16/16.
- VM150 Mail-cloud backup, router manual/recurring Mail-cloud backups, P2 small-stacks and 7 edge restore dry-runs are all confirmed OK.
- Existing backup-restore-coverage-slo.txt is historical model output from 20260625 and does not represent current runtime blocker state.
- Current reconciliation health: STATUS=OK, RUNTIME_BLOCKER=NO, SECRET_PRINTED=no.
- Proof: 208_POST_BACKUP_PASS_SLO_RECONCILIATION_PROOF.txt.
## POST_BACKUP_PASS_ALERTING_20260630
- Prometheus alerting for post-backup-pass health files is installed and loaded.
- Rule file: /opt/stacks/observability-lite/prometheus/rules/homelab-post-backup-pass-alerts.yml.
- Alerts: HomelabPostBackupPassHealthNotOk and HomelabPostBackupPassHealthStale.
- Covered checks include VM150 Mail-cloud, router startup/running config Mail-cloud, P2 small-stacks and seven restore dry-run health files.
- Current proof confirms rule validation, Prometheus API visibility and up targets.
- Proof: 210_POST_BACKUP_PASS_ALERTING_PROOF.txt.
## POST_BACKUP_PASS_ALERTS_NOT_FIRING_20260630
- Post-backup-pass Prometheus alert rules are loaded and currently not firing.
- Corrected proof uses direct PromQL ALERTS query for firing state, not grep over rule names.
- Alerts checked: HomelabPostBackupPassHealthNotOk and HomelabPostBackupPassHealthStale.
- Proof: 213_POST_BACKUP_PASS_ALERTS_NOT_FIRING_CORRECTED_PROOF.txt.
## MAIL_CLOUD_CAPACITY_RETENTION_20260630
- Mail-cloud capacity and backup directory inventory was checked after post-backup-pass closure.
- VM150 Nextcloud, router startup/running config, P0 critical and edge-vm backup directories are present on crypt remotes.
- Observed capacity is sufficient: largest used remote was pve03-mail-03-crypt around 65 GiB of 1 TiB.
- Retention shape observed: P0 critical has 10 visible dirs with keep 14, edge-vm has 3 visible dirs with keep 4, new VM150/router jobs have initial dirs.
- Proof: 216_MAIL_CLOUD_CAPACITY_RETENTION_PROOF.txt.
## ROUTER_BACKUP_SECRET_PERMISSION_20260630
- Router recurring backup uses a dedicated routerbackup credential file on pve01.
- Secret file path: /var/lib/homelab-private/secrets/routerbackup.pass.
- File content must never be printed; only mode/owner/size may be checked.
- Current observed permission target: root-owned mode 600.
- Proof: 218_ROUTER_BACKUP_SECRET_PERMISSION_PROOF.txt.
## NEW_BACKUP_UNITS_SYSTEMD_INVENTORY_20260630
- New backup/restore systemd units and timers were inventoried after post-backup-pass closure.
- pve01 units checked: VM150 Mail-cloud upload and router running-config Mail-cloud backup.
- edge-vm units checked: P2 small-stacks backup/restore, backup dashboard and restore drill index.
- Proof records LoadState, ActiveState, UnitFileState and failed-unit counts for pve01 and edge-vm.
- Proof: 222_NEW_BACKUP_UNITS_SYSTEMD_INVENTORY_PROOF.txt.
## POST_BACKUP_PASS_PROM_HEALTH_COVERAGE_20260630
- Prometheus health coverage was checked for the post-backup-pass checks.
- Covered checks: VM150 Mail-cloud, router startup/running config, P2 small-stacks and seven restore dry-runs.
- Proof records health status and age from Prometheus without printing credential values.
- Proof: 224_POST_BACKUP_PASS_PROM_HEALTH_COVERAGE_PROOF.txt.
## POST_BACKUP_PASS_AUDIT_INDEX_20260630
- Audit proof manifest was generated for post-backup-pass evidence files 192-225.
- Manifest records file names, sha256 hashes, count and missing-number check.
- Proof: 226_POST_BACKUP_PASS_AUDIT_INDEX_PROOF.txt.
## POST_BACKUP_PASS_EXTENDED_FINAL_SNAPSHOT_20260630
- Extended post-backup-pass final snapshot was created after capacity, retention, unit inventory, Prometheus coverage and audit index checks.
- Snapshot copies current reference and assistant error register into the audit directory and records sha256 hashes.
- Covered proof range: 192-227.
- Proof: 228_POST_BACKUP_PASS_EXTENDED_FINAL_SNAPSHOT_PROOF.txt.
## POST_BACKUP_PASS_FINAL_AUDIT_INDEX_192_229_20260630
- Final audit manifest was generated after the extended final snapshot.
- Manifest covers proof numbers 192-229, including snapshot and reference-scan proofs.
- Manifest records file names, sha256 hashes, count and missing-number check.
- Proof: 230_POST_BACKUP_PASS_FINAL_AUDIT_INDEX_192_229_PROOF.txt.
## NEW_BACKUP_UNITS_AND_SCRIPTS_INTEGRITY_20260630
- New backup/restore unit files and executable script paths were inventoried and hashed.
- pve01 services checked: VM150 Mail-cloud upload and router running-config Mail-cloud backup.
- edge-vm services checked: P2 small-stacks backup/restore, backup dashboard and restore drill index.
- Proof records FragmentPath, ExecStart path, mode/owner/size and sha256 hashes without printing script contents.
- Proof: 232_NEW_BACKUP_UNITS_AND_SCRIPTS_INTEGRITY_PROOF.txt.
## NEW_BACKUP_UNITS_AND_SCRIPTS_INTEGRITY_CORRECTED_20260630
- Corrected integrity proof was generated because one edge-vm executable required sudo for sha256sum.
- Corrected proof records hashes for pve01 and edge-vm unit fragments and executable scripts without printing script contents.
- Proof: 234_NEW_BACKUP_UNITS_AND_SCRIPTS_INTEGRITY_CORRECTED_PROOF.txt.
## POST_BACKUP_PASS_FINAL_AUDIT_INDEX_192_235_20260630
- Final audit manifest was regenerated after corrected unit/script integrity proof.
- Manifest covers proof numbers 192-235, including corrected integrity proof 234/235.
- Manifest records file names, sha256 hashes, count and missing-number check.
- Proof: 236_POST_BACKUP_PASS_FINAL_AUDIT_INDEX_192_235_PROOF.txt.
## NEW_BACKUP_TIMERS_INTEGRITY_20260630
- New backup/restore timer unit files were inventoried and hashed.
- pve01 timers checked: VM150 Mail-cloud upload and router running-config Mail-cloud backup.
- edge-vm timers checked: P2 small-stacks backup/restore, backup dashboard and restore drill index.
- Proof records FragmentPath, mode/owner/size, sha256 hashes and next-elapse metadata without printing sensitive values.
- Proof: 238_NEW_BACKUP_TIMERS_INTEGRITY_PROOF.txt.
## POST_BACKUP_PASS_FINAL_AUDIT_INDEX_192_239_20260630
- Final audit manifest was regenerated after timer integrity proof.
- Manifest covers proof numbers 192-239 and records file names, sha256 hashes, count and missing-number check.
- Proof: 240_POST_BACKUP_PASS_FINAL_AUDIT_INDEX_192_239_PROOF.txt.
## POST_BACKUP_PASS_FINAL_SNAPSHOT_AFTER_TIMERS_20260630
- Final snapshot was created after timer integrity and final audit manifest 192-239.
- Snapshot copies current reference and assistant error register into the audit directory and records sha256 hashes.
- Covered proof range: 192-241.
- Proof: 242_POST_BACKUP_PASS_FINAL_SNAPSHOT_AFTER_TIMERS_PROOF.txt.
## ASSISTANT_COMMAND_BATCHING_RULE_20260630
- Operator preference: combine related homelab commands into fewer package-style runs to reduce slow repetitive manual work.
- Applies to proof generation, reference updates, validation scans, hash/index refreshes and snapshot steps when safe.
- Mandatory preflight and sensitive-output hygiene still take priority.
## POST_BACKUP_PASS_AUDIT_INDEX_AFTER_BATCHING_RULE_192_245_20260630
- Audit manifest was regenerated after assistant command batching rule was recorded.
- Manifest covers proof numbers 192-245 and records file names, sha256 hashes, count and missing-number check.
- Proof: 246_POST_BACKUP_PASS_AUDIT_INDEX_AFTER_BATCHING_RULE_192_245_PROOF.txt.
## POST_BACKUP_PASS_SNAPSHOT_AFTER_BATCHING_RULE_20260630
- Snapshot was created after assistant command batching rule and audit manifest 192-245.
- Snapshot copies current reference and assistant error register into the audit directory and records sha256 hashes.
- Covered proof range: 192-247.
- Proof: 248_POST_BACKUP_PASS_SNAPSHOT_AFTER_BATCHING_RULE_PROOF.txt.
## ALERTMANAGER_ROUTING_AND_CONFIG_20260630
- Alertmanager routing/config health was checked after post-backup-pass alert rules were installed.
- Proof records container presence, config validation, readiness endpoint and sanitized route/receiver metadata.
- Sensitive receiver values and URLs are intentionally not printed.
- Proof: 250_ALERTMANAGER_ROUTING_AND_CONFIG_PROOF.txt.
## AUDIT_INDEX_192_251_20260630
- Audit manifest was regenerated after Alertmanager routing/config proof.
- Manifest covers proof numbers 192-251 with count and missing-number check.
- Proof: 252_AUDIT_INDEX_192_251_PROOF.txt.
## RCLONE_CONFIG_PERMISSION_AND_REMOTE_INVENTORY_20260630
- rclone config permissions and crypt remote inventory were checked without printing config contents.
- Proof records config owner/mode/size, remote names and capacity summaries for active Mail-cloud crypt remotes.
- Proof: 254_RCLONE_CONFIG_PERMISSION_AND_REMOTE_INVENTORY_PROOF.txt.
## AUDIT_INDEX_192_255_20260630
- Audit manifest was regenerated after rclone config/remote inventory proof.
- Manifest covers proof numbers 192-255 with count and missing-number check.
- Proof: 256_AUDIT_INDEX_192_255_PROOF.txt.
## CURRENT_OPERATIONAL_ROLLUP_20260630
- Current operational rollup was captured after rclone config/remote inventory proof.
- Rollup records failed-unit counts, next timers, Mail-cloud capacity, Prometheus target count, post-backup health status/age, post-backup firing alerts and Alertmanager readiness.
- Proof: 258_CURRENT_OPERATIONAL_ROLLUP_PROOF.txt.
## AUDIT_INDEX_192_259_20260630
- Audit manifest was regenerated after current operational rollup.
- Manifest covers proof numbers 192-259 with count and missing-number check.
- Proof: 260_AUDIT_INDEX_192_259_PROOF.txt.
## SNAPSHOT_AFTER_CURRENT_OPERATIONAL_ROLLUP_20260630
- Snapshot was created after current operational rollup and audit index 192-259.
- Snapshot copies current reference and assistant error register into the audit directory and records sha256 hashes.
- Covered proof range: 192-261.
- Proof: 262_SNAPSHOT_AFTER_CURRENT_OPERATIONAL_ROLLUP_PROOF.txt.
## AUDIT_INDEX_192_263_20260630
- Audit manifest was regenerated after snapshot following current operational rollup.
- Manifest covers proof numbers 192-263 with count and missing-number check.
- Proof: 264_AUDIT_INDEX_192_263_PROOF.txt.
## NEW_BACKUP_SERVICE_RUNTIME_RESULT_20260630
- Runtime result metadata was captured for new backup/restore service units.
- pve01 services checked: VM150 Mail-cloud upload and router running-config Mail-cloud backup.
- edge-vm services checked: P2 small-stacks backup/restore, backup dashboard and restore drill index.
- Proof records LoadState, ActiveState, Result, ExecMainStatus and restart counters without printing sensitive values.
- Proof: 266_NEW_BACKUP_SERVICE_RUNTIME_RESULT_PROOF.txt.
## AUDIT_INDEX_192_267_20260630
- Audit manifest was regenerated after new backup service runtime-result proof.
- Manifest covers proof numbers 192-267 with count and missing-number check.
- Proof: 268_AUDIT_INDEX_192_267_PROOF.txt.
## NEW_BACKUP_UNITS_JOURNAL_ERROR_SCAN_20260630
- Journal warning/error counters were captured for new backup/restore units and timers over the last 24 hours.
- Proof records only counts, not journal message bodies, to avoid sensitive-output risk.
- Proof: 270_NEW_BACKUP_UNITS_JOURNAL_ERROR_SCAN_PROOF.txt.
## AUDIT_INDEX_192_271_20260630
- Audit manifest was regenerated after journal error scan proof.
- Manifest covers proof numbers 192-271 with count and missing-number check.
- Proof: 272_AUDIT_INDEX_192_271_PROOF.txt.
## NEW_BACKUP_UNITS_JOURNAL_ERROR_TRIAGE_20260630
- Journal warning/error triage was captured after warning/error counters were non-zero.
- Proof records per-unit warning/error counts and redacted journal fragments.
- Proof: 274_NEW_BACKUP_UNITS_JOURNAL_ERROR_TRIAGE_PROOF.txt.
## AUDIT_INDEX_192_275_20260630
- Audit manifest was regenerated after journal error triage proof.
- Manifest covers proof numbers 192-275 with count and missing-number check.
- Proof: 276_AUDIT_INDEX_192_275_PROOF.txt.
## VM150_MAIL_CLOUD_JOURNAL_NOISE_CLASSIFICATION_20260630
- VM150 Mail-cloud upload journal warnings/errors were classified after journal triage found non-zero counters.
- Current service result, Prometheus health and firing-alert status are used to decide whether journal noise is blocking.
- Proof includes redacted journal indicators only, not secrets.
- Proof: 278_VM150_MAIL_CLOUD_JOURNAL_NOISE_CLASSIFICATION_PROOF.txt.
## AUDIT_INDEX_192_279_20260630
- Audit manifest was regenerated after VM150 Mail-cloud journal-noise classification.
- Manifest covers proof numbers 192-279 with count and missing-number check.
- Proof: 280_AUDIT_INDEX_192_279_PROOF.txt.
## SNAPSHOT_AFTER_VM150_JOURNAL_NOISE_CLASSIFICATION_20260630
- Snapshot was created after VM150 Mail-cloud journal noise was classified as non-blocking historical noise.
- Snapshot copies current reference and assistant error register into the audit directory and records sha256 hashes.
- Covered proof range: 192-281.
- Proof: 282_SNAPSHOT_AFTER_VM150_JOURNAL_NOISE_CLASSIFICATION_PROOF.txt.
## AUDIT_INDEX_192_283_20260630
- Audit manifest was regenerated after snapshot following VM150 journal-noise classification.
- Manifest covers proof numbers 192-283 with count and missing-number check.
- Proof: 284_AUDIT_INDEX_192_283_PROOF.txt.
## POST_BACKUP_PASS_CLOSURE_SUMMARY_20260630
- Post-backup-pass closure summary was generated after VM150 journal-noise classification.
- Summary checks backup SLO closure, alert rules, firing-alert status, Prometheus health coverage, failed-unit state, Alertmanager readiness, rclone config hygiene, journal-noise classification and audit index completeness.
- Proof: 286_POST_BACKUP_PASS_CLOSURE_SUMMARY_PROOF.txt.
## SNAPSHOT_AFTER_POST_BACKUP_PASS_CLOSURE_SUMMARY_20260630
- Snapshot was created after post-backup-pass closure summary.
- Snapshot copies current reference and assistant error register into the audit directory and records sha256 hashes.
- Covered proof range: 192-287.
- Proof: 288_SNAPSHOT_AFTER_POST_BACKUP_PASS_CLOSURE_SUMMARY_PROOF.txt.
## AUDIT_INDEX_192_289_20260630
- Audit manifest was regenerated after post-backup-pass closure summary snapshot.
- Manifest covers proof numbers 192-289 with count and missing-number check.
- Proof: 290_AUDIT_INDEX_192_289_PROOF.txt.
## TOMORROW_FIRST_RUN_VERIFICATION_PLAN_20260630
- First scheduled-run verification should be done after 2026-07-01 08:15 MSK.
- Verify P2 small-stacks, router running-config Mail-cloud and VM150 Nextcloud Mail-cloud first automatic runs.
- Also verify service results, health status/age, Mail-cloud directories, dashboard, restore index, Prometheus firing alerts and Alertmanager readiness.
- Proof: 292_TOMORROW_FIRST_RUN_VERIFICATION_PLAN_PROOF.txt.
## AUDIT_INDEX_192_293_20260630
- Audit manifest was regenerated after tomorrow first-run verification plan.
- Manifest covers proof numbers 192-293 with count and missing-number check.
- Proof: 294_AUDIT_INDEX_192_293_PROOF.txt.
## END_OF_DAY_POST_BACKUP_PASS_SNAPSHOT_20260630
- End-of-day snapshot was created after post-backup-pass closure summary and tomorrow first-run verification plan.
- Snapshot copies current reference and assistant error register into the audit directory and records sha256 hashes.
- Covered proof range before this snapshot: 192-295.
- Proof: 296_END_OF_DAY_POST_BACKUP_PASS_SNAPSHOT_PROOF.txt.
## AUDIT_INDEX_192_297_20260630
- Final end-of-day audit manifest was generated after post-backup-pass snapshot.
- Manifest covers proof numbers 192-297 with count and missing-number check.
- Proof: 298_AUDIT_INDEX_192_297_PROOF.txt.
## HOME_PORTAL_DISCOVERY_20260630
- Home portal discovery started for https://home.gram1.ru/.
- Goal: make the portal show and open all web services, including external service cards such as NetBird and mail.
- Discovery proof records DNS, HTTPS/TLS headers, reverse-proxy candidates, compose files, domain references and homepage config candidates without printing secrets.
- Proof: 300_HOME_PORTAL_DISCOVERY_PROOF.txt.
## HOME_PORTAL_CONFIG_AND_SERVICE_INVENTORY_20260630
- Home portal config and service inventory was collected for https://home.gram1.ru/.
- Inventory records portal containers, config files, redacted card lines, compose web hints, domain references and common subdomain probes.
- Proof: 302_HOME_PORTAL_CONFIG_AND_SERVICE_INVENTORY_PROOF.txt.
## HOME_PORTAL_GAP_ANALYSIS_20260630
- Home portal gap analysis was collected for gethomepage/homepage behind npmplus.
- Proof records Homepage config hashes, redacted current cards/bookmarks, current URLs, npmplus route lines, local web-port probes and known service container candidates.
- Proof: 304_HOME_PORTAL_GAP_ANALYSIS_PROOF.txt.
## HOME_PORTAL_AUDIT_INDEX_300_305_20260630
- Home portal audit manifest was generated for proof numbers 300-305.
- Proof: 306_HOME_PORTAL_AUDIT_INDEX_300_305_PROOF.txt.
## HOME_PORTAL_CARD_AND_API_ERROR_ANALYSIS_20260630
- Home portal card/API-error analysis was collected before editing Homepage.
- Proof records current card URLs/titles, redacted widget config, redacted Homepage log errors, NPMPlus internal route files/routes and candidate public/VPN cards.
- Proof: 308_HOME_PORTAL_CARD_AND_API_ERROR_ANALYSIS_PROOF.txt.
## HOME_PORTAL_AUDIT_INDEX_300_309_20260630
- Home portal audit manifest was regenerated after card/API-error analysis.
- Manifest covers proof numbers 300-309 with count and missing-number check.
- Proof: 310_HOME_PORTAL_AUDIT_INDEX_300_309_PROOF.txt.
## HOME_PORTAL_DEDUP_AND_EXTERNAL_CARDS_APPLY_20260630
- Homepage services.yaml was backed up, duplicate VPN cards were removed where a non-VPN card existed, and missing web cards for NetBird, Mail and Webmail were added without VPN suffix in the card names.
- SMTP/IMAP were intentionally not added as cards because they are protocol endpoints, not web UI cards.
- Homepage container was restarted and portal/card URLs were checked.
- Proof: 312_HOME_PORTAL_DEDUP_AND_EXTERNAL_CARDS_APPLY_PROOF.txt.
## HOME_PORTAL_AUDIT_INDEX_300_313_20260630
- Home portal audit manifest was regenerated after duplicate-card cleanup and external-card addition.
- Manifest covers proof numbers 300-313 with count and missing-number check.
- Proof: 314_HOME_PORTAL_AUDIT_INDEX_300_313_PROOF.txt.
## HOME_PORTAL_DEDUP_AND_EXTERNAL_CARDS_CORRECTED_APPLY_20260630
- Corrected Homepage apply removed duplicate VPN cards where a non-VPN card existed and added NetBird, Mail and Webmail cards with non-VPN names.
- The added card URLs use currently reachable vpn.gram1.ru web endpoints because public netbird/mail/webmail hosts did not resolve/open during analysis.
- SMTP/IMAP were not added as cards because they are protocol endpoints, not web UIs.
- Proof: 316_HOME_PORTAL_DEDUP_AND_EXTERNAL_CARDS_CORRECTED_APPLY_PROOF.txt.
## HOME_PORTAL_AUDIT_INDEX_300_317_20260630
- Home portal audit manifest was regenerated after corrected duplicate-card cleanup and external-card addition.
- Manifest covers proof numbers 300-317 with count and missing-number check.
- Proof: 318_HOME_PORTAL_AUDIT_INDEX_300_317_PROOF.txt.
## HOMELAB_COMMAND_SAFETY_HARDENING_20260630
- Command safety rule strengthened after the home portal base64 apply failure.
- Opaque generated payloads are not acceptable for homelab changes; commands must be transparent, bounded, and validated before modification.
- Success requires content-specific checks in addition to service/runtime checks.
- Proof: 320_HOMELAB_COMMAND_SAFETY_HARDENING_RULE_PROOF.txt.
## HOME_PORTAL_AUDIT_INDEX_300_321_20260630
- Home portal audit manifest was regenerated after command-safety hardening rule.
- Manifest covers proof numbers 300-321 with count and missing-number check.
- Proof: 322_HOME_PORTAL_AUDIT_INDEX_300_321_PROOF.txt.
## HOME_PORTAL_POST_APPLY_VALIDATION_AND_API_TRIAGE_20260630
- Home portal was validated after duplicate cleanup and external-card addition.
- Proof checks portal HTTP status, Homepage container state, required cards, duplicate VPN card pairs, current card URLs, redacted API-error logs and widget configuration.
- Proof: 324_HOME_PORTAL_POST_APPLY_VALIDATION_AND_API_TRIAGE_PROOF.txt.
## HOME_PORTAL_AUDIT_INDEX_300_325_20260630
- Home portal audit manifest was regenerated after post-apply validation and API triage.
- Manifest covers proof numbers 300-325 with count and missing-number check.
- Proof: 326_HOME_PORTAL_AUDIT_INDEX_300_325_PROOF.txt.
## HOME_PORTAL_API_ERROR_ROOT_CAUSE_20260630
- Homepage API error root-cause analysis was collected after the UI showed API errors.
- Current evidence points to the Open-Meteo/weather widget timing out from Homepage, separate from service cards.
- HTTP 200 for the portal is not sufficient for UI closure when Homepage logs still contain API errors.
- Proof: 328_HOME_PORTAL_API_ERROR_ROOT_CAUSE_PROOF.txt.
## HOME_PORTAL_AUDIT_INDEX_300_329_20260630
- Home portal audit manifest was regenerated after API-error root-cause analysis.
- Manifest covers proof numbers 300-329 with count and missing-number check.
- Proof: 330_HOME_PORTAL_AUDIT_INDEX_300_329_PROOF.txt.
## HOME_PORTAL_DISABLE_OPENMETEO_WIDGET_20260630
- Open-Meteo/weather widget was disabled after root-cause analysis showed Homepage API errors from api.open-meteo.com timeouts.
- This keeps the portal focused on service-launcher cards and prevents external weather API failures from causing UI API errors.
- Portal closure now requires current Homepage API-error logs to be zero after restart/reload.
- Proof: 332_HOME_PORTAL_DISABLE_OPENMETEO_WIDGET_PROOF.txt.
## HOME_PORTAL_AUDIT_INDEX_300_333_20260630
- Home portal audit manifest was regenerated after disabling Open-Meteo/weather widget.
- Manifest covers proof numbers 300-333 with count and missing-number check.
- Proof: 334_HOME_PORTAL_AUDIT_INDEX_300_333_PROOF.txt.
## HOME_PORTAL_LINK_OPEN_AUDIT_20260630
- Home portal card links were audited after Open-Meteo/weather widget was disabled.
- Audit extracts current Homepage href URLs and checks whether each opens with an acceptable HTTP status.
- Acceptable statuses include success, redirects and auth-gated 401/403 pages because those still indicate a reachable web service.
- Proof: 336_HOME_PORTAL_LINK_OPEN_AUDIT_PROOF.txt.
## HOME_PORTAL_AUDIT_INDEX_300_337_20260630
- Home portal audit manifest was regenerated after link-open audit.
- Manifest covers proof numbers 300-337 with count and missing-number check.
- Proof: 338_HOME_PORTAL_AUDIT_INDEX_300_337_PROOF.txt.
## HOME_PORTAL_EXPLICIT_VPN_DUPLICATE_CLEANUP_CORRECTED_20260630
- Explicit VPN duplicate cards were removed from Homepage with a short transparent perl edit: Homepage VPN, Authentik VPN, Gitea VPN and Uptime Kuma VPN.
- Old router URL [PRIVATE_IP] was replaced with [PRIVATE_IP] where present.
- Proof: 342_HOME_PORTAL_EXPLICIT_VPN_DUPLICATE_CLEANUP_CORRECTED_PROOF.txt.
## HOME_PORTAL_REMAINING_DUPLICATE_OCCURRENCE_ANALYSIS_20260630
- Remaining Homepage duplicate/VPN/router occurrences were inspected after corrected cleanup still reported REVIEW.
- Proof records exact redacted file/line occurrences before another edit.
- Proof: 344_HOME_PORTAL_REMAINING_DUPLICATE_OCCURRENCE_ANALYSIS_PROOF.txt.
## HOME_PORTAL_ACTIVE_CONFIG_SCOPE_RECHECK_20260630
- Homepage duplicate cleanup was rechecked against active config files only, excluding logs and backup files.
- Previous broad grep included historical logs/backups and therefore overstated remaining active duplicates.
- Proof: 346_HOME_PORTAL_ACTIVE_CONFIG_SCOPE_RECHECK_PROOF.txt.
## HOME_PORTAL_ACTIVE_VPN_DUPLICATES_FINAL_CLEANUP_20260630
- Active Homepage services/bookmarks files were cleaned from semantic VPN duplicate cards and old router URL references.
- Removed targets: Homepage VPN, Authentik VPN, Gitea VPN and Uptime Kuma VPN, plus matching vpn.gram1.ru duplicate URLs.
- Proof: 350_HOME_PORTAL_ACTIVE_VPN_DUPLICATES_FINAL_CLEANUP_PROOF.txt.
## HOME_PORTAL_NPMPLUS_DEFAULT_ROUTE_ANALYSIS_20260630
- Home portal NetBird/Mail/Webmail cards were investigated after user saw the NPMPlus default page from client [PRIVATE_IP].
- Analysis checks URL body fingerprints, NPMPlus route config matches, service container candidates and Homepage card lines.
- Proof: 356_HOME_PORTAL_NPMPLUS_DEFAULT_ROUTE_ANALYSIS_PROOF.txt.
## HOME_PORTAL_CURRENT_STATE_20260630
- Active Homepage URL: https://home.gram1.ru.
- Homepage container is running and portal HTTP check returned 200 after latest changes.
- NetBird card points to https://nb.pvepro.ru.
- Mail card points to https://mail.pvepro.ru.
- Webmail card was removed; no separate Webmail card is currently configured.
- Router bookmark points to http://[PRIVATE_IP]:5080 for operator client LAN [PRIVATE_IP].
- Homepage services.yaml was updated via YAML-aware edit for siteMonitor fields.
- Current active services.yaml has SITEMONITOR_COUNT=43.
- Latest validation before this reference update: YAML_ERRORS_LAST_5M=0, HOME_HTTP_CODE=200, HOMEPAGE_RUNNING=true.
- Do not use sed-only insertion for siteMonitor; use YAML-aware editing and validate YAML before restart.
- Related active-state checks: NB_LINK_ACTIVE=yes, MAIL_LINK_ACTIVE=yes, WEBMAIL_ACTIVE=no, ROUTER_192_168_1_1_ACTIVE=yes.
## HOME_PORTAL_DIRECT_LINK_FIX_20260630
- Direct Homepage link correction requested by operator.
- NetBird: https://nb.pvepro.ru.
- Mail: https://mail.pvepro.ru.
- Webmail card removed.
- Router restored to http://[PRIVATE_IP]:5080.
## HOME_PORTAL_YAML_AWARE_SITEMONITOR_20260630
- siteMonitor fields were added using YAML-aware PyYAML editing, not line-based sed insertion.
- Added siteMonitor to 43 service cards.
- Excluded cards: Homepage, NPMplus, Router and Public Domain.
- YAML validation passed before restart: YAML_VALIDATE_RC=0.
- Homepage restarted successfully and reported YAML_ERRORS=0.
## HOMELAB_CLUSTER_BACKLOG_20260630
- Cluster backlog was created at /etc/pve/HOMELAB_CLUSTER_BACKLOG.md.
- PBS is intentionally out of scope; backup strategy remains Mail-cloud based.
- First next action: finish Homepage final validation.
## HOMEPAGE_BACKUP_GAP_CLASSIFICATION_20260630
- Homepage backup coverage gap classification file created at /etc/pve/HOMEPAGE_BACKUP_GAP_CLASSIFICATION.md.
- It defines closure requirements for the 13 remaining Homepage backup coverage gap/special-review rows.
- PBS is intentionally out of scope; closure must use Mail-cloud backups or explicit stateless/config-only classification.
## HOMEPAGE_EXISTING_BACKUP_EVIDENCE_CLOSURE_20260630
- Homepage backup coverage matrix started closing existing-evidence rows.
- Router maps to existing Mail-cloud startup/running-config backup proofs when ROUTER_EXISTING_EVIDENCE=yes.
- MinIO Console maps to existing P2 small-stacks backup/restore proof when MINIO_EXISTING_EVIDENCE=yes.
- Proof: 364_HOMEPAGE_EXISTING_BACKUP_EVIDENCE_CLOSURE_PROOF.txt.
## HOME_EXTERNAL_CLOUDFLARE_ALEXHOST_20260630
- Homepage External group includes Cloudflare card: https://dash.cloudflare.com.
- Homepage External group includes AlexHost Billing card: https://bill.alexhost.com/.
- Both cards have siteMonitor enabled for green status dots.
- YAML validation passed, Homepage returned HTTP 200, container running, and no Homepage YAML errors were seen after restart.
- Proof: 368_HOME_EXTERNAL_CLOUDFLARE_ALEXHOST_PROOF.txt.
## HOME_EXTERNAL_RELAY_REMOVED_20260630
- Homepage External card/link for relay.pvepro.ru was removed because relay.pvepro.ru had DNS but no reachable HTTP/HTTPS endpoint from edge-vm.
- Proof: 382_HOME_EXTERNAL_RELAY_REMOVE_PROOF.txt.
## ROUTER_CLI_PERMISSION_LIMIT_20260630
- Netcraze routerbackup SSH access is read-only for backup: show running-config works, but ACL/config commands are denied.
- Homepage Moscow Router red-dot fix requires router ACL/Web UI/admin-level change, not routerbackup.
- Relevant proofs: 391, 393, 394, 395.
## ROUTER_CLI_PROOF_REPAIR_20260630
- Error register updated for blank proof summary extraction in 395.
- Corrected derived counts are recorded in 396_ROUTER_CLI_ERROR_REGISTER_REPAIR_PROOF.txt.
## HOME_EXTERNAL_NETBIRD_ALEXHOST_NORMALIZE_20260630
- Homepage External NetBird and AlexHost Billing href/siteMonitor URLs normalized with trailing slash.
- Homepage container node checks returned HTTP 200 for both URLs.
- Proof: 397_HOME_EXTERNAL_NETBIRD_ALEXHOST_NORMALIZE_PROOF.txt.
## ROUTER_HTTP_SERVICE_FORBIDDEN_FROM_EDGE_VM_20260701
- Moscow Router ACL is restored and correct after manual Web UI edits.
- Remaining blocker for Homepage green dot is HTTP/HTTPS 403 from router web service to edge-vm/Bridge1, not network reachability.
- Relevant proofs: 399, 400, 401, 402.
## HOME_MOSCOW_ROUTER_HEALTH_ENDPOINT_20260701
- Moscow Router Homepage card href: http://[PRIVATE_IP]:5080.
- Moscow Router Homepage siteMonitor: http://[PRIVATE_IP]:18083/router-moscow.
- Health endpoint service on edge-vm: homelab-router-moscow-health-http.service.
- Endpoint semantics: returns 200 when TCP to router [PRIVATE_IP]:5080 succeeds, else 503.
- Relevant proofs: 405, 406, 407.
## HOME_MOSCOW_ROUTER_HEALTH_ENDPOINT_FIXED_20260701
- Moscow Router Homepage card target after fixed apply: href http://[PRIVATE_IP]:5080, siteMonitor http://[PRIVATE_IP]:18083/router-moscow.
- Fixed updater handles Homepage service-name-key YAML shape and property-style YAML shape.
- Relevant proofs: 405, 406, 407, 408.
## HOME_MOSCOW_ROUTER_HEALTH_ENDPOINT_APPLIED_EXACT_20260701
- Moscow Router Homepage card exact active YAML shape was service-key style.
- Applied href: http://[PRIVATE_IP]:5080.
- Applied siteMonitor: http://[PRIVATE_IP]:18083/router-moscow.
- Health endpoint service: homelab-router-moscow-health-http.service on edge-vm.
- Relevant proofs: 406, 409, 410.
## FORUM_PROD_VM160_REBUILD_BASELINE_20260701
- Old test VM160 was stopped and destroyed cleanly: proof 419, CONFIG_EXISTS=no, VM160_DISKS_LEFT=0.
- New VM160 forum-prod was recreated on pve02 with 4 vCPU, 8 GiB RAM, 200G disk, MAC BC:24:11:B6:45:ED, IP [PRIVATE_IP]/24, gateway [PRIVATE_IP].
- New VM identity confirmed: forum-prod / forum-prod.gram1.ru, Debian 12 bookworm, SSH OK, qemu-agent OK, chrony OK.
- Swapfile 8G enabled with /swapfile mode 600 and fstab entry: proof 430. Proof 429 is invalid due nested command substitution quoting.
- SURY PHP repo added; PHP 8.5.7 available and installed with Nginx 1.22.1, MariaDB 10.11.14 and required XenForo modules.
- Beget-compatible profile applied: memory_limit 256M, post/upload 1024M, max_input_vars 10000, MariaDB utf8mb4/utf8mb4_unicode_ci, innodb_buffer_pool_size 2G.
- Operator reference copies placed in /root/homelab-operator inside VM160 for local VM command preflight.
## FORUM_PROD_CODEVIPE_FIRST_RECIPE_STRATEGY_20260701
- Current VM160 is now laboratory state, not final production closure.
- Bulk import of five forums reached DB/files/nginx/php-fpm ready state, but frontend body stayed empty under PHP 8.5.7.
- Strategy changed: prove CodeVipe first on controlled runtime, likely PHP 8.3 or PHP 8.2, then destroy VM160 and rebuild all five forums from scratch using the proven recipe.
- Do not publish Cloudflare/NPMplus routes for these forums until CodeVipe recipe is proven and final rebuild is complete.
- Proofs: failed/diagnostic chain 451-480; next snapshot proof 481 before further lab work.
## FORUM_PROD_FRESH5_DEPLOYED_OK_20260701
- VM160 forum-prod on pve02, IP [PRIVATE_IP].
- Runtime: Debian 12, Nginx, MariaDB 10.11, PHP 8.3-FPM.
- Five XenForo forums deployed from fresh ZIP backups: codevipe.ru, gamevipe.ru, hkmods.ru, zakrutim.ru, dsmods.ru.
- Webroots: /var/www/forums/{codevipe,gamevipe,hkmods,zakrutim,dsmods}/public.
- DBs: xf_codevipe, xf_gamevipe, xf_hkmods, xf_zakrutim, xf_dsmods.
- Final health proof inside VM: /root/evidence/513_FORUM_PROD_FRESH5_FINAL_HEALTH_PROOF.txt.
- Final result: all five frontend/admin HTTP 200, www redirects 301, internal_data 403, no new XenForo errors.
- Lesson: fresh file ZIPs contain populated internal_data/code_cache; broken branch lacked compiled template cache.
## OPERATOR_RULE_CONTINUE_WITH_NEXT_COMMAND_20260701
- Rule: when an infrastructure step ends at an expected prompt or safe checkpoint, do not stop with only a status summary.
- Rule: immediately provide the next executable command in the same response.
- Applies to: SSH enter/exit points, VM prompt confirmations, snapshot confirmations, file copy confirmations, successful health checks, and other obvious next-step transitions.
- Exception: stop only if the previous output contains an error, ambiguity, dangerous state, missing file, secret exposure risk, or a required user decision.
## OPERATOR_RULE_BATCH_2_3_COMMANDS_20260701
- Rule: when safe, provide 2-3 sequential executable commands in one response instead of stopping after each expected checkpoint.
- Applies after: successful prompt transitions, clean health checks, completed copies, completed snapshots, and non-destructive inventory.
- Do not batch commands across: destructive operations, public exposure, DNS/SSL changes, secret handling, unclear errors, missing files, or required user decisions.
- Each command must still print ERROR_REGISTER_CHECK=OK and REFERENCE_CHECK=OK before work.
## FORUM_PROD_FRESH5_PUBLIC_CUTOVER_OK_20260701
- Public cutover completed for codevipe.ru, gamevipe.ru, hkmods.ru, zakrutim.ru, dsmods.ru.
- Cloudflare A records for main and www names point to edge public IP 95.84.154.183.
- Edge NPMplus manual routes terminate TLS with real Let’s Encrypt certificates and proxy to forum-prod VM160 at [PRIVATE_IP]:80.
- Final public proof: /root/evidence/546_PUBLIC_FORUMS_FINAL_DNS_HTTPS_PROOF.txt on pve01.
- Expected public state: main domains HTTPS 200 with ssl_verify_result 0; HTTP redirects to HTTPS; www redirects to main domain through forum-prod canonical vhosts.
## EDGE_FORUM_PUBLICATION_BACKUP_20260701
- Edge NPMplus forum publication backup created after public cutover.
- Backup location: /opt/npmplus/manual-backups/forum-public-ok-*.tar.gz on edge-vm [PRIVATE_IP].
- Includes manual proxy_host configs 200-204 and Let’s Encrypt forum certificates under /opt/npmplus/tls/forum-certs.
## FORUM_CERT_RENEWAL_CONFIGURED_20260701
- Edge certificate renewal configured on edge-vm [PRIVATE_IP].
- Token stored at /opt/npmplus/secure/forum_cf_token.env mode 600 root-only.
- Renewal script: /opt/npmplus/scripts/renew-forum-certs.sh.
- Cron: /etc/cron.d/forum-cert-renew, daily 03:17.
- Proof: /root/evidence/550_EDGE_FORUM_CERT_RENEWAL_SETUP_PROOF.txt on pve01.
## CLOUDFLARE_FORUM_DNS_AUDIT_OK_20260701
- Full Cloudflare DNS audit completed for codevipe.ru, gamevipe.ru, hkmods.ru, zakrutim.ru, dsmods.ru.
- Proof: /root/evidence/554_CLOUDFLARE_FULL_DNS_AUDIT_PROOF.txt on pve01.
- Result: root and www A records for all five zones point to 95.84.154.183.
- Result: no root/www AAAA records, no root/www CNAME conflicts, no records with old IP 87.236.18.45.
- Remaining cdn.* CNAME and TXT/DMARC/DKIM records were observed and not changed.
## FORUMS_FINAL_DNS_AUDIT_SNAPSHOT_OK_20260701
- Final post-DNS-audit snapshot completed after shortening Proxmox snapshot name.
- Snapshot name: forum-dns-ok-065203Z
- Proof: /root/evidence/557_FINAL_SHORT_SNAPSHOT_AFTER_DNS_AUDIT_PROOF.txt on pve01.
- Current public state: all five main domains resolve to 95.84.154.183 and return HTTPS 200 with valid certificates.
## FORUM_LOCAL_BACKUP_CONFIGURED_20260701
- Local forum backup configured inside VM160 forum-prod.
- Script: /root/scripts/forum-backup.sh.
- Cron: /etc/cron.d/forum-local-backup, daily 02:42.
- Retention: removes per-run directories older than 14 days; compressed archives remain under /var/backups/forums.
- Proof: /root/evidence/560_FORUM_BACKUP_SCRIPT_SETUP_PROOF.txt on pve01.
## FORUM_MAIL_NOREPLY_AND_SPF_CLEANUP_20260701
- XenForo email fields defaultEmailAddress/contactEmailAddress/bounceEmailAddress/emailSenderName set to noreply@pvepro.ru for all five forums.
- SPF TXT on codevipe.ru, gamevipe.ru, hkmods.ru, zakrutim.ru, dsmods.ru changed from stale 87.236.18.45 to v=spf1 -all.
- DKIM, DMARC and cdn.* CNAME records were not changed.
- Proofs: 564_UPDATE_XENFORO_EMAIL_NOREPLY_PROOF.txt, 565_CF_UPDATE_FORUM_SPF_NOREPLY_PROOF.txt, 566_FORUM_MAIL_NOREPLY_FINAL_AUDIT_PROOF.txt.
## FORUM_CDN_RECORDS_DELETED_20260701
- Deleted cdn.* CNAME records for codevipe.ru, gamevipe.ru, hkmods.ru, zakrutim.ru, dsmods.ru.
- Reason: old cdn.* records pointed to previous provider/trbcdn and no current public/local forum references were observed.
- Root and www A records remain on 95.84.154.183.
- Proofs: /root/evidence/567_CF_DELETE_FORUM_CDN_RECORDS_PROOF.txt and /root/evidence/568_CDN_DELETE_FINAL_AUDIT_REFERENCE_PROOF.txt on pve01.
## FORUM_SPF_DEDUPED_AND_CDN_DELETED_20260701
- Fixed duplicate SPF state caused by earlier failed cleanup attempt.
- Each of codevipe.ru, gamevipe.ru, hkmods.ru, zakrutim.ru, dsmods.ru now has exactly one SPF TXT: v=spf1 -all.
- Old SPF references to 87.236.18.* are absent.
- cdn.* CNAME records are absent.
- Public HTTPS remained healthy for all five forums.
- Proofs: 569_CF_FIX_DUPLICATE_SPF_RECORDS_PROOF.txt and 570_MAIL_DNS_FINAL_CLEAN_AUDIT_PROOF.txt.
## FORUM_PUBLIC_DNS_MAIL_CLEAN_FINAL_20260701
- Public web and mail-related DNS cleanup completed for five forum domains.
- @ and www A records point to 95.84.154.183.
- cdn.* CNAME records removed.
- SPF deduped to exactly one TXT: v=spf1 -all on each forum domain.
- Old provider IP 87.236.18.* absent from forum domain TXT records.
- XenForo visible email identity changed to noreply@pvepro.ru on all forums.
- Proofs: 570_MAIL_DNS_FINAL_CLEAN_AUDIT_PROOF.txt, 571_FORUM_MAIL_TRANSPORT_READINESS_PROOF.txt, 572_FORUM_PUBLIC_AND_MAIL_DNS_FINAL_SUMMARY_PROOF.txt.
## FORUM_SMTP_TRANSPORT_PAUSED_20260701
- Forum XenForo email identity is noreply@pvepro.ru, but actual SMTP transport is paused.
- Reason: failed SMTP authentication triggered Mailcow netfilter ban for 95.84.154.183.
- Broken msmtp secret/config files were removed from forum-prod.
- Do not re-enable SMTP until the mailbox/app password is rotated and tested with a single controlled attempt.
- Proof: /root/evidence/590_FORUM_SMTP_FAILED_CONFIG_DISABLED_PROOF.txt.
## POST_INCIDENT_STABLE_STATE_20260701
- Incident after failed forum SMTP testing resolved.
- Mailcow and NetBird are reachable again: mail.pvepro.ru/admin and nb.pvepro.ru return HTTP 200.
- Mailcow netfilter ban for 95.84.154.183 was removed on VPS mail.pvepro.ru.
- Forum sites remain healthy over HTTPS.
- Forum SMTP transport remains paused; broken msmtp configs/secrets removed from forum-prod.
- Do not retry SMTP until the mailbox/app password is rotated.
- Proofs: 589_POST_UNBAN_MAILCOW_NETBIRD_VERIFY.txt on VPS, 590_FORUM_SMTP_FAILED_CONFIG_DISABLED_PROOF.txt, 591_POST_INCIDENT_PUBLIC_STATUS_PROOF.txt, 592_POST_INCIDENT_FINAL_STABLE_STATE_PROOF.txt.
## FORUM_SMTP_ONE_SHOT_TEST_OK_20260701
- One controlled SMTP test from forum-prod to mail.pvepro.ru:587 succeeded using rotated credentials.
- No persistent forum SMTP config was enabled.
- One-shot secret/config files were removed after the test.
- Mailcow and NetBird remained reachable after the test.
- Proofs: /root/evidence/594_FORUM_SMTP_ONE_SHOT_TEST_PROOF.txt and /root/evidence/595_SMTP_TEST_AFTERCHECK_PROOF.txt.
## SMTP_ONESHOT_OK_STABLE_FINAL_20260701
- Rotated SMTP credential was tested once from forum-prod to mail.pvepro.ru:587 successfully.
- No persistent SMTP config or secret was left on forum-prod after the one-shot test.
- Mailcow and NetBird remained reachable after the test.
- Forum websites remained healthy.
- Persistent forum SMTP is still not enabled; enable it only with an explicit command and hidden password input.
- Proofs: 594_FORUM_SMTP_ONE_SHOT_TEST_PROOF.txt, 595_SMTP_TEST_AFTERCHECK_PROOF.txt, 596_SMTP_ONESHOT_OK_FINAL_STABLE_PROOF.txt.
## FORUM_PERSISTENT_MSMTP_ENABLED_20260701
- Persistent msmtp-mta transport enabled on forum-prod as /usr/sbin/sendmail.
- SMTP relay: mail.pvepro.ru:587 STARTTLS, sender noreply@pvepro.ru.
- Secret stored only as root:www-data 0640 base64 file under /etc/msmtp/pvepro.env; no raw SMTP_PASS file remains.
- PHP mail() as www-data succeeded after persistent config installation.
- Mailcow and NetBird remained reachable after enabling persistent forum SMTP.
- Proofs: /root/evidence/597_ENABLE_FORUM_PERSISTENT_MSMTP_PROOF.txt and /root/evidence/598_PERSISTENT_MSMTP_FINAL_VERIFY_PROOF.txt.
## FORUM_PERSISTENT_SMTP_FINAL_STABLE_20260701
- Persistent forum SMTP via msmtp is enabled on forum-prod.
- PHP mail() as www-data succeeded after installation.
- Forum domains remain healthy over HTTPS.
- Mailcow and NetBird remain reachable after enabling persistent forum SMTP.
- Secret storage is root:www-data 0640 under /etc/msmtp/pvepro.env; raw [SENSITIVE_PATH] SMTP_PASS file is absent.
- Snapshot after enable: forum-smtp-on-080840Z.
- Proofs: 597_ENABLE_FORUM_PERSISTENT_MSMTP_PROOF.txt, 598_PERSISTENT_MSMTP_FINAL_VERIFY_PROOF.txt, 600_FORUM_PERSISTENT_SMTP_FINAL_STABLE_PROOF.txt.
## FORUM_PROD_FRESH5_MAILRU_CLOUD_BACKUP_OK_20260701
- Existing Mail.ru Cloud rclone crypt remotes on pve02 were reused; no new cloud account was configured.
- Old codevipe-cloud-backup.timer was stale after fresh five-forum rebuild and is disabled.
- New pve02 timer forum-prod-fresh5-cloud-backup.timer uploads current forum-prod local archives from VM160 /var/backups/forums.
- Current backup contains all five forums: codevipe, gamevipe, hkmods, zakrutim, dsmods.
- Upload target: pve02-mail-01/02/03/04-crypt under app/forums/forum-prod/fresh5/<timestamp>.
- Archive is split into 1500M parts before upload, with SHA256SUMS and MANIFEST.txt.
- First fresh5 cloud upload proof: /root/evidence/608_FORUM_PROD_FRESH5_CLOUD_BACKUP_FIRST_RUN_PROOF.txt.
## FORUM_PROD_FRESH5_MAILRU_CLOUD_RESTORE_DRILL_OK_20260701
- Restore drill from Mail.ru Cloud crypt remote completed for forum-prod fresh5 backup.
- Remote source used: pve02-mail-01-crypt app/forums/forum-prod/fresh5 latest timestamp.
- Downloaded split parts, MANIFEST.txt, SHA256SUMS and SHA256SUMS.local were verified.
- Part SHA256 verification passed.
- Split archive was reconstructed and full archive hash matched SHA256SUMS.local.
- tar listing confirmed DB dumps and file archives for codevipe, gamevipe, hkmods, zakrutim and dsmods.
- Proof: /root/evidence/612_FORUM_FRESH5_CLOUD_RESTORE_DRILL_CORRECTED_PROOF.txt.
## FORUM_POSTLAUNCH_BACKUP_MAIL_FINAL_ROLLUP_OK_20260701
- Five public XenForo forums are healthy over HTTPS.
- Mailcow admin and NetBird UI are reachable after persistent forum SMTP enablement.
- Persistent forum SMTP via msmtp is enabled and tested.
- Local backup exists on VM160 under /var/backups/forums.
- Mail.ru Cloud fresh5 backup is enabled on pve02 via forum-prod-fresh5-cloud-backup.timer.
- Old CodeVipe-only cloud timer is disabled.
- Cloud restore drill passed: split parts verified, archive reconstructed, SHA256 matched, tar contained DB/file archives for all five forums.
- Final rollup proof: /root/evidence/614_FORUM_POSTLAUNCH_BACKUP_MAIL_FINAL_ROLLUP_PROOF.txt.
## FORUM_XENFORO_JOB_RUNNER_CONFIGURED_20260701
- XenForo CLI job runner configured inside VM160 forum-prod.
- Script: /root/scripts/forum-xenforo-run-jobs.sh.
- Cron: /etc/cron.d/forum-xenforo-run-jobs every 5 minutes.
- Purpose: process XenForo job queues and cron tasks independent of forum traffic.
- First run proof: /root/evidence/617_XENFORO_RUN_JOBS_ONCE_PROOF.txt.
- Cron verification proof: /root/evidence/618_XENFORO_JOB_RUNNER_CRON_CONFIG_PROOF.txt.
## FORUM_XENFORO_CRON_DAEMON_AND_JOBRUNNER_OK_20260701
- cron daemon inside VM160 forum-prod is active/enabled and was proven by temporary /etc/cron.d execution proof.
- XenForo job runner script executed successfully after cron daemon verification.
- Due XenForo cron count returned to zero after run.
- Proof: /root/evidence/621_FORUM_CRON_DAEMON_JOBRUNNER_VERIFY_PROOF.txt.
## FORUM_DBTECH_TOR_TIMEOUT_CLASSIFIED_20260701
- Each forum had one historical XenForo error_log row for cURL timeout to check.torproject.org/torbulkexitlist.
- Source addon: DBTech/Security DragonByte Security 5.0.0.
- dbtech_security_tornodes option is 0 on all five forums.
- Classified as non-blocking historical external network timeout/noise; not a forum runtime failure.
- Error rows were not deleted.
- Proof: /root/evidence/622_TOR_TIMEOUT_CLASSIFICATION_FINAL_PROOF.txt.
## FORUM_XENFORO_BUILTIN_MAIL_DELIVERED_BUT_SPAM_20260701
- Built-in XenForo outgoing email test from site "Моды для Hollow Knight" was delivered to aleisaev@yandex.ru.
- Sender was noreply@pvepro.ru.
- Yandex placed the message in Spam with warning that links/images were disabled.
- This proves forum SMTP transport works, but deliverability/reputation needs tuning.
- Custom proof 623 is invalid due /root path permission issue and is superseded by this user-observed built-in test.
- Next evidence needed: Yandex raw message headers, especially Authentication-Results and DKIM-Signature.
- Proof: /root/evidence/627_XENFORO_BUILTIN_MAIL_DELIVERED_SPAM_CLASSIFICATION_PROOF.txt.
## FORUM_YANDEX_DELIVERABILITY_SPAM_WITH_SPF_DKIM_PASS_20260701
- Built-in XenForo outgoing email test was delivered to aleisaev@yandex.ru but placed in Spam.
- User-provided Yandex headers showed SPF pass and DKIM pass for pvepro.ru.
- Return-Path, From and DKIM domain aligned on pvepro.ru.
- Yandex spam score observed: X-Yandex-Spam: 4.
- Classification: SMTP/Mailcow/XenForo transport is OK; remaining issue is deliverability/reputation/content filtering.
- DNS change is not required based on this header proof.
- Next actions: mark as Not spam in Yandex, add/check pvepro.ru in Yandex Postmaster, warm up sender reputation, optionally plan per-forum sender domains later.
## FORUM_PROJECT_FINAL_ACCEPTANCE_OK_20260701
- Five XenForo forums are published and healthy over HTTPS: codevipe.ru, gamevipe.ru, hkmods.ru, zakrutim.ru, dsmods.ru.
- VM160 forum-prod is the active production forum VM on pve02 at [PRIVATE_IP].
- Runtime services are configured: nginx, MariaDB, PHP 8.3 FPM, qemu-agent, cron, msmtp/sendmail transport.
- NPMplus on edge terminates TLS and proxies all five forums to VM160.
- Let’s Encrypt certificates are active for all five domains.
- Forum SMTP transport works through noreply@pvepro.ru; Yandex headers proved SPF pass and DKIM pass. Spam placement is classified as reputation/content deliverability, not server failure.
- XenForo job runner cron is configured and cron daemon execution was proven.
- Local backup is configured on VM160 and Mail.ru Cloud fresh5/all-forums backup is configured on pve02.
- Cloud restore drill passed: split parts verified, archive SHA matched, tar contained DB/file/config archives for all five forums.
- DBTech Tor timeout rows are classified as historical external timeout noise and were not deleted.
- Final proof: /root/evidence/629_FORUM_PROJECT_FINAL_ACCEPTANCE_PROOF.txt.
## FORUM_NEXT_CHAT_HANDOFF_FULL_STATE_20260701
### Start point
- Start shell: root@pve01.
- Canonical truth files:
- /etc/pve/HOMELAB_ASSISTANT_ERROR_REGISTER.md
- /etc/pve/31_HOMELAB_REFERENCE.md
- Before any infra command, strictly check both files and relevant context blocks.
- Every infra command must print:
- ERROR_REGISTER_CHECK=OK
- REFERENCE_CHECK=OK
### Production forum VM
- VMID: 160.
- Name: forum-prod.
- Node: pve02.
- IP: [PRIVATE_IP].
- OS: Debian 12.
- Runtime: nginx, MariaDB 10.11, PHP 8.3 FPM, qemu-agent, cron, msmtp/sendmail transport.
- Final accepted snapshot: forum-final-accepted-091934Z.
- Mail final snapshot: forum-mail-ok-091815Z.
- Postlaunch snapshot: forum-postlaunch-ok-085501Z.
### Public forums
- codevipe.ru -> Форум CodeVipe.
- gamevipe.ru -> Форум GameVipe.
- hkmods.ru -> Моды для Hollow Knight.
- zakrutim.ru -> Консервирование и закрутки.
- dsmods.ru -> Секреты и моды Doom.
- All five are public HTTPS 200 with valid TLS.
- Root/www DNS points through edge public IP 95.84.154.183.
- NPMplus on edge terminates TLS and proxies to http://[PRIVATE_IP]:80.
### Mail
- Forum sender: noreply@pvepro.ru.
- Transport: XenForo sendmail -> msmtp -> mail.pvepro.ru.
- Built-in XenForo mail test reached Yandex.
- Yandex headers showed SPF pass and DKIM pass for pvepro.ru.
- Yandex placed the test in Spam with X-Yandex-Spam: 4.
- Classification: transport/authentication OK; remaining issue is sender reputation/content filtering.
- DNS change is not required from the captured header proof.
- Manual next action: click “Не спам!” in Yandex and add/check pvepro.ru in Yandex Postmaster.
### Backups and restore
- Local backup configured inside VM160.
- Local archive path pattern: /var/backups/forums/forum-backup-*.tar.gz.
- pve02 Mail.ru Cloud fresh5/all-forums backup configured.
- New timer: forum-prod-fresh5-cloud-backup.timer active/enabled.
- Old codevipe-only timer disabled/inactive.
- Cloud remotes:
- pve02-mail-01-crypt
- pve02-mail-02-crypt
- pve02-mail-03-crypt
- pve02-mail-04-crypt
- Latest verified cloud stage during final acceptance: 20260701T083354Z.
- Restore drill passed: split parts verified, archive SHA matched, tar contained DB/file/config archives for all five forums.
### XenForo jobs and cron
- cron daemon was missing, then installed and enabled.
- /etc/cron.d execution was proven by temporary cron proof.
- XenForo job runner configured:
- /root/scripts/forum-xenforo-run-jobs.sh
- /etc/cron.d/forum-xenforo-run-jobs
- Purpose: process XenForo jobs and cron tasks independent of visitor traffic.
### DBTech/Tor timeout
- Each forum had one historical xf_error_log row for cURL timeout to check.torproject.org.
- Source addon: DBTech/Security DragonByte Security 5.0.0.
- dbtech_security_tornodes=0 on all five forums.
- Classified as non-blocking historical external timeout noise.
- Rows were not deleted.
### Key final proofs
- /root/evidence/629_FORUM_PROJECT_FINAL_ACCEPTANCE_PROOF.txt
- /root/evidence/628_FORUM_MAIL_DELIVERABILITY_FINAL_SNAPSHOT_PROOF.txt
- /root/evidence/627_YANDEX_HEADER_DELIVERABILITY_CLASSIFICATION_PROOF.txt
- /root/evidence/622_TOR_TIMEOUT_CLASSIFICATION_FINAL_PROOF.txt
- /root/evidence/621_FORUM_CRON_DAEMON_JOBRUNNER_VERIFY_PROOF.txt
- /root/evidence/614_FORUM_POSTLAUNCH_BACKUP_MAIL_FINAL_ROLLUP_PROOF.txt
- /root/evidence/613_FORUM_FRESH5_CLOUD_RESTORE_DRILL_FINALIZE_PROOF.txt
- /root/evidence/612_FORUM_FRESH5_CLOUD_RESTORE_DRILL_CORRECTED_PROOF.txt
- /root/evidence/609_FORUM_PROD_FRESH5_CLOUD_BACKUP_FINALIZE_PROOF.txt
----- CONTENT END -----
----- FILE=/etc/pve/HOMELAB_ASSISTANT_ERROR_REGISTER.md -----
SIZE=77979 MTIME=2026-07-15 09:27:06.000000000 +0300 MODE=640 OWNER=root:www-data
SHA256=ab8308549db90d235b591af0581580c3c7ba1b287b129cc6111c2eef0a0f9ecc
LINES=934
----- CONTENT BEGIN -----
# HOMELAB ASSISTANT ERROR REGISTER
Назначение: перед каждой следующей командой сверяться с этим файлом.
## Критические ошибки ассистента
1. Повторно дал слишком большой интерактивный paste в shell.
2. Повторно использовал here-doc/Markdown/backticks там, где нужен короткий безопасный файл или отдельный скрипт.
3. Дал генератор справочника прямо в терминал вместо безопасного маленького шага.
4. Нарушил своё же правило: не давать длинные вложенные команды с кавычками.
## Жёсткие правила перед каждой командой
CHECK-1: команда не должна быть большим paste.
CHECK-2: команда не должна содержать большой here-doc.
CHECK-3: команда не должна смешивать Markdown, backticks и shell-логику.
CHECK-4: команда не должна иметь вложенный ssh с несколькими уровнями кавычек.
CHECK-5: команда не должна печатать секреты.
CHECK-6: если создаётся файл, сначала маленький безопасный шаг, потом проверка.
CHECK-7: если команда длиннее 8 строк, её нельзя давать в интерактивный shell.
CHECK-8: для edge-vm использовать debian@[PRIVATE_IP] и sudo.
CHECK-9: для forum-prod использовать pve02 и ключ [SENSITIVE_PATH]
CHECK-10: Corosync не трогать без отдельного плана и rollback.
## Текущие важные факты
Internal network: [PRIVATE_IP]/24.
Migration config: migration: secure,network=[PRIVATE_IP]/24.
Corosync remains on [PRIVATE_IP]/12/13.
VM160 forum-prod is not in Proxmox nightly backup.
VM130 edge-vm has secondary disk backup=0 risk.
05_edge_compose_safe.tgz quarantined.
## Правило для справочника
Не генерировать большой справочник через интерактивную вставку.
Следующий справочник делать только маленькими append-блоками или через файл, созданный вне интерактивного shell.
11. Ошибка: считать offhost OK после failed rsync.
Если rsync/scp упал, нельзя проверять latest offhost-файл без сверки имени.
Проверка должна подтвердить именно новый архив, например дату 2026_06_30-00_20_59.
Старый OFFHOST_ZSTD_OK не закрывает новый backup.
12. Ошибка: широкий secret-поиск по /opt/stacks дал шум.
Не искать Cloudflare/SMTP сразу по /opt/stacks, MkDocs site, backups и HTML-отчётам.
Для ротации сначала смотреть live-конфиги: /opt/npmplus/tls/certbot/renewal/*.conf и реальные credentials-файлы из них.
Значения секретов не печатать; выводить только пути, ключи и redacted-поля.
13. Ошибка: SQL с одинарными кавычками внутри одинарной SSH-команды ломается.
Нельзя писать sqlite SQL вида j.type in ('object','array') внутри ssh '...'.
Для sqlite через ssh использовать SQL без одинарных кавычек: char(36), length(j.atom), двойные внешние кавычки, либо отдельный файл.
Перед запуском проверять, что команда не содержит конфликтующих уровней кавычек.
14. Ошибка: путать контекст входа и узел выполнения.
Стартовая точка оператора: root@pve01 / [PRIVATE_IP].
edge-vm: ssh debian@[PRIVATE_IP], внутри использовать sudo.
pve02/pve03: ssh root@pve02 или ssh root@pve03.
forum-prod: заходить через pve02, ключ [SENSITIVE_PATH]
Перед каждой командой явно понимать: где выполняется локальная часть, где remote-часть, каким пользователем.
15. Ошибка: повторно нарушено правило №13 после его добавления.
Снова был использован SQL JSON-path в одинарных кавычках внутри ssh '...'.
Запрещено использовать в remote sqlite SQL фрагменты вида '$.key' или '.key'.
Для JSON-path использовать только char(...), без одинарных кавычек внутри SQL.
Команду с ошибкой char(36)||.dns_provider считать битой и не использовать.
16. Основополагающее правило: перед каждой командой ассистент обязан явно написать: Сверка с файлом ошибок: CHECK пройден.
После этой строки можно давать только одну короткую команду или один логический блок команд.
Нельзя выдавать команды без предварительной сверки с этим файлом ошибок.
Нельзя продолжать после собственной ошибки без записи ошибки в этот файл.
19. Уточнение формата: не писать отдельную строку сверки перед каждой командой в пакете.
Сверка должна быть внутри самой команды первым логическим блоком: grep правил из файла ошибок и RULE_CHECK_OK.
Разрешено давать до трёх логически связанных команд сразу, по возможности в одну строку каждая.
Не писать фразу "ждём" после команд; указывать только контрольные строки результата.
20. Ошибка: широкий XenForo SMTP-поиск по src дал шум исходников.
Для XenForo SMTP не grep-ать весь /src; сначала смотреть live src/config.php без значений, затем БД XenForo и xf_option/emailTransport.
Значения DB-паролей и SMTP-паролей не печатать.
21. Ошибка: nested PHP php -r дал Parse error на forum-prod.
Команды 102-104 считать битыми: они не подтвердили и не изменили smtpSsl.
Не продолжать длинные вложенные ssh+php -r с кавычками; для forum-prod лучше войти в VM или использовать короткие mysql/php команды без вложенного кода.
Рабочими считать: DB backup перед SMTP, SMTP password update, JSON postcheck; SMTP auth ещё не закрыт из-за CODE=530 на AUTH LOGIN.
22. Ошибка: самодельный base64 PHP для SMTP auth сломан.
Команда 108 дала PHP Parse error на smtpHost и пустой proof-файл.
Не использовать больше сгенерированные большие base64 PHP для SMTP/XenForo.
Для XenForo SMTP использовать короткие mysql-запросы к xf_option и готовые штатные механизмы XenForo.
23. Ошибка: MariaDB не поддержала cast('true' as json) в JSON_SET.
Команда 111 не изменила smtpSsl: OLD_SMTPSSL=false, SQL ERROR 1064, NEW_SMTPSSL=false.
Перед боевым UPDATE сначала проверять JSON_SET синтаксис на тестовом JSON через SELECT.
Не считать PHP_FPM_RELOAD_OK подтверждением изменения БД.
24. Ошибка: exit 1 в interactive-check закрыл SSH-сессию.
Нельзя использовать { echo RULE_CHECK_FAIL; exit 1; } в командах, выполняемых прямо в интерактивном root@pve01 shell.
При failed-check использовать безопасный шаблон: if grep ...; then echo RULE_CHECK_OK; ...; else echo RULE_CHECK_FAIL; fi.
Не делать fragile grep по точной строке с кавычками, если в файле ошибок строка содержит shell/SQL quoting.
25. Ошибка анализа: не надо слепо включать smtpSsl=true при smtpPort=587.
smtpPort=587 обычно означает STARTTLS, а не implicit SMTPS.
CODE=530 на AUTH LOGIN означает, что ручной checker не сделал STARTTLS; это не доказывает неверный SMTP-пароль.
Сначала проверять AUTH через openssl s_client -starttls smtp, без вывода секрета.
26. Ошибка: openssl -crlf вместе с ручным CRLF ломает SMTP AUTH checker.
Команда 116 получила 535 Invalid base64 data in continued response после 334 Username.
Это указывает на битую base64-строку checker-а, а не на неверный SMTP-пароль.
Не использовать -crlf, если команды уже отправляются с явным \r\n.
Для 587 оставлять smtpSsl=false и проверять STARTTLS корректным checker-ом.
27. Ошибка: Python heredoc внутри ssh сломал not_ok диагностику.
Команда 138 пересобрала dashboard, но диагностический Python-блок упал с NameError из-за кавычек.
Не использовать heredoc Python внутри вложенного ssh; для dashboard verification использовать cat/grep JSON или простые команды.
28. Ошибка: docker compose ls --format json | wc -l дал ложный COMPOSE_PROJECT_COUNT=1.
Причина: JSON-вывод может быть одной строкой/структурой, wc -l не является счётчиком проектов.
Для Docker Compose count использовать обычный docker compose ls --all + awk по строкам таблицы или JSON parser, не wc -l.
29. Ошибка: monitoring compact status искал неверные имена health-файлов.
Факт: npmplus-cert-expiry.txt не является подробным файлом; подробный файл называется npmplus-certificate-expiry.txt, а статусный alias npmplus-cert-expiry-health.txt.
Факт: health-metrics.txt не обязан существовать как health-файл; health metrics могут публиковаться через node-exporter textfile collector.
Для Prometheus targets проверять HTTP_CODE/RC явно, не оставлять пустой блок PROMETHEUS_TARGETS_LOCAL.
30. Ошибка/инцидент: Prometheus API HTTP_CODE=000 после monitoring reference.
Факт: в monitoring compact status curl к 127.0.0.1:9090 вернул HTTP_CODE=000, а raw inventory не показал контейнер prometheus среди monitoring containers.
Нельзя считать monitoring слой runtime-closed только по secret scan/reference proof; нужно отдельно проверить Prometheus container/API и обновить справочник по факту.
31. Строгое правило: перед каждой командой сверяться с файлом ошибок и справочником.
Перед любым действием команда обязана проверять /etc/pve/HOMELAB_ASSISTANT_ERROR_REGISTER.md и текущий 31_HOMELAB_REFERENCE.md.
Команда обязана печатать ERROR_REGISTER_CHECK=OK и REFERENCE_CHECK=OK до основного действия.
Если сверка не пройдена, команда не должна выполнять основное действие. Не использовать exit 1, чтобы не закрывать SSH-сессию.
32. Ошибка: 153_PROMETHEUS_TARGETS_SETTLED_PROOF содержит SyntaxError из-за вложенного Python внутри SSH.
Факт: Python -c внутри ssh потерял кавычки вокруг /tmp/prom-targets-settled.json, data, activeTargets, labels, job, health.
Файл 153_PROMETHEUS_TARGETS_SETTLED_PROOF.txt нельзя считать валидным settled proof.
Для таких проверок не использовать вложенный Python; использовать curl + grep/sed/awk либо отдельный локальный файл-скрипт.
34. Ошибка операционного ввода: полный терминальный лог MobaXterm попал в shell.
Факт: bash пытался выполнить строки Authenticating, MobaXterm banner, RULE_CHECK_OK, PROOF_OK и таблицы вывода как команды.
Такие выводы не являются инфраструктурным состоянием; считать невалидными только соответствующие failed command attempts.
Дальше давать короткие команды и не вставлять обратно полный transcript в shell.
33. Security finding: root authorized_keys на PVE-нодах имел права 777.
Факт: inventory 158 показал [SENSITIVE_PATH] и backup-файлы с режимом 777 на pve01/pve02/pve03.
Нужно исправлять на [SENSITIVE_PATH]=700 и authorized_keys/authorized_keys.bak*=600, затем фиксировать before/after proof.
35. Ошибка проверки: 160_PVE_ROOT_AUTHORIZED_KEYS_PERMISSIONS_FIXED не доказал исправление прав.
Факт: AFTER всё ещё показал 777 для [SENSITIVE_PATH] на pve01/pve02/pve03.
Возможная причина: authorized_keys является symlink, а stat без -L показывает режим ссылки 777.
Нужно проверять stat -L целевого файла и считать 160 недостаточным proof.
36. Quality check: Storage block needs integrity and pve03 capacity coverage review.
Факт: terminal output around storage block creation showed possible paste/truncation artifact near smartctCK=FAIL.
Факт: pve03 /mnt/staging observed at 77%, close to WARN threshold 80%, while disk-space.txt did not visibly list pve03_staging.
Before closing storage layer, verify block integrity and pve03 capacity coverage.
37. Coverage gap: pve03_staging missing from disk-space health coverage.
Факт: 169 showed pve03 /mnt/staging at 77%, close to WARN 80%, while disk-space.txt listed pve01_staging, pve02_staging and edge_immich_media only.
Before closing storage fully, add or document pve03_staging capacity monitoring.
38. Quality check: Service Dependency Map block needs integrity review.
Факт: terminal output around command 222 showed paste artifact near "FAIL; fi".
Before closing Service Dependency Map, verify 178_REFERENCE_SERVICE_DEPENDENCY_MAP_BLOCK.txt and the reference section for shell/log garbage or truncation.
39. Ошибка проверки: post deep audit BAD_MARKER scan caught prose false positive.
Факт: строка справочника с фразой No unresolved final REVIEW_REQUIRED... SyntaxError... была принята за BAD_MARKER.
Это не runtime/log мусор, а документированная фраза deep audit closure. В scan нужно исключать эту prose-строку.
40. Ошибка/вывод: VM150 Mail cloud single-file upload failed with 413.
Факт: Mail/WebDAV rejected one 6.6G vzdump file with 413 Request Entity Too Large.
Решение: use VM150 recurring chunked script with 512M parts and download verification.
41. Ошибка: post-backup-pass-slo-reconciliation checked pve01 router health on edge-vm.
Факт: first reconciliation wrote STATUS=ERROR because router-running-config-mail-cloud.txt is on pve01, not edge-vm.
Fix: build reconciliation from pve01 and edge-vm health files by correct owner.
## ASSISTANT_COMMAND_BATCHING_RULE_20260630
- Rule: combine related infrastructure checks, proof generation, reference updates and validation scans into one safe batch command when practical.
- Do not split work into many small commands when the operations are low-risk, read-only, or naturally part of one evidence block.
- Still keep mandatory preflight checks, short bounded commands, no large paste, no here-docs, no interactive shell traps, and no sensitive values in output.
- Split into smaller commands only when safety, rollback, quoting risk, or troubleshooting clarity requires it.
## HOME_PORTAL_BASE64_APPLY_FAILURE_20260630
- Prior home-portal apply attempt 312 failed because a large base64 Python payload decoded as invalid UTF-8.
- Rule: avoid large opaque base64 script payloads for homelab changes; use transparent bounded shell/perl/python commands instead.
## HOMELAB_COMMAND_SAFETY_HARDENING_20260630
- Rule: do not use large opaque payloads, generated base64 scripts, or hidden multi-line script blobs for homelab changes.
- Use transparent bounded shell/perl/python commands, or create a temporary readable script and validate it before execution.
- Any script used for changes must pass a syntax/parse check before it is allowed to modify files or restart services.
- Any Traceback, UnicodeDecodeError, SyntaxError, Permission denied, command-not-found, nonzero critical RC, or failed write marker must force REVIEW or FAILED, never OK.
- Success requires both runtime health and content-specific post-checks: expected files changed, expected cards exist, duplicates absent, service/container alive, and no new critical log pattern.
- HTTP 200 alone is not a success condition for configuration changes.
- If an apply step fails, the next command must explicitly record the failed attempt and then run a corrected transparent apply command.
## HOME_PORTAL_API_ERROR_CLOSURE_RULE_20260630
- Rule: home portal closure requires current Homepage API-error logs to be zero after restart/reload, not only HTTP 200.
- If Homepage UI shows API error or logs contain current httpProxy/API timeout errors, the portal is REVIEW until the widget/API cause is disabled or fixed.
- External informational widgets such as weather/Open-Meteo must not block the service launcher portal.
## HOME_PORTAL_COMPLEX_QUOTING_FAILURE_20260630
- Previous home portal cleanup command failed in local bash before execution: syntax error near unexpected token `('.
- Cause: command was too complex and fragile due to nested shell/perl/python quoting.
- Rule: do not use deeply nested one-liners for config edits; first run short state checks, then apply small transparent edits.
## HOME_PORTAL_ACTIVE_CONFIG_SCOPE_RULE_20260630
- Rule: Homepage cleanup validation must check active config files only, not logs or backup files.
- Active files are services.yaml, bookmarks.yaml, widgets.yaml, settings.yaml and docker.yaml under /opt/stacks/homepage/config.
- Matches in /logs/ or *.bak* are historical evidence only and must not make cleanup REVIEW unless the active config still contains the target.
## HOME_PORTAL_LINK_VALIDATION_NPMPLUS_DEFAULT_RULE_20260630
- Rule: Homepage link validation must not treat HTTP 200 as success if the body is the NPMPlus default “Congratulations” / host-not-set-up page.
- Green siteMonitor dots can be false positives when NPMPlus returns its default vhost page.
- A portal card is openable only when DNS, reverse-proxy host mapping, and target content are all valid.
## ROUTER_CLI_STDIN_APPLY_FAILURE_20260630
- Context: attempted Netcraze router ACL apply through SSH stdin/multiline for Homepage Moscow Router monitor fix.
- Evidence: 391_MOSCOW_ROUTER_EDGE_VM_WEBADMIN_ACL_APPLY_PROOF.txt.
- Failure: STDIN_SHOW_OK=no, APPLY_SKIPPED=stdin_show_failed, CONFIG_HAS_EDGE_5080=no, CONFIG_HAS_EDGE_5083=no.
- Rule: do not apply Netcraze config through stdin/multiline SSH; prove CLI input mode first.
## ROUTER_CLI_PROBE_STDIN_CONSUMPTION_ERROR_20260630
- Context: ACL syntax read-only probe loop executed only one command because ssh consumed the loop stdin.
- Evidence: 393_ROUTER_CLI_ACL_SYNTAX_READONLY_PROBE_PROOF.txt showed COMMAND_COUNT=1.
- Fix pattern: use ssh -n or redirect SSH stdin away from the command-list loop for all future SSH-in-loop probes.
## ROUTERBACKUP_READONLY_CLI_PERMISSION_LIMIT_20260630
- Context: routerbackup user can run show running-config but cannot execute configuration commands such as access-list.
- Evidence: 394_ROUTER_CLI_ACL_SYNTAX_READONLY_PROBE_FIXED_PROOF.txt.
- Observed: CMD_TEXT=access-list ? returned Core::Configurator error execute denied [cli].
- Rule: do not attempt ACL/config changes as routerbackup; use admin/operator router access or router Web UI for write changes.
## PROOF_SUMMARY_EXTRACTION_BLANK_20260630
- Context: proof 395 tried to read COMMAND_COUNT/ZERO_RC_COUNT from proof files 393/394, but those summary fields were printed to console and were not persisted inside the proof files.
- Evidence: 395_ROUTER_CLI_ERROR_REGISTER_UPDATE_PROOF.txt has blank P393_COMMAND_COUNT/P394_COMMAND_COUNT/P394_ZERO_RC_COUNT fields.
- Rule: when creating follow-up proof summaries, derive counts from persisted proof body lines such as CMD_INDEX and CMD_RC, or write summary fields into the proof file itself.
## ROUTER_ACL_UI_RULE_DELETE_AND_RESTORE_20260701
- Context: while moving Moscow Router monitor rules to _WEBADMIN_Bridge1, two _WEBADMIN_Bridge0 Nextcloud Talk TURN rules were accidentally removed.
- Deleted/restored rules: Home [PRIVATE_IP]/24 to Nextcloud [PRIVATE_IP] TCP/UDP 3478.
- Verification proof: 399_ROUTER_ACL_RESTORE_AFTER_UI_PROOF.txt.
- Rule: after manual router Web UI ACL edits, compare _WEBADMIN_Bridge0, _WEBADMIN_Bridge1 and _WEBADMIN_GigabitEthernet1 before continuing.
## ROUTER_MONITOR_NODE_LOOKUP_PROBE_BUG_20260701
- Context: proof 400 tested Moscow router HTTPS monitor from Homepage container with custom Node lookup callback.
- Failure: NODE_HTTPS_RESOLVE_ERROR=ERR_INVALID_IP_ADDRESS and NODE_HTTPS_RESOLVE_INSECURE_ERROR=ERR_INVALID_IP_ADDRESS.
- Impact: Node HTTPS resolve subtest is invalid; curl --resolve results remain valid.
- Rule: do not rely on custom Node lookup callback until separately tested; prefer curl --resolve or simple direct Node HTTP checks.
## ROUTER_HTTP_SERVICE_FORBIDDEN_FROM_EDGE_VM_20260701
- Context: Moscow Router Homepage monitor after ACL fix.
- Evidence: proofs 399, 400, 401.
- Observed: edge-vm TCP to [PRIVATE_IP]:5080/5083 is OK, but router HTTP/HTTPS returns 403 for all tested paths.
- Impact: Homepage siteMonitor cannot be made green via direct router HTTP URL until router web service allows the Bridge1/Proxmox/protected segment or an explicit safe monitor endpoint is used.
- Rule: do not keep changing ACL for this symptom; ACL is already correct. Fix service access scope or use a dedicated monitor endpoint.
## PY_COMPILE_PYC_PERMISSION_ERROR_20260701
- Context: installing edge-vm Moscow router health endpoint.
- Mistake: assistant used `python3 -m py_compile` on a script installed under /usr/local/sbin as root.
- Actual impact: non-root syntax check attempted to create /usr/local/sbin/__pycache__ and failed with Permission denied.
- Recovery: validate syntax with `ast.parse` instead of py_compile, because it does not write .pyc files.
- Rule: do not use py_compile against root-owned system paths from an unprivileged user.
## ROUTER_HOME_IP_DOCKER_ROUTE_CONFLICT_20260701
- Context: Moscow Router Homepage siteMonitor attempt using http://[PRIVATE_IP]:5080.
- Evidence: proof 405 showed route to [PRIVATE_IP] via Docker bridge and Homepage container ECONNRESET.
- Impact: do not use [PRIVATE_IP] directly as Homepage siteMonitor from edge-vm.
- Mitigation: use dedicated edge-vm health endpoint that checks router TCP [PRIVATE_IP]:5080 and returns HTTP 200/503 for Homepage.
## HOMEPAGE_ROUTER_MOSCOW_YAML_TITLE_SHAPE_MISMATCH_20260701
- Context: applying Moscow Router Homepage siteMonitor health endpoint.
- Mistake: assistant apply script matched only property-style cards with `title`, but active Homepage YAML can use service-name-key style like `- Service Name:`.
- Evidence: proof 407 returned YAML_CHANGE_COUNT=0 and APPLY_SKIPPED=unexpected_match_count while health endpoint proof 406 was OK.
- Recovery: use YAML-aware updater that handles both property-style and service-name-key style cards.
## HOMEPAGE_ROUTER_MOSCOW_APPLY_SCRIPT_SYNTAX_ERROR_20260701
- Context: applying Moscow Router Homepage health endpoint.
- Mistake: assistant generated Python script with invalid f-string escaping.
- Evidence: proof 408 showed SyntaxError in /tmp/homepage-router-moscow-apply-fixed.py.
- Actual impact: YAML was not changed, so Homepage green dot could not appear.
- Recovery: stop applying blind updater; first discover exact active YAML shape, then apply minimal YAML-aware edit.
- Rule: validate generated Python syntax locally before scp/remote execution; avoid f-strings in generated maintenance scripts.
## FORUM_PROD_VM160_FIRST_BOOT_SSH255_20260701
- Context: clean rebuilt VM160 first boot.
- Symptom: PING_OK=yes and TCP22_OK=yes, but SSH_RC=255.
- Rule: do not proceed with OS baseline until SSH failure is diagnosed; likely old known_hosts key or cloud-init/root-key issue.
## FORUM_PROD_VM160_NESTED_HOSTNAME_PROOF_QUOTING_20260701
- Context: VM160 first SSH proof after rebuild.
- Issue: command substitution $(hostname) inside nested ssh was expanded on pve02 before entering VM160.
- Impact: VM_HOSTNAME=pve02 in proof 426 is not valid VM identity evidence.
- Rule: for nested SSH identity checks, run literal hostname commands without local command substitution.
## FORUM_PROD_VM160_SWAP_PROOF_QUOTING_20260701
- Context: VM160 swapfile proof 429.
- Issue: nested command substitution expanded on pve02, producing invalid SWAPFILE/FSTAB proof lines.
- Impact: proof 429 is not valid closure evidence even though swap was active.
- Rule: avoid nested $(...) in VM SSH proofs; use literal remote commands and clean proof.
## FORUM_UPLOAD_ARCHIVES_VM_VERIFY_QUOTING_20260701
- Context: proof 446 copy/check archives inside VM160.
- Issue: nested SSH quoting expanded shell variables incorrectly, producing gzip checks against empty .gz and blank TAR_TOP lines.
- Impact: SCP_TO_VM_RC=0 and file size list are valid, but GZIP/TAR proof lines in 446 are invalid.
- Rule: verify VM archive integrity locally inside forum-prod, not through nested quoted loops.
## FORUM_PROD_BULK_IMPORT_PHP85_EMPTY_FRONTEND_20260701
- Context: rebuilt VM160, five XenForo forums imported in bulk on PHP 8.5.7.
- Symptom: all five forums returned HTTP 200 with empty body; XenForo logged Template public:PAGE_CONTAINER/forum_list is unknown.
- Additional failed approach: proof 480 ran xf-dev:rebuild-caches on CodeVipe and returned RC=1, not a valid fix.
- Decision: stop bulk repair, treat current VM160 as lab, find a working CodeVipe recipe first, then destroy/rebuild cleanly for all five forums.
- Rule: no more mass rebuild/repair commands across all forums until single-forum CodeVipe recipe is proven.
## XENFORO_REQUIREMENTS_DIRECT_DOWNLOAD_403_20260701
- Context: Tried to download XenForo requirements ZIP directly from xenforo.com inside forum-prod.
- Issue: curl returned HTTP 403; unzip then failed because the ZIP was not downloaded.
- Evidence: proof 491 showed DOWNLOAD_RC=22, ZIP_LIST_RC=9, UNZIP_RC=9.
- Impact: proof 491 is not a valid server compatibility test.
- Rule: use operator-uploaded xenforo23-requirements-test.zip from /root/forum-upload instead of relying on direct curl download.
## PVE01_UNZIP_MISSING_FOR_XF_REQUIREMENTS_VERIFY_20260701
- Context: proof 492 tried to inspect uploaded xenforo23-requirements-test.zip on pve01.
- Issue: pve01 does not have unzip installed, so ZIP_TEST_RC=127 was not a ZIP integrity result.
- Impact: proof 492 confirmed file presence only, not archive validity.
- Rule: verify ZIP with Python zipfile or inside forum-prod rather than installing unzip on Proxmox just for inspection.
## FRESH5_DEPLOY_SUCCESS_20260701
- Context: VM160 was destroyed/recreated, then five XenForo forums were deployed from fresh ZIP backups.
- Result: proof 513 confirms all five forums locally healthy.
- Note: proof 511 initially failed dsmods path detection; proof 512 fixed dsmods by locating public_html via src/XF.php marker.
- Rule: future restores should preserve internal_data/code_cache or rebuild compiled template cache before smoke testing.
## NPMPLUS_SQLITE_PASTE_FAILURE_20260701
- Context: NPMplus SQLite inspection command was too complex and a Python fragment was pasted into bash.
- Issue: shell entered multiline prompt and produced syntax errors.
- Impact: do not trust that SQLite inspection attempt.
- Rule: avoid nested SSH heredoc/Python for this task; use simpler commands, generated nginx configs, or uploaded/local script files.
## NPMPLUS_PUBLISH_SCRIPT_WRONG_ENV_SOURCE_525_20260701
- Context: proof 525 tried to create forum proxy hosts in NPMplus.
- Issue: script sourced /opt/npmplus/.env, but INITIAL_ADMIN_EMAIL/PASSWORD were not present there; they are in the npmplus container env.
- Impact: no forum proxy hosts were created by proof 525.
- Rule: read NPMplus API login values from docker inspect env internally, never print them.
## NPMPLUS_API_LOGIN_MISSED_HTTPS81_AND_JSON_ESCAPE_20260701
- Context: NPMplus API login attempts in proofs 526/527 failed.
- Issue: commands missed the likely HTTPS admin endpoint on port 81 and built JSON by string interpolation instead of jq escaping.
- Impact: no proxy hosts were created by 526/527.
- Rule: for NPMplus API use jq-generated JSON and test https://127.0.0.1:81/api/tokens with -k before publish.
## NPMPLUS_API_AUTH_UNAVAILABLE_MANUAL_ROUTES_20260701
- Context: NPMplus API login attempts failed even though the forum VM and edge-to-forum connectivity are healthy.
- Issue: API credentials from container initial env are not accepted by current NPMplus API.
- Impact: do not use NPMplus API for this publish path.
- Rule: publish forum routes through backed-up manual nginx proxy_host configs, then issue proper certificates after DNS points to edge.
## EDGE_CERTBOT_TOKEN_FILE_PERMISSION_535_20260701
- Context: proof 535 attempted DNS-01 certificate issue for five forum domains.
- Issue: /tmp/forum_cf_token.env was root-owned mode 600, so debian user could not source it directly.
- Impact: certbot ran without token and all certificate attempts failed; DNS cutover must not proceed from proof 535.
- Rule: read Cloudflare token via sudo without printing it, then create temporary certbot credentials file and remove it after use.
## EDGE_CERTBOT_FAILED_DNS_CUTOVER_ABORTED_20260701
- Context: edge manual routes are working locally, but certbot DNS-01 certificate issue failed in proof 538.
- Issue: all five certbot attempts returned RC=1; proof 539 correctly aborted DNS cutover because edge cert precheck was not ready.
- Impact: Cloudflare DNS records were not changed by proof 539; public DNS still points to old IP 87.236.18.45.
- Rule: inspect certbot logs and Cloudflare token zone permissions before retrying certificate issuance or DNS cutover.
## CLOUDFLARE_TOKEN_NO_ZONE_ACCESS_20260701
- Context: Cloudflare token discovered on edge verified as a token but could not access forum zones.
- Evidence: proof 542 showed all five zones missing and DNS record create probes failed.
- Impact: certbot DNS-01 failed and DNS cutover was correctly aborted; public DNS still points to old hosting.
- Rule: install a new Cloudflare token with Zone Read and DNS Edit for the five forum zones before retrying certs/DNS.
## FORUM_PUBLICATION_FINAL_SUCCESS_20260701
- Context: five XenForo forums were rebuilt on VM160, routed through edge NPMplus, issued Let’s Encrypt certs using a corrected Cloudflare token, and cut over in DNS.
- Result: final public proof 546 passed.
- Important corrections: initial NPMplus API route failed; manual Nginx routes were used. Initial Cloudflare token had no zone access; corrected token passed zone/DNS probe.
## FINAL_SNAPSHOT_NAME_TOO_LONG_556_20260701
- Context: proof 556 final health gate passed for all five public forums.
- Issue: Proxmox snapshot failed because generated snapshot name exceeded 40 characters.
- Evidence: qm snapshot returned snapname value may only be 40 characters long.
- Impact: forum health was OK, but proof 556 snapshot step was not completed.
- Fix: rerun snapshot with short name.
## CLOUDFLARE_DNS_AUDIT_TXT_SUBSTRING_GAP_20260701
- Context: prior Cloudflare DNS audit checked web A/AAAA/CNAME state for root/www records.
- Issue: it did not deeply inspect TXT/SPF contents for old provider IP substrings.
- Evidence: user screenshot showed SPF TXT containing old 87.236.18.* value.
- Impact: web routing is OK, but mail-related DNS may still contain stale provider data.
- Rule: perform full DNS record audit including TXT/SPF/DKIM/DMARC/MX/CDN and substring search for old IPs before DNS cleanup.
## CLOUDFLARE_DNS_CLEANUP_DRYRUN_QUOTE_563_20260701
- Context: SPF cleanup dry-run command 563 had broken shell quoting and left the terminal at multiline prompt.
- Impact: proof 563 is invalid and no DNS cleanup was performed by it.
- Rule: use copied scripts for Cloudflare DNS mutation instead of fragile inline nested quotes.
## SPF_DUPLICATE_AFTER_565_20260701
- Context: SPF cleanup command 565 attempted to replace stale SPF records.
- Issue: backup write permission problem caused record detection failure, so new v=spf1 -all records were created while old v=spf1 ip4:87.236.18.45 records remained.
- Impact: domains temporarily had duplicate SPF records, which is invalid for mail validation.
- Fix: delete all SPF TXT records for the five forum zones, then create exactly one v=spf1 -all TXT per zone.
## MSMTP_SECRET_SOURCE_PARSE_ERROR_576_20260701
- Context: proof 576 installed msmtp but sendmail auth test failed.
- Issue: [SENSITIVE_PATH] was sourced as shell, but SMTP password contained shell-special characters; source failed and auth used an invalid/truncated secret path.
- Impact: msmtp package installed, but mail sending was not proven working.
- Rule: never source SMTP secret files containing arbitrary passwords; store password base64 and use msmtp passwordeval helper accessible to www-data.
## FORUM_MSMTP_MAIL_TRANSPORT_STILL_FAILING_20260701
- Context: attempted to fix msmtp config with passwordeval helper.
- Issue: one or more sendmail/PHP mail tests still failed.
- Impact: XenForo mail sending is not yet proven.
- Next step: verify SMTP credential/password and provider policy for noreply@pvepro.ru.
## SMTP_PASSWORD_EXPOSED_ROTATE_REQUIRED_20260701
- Context: SMTP password was exposed in terminal output during failed msmtp setup.
- Cause: secret file with raw SMTP_PASS was later parsed/sourced by shell; shell error printed the raw assignment line.
- Impact: treat that SMTP password as compromised.
- Required action: rotate the mailbox/app password in the mail provider panel before any further SMTP testing.
- Rule: never store arbitrary passwords as shell-sourceable KEY=value files; use non-printing secret storage only.
## NETBIRD_MAILCOW_REPORTED_DOWN_AFTER_SMTP_INCIDENT_20260701
- Context: user reported NetBird and Mailcow became unavailable after forum mail-transport work.
- Impact: treat as active incident until service reachability and container/VM state are proven.
- Rule: do not change DNS, firewall, proxy, mail, NetBird, or Mailcow configs before read-only triage proof.
## FORUM_SMTP_FAILED_CONFIG_DISABLED_AFTER_MAILCOW_BAN_20260701
- Context: forum-prod SMTP/msmtp test used invalid or compromised credentials and triggered Mailcow netfilter warnings/ban for 95.84.154.183.
- Impact: forum mail transport is not proven and must remain disabled until SMTP password is rotated and tested once carefully.
- Action: remove forum-prod msmtp secret/config files so XenForo cannot keep retrying broken SMTP auth.
- Rule: do not test SMTP auth again until a fresh rotated mailbox/app password is available.
## FORUM_OLD_CODEVIPE_CLOUD_BACKUP_STALE_AFTER_FRESH5_20260701
- Context: pve02 codevipe-cloud-backup.timer existed before fresh five-forum rebuild.
- Issue: old script points to /root/backups/codevipe and old remote path app/codevipe/forum-prod/local.
- Evidence: latest service run failed with missing /root/backups/codevipe; remote inventory contains old codevipe-public/codevipe.sql files, not current five-forum archive.
- Impact: old timer must not be treated as valid current backup for all five forums.
- Fix: create a new forum-prod fresh5 cloud backup using existing pve02 Mail-cloud rclone crypt remotes.
## FORUM_RESTORE_DRILL_610_STATUS_FLAG_BUG_20260701
- Context: restore drill proof 610 downloaded Mail.ru Cloud backup, verified split part SHA256, reconstructed archive, matched full archive SHA256, and confirmed all five forum DB/file archives in tar.
- Issue: final RESTORE_DRILL_OK was set to no because the script tested an unassigned shell variable RCLONE_REMOTE_PVE02_MAIL_01_RC.
- Impact: proof 610 final status flag is invalid, but its detailed integrity checks are valid.
- Fix: rerun corrected restore validation against the downloaded cloud backup artifacts and produce proof 612.
## FORUM_XENFORO_MAIL_SMOKE_TEST_FAILED_20260701
- XenForo-level mail smoke test did not return success for all five forums.
- Check proof 623 and msmtp log before retrying.
## XENFORO_MAIL_SMOKE_623_INVALID_ROOT_PATH_20260701
- Context: custom XenForo mail smoke test 623 failed with Could not open input file /root/scripts/623_xenforo_mail_smoke.php.
- Cause: the script was run as www-data from /root/scripts; /root is not traversable by www-data.
- Impact: proof 623 is invalid and should not be used to judge mail delivery.
- Superseding evidence: user ran XenForo built-in outgoing email test; message was delivered to Yandex but placed in Spam.
- Next step: investigate deliverability/authentication headers and pvepro.ru SPF/DKIM/DMARC/PTR.
## FORUM_NEXT_CHAT_KNOWN_ERRORS_AND_CLOSED_INCIDENTS_20260701
### Closed / classified incidents
- SMTP_PASSWORD_EXPOSED_ROTATE_REQUIRED_20260701:
- A previous bad command sourced a raw SMTP secret and printed it.
- Treat old password as compromised.
- Later persistent SMTP was rebuilt using safe files and verified.
- Never print or package secrets.
- XENFORO_MAIL_SMOKE_623_INVALID_ROOT_PATH_20260701:
- Custom mail proof 623 failed with "Could not open input file".
- Cause: PHP was run as www-data from /root/scripts, but /root is not traversable by www-data.
- Impact: proof 623 is invalid and must not be used to judge mail delivery.
- Superseded by user-observed built-in XenForo test and Yandex header proof.
- FORUM_MSMTP_MAIL_TRANSPORT_STILL_FAILING_20260701 / FORUM_SMTP_FAILED_CONFIG_DISABLED_AFTER_MAILCOW_BAN_20260701:
- Earlier SMTP attempts failed and triggered mailcow netfilter ban for forum public IP.
- Ban was removed.
- Persistent msmtp transport was later enabled and verified.
- Mailcow and NetBird remained reachable after final tests.
- SPF_DUPLICATE_AFTER_565_20260701:
- Earlier SPF cleanup created duplicate SPF records.
- Fixed by deleting duplicates and recreating exactly one SPF per forum domain.
- Final forum domains use v=spf1 -all because sending uses noreply@pvepro.ru.
- FORUM_RESTORE_DRILL_610_STATUS_FLAG_BUG_20260701:
- Restore drill 610 had a status flag bug despite successful detailed checks.
- Corrected restore drill 612 passed.
- FORUM_DBTECH_TOR_TIMEOUT_CLASSIFIED_20260701:
- One historical timeout row per forum to check.torproject.org.
- DBTech/Security active, but dbtech_security_tornodes=0.
- Classified as external timeout noise, not runtime failure.
### Current non-blocking items
- Yandex placed built-in XenForo test mail in Spam despite SPF pass and DKIM pass.
- Classification: deliverability/reputation/content filtering, not server failure.
- Recommended manual action: click “Не спам!” and add/check pvepro.ru in Yandex Postmaster.
### Safety rules for next chat
- Do not print secrets.
- Do not download or upload:
- [SENSITIVE_PATH]
- /etc/msmtprc
- /etc/msmtp/*
- rclone configs
- Cloudflare tokens
- DB dumps
- VM disks
- backup archives
- For handoff, only share the two truth files and selected non-secret proof files.
## PARKED_DOMAINS_STAGE4_DNS01_PREFLIGHT_FAILED_20260701
- Context: parked-domain public apply proof 634.
- Issue: DNS-01 based apply was attempted before a clear CAN_APPLY=yes preflight result was visible in terminal output.
- Observed: stage4 stopped safely with preflight_failed; CERTBOT_PRESENT=no, CERTBOT_DNS_CLOUDFLARE_PLUGIN=no, and CF_ZONE_ACCESS=no for newfi.ru, hapusya.ru and kingofwolk.ru.
- Impact: no parked-domain NPMplus route/certificate changes were applied by stage4; public HTTPS validation still failed certificate verification.
- Rule: do not use Cloudflare DNS-01 for these parked domains unless a token with explicit zone access is installed; because DNS already points to edge, prefer a copied-script HTTP-01 flow with temporary ACME challenge routes and content-specific validation.
## PARKED_DOMAINS_STAGE5_BASH_LOCAL_SETU_BUG_20260701
- Context: parked-domain HTTP-01 apply proof 635.
- Issue: edge script used Bash `local id="$1" ... conf="$WORK/.../$id.conf"` and `local host="$1" ... tmp="$WORK/.../$host.html"` under `set -u`; dependent variables are not safe inside the same local assignment command.
- Observed: `id: unbound variable`, `source can not be empty`, then `host: unbound variable`; certbot did not run, final HTTPS routes were not installed.
- Impact: local parked page remained OK; public HTTPS remained not closed.
- Rule: split dependent Bash local assignments into separate statements, syntax-check copied scripts before execution, and keep content-specific validation for parked routes.
## PARKED_DOMAINS_STAGE6_TEMP_HTTP_ROOT_VALIDATION_BUG_20260701
- Context: parked-domain HTTP-01 fixed apply proof 636.
- Issue: temporary HTTP-01 route validation checked HTTP `/` for parked marker, but expected behavior before certificates is root HTTP 301 to HTTPS.
- Observed: LOCAL_HTTP_VALIDATE returned HTTP_CODE=301 MARKER_OK=no, script rolled back its temporary route configs and did not run certbot.
- Impact: local parked page remained OK; public HTTPS remained not closed.
- Rule: validate `/.well-known/acme-challenge/<token>` for HTTP-01 readiness, not HTTP `/`; also scan all active NPMplus nginx config dirs for unmanaged domain conflicts before writing routes.
## PARKED_DOMAINS_STAGE7_NEWFI_ROOT_ACME_REDIRECT_20260701
- Context: parked domains HTTP-01 Stage7 stopped before certbot because ACME validation path returned 301 for root newfi.ru while www.newfi.ru, hapusya.ru, www.hapusya.ru, kingofwolk.ru and www.kingofwolk.ru returned 200.
- Impact: no final parked-domain HTTPS routes/certificates were applied by Stage7; temporary route configs were rolled back.
- Rule: do not run another apply stage until active NPMplus nginx route selection for newfi.ru is diagnosed with nginx -T and Host-header probes.
## PARKED_DOMAINS_STAGE8_DEFAULT_HTTP_TO_HTTPS_REDIRECT_20260701
- Context: parked-domain route autopsy proof 638.
- Finding: no active NPMplus config contains newfi.ru/hapusya.ru/kingofwolk.ru, but unknown HTTP hosts and ACME paths return 301 to HTTPS from NPMplus default routing.
- Impact: HTTP-01 cannot work through the default server. A dedicated temporary ACME server block must be proven before certbot; do not run another certbot apply without a successful ACME-path probe for all six hostnames.
- Rule: after every parked-domain temp route attempt, rollback the route and prove rollback before continuing.
## PARKED_DOMAINS_STAGE9_NEWFI_EXACT_PROBE_NEEDED_20260701
- Context: parked domains HTTP-01 probe stage9 installed one combined temporary server block for all six hostnames.
- Issue: five hostnames returned ACME probe 200, but root newfi.ru still returned 301.
- Impact: do not continue certbot/apply until exact newfi.ru route behavior is isolated.
- Rule: run a temporary exact newfi.ru probe block and roll it back before any HTTP-01 apply.
## PARKED_DOMAINS_STAGE10_EXACT_NEWFI_PROBE_OK_20260701
- Context: parked-domain Stage10 proof 640.
- Result: exact server block for newfi.ru on NPMplus intercepted both root and ACME paths with HTTP 200 and custom X-Parked-Probe markers.
- Stage9 combined server_name block passed 5/6 but failed bare newfi.ru, so final HTTP-01 cutover must use separate exact per-host server blocks.
- Rule: do not use one combined parked-domain server_name block for these domains; use exact host blocks and content-specific validation.
## PARKED_DOMAINS_STAGE11_LOW_ID_EXACT_ROUTE_FAILED_20260701
- Context: parked-domain Stage11 proof 641.
- Result: exact per-host route files 700-705 passed nginx -t but bare newfi.ru still hit HTTP->HTTPS redirect on ACME path.
- Earlier Stage10 proof showed exact newfi.ru route works when placed at high id 998.
- Hypothesis: file/include priority or NPMplus generated ordering makes low id 700 unsuitable for bare newfi.ru in this runtime.
- Rule: do not retry parked-domain HTTP-01 with low id 700-705. First prove fixed high-id exact blocks, then apply with those exact ids.
## PARKED_DOMAINS_SWITCH_TO_DNS01_AFTER_HTTP01_FAILURES_20260701
- Context: parked domains newfi.ru, hapusya.ru and kingofwolk.ru.
- HTTP-01 attempts hit NPMplus default HTTP-to-HTTPS redirect/include-order problems.
- Stage12 cleanup confirmed temp files 980-985 absent and nginx reload OK.
- Rule: do not continue HTTP-01 for these parked domains; use DNS-01 with a dedicated Cloudflare token that has Zone Read and DNS Edit for the three zones.
## PARKED_DOMAINS_STAGE15_FALSE_LOCAL_VALIDATE_ROLLBACK_20260701
- Context: Stage15 DNS-01 succeeded and public validation showed parked page for most hosts, but local edge curl --resolve validation returned CURL_RC=60 and triggered rollback.
- Impact: do not rerun Stage15 as-is.
- Rule: for parked domains finalization, use existing DNS-01 certificates and validate from pve01 public HTTPS with content marker; do not use the failed local --resolve TLS check as the closure gate.
## DOMAIN_ACTIONS_STAGE18_PLACEHOLDER_AND_RENEWAL_PROOF_20260701
- Context: after parked domains and portfolio audit, operator requested removing domain enumeration from placeholder and checking certificate renewal paths.
- Rule: Stage18 must not change DNS records or NPMplus route files. It may update placeholder HTML, run existing forum renewal script once for runtime proof, and run certbot dry-run for parked certificates using existing DNS-01 hooks.
## GRAM1_ROOT_WWW_PLACEHOLDER_CLOSE_STAGE19_20260701
- Context: operator requested gram1.ru root/www to use the existing placeholder page.
- Rule: only root gram1.ru and www.gram1.ru may be routed. Existing gram1.ru subdomain NPMplus routes must not be changed.
- Rule: use DNS-01 with a dedicated gram1 Cloudflare token; do not use HTTP-01.
- Stage18 parked dry-run note: kingofwolk.ru dry-run hit Let's Encrypt rateLimited/service-busy after active certificate issuance and public HTTPS were already OK. Treat as transient external dry-run failure, not route/cert closure blocker.
## PVEPRO_EDGE_LANDING_STAGE21_20260701
- Context: pvepro.ru root/www currently point to the Mailcow VPS and return Mailcow with TLS hostname mismatch.
- Decision: because pve01 has no SSH key access to the external VPS, do not modify the VPS. Instead move only pvepro.ru and www.pvepro.ru A records to edge and serve an edge landing page.
- Safety rule: do not change mail.pvepro.ru, nb.pvepro.ru, MX, SPF, DKIM, DMARC or forum SMTP configuration.
- Use a dedicated Cloudflare token for pvepro.ru. Do not print token values.
## PVEPRO_STAGE21_LANDING_SERVICE_FAILED_20260701
- Context: Stage21 stopped safely before DNS/cert/NPMplus route because local pvepro landing service returned CURL_RC=7 on 127.0.0.1:18089.
- Impact: DNS root/www pvepro.ru remained on 185.139.214.215 and no route/cert changes were applied.
- Rule: Stage22 must fix and prove the local landing service first, then proceed to DNS-01 cert, NPMplus route and root/www DNS only after local marker PVEPRO_LANDING_OK is reachable.
## PVEPRO_STAGE22_VALIDATION_AND_COPY_GUARD_20260701
- Context: Stage22 succeeded on edge-side landing, cert, NPMplus route and Cloudflare A upserts, but wrapper had a non-critical scp wildcard failure due __pycache__ and public validation used cached DNS for root pvepro.ru.
- Mistakes recorded: do not scp scripts/* after local py_compile because __pycache__ may be present; do not continue after critical scp failure in future apply stages; do not classify PVEPro landing as NetBird just because it contains a link text; use Cloudflare authoritative A records and --resolve validation when DNS propagation may lag.
- Stage23 must be read-only for infrastructure and may close only if edge route, Cloudflare A records, HTTPS landing marker, and mail/nb unchanged checks pass.
## TAFTAUTO_CERT_AUTORENEW_BLOCKED_NO_PRIVATE_ACCESS_20260701
- Context: taftauto.ru points to the dacha router public IP. Operator confirmed the router model is Netcraze-like, same family as Moscow router.
- Current limitation: there is no safe private remote access path to the dacha router yet.
- Rule: do not expose router admin publicly and do not attempt certificate deployment to the router over the public internet.
- Closure status: certificate autodeploy is intentionally blocked until VPN/NetBird/WireGuard or another private management path exists.
## TAFTAUTO_CERT_AUTORENEW_BLOCKED_NO_PRIVATE_ACCESS_20260701
- Context: taftauto.ru points to the dacha router public IP. Operator confirmed the router model is Netcraze-like, same family as Moscow router.
- Current limitation: there is no safe private remote access path to the dacha router yet.
- Rule: do not expose router admin publicly and do not attempt certificate deployment to the router over the public internet.
- Closure status: certificate autodeploy is intentionally blocked until VPN/NetBird/WireGuard or another private management path exists.
## TAFTAUTO_WG_STAGE24_SECRET_AND_IMPORT_RULE_20260701
- Context: configuring private management path for taftauto.ru dacha router.
- Rule: do not print WireGuard private keys, PSK, or client config body in proof/chat.
- Rule: generated client config may be stored only as root/debian mode 600 file and must be imported into the Netcraze router UI manually.
- Rule: do not open router public admin or certificate deployment until WireGuard handshake is proven.
## TAFTAUTO_WG_PSK_ROTATION_SCRIPT_BROKE_DATAPLANE_20260702
- Assistant gave unsafe PSK rotation flow; after profile reimport dacha interface changed from Wireguard0 to Wireguard1 and data-plane broke until security-level, ACL and routes were restored on Wireguard1.
- Proof 666 is superseded because it showed PUBLIC_SSH_22_STILL_OPEN.
- Final fixed proof: /root/evidence/667_TAFTAUTO_WG_PSK_ROTATED_PUBLIC_CLOSED_OK_20260702_PROOF.txt
## TAFTAUTO_CERTBOT_CLOUDFLARE_PLUGIN_MISSING_20260702
- Certbot on edge rejected --dns-cloudflare-* arguments: dns-cloudflare plugin missing/not loaded.
- No taftauto certificate issued in failed run; no DNS A records or NPMplus routes changed.
## HOMEPAGE_SERVICES_YAML_BAD_INDENT_20260702
- Assistant inserted Homepage services.yaml entries with wrong indentation; YAMLException bad indentation at line 227.
- Fix: rebuild SmartApe under External Homelabs with 2-space item indentation, remove Router Moscow, rebuild Sites category.
## HOMEPAGE_SERVICES_YAML_BAD_INDENT_REPAIR_20260702
- Previous apply broke services.yaml indentation and did not follow YAML-aware rule.
- Repair uses latest valid services.yaml backup, PyYAML parse/dump, validates before restart.
## TAFTAUTO_CERTBOT_DRYRUN_BAD_FLAG_20260702
- Assistant used unsupported certbot flag --manual-public-ip-logging-ok with certbot 4.0.0 during taftauto renew dry-run.
- No certificate, DNS, NPMplus route, or deploy state changed by the failed dry-run.
## TAFTAUTO_CERT_RENEW_DRYRUN_RATE_LIMITED_20260702
- Proof 672 is superseded/partial: it printed STATUS=OK even though certbot dry-run failed.
- Failure reason: Let's Encrypt staging/service returned rateLimited / Service busy; retry later.
- Cloudflare manual auth and cleanup hooks did run successfully.
- Deploy hook was installed and manually invoked successfully before the dry-run.
- Do not retry immediately.
## HOMEPAGE_XENFORO_INFO_INLINE_PYTHON_SYNTAX_ERROR_20260702
- Assistant used invalid inline Python syntax with def after semicolon while adding XenForo.info to Homepage External.
- Failed before services.yaml write.
## HOMEPAGE_USEFUL_ROUTER_NAME_MISMATCH_20260702
- Previous apply looked for Russian Router card name, but active Homepage Useful group contains Router and Public Domain.
- Failed before write. Fix uses YAML-aware edit with exact names Router/Public Domain.
## HOMEPAGE_USEFUL_ROUTER_CARD_NOT_FOUND_20260702
- Apply expected Useful -> Router in active services.yaml, but YAML parse did not find it. No write was performed.
- Need verify active Homepage config path, docker mount, and any alternate services.yaml before next apply.
## 20260702_CF_TOKEN_AUDIT_BROKEN_COMMAND
- Ошибка: ассистент дал хрупкую команду аудита Cloudflare-токенов с вложенным heredoc/Python через ssh.
- Факт: bash начал интерпретировать Python-код, появились syntax error и вторичный prompt >.
- Правило: не давать nested heredoc/сложные кавычки через ssh; для длинной логики сначала класть скрипт отдельным файлом, затем запускать его.
- Статус: аудит токенов не выполнен, инфраструктурные изменения не должны считаться применёнными.
## LESSON_20260702_CROWDSEC_NETBIRD_EXIT_ROUTE
- Do not treat e3qxxx.netbird.selfhosted / 100.100.125.70 and e3qxxx-183-106.netbird.selfhosted / 100.100.183.106 as VPS egress nodes; they are user mobile peers.
- Correct egress candidates are relay.pvepro.ru / relay.netbird.selfhosted / 100.100.19.1 for Moldova and mail.pvepro.ru / mail.netbird.selfhosted / 100.100.147.204 for USA.
- Access Policy edge-vm -> relay/mail only proves peer reachability. It is not an internet exit route.
- Required NetBird fix for CrowdSec CAPI was Network Routing exit route 0.0.0.0/0 distributed to edge-vm group, routing peer relay, masquerade/NAT enabled.
- Before attempting CrowdSec CAPI, verify on edge-vm: ip route get 1.1.1.1 uses wt0, wg allowed-ips contains 0.0.0.0/0, and Cloudflare trace no longer shows home IP 95.84.154.183.
## LESSON_20260702_HOMEPAGE_MONITORS_REPAIR_NOT_DELETE
- Do not delete or disable Homepage siteMonitor fields to hide red badges.
- Fix the underlying monitor path, DNS, proxy route, or local health-wrapper instead.
- Do not touch Cloudflare Homepage card when operator says it is green and opens correctly.
- Before changing Homepage, identify the live config host; in this state live Homepage is on core-apps [PRIVATE_IP], not edge-vm.
## LESSON_20260702_KUMA_ADD_ONE_MONITOR_ONLY
- When asked to fix one missing Uptime Kuma service, add only one monitor, then generate proposals separately.
- Before direct Kuma DB mutation, stop the container and create a DB backup.
- Verify DB integrity before starting Kuma again.
- Do not touch Cloudflare when operator says it is green and opens correctly.
## LESSON_20260702_DOCKGE_STALE_STACKS_NOT_CONTAINERS
- Dockge inactive items after migration can be stale compose folders, not stopped containers.
- First classify runtime projects across all Docker hosts before deleting or archiving anything.
- For remote Dockge visibility, projects must be under /opt/stacks or bind-mounted there.
- Do not delete containers to fix Dockge inactive UI; connect agents and archive only confirmed stale moved stack definitions.
## STAGE4B_MASKED_SQL_QUERY_FAILURE_20260714
- Previous Stage4B read-only audit returned empty observation-summary, metadata-key, trigger and function blocks because nested SSH/SQL quoting broke queries while stderr was hidden.
- Empty blocks are query failures, not proof that metadata, triggers or functions are absent.
- Future PostgreSQL audits must use ON_ERROR_STOP, visible stderr, independent query RC and quote-safe dollar literals.
## STAGE4C_SCHEMA_MIGRATIONS_ID_ASSUMPTION_20260714
- Stage4C read-only preflight incorrectly assumed schema_migrations has a column named id.
- PostgreSQL returned column id does not exist and ON_ERROR_STOP correctly stopped all following SQL checks.
- Production DB and application were not changed. Future migration audits must inspect information_schema first and query migration rows without assuming column names.
## STAGE4C_PGDUMP_DEV_NULL_FSYNC_INVALID_PROBE_20260714
- Stage4C custom-format pg_dump capability probe incorrectly used /dev/null as the output file.
- pg_dump failed only because fsync on /dev/null returned Invalid argument.
- Production database and application were not changed.
- Future custom dump capability checks must use a real temporary regular file, validate it with pg_restore --list, and remove it afterward.
## STAGE4C_REMOTE_PREP_THIRD_SUBSTEP_FAILED_20260714
- Isolated migration dry-run did not start: REMOTE_PREP_RC=0|0|1.
- Remote directory creation and SCP succeeded; the third ownership/mode preparation substep returned RC=1.
- Production database and application were not changed.
- Exact cause requires read-only residual directory and permission inspection before cleanup or retry.
## STAGE4C_REMOTE_PREP_GLOB_AFTER_CHMOD_ROOT_CAUSE_20260714
- Exact cause of REMOTE_PREP_RC=0|0|1: after the directory became postgres-owned mode 0700, the unprivileged debian shell could not traverse it and could not expand the wildcard used by chmod.
- Candidate files and SHA256 verification were valid; no temporary database was created and production was unchanged.
- Future preparation must use exact privileged paths or a verified shared group, never an unprivileged wildcard after restrictive chmod.
## STAGE4C_MIGRATION003_FINGERPRINT_DOLLAR_QUOTE_FAILURE_20260714
- Migration 003 isolated dry-run failed at fingerprint CHECK because the regex end anchor collided with the SQL dollar-quote delimiter and produced an unterminated dollar-quoted string.
- The migration transaction rolled back, the temporary database was removed, and production remained unchanged.
- Fingerprint validation must use length plus translate without a regex dollar anchor; corrected candidate must pass a fresh isolated dry-run before production consideration.
## STAGE4C_SEAL_OUTER_RC_MASKING_20260714
- Stage4C final seal correctly refused creation because overall health was WARN, returning internal RC=40.
- The former wrapper then printed a second misleading COMMAND_RC=0 because top-level true masked the internal result.
- Future commands must capture the guarded subshell through PIPESTATUS and print exactly one authoritative COMMAND_RC before the mandatory final true.
- Production database, application and services were unchanged.
## STAGE4C_DIAGNOSIS_EXPECTED_DERIVED_FILE_ON_VM180_20260714
- Previous diagnosis incorrectly treated cluster-admin-restricted-probes.txt as a required source file on VM180.
- The file is a central derived health artifact on pve01; VM180 supplies full-observer evidence instead.
- This false expectation caused diagnostic RC=32 without any infrastructure failure or production change.
## STAGE4C_NO_JOBS_RUNNING_TEXT_COUNTED_AS_JOB_20260714
- Stage4C seal preflight incorrectly counted the systemctl informational text No jobs running as one active job.
- The parser counted every nonempty output line instead of accepting only rows beginning with a numeric systemd Job ID.
- No service was started and no production state changed during the blocked attempt.
- Future job counts must match a numeric first field only.
## STAGE4D_ADAPTER_SOURCE_AUDIT_ABORTED_BEFORE_FINAL_MARKERS_20260714
- The evidence-only Stage4D adapter source audit returned RC=1 before emitting its VM180 completion and final audit markers.
- The six wrapper validation failures are cascading missing-marker checks and do not prove six independent infrastructure faults.
- Production, database, application, services, timers, health and desired-state were unchanged.
- Inspect the preserved runner stdout, stderr and result JSON before correcting or rerunning the task.
## STAGE4D_LOCAL_EXPANSION_OF_REMOTE_Q_UNDER_NOUNSET_20260714
- The first Stage4D adapter source audit embedded a large remote program inside a locally double-quoted SSH argument.
- Local shell expansion reached the remote variable Q while nounset was enabled, causing Q: unbound variable before SSH execution.
- VM180 and PostgreSQL audits did not start; production state was unchanged.
- Remote shell and SQL programs must be separate syntax-checked files passed through SSH stdin.
## STAGE4D_REMOTE_PYTHON311_ADAPTER_SELFTEST_RC1_20260714
- Canonical adapter candidate passed local Python 3.13 self-tests with 48 envelopes but returned RC=1 on VM180 Python 3.11.2.
- Remote upload and cleanup succeeded, and production database remained 0|0|OK.
- The exact traceback and failing assertion must be captured by an isolated identical reproduction before changing the candidate.
## STAGE4D_PYTHON311_FSTRING_COMPATIBILITY_AND_REPRO_VALIDATOR_20260714
- Adapter candidate passed Python 3.13 but failed Python 3.11 parsing at Path(row["path"]) inside a double-quoted f-string.
- Exact fix is Python 3.11-compatible quoting: Path(row['path']).
- The reproduction command captured the expected compile failure but incorrectly returned CAPTURE_BAD=1 because its validator required compile RC=0.
- Production database remained 0|0|OK and desired-state remained clean.
- Future adapter candidates must pass syntax and full self-tests on both pve01 and VM180 runtimes before acceptance.
## STAGE4E_DESIGN_VALIDATOR_ABORTED_BEFORE_LOCAL_VALIDATION_MARKER_20260714
- Restricted probe-agent design candidate compiled successfully but its local validator exited before producing LOCAL_VALIDATION_RC.
- The wrapper validation count is cascading missing-marker evidence and does not represent twenty-one independent faults.
- Active external probes were not executed and production state was unchanged.
- Inspect the preserved validator traceback and exact assertion before modifying the candidate.
## STAGE4E_VALIDATOR_SCANNED_PYC_AND_OWN_FORBIDDEN_LITERALS_20260714
- Stage4E design validator recursively read every candidate file as UTF-8 after py_compile had created binary __pycache__ bytecode.
- This caused UnicodeDecodeError before any design assertion failed.
- The phase-wide forbidden-pattern scan also included validate_design.py itself, which necessarily contains the prohibited literals it verifies.
- Validators must use an explicit text-file allowlist and scans must exclude validator implementation and binary artifacts.
- Production, database, services, timers and desired-state were unchanged; active external probes were not executed.
## STAGE4E_V2_VALIDATOR_FAILED_STATIC_COMPILE_20260714
- Stage4E design candidate v2 failed static compilation before local design validation started.
- The reported validation count is cascading missing-marker evidence, not twenty-five independent design defects.
- The generated validator must be inspected at the exact SyntaxError line before another candidate is created.
- Production, database, application, services, timers, health and desired-state were unchanged; active external probes were not executed.
## STAGE4E_PIPESTATUS_LOST_AFTER_FIRST_ASSIGNMENT_20260714
- VM180 validation copy used a pipeline and read PIPESTATUS in two separate assignments.
- The first assignment reset PIPESTATUS, leaving COPY_RC empty and preventing remote validation and cleanup.
- Retry must use explicit SCP operations without pipeline status parsing.
- Production remained unchanged and active external probes were not executed.
## STAGE4F_VALIDATOR_EXPECTED_HEALTH_REFRESH_FROM_NONWRITING_BACKUP_SCRIPT_20260714
- The controlled backup service completed with Result=success and ExecMainStatus=0.
- Validation incorrectly required cluster-admin-incident-engine-backup.txt to receive a fresh timestamp.
- The executed backup script does not write that health file; therefore the stale health timestamp was not evidence of backup failure.
- Stage4F backup proof must use the service result, fresh latest JSON, produced artifacts and off-host/restore evidence instead.
- The backup must not be rerun merely to satisfy the invalid health-file freshness assertion.
## STRICT_RULE_20260714_CLOSE_TAILS_IMMEDIATELY
- Failure class: переход к следующей задаче при наличии незакрытого хвоста.
- Mandatory anti-regression: после ошибки запрещено продолжать другой scope до точной root cause, исправления, повторной проверки, очистки, proof и seal.
- Closure gate: UNRESOLVED_TAIL_COUNT=0; BLOCKER_COUNT=0; TEMPORARY_ARTIFACT_COUNT=0; ROLLBACK_VERIFIED=YES; HEALTH=OK; PROOF_UPDATED=YES; REFERENCE_UPDATED=YES.
- Если закрытие невозможно из-за внешней зависимости, опасной неоднозначности или обязательного решения пользователя, фиксировать BLOCKED/OPEN и не заявлять CLOSED.
- Proof: /var/lib/homelab-change-sets/operator-rule-close-tails-immediately-v1/20260714T171631Z/report.txt
## ERROR_20260714_DEPENDENCY_FACT_QUERY_GLOBAL_LIMIT
- Symptom: dependency audit вернул только latest_collector_status.
- Root cause: ORDER BY 1 LIMIT 1 в конце UNION ALL применился ко всему набору фактов.
- Correction: latest collector status извлекается скалярным подзапросом внутри CTE facts; глобальный LIMIT отсутствует.
- Anti-regression: multi-fact SQL audit обязан проверять точное ожидаемое число строк до извлечения значений.
- Closure proof: /var/lib/homelab-cluster-admin-stage4/stage4h-collector-provenance-readiness-scope-v1/report.txt
- Status: CLOSED.
## ERROR_20260714_COLLECTOR_PATCH_REQUIREMENT_PATTERN_MISMATCH
- Symptom: MIGRATION_PLAN_COLLECTOR_PATCH_REQUIRED_COUNT=1 при ожидаемых двух подтверждениях.
- Root cause: узкий шаблон учитывал collector_patch_required, но не collector_patch_required_after_stage4c.
- Correction: оба поля проверяются отдельно, затем складываются как semantic requirement count.
- Anti-regression: связанные JSON-контракты проверяются по собственным точным именам полей, а не одним унифицированным grep-шаблоном.
- Closure proof: /var/lib/homelab-cluster-admin-stage4/stage4h-collector-provenance-readiness-scope-v1/report.txt
- Status: CLOSED; PRIOR_TAIL_UNRESOLVED_COUNT=0.
## ERROR_20260714_PRIVILEGED_FILE_REDIRECTION_BEFORE_SUDO
- Symptom: bash reported Permission denied while counting collector.py lines.
- Root cause: `< collector.py` was opened by the unprivileged caller shell before sudo executed wc.
- Correction: run sudo wc -l collector.py without caller-side input redirection.
- Anti-regression: never combine sudo with `< protected-file`; pass the protected path as an argument to the privileged command.
- Closure proof: /var/lib/homelab-cluster-admin-stage4/stage4h-collector-baseline-audit-final-v1/report.txt
- Status: CLOSED.
## ERROR_20260714_COLLECTOR_RUN_COLUMN_ASSUMPTION
- Symptom: SQL failed because completed_at did not exist.
- Root cause: collector_runs columns were assumed instead of read from information_schema; actual names are finished_at and error_text.
- Correction: assert required and forbidden column counts before querying recent runs.
- Anti-regression: every schema-sensitive audit must discover and validate exact column names before the main query.
- Closure proof: /var/lib/homelab-cluster-admin-stage4/stage4h-collector-baseline-audit-final-v1/report.txt
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
## ERROR_20260714_REMOTE_STDIN_ARGUMENT_SHIFT
- Symptom: remote harness выполнил chmod для пути bash.
- Root cause: дополнительный positional token bash был передан после SSH command и стал первым аргументом stdin-скрипта.
- Correction: remote command вызывает bash -s -- с ровно тремя явно встроенными аргументами: candidate path, SHA256 и byte count.
- Anti-regression: stdin-скрипт обязан валидировать $1 по разрешённому path pattern и затем подтверждать точные SHA256 и byte count; запрещено добавлять отдельный bash-token после remote command.
- Closure proof: /var/lib/homelab-change-sets/stage4h-collector-provenance-patch-candidate-v1/20260714T174131Z/report.txt
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
## ERROR_20260714_OVERSIZED_COMMAND_PARSE_FAILURE
- Symptom: Bash получил одиночный фрагмент и сообщил syntax error near unexpected token `(`.
- Root cause: чрезмерно длинная интерактивная команда была обрезана или повреждена при передаче до полного разбора Bash.
- Impact: side-effect audit подтвердил production state 0|0|OK, неизменный collector, активный timer и отсутствие временных БД, каталогов и процессов.
- Correction: монолитные команды запрещены; сложные действия выполняются через отдельный проверяемый task-скрипт и короткий launcher.
- Anti-regression: interactive payload target <= 8000 bytes; syntax-check перед запуском; remote output сохраняется независимо от RC.
- Closure proof: /var/lib/homelab-change-sets/operator-command-size-guard-v1/20260714T182653Z/report.txt
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
## ERROR_20260714_HOMELAB_ADMIN_HELP_RC_ASSUMPTION
- Symptom: template audit завершился HOMELAB_ADMIN_HELP_FAILED.
- Root cause: был ошибочно ожидаем RC=0 от неподдерживаемого аргумента --help.
- Actual contract: неизвестный аргумент печатает usage и возвращает RC=64.
- Correction: проверять version либо считать точную usage-строку вместе с RC=64 штатным результатом.
- Anti-regression: перед проверкой CLI сначала фиксировать поддерживаемые команды; не предполагать GNU-style --help.
- Closure proof: /var/lib/homelab-change-sets/homelab-admin-cli-contract-fix-v1/20260714T183338Z/report.txt
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
## ERROR_20260715_EXPECTED_NEGATIVE_RC_TRAP_CLASSIFICATION
- Symptom: schema-v2 task v3 build stopped while intentionally testing the --invalid argument path.
- Root cause: an expected RC64 was executed while the generic ERR trap remained active.
- Correction: expected failures are evaluated only through an explicit if/else branch with immediate RC capture.
ANTI_REGRESSION_TOKEN=EXPECTED_RC64_INTERCEPTED_ERR_TRAP
- Required pattern: if command; then rc=0; else rc=$?; fi, followed by a separate exact expected-RC assertion.
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
## ERROR_20260715_DUPLICATE_MACHINE_MARKER_COUNT
- Symptom: error-register candidate construction stopped with RC1 before applying the candidate.
- Root cause: the same marker text appeared in both a heading and a descriptive line, while the guard expected one substring occurrence.
- Correction: headings and prose do not contain machine-token values; validation counts only complete exact token lines.
ANTI_REGRESSION_TOKEN=DUPLICATE_MACHINE_TOKEN_IN_HEADING_AND_BODY
- Required pattern: grep -Fxc against a complete machine line, never grep -Foc against an unrestricted substring.
- Production impact: none.
- Temporary artifacts: removed and verified.
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
- Registered at: 2026-07-15T02:56:19Z
## ERROR_20260715_OUTER_WRAPPER_SHELL_SYNTAX
- Symptom: the operator shell rejected the generated one-line wrapper with syntax error near the final subshell parenthesis.
- Root cause: the outer wrapper contained an unbalanced compound shell construct before the closing subshell delimiter.
- Impact: parsing failed before the start marker, guards and homelab-admin invocation; no task phase or infrastructure action executed.
- Correction: use a linear wrapper with explicitly closed if/case blocks and avoid nested brace groups in conditional expressions.
ANTI_REGRESSION_TOKEN=OUTER_WRAPPER_UNBALANCED_SUBSHELL_SYNTAX
- Anti-regression: after any parser-level failure, treat the attempted phase as not executed and verify package and run-state before retrying.
- Production impact: none.
- Task package impact: none.
- Temporary artifacts: none.
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
- Registered at: 2026-07-15T03:24:05Z
## ERROR_20260715_INLINE_REMOTE_SHELL_IN_APPLY_PHASE
- Symptom: homelab-admin lint rejected Stage4H task v6 with ERR-002-INLINE-REMOTE-SHELL.
- Root cause: apply.sh invoked bash -s through an inline quoted SSH program and streamed the remote script through stdin.
- Correction: copy the declared remote script to an isolated fixed path on VM180 and invoke that file directly with positional arguments.
ANTI_REGRESSION_TOKEN=INLINE_REMOTE_SHELL_REPLACED_BY_STAGED_REMOTE_SCRIPT_PATH
- Anti-regression: phase scripts may perform transport orchestration, but remote shell program bodies must exist as separately declared, syntax-checked remote script files.
- Remote stdout and stderr must be preserved before evaluating the remote return code.
- Task v6 status: REJECTED_BY_LINT; never executed and never mutated.
- Production database impact: none.
- Live collector impact: none.
- Temporary database and remote root count after rejection: zero.
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
- Registered at: 2026-07-15T04:13:46Z
## ERROR_20260715_OPTIONAL_RUN_DIRECTORY_FIND_UNDER_PIPEFAIL
- Symptom: task-v7 builder stopped while counting formal runs for rejected task v6.
- Root cause: find was executed against an optional directory that did not exist; under pipefail the assignment returned RC1.
- Correction: test directory existence first and assign zero without invoking find when it is absent.
ANTI_REGRESSION_TOKEN=MISSING_OPTIONAL_RUN_DIRECTORY_TREATED_AS_ZERO
- Anti-regression: optional paths must have an explicit existence branch before find under pipefail.
- Production impact: none.
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
- Registered at: 2026-07-15T04:44:06Z
## ERROR_20260715_PG_CONSTRAINT_CONTYPE_CHAR_CONCAT
- Symptom: isolated Stage4H acceptance failed during schema baseline capture.
- Root cause: pg_constraint.contype uses the internal PostgreSQL char type and was concatenated without an explicit text cast.
- Correction: use contype::text or CAST(contype AS text).
ANTI_REGRESSION_TOKEN=PG_CATALOG_CHAR_CONCAT_REQUIRES_EXPLICIT_TEXT_CAST
- Machine rule: ERR-014-PG-CATALOG-CHAR-CONCAT.
- Negative self-test: uncast expression rejected with RC64.
- Positive self-test: explicit text cast accepted with RC0.
- Task v7 mutated: no.
- Production impact: none.
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
- Registered at: 2026-07-15T06:27:06Z
## ERROR_20260715_LIVE_COLLECTOR_VERIFIER_WRONG_PATH
- Symptom: verified linter installation was rolled back after the final collector integrity guard returned UNKNOWN.
- Root cause: verifier queried a nonexistent path instead of the canonical live collector path declared by the task acceptance script.
- Canonical path: /opt/cluster-admin-incident-engine/collector.py.
- Incorrect path: /opt/homelab-cluster-admin/cluster-admin-collector.py.
ANTI_REGRESSION_TOKEN=LIVE_COLLECTOR_VERIFIER_PATH_MUST_MATCH_TASK_CANONICAL_PATH
- Correction: derive and guard the live collector path from the immutable task contract before executing the hash query.
- Production impact: none; the canonical collector hash remained unchanged.
- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
- Registered at: 2026-07-15T06:27:06Z
----- CONTENT END -----
===== PROXMOX VERSION =====
proxmox-ve: 9.2.0 (running kernel: 7.0.12-1-pve)
pve-manager: 9.2.3 (running version: 9.2.3/d0fde103346cf89a)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0: 7.0.12-1
proxmox-kernel-7.0.12-1-pve-signed: 7.0.12-1
proxmox-kernel-7.0.6-2-pve-signed: 7.0.6-2
proxmox-kernel-7.0.2-6-pve-signed: 7.0.2-6
ceph-fuse: 19.2.3-pve4
corosync: 3.1.10-pve2
criu: 4.1.1-1
frr-pythontools: 10.6.1-1+pve2
ifupdown2: 3.3.0-1+pmx12
intel-microcode: 3.20251111.1~deb13u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.1
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.1
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.1.6
libpve-cluster-perl: 9.1.6
libpve-common-perl: 9.1.16
libpve-guest-common-perl: 6.0.4
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.6
libpve-notify-perl: 9.1.6
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.6
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-1
proxmox-backup-client: 4.2.2-1
proxmox-backup-file-restore: 4.2.2-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.3
proxmox-kernel-helper: 9.2.0
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.5
pve-cluster: 9.1.6
pve-container: 6.1.10
pve-docs: 9.2.2
pve-edk2-firmware: 4.2025.05-2
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.18-4
pve-ha-manager: 5.2.4
pve-i18n: 3.8.0
pve-qemu-kvm: 11.0.0-4
pve-xtermjs: 6.0.0-1
qemu-server: 9.1.17
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.2-pve1
===== CLUSTER STATUS =====
Cluster information
-------------------
Name: homelab
Config Version: 3
Transport: knet
Secure auth: on
Quorum information
------------------
Date: Tue Jul 21 09:11:09 2026
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000001
Ring ID: 1.9d
Quorate: Yes
Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 [PRIVATE_IP] (local)
0x00000002 1 [PRIVATE_IP]
0x00000003 1 [PRIVATE_IP]
===== CLUSTER NODES =====
[
{
"id" : "cluster",
"name" : "homelab",
"nodes" : 3,
"quorate" : 1,
"type" : "cluster",
"version" : 3
},
{
"id" : "node/pve03",
"ip" : "[PRIVATE_IP]",
"level" : "",
"local" : 0,
"name" : "pve03",
"nodeid" : 2,
"online" : 1,
"type" : "node"
},
{
"id" : "node/pve01",
"ip" : "[PRIVATE_IP]",
"level" : "",
"local" : 1,
"name" : "pve01",
"nodeid" : 1,
"online" : 1,
"type" : "node"
},
{
"id" : "node/pve02",
"ip" : "[PRIVATE_IP]",
"level" : "",
"local" : 0,
"name" : "pve02",
"nodeid" : 3,
"online" : 1,
"type" : "node"
}
]
===== CLUSTER RESOURCES =====
[
{
"maxcpu": 1,
"maxdisk": 8350298112,
"maxmem": 536870912,
"name": "dns1",
"node": "pve01",
"plugintype": null,
"status": "running",
"storage": null,
"template": 0,
"type": "lxc",
"vmid": 110
},
{
"maxcpu": 1,
"maxdisk": 8350298112,
"maxmem": 536870912,
"name": "dns2",
"node": "pve02",
"plugintype": null,
"status": "running",
"storage": null,
"template": 0,
"type": "lxc",
"vmid": 111
},
{
"maxcpu": 1,
"maxdisk": 8350298112,
"maxmem": 536870912,
"name": "unbound1",
"node": "pve01",
"plugintype": null,
"status": "running",
"storage": null,
"template": 0,
"type": "lxc",
"vmid": 112
},
{
"maxcpu": 1,
"maxdisk": 8350298112,
"maxmem": 536870912,
"name": "unbound2",
"node": "pve02",
"plugintype": null,
"status": "running",
"storage": null,
"template": 0,
"type": "lxc",
"vmid": 113
},
{
"maxcpu": 4,
"maxdisk": 103079215104,
"maxmem": 12884901888,
"name": "edge-vm",
"node": "pve03",
"plugintype": null,
"status": "running",
"storage": null,
"template": 0,
"type": "qemu",
"vmid": 130
},
{
"maxcpu": 4,
"maxdisk": 68719476736,
"maxmem": 8589934592,
"name": "nextcloud",
"node": "pve01",
"plugintype": null,
"status": "running",
"storage": null,
"template": 0,
"type": "qemu",
"vmid": 150
},
{
"maxcpu": 4,
"maxdisk": 214748364800,
"maxmem": 8589934592,
"name": "forum-prod",
"node": "pve02",
"plugintype": null,
"status": "running",
"storage": null,
"template": 0,
"type": "qemu",
"vmid": 160
},
{
"maxcpu": 4,
"maxdisk": 128849018880,
"maxmem": 10737418240,
"name": "core-apps",
"node": "pve01",
"plugintype": null,
"status": "running",
"storage": null,
"template": 0,
"type": "qemu",
"vmid": 170
},
{
"maxcpu": 4,
"maxdisk": 128849018880,
"maxmem": 8589934592,
"name": "monitoring",
"node": "pve01",
"plugintype": null,
"status": "running",
"storage": null,
"template": 0,
"type": "qemu",
"vmid": 171
},
{
"maxcpu": 2,
"maxdisk": 25769803776,
"maxmem": 2147483648,
"name": "cluster-admin",
"node": "pve02",
"plugintype": null,
"status": "running",
"storage": null,
"template": 0,
"type": "qemu",
"vmid": 180
},
{
"maxcpu": 4,
"maxdisk": 85899345920,
"maxmem": 8589934592,
"name": "homelab-ops-worker",
"node": "pve03",
"plugintype": null,
"status": "running",
"storage": null,
"template": 0,
"type": "qemu",
"vmid": 190
},
{
"maxcpu": 4,
"maxdisk": 103079215104,
"maxmem": 12884901888,
"name": "edge-cold-standby",
"node": "pve02",
"plugintype": null,
"status": "stopped",
"storage": null,
"template": 0,
"type": "qemu",
"vmid": 9130
},
{
"maxcpu": 4,
"maxdisk": 105089261568,
"maxmem": 16536748032,
"name": null,
"node": "pve02",
"plugintype": null,
"status": "online",
"storage": null,
"template": null,
"type": "node",
"vmid": null
},
{
"maxcpu": 8,
"maxdisk": 105089261568,
"maxmem": 33553518592,
"name": null,
"node": "pve03",
"plugintype": null,
"status": "online",
"storage": null,
"template": null,
"type": "node",
"vmid": null
},
{
"maxcpu": 24,
"maxdisk": 105089261568,
"maxmem": 33379028992,
"name": null,
"node": "pve01",
"plugintype": null,
"status": "online",
"storage": null,
"template": null,
"type": "node",
"vmid": null
},
{
"maxcpu": null,
"maxdisk": 105089261568,
"maxmem": null,
"name": null,
"node": "pve02",
"plugintype": "dir",
"status": "available",
"storage": "local",
"template": null,
"type": "storage",
"vmid": null
},
{
"maxcpu": null,
"maxdisk": 105089261568,
"maxmem": null,
"name": null,
"node": "pve03",
"plugintype": "dir",
"status": "available",
"storage": "local",
"template": null,
"type": "storage",
"vmid": null
},
{
"maxcpu": null,
"maxdisk": 105089261568,
"maxmem": null,
"name": null,
"node": "pve01",
"plugintype": "dir",
"status": "available",
"storage": "local",
"template": null,
"type": "storage",
"vmid": null
},
{
"maxcpu": null,
"maxdisk": 846016282624,
"maxmem": null,
"name": null,
"node": "pve02",
"plugintype": "lvmthin",
"status": "available",
"storage": "local-lvm",
"template": null,
"type": "storage",
"vmid": null
},
{
"maxcpu": null,
"maxdisk": 355660201984,
"maxmem": null,
"name": null,
"node": "pve03",
"plugintype": "lvmthin",
"status": "available",
"storage": "local-lvm",
"template": null,
"type": "storage",
"vmid": null
},
{
"maxcpu": null,
"maxdisk": 869168840704,
"maxmem": null,
"name": null,
"node": "pve01",
"plugintype": "lvmthin",
"status": "available",
"storage": "local-lvm",
"template": null,
"type": "storage",
"vmid": null
},
{
"maxcpu": null,
"maxdisk": null,
"maxmem": null,
"name": null,
"node": "pve02",
"plugintype": null,
"status": "ok",
"storage": null,
"template": null,
"type": "network",
"vmid": null
},
{
"maxcpu": null,
"maxdisk": null,
"maxmem": null,
"name": null,
"node": "pve03",
"plugintype": null,
"status": "ok",
"storage": null,
"template": null,
"type": "network",
"vmid": null
},
{
"maxcpu": null,
"maxdisk": null,
"maxmem": null,
"name": null,
"node": "pve01",
"plugintype": null,
"status": "ok",
"storage": null,
"template": null,
"type": "network",
"vmid": null
}
]
===== STORAGE STATUS =====
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
local dir active 102626232 43159780 54207188 42.06%
local-lvm lvmthin active 848797696 280103239 568694456 33.00%
===== PVE01 QEMU =====
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
150 nextcloud running 8192 64.00 10349
170 core-apps running 10240 120.00 10425
171 monitoring running 8192 120.00 10529
===== PVE01 LXC =====
VMID Status Lock Name
110 running dns1
112 running unbound1
===== HOMELAB SYSTEMD UNITS =====
dpkg-db-backup.service static -
dpkg-db-backup.timer enabled enabled
filebrowser-backup.service static -
filebrowser-backup.timer enabled enabled
homelab-alerting-health.service static -
homelab-alerting-health.timer enabled enabled
homelab-alertmanager-backup.service static -
homelab-alertmanager-backup.timer enabled enabled
homelab-appbackup-dockge-stacks.service static -
homelab-appbackup-dockge-stacks.timer enabled enabled
homelab-appbackup-mariadb-batch.service static -
homelab-appbackup-mariadb-batch.timer enabled enabled
homelab-appbackup-npmplus-edge.service static -
homelab-appbackup-npmplus-edge.timer enabled enabled
homelab-appbackup-postgres-batch.service static -
homelab-appbackup-postgres-batch.timer enabled enabled
homelab-appbackup-sqlite-batch.service static -
homelab-appbackup-sqlite-batch.timer enabled enabled
homelab-authentik-backup.service static -
homelab-authentik-backup.timer disabled enabled
homelab-auto-backup.service static -
homelab-auto-backup.timer enabled enabled
homelab-backup-coverage-matrix.service static -
homelab-backup-coverage-matrix.timer enabled enabled
homelab-backup-freshness.service static -
homelab-backup-freshness.timer enabled enabled
homelab-backup-healthcheck.service static -
homelab-backup-healthcheck.timer enabled enabled
homelab-backup-matrix-refresh.service static -
homelab-backup-matrix-refresh.timer disabled enabled
homelab-backup-sla-health.service static -
homelab-backup-sla-health.timer enabled enabled
homelab-beszel-backup.service static -
homelab-beszel-backup.timer enabled enabled
homelab-blackbox-exporter-backup.service static -
homelab-blackbox-exporter-backup.timer enabled enabled
homelab-capacity-risk.service static -
homelab-capacity-risk.timer enabled enabled
homelab-cluster-admin-incident-engine-backup.service static -
homelab-cluster-admin-incident-engine-backup.timer enabled enabled
homelab-cluster-admin-inventory-sync.service static -
homelab-cluster-admin-inventory-sync.timer enabled enabled
homelab-cluster-admin-pull.service static -
homelab-cluster-admin-pull.timer disabled enabled
homelab-cluster-admin-webpanel-sync.service static -
homelab-cluster-admin-webpanel-sync.timer enabled enabled
homelab-cluster-internal-ip.service enabled enabled
homelab-cluster-passport.service static -
homelab-cluster-passport.timer enabled enabled
homelab-crypto-portfolio-backup.service static -
homelab-crypto-portfolio-backup.timer enabled enabled
homelab-crypto-portfolio-chart-health.service static -
homelab-crypto-portfolio-chart-health.timer enabled enabled
homelab-dependency-map-check.service static -
homelab-dependency-map-check.timer enabled enabled
homelab-desired-state-sync.service static -
homelab-desired-state-sync.timer enabled enabled
homelab-disk-space-health.service static -
homelab-disk-space-health.timer enabled enabled
homelab-docker-health.service static -
homelab-docker-health.timer enabled enabled
homelab-drift-check.service static -
homelab-drift-check.timer enabled enabled
homelab-duty-admin-report.service static -
homelab-duty-admin-report.timer enabled enabled
homelab-duty-admin-v2.service static -
homelab-duty-admin-v2.timer enabled enabled
homelab-edge-vm-offhost-freshness.service static -
homelab-edge-vm-offhost-freshness.timer enabled enabled
homelab-edge-vm-vzdump-backup.service static -
homelab-edge-vm-vzdump-backup.timer disabled enabled
homelab-emergency-backup.service static -
homelab-emergency-backup.timer enabled enabled
homelab-evidence-catalog.service static -
homelab-evidence-catalog.timer enabled enabled
homelab-evidence-seal.service static -
homelab-evidence-seal.timer disabled enabled
homelab-extended-appbackup.service static -
homelab-extended-appbackup.timer enabled enabled
homelab-external-probe-vps-health.service static -
homelab-external-probe-vps-health.timer enabled enabled
homelab-final-readiness-gate.service static -
homelab-final-readiness-gate.timer enabled enabled
homelab-forum-snuffleupagus-health.service static -
homelab-forum-snuffleupagus-health.timer enabled enabled
homelab-gitea-backup.service static -
homelab-gitea-backup.timer disabled enabled
homelab-gitea-secondary-backup.service static -
homelab-gitea-secondary-backup.timer enabled enabled
homelab-golden-state-index.service static -
homelab-golden-state-index.timer enabled enabled
homelab-health-http-9101.service enabled enabled
homelab-health-http.service enabled enabled
homelab-health-metrics.service static -
homelab-health-metrics.timer enabled enabled
homelab-incident-journal.service static -
homelab-incident-journal.timer enabled enabled
homelab-ingress-standby-refresh.service static -
homelab-ingress-standby-refresh.timer disabled enabled
homelab-kuma-monitor-policy.service static -
homelab-kuma-monitor-policy.timer enabled enabled
homelab-legacy-monitoring-backups.service static -
homelab-legacy-monitoring-backups.timer enabled enabled
homelab-mail-cloud-critical-upload.service static -
homelab-mail-cloud-critical-upload.timer enabled enabled
homelab-mail-cloud-edge-vm-upload.service static -
homelab-mail-cloud-edge-vm-upload.timer enabled enabled
homelab-mail-cloud-nextcloud-vm-upload.service static -
homelab-mail-cloud-nextcloud-vm-upload.timer enabled enabled
homelab-mail-cloud-restore-drill.service static -
homelab-mail-cloud-restore-drill.timer enabled enabled
homelab-mkdocs-auto-refresh.service static -
homelab-mkdocs-auto-refresh.timer enabled enabled
homelab-npmplus-kuma-config-backup.service static -
homelab-npmplus-kuma-config-backup.timer disabled enabled
homelab-overall-health.service static -
homelab-overall-health.timer enabled enabled
homelab-private-vpn-hosts-health.service static -
homelab-private-vpn-hosts-health.timer enabled enabled
homelab-pve03-staging-capacity-health.service static -
homelab-pve03-staging-capacity-health.timer enabled enabled
homelab-quality-gate.service static -
homelab-quality-gate.timer enabled enabled
homelab-router-running-config-mail-cloud.service static -
homelab-router-running-config-mail-cloud.timer enabled enabled
homelab-router-watchdog.service static -
homelab-router-watchdog.timer enabled enabled
homelab-runbook-generate.service static -
homelab-runbook-generate.timer enabled enabled
homelab-safe-autoheal.service static -
homelab-safe-autoheal.timer enabled enabled
homelab-safe-exec-health.service static -
homelab-safe-exec-health.timer disabled enabled
homelab-scrutiny.service enabled enabled
homelab-secret-exposure-guard.service static -
homelab-secret-exposure-guard.timer enabled enabled
homelab-secret-sanity.service static -
homelab-secret-sanity.timer enabled enabled
homelab-service-registry-check.service static -
homelab-service-registry-check.timer enabled enabled
homelab-smartctl-textfile.service static -
homelab-smartctl-textfile.timer enabled enabled
homelab-sops-edge-secret-coverage.service static -
homelab-sops-edge-secret-coverage.timer enabled enabled
homelab-storage-capacity.service static -
homelab-storage-capacity.timer enabled enabled
homelab-vaultwarden-backup.service static -
homelab-vaultwarden-backup.timer disabled enabled
homelab-verified-backup-health.service static -
homelab-verified-backup-health.timer enabled enabled
homelab-vm160-remote-backup.service static -
homelab-vm160-remote-backup.timer disabled enabled
homelab-vm-backup-health.service static -
homelab-vm-backup-health.timer enabled enabled
homelab-vm-backup-policy.service static -
homelab-vm-backup-policy.timer enabled enabled
homelab-vm-full-backup.service static -
homelab-vm-full-backup.timer enabled enabled
homelab-vm-full-restore-drill.service static -
homelab-vm-full-restore-drill.timer enabled enabled
homelab-vm-local-dumps-2cloud.service static -
homelab-vm-local-dumps-2cloud.timer enabled enabled
homelab-vm-local-dumps-retention.service static -
homelab-vm-local-dumps-retention.timer enabled enabled
homelab-vps-identity-audit.service static -
homelab-vps-identity-audit.timer enabled enabled
immich-media-backup.service static -
immich-media-backup.timer disabled enabled
memos-backup.service static -
memos-backup.timer enabled enabled
netbird-vps-backup.service static -
netbird-vps-backup.timer disabled enabled
paperless-backup.service static -
paperless-backup.timer disabled enabled
skladchik-reports-monitor-backup-restore-check.service static -
skladchik-reports-monitor-backup-restore-check.timer enabled enabled
skladchik-reports-monitor-backup.service static -
skladchik-reports-monitor-backup.timer enabled enabled
===== HOMELAB TIMERS =====
Tue 2026-07-21 09:11:14 MSK 3s Tue 2026-07-21 09:10:14 MSK 56s ago homelab-cluster-admin-webpanel-sync.timer homelab-cluster-admin-webpanel-sync.service
Tue 2026-07-21 09:11:14 MSK 3s Tue 2026-07-21 09:10:14 MSK 56s ago homelab-private-vpn-hosts-health.timer homelab-private-vpn-hosts-health.service
Tue 2026-07-21 09:11:49 MSK 37s Tue 2026-07-21 09:10:45 MSK 25s ago homelab-router-watchdog.timer homelab-router-watchdog.service
Tue 2026-07-21 09:12:16 MSK 1min 5s Tue 2026-07-21 09:10:14 MSK 56s ago homelab-cluster-admin-inventory-sync.timer homelab-cluster-admin-inventory-sync.service
Tue 2026-07-21 09:14:26 MSK 3min 14s Tue 2026-07-21 09:09:25 MSK 1min 45s ago homelab-health-metrics.timer homelab-health-metrics.service
Tue 2026-07-21 09:15:00 MSK 3min 48s Tue 2026-07-21 09:00:14 MSK 10min ago homelab-incident-journal.timer homelab-incident-journal.service
Tue 2026-07-21 09:15:00 MSK 3min 48s Tue 2026-07-21 09:00:14 MSK 10min ago homelab-safe-autoheal.timer homelab-safe-autoheal.service
Tue 2026-07-21 09:15:26 MSK 4min 14s Tue 2026-07-21 09:01:14 MSK 9min ago homelab-duty-admin-v2.timer homelab-duty-admin-v2.service
Tue 2026-07-21 09:15:30 MSK 4min 18s Tue 2026-07-21 09:00:35 MSK 10min ago homelab-external-probe-vps-health.timer homelab-external-probe-vps-health.service
Tue 2026-07-21 09:15:30 MSK 4min 19s Tue 2026-07-21 09:10:14 MSK 56s ago homelab-crypto-portfolio-chart-health.timer homelab-crypto-portfolio-chart-health.service
Tue 2026-07-21 09:15:45 MSK 4min 34s Tue 2026-07-21 09:10:45 MSK 25s ago homelab-smartctl-textfile.timer homelab-smartctl-textfile.service
Tue 2026-07-21 09:16:28 MSK 5min Tue 2026-07-21 09:00:41 MSK 10min ago homelab-forum-snuffleupagus-health.timer homelab-forum-snuffleupagus-health.service
Tue 2026-07-21 09:16:30 MSK 5min Tue 2026-07-21 09:01:30 MSK 9min ago homelab-disk-space-health.timer homelab-disk-space-health.service
Tue 2026-07-21 09:19:30 MSK 8min Tue 2026-07-21 09:04:30 MSK 6min ago homelab-pve03-staging-capacity-health.timer homelab-pve03-staging-capacity-health.service
Tue 2026-07-21 09:30:55 MSK 19min Tue 2026-07-21 09:02:14 MSK 8min ago homelab-mkdocs-auto-refresh.timer homelab-mkdocs-auto-refresh.service
Tue 2026-07-21 11:42:25 MSK 2h 31min Tue 2026-07-21 05:42:25 MSK 3h 28min ago homelab-vps-identity-audit.timer homelab-vps-identity-audit.service
Wed 2026-07-22 00:00:00 MSK 14h Tue 2026-07-21 00:00:14 MSK 9h ago dpkg-db-backup.timer dpkg-db-backup.service
Wed 2026-07-22 00:01:28 MSK 14h Tue 2026-07-21 00:06:03 MSK 9h ago homelab-storage-capacity.timer homelab-storage-capacity.service
Wed 2026-07-22 00:06:04 MSK 14h Tue 2026-07-21 00:03:14 MSK 9h ago homelab-quality-gate.timer homelab-quality-gate.service
Wed 2026-07-22 00:08:06 MSK 14h Tue 2026-07-21 00:09:32 MSK 9h ago homelab-evidence-catalog.timer homelab-evidence-catalog.service
Wed 2026-07-22 00:09:43 MSK 14h Tue 2026-07-21 00:09:32 MSK 9h ago homelab-backup-freshness.timer homelab-backup-freshness.service
Wed 2026-07-22 00:10:12 MSK 14h Tue 2026-07-21 00:16:14 MSK 8h ago homelab-docker-health.timer homelab-docker-health.service
Wed 2026-07-22 02:15:00 MSK 17h Tue 2026-07-21 02:15:05 MSK 6h ago homelab-vm-full-backup.timer homelab-vm-full-backup.service
Wed 2026-07-22 03:10:02 MSK 17h Tue 2026-07-21 03:13:14 MSK 5h 57min ago homelab-emergency-backup.timer homelab-emergency-backup.service
Wed 2026-07-22 03:35:00 MSK 18h Tue 2026-07-21 03:35:14 MSK 5h 35min ago filebrowser-backup.timer filebrowser-backup.service
Wed 2026-07-22 03:55:02 MSK 18h Tue 2026-07-21 03:51:25 MSK 5h 19min ago homelab-appbackup-dockge-stacks.timer homelab-appbackup-dockge-stacks.service
Wed 2026-07-22 04:17:29 MSK 19h Tue 2026-07-21 04:19:17 MSK 4h 51min ago homelab-appbackup-npmplus-edge.timer homelab-appbackup-npmplus-edge.service
Wed 2026-07-22 04:26:54 MSK 19h Tue 2026-07-21 04:28:14 MSK 4h 42min ago homelab-gitea-secondary-backup.timer homelab-gitea-secondary-backup.service
Wed 2026-07-22 04:38:04 MSK 19h Tue 2026-07-21 04:34:48 MSK 4h 36min ago skladchik-reports-monitor-backup.timer skladchik-reports-monitor-backup.service
Wed 2026-07-22 04:39:28 MSK 19h Tue 2026-07-21 04:48:40 MSK 4h 22min ago homelab-edge-vm-offhost-freshness.timer homelab-edge-vm-offhost-freshness.service
Wed 2026-07-22 05:01:44 MSK 19h Tue 2026-07-21 04:57:14 MSK 4h 13min ago homelab-appbackup-sqlite-batch.timer homelab-appbackup-sqlite-batch.service
Wed 2026-07-22 05:13:38 MSK 20h Tue 2026-07-21 04:55:35 MSK 4h 15min ago homelab-sops-edge-secret-coverage.timer homelab-sops-edge-secret-coverage.service
Wed 2026-07-22 05:23:56 MSK 20h Tue 2026-07-21 05:17:57 MSK 3h 53min ago homelab-router-running-config-mail-cloud.timer homelab-router-running-config-mail-cloud.service
Wed 2026-07-22 05:27:13 MSK 20h Tue 2026-07-21 05:29:05 MSK 3h 42min ago homelab-appbackup-postgres-batch.timer homelab-appbackup-postgres-batch.service
Wed 2026-07-22 05:47:19 MSK 20h Tue 2026-07-21 05:46:14 MSK 3h 24min ago homelab-appbackup-mariadb-batch.timer homelab-appbackup-mariadb-batch.service
Wed 2026-07-22 06:01:19 MSK 20h Tue 2026-07-21 05:57:30 MSK 3h 13min ago homelab-extended-appbackup.timer homelab-extended-appbackup.service
Wed 2026-07-22 06:27:43 MSK 21h Tue 2026-07-21 06:32:25 MSK 2h 38min ago homelab-crypto-portfolio-backup.timer homelab-crypto-portfolio-backup.service
Wed 2026-07-22 06:38:51 MSK 21h Tue 2026-07-21 06:43:54 MSK 2h 27min ago homelab-verified-backup-health.timer homelab-verified-backup-health.service
Wed 2026-07-22 06:39:22 MSK 21h Tue 2026-07-21 06:40:25 MSK 2h 30min ago homelab-alertmanager-backup.timer homelab-alertmanager-backup.service
Wed 2026-07-22 06:44:16 MSK 21h Tue 2026-07-21 06:40:25 MSK 2h 30min ago homelab-mail-cloud-critical-upload.timer homelab-mail-cloud-critical-upload.service
Wed 2026-07-22 06:52:06 MSK 21h Tue 2026-07-21 06:47:47 MSK 2h 23min ago homelab-beszel-backup.timer homelab-beszel-backup.service
Wed 2026-07-22 06:58:58 MSK 21h Tue 2026-07-21 06:57:30 MSK 2h 13min ago homelab-blackbox-exporter-backup.timer homelab-blackbox-exporter-backup.service
Wed 2026-07-22 07:02:27 MSK 21h Tue 2026-07-21 07:05:30 MSK 2h 5min ago homelab-backup-healthcheck.timer homelab-backup-healthcheck.service
Wed 2026-07-22 07:07:14 MSK 21h Tue 2026-07-21 07:18:55 MSK 1h 52min ago homelab-legacy-monitoring-backups.timer homelab-legacy-monitoring-backups.service
Wed 2026-07-22 07:16:17 MSK 22h Tue 2026-07-21 07:15:45 MSK 1h 55min ago homelab-drift-check.timer homelab-drift-check.service
Wed 2026-07-22 07:35:06 MSK 22h Tue 2026-07-21 07:36:59 MSK 1h 34min ago homelab-vm-backup-health.timer homelab-vm-backup-health.service
Wed 2026-07-22 07:36:00 MSK 22h Tue 2026-07-21 07:21:14 MSK 1h 49min ago homelab-service-registry-check.timer homelab-service-registry-check.service
Wed 2026-07-22 07:38:21 MSK 22h Tue 2026-07-21 07:36:14 MSK 1h 34min ago homelab-cluster-admin-incident-engine-backup.timer homelab-cluster-admin-incident-engine-backup.service
Wed 2026-07-22 07:41:49 MSK 22h Tue 2026-07-21 07:53:41 MSK 1h 17min ago homelab-mail-cloud-nextcloud-vm-upload.timer homelab-mail-cloud-nextcloud-vm-upload.service
Wed 2026-07-22 07:47:14 MSK 22h Tue 2026-07-21 07:51:41 MSK 1h 19min ago homelab-dependency-map-check.timer homelab-dependency-map-check.service
Wed 2026-07-22 07:49:26 MSK 22h Tue 2026-07-21 07:56:14 MSK 1h 14min ago homelab-runbook-generate.timer homelab-runbook-generate.service
Wed 2026-07-22 07:53:13 MSK 22h Tue 2026-07-21 07:51:14 MSK 1h 19min ago homelab-alerting-health.timer homelab-alerting-health.service
Wed 2026-07-22 07:58:00 MSK 22h Tue 2026-07-21 08:09:34 MSK 1h 1min ago homelab-duty-admin-report.timer homelab-duty-admin-report.service
Wed 2026-07-22 08:03:06 MSK 22h Tue 2026-07-21 07:49:14 MSK 1h 21min ago homelab-desired-state-sync.timer homelab-desired-state-sync.service
Wed 2026-07-22 08:10:37 MSK 22h Tue 2026-07-21 08:07:25 MSK 1h 3min ago homelab-backup-sla-health.timer homelab-backup-sla-health.service
Wed 2026-07-22 08:18:43 MSK 23h Tue 2026-07-21 08:01:14 MSK 1h 9min ago homelab-overall-health.timer homelab-overall-health.service
Wed 2026-07-22 08:24:56 MSK 23h Tue 2026-07-21 08:29:14 MSK 41min ago homelab-capacity-risk.timer homelab-capacity-risk.service
Wed 2026-07-22 08:24:58 MSK 23h Tue 2026-07-21 08:17:02 MSK 54min ago homelab-kuma-monitor-policy.timer homelab-kuma-monitor-policy.service
Wed 2026-07-22 08:28:27 MSK 23h Tue 2026-07-21 08:31:30 MSK 39min ago homelab-final-readiness-gate.timer homelab-final-readiness-gate.service
Wed 2026-07-22 08:37:02 MSK 23h Tue 2026-07-21 08:30:07 MSK 41min ago homelab-vm-backup-policy.timer homelab-vm-backup-policy.service
Wed 2026-07-22 08:40:13 MSK 23h Tue 2026-07-21 08:50:14 MSK 20min ago homelab-backup-coverage-matrix.timer homelab-backup-coverage-matrix.service
Wed 2026-07-22 08:43:29 MSK 23h Tue 2026-07-21 08:54:07 MSK 17min ago homelab-golden-state-index.timer homelab-golden-state-index.service
Wed 2026-07-22 08:44:57 MSK 23h Tue 2026-07-21 08:35:41 MSK 35min ago homelab-secret-exposure-guard.timer homelab-secret-exposure-guard.service
Wed 2026-07-22 08:53:01 MSK 23h Tue 2026-07-21 08:51:14 MSK 19min ago homelab-cluster-passport.timer homelab-cluster-passport.service
Wed 2026-07-22 08:53:36 MSK 23h Tue 2026-07-21 08:41:14 MSK 29min ago memos-backup.timer memos-backup.service
Sun 2026-07-26 03:34:47 MSK 4 days Sun 2026-07-19 03:45:37 MSK 2 days ago homelab-auto-backup.timer homelab-auto-backup.service
Sun 2026-07-26 05:17:43 MSK 4 days Sun 2026-07-19 05:11:25 MSK 2 days ago skladchik-reports-monitor-backup-restore-check.timer skladchik-reports-monitor-backup-restore-check.service
Sun 2026-07-26 09:32:36 MSK 5 days Sun 2026-07-19 09:32:29 MSK 1 day 23h ago homelab-mail-cloud-edge-vm-upload.timer homelab-mail-cloud-edge-vm-upload.service
Sun 2026-07-26 10:29:15 MSK 5 days Sun 2026-07-19 10:06:07 MSK 1 day 23h ago homelab-vm-local-dumps-2cloud.timer homelab-vm-local-dumps-2cloud.service
Sun 2026-07-26 11:13:32 MSK 5 days Sun 2026-07-19 11:08:41 MSK 1 day 22h ago homelab-vm-local-dumps-retention.timer homelab-vm-local-dumps-retention.service
Mon 2026-07-27 00:09:19 MSK 5 days Mon 2026-07-20 00:00:29 MSK 1 day 9h ago homelab-secret-sanity.timer homelab-secret-sanity.service
Mon 2026-07-27 07:34:51 MSK 5 days Mon 2026-07-20 07:25:14 MSK 1 day 1h ago homelab-mail-cloud-restore-drill.timer homelab-mail-cloud-restore-drill.service
Thu 2026-10-15 06:00:00 MSK 2 months 24 days Wed 2026-07-15 06:00:00 MSK - homelab-vm-full-restore-drill.timer homelab-vm-full-restore-drill.service
===== FAILED UNITS =====
UNIT LOAD ACTIVE SUB DESCRIPTION
● homelab-backup-matrix-refresh.service loaded failed failed Generate unified homelab backup matrix state
● homelab-cluster-admin-pull.service loaded failed failed Pull cluster-admin observer health into pve01
● homelab-vm160-remote-backup.service loaded failed failed VM160 full-image backup streamed from pve02 to pve03
Legend: LOAD → Reflects whether the unit definition was properly loaded.
ACTIVE → The high-level unit activation state, i.e. generalization of SUB.
SUB → The low-level unit activation state, values depend on unit type.
3 loaded units listed.
===== HOMELAB UNIT FILE PATHS =====
2026-06-15T13:00:42.7590703540 644 root:root 259 /etc/systemd/system/homelab-auto-backup.service
2026-06-15T13:00:58.1712446040 644 root:root 205 /etc/systemd/system/homelab-auto-backup.timer
2026-06-15T13:04:11.9638937260 644 root:root 124 /etc/systemd/system/homelab-backup-healthcheck.service
2026-06-15T13:04:11.9649165750 644 root:root 204 /etc/systemd/system/homelab-backup-healthcheck.timer
2026-06-16T19:56:57.2280572190 644 root:root 133 /etc/systemd/system/homelab-disk-space-health.service
2026-06-16T19:56:57.2283922310 644 root:root 188 /etc/systemd/system/homelab-disk-space-health.timer
2026-06-18T06:26:08.5462695380 644 root:root 446 /etc/systemd/system/homelab-private-vpn-hosts-health.service
2026-06-18T06:26:08.5473243400 644 root:root 232 /etc/systemd/system/homelab-private-vpn-hosts-health.timer
2026-06-18T07:10:14.4453177530 644 root:root 204 /etc/systemd/system/homelab-npmplus-kuma-config-backup.service
2026-06-18T07:10:14.4468289380 644 root:root 230 /etc/systemd/system/homelab-npmplus-kuma-config-backup.timer
2026-06-18T16:26:18.3740018850 644 root:root 181 /etc/systemd/system/homelab-authentik-backup.service
2026-06-18T16:26:18.3753734820 644 root:root 208 /etc/systemd/system/homelab-authentik-backup.timer
2026-06-18T17:09:28.1614174140 644 root:root 173 /etc/systemd/system/homelab-gitea-backup.service
2026-06-18T17:09:28.1628104380 644 root:root 200 /etc/systemd/system/homelab-gitea-backup.timer
2026-06-18T17:35:10.8723268760 644 root:root 185 /etc/systemd/system/homelab-vaultwarden-backup.service
2026-06-18T17:35:10.8735951370 644 root:root 212 /etc/systemd/system/homelab-vaultwarden-backup.timer
2026-06-18T18:56:51.5685208630 644 root:root 200 /etc/systemd/system/homelab-edge-vm-vzdump-backup.service
2026-06-18T18:56:51.5698180780 644 root:root 225 /etc/systemd/system/homelab-edge-vm-vzdump-backup.timer
2026-06-18T22:13:10.8353373180 644 root:root 271 /etc/systemd/system/homelab-mail-cloud-critical-upload.service
2026-06-18T22:13:10.8363221220 644 root:root 231 /etc/systemd/system/homelab-mail-cloud-critical-upload.timer
2026-06-18T22:23:32.0559055120 644 root:root 268 /etc/systemd/system/homelab-mail-cloud-edge-vm-upload.service
2026-06-18T22:23:32.0560225000 644 root:root 233 /etc/systemd/system/homelab-mail-cloud-edge-vm-upload.timer
2026-06-18T22:41:10.0626047720 644 root:root 252 /etc/systemd/system/homelab-mail-cloud-restore-drill.service
2026-06-18T22:41:10.0636047870 644 root:root 217 /etc/systemd/system/homelab-mail-cloud-restore-drill.timer
2026-06-20T23:10:27.1119168660 644 root:root 134 /etc/systemd/system/homelab-smartctl-textfile.service
2026-06-20T23:10:27.1128425330 644 root:root 208 /etc/systemd/system/homelab-smartctl-textfile.timer
2026-06-21T00:30:59.6377784930 644 root:root 619 /etc/systemd/system/homelab-scrutiny.service
2026-06-21T10:39:21.2127986960 644 root:root 205 /etc/systemd/system/homelab-sops-edge-secret-coverage.service
2026-06-21T10:39:21.2139615830 644 root:root 230 /etc/systemd/system/homelab-sops-edge-secret-coverage.timer
2026-06-21T11:06:12.5516186940 644 root:root 191 /etc/systemd/system/homelab-mkdocs-auto-refresh.timer
2026-06-21T15:54:26.1428955170 644 root:root 295 /etc/systemd/system/homelab-mkdocs-auto-refresh.service
2026-06-22T12:34:40.6028014260 644 root:root 416 /etc/systemd/system/homelab-health-metrics.service
2026-06-22T12:34:40.6037061340 644 root:root 219 /etc/systemd/system/homelab-health-metrics.timer
2026-06-24T16:00:07.1378187060 644 root:root 127 /etc/systemd/system/homelab-vps-identity-audit.service
2026-06-24T16:00:07.1388187210 644 root:root 147 /etc/systemd/system/homelab-vps-identity-audit.timer
2026-06-27T04:30:28.4973721150 644 root:root 169 /etc/systemd/system/homelab-secret-sanity.timer
2026-06-27T04:30:28.4973721150 644 root:root 249 /etc/systemd/system/homelab-secret-sanity.service
2026-06-27T05:04:37.4313772970 644 root:root 201 /etc/systemd/system/homelab-backup-freshness.timer
2026-06-27T05:06:45.3586435000 644 root:root 285 /etc/systemd/system/homelab-docker-health.service
2026-06-27T05:06:45.3594437270 644 root:root 204 /etc/systemd/system/homelab-docker-health.timer
2026-06-27T05:08:30.6395106090 644 root:root 259 /etc/systemd/system/homelab-storage-capacity.service
2026-06-27T05:08:30.6403077050 644 root:root 211 /etc/systemd/system/homelab-storage-capacity.timer
2026-06-27T05:09:24.1641095310 644 root:root 259 /etc/systemd/system/homelab-evidence-catalog.service
2026-06-27T05:09:24.1648443770 644 root:root 201 /etc/systemd/system/homelab-evidence-catalog.timer
2026-06-27T05:16:32.9994853080 644 root:root 189 /etc/systemd/system/homelab-quality-gate.timer
2026-06-27T05:16:32.9994853080 644 root:root 213 /etc/systemd/system/homelab-quality-gate.service
2026-06-29T21:00:02.0934499640 644 root:root 314 /etc/systemd/system/homelab-cluster-internal-ip.service
2026-06-30T13:42:51.4303494920 644 root:root 151 /etc/systemd/system/homelab-pve03-staging-capacity-health.service
2026-06-30T13:42:51.4310788310 644 root:root 180 /etc/systemd/system/homelab-pve03-staging-capacity-health.timer
2026-06-30T14:51:25.1233593980 644 root:root 271 /etc/systemd/system/homelab-mail-cloud-nextcloud-vm-upload.service
2026-06-30T14:51:25.1244045430 644 root:root 176 /etc/systemd/system/homelab-mail-cloud-nextcloud-vm-upload.timer
2026-06-30T16:30:01.3301453960 644 root:root 214 /etc/systemd/system/homelab-router-running-config-mail-cloud.service
2026-06-30T16:30:01.3311218770 644 root:root 185 /etc/systemd/system/homelab-router-running-config-mail-cloud.timer
2026-07-03T11:51:30.8268733400 644 root:root 171 /etc/systemd/system/homelab-emergency-backup.timer
2026-07-03T11:51:30.8268733400 644 root:root 197 /etc/systemd/system/homelab-emergency-backup.service
2026-07-03T12:04:53.4624196890 644 root:root 182 /etc/systemd/system/homelab-appbackup-dockge-stacks.timer
2026-07-03T12:04:53.4624196890 644 root:root 207 /etc/systemd/system/homelab-appbackup-dockge-stacks.service
2026-07-03T12:12:11.4472306810 644 root:root 246 /etc/systemd/system/homelab-backup-freshness.service
2026-07-03T12:29:06.4444996780 644 root:root 181 /etc/systemd/system/homelab-appbackup-npmplus-edge.timer
2026-07-03T12:29:06.4444996780 644 root:root 212 /etc/systemd/system/homelab-appbackup-npmplus-edge.service
2026-07-03T12:37:48.5320271520 644 root:root 140 /etc/systemd/system/homelab-verified-backup-health.service
2026-07-03T12:37:48.5320271520 644 root:root 173 /etc/systemd/system/homelab-verified-backup-health.timer
2026-07-03T12:44:42.9312711690 644 root:root 109 /etc/systemd/system/homelab-drift-check.service
2026-07-03T12:44:42.9312711690 644 root:root 153 /etc/systemd/system/homelab-drift-check.timer
2026-07-03T13:16:53.8209829850 644 root:root 171 /etc/systemd/system/homelab-appbackup-sqlite-batch.timer
2026-07-03T13:16:53.8209829850 644 root:root 417 /etc/systemd/system/homelab-appbackup-sqlite-batch.service
2026-07-03T13:20:57.4660979330 644 root:root 173 /etc/systemd/system/homelab-appbackup-postgres-batch.timer
2026-07-03T13:25:04.7183999550 644 root:root 418 /etc/systemd/system/homelab-appbackup-postgres-batch.service
2026-07-03T13:33:49.3146898050 644 root:root 172 /etc/systemd/system/homelab-appbackup-mariadb-batch.timer
2026-07-03T13:33:49.3146898050 644 root:root 200 /etc/systemd/system/homelab-appbackup-mariadb-batch.service
2026-07-03T13:41:24.9497954780 644 root:root 138 /etc/systemd/system/homelab-service-registry-check.service
2026-07-03T13:41:24.9497954780 644 root:root 171 /etc/systemd/system/homelab-service-registry-check.timer
2026-07-03T13:42:08.5796973730 644 root:root 127 /etc/systemd/system/homelab-dependency-map-check.service
2026-07-03T13:42:08.5796973730 644 root:root 162 /etc/systemd/system/homelab-dependency-map-check.timer
2026-07-03T13:43:01.6263487930 644 root:root 115 /etc/systemd/system/homelab-overall-health.service
2026-07-03T13:43:01.6263487930 644 root:root 156 /etc/systemd/system/homelab-overall-health.timer
2026-07-03T13:44:21.6709762300 644 root:root 268 /etc/systemd/system/homelab-health-http.service
2026-07-03T13:46:25.4886845470 644 root:root 119 /etc/systemd/system/homelab-vm-backup-health.service
2026-07-03T13:46:25.4886845470 644 root:root 158 /etc/systemd/system/homelab-vm-backup-health.timer
2026-07-03T13:51:34.0440941410 644 root:root 273 /etc/systemd/system/homelab-health-http-9101.service
2026-07-03T14:07:11.3293592880 644 root:root 184 /etc/systemd/system/homelab-alerting-health.service
2026-07-03T14:07:11.3303593020 644 root:root 163 /etc/systemd/system/homelab-alerting-health.timer
2026-07-03T14:19:38.2269689900 644 root:root 168 /etc/systemd/system/homelab-runbook-generate.timer
2026-07-03T14:19:38.2269689900 644 root:root 174 /etc/systemd/system/homelab-runbook-generate.service
2026-07-03T14:30:48.2259704210 644 root:root 159 /etc/systemd/system/homelab-desired-state-sync.service
2026-07-03T14:30:48.2259704210 644 root:root 168 /etc/systemd/system/homelab-desired-state-sync.timer
2026-07-03T14:35:20.4059353590 644 root:root 159 /etc/systemd/system/homelab-duty-admin-report.service
2026-07-03T14:35:20.4059353590 644 root:root 169 /etc/systemd/system/homelab-duty-admin-report.timer
2026-07-06T16:38:41.0738565590 644 root:root 127 /etc/systemd/system/homelab-incident-journal.timer
2026-07-06T16:38:41.0738565590 644 root:root 147 /etc/systemd/system/homelab-incident-journal.service
2026-07-06T16:46:56.0853501450 644 root:root 159 /etc/systemd/system/homelab-backup-sla-health.service
2026-07-06T16:46:56.0853501450 644 root:root 169 /etc/systemd/system/homelab-backup-sla-health.timer
2026-07-06T17:07:42.1531722210 644 root:root 160 /etc/systemd/system/homelab-kuma-monitor-policy.service
2026-07-06T17:07:42.1531722210 644 root:root 168 /etc/systemd/system/homelab-kuma-monitor-policy.timer
2026-07-06T17:17:27.4890025440 644 root:root 134 /etc/systemd/system/homelab-safe-autoheal.timer
2026-07-06T17:17:27.4890025440 644 root:root 151 /etc/systemd/system/homelab-safe-autoheal.service
2026-07-06T17:28:46.8602514040 644 root:root 155 /etc/systemd/system/homelab-final-readiness-gate.service
2026-07-06T17:28:46.8602514040 644 root:root 162 /etc/systemd/system/homelab-final-readiness-gate.timer
2026-07-06T17:51:59.0742410960 644 root:root 170 /etc/systemd/system/homelab-vm-backup-policy.service
2026-07-06T17:51:59.0742410960 644 root:root 181 /etc/systemd/system/homelab-vm-backup-policy.timer
2026-07-06T18:27:29.1535313910 644 root:root 181 /etc/systemd/system/homelab-vm-local-dumps-2cloud.service
2026-07-06T18:27:29.1535313910 644 root:root 191 /etc/systemd/system/homelab-vm-local-dumps-2cloud.timer
2026-07-06T18:56:19.5296144310 644 root:root 176 /etc/systemd/system/homelab-vm-local-dumps-retention.timer
2026-07-06T18:56:19.5296144310 644 root:root 177 /etc/systemd/system/homelab-vm-local-dumps-retention.service
2026-07-06T19:08:11.3245947220 644 root:root 147 /etc/systemd/system/homelab-capacity-risk.service
2026-07-06T19:08:11.3245947220 644 root:root 161 /etc/systemd/system/homelab-capacity-risk.timer
2026-07-06T19:27:15.6118940830 644 root:root 157 /etc/systemd/system/homelab-secret-exposure-guard.service
2026-07-06T19:27:15.6118940830 644 root:root 163 /etc/systemd/system/homelab-secret-exposure-guard.timer
2026-07-06T19:44:22.9146001910 644 root:root 147 /etc/systemd/system/homelab-cluster-passport.service
2026-07-06T19:44:22.9146001910 644 root:root 158 /etc/systemd/system/homelab-cluster-passport.timer
2026-07-06T20:01:29.9500539270 644 root:root 151 /etc/systemd/system/homelab-golden-state-index.service
2026-07-06T20:01:29.9500539270 644 root:root 160 /etc/systemd/system/homelab-golden-state-index.timer
2026-07-07T00:22:27.7023075630 644 root:root 131 /etc/systemd/system/homelab-backup-coverage-matrix.service
2026-07-07T00:22:27.7031435610 644 root:root 164 /etc/systemd/system/homelab-backup-coverage-matrix.timer
2026-07-07T04:22:28.9708924180 644 root:root 165 /etc/systemd/system/homelab-extended-appbackup.service
2026-07-07T04:22:28.9708924180 644 root:root 166 /etc/systemd/system/homelab-extended-appbackup.timer
2026-07-07T06:37:16.6351692020 644 root:root 160 /etc/systemd/system/homelab-forum-snuffleupagus-health.timer
2026-07-07T06:37:16.6351692020 644 root:root 167 /etc/systemd/system/homelab-forum-snuffleupagus-health.service
2026-07-07T07:28:55.1971196770 644 root:root 168 /etc/systemd/system/homelab-external-probe-vps-health.timer
2026-07-07T07:28:55.1971196770 644 root:root 174 /etc/systemd/system/homelab-external-probe-vps-health.service
2026-07-07T08:01:44.7118958700 644 root:root 150 /etc/systemd/system/homelab-duty-admin-v2.service
2026-07-07T08:01:44.7118958700 644 root:root 156 /etc/systemd/system/homelab-duty-admin-v2.timer
2026-07-08T15:08:13.7211859980 644 root:root 235 /etc/systemd/system/homelab-edge-vm-offhost-freshness.service
2026-07-08T15:08:13.7216805560 644 root:root 252 /etc/systemd/system/homelab-edge-vm-offhost-freshness.timer
2026-07-09T02:10:26.8244661930 644 root:root 142 /etc/systemd/system/homelab-cluster-admin-pull.service
2026-07-09T02:10:26.8256177450 644 root:root 145 /etc/systemd/system/homelab-cluster-admin-pull.timer
2026-07-09T08:11:07.2434837370 644 root:root 162 /etc/systemd/system/homelab-cluster-admin-webpanel-sync.service
2026-07-09T08:11:07.2446282660 644 root:root 169 /etc/systemd/system/homelab-cluster-admin-webpanel-sync.timer
2026-07-10T02:50:51.8861195400 644 root:root 361 /etc/systemd/system/homelab-vm-full-backup.service
2026-07-10T02:50:51.8871195540 644 root:root 201 /etc/systemd/system/homelab-vm-full-backup.timer
2026-07-10T02:51:05.4785581640 644 root:root 357 /etc/systemd/system/homelab-vm-full-restore-drill.service
2026-07-10T02:51:05.4796181450 644 root:root 234 /etc/systemd/system/homelab-vm-full-restore-drill.timer
2026-07-10T11:28:07.6943454530 644 root:root 187 /etc/systemd/system/homelab-backup-matrix-refresh.service
2026-07-10T11:28:07.6943454530 644 root:root 217 /etc/systemd/system/homelab-backup-matrix-refresh.timer
2026-07-10T20:33:12.8858520910 644 root:root 162 /etc/systemd/system/homelab-crypto-portfolio-backup.timer
2026-07-10T20:33:12.8858520910 644 root:root 164 /etc/systemd/system/homelab-crypto-portfolio-backup.service
2026-07-10T21:46:05.4474358550 644 root:root 204 /etc/systemd/system/homelab-alertmanager-backup.timer
2026-07-10T21:46:05.4474358550 644 root:root 209 /etc/systemd/system/homelab-alertmanager-backup.service
2026-07-10T22:04:53.2805465050 644 root:root 197 /etc/systemd/system/homelab-beszel-backup.service
2026-07-10T22:04:53.2805465050 644 root:root 202 /etc/systemd/system/homelab-beszel-backup.timer
2026-07-10T22:19:57.8193011610 644 root:root 219 /etc/systemd/system/homelab-blackbox-exporter-backup.service
2026-07-10T22:19:57.8193011610 644 root:root 224 /etc/systemd/system/homelab-blackbox-exporter-backup.timer
2026-07-10T22:44:47.4338992100 644 root:root 227 /etc/systemd/system/homelab-legacy-monitoring-backups.service
2026-07-10T22:44:47.4338992100 644 root:root 234 /etc/systemd/system/homelab-legacy-monitoring-backups.timer
2026-07-11T00:36:01.9494545100 644 root:root 235 /etc/systemd/system/homelab-cluster-admin-inventory-sync.service
2026-07-11T00:36:01.9494545100 644 root:root 235 /etc/systemd/system/homelab-cluster-admin-inventory-sync.timer
2026-07-11T01:27:09.7763208170 644 root:root 231 /etc/systemd/system/homelab-cluster-admin-incident-engine-backup.service
2026-07-11T01:27:09.7763208170 644 root:root 238 /etc/systemd/system/homelab-cluster-admin-incident-engine-backup.timer
2026-07-11T22:01:25.6167081820 644 root:root 199 /etc/systemd/system/homelab-router-watchdog.service
2026-07-11T22:01:25.6179399040 644 root:root 236 /etc/systemd/system/homelab-router-watchdog.timer
2026-07-12T21:35:41.6488572630 644 root:root 197 /etc/systemd/system/homelab-crypto-portfolio-chart-health.service
2026-07-12T21:35:41.6488811480 644 root:root 230 /etc/systemd/system/homelab-crypto-portfolio-chart-health.timer
2026-07-18T01:38:18.1727154050 644 root:root 245 /etc/systemd/system/homelab-vm160-remote-backup.service
2026-07-18T01:38:18.1738421500 644 root:root 189 /etc/systemd/system/homelab-vm160-remote-backup.timer
2026-07-20T22:15:38.8869753850 644 root:root 316 /etc/systemd/system/homelab-gitea-secondary-backup.service
2026-07-20T22:15:38.8883231940 644 root:root 208 /etc/systemd/system/homelab-gitea-secondary-backup.timer
2026-07-20T23:59:13.9750909030 644 root:root 361 /etc/systemd/system/homelab-ingress-standby-refresh.service
2026-07-20T23:59:13.9756878160 644 root:root 227 /etc/systemd/system/homelab-ingress-standby-refresh.timer
2026-07-21T00:09:32.1964449440 644 root:root 510 /etc/systemd/system/homelab-safe-exec-health.service
2026-07-21T00:09:32.1984090270 644 root:root 236 /etc/systemd/system/homelab-safe-exec-health.timer
2026-07-21T00:33:12.4631182460 644 root:root 527 /etc/systemd/system/homelab-evidence-seal.service
2026-07-21T00:33:12.4638041710 644 root:root 218 /etc/systemd/system/homelab-evidence-seal.timer
===== PVE01 DOCKER =====
DOCKER_INSTALLED=NO
===== EXISTING HOMELAB STATE DIRECTORIES =====
----- DIR=/var/lib/homelab-health -----
2026-07-09T15:29:41.9851382980 236 /var/lib/homelab-health/proxmox-pve01-offhost.txt
2026-07-09T15:29:41.9851382980 236 /var/lib/homelab-health/pve01-offhost.txt
2026-07-09T15:29:41.9851382980 276 /var/lib/homelab-health/proxmox-pve01-backup.txt
2026-07-09T15:29:41.9851382980 276 /var/lib/homelab-health/pve01-backup.txt
2026-07-09T15:33:52.8128905890 397 /var/lib/homelab-health/service-readiness-proxmox-pve01.txt
2026-07-09T15:42:17.7925620460 203 /var/lib/homelab-health/filebrowser-offhost.txt
2026-07-09T15:47:18.8311350910 276 /var/lib/homelab-health/service-readiness-filebrowser.txt
2026-07-09T15:53:45.2480051850 207 /var/lib/homelab-health/actual-budget-offhost.txt
2026-07-09T15:53:45.2480051850 207 /var/lib/homelab-health/actual-offhost.txt
2026-07-09T15:53:45.2480051850 213 /var/lib/homelab-health/actual-budget-restore.txt
2026-07-09T15:53:45.2480051850 213 /var/lib/homelab-health/actual-restore.txt
2026-07-09T15:53:45.2480051850 269 /var/lib/homelab-health/actual-backup.txt
2026-07-09T15:53:45.2480051850 269 /var/lib/homelab-health/actual-budget-backup.txt
2026-07-09T16:04:50.0111067090 281 /var/lib/homelab-health/service-readiness-actual-budget.txt
2026-07-09T19:01:20.7901933280 195 /var/lib/homelab-health/homebox-offhost.txt
2026-07-09T19:01:20.7901933280 201 /var/lib/homelab-health/homebox-restore.txt
2026-07-09T19:01:20.7901933280 251 /var/lib/homelab-health/homebox-backup.txt
2026-07-09T19:06:43.1941052730 268 /var/lib/homelab-health/service-readiness-homebox.txt
2026-07-09T19:17:35.0170448070 193 /var/lib/homelab-health/mealie-offhost.txt
2026-07-09T19:17:35.0170448070 199 /var/lib/homelab-health/mealie-restore.txt
2026-07-09T19:17:35.0170448070 250 /var/lib/homelab-health/mealie-backup.txt
2026-07-09T19:23:39.3486047240 283 /var/lib/homelab-health/service-readiness-mealie.txt
2026-07-09T19:46:33.3515780290 299 /var/lib/homelab-health/service-readiness-paperless.txt
2026-07-09T19:49:28.4202504550 212 /var/lib/homelab-health/memos-restore.txt
2026-07-09T19:50:22.5980774800 261 /var/lib/homelab-health/service-readiness-memos.txt
2026-07-09T20:00:19.8381943590 214 /var/lib/homelab-health/linkding-backup.txt
2026-07-09T20:00:19.8381943590 227 /var/lib/homelab-health/linkding-offhost.txt
2026-07-09T20:00:19.8381943590 227 /var/lib/homelab-health/linkding-restore.txt
2026-07-09T20:40:58.8843463730 270 /var/lib/homelab-health/service-readiness-linkding.txt
2026-07-09T20:47:41.7774675140 301 /var/lib/homelab-health/healthchecks-canonical-domain-repair.txt
2026-07-09T20:47:41.7774675140 340 /var/lib/homelab-health/service-readiness-healthchecks.txt
2026-07-09T20:51:20.8507958870 210 /var/lib/homelab-health/vikunja-backup.txt
2026-07-09T20:51:20.8507958870 222 /var/lib/homelab-health/vikunja-offhost.txt
2026-07-09T20:51:20.8507958870 222 /var/lib/homelab-health/vikunja-restore.txt
2026-07-09T20:55:48.6988688880 269 /var/lib/homelab-health/service-readiness-vikunja.txt
2026-07-09T20:58:33.5284046710 218 /var/lib/homelab-health/bookstack-backup.txt
2026-07-09T20:58:33.5284046710 232 /var/lib/homelab-health/bookstack-offhost.txt
2026-07-09T20:58:33.5284046710 232 /var/lib/homelab-health/bookstack-restore.txt
2026-07-09T21:01:03.7247153370 297 /var/lib/homelab-health/service-readiness-bookstack.txt
2026-07-09T21:08:55.5329737450 243 /var/lib/homelab-health/stirling-pdf-backup.txt
2026-07-09T21:08:55.5329737450 247 /var/lib/homelab-health/stirling-pdf-offhost.txt
2026-07-09T21:08:55.5329737450 247 /var/lib/homelab-health/stirling-pdf-restore.txt
2026-07-09T21:13:22.9720909840 269 /var/lib/homelab-health/service-readiness-stirling-pdf.txt
2026-07-09T21:16:17.6627935760 214 /var/lib/homelab-health/jellyfin-backup.txt
2026-07-09T21:16:17.6627935760 227 /var/lib/homelab-health/jellyfin-offhost.txt
2026-07-09T21:16:17.6627935760 227 /var/lib/homelab-health/jellyfin-restore.txt
2026-07-09T21:19:11.7814873230 270 /var/lib/homelab-health/service-readiness-jellyfin.txt
2026-07-09T21:22:25.6704869300 238 /var/lib/homelab-health/audiobookshelf-backup.txt
2026-07-09T21:22:25.6704869300 257 /var/lib/homelab-health/audiobookshelf-offhost.txt
2026-07-09T21:22:25.6704869300 257 /var/lib/homelab-health/audiobookshelf-restore.txt
2026-07-09T21:25:22.5602235460 279 /var/lib/homelab-health/service-readiness-audiobookshelf.txt
2026-07-09T21:46:20.9338292150 225 /var/lib/homelab-health/calibre-web-backup.txt
2026-07-09T21:46:20.9338292150 242 /var/lib/homelab-health/calibre-web-offhost.txt
2026-07-09T21:46:20.9338292150 242 /var/lib/homelab-health/calibre-web-restore.txt
2026-07-09T21:57:20.3030989240 271 /var/lib/homelab-health/service-readiness-calibre-web.txt
2026-07-09T22:04:04.0733864250 236 /var/lib/homelab-health/homeassistant-backup.txt
2026-07-09T22:04:04.0733864250 252 /var/lib/homelab-health/homeassistant-offhost.txt
2026-07-09T22:04:04.0733864250 252 /var/lib/homelab-health/homeassistant-restore.txt
2026-07-09T22:07:41.2227931800 281 /var/lib/homelab-health/service-readiness-homeassistant.txt
2026-07-09T22:13:14.6980274280 225 /var/lib/homelab-health/it-tools-backup.txt
2026-07-09T22:13:14.6980274280 227 /var/lib/homelab-health/it-tools-offhost.txt
2026-07-09T22:13:14.6980274280 227 /var/lib/homelab-health/it-tools-restore.txt
2026-07-09T22:23:57.3350983150 271 /var/lib/homelab-health/service-readiness-it-tools.txt
2026-07-09T22:28:35.5724369860 214 /var/lib/homelab-health/karakeep-backup.txt
2026-07-09T22:28:35.5724369860 227 /var/lib/homelab-health/karakeep-offhost.txt
2026-07-09T22:28:35.5724369860 227 /var/lib/homelab-health/karakeep-restore.txt
2026-07-09T22:32:36.7481977520 327 /var/lib/homelab-health/service-readiness-karakeep.txt
2026-07-09T22:39:24.2375519320 194 /var/lib/homelab-health/n8n-backup.txt
2026-07-09T22:39:24.2375519320 202 /var/lib/homelab-health/n8n-offhost.txt
2026-07-09T22:39:24.2375519320 202 /var/lib/homelab-health/n8n-restore.txt
2026-07-09T22:42:27.8124145040 261 /var/lib/homelab-health/service-readiness-n8n.txt
2026-07-09T22:46:45.7324363630 214 /var/lib/homelab-health/node-red-backup.txt
2026-07-09T22:46:45.7324363630 227 /var/lib/homelab-health/node-red-offhost.txt
2026-07-09T22:46:45.7324363630 227 /var/lib/homelab-health/node-red-restore.txt
2026-07-09T22:50:18.7317577520 270 /var/lib/homelab-health/service-readiness-node-red.txt
2026-07-09T22:55:13.5023542440 217 /var/lib/homelab-health/syncthing-backup.txt
2026-07-09T22:55:13.5023542440 232 /var/lib/homelab-health/syncthing-offhost.txt
2026-07-09T22:55:13.5023542440 232 /var/lib/homelab-health/syncthing-restore.txt
2026-07-09T22:59:10.6320324800 273 /var/lib/homelab-health/service-readiness-syncthing.txt
2026-07-09T23:09:25.0745502100 254 /var/lib/homelab-health/immich-restore.txt
2026-07-09T23:09:25.0745502100 278 /var/lib/homelab-health/immich-backup.txt
2026-07-09T23:09:25.0745502100 287 /var/lib/homelab-health/immich-offhost.txt
2026-07-09T23:12:55.3968081120 349 /var/lib/homelab-health/service-readiness-immich.txt
2026-07-09T23:39:37.9114415990 206 /var/lib/homelab-health/gotify-backup.txt
2026-07-09T23:39:37.9124416140 217 /var/lib/homelab-health/gotify-offhost.txt
2026-07-09T23:39:37.9124416140 217 /var/lib/homelab-health/gotify-restore.txt
2026-07-09T23:39:41.3274938750 198 /var/lib/homelab-health/ntfy-backup.txt
2026-07-09T23:39:41.3274938750 207 /var/lib/homelab-health/ntfy-offhost.txt
2026-07-09T23:39:41.3274938750 207 /var/lib/homelab-health/ntfy-restore.txt
2026-07-09T23:39:50.5946356910 229 /var/lib/homelab-health/uptime-kuma-backup.txt
2026-07-09T23:39:50.5946356910 242 /var/lib/homelab-health/uptime-kuma-offhost.txt
2026-07-09T23:39:50.5946356910 242 /var/lib/homelab-health/uptime-kuma-restore.txt
2026-07-09T23:45:51.3911568860 311 /var/lib/homelab-health/service-readiness-nextcloud.txt
2026-07-09T23:55:38.2600376160 277 /var/lib/homelab-health/service-readiness-dns1.txt
2026-07-09T23:55:38.2600376160 277 /var/lib/homelab-health/service-readiness-dns2.txt
2026-07-09T23:55:38.2600376160 284 /var/lib/homelab-health/service-readiness-homepage.txt
2026-07-09T23:58:29.2076174390 405 /var/lib/homelab-health/external-link-aleisaevn-netcraze-5083.txt
2026-07-10T00:12:45.3264837560 292 /var/lib/homelab-health/deep-service-backup-audit.txt
2026-07-10T21:46:36.9609141470 200 /var/lib/homelab-health/alertmanager-two-cloud-backup.txt
2026-07-10T22:05:25.8570406280 189 /var/lib/homelab-health/beszel-two-cloud-backup.txt
2026-07-10T22:20:26.1397323180 209 /var/lib/homelab-health/blackbox-exporter-two-cloud-backup.txt
2026-07-10T23:00:03.1168349530 151 /var/lib/homelab-health/legacy-monitoring-backups-migrated.txt
2026-07-10T23:25:23.2350187140 422 /var/lib/homelab-health/drift-baseline-acceptance.txt
2026-07-11T00:52:36.4697413270 151 /var/lib/homelab-health/cluster-admin-inventory-sync.txt
2026-07-11T02:25:25.1542644770 188 /var/lib/homelab-health/cluster-admin-public-route.txt
2026-07-11T02:26:01.7218294760 176 /var/lib/homelab-health/cluster-admin-prometheus-integration.txt
2026-07-11T02:30:49.6252637080 249 /var/lib/homelab-health/cluster-admin-incident-engine-backup.txt
2026-07-11T04:07:23.3393000650 361 /var/lib/homelab-health/service-readiness-cluster-admin-incident-engine.txt
2026-07-11T04:39:28.9976540970 212 /var/lib/homelab-health/cluster-admin-webpanel.txt
2026-07-11T06:23:40.9114023280 361 /var/lib/homelab-health/cluster-mail-cloud-cleanup.txt
2026-07-13T01:15:40.0109283350 386 /var/lib/homelab-health/cluster-admin-stage0-remediation.txt
2026-07-13T02:08:25.4011133770 436 /var/lib/homelab-health/cluster-admin-stage0-desired-state-sync.txt
2026-07-13T02:14:59.9458929520 400 /var/lib/homelab-health/cluster-admin-stage0-convergence.txt
2026-07-13T02:22:57.9862108430 523 /var/lib/homelab-health/cluster-admin-stage0-readiness-fix.txt
2026-07-13T02:26:02.8616913120 587 /var/lib/homelab-health/cluster-admin-stage0-baseline.txt
2026-07-13T02:32:35.0280552640 515 /var/lib/homelab-health/cluster-admin-stage1-capture.txt
2026-07-13T02:35:09.6424265770 727 /var/lib/homelab-health/cluster-admin-stage1-design-inputs.txt
2026-07-13T02:46:43.8820757430 698 /var/lib/homelab-health/cluster-admin-stage1-data-shape.txt
2026-07-13T03:11:06.6805116280 1065 /var/lib/homelab-health/cluster-admin-stage1-migration-dry-run.txt
2026-07-13T03:11:06.6805116280 406 /var/lib/homelab-health/cluster-admin-stage1-migration-compatibility.txt
2026-07-13T03:11:06.7005119340 926 /var/lib/homelab-health/cluster-admin-stage1-preapply-backup.txt
2026-07-13T03:22:22.4838707540 898 /var/lib/homelab-health/cluster-admin-stage1-foundation-apply.txt
2026-07-13T03:26:26.3016083230 956 /var/lib/homelab-health/cluster-admin-stage1-final-seal.txt
2026-07-13T03:29:40.0025776360 821 /var/lib/homelab-health/appbackupctl-routing.txt
2026-07-13T03:35:17.3427496850 1215 /var/lib/homelab-health/cluster-admin-stage2-design.txt
2026-07-13T03:39:46.1788887390 1361 /var/lib/homelab-health/cluster-admin-stage2-dry-run.txt
2026-07-13T04:05:34.7337002360 854 /var/lib/homelab-health/cluster-admin-stage2-recovery.txt
2026-07-13T04:15:16.8986427760 1035 /var/lib/homelab-health/cluster-admin-stage2-placeholder-diagnosis.txt
2026-07-13T04:20:17.9572677960 1318 /var/lib/homelab-health/cluster-admin-stage2-dynamic-redryrun.txt
2026-07-13T04:24:04.9957556580 1727 /var/lib/homelab-health/cluster-admin-stage2-preapply-backup.txt
2026-07-13T04:39:27.6419256780 1705 /var/lib/homelab-health/cluster-admin-stage2-apply.txt
2026-07-13T04:49:49.9254840660 1338 /var/lib/homelab-health/cluster-admin-stage2-final-seal.txt
2026-07-13T05:04:04.9066160570 1349 /var/lib/homelab-health/cluster-admin-stage3-api-design.txt
2026-07-13T05:36:13.0711962680 854 /var/lib/homelab-health/cluster-admin-stage3-api-auth-diagnosis.txt
2026-07-13T05:45:21.0465781570 1724 /var/lib/homelab-health/cluster-admin-stage3-api-dry-run.txt
2026-07-13T05:53:43.0632562670 1433 /var/lib/homelab-health/cluster-admin-stage3-api-runtime-dry-run.txt
2026-07-13T06:05:42.4882482970 728 /var/lib/homelab-health/cluster-admin-stage3-preapply-checkpoint.txt
2026-07-13T06:15:56.2396234100 1118 /var/lib/homelab-health/cluster-admin-stage3-preapply-backup.txt
2026-07-13T06:20:26.7327543700 1191 /var/lib/homelab-health/cluster-admin-stage3-api-candidate-normalized.txt
2026-07-13T06:22:40.5977987490 1106 /var/lib/homelab-health/cluster-admin-stage3-api-apply.txt
2026-07-13T06:25:40.2405422500 732 /var/lib/homelab-health/cluster-admin-stage3-production-verifier-prepared.txt
2026-07-13T06:28:56.1285342090 1207 /var/lib/homelab-health/cluster-admin-stage3-production-verify.txt
2026-07-13T06:44:42.7609957740 972 /var/lib/homelab-health/cluster-admin-stage3-postapply-backup.txt
2026-07-13T06:47:15.5353299170 1281 /var/lib/homelab-health/cluster-admin-stage3-final-seal.txt
2026-07-13T06:50:50.2356102460 507 /var/lib/homelab-health/crypto-portfolio-remove-grt-discovery.txt
2026-07-13T06:52:07.9807980800 488 /var/lib/homelab-health/crypto-portfolio-remove-grt-target.txt
2026-07-13T07:00:02.5170483350 822 /var/lib/homelab-health/crypto-portfolio-remove-grt-current.txt
2026-07-13T07:05:43.6132589850 879 /var/lib/homelab-health/crypto-portfolio-remove-grt-surface.txt
2026-07-13T07:20:46.9160557760 1741 /var/lib/homelab-health/crypto-portfolio-remove-grt-candidate.txt
2026-07-13T07:35:30.8895464510 1403 /var/lib/homelab-health/crypto-portfolio-remove-grt-runtime-verifier-prepared.txt
2026-07-13T07:41:31.9430520860 1497 /var/lib/homelab-health/crypto-portfolio-remove-grt-runtime-dry-run.txt
2026-07-13T07:52:08.1187504330 1724 /var/lib/homelab-health/crypto-portfolio-remove-grt-browser-smoke.txt
2026-07-13T08:05:15.1507543740 762 /var/lib/homelab-health/crypto-portfolio-remove-grt-desired-state-diagnostic.txt
2026-07-13T08:07:21.2856783820 1184 /var/lib/homelab-health/crypto-portfolio-remove-grt-desired-state-model.txt
2026-07-13T08:10:44.7277816120 1647 /var/lib/homelab-health/crypto-portfolio-remove-grt-prechange-backup.txt
2026-07-13T08:39:10.8547948980 330 /var/lib/homelab-health/crypto-portfolio-remove-grt-recovery-audit.txt
2026-07-13T08:41:01.3664795150 684 /var/lib/homelab-health/crypto-portfolio-remove-grt-failed-units-diagnostic.txt
2026-07-13T08:48:13.2600633440 1640 /var/lib/homelab-health/crypto-portfolio-remove-grt-production-apply.txt
2026-07-13T09:11:24.1542650490 1347 /var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-candidate.txt
2026-07-13T10:04:47.2300377080 1490 /var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-browser-smoke.txt
2026-07-13T10:42:57.1487691660 1851 /var/lib/homelab-health/crypto-portfolio-ui-preapply-backup.txt
2026-07-13T11:12:13.3023976030 192 /var/lib/homelab-health/crypto-portfolio-restore.txt
2026-07-13T11:12:13.3023976030 266 /var/lib/homelab-health/crypto-portfolio-offhost.txt
2026-07-13T11:12:13.3023976030 325 /var/lib/homelab-health/service-readiness-crypto-portfolio.txt
2026-07-13T11:12:13.3023976030 353 /var/lib/homelab-health/crypto-portfolio-backup.txt
2026-07-13T11:12:14.9294222730 757 /var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-production-closure.txt
2026-07-13T11:37:59.5159486840 1407 /var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-final-seal.txt
2026-07-13T11:37:59.5171037700 1407 /var/lib/homelab-health/crypto-portfolio-runtime-final-seal.txt
2026-07-13T14:03:06.0152234300 388 /var/lib/homelab-health/hapusya-site-preapply-backup.txt
2026-07-13T14:34:59.2083653510 533 /var/lib/homelab-health/hapusya-site-v2-preapply-backup.txt
2026-07-13T14:45:27.4649272800 793 /var/lib/homelab-health/hapusya-site-production-apply.txt
2026-07-13T14:48:15.6134864600 694 /var/lib/homelab-health/hapusya-site-production-browser.txt
2026-07-13T15:10:58.1372215280 156 /var/lib/homelab-health/hapusya-site-offhost.txt
2026-07-13T15:10:58.1372215280 185 /var/lib/homelab-health/hapusya-site-retention.txt
2026-07-13T15:10:58.1372215280 194 /var/lib/homelab-health/hapusya-site-restore.txt
2026-07-13T15:10:58.1372215280 499 /var/lib/homelab-health/hapusya-site-backup.txt
2026-07-13T15:10:58.1382215430 424 /var/lib/homelab-health/hapusya-site-backup-path-mapping.txt
2026-07-13T15:26:19.5872390600 651 /var/lib/homelab-health/hapusya-site-desired-state-sync.txt
2026-07-13T15:33:08.6444630100 1520 /var/lib/homelab-health/hapusya-site-final-seal.txt
2026-07-13T20:29:02.8967807310 628 /var/lib/homelab-health/backup-matrix-vm-health-feedback-fix.txt
2026-07-13T20:40:37.0653547720 648 /var/lib/homelab-health/backup-matrix-desired-state-persistence.txt
2026-07-13T20:48:14.6313253050 943 /var/lib/homelab-health/backup-matrix-cluster-preseal-convergence.txt
2026-07-13T20:51:32.4163384480 887 /var/lib/homelab-health/backup-matrix-cluster-desired-state-sync.txt
2026-07-13T20:51:32.4223385400 1445 /var/lib/homelab-health/backup-matrix-cluster-final-seal.txt
2026-07-13T21:13:01.1129724320 556 /var/lib/homelab-health/mail-cloud-edge-vm-runtime-contract-fix.txt
2026-07-13T21:15:43.9664537840 670 /var/lib/homelab-health/mail-cloud-edge-vm-readonly-probe.txt
2026-07-13T21:27:22.2230931620 630 /var/lib/homelab-health/mail-cloud-edge-vm-controlled-no-retention-mode.txt
2026-07-13T21:39:22.4540680390 719 /var/lib/homelab-health/mail-cloud-edge-vm-controlled-upload.txt
2026-07-13T21:40:21.6099694610 408 /var/lib/homelab-health/mail-cloud-edge-vm-service-convergence.txt
2026-07-13T21:56:30.9657631360 881 /var/lib/homelab-health/mail-cloud-edge-vm-matrix-registration.txt
2026-07-14T04:02:58.3995631560 1245 /var/lib/homelab-health/mail-cloud-edge-vm-final-seal.txt
2026-07-14T04:12:54.7747139510 707 /var/lib/homelab-health/remaining-special-families-matrix-registration.txt
2026-07-14T04:15:48.2773761880 1115 /var/lib/homelab-health/special-vm-emergency-non-app-final-seal.txt
2026-07-14T04:47:22.1524259850 902 /var/lib/homelab-health/cluster-admin-node-online-green-dot-fix.txt
2026-07-14T04:54:55.1333752320 1021 /var/lib/homelab-health/cluster-admin-collapsible-health-sections.txt
2026-07-14T08:33:26.7124814180 629 /var/lib/homelab-health/cluster-admin-stage4c-database-migration-dry-run.txt
2026-07-14T08:48:04.5210170900 482 /var/lib/homelab-health/homelab-admin-runner.txt
2026-07-14T09:14:58.6397106130 702 /var/lib/homelab-health/cluster-admin-stage4d-adapter-candidates-in-isolation.txt
2026-07-14T09:39:00.0749801060 610 /var/lib/homelab-health/cluster-admin-stage4e-restricted-probe-agent-design.txt
2026-07-14T10:39:53.2848761800 600 /var/lib/homelab-health/homelab-admin-guardrails.txt
2026-07-14T10:54:50.6815482590 692 /var/lib/homelab-health/cluster-admin-stage4f-preapply-backup.txt
2026-07-14T20:02:48.4191051180 266 /var/lib/homelab-health/cluster-admin-stage4g-production-apply-read-only.txt
2026-07-14T20:16:32.3610333130 213 /var/lib/homelab-health/operator-rule-close-tails-immediately.txt
2026-07-14T20:23:46.5434583710 328 /var/lib/homelab-health/cluster-admin-stage4h-scope-definition.txt
2026-07-14T20:29:41.0475984260 268 /var/lib/homelab-health/cluster-admin-stage4h-collector-baseline-audit.txt
2026-07-14T20:41:34.5353531810 455 /var/lib/homelab-health/cluster-admin-stage4h-collector-patch-candidate.txt
2026-07-14T21:26:54.1724254520 226 /var/lib/homelab-health/operator-command-size-guard.txt
2026-07-14T21:29:30.0818406610 312 /var/lib/homelab-health/cluster-admin-stage4h-final-acceptance-contract.txt
2026-07-14T21:33:40.0697208200 209 /var/lib/homelab-health/homelab-admin-cli-contract.txt
2026-07-14T21:53:16.5230291610 357 /var/lib/homelab-health/cluster-admin-stage4h-task-package-build.txt
2026-07-15T10:10:19.9159742040 1732 /var/lib/homelab-health/cluster-admin-stage4h-final-acceptance.txt
2026-07-15T10:17:53.0215082300 921 /var/lib/homelab-health/cluster-admin-stage4i-scope-definition.txt
2026-07-16T01:32:11.0876121490 237 /var/lib/homelab-health/homelab-ops-bootstrap.txt
2026-07-16T09:20:34.5941094330 75 /var/lib/homelab-health/router-hard-power.txt
2026-07-16T12:14:29.4893285610 202 /var/lib/homelab-health/xenforo-branding-rollout.txt
2026-07-16T12:41:28.8422409190 192 /var/lib/homelab-health/xenforo-branding-header-visibility.txt
2026-07-16T12:48:31.4566023130 185 /var/lib/homelab-health/xenforo-branding-header-order.txt
2026-07-16T13:01:16.2035467580 250 /var/lib/homelab-health/homelab-ops-worker-vm190.txt
2026-07-17T04:59:42.6476420400 242 /var/lib/homelab-health/netbird-vps-offhost.txt
2026-07-17T09:31:15.5873108310 443 /var/lib/homelab-health/paperless-offhost.txt
2026-07-17T14:21:55.3567278930 272 /var/lib/homelab-health/cluster-storage-retention.txt
2026-07-18T01:52:29.8931662090 684 /var/lib/homelab-health/paperless-backup.txt
2026-07-18T01:54:49.4850009180 728 /var/lib/homelab-health/netbird-vps-trust.txt
2026-07-18T02:54:28.2167554790 200 /var/lib/homelab-health/full-handoff-recovery.txt
2026-07-19T09:41:56.0870397860 437 /var/lib/homelab-health/mail-cloud-edge-vm.txt
2026-07-20T09:13:15.5923007820 193 /var/lib/homelab-health/memos-offhost.txt
2026-07-21T00:33:12.9785038850 214 /var/lib/homelab-health/evidence-seal/latest.env
2026-07-21T02:37:04.7969419110 195 /var/lib/homelab-health/cluster-admin-observer.txt
2026-07-21T02:37:04.7969419110 200 /var/lib/homelab-health/cluster-admin-restricted-probes.txt
2026-07-21T02:37:04.7969419110 219 /var/lib/homelab-health/cluster-admin-full-observer.txt
2026-07-21T02:39:14.3120870600 153 /var/lib/homelab-health/vm160-remote-backup.txt
2026-07-21T04:55:35.3129057980 316 /var/lib/homelab-health/edge-vm-vzdump-offhost.txt
2026-07-21T04:55:35.3129057980 334 /var/lib/homelab-health/edge-vm-vzdump-restore.txt
2026-07-21T04:55:35.3129057980 409 /var/lib/homelab-health/edge-vm-offhost-freshness.txt
2026-07-21T04:55:35.3408368720 314 /var/lib/homelab-health/sops-edge-secret-coverage.txt
2026-07-21T05:18:05.9056051500 633 /var/lib/homelab-health/router-running-config-mail-cloud.txt
2026-07-21T05:42:36.0601490370 236 /var/lib/homelab-health/vps-identity-audit.txt
2026-07-21T06:42:44.4264139010 339 /var/lib/homelab-health/mail-cloud-critical.txt
2026-07-21T07:05:43.0783306490 1583 /var/lib/homelab-health/backup-last-run.log
2026-07-21T07:05:43.0870410520 217 /var/lib/homelab-health/backup.txt
2026-07-21T07:05:43.0879592500 274 /var/lib/homelab-health/backup.json
2026-07-21T07:15:46.8155363020 232 /var/lib/homelab-health/drift-check.txt
2026-07-21T07:36:59.5343449800 170 /var/lib/homelab-health/vm-backup.txt
2026-07-21T07:49:15.8201883610 136 /var/lib/homelab-health/desired-state.txt
2026-07-21T07:56:14.9677282530 146 /var/lib/homelab-health/runbooks.txt
2026-07-21T07:56:26.4127609750 475 /var/lib/homelab-health/mail-cloud-nextcloud-vm.txt
2026-07-21T08:01:15.0701844650 1522 /var/lib/homelab-health/overall.txt
2026-07-21T08:10:47.5199010040 117 /var/lib/homelab-health/duty-admin.txt
2026-07-21T08:29:16.9008054160 178 /var/lib/homelab-health/capacity-risk.txt
2026-07-21T08:31:47.8421084810 128 /var/lib/homelab-health/final-readiness.txt
2026-07-21T08:35:41.8646792150 147 /var/lib/homelab-health/secret-exposure.txt
2026-07-21T08:50:15.1546349960 698 /var/lib/homelab-health/filebrowser-backup.txt
2026-07-21T08:50:15.1749916820 796 /var/lib/homelab-health/filebrowser-restore.txt
2026-07-21T08:50:15.2338941910 811 /var/lib/homelab-health/npmplus-kuma-config-backup.txt
2026-07-21T08:50:15.2339779780 812 /var/lib/homelab-health/npmplus-kuma-config-offhost.txt
2026-07-21T08:50:15.2340536730 812 /var/lib/homelab-health/npmplus-kuma-config-restore.txt
2026-07-21T08:50:15.2530255710 754 /var/lib/homelab-health/authentik-backup.txt
2026-07-21T08:50:15.2531096640 755 /var/lib/homelab-health/authentik-offhost.txt
2026-07-21T08:50:15.2531846660 755 /var/lib/homelab-health/authentik-restore.txt
2026-07-21T08:50:15.2737214530 738 /var/lib/homelab-health/gitea-backup.txt
2026-07-21T08:50:15.2738013060 739 /var/lib/homelab-health/gitea-offhost.txt
2026-07-21T08:50:15.2738768280 739 /var/lib/homelab-health/gitea-restore.txt
2026-07-21T08:50:15.2739499850 744 /var/lib/homelab-health/vaultwarden-backup.txt
2026-07-21T08:50:15.2740214440 745 /var/lib/homelab-health/vaultwarden-offhost.txt
2026-07-21T08:50:15.2740905310 745 /var/lib/homelab-health/vaultwarden-restore.txt
2026-07-21T08:50:15.3412420180 805 /var/lib/homelab-health/backup-coverage-matrix.txt
2026-07-21T08:50:15.4230013070 769 /var/lib/homelab-health/edge-vm-vzdump-backup.txt
2026-07-21T08:50:15.4602967240 776 /var/lib/homelab-health/extended-appbackup.txt
2026-07-21T08:50:15.6009660870 829 /var/lib/homelab-health/mail-cloud-restore-drill.txt
2026-07-21T08:50:15.6983060740 802 /var/lib/homelab-health/backup-framework.txt
2026-07-21T08:50:15.7459043920 760 /var/lib/homelab-health/vm-backup-policy.txt
2026-07-21T08:50:15.7701962780 763 /var/lib/homelab-health/vm170-vm171-full-backup.txt
2026-07-21T08:50:15.7943029380 822 /var/lib/homelab-health/vm170-vm171-full-restore-drill.txt
2026-07-21T08:50:15.8187786300 800 /var/lib/homelab-health/vm-local-dumps-cloud.txt
2026-07-21T08:50:15.8439264310 826 /var/lib/homelab-health/vm-local-dumps-retention.txt
2026-07-21T08:50:15.8671718640 718 /var/lib/homelab-health/immich-media-backup.txt
2026-07-21T08:50:15.8912318140 656 /var/lib/homelab-health/memos-backup.txt
2026-07-21T08:50:15.9706145680 773 /var/lib/homelab-health/paperless-restore.txt
2026-07-21T08:51:15.1419524730 115 /var/lib/homelab-health/cluster-passport.txt
2026-07-21T08:51:15.4733058530 323 /var/lib/homelab-health/immich-media-offhost.txt
2026-07-21T08:54:09.3706207380 509 /var/lib/homelab-health/golden-state.txt
2026-07-21T09:00:14.9472448680 142 /var/lib/homelab-health/incident-journal.txt
2026-07-21T09:00:16.1272438700 295 /var/lib/homelab-health/alerting.txt
2026-07-21T09:00:16.5952705670 327 /var/lib/homelab-health/kuma-monitor-policy.txt
2026-07-21T09:00:17.1552793000 154 /var/lib/homelab-health/backup-sla.txt
2026-07-21T09:00:17.1710283480 141 /var/lib/homelab-health/service-registry.txt
2026-07-21T09:00:17.1903657410 128 /var/lib/homelab-health/dependency-map.txt
2026-07-21T09:00:17.1922798770 142 /var/lib/homelab-health/safe-autoheal.txt
2026-07-21T09:00:35.8775712520 539 /var/lib/homelab-health/external-probe-vps.txt
2026-07-21T09:00:42.8766803960 233 /var/lib/homelab-health/forum-snuffleupagus.txt
2026-07-21T09:01:18.5488450380 317 /var/lib/homelab-health/duty-admin-v2.txt
2026-07-21T09:01:30.9278621010 489 /var/lib/homelab-health/disk-space.txt
2026-07-21T09:02:24.8882711660 170 /var/lib/homelab-health/mkdocs-auto-refresh.txt
2026-07-21T09:04:30.1762249020 0 /var/lib/homelab-health/pve03-staging-capacity.txt.err
2026-07-21T09:04:30.5477223940 207 /var/lib/homelab-health/pve03-staging-capacity.txt
2026-07-21T09:10:15.0418809250 139 /var/lib/homelab-health/crypto-portfolio-chart-health.txt
2026-07-21T09:10:15.5101489600 215 /var/lib/homelab-health/cluster-admin-webpanel-sync.txt
2026-07-21T09:10:17.6889902650 641 /var/lib/homelab-health/private-vpn-hosts.txt
2026-07-21T09:10:47.2970978580 227 /var/lib/homelab-health/router-watchdog.txt
2026-07-21T09:10:48.3731143150 477 /var/lib/homelab-health/netbird-peers.txt
----- DIR=/var/lib/homelab-backup-matrix -----
2026-07-01T04:31:05.4336469050 312 /var/lib/homelab-backup-matrix/evidence/netbird_vps_backup/0-netbird-vps-backup.txt
2026-07-07T18:24:59.9494760330 357 /var/lib/homelab-backup-matrix/evidence/netbird_vps_backup/0-netbird-vps-trust.txt
2026-07-09T10:22:35.4762594970 286 /var/lib/homelab-backup-matrix/evidence/immich_media_backup/0-immich-media-backup.txt
2026-07-09T10:30:21.2893338800 393 /var/lib/homelab-backup-matrix/evidence/homelab_npmplus_kuma_config_backup/1-npmplus-kuma-config-offhost.txt
2026-07-09T10:30:21.2893338800 399 /var/lib/homelab-backup-matrix/evidence/homelab_npmplus_kuma_config_backup/2-npmplus-kuma-config-restore.txt
2026-07-09T10:30:21.2893338800 416 /var/lib/homelab-backup-matrix/evidence/homelab_npmplus_kuma_config_backup/0-npmplus-kuma-config-backup.txt
2026-07-09T15:42:17.7925620460 209 /var/lib/homelab-backup-matrix/evidence/filebrowser_restore_validate/0-filebrowser-restore.txt
2026-07-09T15:42:17.7925620460 263 /var/lib/homelab-backup-matrix/evidence/filebrowser_backup/0-filebrowser-backup.txt
2026-07-09T19:42:55.8422576240 221 /var/lib/homelab-backup-matrix/evidence/paperless_backup/0-paperless-backup.txt
2026-07-09T19:42:55.8422576240 232 /var/lib/homelab-backup-matrix/evidence/paperless_restore_dry_run/0-paperless-restore.txt
2026-07-09T19:49:28.4202504550 201 /var/lib/homelab-backup-matrix/evidence/memos_backup/0-memos-backup.txt
2026-07-09T23:31:32.0510186090 353 /var/lib/homelab-backup-matrix/evidence/homelab_mail_cloud_restore_drill/0-mail-cloud-restore-drill.txt
2026-07-09T23:39:31.6643460000 221 /var/lib/homelab-backup-matrix/evidence/homelab_authentik_backup/0-authentik-backup.txt
2026-07-09T23:39:31.6643460000 232 /var/lib/homelab-backup-matrix/evidence/homelab_authentik_backup/1-authentik-offhost.txt
2026-07-09T23:39:31.6643460000 232 /var/lib/homelab-backup-matrix/evidence/homelab_authentik_backup/2-authentik-restore.txt
2026-07-09T23:39:36.8524253930 205 /var/lib/homelab-backup-matrix/evidence/homelab_gitea_backup/0-gitea-backup.txt
2026-07-09T23:39:36.8524253930 212 /var/lib/homelab-backup-matrix/evidence/homelab_gitea_backup/1-gitea-offhost.txt
2026-07-09T23:39:36.8524253930 212 /var/lib/homelab-backup-matrix/evidence/homelab_gitea_backup/2-gitea-restore.txt
2026-07-09T23:39:53.9766874470 226 /var/lib/homelab-backup-matrix/evidence/homelab_vaultwarden_backup/0-vaultwarden-backup.txt
2026-07-09T23:39:53.9766874470 242 /var/lib/homelab-backup-matrix/evidence/homelab_vaultwarden_backup/1-vaultwarden-offhost.txt
2026-07-09T23:39:53.9766874470 242 /var/lib/homelab-backup-matrix/evidence/homelab_vaultwarden_backup/2-vaultwarden-restore.txt
2026-07-10T04:59:04.1128229600 277 /var/lib/homelab-backup-matrix/evidence/homelab_vm_backup_health/1-edge-vm-vzdump-backup.txt
2026-07-10T12:47:24.3262767100 11477 /var/lib/homelab-backup-matrix/evidence-map.json
2026-07-10T12:57:44.4526176160 1657 /var/lib/homelab-backup-matrix/legacy-health-cutover-paths.txt
2026-07-10T13:10:05.0808425340 11888 /var/lib/homelab-backup-matrix/migration-plan.json
2026-07-12T04:56:52.9608056990 277 /var/lib/homelab-backup-matrix/evidence/homelab_edge_vm_vzdump_backup/0-edge-vm-vzdump-backup.txt
2026-07-13T20:26:36.9252845450 748 /var/lib/homelab-backup-matrix/compat/homelab_vm_backup_health--backup.txt--62f47c24aa6b.txt
2026-07-13T20:26:36.9253893680 764 /var/lib/homelab-backup-matrix/compat/homelab_vm_backup_health--edge-vm-vzdump-offhost.txt--6b16ee077767.txt
2026-07-13T20:26:36.9254686050 764 /var/lib/homelab-backup-matrix/compat/homelab_vm_backup_health--edge-vm-vzdump-restore.txt--bfe4c7a45ee2.txt
2026-07-13T20:26:36.9255441420 751 /var/lib/homelab-backup-matrix/compat/homelab_vm_backup_health--vm-backup.txt--11c1b9f3c8ae.txt
2026-07-15T08:44:36.3676757710 515 /var/lib/homelab-backup-matrix/evidence/homelab_vm_full_restore_drill/1-latest-restore-drill.txt
2026-07-15T08:44:36.3692351120 515 /var/lib/homelab-backup-matrix/evidence/homelab_vm_full_restore_drill/0-vm170-vm171-full-restore-drill.txt
2026-07-18T01:52:29.8722644570 731 /var/lib/homelab-backup-matrix/compat/netbird_vps_backup--netbird-vps-trust.txt--44dfdc0d5d0a.txt
2026-07-18T01:52:29.8931212020 684 /var/lib/homelab-backup-matrix/compat/paperless_backup--paperless-backup.txt--b3a9ed8e7905.txt
2026-07-19T05:11:28.9578382090 404 /var/lib/homelab-backup-matrix/evidence/skladchik_reports_monitor_backup_restore_check/0-restore-check-health.txt
2026-07-19T09:41:56.0870397860 437 /var/lib/homelab-backup-matrix/evidence/homelab_mail_cloud_edge_vm_upload/0-mail-cloud-edge-vm.txt
2026-07-19T10:13:40.6950277460 544 /var/lib/homelab-backup-matrix/evidence/homelab_vm_local_dumps_retention/1-latest-vm-local-dumps-cloud.json
2026-07-19T10:13:40.6960277610 445 /var/lib/homelab-backup-matrix/evidence/homelab_vm_local_dumps_2cloud/0-vm-local-dumps-cloud.txt
2026-07-19T11:08:42.8436095010 184 /var/lib/homelab-backup-matrix/evidence/homelab_vm_local_dumps_retention/0-vm-local-dumps-retention.txt
2026-07-20T08:47:17.4079908050 509 /var/lib/homelab-backup-matrix/evidence/homelab_golden_state_index/0-golden-state.txt
2026-07-20T17:21:30.2733358650 641 /var/lib/homelab-backup-matrix/evidence/homelab_npmplus_kuma_config_backup/3-private-vpn-hosts.txt
2026-07-21T02:39:14.3120870600 153 /var/lib/homelab-backup-matrix/evidence/vm160_remote_backup/0-vm160-remote-backup.txt
2026-07-21T03:09:43.9100609310 485 /var/lib/homelab-backup-matrix/evidence/homelab_vm_full_backup/1-latest-full-backup.txt
2026-07-21T03:09:43.9161554950 485 /var/lib/homelab-backup-matrix/evidence/homelab_vm_full_backup/0-vm170-vm171-full-backup.txt
2026-07-21T03:13:42.8950037020 553 /var/lib/homelab-backup-matrix/evidence/homelab_emergency_backup/0-latest-emergency-pack.json
2026-07-21T03:13:42.8950037020 553 /var/lib/homelab-backup-matrix/evidence/homelab_verified_backup_health/1-latest-emergency-pack.json
2026-07-21T03:51:53.5230549890 601 /var/lib/homelab-backup-matrix/evidence/homelab_appbackup_dockge_stacks/0-latest-dockge-stacks.json
2026-07-21T04:19:48.6158169700 880 /var/lib/homelab-backup-matrix/evidence/homelab_appbackup_npmplus_edge/0-latest-npmplus-edge.json
2026-07-21T04:35:22.4622843520 484 /var/lib/homelab-backup-matrix/evidence/skladchik_reports_monitor_backup/0-health.txt
2026-07-21T04:55:35.3129057980 316 /var/lib/homelab-backup-matrix/evidence/homelab_vm_backup_health/2-edge-vm-vzdump-offhost.txt
2026-07-21T04:55:35.3129057980 334 /var/lib/homelab-backup-matrix/evidence/homelab_vm_backup_health/3-edge-vm-vzdump-restore.txt
2026-07-21T05:18:05.9056051500 633 /var/lib/homelab-backup-matrix/evidence/homelab_router_running_config_mail_cloud/0-router-running-config-mail-cloud.txt
2026-07-21T06:06:45.3183873770 213 /var/lib/homelab-backup-matrix/evidence/homelab_extended_appbackup/0-extended-appbackup.txt
2026-07-21T06:42:44.4264139010 339 /var/lib/homelab-backup-matrix/evidence/homelab_mail_cloud_critical_upload/0-mail-cloud-critical.txt
2026-07-21T06:45:16.5316587930 155 /var/lib/homelab-backup-matrix/evidence/homelab_verified_backup_health/2-backup-framework.txt
2026-07-21T06:45:16.5316587930 7256 /var/lib/homelab-backup-matrix/evidence/homelab_verified_backup_health/0-backup-registry-latest.txt
2026-07-21T07:05:43.0870410520 217 /var/lib/homelab-backup-matrix/evidence/homelab_vm_backup_health/0-backup.txt
2026-07-21T07:36:59.5343449800 170 /var/lib/homelab-backup-matrix/evidence/homelab_vm_backup_health/4-vm-backup.txt
2026-07-21T07:56:26.4127609750 475 /var/lib/homelab-backup-matrix/evidence/homelab_mail_cloud_nextcloud_vm_upload/0-mail-cloud-nextcloud-vm.txt
2026-07-21T08:30:13.9066752070 185 /var/lib/homelab-backup-matrix/evidence/homelab_vm_backup_policy/0-vm-backup-policy.txt
2026-07-21T08:45:17.1624701130 154 /var/lib/homelab-backup-matrix/evidence/homelab_backup_sla_health/0-backup-sla.txt
2026-07-21T08:50:15.0600323370 317 /var/lib/homelab-backup-matrix/evidence/homelab_backup_coverage_matrix/0-backup-coverage-matrix.txt
2026-07-21T08:50:15.1342359450 492 /var/lib/homelab-backup-matrix/rendered/dpkg-db-backup.txt
2026-07-21T08:50:15.1540337770 538 /var/lib/homelab-backup-matrix/rendered/filebrowser-backup.txt
2026-07-21T08:50:15.1545768810 698 /var/lib/homelab-backup-matrix/compat/filebrowser_backup--filebrowser-backup.txt--51237e75817c.txt
2026-07-21T08:50:15.1748329110 625 /var/lib/homelab-backup-matrix/rendered/filebrowser-restore-validate.txt
2026-07-21T08:50:15.1749419720 796 /var/lib/homelab-backup-matrix/compat/filebrowser_restore_validate--filebrowser-restore.txt--0340a8b1a5de.txt
2026-07-21T08:50:15.1958070440 637 /var/lib/homelab-backup-matrix/rendered/homelab-appbackup-dockge-stacks.txt
2026-07-21T08:50:15.2143009170 577 /var/lib/homelab-backup-matrix/rendered/homelab-appbackup-mariadb-batch.txt
2026-07-21T08:50:15.2337385240 631 /var/lib/homelab-backup-matrix/rendered/homelab-appbackup-npmplus-edge.txt
2026-07-21T08:50:15.2338453140 811 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_npmplus_edge--npmplus-kuma-config-backup.txt--f5c6697bf2fc.txt
2026-07-21T08:50:15.2339418310 812 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_npmplus_edge--npmplus-kuma-config-offhost.txt--1638681c118d.txt
2026-07-21T08:50:15.2340198050 812 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_npmplus_edge--npmplus-kuma-config-restore.txt--6f07430aa43a.txt
2026-07-21T08:50:15.2340955590 802 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_npmplus_edge--private-vpn-hosts.txt--a14118858bd9.txt
2026-07-21T08:50:15.2528809530 582 /var/lib/homelab-backup-matrix/rendered/homelab-appbackup-postgres-batch.txt
2026-07-21T08:50:15.2529784080 754 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_postgres_batch--authentik-backup.txt--800cae69eb31.txt
2026-07-21T08:50:15.2530726290 755 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_postgres_batch--authentik-offhost.txt--d42912853ad6.txt
2026-07-21T08:50:15.2531503130 755 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_postgres_batch--authentik-restore.txt--b81c6165c2a7.txt
2026-07-21T08:50:15.2735658650 572 /var/lib/homelab-backup-matrix/rendered/homelab-appbackup-sqlite-batch.txt
2026-07-21T08:50:15.2736758420 738 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_sqlite_batch--gitea-backup.txt--4b591c9d4b9b.txt
2026-07-21T08:50:15.2737669460 739 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_sqlite_batch--gitea-offhost.txt--0fe7014b9333.txt
2026-07-21T08:50:15.2738429560 739 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_sqlite_batch--gitea-restore.txt--d0854df47f34.txt
2026-07-21T08:50:15.2739171710 744 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_sqlite_batch--vaultwarden-backup.txt--0c7639e2a8c2.txt
2026-07-21T08:50:15.2739891140 745 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_sqlite_batch--vaultwarden-offhost.txt--41b1d10c7f2a.txt
2026-07-21T08:50:15.2740584210 745 /var/lib/homelab-backup-matrix/compat/homelab_appbackup_sqlite_batch--vaultwarden-restore.txt--2247a2977d58.txt
2026-07-21T08:50:15.3019034840 607 /var/lib/homelab-backup-matrix/rendered/homelab-authentik-backup.txt
2026-07-21T08:50:15.3210502310 518 /var/lib/homelab-backup-matrix/rendered/homelab-auto-backup.txt
2026-07-21T08:50:15.3409347590 629 /var/lib/homelab-backup-matrix/rendered/homelab-backup-coverage-matrix.txt
2026-07-21T08:50:15.3411214110 805 /var/lib/homelab-backup-matrix/compat/homelab_backup_coverage_matrix--backup-coverage-matrix.txt--fcdfdafd14d8.txt
2026-07-21T08:50:15.3609141800 542 /var/lib/homelab-backup-matrix/rendered/homelab-backup-freshness.txt
2026-07-21T08:50:15.3807132660 551 /var/lib/homelab-backup-matrix/rendered/homelab-backup-healthcheck.txt
2026-07-21T08:50:15.3997651780 598 /var/lib/homelab-backup-matrix/rendered/homelab-backup-sla-health.txt
2026-07-21T08:50:15.3998838880 757 /var/lib/homelab-backup-matrix/compat/homelab_backup_sla_health--backup-sla.txt--b7b83ec539a3.txt
2026-07-21T08:50:15.4228298640 595 /var/lib/homelab-backup-matrix/rendered/homelab-edge-vm-vzdump-backup.txt
2026-07-21T08:50:15.4229475270 769 /var/lib/homelab-backup-matrix/compat/homelab_edge_vm_vzdump_backup--edge-vm-vzdump-backup.txt--112363b34713.txt
2026-07-21T08:50:15.4427017570 603 /var/lib/homelab-backup-matrix/rendered/homelab-emergency-backup.txt
2026-07-21T08:50:15.4601250520 608 /var/lib/homelab-backup-matrix/rendered/homelab-extended-appbackup.txt
2026-07-21T08:50:15.4602433910 776 /var/lib/homelab-backup-matrix/compat/homelab_extended_appbackup--extended-appbackup.txt--9e6d2ebe69e3.txt
2026-07-21T08:50:15.4846247920 583 /var/lib/homelab-backup-matrix/rendered/homelab-gitea-backup.txt
2026-07-21T08:50:15.5053264870 603 /var/lib/homelab-backup-matrix/rendered/homelab-golden-state-index.txt
2026-07-21T08:50:15.5261090100 649 /var/lib/homelab-backup-matrix/rendered/homelab-mail-cloud-critical-upload.txt
2026-07-21T08:50:15.5497812040 645 /var/lib/homelab-backup-matrix/rendered/homelab-mail-cloud-edge-vm-upload.txt
2026-07-21T08:50:15.5751798050 673 /var/lib/homelab-backup-matrix/rendered/homelab-mail-cloud-nextcloud-vm-upload.txt
2026-07-21T08:50:15.6005257930 649 /var/lib/homelab-backup-matrix/rendered/homelab-mail-cloud-restore-drill.txt
2026-07-21T08:50:15.6008056360 829 /var/lib/homelab-backup-matrix/compat/homelab_mail_cloud_restore_drill--mail-cloud-restore-drill.txt--aec14f187258.txt
2026-07-21T08:50:15.6302773490 657 /var/lib/homelab-backup-matrix/rendered/homelab-npmplus-kuma-config-backup.txt
2026-07-21T08:50:15.6516842010 693 /var/lib/homelab-backup-matrix/rendered/homelab-router-running-config-mail-cloud.txt
2026-07-21T08:50:15.6783067700 619 /var/lib/homelab-backup-matrix/rendered/homelab-vaultwarden-backup.txt
2026-07-21T08:50:15.6981173180 632 /var/lib/homelab-backup-matrix/rendered/homelab-verified-backup-health.txt
2026-07-21T08:50:15.6982460880 802 /var/lib/homelab-backup-matrix/compat/homelab_verified_backup_health--backup-framework.txt--5cdc2f4dfaac.txt
2026-07-21T08:50:15.7231207570 589 /var/lib/homelab-backup-matrix/rendered/homelab-vm-backup-health.txt
2026-07-21T08:50:15.7457453080 596 /var/lib/homelab-backup-matrix/rendered/homelab-vm-backup-policy.txt
2026-07-21T08:50:15.7458568190 760 /var/lib/homelab-backup-matrix/compat/homelab_vm_backup_policy--vm-backup-policy.txt--b7c9d2d099d3.txt
2026-07-21T08:50:15.7698895230 594 /var/lib/homelab-backup-matrix/rendered/homelab-vm-full-backup.txt
2026-07-21T08:50:15.7700805790 763 /var/lib/homelab-backup-matrix/compat/homelab_vm_full_backup--vm170-vm171-full-backup.txt--84ca9b2d5334.txt
2026-07-21T08:50:15.7941438050 639 /var/lib/homelab-backup-matrix/rendered/homelab-vm-full-restore-drill.txt
2026-07-21T08:50:15.7942564800 822 /var/lib/homelab-backup-matrix/compat/homelab_vm_full_restore_drill--vm170-vm171-full-restore-drill.txt--296c241364b3.txt
2026-07-21T08:50:15.8186148570 627 /var/lib/homelab-backup-matrix/rendered/homelab-vm-local-dumps-2cloud.txt
2026-07-21T08:50:15.8187301490 800 /var/lib/homelab-backup-matrix/compat/homelab_vm_local_dumps_2cloud--vm-local-dumps-cloud.txt--b8453dc56a16.txt
2026-07-21T08:50:15.8437441190 646 /var/lib/homelab-backup-matrix/rendered/homelab-vm-local-dumps-retention.txt
2026-07-21T08:50:15.8438694010 826 /var/lib/homelab-backup-matrix/compat/homelab_vm_local_dumps_retention--vm-local-dumps-retention.txt--bba459c6cd8c.txt
2026-07-21T08:50:15.8667988910 556 /var/lib/homelab-backup-matrix/rendered/immich-media-backup.txt
2026-07-21T08:50:15.8670229770 718 /var/lib/homelab-backup-matrix/compat/immich_media_backup--immich-media-backup.txt--8994915171ad.txt
2026-07-21T08:50:15.8909020950 508 /var/lib/homelab-backup-matrix/rendered/memos-backup.txt
2026-07-21T08:50:15.8911074740 656 /var/lib/homelab-backup-matrix/compat/memos_backup--memos-backup.txt--ef6fd685e9e3.txt
2026-07-21T08:50:15.9190603340 576 /var/lib/homelab-backup-matrix/rendered/netbird-vps-backup.txt
2026-07-21T08:50:15.9488250690 537 /var/lib/homelab-backup-matrix/rendered/paperless-backup.txt
2026-07-21T08:50:15.9702101810 607 /var/lib/homelab-backup-matrix/rendered/paperless-restore-dry-run.txt
2026-07-21T08:50:15.9704189420 773 /var/lib/homelab-backup-matrix/compat/paperless_restore_dry_run--paperless-restore.txt--acfbc4e465fa.txt
2026-07-21T08:50:15.9918669420 627 /var/lib/homelab-backup-matrix/rendered/skladchik-reports-monitor-backup.txt
2026-07-21T08:50:16.0138324250 712 /var/lib/homelab-backup-matrix/rendered/skladchik-reports-monitor-backup-restore-check.txt
2026-07-21T08:50:16.0376266730 600 /var/lib/homelab-backup-matrix/rendered/vm160-remote-backup.txt
2026-07-21T08:50:16.0377791200 7616 /var/lib/homelab-backup-matrix/compat-map.json
2026-07-21T08:50:16.0380568130 36830 /var/lib/homelab-backup-matrix/state.json
2026-07-21T08:50:16.0381265160 357 /var/lib/homelab-backup-matrix/health.txt
----- DIR=/var/lib/homelab-cluster-admin -----
2026-07-09T02:15:16.4348302020 212 /var/lib/homelab-cluster-admin/restricted-probes-latest.md
2026-07-13T01:12:15.2820928160 357 /var/lib/homelab-cluster-admin/central-pull-latest.md
2026-07-21T02:37:04.7963344210 259 /var/lib/homelab-cluster-admin/full-observer-central-latest.md
2026-07-21T02:37:04.7963344210 272 /var/lib/homelab-cluster-admin/observer-remote-latest.txt
----- DIR=/var/lib/homelab-admin -----
2026-07-14T08:48:04.5070168760 2594 /var/lib/homelab-admin/current-state.json
2026-07-16T01:13:21.5726815610 93 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v1/.gitignore
2026-07-16T01:13:21.5727162640 1106 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v1/ACTIVE_CHANGES.md
2026-07-16T01:13:21.5727637410 3683 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v1/AGENTS.md
2026-07-16T01:13:21.5728454810 1206 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v1/README.md
2026-07-16T01:17:33.7922430400 25 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v1/.bootstrap-task-id
2026-07-16T01:17:34.8439897360 1147 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v1/CURRENT_CONTEXT.md
2026-07-16T01:24:23.7545066840 1206 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v2/README.md
2026-07-16T01:24:23.7545631560 3683 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v2/AGENTS.md
2026-07-16T01:24:23.7546125270 1106 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v2/ACTIVE_CHANGES.md
2026-07-16T01:24:23.7546326130 93 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v2/.gitignore
2026-07-16T01:26:14.8577149440 25 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v2/.bootstrap-task-id
2026-07-16T01:26:15.9203183270 1147 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v2/CURRENT_CONTEXT.md
2026-07-16T01:29:17.1899811830 216 /var/lib/homelab-admin/staging/homelab-ops-bootstrap-v2/apply-state.json
2026-07-16T03:11:54.7617119420 100 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-v3/bootstrap-key.pub
2026-07-16T03:11:54.7619281900 28 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-v3/.task-id
2026-07-16T03:11:54.7679096450 832 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-v3/plan.json
2026-07-16T12:12:29.3146422310 1352 /var/lib/homelab-admin/staging/xenforo-branding-rollout-v3/forum-map.tsv
2026-07-16T12:12:29.3157094100 92160 /var/lib/homelab-admin/staging/xenforo-branding-rollout-v3/payload.tar
2026-07-16T12:12:33.3952889200 28 /var/lib/homelab-admin/staging/xenforo-branding-rollout-v3/.task-id
2026-07-16T12:12:33.4021048840 371 /var/lib/homelab-admin/staging/xenforo-branding-rollout-v3/plan.json
2026-07-16T12:13:31.2199747110 323 /var/lib/homelab-admin/staging/xenforo-branding-rollout-v3/apply-state.json
2026-07-16T12:14:29.3376202110 242 /var/lib/homelab-admin/staging/xenforo-branding-rollout-v3/verify-state.json
2026-07-16T12:35:08.5536209670 340 /var/lib/homelab-admin/staging/xenforo-branding-header-visibility-v3/forum-map.tsv
2026-07-16T12:35:12.7600519140 38 /var/lib/homelab-admin/staging/xenforo-branding-header-visibility-v3/.task-id
2026-07-16T12:35:12.7617956360 259 /var/lib/homelab-admin/staging/xenforo-branding-header-visibility-v3/prepare.json
2026-07-16T12:35:15.3572982170 985 /var/lib/homelab-admin/staging/xenforo-branding-header-visibility-v3/forum-plan.tsv
2026-07-16T12:36:17.0107678250 292 /var/lib/homelab-admin/staging/xenforo-branding-header-visibility-v3/apply-state.json
2026-07-16T12:41:28.6907635740 282 /var/lib/homelab-admin/staging/xenforo-branding-header-visibility-v3/verify-state.json
2026-07-16T12:47:29.5876473720 340 /var/lib/homelab-admin/staging/xenforo-branding-header-order-v1/forum-map.tsv
2026-07-16T12:47:33.7730329860 33 /var/lib/homelab-admin/staging/xenforo-branding-header-order-v1/.task-id
2026-07-16T12:47:33.7750294120 254 /var/lib/homelab-admin/staging/xenforo-branding-header-order-v1/prepare.json
2026-07-16T12:47:36.3380684650 985 /var/lib/homelab-admin/staging/xenforo-branding-header-order-v1/forum-plan.tsv
2026-07-16T12:48:22.1017657740 287 /var/lib/homelab-admin/staging/xenforo-branding-header-order-v1/apply-state.json
2026-07-16T12:48:31.2981607510 287 /var/lib/homelab-admin/staging/xenforo-branding-header-order-v1/verify-state.json
2026-07-16T12:57:01.2226647630 100 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-v4/bootstrap-key.pub
2026-07-16T12:57:01.2229047360 28 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-v4/.task-id
2026-07-16T12:57:01.2280411540 832 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-v4/plan.json
2026-07-16T12:58:33.9325456800 251 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-v4/apply-state.json
2026-07-16T13:30:34.7942944220 27 /var/lib/homelab-admin/staging/pve03-kpartx-dependency-v1/.task-id
2026-07-16T13:30:35.6961419590 269 /var/lib/homelab-admin/staging/pve03-kpartx-dependency-v1/plan.json
2026-07-16T13:32:19.0726550040 46 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v3/.task-id
2026-07-16T13:32:19.0737338470 311 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v3/plan.json
2026-07-16T13:37:47.0276492630 46 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v4/.task-id
2026-07-16T13:37:47.0287839370 311 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v4/plan.json
2026-07-16T13:45:58.6002136100 46 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v5/.task-id
2026-07-16T13:45:58.6013488190 311 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v5/plan.json
2026-07-16T13:51:23.3271017050 46 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v6/.task-id
2026-07-16T13:51:23.3293401540 311 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v6/plan.json
2026-07-16T13:51:25.8933795560 692 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v6/guest-state-audit.json
2026-07-16T13:51:25.9020688350 95 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-guest-state-audit-v6/known_hosts.ip
2026-07-16T14:08:50.9476829420 42 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-jq-dependency-v1/.task-id
2026-07-16T14:08:50.9485294130 95 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-jq-dependency-v1/known_hosts.ip
2026-07-16T14:08:51.4604803550 287 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-jq-dependency-v1/plan.json
2026-07-16T14:12:22.9698312740 42 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-jq-dependency-v2/.task-id
2026-07-16T14:12:22.9707119200 95 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-jq-dependency-v2/known_hosts.ip
2026-07-16T14:12:23.4917512910 287 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-jq-dependency-v2/plan.json
2026-07-16T14:13:50.1103799100 43 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-base-bootstrap-v1/.task-id
2026-07-16T14:13:50.1111850760 95 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-base-bootstrap-v1/known_hosts.ip
2026-07-16T14:13:50.8632590890 979 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-base-bootstrap-v1/plan.json
2026-07-16T15:56:47.1548416000 46 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-core-registration-v2/.task-id
2026-07-16T16:06:12.1374995170 46 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-core-registration-v3/.task-id
2026-07-16T16:06:31.4132842520 918 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-core-registration-v3/plan.json
2026-07-16T16:09:04.9578565910 40 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-core-registration-v3/netbox-created.json
2026-07-16T22:49:51.9162150280 49 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-gitea-actions-runner-v2/.task-id
2026-07-16T22:49:51.9169336150 95 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-gitea-actions-runner-v2/known_hosts.ip
2026-07-16T22:49:52.9123866110 546 /var/lib/homelab-admin/staging/homelab-ops-worker-vm190-gitea-actions-runner-v2/plan.json
2026-07-18T00:45:40.5710369220 37 /var/lib/homelab-admin/repair-stage/homelab-dev-null-repair-v1/before.txt
2026-07-18T00:45:41.1140451260 163 /var/lib/homelab-admin/seals/homelab-dev-null-repair-v1.json
2026-07-18T02:04:55.1701966370 376 /var/lib/homelab-admin/seals/homelab-backup-tail-closure-v1.json
2026-07-18T03:00:03.7692669430 682 /var/lib/homelab-admin/seals/homelab-full-handoff-recovery-v1.json
2026-07-18T03:00:04.0863952050 378 /var/lib/homelab-admin/seals/homelab-full-handoff-seal-normalization-v1.json
2026-07-18T04:06:17.0615961730 373 /var/lib/homelab-admin/seals/homelab-logrotate-namespace-latch-clearance-v1.json
2026-07-18T13:53:02.8576614270 469 /var/lib/homelab-admin/seals/skladchik-reports-monitor-controlled-reauth-v1.json
2026-07-18T13:53:05.3889306280 235 /var/lib/homelab-admin/seals/skladchik-reports-monitor-az52-url-hotfix-v1.json
2026-07-19T18:02:00.5391470400 30 /var/lib/homelab-admin/staging/pve03-r8152-rx-ring-canary-v1/.task-id
2026-07-19T18:02:00.5400167640 89 /var/lib/homelab-admin/staging/pve03-r8152-rx-ring-canary-v1/known-hosts.sha256
2026-07-19T18:02:00.5410167800 41 /var/lib/homelab-admin/staging/pve03-r8152-rx-ring-canary-v1/desired-head
2026-07-19T18:02:02.2810433540 275 /var/lib/homelab-admin/staging/pve03-r8152-rx-ring-canary-v1/remote-baseline.json
2026-07-19T18:02:02.2810433540 65 /var/lib/homelab-admin/staging/pve03-r8152-rx-ring-canary-v1/remote-script.sha256
2026-07-19T18:02:39.2949351360 30 /var/lib/homelab-admin/staging/pve03-r8152-rx-ring-canary-v1/seal-run-id
2026-07-19T18:02:39.3036087780 1000 /var/lib/homelab-admin/seals/pve03-r8152-rx-ring-canary-v1.json
2026-07-19T18:02:39.3076088390 65 /var/lib/homelab-admin/staging/pve03-r8152-rx-ring-canary-v1/canonical-seal.sha256
----- DIR=/var/lib/homelab-chat-run -----
2026-07-21T08:54:39.0070746130 133 /var/lib/homelab-chat-run/raw/TRANSPORT-WRAPPER-001-20260721T055438Z.log
2026-07-21T08:54:39.0280243940 133 /var/lib/homelab-chat-run/public/TRANSPORT-WRAPPER-001-20260721T055438Z.txt
2026-07-21T09:01:42.5216104950 96 /var/lib/homelab-chat-run/raw/TRANSPORT-WRAPPER-004-20260721T060142Z.log
2026-07-21T09:01:42.5447263200 96 /var/lib/homelab-chat-run/public/TRANSPORT-WRAPPER-004-20260721T060142Z.txt
2026-07-21T09:01:42.5660598920 1325 /var/lib/homelab-chat-run/public/TRANSPORT-WRAPPER-004-20260721T060142Z.json
2026-07-21T09:01:42.5676112120 872 /var/lib/homelab-chat-run/public/TRANSPORT-WRAPPER-004-20260721T060142Z-latest.txt
2026-07-21T09:05:16.9049535880 82 /var/lib/homelab-chat-run/raw/TRANSPORT-WRAPPER-INNER-005-20260721T060516Z.log
2026-07-21T09:05:16.9253045330 82 /var/lib/homelab-chat-run/public/TRANSPORT-WRAPPER-INNER-005-20260721T060516Z.txt
2026-07-21T09:05:16.9482306600 1316 /var/lib/homelab-chat-run/public/TRANSPORT-WRAPPER-INNER-005-20260721T060516Z.json
2026-07-21T09:05:16.9509543050 864 /var/lib/homelab-chat-run/public/TRANSPORT-WRAPPER-INNER-005-20260721T060516Z-latest.txt
2026-07-21T09:11:08.4524214090 248 /var/lib/homelab-chat-run/state/command-ledger.tsv
2026-07-21T09:11:08.4544214400 0 /var/lib/homelab-chat-run/state/runner.lock
2026-07-21T09:11:11.5614689600 319582 /var/lib/homelab-chat-run/raw/CONTEXT-SOURCES-001-20260721T061108Z.log
===== DISCOVERY COMPLETE =====
CHANGES_MADE=NO
OUTPUT_END
CHAT_OUTPUT_END
View Git Blame Copy Permalink