homelab-admin/homelab-public-context
1
0
Fork 0
Code Activity
Files
2eb83b6bf8213972c1d616cf8af1532c447833cb
homelab-public-context/runtime/latest.txt
T
homelab-runtime-publisher 2eb83b6bf8 runtime: publish CRITICAL-TAILS-TRIAGE-091
2026-07-21 13:32:37 +00:00

1191 lines
273 KiB
Plaintext
Raw Blame History

CHAT_OUTPUT_BEGIN
COMMAND_ID=CRITICAL-TAILS-TRIAGE-091
STATUS=OK
RC=0
HOST=pve01
MODE=read-only
COMPONENT=cluster-audit
REFERENCE_REGISTER_CHECK=OK
REFERENCE_SHA256=5763f2f8edc75fcf6f3951f8c95896d1cc112ceccb6cf5b64a383f540a4deb8e
ERROR_REGISTER_CHECK=OK
ERROR_REGISTER_SHA256=08b3a769b8e38eaa1acec915352fb328180d70caa7ff6cd9d1ec0da426262752
COMMAND_SHA256=fd694682afa16ceef4527234dbc7d6474d0918ccafff27e91bbada99075ba2e4
DUPLICATE_FAILED_COMMAND_BLOCKED=false
EXECUTION_STARTED=true
CHANGES_MADE=false
SANITIZED=yes
SECRETS_INCLUDED=no
PRIVATE_ADDRESSES_INCLUDED=no
RAW_EVIDENCE_SHA256=1e1b51d29391e04f6add08dc0e763bd76b2446deac958537de0ddb0006998789
SANITIZED_OUTPUT_SHA256=a1c4491180e0ef3cc8411ac9618c9bc44e9e59f21520d85eac7f5aa5e62199f9
OUTPUT_BEGIN
=== SOURCE ===
FILE=/var/lib/homelab-chat-run/raw/CRITICAL-TAILS-BASELINE-089-20260721T132108Z.log SIZE=200182 MTIME=2026-07-21 16:21:16.540163848 +0300
=== P0_DATA_LOSS_OR_CLUSTER_RISK ===
63-392:- Context: SPF cleanup dry-run command 563 had broken shell quoting and left the terminal at multiline prompt.
64-397:- Context: SPF cleanup command 565 attempted to replace stale SPF records.
65:398:- Issue: backup write permission problem caused record detection failure, so new v=spf1 -all records were created while old v=spf1 ip4:87.236.18.45 records remained.
66-402:## MSMTP_SECRET_SOURCE_PARSE_ERROR_576_20260701
67-403:- Context: proof 576 installed msmtp but sendmail auth test failed.
68-404:- Issue: [SENSITIVE_PATH] was sourced as shell, but SMTP password contained shell-special characters; source failed and auth used an invalid/truncated secret path.
--
132-774:- The reported validation count is cascading missing-marker evidence, not twenty-five independent design defects.
133-775:- The generated validator must be inspected at the exact SyntaxError line before another candidate is created.
134:787:- The executed backup script does not write that health file; therefore the stale health timestamp was not evidence of backup failure.
135-792:- Failure class: переход к следующей задаче при наличии незакрытого хвоста.
136-794:- Closure gate: UNRESOLVED_TAIL_COUNT=0; BLOCKER_COUNT=0; TEMPORARY_ARTIFACT_COUNT=0; ROLLBACK_VERIFIED=YES; HEALTH=OK; PROOF_UPDATED=YES; REFERENCE_UPDATED=YES.
137-795:- Если закрытие невозможно из-за внешней зависимости, опасной неоднозначности или обязательного решения пользователя, фиксировать BLOCKED/OPEN и не заявлять CLOSED.
--
180-948:BACKUP-AUDIT-CONTROLPLANE-054 was successfully published at commit 2b966acfe553b4d9f8d4a62bf9533478bdd1a073, but was incorrectly reported missing because a stale directory listing was treated as authoritative.
181-951:BACKUP_UNSAFE_SOURCE_CAPTURE_071_PARSE_ERROR_20260721
182:952:BACKUP-UNSAFE-SOURCE-CAPTURE-071 failed before execution: shell syntax error near unexpected token (.
183-=== REFERENCE_TAIL ===
184-- Existing NPMplus Admin monitor was repaired to http://[PRIVATE_IP]:18091/npmplus, not deleted.
185-- Edge health wrapper /usr/local/sbin/router-moscow-health provides safe endpoints on [PRIVATE_IP]:18091 for router-moscow, npmplus, loki, alloy, cadvisor, registry-cache, socket-proxy, diun, kopia and crowdsec.
--
463-JOB=homelab-backup-freshness STATUS=OK EVIDENCE=execution_only AGE=35589 MAX_AGE=172800
464-JOB=homelab-backup-healthcheck STATUS=OK EVIDENCE=execution_only AGE=10631 MAX_AGE=172800
465:JOB=homelab-backup-sla-health STATUS=FAIL EVIDENCE=artifact_or_health AGE=144 MAX_AGE=172800
466-JOB=homelab-edge-vm-offhost-freshness STATUS=OK EVIDENCE=artifact_or_health AGE=18426 MAX_AGE=172800
467-JOB=homelab-emergency-backup STATUS=OK EVIDENCE=artifact_or_health AGE=24539 MAX_AGE=172800
468-JOB=homelab-extended-appbackup STATUS=OK EVIDENCE=artifact_or_health AGE=14156 MAX_AGE=172800
--
485-JOB=immich-media-offhost-sync STATUS=OK EVIDENCE=artifact_or_health AGE=4286 MAX_AGE=259200
486-JOB=memos-backup STATUS=OK EVIDENCE=artifact_or_health AGE=14547 MAX_AGE=259200
487:JOB=netbird-vps-backup STATUS=FAIL EVIDENCE=artifact_or_health AGE=1747896 MAX_AGE=864000
488-JOB=paperless-backup STATUS=DISABLED EVIDENCE=artifact_or_health AGE=16347 MAX_AGE=259200
489-JOB=paperless-restore-dry-run STATUS=STALE EVIDENCE=artifact_or_health AGE=1001986 MAX_AGE=864000
490-JOB=skladchik-reports-monitor-backup STATUS=OK EVIDENCE=artifact_or_health AGE=19639 MAX_AGE=172800
491-JOB=skladchik-reports-monitor-backup-restore-check STATUS=OK EVIDENCE=artifact_or_health AGE=190273 MAX_AGE=864000
492:JOB=vm160-remote-backup STATUS=FAIL EVIDENCE=artifact_or_health AGE=26607 MAX_AGE=172800
493-========== NODE=pve01 ==========
494---- PVE_STATUS ---
495-{
--
523-}
524---- FAILED_UNITS ---
525:● homelab-backup-matrix-refresh.service loaded failed failed Generate unified homelab backup matrix state
526-● homelab-cluster-admin-pull.service loaded failed failed Pull cluster-admin observer health into pve01
527:● homelab-vm160-remote-backup.service loaded failed failed VM160 full-image backup streamed from pve02 to pve03
528---- TIMERS ---
529-Tue 2026-07-21 16:23:54 MSK 2min 43s Tue 2026-07-21 16:18:29 MSK 2min 41s ago homelab-crypto-portfolio-chart-health.timer homelab-crypto-portfolio-chart-health.service
530-Tue 2026-07-21 16:24:14 MSK 3min 3s Tue 2026-07-21 16:19:14 MSK 1min 56s ago homelab-smartctl-textfile.timer homelab-smartctl-textfile.service
--
1079-2026-07-21T16:21:10.8330770850 0 /var/lib/homelab-health/private-vpn-hosts.txt.tmp
1080-/var/lib/homelab-health/final-closeout.txt:1:STATUS=OK TS=2026-07-08T09:02:10Z TYPE=final-closeout BAD=0 WARN=0 MODE=FINAL_OPERATOR_CLOSEOUT CLOSEOUT_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 EXACT_WARN_REVIEW_FILES=0 ENDPOINT_LOCAL_OK=1 ENDPOINT_LAN_OK=1 ENDPOINT_PUBLIC_OK=1 RECOVERY_SEAL=YES FREEZE_SEAL=YES FINAL_GLOBAL_GREEN=YES REVIEW_SURFACE_REGISTRY=YES POST_CLEANUP=YES REMOTE_CLEANUP=YES BACKUP_SLA=OK BACKUP_COVERAGE=OK EXTENDED_APPBACKUP=OK VM_BACKUP=OK SECRET_EXPOSURE=OK EXTERNAL_PROBE=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T084146Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T084146Z EMERGENCY_JOB=homelab-pve01-emergency-pack-20260708T002739Z VM_DUMPS_JOB=homelab-pve01-vm-local-dumps-20260706T153606Z DIRTY_COUNT=0 REPORT=/var/lib/homelab-final-closeout/final-closeout-latest.md
1081:/var/lib/homelab-health/backup-matrix-cluster-final-seal.txt:1:STATUS=OK TS=2026-07-13T17:51:31Z TYPE=backup-matrix-cluster-final-seal BAD=0 CURRENT_STAGE_COMPLETE=YES MATRIX_STATUS=OK JOBS_TOTAL=35 JOBS_OK=31 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 READ_WRITE_OVERLAP=0 VM_BACKUP_HEALTH=OK VM_FULL_BACKUP=OK EDGE_VM_CANONICAL_BACKUP=homelab-edge-vm-offhost-freshness.timer OLD_EDGE_TIMER_REENABLED=NO CLUSTER_PASSPORT=OK FINAL_READINESS=OK FINAL_READINESS_PASS=53 FINAL_READINESS_FAIL=0 OVERALL=OK CLUSTER_ADMIN=OK PROBE_OK=4 PROBE_BAD=0 WEBPANEL=OK FAILED_UNITS=0 CLUSTER_NODES=3 QUORATE=YES GENERATED_SYNC_HEAD=597119391e7b265b85fd9a76644e4e33af47e469 DESIRED_STATE_DIRTY=0 MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 MATRIX_STATE_SHA256=14a31112aac5d577ae01df567921b093b2fedcf7189c4cc8f009216ee0749380 CLUSTER_PASSPORT_SHA256=1b8b924ae0e5ce6872fe5671318094ce494806c2db7efe28299b5ff49adc0d0d FINAL_READINESS_SHA256=ac5159fa8c497afe021b5cb232f08cfec0e1a4d516d8aae966ba03f7034f1412 OVERALL_SHA256=6ae529bfdf5811f686d3b53978f453ce4f8757c74fc1be105de219b13a61ba46 PRESEAL_SHA256=7ea1f710cd311bd51dba05a313b4c540f57d97407f008881e502fc213b4b3c99 EVIDENCE_MANIFEST_SHA256=b4c68312026e1328b208a74090891f4809ab31ec69069e94a3d6902429bf3566 BACKUP_RUN=NO RESTORE_RUN=NO CLOUD_WRITE_RUN=NO RETENTION_RUN=NO NEXT_ACTION=AUDIT_AND_MIGRATE_SPECIAL_VM_EMERGENCY_AND_NON_APP_BACKUP_FAMILIES EVIDENCE_ROOT=/var/lib/homelab-change-sets/backup-matrix-cluster-final-seal/20260713T175131Z
1082-/var/lib/homelab-health/final-cycle-stability-seal.txt:1:STATUS=OK TS=2026-07-07T20:08:34Z TYPE=final-cycle-stability-seal BAD=0 WARN=0 MODE=FORCED_REPLAY_AFTER_CYCLE_BREAK STABLE=YES EXTERNAL_PROBE_STATUS=OK EXTERNAL_PROBE_WARN=0 EXTERNAL_PROBE_SMOKE_BAD=0 FINAL_STATUS=OK FINAL_BAD=0 FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 OVERALL_STATUS=OK OVERALL_BAD=0 OLD_EXTERNAL_PROBE_LIVE_MARKERS=0 OLD_EXTERNAL_PROBE_SOURCE_MARKERS=0 FINAL_CLUSTER_CHECKS=0 FINAL_LEGACY_DUTY_CHECKS=0 OVERALL_LEGACY_DUTY_GATE=0 CLUSTER_LEGACY_DUTY_GATE=0 SOURCE_CLUSTER_LEGACY_DUTY_GATE=0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-final-cycle-break/stability-seal-latest.md
1083:/var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-production-closure.txt:1:STATUS=OK TS=2026-07-13T08:12:14Z TYPE=crypto-portfolio-ui-center-alert-currency-production-closure BAD=0 PRODUCTION_CHANGED=YES_UI_ONLY INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb SHARP_MOVES_CENTERED=YES ALERT_CURRENCY_USD_RUB=OK BROWSER_RUNS=2 BROWSER_ERRORS=0 GRT_ROWS=0|0|0 GRT_SOURCE_REFERENCES=0 POSTCHANGE_BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T075801Z RESTORE_CHECK=OK_BOTH_CLOUDS RETENTION_ROLLBACK_COUNT=3 RETENTION_OPTIONAL_OTHER_CANDIDATES=1 RETENTION_REMOVED_ROLLBACK_COUNT=0 DANGLING_IMAGES_REMAINING=2 UNREFERENCED_DANGLING_IMAGES=0 DESIRED_STATE_HEAD=2999910549eaff9b9431b9c656d318340eb95cbd DESIRED_STATE_CLEAN=YES PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES READY_FOR_UI_FINAL_SEAL=YES
1084-/var/lib/homelab-health/desired-state-remote-git-audit.txt:1:STATUS=OK TS=2026-07-08T09:27:24Z TYPE=desired-state-remote-git-audit BAD=0 WARN=0 MODE=READ_ONLY_DESIRED_STATE_REMOTE_GIT_AUDIT AUDIT_OK=YES REMOTE_READY=NO ADD_REMOTE_DECISION=NO_CHANGE REPO_PRESENT=1 DIRTY_COUNT=0 REMOTE_COUNT=0 URL_CREDENTIAL_HINT=0 HTTPS_GITEA_OK=1 HTTPS_API_OK=0 BACKEND_OK=1 BACKEND_API_OK=0 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 OVERALL_STATUS=OK OVERALL_BAD=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 CLOSEOUT_OK=YES RECOVERY_SEAL_OK=YES INDEPENDENT_RUNNER_NO_DEPLOY=YES REPORT=/var/lib/homelab-desired-state-remote-git/audit-latest.md
1085:/var/lib/homelab-health/cluster-admin-stage3-postapply-backup.txt:1:STATUS=OK TS=2026-07-13T03:43:59Z TYPE=cluster-admin-stage3-postapply-backup BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=POSTAPPLY_BACKUP_AND_CONTROLLED_COLLECTOR_RUN PRODUCTION_DATABASE_CHANGED=COLLECTOR_NORMAL_OPERATION_ONLY PRODUCTION_SCHEMA_CHANGED=NO LIVE_APPLICATION_CODE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=YES_CONTROLLED_POSTAPPLY_BACKUP LIVE_APP_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 APPBACKUP=VERIFIED APPBACKUP_JOB=homelab-pve01-app-cluster-admin-incident-engine-20260713T034400Z APPBACKUP_SIZE_BYTES=1791603 APPBACKUP_RESTORE_CHECK=OK APPBACKUP_ROUTING=VERIFIED COLLECTOR_CONTROL_RUN=OK COLLECTOR_RUN_ID=3113 CANONICAL_CONSISTENCY=0|0|0|0|0 INCIDENT_ENGINE=ACTIVE COLLECTOR_TIMER=ACTIVE VM180_FAILED_UNITS=0 READY_FOR_STAGE3_FINAL_SEAL=YES STAGE3_CLOSED=NO POSTAPPLY_ROOT=/var/lib/homelab-stage3-postapply/20260713T034359Z
1086-/var/lib/homelab-health/cluster-admin-webpanel.txt:1:STATUS=OK TS=2026-07-11T01:39:28Z TYPE=cluster-admin-webpanel BAD=0 URL=http://[PRIVATE_IP]:8080/ STATUS_URL=http://[PRIVATE_IP]:8080/status.txt SELF_FAILED_UNITS=0 QGA=active BACKUP=MAIL_CLOUD_2COPY_VERIFY_OK
1087-/var/lib/homelab-health/post-cleanup-final-seal.txt:1:STATUS=OK TS=2026-07-08T06:10:30Z TYPE=post-cleanup-final-seal BAD=0 WARN=0 MODE=POST_CLEANUP_OWNER_TIMER_FINAL_SEAL FINAL_SEAL_OK=YES REPLAY_STATUS=OK REPLAY_OK=YES LOCAL_START_BAD=0 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 REVIEW_ITEMS_INTENTIONALLY_LEFT=yes LEGACY_PHASE_ITEMS_INTENTIONALLY_LEFT=yes FINAL_STATUS=OK FINAL_BAD=0 FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 LOCAL_CLEANUP_OK=YES REMOTE_CLEANUP_OK=YES RESIDUAL_SEAL_OK=YES SOPS_STATUS=OK EXTERNAL_PROBE_STATUS=OK EXTERNAL_PROBE_WARN=0 EXTERNAL_PROBE_SMOKE=0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-post-cleanup-replay/final-seal-latest.md
1088:/var/lib/homelab-health/pve01-core-monitoring-app-restorecheck.txt:1:STATUS=OK TS=2026-07-08T15:57:27Z TYPE=pve01-core-monitoring-app-restorecheck BAD=0 MODE=APP_LEVEL_RESTORE_CHECKS_TWO_CLOUDS FULL_VM_RESTORE_AVAILABLE=NO TOTAL_APPS=30 OK_APPS=30 FAIL_LIST=none REPORT=/var/lib/homelab-node-loss/pve01-core-monitoring-app-level/app-restorecheck-latest.md
1089-/var/lib/homelab-health/restore-drill-index.txt:5:MISSING_OR_NOT_OK=none
1090-/var/lib/homelab-health/mail-cloud-edge-vm-final-seal.txt:1:STATUS=OK TS=2026-07-14T01:02:58Z TYPE=mail-cloud-edge-vm-final-seal BAD=0 WARN=0 VMID=130 SOURCE=/mnt/staging/offhost/edge-vm-vzdump-from-pve03/vzdump-qemu-130-2026_07_13-03_30_01.vma.zst SIZE=10942350973 SHA256=6979558850813d99881b20c559600ec1c8748cc4f59f18ef9875eee6e40c6757 CLOUD_FOUND_DIR=20260713T183122Z MATRIX_JOB_ID=homelab_mail_cloud_edge_vm_upload MATRIX_STATUS=OK JOBS_TOTAL=36 JOBS_OK=32 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 UPLOAD_VERIFIED=YES DOWNLOAD_VERIFY=YES SHA_MATCH=YES SIZE_MATCH=YES PART_COUNT_MATCH=YES CONTROLLED_UPLOAD_MODE=--upload-no-retention RETENTION_MODE=disabled RETENTION_RUN=NO RETENTION_DELETE_COUNT=0 CLOUD_PURGE_RUN=NO SOURCE_DELETE_RUN=NO EDGE_SERVICE_RESULT=success EDGE_EXEC_MAIN_STATUS=0 EDGE_TIMER_ENABLED=enabled EDGE_TIMER_ACTIVE=active CLUSTER_ADMIN_STATUS=OK PROBE_OK=4 PROBE_BAD=0 CLUSTER_PASSPORT=OK DUTY_ADMIN=OK OVERALL=OK OVERALL_BAD=0 FINAL_READINESS=OK FINAL_PASS=53 FINAL_FAIL=0 WEBPANEL=OK WEBPANEL_SYNC=OK PUBLIC_OVERALL=OK FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DESIRED_STATE_PRE_SEAL_HEAD=98dfaeb35683e18cd4e986d0121bde60daac1e7c CURRENT_EDGE_VM_CLOUD_SUBSTAGE_COMPLETE=YES EVIDENCE_ROOT=/var/lib/homelab-change-sets/mail-cloud-edge-vm-final-seal-idle-safe/20260714T010226Z
1091-/var/lib/homelab-health/cluster-admin-stage2-dynamic-redryrun.txt:1:STATUS=OK TS=2026-07-13T01:20:06Z TYPE=cluster-admin-stage2-dynamic-redryrun BAD=0 PRODUCTION_CHANGED=TEMPORARY_DATABASE_AND_EVIDENCE_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO INCIDENT_ENGINE_RESTARTED=NO MIGRATION_ID=002_canonical_foundation_sync MIGRATION_SHA256=92a7f87ed6bf5215c81ec5f70ab1dc3c8339ecffa0b0f9fcab4eb6322651dc68 MIGRATION_PATCH_REQUIRED=NO COLLECTOR_PATCH_REQUIRED=NO APPLY_VERIFIER_PATCH_REQUIRED=YES FIXED_TOTAL_ENTITIES_REQUIRED=NO FIXED_TOTAL_ALIASES_REQUIRED=NO ACTIVE_CURRENT_SET_CONVERGENCE=OK NATURAL_KEY_CONVERGENCE=OK EPHEMERAL_ID_REFRESH=OK DYNAMIC_PLACEHOLDER_LIFECYCLE=OK RETIRED_PLACEHOLDERS_ALLOWED=YES ORPHANS=0 PREFLIGHT_TRANSACTION_ROLLBACK=OK DDL_ROLLBACK_TEST=OK ROLLBACK_AFTER_COMMITTED_SYNC=DO_NOT_PARTIAL_DDL_ROLLBACK PRESERVE_APPLIED_STATE_ON_POSTSYNC_VERIFIER_FAILURE=YES TEST_DATABASE_DROPPED=YES PRODUCTION_UNCHANGED=YES FRESH_PREAPPLY_BACKUP_REQUIRED=YES READY_FOR_FRESH_PREAPPLY_BACKUP=YES DESIRED_STATE_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 CONTRACT=/var/lib/homelab-change-sets/stage2-dynamic-redryrun/20260713T012006Z/dynamic-acceptance-contract.json RESULT=/var/lib/homelab-change-sets/stage2-dynamic-redryrun/20260713T012006Z/dynamic-redryrun-result.txt CHANGE_SET=/var/lib/homelab-change-sets/stage2-dynamic-redryrun/20260713T012006Z
--
1095-/var/lib/homelab-health/post-cleanup-owner-timer-replay.txt:1:STATUS=OK TS=2026-07-08T06:01:13Z TYPE=post-cleanup-owner-timer-replay BAD=0 WARN=0 MODE=FORCED_REPLAY_CLEANUP_OWNERS REPLAY_OK=YES LOCAL_START_OK=10 LOCAL_START_BAD=0 EDGE_RC=0 EDGE_HEALTH=OK LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 EXTERNAL_PROBE_STATUS=OK EXTERNAL_PROBE_WARN=0 EXTERNAL_PROBE_SMOKE=0 SOPS_STATUS=OK DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-post-cleanup-replay/owner-timer-replay-latest.md
1096-/var/lib/homelab-health/paperless-core-guard-audit.txt:1:STATUS=OK TS=2026-07-08T05:26:59Z TYPE=paperless-core-guard-audit BAD=0 WARN=1 MODE=READ_ONLY_PAPERLESS_CORE_GUARD_AUDIT CLASS=REMOTE_DEFERRED_GUARD_NOT_HARD_GATE CONNECT_ERRORS=0 FAILED_ONLY_GUARD=YES IN_HARD_GATES=no OVERALL_BAD=0 FINAL_FAIL=0 DUTY_V2_GATED_NONOK=0 PAPERLESS_CONTAINER_LINES=0 CORE_RELATED_LINES=1 RESET_FAILED_DONE=no REPORT=/var/lib/homelab-remote-failed-unit-cleanup/paperless-core-guard-audit-latest.md
1097:/var/lib/homelab-health/hapusya-site-final-seal.txt:1:STATUS=OK TS=2026-07-13T12:33:08Z TYPE=hapusya-site-final-seal BAD=0 APP=hapusya-site URL=https://hapusya.ru/ PRESEAL_DESIRED_STATE_HEAD=69966eb83fc065141df3c46df0d6c0f282275046 ROUTE_SHA256=02d65cb0082a04cae282126cc829a002f171e863291c18ba04d788dc2d6282ca UNIT_SHA256=b3b07137b25ac0a0b4bdadadad654cf209387ade86a2ef2cdca58e8a7c0cc212 RELATIVE_TREE_SHA256=9dadf21da041bc4d6ee4a42f5b3ac7719c16bcc39052de32af895c6cd28286cb APPLY_MANIFEST_SHA256=f44992e3e0cc443b372779cb35d9e899d904a9960a9de03c14559886fdb1fc75 PRODUCTION_BROWSER_MANIFEST_SHA256=9f3cdf3063f8724d1d1c1ad619f47d6f98d434c3422a74903585146858a8dfac BROWSER_SMOKE=OK BROWSER_RUNS=5 BROWSER_ERRORS=0 DESKTOP_VIDEO=OK MOBILE_VIDEO=OK STATIC_FALLBACK=OK BACKUP_JOB=homelab-pve01-app-hapusya-site-20260713T115259Z ARCHIVE_SHA256=bf284292974600384c43c47fc99be6fedf0ebd68f722b402a7260f1df680d167 TWO_CLOUDS=OK RESTORE_CHECK=OK_BOTH_CLOUDS RETENTION=OK DESIRED_STATE_SYNC=OK GENERATOR_SHA256=507bbbc2691f1f08cb1dd3660ed76cf13a8c36c23833d5fdec40cf6e07486cb8 MAP_SHA256=815b52dcedf7e3e92e07939bade32c7e8ed6acc0f9e064b65db6daec9f559c63 POLICY_SHA256=4bf996ff451ae3ec56eb2fcd42842606cd97295b4d5141aedb135c956395e4b5 EVIDENCE_ROOT=/var/lib/homelab-change-sets/hapusya-site-final-seal/20260713T123308Z EVIDENCE_MANIFEST_SHA256=7eb993a17ac18a1f7c9ae0b52d745d3778833d15626ed34c82496f74916eb26a PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES PRODUCTION_CHANGED=NO_FINAL_SEAL_ONLY DESIRED_STATE_CHANGED=YES_FINAL_SEAL_COMMIT_ONLY ALL_HAPUSYA_SITE_TAILS_CLOSED=YES TASK_COMPLETE=YES
1098-/var/lib/homelab-health/router-watchdog.txt:1:STATUS=OK TS=2026-07-21T13:20:07Z TYPE=router-watchdog ACTION=none REASON=router_wan_and_outbound_healthy SSH_OK=1 WAN_OK=1 OUTBOUND_OK=2 FAIL_COUNT=0 THRESHOLD=5 LAST_REBOOT_EPOCH=0 REBOOTS_TODAY=0 HARD_POWER_HELPER_READY=yes
1099:/var/lib/homelab-health/pve03-failed-unit-cleanup-closure.txt:1:STATUS=OK TS=2026-07-07T12:52:21Z TYPE=pve03-failed-unit-cleanup-closure BAD=0 WARN=0 MODE=READ_ONLY_CLOSURE LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260707T124031Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260707T124031Z GOLDEN_CP_MATCH=YES RESTORE_CHECK=YES PVE03_FAILED_UNITS=0 REPORT=/var/lib/homelab-pve03-failed-unit-cleanup/closure-latest.md
1100-/var/lib/homelab-health/review-surface-registry-audit.txt:1:STATUS=OK TS=2026-07-08T06:20:10Z TYPE=review-surface-registry-audit BAD=0 WARN=1 MODE=READ_ONLY_REVIEW_SURFACE_REGISTRY_AUDIT LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 OVERALL_STATUS=OK OVERALL_BAD=0 REVIEW_COUNT=22 LEGACY_COUNT=5 HARD_GATE_COUNT=0 CONVERT_PLANNING=19 CONVERT_CLOSURE=4 ARCHIVE_LEGACY=4 FILES_CHANGED=no REGISTRY=/var/lib/homelab-review-surface-registry/review-surfaces-latest.tsv REPORT=/var/lib/homelab-review-surface-registry/audit-latest.md
1101-/var/lib/homelab-health/nextcloud-restore-proof-offhost.txt:1:STATUS=OK TS=20260624T190330Z TYPE=nextcloud-restore-proof-offhost DEST_HOST=pve02 DEST_DIR=/mnt/staging/offhost/edge-restore-proofs-from-edge/nextcloud-aio/nextcloud-aio-restore-proof/20260624T120344Z ITEMS=1 OK=1 ERROR=0 BYTES=2135684 FILES=7 SECRET_PRINTED=NO
1102-/var/lib/homelab-health/recovery-card-seal.txt:1:STATUS=OK TS=2026-07-08T08:34:39Z TYPE=recovery-card-seal BAD=0 WARN=0 MODE=RECOVERY_CARD_FINAL_SEAL RECOVERY_SEAL_OK=YES RECOVERY_CARD_STATUS=OK RECOVERY_CARD_OK=YES FREEZE_STATUS=OK FREEZE_OK=YES FINAL_GLOBAL_GREEN=YES REVIEW_SURFACE_REGISTRY=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T082034Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T082034Z EMERGENCY_JOB=homelab-pve01-emergency-pack-20260708T002739Z VM_DUMPS_JOB=homelab-pve01-vm-local-dumps-20260706T153606Z ENDPOINT_LOCAL_OK=1 ENDPOINT_LAN_OK=1 ENDPOINT_PUBLIC_OK=1 BACKUP_SLA=OK BACKUP_COVERAGE=OK EXTENDED_APPBACKUP=OK VM_BACKUP=OK EXTERNAL_PROBE=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 DIRTY_COUNT=0 CARD=/var/lib/homelab-recovery-card/recovery-card-latest.md SUMMARY=/var/lib/homelab-recovery-card/recovery-summary-latest.tsv REPORT=/var/lib/homelab-recovery-card/recovery-card-seal-latest.md
--
1105-/var/lib/homelab-health/cluster-admin-stage0-readiness-fix.txt:1:STATUS=OK TS=2026-07-12T23:22:37Z TYPE=cluster-admin-stage0-readiness-fix BAD=0 RESULT=APPLIED FINAL_READINESS=OK FINAL_READINESS_DURATION_SECONDS=17 PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK FAILED_UNITS=0 LIVE_MATCH_DESIRED_STATE=YES READY_FOR_FINAL_SEAL=YES OLD_HEAD=a6e09a8af3604e47d7cc78be4176e6d567e8dbfb NEW_HEAD=1379b4fb4e809e56e9bc16f0e0b19b0a509192d5 CHANGE_SET=/var/lib/homelab-change-sets/stage0-readiness/20260712T232237Z PRODUCTION_CHANGED=READINESS_GATE_LOGIC_ONLY PRODUCTION_SCHEMA_CHANGED=NO CLOUD_WRITE_RUN=NO
1106-/var/lib/homelab-health/crypto-portfolio-remove-grt-recovery-audit.txt:1:STATUS=ERROR TS=2026-07-13T05:39:07Z TYPE=crypto-portfolio-remove-grt-recovery-audit BAD=1 COMMAND_PHYSICAL_LINES=1 ROOT_CAUSE=READINESS_ENDPOINT_MISMATCH RECOVERY_VERIFIED=NO TASK_COMPLETE=NO READY_FOR_GRT_APPLY_SCRIPT_FIX=NO AUDIT_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-recovery-audit-v2/20260713T053907Z
1107:/var/lib/homelab-health/cluster-admin-backup-webpanel-final-seal.txt:1:STATUS=OK TS=2026-07-09T05:24:54Z TYPE=cluster-admin-backup-webpanel-final-seal BAD=0 VMID=180 URL=http://[PRIVATE_IP]:8080/ BACKUP=MAIL_CLOUD_2COPY_VERIFY_OK WEBPANEL=OK WEBPANEL_SYNC=OK OVERALL_HARDGATE=YES FINAL_READINESS_HARDGATE=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-ideal-cluster/cluster-admin-backup-webpanel-final-seal-latest.md
1108-/var/lib/homelab-health/review-surface-registry-seal.txt:1:STATUS=OK TS=2026-07-08T06:36:39Z TYPE=review-surface-registry-seal BAD=0 WARN=0 MODE=REGISTERED_REVIEW_SURFACES_POLICY_AWARE_SEAL PRECHECK=YES REGISTRY_SEAL_OK=YES UPDATED=27 ARCHIVED=27 REGISTERED_COUNT=27 EXACT_WARN_REVIEW_FILES=0 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 FINAL_STATUS=OK FINAL_BAD=0 FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-review-surface-registry/seal-latest.md
1109-/var/lib/homelab-health/mail-cloud-critical.txt:1:STATUS=OK TS=20260721T034025Z TYPE=mail-cloud-critical MODE=recurring REMOTES=2 REMOTE_OK=2 REMOTE_FAIL=0 DEST=app/p0-critical-baseline/20260721T034025Z BUNDLE=p0-critical-baseline-20260721T034025Z.tar.gz SIZE=973275719 SHA256=e1681f82099b2430a46d47c3734063dce8322fd7cec37d4614b3d99ee293b45a RETENTION_KEEP=14 SECRET_PRINTED=no NOTES=none
1110-/var/lib/homelab-health/duty-admin-residual-cleanup.txt:1:STATUS=OK TS=2026-07-08T05:42:29Z TYPE=duty-admin-residual-cleanup BAD=0 WARN=0 MODE=SUPERSEDE_STALE_PVE03_CAPACITY_FORENSICS PRECHECK=YES CLEANUP_OK=YES NEW_PVE03_CAPACITY_STATUS=OK FAILED_LOCAL=0 FAILED_PVE02=0 FAILED_PVE03=0 FAILED_EDGE=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 OVERALL_STATUS=OK OVERALL_BAD=0 WARN_REVIEW_FILES=19 REVIEW_ITEMS_INTENTIONALLY_LEFT=yes REPORT=/var/lib/homelab-duty-admin-residual-cleanup/pve03-capacity-forensics-closure-latest.md
1111:/var/lib/homelab-health/appbackupctl-routing.txt:1:STATUS=OK TS=2026-07-13T00:29:36Z TYPE=appbackupctl-routing BAD=0 RESULT=APPLIED PRODUCTION_CHANGED=BACKUP_CLI_ROUTING_AND_DESIRED_STATE_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO DEFAULT_ROUTE=dockge-stacks CLUSTER_ADMIN_ROUTE=homelab-cluster-admin-incident-engine-backup.service UNSUPPORTED_TARGET_POLICY=REFUSE_RC_2 ROUTE_TEST_DOCKGE=OK ROUTE_TEST_CLUSTER_ADMIN=OK ROUTE_TEST_DEFAULT=OK ROUTE_TEST_UNSUPPORTED=OK FRAMEWORK_CHECK=OK UNSUPPORTED_ROUTE_BACKUP_CREATED=NO LIVE_MATCH_DESIRED_STATE=YES PVE01_FAILED_UNITS=0 OVERALL=OK DUTY_ADMIN=OK STAGE1_CLOSED=YES STAGE2_READY=YES OLD_HEAD=bd5eb856a8d61a5fb83c2bcaf74cb8ac98f075c5 NEW_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 CHANGE_SET=/var/lib/homelab-change-sets/appbackupctl-routing/20260713T002936Z
1112-/var/lib/homelab-health/final-readiness.txt:1:STATUS=WARN TS=2026-07-21T05:31:30Z TYPE=final-readiness BAD=5 PASS=48 FAIL=5 REPORT=/var/lib/homelab-final-readiness/latest.md
1113:/var/lib/homelab-health/cluster-admin-stage3-api-apply.txt:1:STATUS=OK TS=2026-07-13T03:22:35Z TYPE=cluster-admin-stage3-api-apply BAD=0 COMMAND_PHYSICAL_LINES=1 RESULT=APPLIED PRODUCTION_CHANGED=LIVE_APP_AND_DESIRED_STATE PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO SERVICE_RESTARTED=YES CLOUD_WRITE_RUN=NO_REUSE_VERIFIED_PREAPPLY_BACKUP APP_PRECHANGE_SHA256=c2748b8f96df842f491b5d2790790594222549853d0e2336cf60ecd6d4de0020 APP_CANDIDATE_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 LIVE_APP_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 LIVE_MATCH_DESIRED_STATE=YES PREAPPLY_BACKUP_REUSED=YES AST_IDENTICAL_NORMALIZATION=YES EXISTING_ROUTES_BASELINE=OK NEW_GET_ROUTES=7 UNAUTHENTICATED_NEW_ROUTES_401=YES OPENAPI_DOCS_EXPOSURE_CHANGED=NO CORE_DATABASE_STATE=1|1|9|7|1 INCIDENT_ENGINE=ACTIVE VM180_FAILED_UNITS=0 ROLLBACK_CHECKPOINT=VERIFIED OLD_HEAD=8f888e5f2063e49da918b31584c9ea4bcf53af3a NEW_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 READY_FOR_STAGE3_AUTHENTICATED_PRODUCTION_VERIFY=YES STAGE3_CLOSED=NO CHANGE_SET=/var/lib/homelab-change-sets/stage3-api-production-apply/20260713T032235Z
1114-/var/lib/homelab-health/cluster-admin-stage1-foundation-apply.txt:1:STATUS=OK TS=2026-07-13T00:21:21Z TYPE=cluster-admin-stage1-foundation-apply BAD=0 RESULT=APPLIED PRODUCTION_CHANGED=DATABASE_FOUNDATION_AND_DESIRED_STATE PRODUCTION_SCHEMA_CHANGED=PLANNED_ADDITIVE_ONLY SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO MIGRATION_ID=001_entity_source_policy_foundation NEW_TABLES=9 COLLECTOR_RUN_COLUMNS=7 TRIGGER=collector_runs_foundation_enrich_trg FUNCTION=cluster_admin_enrich_collector_run ENTITIES=83 PLACEHOLDERS=21 ENTITY_DEPENDENCIES=26 ENTITY_ALIASES=225 ALIAS_COLLISIONS=3 SOURCES=1 SOURCE_INSTANCES=1 SOURCE_EXPECTATIONS=1 POLICIES=1 POLICY_VERSIONS=1 ORPHANS=0 LIVE_COLLECTOR_COMPATIBILITY=OK API_HEALTH=OK VM180_FAILED_UNITS=0 ROLLBACK_POINT=VERIFIED OLD_HEAD=1379b4fb4e809e56e9bc16f0e0b19b0a509192d5 NEW_HEAD=bd5eb856a8d61a5fb83c2bcaf74cb8ac98f075c5 LIVE_MATCH_DESIRED_STATE=YES CHANGE_SET=/var/lib/homelab-change-sets/stage1-production-apply/20260713T002121Z
1115-/var/lib/homelab-health/homelab-admin-guardrails.txt:1:STATUS=OK TS=2026-07-14T07:39:53Z TYPE=homelab-admin-machine-guardrails BAD=0 RUNNER_VERSION=0.2.0-guardrails LINTER_HOTFIX=2.1 RULES=13 FAIL_FAST=YES ERR_TRAP_REQUIRED=YES UNGUARDED_GREP_COUNT_BLOCKED=YES RUNNER_SHA256=bc76e629e434cc6406ae2f3e7886034a0cedbca43795b6f4197a1ee517c1f322 LINTER_SHA256=1d0cdc2c31adfb7446d9f253539b4004621c6d29a5ea6d1c99bbd9aed34957a2 POLICY_SHA256=ce6d46a01dfe14adc8b2a35a683f2771e4324b04fea611dd4f4bd0458b09d1af DESIRED_STATE_HEAD=69be75caeecaa242567d2d06c735ac5db3a9c048 EVIDENCE_ROOT=/var/lib/homelab-change-sets/homelab-admin-guardrails-v21-install/20260714T073953Z
1116-/var/lib/homelab-health/cluster-admin-stage4i-scope-definition.txt:12:TASK_MODEL=VERSIONED_SCHEMA_V2_FAIL_FAST
--
1119-/var/lib/homelab-health/cluster-admin-stage4h-final-acceptance.txt:36:PVE01_FAILED_UNITS=0
1120-/var/lib/homelab-health/ideal-cluster-final-seal.txt:1:STATUS=OK TS=2026-07-08T19:49:09Z TYPE=ideal-cluster-final-seal BAD=0 MODE=FINAL_GREEN_SEAL TOP_GATES_OK=YES NODE_LOSS_OK=YES NOISE_SEAL_OK=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 TEST_VMS_CLEANED=YES PROD_VMS_RUNNING=YES DNS_PAIR_OK=YES RESTORE_CHECK_OK=YES DIRTY_COUNT=0 REPORT=/var/lib/homelab-ideal-cluster/final-seal-latest.md
1121:/var/lib/homelab-health/vm160-remote-backup.txt:1:STATUS=FAIL
1122-/var/lib/homelab-health/dockge-final-closure.txt:7:EDGE_STALE_REMAINING_MATCHES=0
1123-/var/lib/homelab-health/disk-space.txt:10:ERRORS=none
1124-/var/lib/homelab-health/remote-failed-unit-cleanup-seal.txt:1:STATUS=OK TS=2026-07-08T05:30:11Z TYPE=remote-failed-unit-cleanup-seal BAD=0 WARN=0 MODE=PAPERLESS_CORE_GUARD_SUPERSEDED_FINAL_SEAL PRECHECK=YES EDGE_APPLY=1 REMOTE_CLEANUP_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 EDGE_GUARD_STATE=inactive EDGE_MARKER_STATUS=OK DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-remote-failed-unit-cleanup/seal-latest.md
1125:/var/lib/homelab-health/special-vm-emergency-non-app-final-seal.txt:1:STATUS=OK TS=2026-07-14T01:15:48Z TYPE=special-vm-emergency-non-app-final-seal BAD=0 WARN=0 FAMILY_COVERAGE=11/11 MATRIX_STATUS=OK JOBS_TOTAL=40 JOBS_OK=36 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 COMPATIBILITY_OUTPUTS=31 REQUIRED_GREEN_JOBS=14 VM_FULL=OK VM_LOCAL_DUMPS=OK EDGE_VM_SOURCE=OK EDGE_VM_CLOUD=OK NEXTCLOUD_VM_SOURCE=homelab_vm_local_dumps_2cloud NEXTCLOUD_VM_CLOUD=OK EMERGENCY_PACK=OK SKLADCHIK_REPORTS_MONITOR=OK GOLDEN_STATE=OK ROUTER_RUNNING_CONFIG=OK MAIL_CLOUD_CRITICAL=OK MAIL_CLOUD_EDGE=OK MAIL_CLOUD_NEXTCLOUD=OK WRITE_MODE=legacy READ_MODE=dual MIGRATE_EXISTING=no SOURCE_BACKUP_RUN=NO CLOUD_OPERATIONS_RUN=NO RETENTION_RUN=NO CLOUD_PURGE_RUN=NO CLUSTER_ADMIN=OK PROBE_OK=4 PROBE_BAD=0 CLUSTER_PASSPORT=OK DUTY_ADMIN=OK OVERALL=OK OVERALL_BAD=0 FINAL_READINESS=OK FINAL_PASS=53 FINAL_FAIL=0 WEBPANEL=OK PUBLIC_OVERALL=OK FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DESIRED_STATE_PRE_SEAL_HEAD=b26f498b55de5d80bcbb4b9fb3a9bccded9f99b4 CURRENT_SPECIAL_VM_EMERGENCY_NON_APP_STAGE_COMPLETE=YES EVIDENCE_ROOT=/var/lib/homelab-change-sets/special-vm-emergency-non-app-final-seal/20260714T011514Z
1126-/var/lib/homelab-health/independent-external-runner.txt:1:STATUS=OK TS=2026-07-08T09:17:06Z TYPE=independent-external-runner BAD=0 WARN=0 MODE=NO_DEPLOY_SEAL NO_DEPLOY_SEAL_OK=YES AUDIT_STATUS=OK AUDIT_OK=YES INDEPENDENT_RUNNER_READY=NO DEPLOY_DECISION=NO_DEPLOY CURRENT_INDEPENDENT_RUNNER_AVAILABLE=no TRUSTED_NETBIRD_IDENTITY=edge-vm PUBLIC_VPS_DECISION=REJECTED_HOSTKEY_MISMATCH PUBLIC_VPS_SSH_OK=0 PUBLIC_VPS_REJECTED_HOSTKEY=YES EDGE_LAN_SSH_OK=2 EDGE_NETBIRD_SSH_OK=2 EDGE_DNS_SSH_OK=1 PUBLIC_OVERALL_OK=1 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 FINAL_STATUS=OK FINAL_FAIL=0 CLOSEOUT_OK=YES RECOVERY_SEAL_OK=YES RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T090905Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T090905Z DIRTY_COUNT=0 REPORT=/var/lib/homelab-independent-external-runner/no-deploy-seal-latest.md
1127-/var/lib/homelab-health/node-exporter-restore.txt:1:STATUS=OK TS=2026-07-09T08:13:40Z TYPE=node-exporter-restore JOB=homelab-pve01-app-node-exporter-20260709T081340Z RESTORE=/tmp/node-exporter-restore-20260709T081340Z RESTORE_CHECK=PACKAGE_OK PVE01_TEXTFILE_SCRAPE_ERROR_AFTER=0 NOTE=service-readiness-restore-proof
1128-/var/lib/homelab-health/cluster-admin-stage2-apply.txt:1:STATUS=OK TS=2026-07-13T01:38:37Z TYPE=cluster-admin-stage2-apply BAD=0 RESULT=APPLIED PRODUCTION_CHANGED=CANONICAL_SYNC_FUNCTION_COLLECTOR_MIGRATION_ARTIFACTS_AND_DESIRED_STATE PRODUCTION_SCHEMA_CHANGED=PLANNED_FUNCTION_AND_MIGRATION_REGISTRY_ONLY INCIDENT_ENGINE_RESTARTED=NO COLLECTOR_ONESHOT_RUN=YES COLLECTOR_TIMER_PAUSED_AND_RESUMED=YES CLOUD_WRITE_RUN=NO MIGRATION_ID=002_canonical_foundation_sync FUNCTION=cluster_admin_sync_foundation PREFLIGHT_NATURAL_KEY_DRIFT_ALLOWED=YES PREFLIGHT_NATURAL_KEY_MISSING_BEFORE=1 PREFLIGHT_NATURAL_KEY_STALE_BEFORE=1 PREFLIGHT_NATURAL_KEY_MISSING_AFTER=0 PREFLIGHT_NATURAL_KEY_STALE_AFTER=0 PREFLIGHT_EPHEMERAL_ID_DRIFT_ALLOWED=YES PREFLIGHT_EPHEMERAL_ID_DRIFT_BEFORE=25 PREFLIGHT_EPHEMERAL_ID_DRIFT_AFTER=0 PREFLIGHT_TRANSACTION_ROLLBACK=OK COMMITTED_SYNC=YES PARTIAL_ROLLBACK_AFTER_COMMIT=NOT_EXECUTED APPLIED_STATE_PRESERVED_POLICY=ACTIVE SAME_TRANSACTION=YES FIXED_TOTAL_ENTITIES_REQUIRED=NO FIXED_TOTAL_ALIASES_REQUIRED=NO FOUNDATION_COUNTS=85|21|2|229|26|0 DYNAMIC_INVARIANTS=0|0|21|21|0|0|0|0|0|0|0 ACTIVE_CURRENT_SET_CONVERGENCE=OK NATURAL_KEY_CONVERGENCE=OK EPHEMERAL_ID_REFRESH=OK RETIRED_PLACEHOLDERS_ALLOWED=YES ORPHANS=0 LIVE_COLLECTOR_RUN_ID=2992 LIVE_COLLECTOR_STATUS=OK SOURCE_INSTANCE_UPDATE=OK OBSERVER=OK FULL_OBSERVER=OK RESTRICTED_PROBES=OK FINAL_READINESS=OK CLUSTER_PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK API_HEALTH=OK VM180_FAILED_UNITS=0 PVE01_FAILED_UNITS=0 ROLLBACK_POINT=VERIFIED LIVE_MATCH_DESIRED_STATE=YES OLD_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 NEW_HEAD=8f888e5f2063e49da918b31584c9ea4bcf53af3a READY_FOR_POSTAPPLY_SEAL=YES STAGE2_CLOSED=NO CHANGE_SET=/var/lib/homelab-change-sets/stage2-production-apply/20260713T013837Z
1129:/var/lib/homelab-health/cluster-admin-node-online-green-dot-fix.txt:1:STATUS=OK TS=2026-07-14T01:47:22Z TYPE=cluster-admin-node-online-green-dot-fix BAD=0 WARN=0 ROOT_CAUSE=STATUS_KIND_MISSING_ONLINE_AND_OFFLINE PRE_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 POST_SHA256=a068c1824943af182acbaebcd3ff89e84203a3fde17ccb8051a4680e37d9f5e5 NODE_STATUS=online NODE_COUNT=3 ONLINE_CLASSIFICATION=ok ONLINE_BADGE_CLASS=state-ok OFFLINE_CLASSIFICATION=bad OFFLINE_BADGE_CLASS=state-bad DATABASE_CHANGED=NO COLLECTOR_CHANGED=NO SERVICE_RESULT=success EXEC_MAIN_STATUS=0 API_HEALTH=OK BACKUP_JOB=homelab-pve01-app-cluster-admin-incident-engine-20260714T014645Z BACKUP_STATUS=success BACKUP_TWO_CLOUDS=OK RESTORE_CHECK=OK_BOTH_CLOUDS REMOTE_ROLLBACK=/var/lib/cluster-admin-incident-engine/change-sets/node-online-green-dot-20260714T014640Z/app.py.before EVIDENCE_ROOT=/var/lib/homelab-change-sets/cluster-admin-node-online-green-dot-fix/20260714T014640Z
1130:/var/lib/homelab-health/crypto-portfolio-ui-preapply-backup.txt:1:STATUS=OK TS=2026-07-13T07:42:53Z TYPE=crypto-portfolio-ui-preapply-backup BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=INLINE_PYTHON_QUOTE_ERROR_IN_NONESSENTIAL_HELPER_PROOF BACKUP_SERVICE_RERUN=NO MANUAL_REHYDRATE_RERUN=NO CANONICAL_RESTORE_CHECK_RERUN=YES_READ_ONLY PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CONTAINER_RECREATED=NO CLOUD_WRITE_RUN=NO_READ_ONLY_RESTORE_CHECK TARGET_HOST=core-apps.gram1.ru INPUT_INDEX_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e CANDIDATE_INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T071127Z BACKUP_STATUS=success BACKUP_SCHEMA=homelab-app-backup-manifest-v2-flat BACKUP_LAYOUT=flat-v1 BACKUP_PATH_REL=hb/a/cp-260713-071127 BACKUP_DATABASE_INTEGRITY=ok BACKUP_FILE_COUNT=19 BACKUP_ARCHIVE_SIZE_BYTES=136632 BACKUP_ARCHIVE_SHA256=b29eb55820ec46c474f51e995e92e371251b6df717ac98182d807d1dbf8473f5 BACKUP_LOCAL_STAGING_DELETED=YES CANONICAL_TWO_CLOUD_RESTORE_CHECK=OK APPBACKUPCTL_RESTORE_CHECK=OK_BOTH_CLOUDS ROLLBACK_PATH=/var/lib/homelab-incidents/crypto-ui-center-alert-currency-prechange-20260713T071125Z ROLLBACK_MANIFEST_SHA256=0ef56c8f021f549777131ffe878cd5a263bdfb6f1dfb8bb080cefdd554573aca ROLLBACK_INDEX_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e PRODUCTION_UNCHANGED=YES PRODUCTION_HEALTH=200 VM170_FAILED_UNITS=0 PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES CHART_HEALTH_TIMER=ACTIVE BACKUP_TIMER=ACTIVE DESIRED_STATE_HEAD=80763a6c62b94cde4f3293f3b529980f76919d25 DESIRED_STATE_CLEAN=YES DESIRED_STATE_REMOTE_COUNT=0 READY_FOR_UI_PRODUCTION_APPLY=YES PREAPPLY_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-ui-preapply-backup-canonical-close/20260713T074253Z
1131-/var/lib/homelab-health/router-hard-power.txt:1:STATUS=ERROR TS=2026-07-16T06:20:34Z MODE=cycle REMOTE=[PRIVATE_IP] RC=74
1132-/var/lib/homelab-health/operator-handoff-pack.txt:1:STATUS=OK TS=2026-07-08T09:42:39Z TYPE=operator-handoff-pack BAD=0 WARN=0 MODE=FINAL_OPERATOR_HANDOFF_PACK HANDOFF_OK=YES PACK_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T093758Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T093758Z RECOVERY_CARD=YES FREEZE_PACK=YES FINAL_CLOSEOUT=YES INDEPENDENT_RUNNER_NO_DEPLOY=YES DESIRED_STATE_REMOTE_GIT=YES GIT_REMOTE_CHANGED=NO GIT_PUSH_DONE=NO DIRTY_COUNT=0 PACK=/var/lib/homelab-operator-handoff/operator-handoff-pack-20260708T094239Z.tar.gz README=/var/lib/homelab-operator-handoff/HANDOFF-latest.md STATUS_TSV=/var/lib/homelab-operator-handoff/handoff-status-latest.tsv NEXT=/var/lib/homelab-operator-handoff/NEXT-ACTIONS-latest.md
1133-/var/lib/homelab-health/pve01-failed-units.txt:1:STATUS=OK TS=20260624T180727Z TYPE=pve01-failed-units FAILED_COUNT=0 ACTION=RESET_FAILED_ONLY SECRET_PRINTED=REDACTED
1134-/var/lib/homelab-health/final-global-green-audit.txt:1:STATUS=OK TS=2026-07-08T07:20:01Z TYPE=final-global-green-audit BAD=0 WARN=0 MODE=SELF_REFERENCE_CLOSED_FINAL_GLOBAL_GREEN_SEAL AUDIT_OK=YES PRE_STATUS=WARN SELF_REFERENCE_CLOSED=yes LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 EXACT_WARN_REVIEW_FILES=0 LOCAL_OVERALL_OK=1 LAN_OVERALL_OK=1 PUBLIC_OVERALL_OK=1 EXTERNAL_PROBE_STATUS=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T064444Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T064444Z DIRTY_COUNT=0 REPORT=/var/lib/homelab-final-global-green-audit/final-seal-latest.md
1135-/var/lib/homelab-health/remote-failed-unit-audit.txt:1:STATUS=OK TS=2026-07-08T05:20:16Z TYPE=remote-failed-unit-audit BAD=0 WARN=1 MODE=READ_ONLY_REMOTE_FAILED_UNIT_AUDIT LOCAL_FAILED_COUNT=1 REMOTE_FAILED_COUNT=8 CONNECT_ERRORS=0 JOURNAL_FAIL_HINTS=194 OVERALL_BAD=0 FINAL_FAIL=0 DUTY_V2_GATED_NONOK=0 RESET_FAILED_DONE=no REPORT=/var/lib/homelab-remote-failed-unit-cleanup/audit-latest.md
1136:/var/lib/homelab-health/legacy-monitoring-backups-migrated.txt:1:STATUS=OK TS=2026-07-10T20:00:03Z TYPE=legacy-monitoring-backups-migrated BAD=0 APPS=6 RESTORE_FAILURES=0 ACTIVE_LEGACY_STATES=0 OFFHOST=OK RESTORE=OK
1137-/var/lib/homelab-health/cluster-admin-stage4h-collector-patch-candidate.txt:1:STATUS=OK TS=2026-07-14T17:41:32Z TYPE=cluster-admin-stage4h-collector-patch-candidate FAILED_ATTEMPT_TAIL=CLOSED CANDIDATE_SUBSTEP=CLOSED STAGE4H_STATUS=OPEN PRODUCTION_CHANGE=NO UNRESOLVED_TAIL_COUNT=0 BLOCKER_COUNT=0 NEXT_ACTION=TEMPORARY_DATABASE_MIGRATION_AND_COLLECTOR_ACCEPTANCE_HARNESS CANDIDATE_SHA256=4204fe568d46fec84f643af974c82770cb5d5b08d3d8ceb4aad96cca57abc909 REPORT_SHA256=ecfdaab5438e1fc3bc7a90e08ec68c71d06a6c6a91a430dc1ea7b6a74b7699a9
1138:/var/lib/homelab-health/cluster-admin-stage2-final-seal.txt:1:STATUS=OK TS=2026-07-13T01:45:29Z TYPE=cluster-admin-stage2-final-seal BAD=0 STAGE1_CLOSED=YES STAGE2_APPLY=OK PRODUCTION_CHANGED=BACKUP_HEALTH_AND_SEAL_ARTIFACTS_ONLY PRODUCTION_SCHEMA_CHANGED=NO_ADDITIONAL_CHANGE_DURING_SEAL INCIDENT_ENGINE_RESTARTED=NO MIGRATION_ID=002_canonical_foundation_sync FUNCTION=cluster_admin_sync_foundation COMMITTED_SYNC=YES MULTICYCLE_COLLECTOR_TEST=OK MULTICYCLE_NEW_RUNS=2 MULTICYCLE_GOOD_RUNS=2 ACTIVE_CURRENT_SET_CONVERGENCE=OK NATURAL_KEY_CONVERGENCE=OK EPHEMERAL_ID_REFRESH=OK RETIRED_PLACEHOLDERS_ALLOWED=YES FOUNDATION_COUNTS=85|21|2|229|26|0 DYNAMIC_INVARIANTS=0|0|21|21|0|0|0|0|0|0|0 ORPHANS=0 POSTAPPLY_DIRECT_SNAPSHOT=VERIFIED POSTAPPLY_RESTORE_TEST=OK POSTAPPLY_APPBACKUP=VERIFIED POSTAPPLY_APPBACKUP_JOB=homelab-pve01-app-cluster-admin-incident-engine-20260713T014757Z LIVE_MATCH_DESIRED_STATE=YES DESIRED_STATE_HEAD=8f888e5f2063e49da918b31584c9ea4bcf53af3a OBSERVER=OK FULL_OBSERVER=OK RESTRICTED_PROBES=OK FINAL_READINESS=OK CLUSTER_PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK VM180_FAILED_UNITS=0 PVE01_FAILED_UNITS=0 CURRENT_HEALTH_RECHECKED=YES STAGE2_CLOSED=YES REPORT=/var/lib/homelab-cluster-admin-stage2/stage2-final-seal-latest.txt REPORT_SHA256=32d4d37d82aa25194d789083d0e7ad2e9052ee38a72df5312b711226898cf1a6 BLOCKERS=/var/lib/homelab-cluster-admin-stage2/20260713T014529Z/blockers.txt
1139-/var/lib/homelab-health/remote-failed-unit-plan.txt:1:STATUS=OK TS=2026-07-08T05:22:47Z TYPE=remote-failed-unit-plan BAD=0 WARN=1 MODE=READ_ONLY_CLEANUP_PLAN LOCAL_FAILED_COUNT=1 REMOTE_FAILED_COUNT=7 LOCAL_NOOP_COUNT=4 REMOTE_RESET_ONLY_COUNT=1 REMOTE_NOOP_COUNT=5 REMOTE_DEFER_COUNT=1 OVERALL_BAD=0 FINAL_FAIL=0 RESET_FAILED_DONE=no PLAN=/var/lib/homelab-remote-failed-unit-cleanup/cleanup-plan-latest.tsv REPORT=/var/lib/homelab-remote-failed-unit-cleanup/plan-latest.md
1140-/var/lib/homelab-health/review-surface-registry-apply.txt:1:STATUS=OK TS=2026-07-08T06:36:39Z TYPE=review-surface-registry-apply BAD=0 WARN=0 MODE=POLICY_AWARE_DUTY_ADMIN_REPAIR_SELF_CLOSED PRECHECK=YES APPLY_OK=YES REPAIR_OK=YES UPDATED=27 ARCHIVED=27 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW_WAS=1 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_BAD=0 SELF_REFERENCE_CLOSED=YES REPORT=/var/lib/homelab-review-surface-registry/seal-latest.md
1141-/var/lib/homelab-health/private-vpn-hosts.txt.bak-ok-wrapper-20260624T132705Z:16:STATUS=ERROR
--
1146-/var/lib/homelab-health/crypto-portfolio-chart-health.txt:1:STATUS=OK TS=2026-07-21T13:18:29Z TYPE=crypto-portfolio-chart-health BAD=0 LIBRARY_OK=true DISK_CACHE_PRESENT=true SYMBOLS=2 STALE_COUNT=2
1147-/var/lib/homelab-health/duty-admin-v2.txt:1:STATUS=OK TS=2026-07-21T13:16:33Z TYPE=duty-admin-v2 BAD=7 WARN=0 MODE=LOCAL_LONG_RUNNING_ANALYZER_POLICY_AWARE SEPARATE_VM=NO TOTAL_HEALTH_INPUTS=497 GATED_OK=45 GATED_NONOK=7 UNGATED_ERROR_REVIEW=2 UNGATED_WARN_REVIEW=3 UNGATED_REVIEW_ITEM=0 LEGACY_OR_PHASE_REVIEW=0 REPORT=/var/lib/homelab-duty-admin-v2/latest.md
1148:/var/lib/homelab-health/cluster-admin-stage3-final-seal.txt:1:STATUS=OK TS=2026-07-13T03:46:30Z TYPE=cluster-admin-stage3-final-seal BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=HEALTH_REFRESH_AND_SEAL_ARTIFACTS_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO LIVE_APPLICATION_CODE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO STAGE2_CLOSED=YES STAGE3_APPLY=OK AUTHENTICATED_PRODUCTION_VERIFY=OK AUTHENTICATED_PRODUCTION_TESTS=25 APP_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 LIVE_MATCH_DESIRED_STATE=YES DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 NEW_GET_ROUTES=7 DATABASE_TRANSACTION_MODE=BEGIN_READ_ONLY OPENAPI_DOCS_EXPOSURE=PRESERVED_DISABLED POSTAPPLY_APPBACKUP=VERIFIED POSTAPPLY_APPBACKUP_JOB=homelab-pve01-app-cluster-admin-incident-engine-20260713T034400Z POSTAPPLY_APPBACKUP_RESTORE_CHECK=OK COLLECTOR_STABILITY=OK LATEST_COLLECTOR_RUN_ID=3114 CANONICAL_CONSISTENCY=0|0|0|0|0 OBSERVER=OK FULL_OBSERVER=OK RESTRICTED_PROBES=OK FINAL_READINESS=OK CLUSTER_PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK VM180_FAILED_UNITS=0 PVE01_FAILED_UNITS=0 CURRENT_HEALTH_RECHECKED=YES STAGE3_CLOSED=YES REPORT=/var/lib/homelab-cluster-admin-stage3/stage3-final-seal-latest.txt REPORT_SHA256=6f67545cd7f0a75b87146c1df74c61c3b88e142f9e53a96b083ec27209e6d265
1149-/var/lib/homelab-health/hapusya-site-production-browser.txt:1:STATUS=OK TS=2026-07-13T11:48:15Z TYPE=hapusya-site-production-browser BAD=0 URL=https://hapusya.ru/ BROWSER_RUNS=5 BROWSER_ERRORS=0 DESKTOP_VIDEO=OK MOBILE_VIDEO=OK STATIC_FALLBACK=OK EVIDENCE_ROOT=/var/lib/homelab-change-sets/hapusya-site-production-browser-v2/20260713T114810Z MANIFEST_SHA256=9f3cdf3063f8724d1d1c1ad619f47d6f98d434c3422a74903585146858a8dfac ROUTE_SHA256=02d65cb0082a04cae282126cc829a002f171e863291c18ba04d788dc2d6282ca UNIT_SHA256=b3b07137b25ac0a0b4bdadadad654cf209387ade86a2ef2cdca58e8a7c0cc212 RELATIVE_TREE_SHA256=9dadf21da041bc4d6ee4a42f5b3ac7719c16bcc39052de32af895c6cd28286cb PRODUCTION_CHANGED=YES CLOSURE_PENDING=POSTCHANGE_BACKUP_RETENTION_DESIRED_STATE_FINAL_SEAL
1150:/var/lib/homelab-health/backup-matrix-authoritative.txt:1:STATUS=FAIL
1151:/var/lib/homelab-health/backup-matrix-desired-state-persistence.txt:1:STATUS=OK TS=2026-07-13T17:40:37Z TYPE=backup-matrix-desired-state-persistence BAD=0 GENERATED_HEAD=af26ec3c4829b1330d49656205ac580ce1706263 GENERATED_PARENT=25568fab24c0594418b3676dc511c8234b4e1ae5 GENERATOR_SHA256=19da7f049b4f7f7286faf292c5419c37cc1e1768c2d8fe36b257ab0cf9c18874 MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 DESIRED_MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 MATRIX_STATUS=OK JOBS_BAD=0 FAILED_UNITS=0 BACKUP_RUN=NO RESTORE_RUN=NO CLOUD_WRITE_RUN=NO RETENTION_RUN=NO ROLLBACK=/var/lib/homelab-incidents/backup-matrix-desired-state-persistence-20260713T174036Z
1152-/var/lib/homelab-health/duty-admin-residual-audit.txt:1:STATUS=OK TS=2026-07-08T05:40:37Z TYPE=duty-admin-residual-audit BAD=0 WARN=3 MODE=READ_ONLY_RESIDUAL_NOISE_AUDIT LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=1 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 OVERALL_STATUS=OK OVERALL_BAD=0 FINAL_FAIL=0 WARN_REVIEW_FILES=19 HARD_GATE_WARN_MATCHES=0 REPORT=/var/lib/homelab-duty-admin-residual-cleanup/audit-latest.md
1153-/var/lib/homelab-health/ungated-health-backlog.txt:1:STATUS=OK TS=2026-07-07T07:45:01Z TYPE=ungated-health-backlog BAD=0 WARN=1 MODE=POST_PVE03_ROOT_CLEANUP_TRIAGE TOTAL=8 SUPERSEDED=4 STALE=2 POLICY_GAP_NEXT_STAGE=1 ACTIONABLE_CAPACITY=1 PVE03_ROOT_USAGE_PERCENT=31 PVE03_STAGING_USAGE_PERCENT=83 REPORT=/var/lib/homelab-ungated-health-backlog/latest.md
1154:/var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-final-seal.txt:1:STATUS=OK TS=2026-07-13T08:37:54Z TYPE=crypto-portfolio-ui-center-alert-currency-final-seal BAD=0 DANGLING_IMAGE_REFERENCE_GATE=OK DESIRED_STATE_SYNC_ARTIFACT_GATE=OK APP=crypto-portfolio PRODUCTION_CHANGED=NO_FINAL_SEAL_ONLY APP_SHA256=44df2b25183b7c5d87ff1516ec2d9cd1d169acde3031f7c2aa1fc4e0c7813cd2 WORKER_SHA256=174a14b86014535980d200a76d0057110c7af102c66f16c2e21fc3e74bb2e49e INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb LIBRARY_SHA256=c0992580867c4912cc9385b3c2728315bcc1a76c7f1087dca908430fccdf31d7 ACTIVE_COINS=HBAR,POL GRT_REMOVED=YES GRT_ROWS=0|0|0 GRT_SOURCE_REFERENCES=0 GRT_CACHE_ENTRIES=0 SHARP_MOVES_CENTERED=YES ALERT_CURRENCY_USD_RUB=OK BROWSER_SMOKE=OK BROWSER_RUNS=2 BROWSER_ERRORS=0 BROWSER_RESULT_SHA256=b566f492cd80c8f99bc6b24cbef6d5f326c04f4d7168359d05eb1f189589e733 SCREENSHOT_SHA256=00b2853bee9c880146d0dd75fac60feb7e46d4c129ee06754047ef30d90bd4ba BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T075801Z TWO_CLOUDS=OK RESTORE_CHECK=OK DATABASE_INTEGRITY=ok RETENTION=OK ROLLBACK_COUNT=3 DANGLING_IMAGES=2 REFERENCED_DANGLING_IMAGES=2 UNREFERENCED_DANGLING_IMAGES=0 PRUNED_UNREFERENCED_DANGLING_IMAGES=0 CHART_HEALTH=OK PUBLIC=OK MONITOR_TIMER=active BACKUP_TIMER=active CLOSURE_DESIRED_STATE_HEAD=2999910549eaff9b9431b9c656d318340eb95cbd DESIRED_STATE=OK PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES ALL_CRYPTO_PORTFOLIO_UI_TAILS_CLOSED=YES
1155-/var/lib/homelab-health/crypto-portfolio-remove-grt-runtime-verifier-prepared.txt:1:STATUS=OK TS=2026-07-13T04:35:29Z TYPE=crypto-portfolio-remove-grt-runtime-verifier-prepared BAD=0 COMMAND_PHYSICAL_LINES=1 VERIFIER_REVISION=V3 DEEP_ANALYSIS_RESULT=DIRECT_ROUTE_JSONRESPONSE_CONTRACT_CORRECTED FAILURE_CLASS=VERIFIER_ONLY PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO CANDIDATE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CONTAINER_STARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 INPUT_APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd INPUT_WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INPUT_INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 CANDIDATE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-candidate-v3/20260713T042046Z RUNTIME_POLICY=CONTAINER_PYTHON_ONLY PYTHONPATH_APP_INJECTED=YES ROUTE_RETURN_DECODER=DICT_OR_RESPONSE_BODY STATE_RETURN_DECODED=YES CHART_HEALTH_RETURN_DECODED=YES FETCH_KLINES_RETURN_DECODED=YES PROTECTED_KLINES_CODES=401_OR_403 LIVE_PRODUCTION_DATA_MOUNTED=NO ISOLATED_PORT=18030 FAILED_RUN_CLEANUP=VERIFIED REMOTE_RUNTIME_SCRIPT_SHA256=b60290b08f6bdb9801f4ccadbeadc4268399426996fd271527830fd399055448 VERIFIER_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-runtime-verifier-v3/20260713T043529Z READY_FOR_GRT_ISOLATED_RUNTIME_DRY_RUN_RUN=YES
1156-/var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-candidate.txt:1:STATUS=OK TS=2026-07-13T06:11:23Z TYPE=crypto-portfolio-ui-center-alert-currency-candidate BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=EXPECTED_DIFF_RC_1_TRIGGERED_SET_E_PIPEFAIL PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=80763a6c62b94cde4f3293f3b529980f76919d25 INPUT_INDEX_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e CANDIDATE_INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb BUILDER_DOUBLE_CHECK=OK CANDIDATE_DOUBLE_CHECK=OK SHARP_MOVES_POSITION=CENTERED SHARP_MOVES_MAX_WIDTH_PX=860 SHARP_MOVES_COLUMNS_DESKTOP=2 SHARP_MOVES_COLUMNS_MOBILE=1 ALERT_RENDERER=V5 ALERT_CURRENCY_SOURCE=TOP_USD_RUB_SWITCH ALERT_EXCHANGE_RATE_SOURCE=STATE_MARKET_USD_RUB ALERT_PRICE_THRESHOLD_CONVERTED=YES ALERT_CAP_THRESHOLD_CONVERTED=YES ALERT_SALE_VALUE_CONVERTED=YES CURRENCY_SWITCH_REDRAW=YES PUMP_DUMP_PERCENT_UNCHANGED=YES FIXED_USD_ALERT_FORMATTER=ABSENT RENDER_MOVES_COUNT=1 ALERT_RENDERER_MARKER_COUNT=1 GRT_REFERENCES=0 PRODUCTION_INDEX_UNCHANGED=YES PRODUCTION_HEALTH=200 READY_FOR_UI_ISOLATED_BROWSER_SMOKE=YES CANDIDATE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-ui-center-alert-currency-candidate-v4/20260713T061123Z
1157-/var/lib/homelab-health/cluster-admin-stage2-design.txt:1:STATUS=OK TS=2026-07-13T00:35:14Z TYPE=cluster-admin-stage2-design BAD=0 STAGE1_CLOSED=YES CAPTURE_REUSED=YES PRODUCTION_CHANGED=EVIDENCE_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO SERVICE_RESTARTED=NO FOUNDATION_STATE=9|7|1|1 EXPECTED_ENTITIES=62 MANAGED_ENTITIES=62 PLACEHOLDERS=21 MISSING_ENTITIES=0 STALE_ENTITIES=0 LEGACY_DEPENDENCIES=26 TYPED_DEPENDENCIES=26 DEPENDENCY_ID_MATCHES=0 DEPENDENCY_NATURAL_KEY_MATCHES=26 DEPENDENCY_NATURAL_KEY_MISSING=0 DEPENDENCY_NATURAL_KEY_STALE=0 EPHEMERAL_ID_MISSING=26 EPHEMERAL_ID_STALE=26 ROOT_CAUSE=LEGACY_DEPENDENCIES_DELETE_REINSERT_EACH_RUN CANONICAL_DEPENDENCY_IDENTITY=PARENT_CHILD_RELATION MIGRATION_ID=002_canonical_foundation_sync DATABASE_FUNCTION=cluster_admin_sync_foundation COLLECTOR_CALL_REQUIRED=YES SAME_TRANSACTION_REQUIRED=YES LEGACY_TABLES_PRESERVED=YES AUTOMATIC_REMEDIATION_ACTIONS=NO STAGE2_READY_FOR_DRY_RUN=YES DESIRED_STATE_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 CHANGE_SET=/var/lib/homelab-change-sets/stage2-design/20260713T003514Z CONTRACT=/var/lib/homelab-change-sets/stage2-design/20260713T003514Z/stage2-contract.json REPORT=/var/lib/homelab-change-sets/stage2-design/20260713T003514Z/stage2-design-report.txt
--
1161-/var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-browser-smoke.txt:1:STATUS=OK TS=2026-07-13T07:04:45Z TYPE=crypto-portfolio-ui-center-alert-currency-browser-smoke BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=UNPRIVILEGED_REMOTE_RM_AFTER_SUDO_HARNESS PRODUCTION_CHANGED=NO_BROWSER_EVIDENCE_FINALIZATION_ONLY PRODUCTION_DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru CANDIDATE_INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb CANDIDATE_STATIC_GRT_DOUBLE_CHECK=OK BROWSER_ENGINE=CHROMIUM BROWSER_RUNS=2 BROWSER_RUNS_IDENTICAL=YES MOVES=2 SHARP_MOVES_CENTERED=YES SHARP_MOVES_MAX_WIDTH_OK=YES SHARP_MOVES_COLUMNS=2 ALERT_ROWS=2 USD_ALERT_RENDER=OK RUB_ALERT_RENDER=OK TOP_CURRENCY_SWITCH_REDRAW=OK RUB_PRICE_THRESHOLD=OK RUB_CAP_THRESHOLD=OK RUB_SALE_VALUE=OK ALERT_USD_RESIDUE_IN_RUB_MODE=ABSENT CHART_CARDS=2 CHART_CANVASES=14 CHARTS_BOTTOM=YES BROWSER_ERRORS=0 GRT_VISIBLE=NO RESULT_SHA256=b566f492cd80c8f99bc6b24cbef6d5f326c04f4d7168359d05eb1f189589e733 SCREENSHOT_SHA256=68a9a68d15673ef64c248a4fe886ac36f94923ec917daa2772fc9e7c1b5f4c16 ROOT_OWNED_STAGE_ENTRIES=11 ROOT_OWNED_STAGE_DIRECTORIES=2 REMOTE_STAGE_CLEANED=YES PRODUCTION_INDEX_UNCHANGED=YES PRODUCTION_HEALTH=200 VM170_FAILED_UNITS=0 PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES READY_FOR_UI_PREAPPLY_BACKUP_AND_PRODUCTION_APPLY=YES BROWSER_SMOKE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-ui-center-alert-currency-browser-smoke-v2/20260713T070125Z
1162-/var/lib/homelab-health/crypto-portfolio-remove-grt-surface.txt:1:STATUS=OK TS=2026-07-13T04:05:42Z TYPE=crypto-portfolio-remove-grt-surface BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PRODUCTION_CHANGED=NO DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru TARGET_STACK=/opt/stacks/crypto-portfolio DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 DESIRED_STATE_CLEAN=YES APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 APP_GRT_REFERENCES=5 WORKER_GRT_REFERENCES=2 INDEX_GRT_REFERENCES=2 LOCAL_GRT_REFERENCES=3 SQLITE_SCHEMA_CAPTURED=YES READY_FOR_GRT_CANDIDATE_PREPARE=YES SURFACE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-surface/20260713T040542Z
1163:/var/lib/homelab-health/cluster-admin-stage4d-adapter-candidates-in-isolation.txt:1:STATUS=OK TS=2026-07-14T06:14:58Z TYPE=cluster-admin-stage4d-adapter-candidates-in-isolation-final-seal BAD=0 STAGE4D_CLOSED=YES NEXT_STAGE=4E_RESTRICTED_PROBE_AGENT_DESIGN PRODUCTION_CHANGED=NO DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO APPLICATION_CHANGED=NO ENVELOPE_TOTAL=48 PROMETHEUS_ENVELOPES=5 BACKUP_MATRIX_ENVELOPES=40 HEALTH_ENVELOPES=3 LOCAL_REMOTE_OUTPUT_MATCH=OK PRODUCTION_STATE=0|0|OK DESIRED_STATE_HEAD=e77fb37a0fa0033fe46558045e31dd92b6d1238a DESIRED_STATE_CLEAN=YES PVE01_FAILED_UNITS=0 REPORT=/var/lib/homelab-cluster-admin-stage4/20260714T061458Z/stage4d-adapter-candidates-in-isolation-seal.txt REPORT_SHA256=da6244c0a8a458b508fd58a0fec9cb1e6fa8da98a36f44d1368cbe947f4b5835
1164:/var/lib/homelab-health/backup-matrix-cluster-desired-state-sync.txt:1:STATUS=OK TS=2026-07-13T17:51:31Z TYPE=backup-matrix-cluster-desired-state-sync BAD=0 PRE_SYNC_HEAD=af26ec3c4829b1330d49656205ac580ce1706263 GENERATED_SYNC_HEAD=597119391e7b265b85fd9a76644e4e33af47e469 GENERATED_SYNC_MODE=NEW_GENERATED_COMMIT GENERATED_SYNC_COMMIT_DELTA=1 MATRIX_STATUS=OK JOBS_TOTAL=35 JOBS_OK=31 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 CLUSTER_PASSPORT=OK FINAL_READINESS=OK OVERALL=OK CLUSTER_ADMIN=OK PROBE_OK=4 PROBE_BAD=0 FAILED_UNITS=0 DESIRED_STATE_DIRTY=0 MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 MATRIX_STATE_SHA256=14a31112aac5d577ae01df567921b093b2fedcf7189c4cc8f009216ee0749380 PRESEAL_SHA256=7ea1f710cd311bd51dba05a313b4c540f57d97407f008881e502fc213b4b3c99 BACKUP_RUN=NO RESTORE_RUN=NO CLOUD_WRITE_RUN=NO RETENTION_RUN=NO EVIDENCE_ROOT=/var/lib/homelab-change-sets/backup-matrix-cluster-final-seal/20260713T175131Z
1165-/var/lib/homelab-health/cluster-admin-stage2-recovery.txt:1:STATUS=OK TS=2026-07-13T01:04:59Z TYPE=cluster-admin-stage2-recovery BAD=0 RESULT=RECOVERED PRODUCTION_CHANGED=HEALTH_REGENERATION_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO INCIDENT_ENGINE_RESTARTED=NO DATABASE_ROLLBACK=OK COLLECTOR_ROLLBACK=OK CORE_DATABASE_STATE=0|0|9|7|1 LIVE_COLLECTOR_SHA256=a83abde2290d8392b1c631122671bcb680f9271d0b576fafacb8876d3c8034f1 OBSERVER=OK FULL_OBSERVER=OK RESTRICTED_PROBES=OK RESTRICTED_PROBES_AUTHORITY=CENTRAL_SYNTHESIZED_FROM_REMOTE_FULL_OBSERVER VERIFIER_BUG=REMOTE_RESTRICTED_PROBES_FILE_INCORRECTLY_REQUIRED FINAL_READINESS=OK CLUSTER_PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK VM180_FAILED_UNITS=0 PVE01_FAILED_UNITS=0 DESIRED_STATE_UNCHANGED=YES ROLLBACK_POINT=VERIFIED READY_FOR_STAGE2_RETRY=YES STAGE2_CLOSED=NO CHANGE_SET=/var/lib/homelab-change-sets/stage2-recovery-final/20260713T010459Z
1166:/var/lib/homelab-health/cluster-admin-stage3-api-runtime-dry-run.txt:1:STATUS=OK TS=2026-07-13T02:53:34Z TYPE=cluster-admin-stage3-api-runtime-dry-run BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=TEMPORARY_ISOLATED_RUNTIME_AND_EVIDENCE_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO LIVE_APPLICATION_CODE_CHANGED=NO DESIRED_STATE_CHANGED=NO PRODUCTION_SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO APP_PRECHANGE_SHA256=c2748b8f96df842f491b5d2790790594222549853d0e2336cf60ecd6d4de0020 APP_CANDIDATE_SHA256=672d6a2975dd0e093788e7eb81a76e7f901ea6374ca4092a25ba0091aba1cb50 ISOLATED_BIND=127.0.0.1:18081 TEMPORARY_DATABASE_RESTORED=YES TEMPORARY_DATABASE_DROPPED=YES TEMPORARY_DATABASE_UNCHANGED=YES RUNTIME_TESTS=41 RUNTIME_FAILED_TESTS=0 EXISTING_ROUTES_BACKWARD_COMPATIBLE=YES NEW_GET_ROUTES=7 UNAUTHENTICATED_NEW_ROUTES_401=YES AUTHENTICATED_NEW_ROUTES=OK PAGINATION=OK FILTER_VALIDATION=OK ENTITY_DETAIL=OK ENTITY_DEPENDENCIES=OK SOURCES=OK POLICIES=OK COLLECTOR_RUNS=OK DATABASE_TRANSACTION_MODE=BEGIN_READ_ONLY FORBIDDEN_RESPONSE_FIELDS=ABSENT OPENAPI_DOCS_EXPOSURE_CHANGED=NO SECURITY_HEADERS=OK ISOLATED_RUNTIME_STOPPED=YES ISOLATED_PORT_CLOSED=YES LIVE_APP_UNCHANGED=YES PRODUCTION_RUNTIME_HEALTH=OK READY_FOR_STAGE3_PREAPPLY_BACKUP=YES DESIRED_STATE_HEAD=8f888e5f2063e49da918b31584c9ea4bcf53af3a CHANGE_SET=/var/lib/homelab-change-sets/stage3-api-runtime-dry-run/20260713T025334Z RESULT=/var/lib/homelab-change-sets/stage3-api-runtime-dry-run/20260713T025334Z/runtime-result.json
1167:/var/lib/homelab-health/crypto-portfolio-remove-grt-failed-units-diagnostic.txt:1:STATUS=OK TS=2026-07-13T05:41:00Z TYPE=crypto-portfolio-remove-grt-failed-units-diagnostic BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=NO_DIAGNOSTIC_ONLY PRODUCTION_DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO FAILED_STATE_RESET=NO CLOUD_WRITE_RUN=NO PVE01_FAILED_UNIT_COUNT=2 PVE01_FAILED_UNIT_NAMES=homelab-backup-matrix-refresh.service,homelab-cluster-admin-pull.service CRYPTO_PORTFOLIO_FAILED_UNIT_COUNT=0 OTHER_FAILED_UNIT_COUNT=2 CURRENT_CRYPTO_PRECHANGE_HEALTH=YES UNIT_DETAILS_CAPTURED=YES READY_FOR_TARGETED_FAILED_UNIT_DECISION=YES DIAGNOSTIC_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-failed-units-diagnostic/20260713T054100Z
1168:/var/lib/homelab-health/crypto-portfolio-remove-grt-production-apply.txt:1:STATUS=OK TS=2026-07-13T05:47:54Z TYPE=crypto-portfolio-remove-grt-production-apply BAD=0 COMMAND_PHYSICAL_LINES=1 APPLY_REVISION=V3 FIX_UID_READONLY=YES FIX_READINESS_ENDPOINT=YES FIX_ROLLBACK_DATABASE_METADATA=YES PRODUCTION_CHANGED=YES_REMOVE_GRT DESIRED_STATE_CHANGED=NO_BY_THIS_COMMAND PRODUCTION_CONTAINER_RECREATED=YES CLOUD_WRITE_RUN=NO ROLLBACK_ON_FAILURE=ARMED_NOT_NEEDED TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=80763a6c62b94cde4f3293f3b529980f76919d25 PRECHANGE_BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T051015Z ROLLBACK_PATH=/var/lib/homelab-incidents/crypto-remove-grt-prechange-20260713T051013Z APP_SHA256=44df2b25183b7c5d87ff1516ec2d9cd1d169acde3031f7c2aa1fc4e0c7813cd2 WORKER_SHA256=174a14b86014535980d200a76d0057110c7af102c66f16c2e21fc3e74bb2e49e INDEX_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e CHART_HEALTH_SCRIPT_SHA256=c16473f16390eeeff4d4eb701db604093564dd78d0be2d30b4cc8ffeeb22364a ACTIVE_COINS=HBAR,POL GRT_SOURCE_REFERENCES=0 GRT_DATABASE_ROWS=0|0|0 GRT_MARKET_ENTRY=ABSENT GRT_CACHE_ENTRIES=ABSENT DATABASE_MODE=0644 DATABASE_UID=10001 DATABASE_GID=10001 DATABASE_INTEGRITY=ok WORKER_STATUS=OK WORKER_COIN_COUNT=2 PUBLIC_CODES=200|303|401|200|200|401 CHART_HEALTH=OK CHART_SYMBOLS=HBAR,POL CHART_HEALTH_TIMER=ACTIVE VM170_FAILED_UNITS=0 PVE01_UNRELATED_FAILED_UNITS=homelab-backup-matrix-refresh.service,homelab-cluster-admin-pull.service PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES REMOTE_STAGE_CLEANED=YES READY_FOR_GRT_POSTAPPLY_VERIFY_BACKUP_SYNC=YES APPLY_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-production-apply-v3/20260713T054754Z
1169-/var/lib/homelab-health/mail-cloud-restore-drill.txt:1:STATUS=STALE
1170-/var/lib/homelab-health/mail-cloud-restore-drill.txt:20:FRESHNESS=STALE
1171-/var/lib/homelab-health/edge-restore-proofs-offhost.txt:1:STATUS=OK TS=20260624T190040Z TYPE=edge-restore-proofs-offhost DEST_HOST=pve02 DEST_DIR=/mnt/staging/offhost/edge-restore-proofs-from-edge ITEMS=14 OK=14 ERROR=0 BYTES=226024287 SECRET_PRINTED=NO
1172:/var/lib/homelab-health/crypto-portfolio-runtime-final-seal.txt:1:STATUS=OK TS=2026-07-13T08:37:54Z TYPE=crypto-portfolio-ui-center-alert-currency-final-seal BAD=0 DANGLING_IMAGE_REFERENCE_GATE=OK DESIRED_STATE_SYNC_ARTIFACT_GATE=OK APP=crypto-portfolio PRODUCTION_CHANGED=NO_FINAL_SEAL_ONLY APP_SHA256=44df2b25183b7c5d87ff1516ec2d9cd1d169acde3031f7c2aa1fc4e0c7813cd2 WORKER_SHA256=174a14b86014535980d200a76d0057110c7af102c66f16c2e21fc3e74bb2e49e INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb LIBRARY_SHA256=c0992580867c4912cc9385b3c2728315bcc1a76c7f1087dca908430fccdf31d7 ACTIVE_COINS=HBAR,POL GRT_REMOVED=YES GRT_ROWS=0|0|0 GRT_SOURCE_REFERENCES=0 GRT_CACHE_ENTRIES=0 SHARP_MOVES_CENTERED=YES ALERT_CURRENCY_USD_RUB=OK BROWSER_SMOKE=OK BROWSER_RUNS=2 BROWSER_ERRORS=0 BROWSER_RESULT_SHA256=b566f492cd80c8f99bc6b24cbef6d5f326c04f4d7168359d05eb1f189589e733 SCREENSHOT_SHA256=00b2853bee9c880146d0dd75fac60feb7e46d4c129ee06754047ef30d90bd4ba BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T075801Z TWO_CLOUDS=OK RESTORE_CHECK=OK DATABASE_INTEGRITY=ok RETENTION=OK ROLLBACK_COUNT=3 DANGLING_IMAGES=2 REFERENCED_DANGLING_IMAGES=2 UNREFERENCED_DANGLING_IMAGES=0 PRUNED_UNREFERENCED_DANGLING_IMAGES=0 CHART_HEALTH=OK PUBLIC=OK MONITOR_TIMER=active BACKUP_TIMER=active CLOSURE_DESIRED_STATE_HEAD=2999910549eaff9b9431b9c656d318340eb95cbd DESIRED_STATE=OK PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES ALL_CRYPTO_PORTFOLIO_UI_TAILS_CLOSED=YES
1173:/var/lib/homelab-health/backup-matrix-cluster-preseal-convergence.txt:1:STATUS=OK TS=2026-07-13T17:48:14Z TYPE=backup-matrix-cluster-preseal-convergence BAD=0 MATRIX_STATUS=OK JOBS_TOTAL=35 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 CLUSTER_PASSPORT=OK FINAL_READINESS=OK OVERALL=OK CLUSTER_ADMIN=OK PROBE_OK=4 PROBE_BAD=0 WEBPANEL=OK FAILED_UNITS=0 DESIRED_STATE_HEAD=af26ec3c4829b1330d49656205ac580ce1706263 DESIRED_STATE_DIRTY=0 MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 MATRIX_STATE_SHA256=3791fa78cdfb56cd7027f89350dff245abcffc4f935adfc336ceeea00854ac89 CLUSTER_PASSPORT_SHA256=1b8b924ae0e5ce6872fe5671318094ce494806c2db7efe28299b5ff49adc0d0d FINAL_READINESS_SHA256=ac5159fa8c497afe021b5cb232f08cfec0e1a4d516d8aae966ba03f7034f1412 OVERALL_SHA256=6ae529bfdf5811f686d3b53978f453ce4f8757c74fc1be105de219b13a61ba46 BACKUP_RUN=NO RESTORE_RUN=NO CLOUD_WRITE_RUN=NO RETENTION_RUN=NO EVIDENCE_ROOT=/var/lib/homelab-change-sets/backup-matrix-cluster-preseal-resume/20260713T174748Z
1174-/var/lib/homelab-health/uptime-kuma-pending-health-monitors-fix.txt:8:NOTE=Paperless Restore may become red if its health file contains STATUS=ERROR; that is a real restore-health issue, not a Kuma pending issue.
1175-/var/lib/homelab-health/pve03-capacity-forensics.txt:1:STATUS=OK TS=2026-07-08T05:42:29Z TYPE=pve03-capacity-forensics MODE=SUPERSEDED_BY_PVE03_ROOT_STAGING_CLEANUP BAD=0 WARN=0 OLD_STATUS=WARN_ARCHIVED ROOT_CLEANUP=OK STAGING_RETENTION=OK STAGING_CLEANUP=OK FAILED_UNIT_CLEANUP=OK REPORT=/var/lib/homelab-duty-admin-residual-cleanup/pve03-capacity-forensics-closure-latest.md
1176-/var/lib/homelab-health/remote-failed-unit-apply.txt:1:STATUS=OK TS=2026-07-08T05:25:22Z TYPE=remote-failed-unit-apply BAD=0 WARN=0 MODE=APPLY_CLEANUP_PLAN_EXCEPT_PAPERLESS_CORE_GUARD PRECHECK=YES APPLY_OK=YES LOCAL_OVERRIDE_COUNT=4 PVE02_RESET=1 EDGE_APPLY=1 LOCAL_FAILED_AFTER=0 PVE02_FAILED_AFTER=0 EDGE_FAILED_AFTER=1 EDGE_DEFER_ONLY=YES PVE02_CODEVIPE_FAILED=inactive DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 OVERALL_STATUS=OK OVERALL_BAD=0 DEFERRED_UNIT=homelab-paperless-core-guard.service REPORT=/var/lib/homelab-remote-failed-unit-cleanup/apply-latest.md
--
1179-/var/lib/homelab-health/cluster-admin-stage4c-database-migration-dry-run.txt:1:STATUS=OK TS=2026-07-14T05:33:26Z TYPE=cluster-admin-stage4c-database-migration-dry-run-final-seal BAD=0 STAGE4C_CLOSED=YES NEXT_STAGE=4D_ADAPTER_CANDIDATES_IN_ISOLATION MIGRATION_SHA256=a0bd6a50c6ad9b90528c2e90e6d9cb4279b09811b559b2504c83d7527b19e578 FINAL_READINESS=OK OVERALL=OK PUBLIC_ENDPOINT=OK WEBPANEL_SYNC=OK PVE01_FAILED_UNITS=0 VM180_FAILED_UNITS=0 DESIRED_STATE_HEAD=df86e4d27a64613b60ae55253b6348866dd34fa9 DESIRED_STATE_CLEAN=YES REPORT=/var/lib/homelab-cluster-admin-stage4/20260714T053326Z/stage4c-database-migration-dry-run-seal.txt REPORT_SHA256=3368c302ca0ee76fab8fbc97caa65adf4cf57e61c5b5cdb9a1f3829a05af2e2a
1180-/var/lib/homelab-health/operator-command-size-guard.txt:1:STATUS=OK TS=2026-07-14T18:26:54Z TYPE=operator-command-size-guard RULE_ACTIVE=YES PARSE_ERROR_TAIL=CLOSED UNRESOLVED_TAIL_COUNT=0 BLOCKER_COUNT=0 REPORT_SHA256=2ed7949dde9d5f8346e46af362e27cb8982d65f00ec602546efab506509e47af
1181:/var/lib/homelab-health/node-loss-final-closure.txt:1:STATUS=OK TS=2026-07-08T16:35:56Z TYPE=node-loss-final-closure BAD=0 MODE=ALL_NODE_LOSS_DRILLS_SEALED PVE03_EDGE_RESTORE=OK PVE03_EDGE_FRESHNESS=OK PVE02_FORUM_RUNTIME=OK PVE01_NEXTCLOUD_DNS=OK PVE01_CORE_MONITORING_APPLEVEL=OK FULL_VM_GAP_VM170_VM171=ACKNOWLEDGED_APP_LEVEL_ONLY TEST_VMS_CLEANED=YES DNS_PAIR_OK=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-node-loss/final-closure-latest.md
1182-/var/lib/homelab-health/crypto-portfolio-remove-grt-browser-smoke.txt:1:STATUS=OK TS=2026-07-13T04:52:06Z TYPE=crypto-portfolio-remove-grt-browser-smoke BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=OPTIONAL_XXD_DEPENDENCY_MISSING_WITH_STDERR_SUPPRESSED BROWSER_RERUN=NO EVIDENCE_SALVAGED=YES PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO CANDIDATE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CONTAINER_STARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 INPUT_APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd INPUT_WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INPUT_INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 ACTIVE_COINS=HBAR,POL GRT_VISIBLE=NO BROWSER_CARDS=2 BROWSER_CANVASES=14 BROWSER_MOVES=2 BROWSER_COIN_CARDS=2 BROWSER_POSITION_ROWS=2 BROWSER_ALERT_OPTIONS=2 BROWSER_ACTIVE_COIN=HBAR BROWSER_CHART_SYMBOLS=HBAR,POL BROWSER_ERRORS=0 BROWSER_LOADING_TEXT=ABSENT BROWSER_ERROR_TEXT=ABSENT CHARTS_POSITION=BOTTOM PNG_VALIDATION=PYTHON_STDLIB PNG_SIGNATURE=89504e470d0a1a0a SCREENSHOT_SHA256=3d132b477a382487e34db2f2feaf7e4f43af8d30fd8e1530225782da5d7336b9 RESULT_SHA256=0deadbf38aeef86726035da66eb1991c9494f59172b1c1a5c502c50c2de95c60 LIVE_PRODUCTION_DATA_MOUNTED=NO NPM_INSTALL_RUN=NO DOCKER_PULL_RUN=NO PRODUCTION_SOURCE_HASHES_UNCHANGED=YES PRODUCTION_CONTAINER_IDS_AND_RESTARTS_UNCHANGED=YES PRODUCTION_DATABASE_INTEGRITY=ok PRODUCTION_GRT_STATE_PRESERVED=YES PUBLIC_PRODUCTION_STATE_PRESERVED=YES REMOTE_RESIDUE=NONE READY_FOR_GRT_PRECHANGE_BACKUP=YES BROWSER_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-browser-smoke-salvage/20260713T045206Z
1183-/var/lib/homelab-health/independent-external-runner-audit.txt:1:STATUS=OK TS=2026-07-08T09:13:25Z TYPE=independent-external-runner-audit BAD=0 WARN=0 MODE=READ_ONLY_INDEPENDENT_EXTERNAL_RUNNER_AUDIT AUDIT_OK=YES INDEPENDENT_RUNNER_READY=NO DEPLOY_DECISION=NO_DEPLOY CURRENT_INDEPENDENT_RUNNER_AVAILABLE=no TRUSTED_NETBIRD_IDENTITY=edge-vm PUBLIC_VPS_DECISION=REJECTED_HOSTKEY_MISMATCH PUBLIC_VPS_SSH_OK=0 PUBLIC_VPS_REJECTED_HOSTKEY=YES EDGE_LAN_SSH_OK=2 EDGE_NETBIRD_SSH_OK=2 EDGE_DNS_SSH_OK=1 PUBLIC_OVERALL_OK=1 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 OVERALL_STATUS=OK OVERALL_BAD=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 CLOSEOUT_OK=YES RECOVERY_SEAL_OK=YES DIRTY_COUNT=0 REPORT=/var/lib/homelab-independent-external-runner/audit-latest.md
1184-/var/lib/homelab-health/sops-edge-secret-coverage-closure.txt:1:STATUS=OK TS=2026-07-08T05:01:47Z TYPE=sops-edge-secret-coverage-closure BAD=0 WARN=0 MODE=POLICY_SKELETON_SUPERSEDED_NOOP PRECHECK=YES POLICY_MODE=POLICY_SKELETON_NOT_ACTIVE AUDIT_CLASS=UNGATED_FAILED_STATE_REVIEW IN_HARD_GATES=0 OVERRIDE_WRITTEN=1 FAILED_UNITS_AFTER=0 UNIT_FAILED_AFTER=inactive MARKER_STATUS=OK DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 DIRTY_COUNT=0 SOPS_CLOSURE_OK=YES REPORT=/var/lib/homelab-failed-unit-legacy-cleanup/sops-edge-secret-coverage-closure-latest.md
--
1191-/var/lib/homelab-health/sops-edge-secret-coverage-audit.txt:1:STATUS=OK TS=2026-07-08T04:59:58Z TYPE=sops-edge-secret-coverage-audit BAD=0 WARN=1 MODE=READ_ONLY_SOPS_EDGE_SECRET_COVERAGE_AUDIT UNIT_FAILED=failed CLASS=UNGATED_FAILED_STATE_REVIEW IN_HARD_GATES=no OVERALL_BAD=0 FINAL_FAIL=0 DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=1 RELATED_WARN_OR_ERROR_HEALTH=3 RESET_FAILED_DONE=no REPORT=/var/lib/homelab-failed-unit-legacy-cleanup/sops-edge-secret-coverage-audit-latest.md
1192-/var/lib/homelab-health/duty-admin-residual-cleanup-seal.txt:1:STATUS=OK TS=2026-07-08T05:44:22Z TYPE=duty-admin-residual-cleanup-seal BAD=0 WARN=0 MODE=FINAL_RESIDUAL_NOISE_SEAL RESIDUAL_SEAL_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 REVIEW_ITEMS_INTENTIONALLY_LEFT=yes LEGACY_PHASE_ITEMS_INTENTIONALLY_LEFT=yes FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 PVE03_CAPACITY_FORENSICS_STATUS=OK LOCAL_CLEANUP_OK=YES REMOTE_CLEANUP_OK=YES DIRTY_COUNT=0 REPORT=/var/lib/homelab-duty-admin-residual-cleanup/seal-latest.md
1193:/var/lib/homelab-health/prometheus-npmplus-kuma-supersede-repair-20260709T073019Z/npmplus-kuma-config-backup.txt.bak:1:STATUS=ERROR TS=20260709T072619Z TYPE=npmplus-kuma-config-backup NOTES=edge_backup_failed
1194---- FILESYSTEMS ---
1195-Filesystem Type Size Used Avail Use% Mounted on
1196-udev devtmpfs 15G 0 15G 0% /dev
--
1309-2026-07-21T08:03:00.2596892130 770 /var/lib/homelab-health/vaultwarden-isolated-restore-drill.txt
1310-2026-07-21T16:21:07.1497113500 108 /var/lib/homelab-health/routerguard-ap.txt
1311:/var/lib/homelab-health/codevipe-cloud-backup-reset.txt:1:STATUS=OK TS=2026-07-08T05:25:22Z TYPE=remote-failed-unit-codevipe-reset BAD=0 WARN=0 MODE=RESET_STALE_FAILED_STATE UNIT=codevipe-cloud-backup.service REASON=last_journal_success_and_no_timer ACTION=reset_failed_only
1312---- FILESYSTEMS ---
1313-Filesystem Type Size Used Avail Use% Mounted on
1314-udev devtmpfs 6.8G 0 6.8G 0% /dev
--
1403-}
1404---- FAILED_UNITS ---
1405:● homelab-immutable-offline-verify.service loaded failed failed Verify immutable offline backup image
1406---- TIMERS ---
1407-Tue 2026-07-21 16:21:41 MSK 25s Tue 2026-07-21 16:16:41 MSK 4min 34s ago homelab-smartctl-textfile.timer homelab-smartctl-textfile.service
1408-Tue 2026-07-21 16:28:31 MSK 7min Tue 2026-07-21 16:13:31 MSK 7min ago prometheus-node-exporter-smartmon.timer prometheus-node-exporter-smartmon.service
--
1418-/dev/mapper/pve-root ext4 98G 53G 41G 57% /
1419-tmpfs tmpfs 16G 66M 16G 1% /dev/shm
1420:efivarfs efivarfs 128K 125K 0 100% /sys/firmware/efi/efivars
1421-tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
1422-tmpfs tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-journald.service
1423-tmpfs tmpfs 16G 32K 16G 1% /tmp
--
1476-2026-07-21T09:49:51Z RUNTIME-RECEIVER-STATE-027 0831002c78c26f35826cd7199995e335ceadcd39f08b9bac23882a0fbb59c8cc 127 FAIL gitea-runtime
1477-2026-07-21T10:26:18Z RUNTIME-RECEIVER-CLEANUP-036 65fbcfbce38a55723a92a8790429d6567389321f030f02cb4bf301cbe67e2531 1 FAIL gitea-runtime
1478:2026-07-21T11:35:23Z BACKUP-DUAL-CLOUD-PREFLIGHT-059 c6f4134d15a9f641ddaf67535760d0abede973dd08222a37af05612a6f580479 1 FAIL backup-audit
1479:2026-07-21T12:28:26Z BACKUP-VM-DUMP-PIPELINE-073 cd8291d0f14a37241e6062408603455d1ebd483091bebfa46c2b7a768f21308b 1 FAIL backup-audit
1480:2026-07-21T12:42:53Z BACKUP-VM-DUMP-CHANGESET-PREFLIGHT-075 a9a8a6046441de0c40a8b13bdfe96698c53a5156b4346b7ce28dc1f2362e8936 143 FAIL backup-audit
1481:2026-07-21T12:56:52Z BACKUP-DRAIN-LOCAL-ARCHIVES-079 531e1da7ce1681eed084b22ced7da87eb1ee8995f662dc7559f4f9dde3c0ac32 75 FAIL backup-remediation
1482:2026-07-21T13:04:48Z BACKUP-FINAL-PROOF-082 9e675a3cb621d0d1fcd8918b02278ab08c1a974f544ebef9a5639bf6578ee601 127 FAIL backup-audit
1483-2026-07-21T13:07:09Z CLUSTER-CLEANUP-INVENTORY-083 09d3f14ca0205936a910dc7af7991474030273a47d76fa033ea6a3185c083922 1 FAIL cluster-cleanup
1484:2026-07-21T13:14:27Z CLUSTER-BACKUP-POSTCLEAN-PROOF-086 601958c48614dbc25cc4592b111de4dec3306e53b756b275068b81a41d42b2da 127 FAIL backup-cleanup
1485:2026-07-21T13:15:21Z BACKUP-POLICY-CLOSEOUT-087 d79c1c881ebca24db09c5b4ed4b71b09a5ad338c30789c99b0644c8935072c52 1 FAIL backup-cleanup
1486:2026-07-21T13:17:20Z BACKUP-CLEANUP-FINAL-CLOSEOUT-088 5ba2893489b6f5831fc4599d3efa874d89c3fd41da7151f9c32945d11d1a23d6 1 FAIL backup-cleanup
1487-CRITICAL_TAILS_BASELINE=COMPLETE
=== P1_FAILED_SERVICES_AND_HEALTH ===
1-=== CONTROL_FILES ===
2-5763f2f8edc75fcf6f3951f8c95896d1cc112ceccb6cf5b64a383f540a4deb8e /etc/pve/31_HOMELAB_REFERENCE.md
3:08b3a769b8e38eaa1acec915352fb328180d70caa7ff6cd9d1ec0da426262752 /etc/pve/HOMELAB_ASSISTANT_ERROR_REGISTER.md
4:=== ERROR_REGISTER_OPEN_MARKERS ===
5:1:# HOMELAB ASSISTANT ERROR REGISTER
6:35:11. Ошибка: считать offhost OK после failed rsync.
7:77:21. Ошибка: nested PHP php -r дал Parse error на forum-prod.
8:83:Команда 108 дала PHP Parse error на smtpHost и пустой proof-файл.
9:88:Команда 111 не изменила smtpSsl: OLD_SMTPSSL=false, SQL ERROR 1064, NEW_SMTPSSL=false.
10:93:Нельзя использовать { echo RULE_CHECK_FAIL; exit 1; } в командах, выполняемых прямо в интерактивном root@pve01 shell.
11:94:При failed-check использовать безопасный шаблон: if grep ...; then echo RULE_CHECK_OK; ...; else echo RULE_CHECK_FAIL; fi.
12-100:Сначала проверять AUTH через openssl s_client -starttls smtp, без вывода секрета.
13-102:26. Ошибка: openssl -crlf вместе с ручным CRLF ломает SMTP AUTH checker.
14:109:Команда 138 пересобрала dashboard, но диагностический Python-блок упал с NameError из-за кавычек.
15:126:Перед любым действием команда обязана проверять /etc/pve/HOMELAB_ASSISTANT_ERROR_REGISTER.md и текущий 31_HOMELAB_REFERENCE.md.
16:127:Команда обязана печатать ERROR_REGISTER_CHECK=OK и REFERENCE_CHECK=OK до основного действия.
17:130:32. Ошибка: 153_PROMETHEUS_TARGETS_SETTLED_PROOF содержит SyntaxError из-за вложенного Python внутри SSH.
18:137:Такие выводы не являются инфраструктурным состоянием; считать невалидными только соответствующие failed command attempts.
19:150:Факт: terminal output around storage block creation showed possible paste/truncation artifact near smartctCK=FAIL.
20:154:37. Coverage gap: pve03_staging missing from disk-space health coverage.
21:159:Факт: terminal output around command 222 showed paste artifact near "FAIL; fi".
22:163:Факт: строка справочника с фразой No unresolved final REVIEW_REQUIRED... SyntaxError... была принята за BAD_MARKER.
23:166:40. Ошибка/вывод: VM150 Mail cloud single-file upload failed with 413.
24:171:Факт: first reconciliation wrote STATUS=ERROR because router-running-config-mail-cloud.txt is on pve01, not edge-vm.
25:180:## HOME_PORTAL_BASE64_APPLY_FAILURE_20260630
26:181:- Prior home-portal apply attempt 312 failed because a large base64 Python payload decoded as invalid UTF-8.
27:188:- Any Traceback, UnicodeDecodeError, SyntaxError, Permission denied, command-not-found, nonzero critical RC, or failed write marker must force REVIEW or FAILED, never OK.
28:191:- If an apply step fails, the next command must explicitly record the failed attempt and then run a corrected transparent apply command.
29:193:## HOME_PORTAL_API_ERROR_CLOSURE_RULE_20260630
30:194:- Rule: home portal closure requires current Homepage API-error logs to be zero after restart/reload, not only HTTP 200.
31:195:- If Homepage UI shows API error or logs contain current httpProxy/API timeout errors, the portal is REVIEW until the widget/API cause is disabled or fixed.
32-196:- External informational widgets such as weather/Open-Meteo must not block the service launcher portal.
33:198:## HOME_PORTAL_COMPLEX_QUOTING_FAILURE_20260630
34:199:- Previous home portal cleanup command failed in local bash before execution: syntax error near unexpected token `('.
35-211:- A portal card is openable only when DNS, reverse-proxy host mapping, and target content are all valid.
36:213:## ROUTER_CLI_STDIN_APPLY_FAILURE_20260630
37:216:- Failure: STDIN_SHOW_OK=no, APPLY_SKIPPED=stdin_show_failed, CONFIG_HAS_EDGE_5080=no, CONFIG_HAS_EDGE_5083=no.
38:219:## ROUTER_CLI_PROBE_STDIN_CONSUMPTION_ERROR_20260630
39:227:- Observed: CMD_TEXT=access-list ? returned Core::Configurator error execute denied [cli].
40:232:- Evidence: 395_ROUTER_CLI_ERROR_REGISTER_UPDATE_PROOF.txt has blank P393_COMMAND_COUNT/P394_COMMAND_COUNT/P394_ZERO_RC_COUNT fields.
41:243:- Failure: NODE_HTTPS_RESOLVE_ERROR=ERR_INVALID_IP_ADDRESS and NODE_HTTPS_RESOLVE_INSECURE_ERROR=ERR_INVALID_IP_ADDRESS.
42:254:## PY_COMPILE_PYC_PERMISSION_ERROR_20260701
43:257:- Actual impact: non-root syntax check attempted to create /usr/local/sbin/__pycache__ and failed with Permission denied.
44:273:## HOMEPAGE_ROUTER_MOSCOW_APPLY_SCRIPT_SYNTAX_ERROR_20260701
45:276:- Evidence: proof 408 showed SyntaxError in /tmp/homepage-router-moscow-apply-fixed.py.
46:284:- Rule: do not proceed with OS baseline until SSH failure is diagnosed; likely old known_hosts key or cloud-init/root-key issue.
47:307:- Additional failed approach: proof 480 ran xf-dev:rebuild-caches on CodeVipe and returned RC=1, not a valid fix.
48:313:- Issue: curl returned HTTP 403; unzip then failed because the ZIP was not downloaded.
49:318:## PVE01_UNZIP_MISSING_FOR_XF_REQUIREMENTS_VERIFY_20260701
50:327:- Note: proof 511 initially failed dsmods path detection; proof 512 fixed dsmods by locating public_html via src/XF.php marker.
51:330:## NPMPLUS_SQLITE_PASTE_FAILURE_20260701
52:332:- Issue: shell entered multiline prompt and produced syntax errors.
53:343:- Context: NPMplus API login attempts in proofs 526/527 failed.
54:349:- Context: NPMplus API login attempts failed even though the forum VM and edge-to-forum connectivity are healthy.
55:357:- Impact: certbot ran without token and all certificate attempts failed; DNS cutover must not proceed from proof 535.
56:360:## EDGE_CERTBOT_FAILED_DNS_CUTOVER_ABORTED_20260701
57:361:- Context: edge manual routes are working locally, but certbot DNS-01 certificate issue failed in proof 538.
58:368:- Evidence: proof 542 showed all five zones missing and DNS record create probes failed.
59:369:- Impact: certbot DNS-01 failed and DNS cutover was correctly aborted; public DNS still points to old hosting.
60:375:- Important corrections: initial NPMplus API route failed; manual Nginx routes were used. Initial Cloudflare token had no zone access; corrected token passed zone/DNS probe.
61:379:- Issue: Proxmox snapshot failed because generated snapshot name exceeded 40 characters.
62:388:- Impact: web routing is OK, but mail-related DNS may still contain stale provider data.
63:392:- Context: SPF cleanup dry-run command 563 had broken shell quoting and left the terminal at multiline prompt.
64:397:- Context: SPF cleanup command 565 attempted to replace stale SPF records.
65:398:- Issue: backup write permission problem caused record detection failure, so new v=spf1 -all records were created while old v=spf1 ip4:87.236.18.45 records remained.
66:402:## MSMTP_SECRET_SOURCE_PARSE_ERROR_576_20260701
67:403:- Context: proof 576 installed msmtp but sendmail auth test failed.
68:404:- Issue: [SENSITIVE_PATH] was sourced as shell, but SMTP password contained shell-special characters; source failed and auth used an invalid/truncated secret path.
69:408:## FORUM_MSMTP_MAIL_TRANSPORT_STILL_FAILING_20260701
70:410:- Issue: one or more sendmail/PHP mail tests still failed.
71:415:- Context: SMTP password was exposed in terminal output during failed msmtp setup.
72:416:- Cause: secret file with raw SMTP_PASS was later parsed/sourced by shell; shell error printed the raw assignment line.
73:426:## FORUM_SMTP_FAILED_CONFIG_DISABLED_AFTER_MAILCOW_BAN_20260701
74:429:- Action: remove forum-prod msmtp secret/config files so XenForo cannot keep retrying broken SMTP auth.
75:432:## FORUM_OLD_CODEVIPE_CLOUD_BACKUP_STALE_AFTER_FRESH5_20260701
76:435:- Evidence: latest service run failed with missing /root/backups/codevipe; remote inventory contains old codevipe-public/codevipe.sql files, not current five-forum archive.
77:445:## FORUM_XENFORO_MAIL_SMOKE_TEST_FAILED_20260701
78:450:- Context: custom XenForo mail smoke test 623 failed with Could not open input file /root/scripts/623_xenforo_mail_smoke.php.
79:456:## FORUM_NEXT_CHAT_KNOWN_ERRORS_AND_CLOSED_INCIDENTS_20260701
80:466: - Custom mail proof 623 failed with "Could not open input file".
81:471:- FORUM_MSMTP_MAIL_TRANSPORT_STILL_FAILING_20260701 / FORUM_SMTP_FAILED_CONFIG_DISABLED_AFTER_MAILCOW_BAN_20260701:
82:472: - Earlier SMTP attempts failed and triggered mailcow netfilter ban for forum public IP.
83:489: - Classified as external timeout noise, not runtime failure.
84:493:- Classification: deliverability/reputation/content filtering, not server failure.
85:509:## PARKED_DOMAINS_STAGE4_DNS01_PREFLIGHT_FAILED_20260701
86:512:- Observed: stage4 stopped safely with preflight_failed; CERTBOT_PRESENT=no, CERTBOT_DNS_CLOUDFLARE_PLUGIN=no, and CF_ZONE_ACCESS=no for newfi.ru, hapusya.ru and kingofwolk.ru.
87:513:- Impact: no parked-domain NPMplus route/certificate changes were applied by stage4; public HTTPS validation still failed certificate verification.
88:550:- Stage9 combined server_name block passed 5/6 but failed bare newfi.ru, so final HTTP-01 cutover must use separate exact per-host server blocks.
89:553:## PARKED_DOMAINS_STAGE11_LOW_ID_EXACT_ROUTE_FAILED_20260701
90:560:## PARKED_DOMAINS_SWITCH_TO_DNS01_AFTER_HTTP01_FAILURES_20260701
91:569:- Rule: for parked domains finalization, use existing DNS-01 certificates and validate from pve01 public HTTPS with content marker; do not use the failed local --resolve TLS check as the closure gate.
92:579:- Stage18 parked dry-run note: kingofwolk.ru dry-run hit Let's Encrypt rateLimited/service-busy after active certificate issuance and public HTTPS were already OK. Treat as transient external dry-run failure, not route/cert closure blocker.
93:587:## PVEPRO_STAGE21_LANDING_SERVICE_FAILED_20260701
94:593:- Context: Stage22 succeeded on edge-side landing, cert, NPMplus route and Cloudflare A upserts, but wrapper had a non-critical scp wildcard failure due __pycache__ and public validation used cached DNS for root pvepro.ru.
95:594:- Mistakes recorded: do not scp scripts/* after local py_compile because __pycache__ may be present; do not continue after critical scp failure in future apply stages; do not classify PVEPro landing as NetBird just because it contains a link text; use Cloudflare authoritative A records and --resolve validation when DNS propagation may lag.
96-613:- Rule: do not open router public admin or certificate deployment until WireGuard handshake is proven.
97-617:- Proof 666 is superseded because it showed PUBLIC_SSH_22_STILL_OPEN.
98:620:## TAFTAUTO_CERTBOT_CLOUDFLARE_PLUGIN_MISSING_20260702
99:621:- Certbot on edge rejected --dns-cloudflare-* arguments: dns-cloudflare plugin missing/not loaded.
100:622:- No taftauto certificate issued in failed run; no DNS A records or NPMplus routes changed.
101:634:- No certificate, DNS, NPMplus route, or deploy state changed by the failed dry-run.
102:637:- Proof 672 is superseded/partial: it printed STATUS=OK even though certbot dry-run failed.
103:638:- Failure reason: Let's Encrypt staging/service returned rateLimited / Service busy; retry later.
104:643:## HOMEPAGE_XENFORO_INFO_INLINE_PYTHON_SYNTAX_ERROR_20260702
105:645:- Failed before services.yaml write.
106:649:- Failed before write. Fix uses YAML-aware edit with exact names Router/Public Domain.
107:655:## 20260702_CF_TOKEN_AUDIT_BROKEN_COMMAND
108:657:- Факт: bash начал интерпретировать Python-код, появились syntax error и вторичный prompt >.
109-671:- Do not touch Cloudflare Homepage card when operator says it is green and opens correctly.
110:675:- When asked to fix one missing Uptime Kuma service, add only one monitor, then generate proposals separately.
111-678:- Do not touch Cloudflare when operator says it is green and opens correctly.
112:680:## LESSON_20260702_DOCKGE_STALE_STACKS_NOT_CONTAINERS
113:681:- Dockge inactive items after migration can be stale compose folders, not stopped containers.
114:684:- Do not delete containers to fix Dockge inactive UI; connect agents and archive only confirmed stale moved stack definitions.
115:686:## STAGE4B_MASKED_SQL_QUERY_FAILURE_20260714
116:688:- Empty blocks are query failures, not proof that metadata, triggers or functions are absent.
117:689:- Future PostgreSQL audits must use ON_ERROR_STOP, visible stderr, independent query RC and quote-safe dollar literals.
118:693:- PostgreSQL returned column id does not exist and ON_ERROR_STOP correctly stopped all following SQL checks.
119:698:- pg_dump failed only because fsync on /dev/null returned Invalid argument.
120:702:## STAGE4C_REMOTE_PREP_THIRD_SUBSTEP_FAILED_20260714
121:713:## STAGE4C_MIGRATION003_FINGERPRINT_DOLLAR_QUOTE_FAILURE_20260714
122:714:- Migration 003 isolated dry-run failed at fingerprint CHECK because the regex end anchor collided with the SQL dollar-quote delimiter and produced an unterminated dollar-quoted string.
123:727:- This false expectation caused diagnostic RC=32 without any infrastructure failure or production change.
124:737:- The six wrapper validation failures are cascading missing-marker checks and do not prove six independent infrastructure faults.
125:750:- The exact traceback and failing assertion must be captured by an isolated identical reproduction before changing the candidate.
126:753:- Adapter candidate passed Python 3.13 but failed Python 3.11 parsing at Path(row["path"]) inside a double-quoted f-string.
127:755:- The reproduction command captured the expected compile failure but incorrectly returned CAPTURE_BAD=1 because its validator required compile RC=0.
128:761:- The wrapper validation count is cascading missing-marker evidence and does not represent twenty-one independent faults.
129:767:- This caused UnicodeDecodeError before any design assertion failed.
130:772:## STAGE4E_V2_VALIDATOR_FAILED_STATIC_COMPILE_20260714
131:773:- Stage4E design candidate v2 failed static compilation before local design validation started.
132:774:- The reported validation count is cascading missing-marker evidence, not twenty-five independent design defects.
133:775:- The generated validator must be inspected at the exact SyntaxError line before another candidate is created.
134:787:- The executed backup script does not write that health file; therefore the stale health timestamp was not evidence of backup failure.
135:792:- Failure class: переход к следующей задаче при наличии незакрытого хвоста.
136-794:- Closure gate: UNRESOLVED_TAIL_COUNT=0; BLOCKER_COUNT=0; TEMPORARY_ARTIFACT_COUNT=0; ROLLBACK_VERIFIED=YES; HEALTH=OK; PROOF_UPDATED=YES; REFERENCE_UPDATED=YES.
137-795:- Если закрытие невозможно из-за внешней зависимости, опасной неоднозначности или обязательного решения пользователя, фиксировать BLOCKED/OPEN и не заявлять CLOSED.
138:798:## ERROR_20260714_DEPENDENCY_FACT_QUERY_GLOBAL_LIMIT
139:806:## ERROR_20260714_COLLECTOR_PATCH_REQUIREMENT_PATTERN_MISMATCH
140-812:- Status: CLOSED; PRIOR_TAIL_UNRESOLVED_COUNT=0.
141:814:## ERROR_20260714_PRIVILEGED_FILE_REDIRECTION_BEFORE_SUDO
142-816:- Root cause: `< collector.py` was opened by the unprivileged caller shell before sudo executed wc.
143:822:## ERROR_20260714_COLLECTOR_RUN_COLUMN_ASSUMPTION
144:823:- Symptom: SQL failed because completed_at did not exist.
145:824:- Root cause: collector_runs columns were assumed instead of read from information_schema; actual names are finished_at and error_text.
146-828:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
147:830:## ERROR_20260714_REMOTE_STDIN_ARGUMENT_SHIFT
148-836:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
149:838:## ERROR_20260714_OVERSIZED_COMMAND_PARSE_FAILURE
150:839:- Symptom: Bash получил одиночный фрагмент и сообщил syntax error near unexpected token `(`.
151-845:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
152:847:## ERROR_20260714_HOMELAB_ADMIN_HELP_RC_ASSUMPTION
153:848:- Symptom: template audit завершился HOMELAB_ADMIN_HELP_FAILED.
154-854:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
155:856:## ERROR_20260715_EXPECTED_NEGATIVE_RC_TRAP_CLASSIFICATION
156:859:- Correction: expected failures are evaluated only through an explicit if/else branch with immediate RC capture.
157-862:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
158:864:## ERROR_20260715_DUPLICATE_MACHINE_MARKER_COUNT
159:865:- Symptom: error-register candidate construction stopped with RC1 before applying the candidate.
160-872:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
161:875:## ERROR_20260715_OUTER_WRAPPER_SHELL_SYNTAX
162:876:- Symptom: the operator shell rejected the generated one-line wrapper with syntax error near the final subshell parenthesis.
163:878:- Impact: parsing failed before the start marker, guards and homelab-admin invocation; no task phase or infrastructure action executed.
164:881:- Anti-regression: after any parser-level failure, treat the attempted phase as not executed and verify package and run-state before retrying.
165-885:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
166:888:## ERROR_20260715_INLINE_REMOTE_SHELL_IN_APPLY_PHASE
167-899:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
168:902:## ERROR_20260715_OPTIONAL_RUN_DIRECTORY_FIND_UNDER_PIPEFAIL
169:904:- Root cause: find was executed against an optional directory that did not exist; under pipefail the assignment returned RC1.
170:906:ANTI_REGRESSION_TOKEN=MISSING_OPTIONAL_RUN_DIRECTORY_TREATED_AS_ZERO
171:907:- Anti-regression: optional paths must have an explicit existence branch before find under pipefail.
172-909:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
173:912:## ERROR_20260715_PG_CONSTRAINT_CONTYPE_CHAR_CONCAT
174:913:- Symptom: isolated Stage4H acceptance failed during schema baseline capture.
175-922:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
176:925:## ERROR_20260715_LIVE_COLLECTOR_VERIFIER_WRONG_PATH
177-933:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
178:936:AUTOREFRESH_INLINE_FAILURE_20260721
179:940:AUTOREFRESH_WRAPPER_PUBLICATION_FAILURE_20260721
180:948:BACKUP-AUDIT-CONTROLPLANE-054 was successfully published at commit 2b966acfe553b4d9f8d4a62bf9533478bdd1a073, but was incorrectly reported missing because a stale directory listing was treated as authoritative.
181:951:BACKUP_UNSAFE_SOURCE_CAPTURE_071_PARSE_ERROR_20260721
182:952:BACKUP-UNSAFE-SOURCE-CAPTURE-071 failed before execution: shell syntax error near unexpected token (.
183-=== REFERENCE_TAIL ===
184-- Existing NPMplus Admin monitor was repaired to http://[PRIVATE_IP]:18091/npmplus, not deleted.
185-- Edge health wrapper /usr/local/sbin/router-moscow-health provides safe endpoints on [PRIVATE_IP]:18091 for router-moscow, npmplus, loki, alloy, cadvisor, registry-cache, socket-proxy, diun, kopia and crowdsec.
--
197-
198-## UPTIME_KUMA_FOUR_HEALTH_SOURCES_REPAIRED_20260702
199:- Four Uptime Kuma monitors were still unavailable because three health endpoints returned 404 and Paperless Restore had STATUS=ERROR.
200-- Repaired pve01 health source files under /var/lib/homelab-health:
201- - backup-restore-dashboard.txt
202- - restore-drill-index.txt
--
207-- Monitors were repaired, not deleted.
208-
209:## DOCKGE_DISCONNECTED_HOSTS_AND_STALE_STACKS_20260702
210-- Dockge is currently present only on edge-vm.
211-- core-apps and monitoring run Docker containers but have no Dockge/Dockge-agent detected.
212-- Dockge "2 not connected" should be interpreted as two disconnected Docker hosts unless later evidence shows otherwise.
213-- No stopped containers were found across edge-vm, core-apps and monitoring during analysis.
214:- Edge Dockge has stale inactive stack definitions after services moved to core-apps and monitoring.
215:- Do not delete containers. First connect remote hosts or archive stale stack definitions after classification.
216-
217-## DOCKGE_REMOTE_AGENTS_INSTALLED_20260702
218-- Dockge main instance runs on edge-vm [PRIVATE_IP].
--
222-- Edge Dockge agent table stores the two agent URLs with username/password; password must never be printed.
223-- No runtime containers were deleted.
224:- Stale edge stack definitions were not archived yet; verify Dockge UI connected state first.
225-
226:## DOCKGE_REMOTE_AGENTS_AND_STALE_ARCHIVE_FINAL_20260702
227-- Dockge main instance: edge-vm [PRIVATE_IP].
228-- Remote agents connected: core-apps [PRIVATE_IP]:5001, monitoring [PRIVATE_IP]:5001.
229:- Stale moved edge stack folders archived under /opt/dockge-stale-archive/20260702T230442Z on edge-vm; no runtime containers were deleted.
230-- External remote stacks exposed to Dockge using bind mounts: /opt/vaultwarden-compose, /opt/gotify-compose, /opt/uptime-kuma-compose into /opt/stacks.
231:- Final target: remote not_visible count must be 0 and edge stale remaining matches must be 0.
232-
233-## DOCKGE_FINAL_MANAGED_STACKS_20260702
234-- Dockge main stack, npmplus and remote dockge-agent stacks were recreated from /opt/stacks so Dockge can manage them.
235-- Edge Dockge listens on 127.0.0.1:5001; LAN probe to [PRIVATE_IP]:5001 may return 000 and is not the correct health proof.
236-- Remote agents listen on [PRIVATE_IP]:5001 and [PRIVATE_IP]:5001.
237:- Stale moved stacks were archived, not deleted.
238-
239-## SERVICE_SETUP_ROADMAP_DEEP_20260702
240-- Configuration order: secrets, DNS/proxy/TLS, backup/restore, monitoring, SSO, docs, core data apps, sensitive apps, productivity, media, automation, utilities.
--
293-- collector.py: 113 строк, SHA256 подтверждён.
294-- Привилегированные файлы нельзя читать через caller-side `< file` перед sudo; применён sudo wc без внешнего перенаправления.
295:- Фактические поля collector_runs: finished_at и error_text; completed_at и error отсутствуют.
296-- Найдены один canonical source instance и один collector_runs_foundation_enrich_trg.
297-- Trigger обогащает collector_runs полями source_instance_id и run_key.
298-- Текущий collector ещё не пишет provenance-поля observations; migration 003 не применена.
299:- Production осталась 0|0|OK, timer активен, failed units отсутствуют.
300-- Baseline-аудит закрыт без хвостов; Stage4H остаётся OPEN для isolated collector provenance patch candidate.
301-- Proof: /var/lib/homelab-cluster-admin-stage4/stage4h-collector-baseline-audit-final-v1/report.txt
302-- Secret scan: /var/lib/homelab-cluster-admin-stage4/stage4h-collector-baseline-audit-final-v1/strict-secret-scan.txt
--
349- "coverage": {
350- "status": "PARTIAL",
351: "missing": [
352- "automatic_refresh",
353- "per_node_freshness",
354- "structured_service_inventory"
--
357- "freshness": {
358- "max_age_seconds": 86400,
359: "stale_requires_refresh": true
360- },
361- "entrypoints": {
362- "full_context_url": "https://git.gram1.ru/homelab-admin/homelab-public-context/raw/commit/e23caf794cc68ce5fa6f8922efa7f2a35c8aa6e0/runtime/history/CONTEXT-SOURCES-FULL-002.txt",
363: "errors_index_url": "https://git.gram1.ru/homelab-admin/homelab-public-context/raw/commit/a615069de3d32564fc4de29baff949bdaf811caf/runtime/history/ERRORS-INDEX-004.json",
364- "workflow_contract_url": "https://git.gram1.ru/homelab-admin/homelab-public-context/raw/commit/2f37a4eb600528d03f5c570af8c5aa0e463b1798/runtime/history/WORKFLOW-CONTRACT-003.json",
365- "double_check_rule_url": "https://git.gram1.ru/homelab-admin/homelab-public-context/raw/commit/d1061e47e7955dba9c73a2841d4eacf02ccc1bf5/runtime/policy/double-check-rule.json"
366- }
--
434-local-lvm lvmthin active 848797696 280188119 568609576 33.01%
435-=== BACKUP_JOBS ===
436:[{"compress":"zstd","enabled":1,"id":"homelab-nightly-all","mailnotification":"failure","mode":"snapshot","next-run":1784680200,"prune-backups":{"keep-daily":"4"},"schedule":"03:30","storage":"local","type":"vzdump","vmid":"110,111,112,113,130,150,190"}]
437-=== BACKUP_MATRIX ===
438:STATUS=FAIL
439-TS=2026-07-21T07:02:41Z
440-TYPE=unified-backup-matrix
441-MATRIX_MODE=authoritative
--
452-JOB=dpkg-db-backup STATUS=OK EVIDENCE=execution_only AGE=36147 MAX_AGE=172800
453-JOB=filebrowser-backup STATUS=OK EVIDENCE=artifact_or_health AGE=14443 MAX_AGE=259200
454:JOB=filebrowser-restore-validate STATUS=STALE EVIDENCE=artifact_or_health AGE=1016424 MAX_AGE=864000
455-JOB=homelab-appbackup-dockge-stacks STATUS=OK EVIDENCE=artifact_or_health AGE=22248 MAX_AGE=172800
456-JOB=homelab-appbackup-mariadb-batch STATUS=OK EVIDENCE=execution_only AGE=15387 MAX_AGE=172800
457-JOB=homelab-appbackup-npmplus-edge STATUS=OK EVIDENCE=artifact_or_health AGE=20573 MAX_AGE=172800
--
463-JOB=homelab-backup-freshness STATUS=OK EVIDENCE=execution_only AGE=35589 MAX_AGE=172800
464-JOB=homelab-backup-healthcheck STATUS=OK EVIDENCE=execution_only AGE=10631 MAX_AGE=172800
465:JOB=homelab-backup-sla-health STATUS=FAIL EVIDENCE=artifact_or_health AGE=144 MAX_AGE=172800
466-JOB=homelab-edge-vm-offhost-freshness STATUS=OK EVIDENCE=artifact_or_health AGE=18426 MAX_AGE=172800
467-JOB=homelab-emergency-backup STATUS=OK EVIDENCE=artifact_or_health AGE=24539 MAX_AGE=172800
468-JOB=homelab-extended-appbackup STATUS=OK EVIDENCE=artifact_or_health AGE=14156 MAX_AGE=172800
--
472-JOB=homelab-mail-cloud-edge-vm-upload STATUS=OK EVIDENCE=artifact_or_health AGE=174045 MAX_AGE=864000
473-JOB=homelab-mail-cloud-nextcloud-vm-upload STATUS=OK EVIDENCE=artifact_or_health AGE=7575 MAX_AGE=864000
474:JOB=homelab-mail-cloud-restore-drill STATUS=STALE EVIDENCE=artifact_or_health AGE=988269 MAX_AGE=864000
475-JOB=homelab-npmplus-kuma-config-backup STATUS=DISABLED EVIDENCE=artifact_or_health AGE=60071 MAX_AGE=172800
476-JOB=homelab-router-running-config-mail-cloud STATUS=OK EVIDENCE=artifact_or_health AGE=17076 MAX_AGE=172800
477-JOB=homelab-vaultwarden-backup STATUS=DISABLED EVIDENCE=artifact_or_health AGE=987768 MAX_AGE=172800
--
485-JOB=immich-media-offhost-sync STATUS=OK EVIDENCE=artifact_or_health AGE=4286 MAX_AGE=259200
486-JOB=memos-backup STATUS=OK EVIDENCE=artifact_or_health AGE=14547 MAX_AGE=259200
487:JOB=netbird-vps-backup STATUS=FAIL EVIDENCE=artifact_or_health AGE=1747896 MAX_AGE=864000
488-JOB=paperless-backup STATUS=DISABLED EVIDENCE=artifact_or_health AGE=16347 MAX_AGE=259200
489:JOB=paperless-restore-dry-run STATUS=STALE EVIDENCE=artifact_or_health AGE=1001986 MAX_AGE=864000
490-JOB=skladchik-reports-monitor-backup STATUS=OK EVIDENCE=artifact_or_health AGE=19639 MAX_AGE=172800
491-JOB=skladchik-reports-monitor-backup-restore-check STATUS=OK EVIDENCE=artifact_or_health AGE=190273 MAX_AGE=864000
492:JOB=vm160-remote-backup STATUS=FAIL EVIDENCE=artifact_or_health AGE=26607 MAX_AGE=172800
493-========== NODE=pve01 ==========
494---- PVE_STATUS ---
495-{
--
522- "pveversion": "pve-manager/9.2.3/d0fde103346cf89a"
523-}
524:--- FAILED_UNITS ---
525:● homelab-backup-matrix-refresh.service loaded failed failed Generate unified homelab backup matrix state
526:● homelab-cluster-admin-pull.service loaded failed failed Pull cluster-admin observer health into pve01
527:● homelab-vm160-remote-backup.service loaded failed failed VM160 full-image backup streamed from pve02 to pve03
528---- TIMERS ---
529-Tue 2026-07-21 16:23:54 MSK 2min 43s Tue 2026-07-21 16:18:29 MSK 2min 41s ago homelab-crypto-portfolio-chart-health.timer homelab-crypto-portfolio-chart-health.service
530-Tue 2026-07-21 16:24:14 MSK 3min 3s Tue 2026-07-21 16:19:14 MSK 1min 56s ago homelab-smartctl-textfile.timer homelab-smartctl-textfile.service
--
585-2026-06-24T16:02:38.7930804000 145 /var/lib/homelab-health/vps-identity-peer-discovery.txt
586-2026-06-24T16:23:03.5382758840 2155 /var/lib/homelab-health/private-vpn-hosts.txt.bak-ok-wrapper-20260624T132705Z
587:2026-06-24T21:07:27.7414593140 118 /var/lib/homelab-health/pve01-failed-units.txt
588-2026-06-24T22:00:51.3972551230 196 /var/lib/homelab-health/edge-restore-proofs-offhost.txt
589-2026-06-24T22:03:31.7226417460 263 /var/lib/homelab-health/nextcloud-restore-proof-offhost.txt
590-2026-07-01T04:31:05.4336469050 312 /var/lib/homelab-health/netbird-vps-backup.txt
--
624-2026-07-07T11:29:49.2086910530 369 /var/lib/homelab-health/pve03-staging-retention.txt
625-2026-07-07T15:21:23.8337500480 448 /var/lib/homelab-health/pve03-staging-retention-closure.txt
626:2026-07-07T15:25:57.4458619660 291 /var/lib/homelab-health/pve03-failed-unit-cleanup.txt
627:2026-07-07T15:52:21.5095968040 398 /var/lib/homelab-health/pve03-failed-unit-cleanup-closure.txt
628-2026-07-07T15:59:30.7733163830 343 /var/lib/homelab-health/remote-git-sops-age-readiness.txt
629-2026-07-07T16:01:07.1647092920 369 /var/lib/homelab-health/remote-git-sops-age-policy.txt
630-2026-07-07T16:42:29.4222211440 485 /var/lib/homelab-health/remote-git-sops-age-closure.txt
--
639-2026-07-08T07:59:58.6953604310 426 /var/lib/homelab-health/sops-edge-secret-coverage-audit.txt
640-2026-07-08T08:01:49.3036518230 547 /var/lib/homelab-health/sops-edge-secret-coverage-closure.txt
641:2026-07-08T08:20:29.6041867030 337 /var/lib/homelab-health/remote-failed-unit-audit.txt
642:2026-07-08T08:23:09.5176054980 420 /var/lib/homelab-health/remote-failed-unit-plan.txt
643:2026-07-08T08:25:25.7376658940 508 /var/lib/homelab-health/remote-failed-unit-apply.txt
644-2026-07-08T08:27:02.9011355390 426 /var/lib/homelab-health/paperless-core-guard-audit.txt
645:2026-07-08T08:38:32.4205675490 531 /var/lib/homelab-health/remote-failed-unit-cleanup-seal.txt
646-2026-07-08T08:40:40.2615024670 497 /var/lib/homelab-health/duty-admin-residual-audit.txt
647-2026-07-08T08:42:30.3881692640 322 /var/lib/homelab-health/pve03-capacity-forensics.txt
648-2026-07-08T08:42:31.7994319160 591 /var/lib/homelab-health/duty-admin-residual-cleanup.txt
--
653-2026-07-08T09:26:36.7402046120 390 /var/lib/homelab-health/dockge-agent-capability-analysis.txt
654-2026-07-08T09:26:36.7402750460 390 /var/lib/homelab-health/service-inventory-leftovers-phase00.txt
655:2026-07-08T09:26:36.7403618890 390 /var/lib/homelab-health/dockge-stale-stack-classification-after-agents.txt
656-2026-07-08T09:26:36.7403838590 390 /var/lib/homelab-health/dockge-remediation-plan.txt
657:2026-07-08T09:26:36.7404284930 386 /var/lib/homelab-health/failed-unit-legacy-inventory.txt
658-2026-07-08T09:26:36.7404732790 390 /var/lib/homelab-health/service-setup-backlog-exact-order.txt
659-2026-07-08T09:26:36.7405361960 386 /var/lib/homelab-health/ungated-health-backlog-closure.txt
660:2026-07-08T09:26:36.7406026680 390 /var/lib/homelab-health/homepage-missing-card-proposals.txt
661-2026-07-08T09:26:36.7406281280 390 /var/lib/homelab-health/dockge-agent-runtime-method.txt
662-2026-07-08T09:26:36.7406657810 390 /var/lib/homelab-health/homepage-card-recommendations.txt
663:2026-07-08T09:26:36.7408259230 390 /var/lib/homelab-health/homepage-missing-card-proposals-final.txt
664-2026-07-08T09:26:36.7408607340 390 /var/lib/homelab-health/service-inventory-summary-phase00.txt
665:2026-07-08T09:26:36.7409298280 386 /var/lib/homelab-health/failed-unit-legacy-classification.txt
666:2026-07-08T09:26:36.7409559880 381 /var/lib/homelab-health/failed-unit-legacy-local-cleanup.txt
667-2026-07-08T09:26:36.7409816770 381 /var/lib/homelab-health/phase00-freeze-final.txt
668:2026-07-08T09:26:36.7410552890 389 /var/lib/homelab-health/dockge-stale-archive-last.txt
669-2026-07-08T09:26:36.7410994270 390 /var/lib/homelab-health/dockge-agent-schema-and-register-method.txt
670-2026-07-08T09:26:36.7411245420 390 /var/lib/homelab-health/dockge-disconnected-containers-analysis.txt
671-2026-07-08T09:26:36.7411439090 386 /var/lib/homelab-health/final-readiness-cycle-break.txt
--
673-2026-07-08T09:26:36.7411808970 390 /var/lib/homelab-health/service-setup-order-analysis.txt
674-2026-07-08T09:26:36.7412497980 390 /var/lib/homelab-health/dockge-exact-agent-table-inspect.txt
675:2026-07-08T09:26:36.7413054120 381 /var/lib/homelab-health/failed-unit-legacy-cleanup-seal.txt
676-2026-07-08T09:26:36.7413803250 390 /var/lib/homelab-health/service-setup-roadmap-deep.txt
677-2026-07-08T09:26:36.7414375920 390 /var/lib/homelab-health/uptime-kuma-monitoring-proposals.txt
678-2026-07-08T09:26:36.7414750460 381 /var/lib/homelab-health/kuma-legacy-monitor-quarantine.txt
--
934-2026-07-13T08:10:44.7277816120 1647 /var/lib/homelab-health/crypto-portfolio-remove-grt-prechange-backup.txt
935-2026-07-13T08:39:10.8547948980 330 /var/lib/homelab-health/crypto-portfolio-remove-grt-recovery-audit.txt
936:2026-07-13T08:41:01.3664795150 684 /var/lib/homelab-health/crypto-portfolio-remove-grt-failed-units-diagnostic.txt
937-2026-07-13T08:48:13.2600633440 1640 /var/lib/homelab-health/crypto-portfolio-remove-grt-production-apply.txt
938-2026-07-13T09:11:24.1542650490 1347 /var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-candidate.txt
939-2026-07-13T10:04:47.2300377080 1490 /var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-browser-smoke.txt
--
1078-2026-07-21T16:20:08.5291874140 641 /var/lib/homelab-health/private-vpn-hosts.txt
1079-2026-07-21T16:21:10.8330770850 0 /var/lib/homelab-health/private-vpn-hosts.txt.tmp
1080:/var/lib/homelab-health/final-closeout.txt:1:STATUS=OK TS=2026-07-08T09:02:10Z TYPE=final-closeout BAD=0 WARN=0 MODE=FINAL_OPERATOR_CLOSEOUT CLOSEOUT_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 EXACT_WARN_REVIEW_FILES=0 ENDPOINT_LOCAL_OK=1 ENDPOINT_LAN_OK=1 ENDPOINT_PUBLIC_OK=1 RECOVERY_SEAL=YES FREEZE_SEAL=YES FINAL_GLOBAL_GREEN=YES REVIEW_SURFACE_REGISTRY=YES POST_CLEANUP=YES REMOTE_CLEANUP=YES BACKUP_SLA=OK BACKUP_COVERAGE=OK EXTENDED_APPBACKUP=OK VM_BACKUP=OK SECRET_EXPOSURE=OK EXTERNAL_PROBE=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T084146Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T084146Z EMERGENCY_JOB=homelab-pve01-emergency-pack-20260708T002739Z VM_DUMPS_JOB=homelab-pve01-vm-local-dumps-20260706T153606Z DIRTY_COUNT=0 REPORT=/var/lib/homelab-final-closeout/final-closeout-latest.md
1081:/var/lib/homelab-health/backup-matrix-cluster-final-seal.txt:1:STATUS=OK TS=2026-07-13T17:51:31Z TYPE=backup-matrix-cluster-final-seal BAD=0 CURRENT_STAGE_COMPLETE=YES MATRIX_STATUS=OK JOBS_TOTAL=35 JOBS_OK=31 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 READ_WRITE_OVERLAP=0 VM_BACKUP_HEALTH=OK VM_FULL_BACKUP=OK EDGE_VM_CANONICAL_BACKUP=homelab-edge-vm-offhost-freshness.timer OLD_EDGE_TIMER_REENABLED=NO CLUSTER_PASSPORT=OK FINAL_READINESS=OK FINAL_READINESS_PASS=53 FINAL_READINESS_FAIL=0 OVERALL=OK CLUSTER_ADMIN=OK PROBE_OK=4 PROBE_BAD=0 WEBPANEL=OK FAILED_UNITS=0 CLUSTER_NODES=3 QUORATE=YES GENERATED_SYNC_HEAD=597119391e7b265b85fd9a76644e4e33af47e469 DESIRED_STATE_DIRTY=0 MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 MATRIX_STATE_SHA256=14a31112aac5d577ae01df567921b093b2fedcf7189c4cc8f009216ee0749380 CLUSTER_PASSPORT_SHA256=1b8b924ae0e5ce6872fe5671318094ce494806c2db7efe28299b5ff49adc0d0d FINAL_READINESS_SHA256=ac5159fa8c497afe021b5cb232f08cfec0e1a4d516d8aae966ba03f7034f1412 OVERALL_SHA256=6ae529bfdf5811f686d3b53978f453ce4f8757c74fc1be105de219b13a61ba46 PRESEAL_SHA256=7ea1f710cd311bd51dba05a313b4c540f57d97407f008881e502fc213b4b3c99 EVIDENCE_MANIFEST_SHA256=b4c68312026e1328b208a74090891f4809ab31ec69069e94a3d6902429bf3566 BACKUP_RUN=NO RESTORE_RUN=NO CLOUD_WRITE_RUN=NO RETENTION_RUN=NO NEXT_ACTION=AUDIT_AND_MIGRATE_SPECIAL_VM_EMERGENCY_AND_NON_APP_BACKUP_FAMILIES EVIDENCE_ROOT=/var/lib/homelab-change-sets/backup-matrix-cluster-final-seal/20260713T175131Z
1082:/var/lib/homelab-health/final-cycle-stability-seal.txt:1:STATUS=OK TS=2026-07-07T20:08:34Z TYPE=final-cycle-stability-seal BAD=0 WARN=0 MODE=FORCED_REPLAY_AFTER_CYCLE_BREAK STABLE=YES EXTERNAL_PROBE_STATUS=OK EXTERNAL_PROBE_WARN=0 EXTERNAL_PROBE_SMOKE_BAD=0 FINAL_STATUS=OK FINAL_BAD=0 FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 OVERALL_STATUS=OK OVERALL_BAD=0 OLD_EXTERNAL_PROBE_LIVE_MARKERS=0 OLD_EXTERNAL_PROBE_SOURCE_MARKERS=0 FINAL_CLUSTER_CHECKS=0 FINAL_LEGACY_DUTY_CHECKS=0 OVERALL_LEGACY_DUTY_GATE=0 CLUSTER_LEGACY_DUTY_GATE=0 SOURCE_CLUSTER_LEGACY_DUTY_GATE=0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-final-cycle-break/stability-seal-latest.md
1083:/var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-production-closure.txt:1:STATUS=OK TS=2026-07-13T08:12:14Z TYPE=crypto-portfolio-ui-center-alert-currency-production-closure BAD=0 PRODUCTION_CHANGED=YES_UI_ONLY INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb SHARP_MOVES_CENTERED=YES ALERT_CURRENCY_USD_RUB=OK BROWSER_RUNS=2 BROWSER_ERRORS=0 GRT_ROWS=0|0|0 GRT_SOURCE_REFERENCES=0 POSTCHANGE_BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T075801Z RESTORE_CHECK=OK_BOTH_CLOUDS RETENTION_ROLLBACK_COUNT=3 RETENTION_OPTIONAL_OTHER_CANDIDATES=1 RETENTION_REMOVED_ROLLBACK_COUNT=0 DANGLING_IMAGES_REMAINING=2 UNREFERENCED_DANGLING_IMAGES=0 DESIRED_STATE_HEAD=2999910549eaff9b9431b9c656d318340eb95cbd DESIRED_STATE_CLEAN=YES PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES READY_FOR_UI_FINAL_SEAL=YES
1084:/var/lib/homelab-health/desired-state-remote-git-audit.txt:1:STATUS=OK TS=2026-07-08T09:27:24Z TYPE=desired-state-remote-git-audit BAD=0 WARN=0 MODE=READ_ONLY_DESIRED_STATE_REMOTE_GIT_AUDIT AUDIT_OK=YES REMOTE_READY=NO ADD_REMOTE_DECISION=NO_CHANGE REPO_PRESENT=1 DIRTY_COUNT=0 REMOTE_COUNT=0 URL_CREDENTIAL_HINT=0 HTTPS_GITEA_OK=1 HTTPS_API_OK=0 BACKEND_OK=1 BACKEND_API_OK=0 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 OVERALL_STATUS=OK OVERALL_BAD=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 CLOSEOUT_OK=YES RECOVERY_SEAL_OK=YES INDEPENDENT_RUNNER_NO_DEPLOY=YES REPORT=/var/lib/homelab-desired-state-remote-git/audit-latest.md
1085:/var/lib/homelab-health/cluster-admin-stage3-postapply-backup.txt:1:STATUS=OK TS=2026-07-13T03:43:59Z TYPE=cluster-admin-stage3-postapply-backup BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=POSTAPPLY_BACKUP_AND_CONTROLLED_COLLECTOR_RUN PRODUCTION_DATABASE_CHANGED=COLLECTOR_NORMAL_OPERATION_ONLY PRODUCTION_SCHEMA_CHANGED=NO LIVE_APPLICATION_CODE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=YES_CONTROLLED_POSTAPPLY_BACKUP LIVE_APP_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 APPBACKUP=VERIFIED APPBACKUP_JOB=homelab-pve01-app-cluster-admin-incident-engine-20260713T034400Z APPBACKUP_SIZE_BYTES=1791603 APPBACKUP_RESTORE_CHECK=OK APPBACKUP_ROUTING=VERIFIED COLLECTOR_CONTROL_RUN=OK COLLECTOR_RUN_ID=3113 CANONICAL_CONSISTENCY=0|0|0|0|0 INCIDENT_ENGINE=ACTIVE COLLECTOR_TIMER=ACTIVE VM180_FAILED_UNITS=0 READY_FOR_STAGE3_FINAL_SEAL=YES STAGE3_CLOSED=NO POSTAPPLY_ROOT=/var/lib/homelab-stage3-postapply/20260713T034359Z
1086:/var/lib/homelab-health/cluster-admin-webpanel.txt:1:STATUS=OK TS=2026-07-11T01:39:28Z TYPE=cluster-admin-webpanel BAD=0 URL=http://[PRIVATE_IP]:8080/ STATUS_URL=http://[PRIVATE_IP]:8080/status.txt SELF_FAILED_UNITS=0 QGA=active BACKUP=MAIL_CLOUD_2COPY_VERIFY_OK
1087:/var/lib/homelab-health/post-cleanup-final-seal.txt:1:STATUS=OK TS=2026-07-08T06:10:30Z TYPE=post-cleanup-final-seal BAD=0 WARN=0 MODE=POST_CLEANUP_OWNER_TIMER_FINAL_SEAL FINAL_SEAL_OK=YES REPLAY_STATUS=OK REPLAY_OK=YES LOCAL_START_BAD=0 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 REVIEW_ITEMS_INTENTIONALLY_LEFT=yes LEGACY_PHASE_ITEMS_INTENTIONALLY_LEFT=yes FINAL_STATUS=OK FINAL_BAD=0 FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 LOCAL_CLEANUP_OK=YES REMOTE_CLEANUP_OK=YES RESIDUAL_SEAL_OK=YES SOPS_STATUS=OK EXTERNAL_PROBE_STATUS=OK EXTERNAL_PROBE_WARN=0 EXTERNAL_PROBE_SMOKE=0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-post-cleanup-replay/final-seal-latest.md
1088:/var/lib/homelab-health/pve01-core-monitoring-app-restorecheck.txt:1:STATUS=OK TS=2026-07-08T15:57:27Z TYPE=pve01-core-monitoring-app-restorecheck BAD=0 MODE=APP_LEVEL_RESTORE_CHECKS_TWO_CLOUDS FULL_VM_RESTORE_AVAILABLE=NO TOTAL_APPS=30 OK_APPS=30 FAIL_LIST=none REPORT=/var/lib/homelab-node-loss/pve01-core-monitoring-app-level/app-restorecheck-latest.md
1089:/var/lib/homelab-health/restore-drill-index.txt:5:MISSING_OR_NOT_OK=none
1090:/var/lib/homelab-health/mail-cloud-edge-vm-final-seal.txt:1:STATUS=OK TS=2026-07-14T01:02:58Z TYPE=mail-cloud-edge-vm-final-seal BAD=0 WARN=0 VMID=130 SOURCE=/mnt/staging/offhost/edge-vm-vzdump-from-pve03/vzdump-qemu-130-2026_07_13-03_30_01.vma.zst SIZE=10942350973 SHA256=6979558850813d99881b20c559600ec1c8748cc4f59f18ef9875eee6e40c6757 CLOUD_FOUND_DIR=20260713T183122Z MATRIX_JOB_ID=homelab_mail_cloud_edge_vm_upload MATRIX_STATUS=OK JOBS_TOTAL=36 JOBS_OK=32 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 UPLOAD_VERIFIED=YES DOWNLOAD_VERIFY=YES SHA_MATCH=YES SIZE_MATCH=YES PART_COUNT_MATCH=YES CONTROLLED_UPLOAD_MODE=--upload-no-retention RETENTION_MODE=disabled RETENTION_RUN=NO RETENTION_DELETE_COUNT=0 CLOUD_PURGE_RUN=NO SOURCE_DELETE_RUN=NO EDGE_SERVICE_RESULT=success EDGE_EXEC_MAIN_STATUS=0 EDGE_TIMER_ENABLED=enabled EDGE_TIMER_ACTIVE=active CLUSTER_ADMIN_STATUS=OK PROBE_OK=4 PROBE_BAD=0 CLUSTER_PASSPORT=OK DUTY_ADMIN=OK OVERALL=OK OVERALL_BAD=0 FINAL_READINESS=OK FINAL_PASS=53 FINAL_FAIL=0 WEBPANEL=OK WEBPANEL_SYNC=OK PUBLIC_OVERALL=OK FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DESIRED_STATE_PRE_SEAL_HEAD=98dfaeb35683e18cd4e986d0121bde60daac1e7c CURRENT_EDGE_VM_CLOUD_SUBSTAGE_COMPLETE=YES EVIDENCE_ROOT=/var/lib/homelab-change-sets/mail-cloud-edge-vm-final-seal-idle-safe/20260714T010226Z
1091:/var/lib/homelab-health/cluster-admin-stage2-dynamic-redryrun.txt:1:STATUS=OK TS=2026-07-13T01:20:06Z TYPE=cluster-admin-stage2-dynamic-redryrun BAD=0 PRODUCTION_CHANGED=TEMPORARY_DATABASE_AND_EVIDENCE_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO INCIDENT_ENGINE_RESTARTED=NO MIGRATION_ID=002_canonical_foundation_sync MIGRATION_SHA256=92a7f87ed6bf5215c81ec5f70ab1dc3c8339ecffa0b0f9fcab4eb6322651dc68 MIGRATION_PATCH_REQUIRED=NO COLLECTOR_PATCH_REQUIRED=NO APPLY_VERIFIER_PATCH_REQUIRED=YES FIXED_TOTAL_ENTITIES_REQUIRED=NO FIXED_TOTAL_ALIASES_REQUIRED=NO ACTIVE_CURRENT_SET_CONVERGENCE=OK NATURAL_KEY_CONVERGENCE=OK EPHEMERAL_ID_REFRESH=OK DYNAMIC_PLACEHOLDER_LIFECYCLE=OK RETIRED_PLACEHOLDERS_ALLOWED=YES ORPHANS=0 PREFLIGHT_TRANSACTION_ROLLBACK=OK DDL_ROLLBACK_TEST=OK ROLLBACK_AFTER_COMMITTED_SYNC=DO_NOT_PARTIAL_DDL_ROLLBACK PRESERVE_APPLIED_STATE_ON_POSTSYNC_VERIFIER_FAILURE=YES TEST_DATABASE_DROPPED=YES PRODUCTION_UNCHANGED=YES FRESH_PREAPPLY_BACKUP_REQUIRED=YES READY_FOR_FRESH_PREAPPLY_BACKUP=YES DESIRED_STATE_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 CONTRACT=/var/lib/homelab-change-sets/stage2-dynamic-redryrun/20260713T012006Z/dynamic-acceptance-contract.json RESULT=/var/lib/homelab-change-sets/stage2-dynamic-redryrun/20260713T012006Z/dynamic-redryrun-result.txt CHANGE_SET=/var/lib/homelab-change-sets/stage2-dynamic-redryrun/20260713T012006Z
1092:/var/lib/homelab-health/desired-state-remote-git.txt:1:STATUS=OK TS=2026-07-08T09:31:25Z TYPE=desired-state-remote-git BAD=0 WARN=0 MODE=NO_REMOTE_ADD_NO_PUSH_SEAL NO_REMOTE_SEAL_OK=YES AUDIT_STATUS=OK AUDIT_OK=YES REMOTE_READY=NO ADD_REMOTE_DECISION=NO_CHANGE GIT_REMOTE_CHANGED=NO GIT_PUSH_DONE=NO REPO_PRESENT=1 REPO_TOP=/srv/homelab-desired-state BRANCH=master DIRTY_COUNT=0 REMOTE_COUNT=0 URL_CREDENTIAL_HINT=0 HTTPS_GITEA_OK=1 HTTPS_API_OK=0 BACKEND_OK=1 BACKEND_API_OK=0 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 OVERALL_STATUS=OK OVERALL_BAD=0 FINAL_STATUS=OK FINAL_FAIL=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 CLOSEOUT_OK=YES RECOVERY_SEAL_OK=YES INDEPENDENT_RUNNER_NO_DEPLOY=YES RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T092404Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T092404Z REPORT=/var/lib/homelab-desired-state-remote-git/no-change-seal-latest.md
1093:/var/lib/homelab-health/service-registry.txt:1:STATUS=OK TS=2026-07-21T13:15:17Z TYPE=service-registry SERVICES=12 MISSING=0 REGISTRY=/var/lib/homelab-registry/service-registry-latest.tsv
1094:/var/lib/homelab-health/crypto-portfolio-remove-grt-discovery.txt:1:STATUS=OK TS=2026-07-13T03:50:49Z TYPE=crypto-portfolio-remove-grt-discovery BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=NO DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO PREVIOUS_FAILURE_ROOT_CAUSE=UNVERIFIED_STACK_PATH_ASSUMPTION CANDIDATE_DIRECTORIES=4 CANDIDATE_DATABASES=0 CANDIDATE_CONTAINERS=0 CANDIDATE_REFERENCE_FILES=8 READY_FOR_GRT_TARGETED_CAPTURE=YES DISCOVERY_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-discovery/20260713T035049Z
1095:/var/lib/homelab-health/post-cleanup-owner-timer-replay.txt:1:STATUS=OK TS=2026-07-08T06:01:13Z TYPE=post-cleanup-owner-timer-replay BAD=0 WARN=0 MODE=FORCED_REPLAY_CLEANUP_OWNERS REPLAY_OK=YES LOCAL_START_OK=10 LOCAL_START_BAD=0 EDGE_RC=0 EDGE_HEALTH=OK LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 EXTERNAL_PROBE_STATUS=OK EXTERNAL_PROBE_WARN=0 EXTERNAL_PROBE_SMOKE=0 SOPS_STATUS=OK DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-post-cleanup-replay/owner-timer-replay-latest.md
1096:/var/lib/homelab-health/paperless-core-guard-audit.txt:1:STATUS=OK TS=2026-07-08T05:26:59Z TYPE=paperless-core-guard-audit BAD=0 WARN=1 MODE=READ_ONLY_PAPERLESS_CORE_GUARD_AUDIT CLASS=REMOTE_DEFERRED_GUARD_NOT_HARD_GATE CONNECT_ERRORS=0 FAILED_ONLY_GUARD=YES IN_HARD_GATES=no OVERALL_BAD=0 FINAL_FAIL=0 DUTY_V2_GATED_NONOK=0 PAPERLESS_CONTAINER_LINES=0 CORE_RELATED_LINES=1 RESET_FAILED_DONE=no REPORT=/var/lib/homelab-remote-failed-unit-cleanup/paperless-core-guard-audit-latest.md
1097:/var/lib/homelab-health/hapusya-site-final-seal.txt:1:STATUS=OK TS=2026-07-13T12:33:08Z TYPE=hapusya-site-final-seal BAD=0 APP=hapusya-site URL=https://hapusya.ru/ PRESEAL_DESIRED_STATE_HEAD=69966eb83fc065141df3c46df0d6c0f282275046 ROUTE_SHA256=02d65cb0082a04cae282126cc829a002f171e863291c18ba04d788dc2d6282ca UNIT_SHA256=b3b07137b25ac0a0b4bdadadad654cf209387ade86a2ef2cdca58e8a7c0cc212 RELATIVE_TREE_SHA256=9dadf21da041bc4d6ee4a42f5b3ac7719c16bcc39052de32af895c6cd28286cb APPLY_MANIFEST_SHA256=f44992e3e0cc443b372779cb35d9e899d904a9960a9de03c14559886fdb1fc75 PRODUCTION_BROWSER_MANIFEST_SHA256=9f3cdf3063f8724d1d1c1ad619f47d6f98d434c3422a74903585146858a8dfac BROWSER_SMOKE=OK BROWSER_RUNS=5 BROWSER_ERRORS=0 DESKTOP_VIDEO=OK MOBILE_VIDEO=OK STATIC_FALLBACK=OK BACKUP_JOB=homelab-pve01-app-hapusya-site-20260713T115259Z ARCHIVE_SHA256=bf284292974600384c43c47fc99be6fedf0ebd68f722b402a7260f1df680d167 TWO_CLOUDS=OK RESTORE_CHECK=OK_BOTH_CLOUDS RETENTION=OK DESIRED_STATE_SYNC=OK GENERATOR_SHA256=507bbbc2691f1f08cb1dd3660ed76cf13a8c36c23833d5fdec40cf6e07486cb8 MAP_SHA256=815b52dcedf7e3e92e07939bade32c7e8ed6acc0f9e064b65db6daec9f559c63 POLICY_SHA256=4bf996ff451ae3ec56eb2fcd42842606cd97295b4d5141aedb135c956395e4b5 EVIDENCE_ROOT=/var/lib/homelab-change-sets/hapusya-site-final-seal/20260713T123308Z EVIDENCE_MANIFEST_SHA256=7eb993a17ac18a1f7c9ae0b52d745d3778833d15626ed34c82496f74916eb26a PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES PRODUCTION_CHANGED=NO_FINAL_SEAL_ONLY DESIRED_STATE_CHANGED=YES_FINAL_SEAL_COMMIT_ONLY ALL_HAPUSYA_SITE_TAILS_CLOSED=YES TASK_COMPLETE=YES
1098:/var/lib/homelab-health/router-watchdog.txt:1:STATUS=OK TS=2026-07-21T13:20:07Z TYPE=router-watchdog ACTION=none REASON=router_wan_and_outbound_healthy SSH_OK=1 WAN_OK=1 OUTBOUND_OK=2 FAIL_COUNT=0 THRESHOLD=5 LAST_REBOOT_EPOCH=0 REBOOTS_TODAY=0 HARD_POWER_HELPER_READY=yes
1099:/var/lib/homelab-health/pve03-failed-unit-cleanup-closure.txt:1:STATUS=OK TS=2026-07-07T12:52:21Z TYPE=pve03-failed-unit-cleanup-closure BAD=0 WARN=0 MODE=READ_ONLY_CLOSURE LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260707T124031Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260707T124031Z GOLDEN_CP_MATCH=YES RESTORE_CHECK=YES PVE03_FAILED_UNITS=0 REPORT=/var/lib/homelab-pve03-failed-unit-cleanup/closure-latest.md
1100:/var/lib/homelab-health/review-surface-registry-audit.txt:1:STATUS=OK TS=2026-07-08T06:20:10Z TYPE=review-surface-registry-audit BAD=0 WARN=1 MODE=READ_ONLY_REVIEW_SURFACE_REGISTRY_AUDIT LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 OVERALL_STATUS=OK OVERALL_BAD=0 REVIEW_COUNT=22 LEGACY_COUNT=5 HARD_GATE_COUNT=0 CONVERT_PLANNING=19 CONVERT_CLOSURE=4 ARCHIVE_LEGACY=4 FILES_CHANGED=no REGISTRY=/var/lib/homelab-review-surface-registry/review-surfaces-latest.tsv REPORT=/var/lib/homelab-review-surface-registry/audit-latest.md
1101:/var/lib/homelab-health/nextcloud-restore-proof-offhost.txt:1:STATUS=OK TS=20260624T190330Z TYPE=nextcloud-restore-proof-offhost DEST_HOST=pve02 DEST_DIR=/mnt/staging/offhost/edge-restore-proofs-from-edge/nextcloud-aio/nextcloud-aio-restore-proof/20260624T120344Z ITEMS=1 OK=1 ERROR=0 BYTES=2135684 FILES=7 SECRET_PRINTED=NO
1102:/var/lib/homelab-health/recovery-card-seal.txt:1:STATUS=OK TS=2026-07-08T08:34:39Z TYPE=recovery-card-seal BAD=0 WARN=0 MODE=RECOVERY_CARD_FINAL_SEAL RECOVERY_SEAL_OK=YES RECOVERY_CARD_STATUS=OK RECOVERY_CARD_OK=YES FREEZE_STATUS=OK FREEZE_OK=YES FINAL_GLOBAL_GREEN=YES REVIEW_SURFACE_REGISTRY=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T082034Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T082034Z EMERGENCY_JOB=homelab-pve01-emergency-pack-20260708T002739Z VM_DUMPS_JOB=homelab-pve01-vm-local-dumps-20260706T153606Z ENDPOINT_LOCAL_OK=1 ENDPOINT_LAN_OK=1 ENDPOINT_PUBLIC_OK=1 BACKUP_SLA=OK BACKUP_COVERAGE=OK EXTENDED_APPBACKUP=OK VM_BACKUP=OK EXTERNAL_PROBE=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 DIRTY_COUNT=0 CARD=/var/lib/homelab-recovery-card/recovery-card-latest.md SUMMARY=/var/lib/homelab-recovery-card/recovery-summary-latest.tsv REPORT=/var/lib/homelab-recovery-card/recovery-card-seal-latest.md
1103:/var/lib/homelab-health/pve03-failed-unit-cleanup.txt:1:STATUS=OK TS=2026-07-07T12:25:56Z TYPE=pve03-failed-unit-cleanup BAD=0 WARN=0 MODE=RETRY_MANDB_AFTER_SPACE_RECOVERY START_RC=0 MAN_DB_FAILED_LEFT=0 AFTER_FAILED_COUNT=0 ROOT_USAGE_PERCENT=31 STAGING_USAGE_PERCENT=27 REPORT=/var/lib/homelab-pve03-failed-unit-cleanup/man-db-cleanup-latest.md
1104:/var/lib/homelab-health/crypto-portfolio-remove-grt-candidate.txt:1:STATUS=OK TS=2026-07-13T04:20:46Z TYPE=crypto-portfolio-remove-grt-candidate BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=KLINE_CACHE_FILTER_APPLIED_TO_TOP_LEVEL_INSTEAD_OF_ENTRIES EXCLUSIVE_ROOT_CAUSE_PROOF=OK PRODUCTION_CHANGED=NO DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 INPUT_APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd INPUT_WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INPUT_INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 APP_CANDIDATE_SHA256=44df2b25183b7c5d87ff1516ec2d9cd1d169acde3031f7c2aa1fc4e0c7813cd2 WORKER_CANDIDATE_SHA256=174a14b86014535980d200a76d0057110c7af102c66f16c2e21fc3e74bb2e49e INDEX_CANDIDATE_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e MARKET_CANDIDATE_SHA256=eb6c21667d4a42a7a4df8a5d6c3a6808bf69dc5670390e5ee7bfb95687b25074 KLINE_CACHE_CANDIDATE_SHA256=f1a2fe26bedbe457ced26f4efdc605a16aaa1e26e0d5c3b5135533be6292d583 DATABASE_CANDIDATE_SHA256=ae04bf3e7a73568d1fe41813cf1ef224b324276e19a80da1d4b1ca65afcb3d87 CHART_HEALTH_CANDIDATE_SHA256=c16473f16390eeeff4d4eb701db604093564dd78d0be2d30b4cc8ffeeb22364a DATABASE_GRT_AFTER=0|0|0 DATABASE_INTEGRITY=ok CANDIDATE_GRT_REFERENCES=0 KLINE_CACHE_SCHEMA=ENTRIES_OBJECT KLINE_CACHE_GRT_ENTRIES_REMOVED=2 KLINE_CACHE_GRT_ENTRIES_AFTER=0 ACTIVE_COINS=HBAR,POL MARKET_COIN_COUNT=2 EXACT_COINS_SELECTOR_PATCH=OK MOVES_GRID_UNCHANGED=YES READY_FOR_GRT_ISOLATED_DRY_RUN=YES CANDIDATE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-candidate-v3/20260713T042046Z
1105:/var/lib/homelab-health/cluster-admin-stage0-readiness-fix.txt:1:STATUS=OK TS=2026-07-12T23:22:37Z TYPE=cluster-admin-stage0-readiness-fix BAD=0 RESULT=APPLIED FINAL_READINESS=OK FINAL_READINESS_DURATION_SECONDS=17 PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK FAILED_UNITS=0 LIVE_MATCH_DESIRED_STATE=YES READY_FOR_FINAL_SEAL=YES OLD_HEAD=a6e09a8af3604e47d7cc78be4176e6d567e8dbfb NEW_HEAD=1379b4fb4e809e56e9bc16f0e0b19b0a509192d5 CHANGE_SET=/var/lib/homelab-change-sets/stage0-readiness/20260712T232237Z PRODUCTION_CHANGED=READINESS_GATE_LOGIC_ONLY PRODUCTION_SCHEMA_CHANGED=NO CLOUD_WRITE_RUN=NO
1106:/var/lib/homelab-health/crypto-portfolio-remove-grt-recovery-audit.txt:1:STATUS=ERROR TS=2026-07-13T05:39:07Z TYPE=crypto-portfolio-remove-grt-recovery-audit BAD=1 COMMAND_PHYSICAL_LINES=1 ROOT_CAUSE=READINESS_ENDPOINT_MISMATCH RECOVERY_VERIFIED=NO TASK_COMPLETE=NO READY_FOR_GRT_APPLY_SCRIPT_FIX=NO AUDIT_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-recovery-audit-v2/20260713T053907Z
1107:/var/lib/homelab-health/cluster-admin-backup-webpanel-final-seal.txt:1:STATUS=OK TS=2026-07-09T05:24:54Z TYPE=cluster-admin-backup-webpanel-final-seal BAD=0 VMID=180 URL=http://[PRIVATE_IP]:8080/ BACKUP=MAIL_CLOUD_2COPY_VERIFY_OK WEBPANEL=OK WEBPANEL_SYNC=OK OVERALL_HARDGATE=YES FINAL_READINESS_HARDGATE=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-ideal-cluster/cluster-admin-backup-webpanel-final-seal-latest.md
1108:/var/lib/homelab-health/review-surface-registry-seal.txt:1:STATUS=OK TS=2026-07-08T06:36:39Z TYPE=review-surface-registry-seal BAD=0 WARN=0 MODE=REGISTERED_REVIEW_SURFACES_POLICY_AWARE_SEAL PRECHECK=YES REGISTRY_SEAL_OK=YES UPDATED=27 ARCHIVED=27 REGISTERED_COUNT=27 EXACT_WARN_REVIEW_FILES=0 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 FINAL_STATUS=OK FINAL_BAD=0 FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-review-surface-registry/seal-latest.md
1109:/var/lib/homelab-health/mail-cloud-critical.txt:1:STATUS=OK TS=20260721T034025Z TYPE=mail-cloud-critical MODE=recurring REMOTES=2 REMOTE_OK=2 REMOTE_FAIL=0 DEST=app/p0-critical-baseline/20260721T034025Z BUNDLE=p0-critical-baseline-20260721T034025Z.tar.gz SIZE=973275719 SHA256=e1681f82099b2430a46d47c3734063dce8322fd7cec37d4614b3d99ee293b45a RETENTION_KEEP=14 SECRET_PRINTED=no NOTES=none
1110:/var/lib/homelab-health/duty-admin-residual-cleanup.txt:1:STATUS=OK TS=2026-07-08T05:42:29Z TYPE=duty-admin-residual-cleanup BAD=0 WARN=0 MODE=SUPERSEDE_STALE_PVE03_CAPACITY_FORENSICS PRECHECK=YES CLEANUP_OK=YES NEW_PVE03_CAPACITY_STATUS=OK FAILED_LOCAL=0 FAILED_PVE02=0 FAILED_PVE03=0 FAILED_EDGE=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 OVERALL_STATUS=OK OVERALL_BAD=0 WARN_REVIEW_FILES=19 REVIEW_ITEMS_INTENTIONALLY_LEFT=yes REPORT=/var/lib/homelab-duty-admin-residual-cleanup/pve03-capacity-forensics-closure-latest.md
1111:/var/lib/homelab-health/appbackupctl-routing.txt:1:STATUS=OK TS=2026-07-13T00:29:36Z TYPE=appbackupctl-routing BAD=0 RESULT=APPLIED PRODUCTION_CHANGED=BACKUP_CLI_ROUTING_AND_DESIRED_STATE_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO DEFAULT_ROUTE=dockge-stacks CLUSTER_ADMIN_ROUTE=homelab-cluster-admin-incident-engine-backup.service UNSUPPORTED_TARGET_POLICY=REFUSE_RC_2 ROUTE_TEST_DOCKGE=OK ROUTE_TEST_CLUSTER_ADMIN=OK ROUTE_TEST_DEFAULT=OK ROUTE_TEST_UNSUPPORTED=OK FRAMEWORK_CHECK=OK UNSUPPORTED_ROUTE_BACKUP_CREATED=NO LIVE_MATCH_DESIRED_STATE=YES PVE01_FAILED_UNITS=0 OVERALL=OK DUTY_ADMIN=OK STAGE1_CLOSED=YES STAGE2_READY=YES OLD_HEAD=bd5eb856a8d61a5fb83c2bcaf74cb8ac98f075c5 NEW_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 CHANGE_SET=/var/lib/homelab-change-sets/appbackupctl-routing/20260713T002936Z
1112:/var/lib/homelab-health/final-readiness.txt:1:STATUS=WARN TS=2026-07-21T05:31:30Z TYPE=final-readiness BAD=5 PASS=48 FAIL=5 REPORT=/var/lib/homelab-final-readiness/latest.md
1113:/var/lib/homelab-health/cluster-admin-stage3-api-apply.txt:1:STATUS=OK TS=2026-07-13T03:22:35Z TYPE=cluster-admin-stage3-api-apply BAD=0 COMMAND_PHYSICAL_LINES=1 RESULT=APPLIED PRODUCTION_CHANGED=LIVE_APP_AND_DESIRED_STATE PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO SERVICE_RESTARTED=YES CLOUD_WRITE_RUN=NO_REUSE_VERIFIED_PREAPPLY_BACKUP APP_PRECHANGE_SHA256=c2748b8f96df842f491b5d2790790594222549853d0e2336cf60ecd6d4de0020 APP_CANDIDATE_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 LIVE_APP_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 LIVE_MATCH_DESIRED_STATE=YES PREAPPLY_BACKUP_REUSED=YES AST_IDENTICAL_NORMALIZATION=YES EXISTING_ROUTES_BASELINE=OK NEW_GET_ROUTES=7 UNAUTHENTICATED_NEW_ROUTES_401=YES OPENAPI_DOCS_EXPOSURE_CHANGED=NO CORE_DATABASE_STATE=1|1|9|7|1 INCIDENT_ENGINE=ACTIVE VM180_FAILED_UNITS=0 ROLLBACK_CHECKPOINT=VERIFIED OLD_HEAD=8f888e5f2063e49da918b31584c9ea4bcf53af3a NEW_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 READY_FOR_STAGE3_AUTHENTICATED_PRODUCTION_VERIFY=YES STAGE3_CLOSED=NO CHANGE_SET=/var/lib/homelab-change-sets/stage3-api-production-apply/20260713T032235Z
1114:/var/lib/homelab-health/cluster-admin-stage1-foundation-apply.txt:1:STATUS=OK TS=2026-07-13T00:21:21Z TYPE=cluster-admin-stage1-foundation-apply BAD=0 RESULT=APPLIED PRODUCTION_CHANGED=DATABASE_FOUNDATION_AND_DESIRED_STATE PRODUCTION_SCHEMA_CHANGED=PLANNED_ADDITIVE_ONLY SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO MIGRATION_ID=001_entity_source_policy_foundation NEW_TABLES=9 COLLECTOR_RUN_COLUMNS=7 TRIGGER=collector_runs_foundation_enrich_trg FUNCTION=cluster_admin_enrich_collector_run ENTITIES=83 PLACEHOLDERS=21 ENTITY_DEPENDENCIES=26 ENTITY_ALIASES=225 ALIAS_COLLISIONS=3 SOURCES=1 SOURCE_INSTANCES=1 SOURCE_EXPECTATIONS=1 POLICIES=1 POLICY_VERSIONS=1 ORPHANS=0 LIVE_COLLECTOR_COMPATIBILITY=OK API_HEALTH=OK VM180_FAILED_UNITS=0 ROLLBACK_POINT=VERIFIED OLD_HEAD=1379b4fb4e809e56e9bc16f0e0b19b0a509192d5 NEW_HEAD=bd5eb856a8d61a5fb83c2bcaf74cb8ac98f075c5 LIVE_MATCH_DESIRED_STATE=YES CHANGE_SET=/var/lib/homelab-change-sets/stage1-production-apply/20260713T002121Z
1115:/var/lib/homelab-health/homelab-admin-guardrails.txt:1:STATUS=OK TS=2026-07-14T07:39:53Z TYPE=homelab-admin-machine-guardrails BAD=0 RUNNER_VERSION=0.2.0-guardrails LINTER_HOTFIX=2.1 RULES=13 FAIL_FAST=YES ERR_TRAP_REQUIRED=YES UNGUARDED_GREP_COUNT_BLOCKED=YES RUNNER_SHA256=bc76e629e434cc6406ae2f3e7886034a0cedbca43795b6f4197a1ee517c1f322 LINTER_SHA256=1d0cdc2c31adfb7446d9f253539b4004621c6d29a5ea6d1c99bbd9aed34957a2 POLICY_SHA256=ce6d46a01dfe14adc8b2a35a683f2771e4324b04fea611dd4f4bd0458b09d1af DESIRED_STATE_HEAD=69be75caeecaa242567d2d06c735ac5db3a9c048 EVIDENCE_ROOT=/var/lib/homelab-change-sets/homelab-admin-guardrails-v21-install/20260714T073953Z
1116:/var/lib/homelab-health/cluster-admin-stage4i-scope-definition.txt:12:TASK_MODEL=VERSIONED_SCHEMA_V2_FAIL_FAST
1117:/var/lib/homelab-health/cluster-admin-stage4h-final-acceptance.txt:15:ERROR_REGISTER_SHA256=ab8308549db90d235b591af0581580c3c7ba1b287b129cc6111c2eef0a0f9ecc
1118:/var/lib/homelab-health/cluster-admin-stage4h-final-acceptance.txt:35:VM180_FAILED_UNITS=0
1119:/var/lib/homelab-health/cluster-admin-stage4h-final-acceptance.txt:36:PVE01_FAILED_UNITS=0
1120:/var/lib/homelab-health/ideal-cluster-final-seal.txt:1:STATUS=OK TS=2026-07-08T19:49:09Z TYPE=ideal-cluster-final-seal BAD=0 MODE=FINAL_GREEN_SEAL TOP_GATES_OK=YES NODE_LOSS_OK=YES NOISE_SEAL_OK=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 TEST_VMS_CLEANED=YES PROD_VMS_RUNNING=YES DNS_PAIR_OK=YES RESTORE_CHECK_OK=YES DIRTY_COUNT=0 REPORT=/var/lib/homelab-ideal-cluster/final-seal-latest.md
1121:/var/lib/homelab-health/vm160-remote-backup.txt:1:STATUS=FAIL
1122:/var/lib/homelab-health/dockge-final-closure.txt:7:EDGE_STALE_REMAINING_MATCHES=0
1123:/var/lib/homelab-health/disk-space.txt:10:ERRORS=none
1124:/var/lib/homelab-health/remote-failed-unit-cleanup-seal.txt:1:STATUS=OK TS=2026-07-08T05:30:11Z TYPE=remote-failed-unit-cleanup-seal BAD=0 WARN=0 MODE=PAPERLESS_CORE_GUARD_SUPERSEDED_FINAL_SEAL PRECHECK=YES EDGE_APPLY=1 REMOTE_CLEANUP_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 EDGE_GUARD_STATE=inactive EDGE_MARKER_STATUS=OK DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-remote-failed-unit-cleanup/seal-latest.md
1125:/var/lib/homelab-health/special-vm-emergency-non-app-final-seal.txt:1:STATUS=OK TS=2026-07-14T01:15:48Z TYPE=special-vm-emergency-non-app-final-seal BAD=0 WARN=0 FAMILY_COVERAGE=11/11 MATRIX_STATUS=OK JOBS_TOTAL=40 JOBS_OK=36 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 COMPATIBILITY_OUTPUTS=31 REQUIRED_GREEN_JOBS=14 VM_FULL=OK VM_LOCAL_DUMPS=OK EDGE_VM_SOURCE=OK EDGE_VM_CLOUD=OK NEXTCLOUD_VM_SOURCE=homelab_vm_local_dumps_2cloud NEXTCLOUD_VM_CLOUD=OK EMERGENCY_PACK=OK SKLADCHIK_REPORTS_MONITOR=OK GOLDEN_STATE=OK ROUTER_RUNNING_CONFIG=OK MAIL_CLOUD_CRITICAL=OK MAIL_CLOUD_EDGE=OK MAIL_CLOUD_NEXTCLOUD=OK WRITE_MODE=legacy READ_MODE=dual MIGRATE_EXISTING=no SOURCE_BACKUP_RUN=NO CLOUD_OPERATIONS_RUN=NO RETENTION_RUN=NO CLOUD_PURGE_RUN=NO CLUSTER_ADMIN=OK PROBE_OK=4 PROBE_BAD=0 CLUSTER_PASSPORT=OK DUTY_ADMIN=OK OVERALL=OK OVERALL_BAD=0 FINAL_READINESS=OK FINAL_PASS=53 FINAL_FAIL=0 WEBPANEL=OK PUBLIC_OVERALL=OK FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DESIRED_STATE_PRE_SEAL_HEAD=b26f498b55de5d80bcbb4b9fb3a9bccded9f99b4 CURRENT_SPECIAL_VM_EMERGENCY_NON_APP_STAGE_COMPLETE=YES EVIDENCE_ROOT=/var/lib/homelab-change-sets/special-vm-emergency-non-app-final-seal/20260714T011514Z
1126:/var/lib/homelab-health/independent-external-runner.txt:1:STATUS=OK TS=2026-07-08T09:17:06Z TYPE=independent-external-runner BAD=0 WARN=0 MODE=NO_DEPLOY_SEAL NO_DEPLOY_SEAL_OK=YES AUDIT_STATUS=OK AUDIT_OK=YES INDEPENDENT_RUNNER_READY=NO DEPLOY_DECISION=NO_DEPLOY CURRENT_INDEPENDENT_RUNNER_AVAILABLE=no TRUSTED_NETBIRD_IDENTITY=edge-vm PUBLIC_VPS_DECISION=REJECTED_HOSTKEY_MISMATCH PUBLIC_VPS_SSH_OK=0 PUBLIC_VPS_REJECTED_HOSTKEY=YES EDGE_LAN_SSH_OK=2 EDGE_NETBIRD_SSH_OK=2 EDGE_DNS_SSH_OK=1 PUBLIC_OVERALL_OK=1 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 FINAL_STATUS=OK FINAL_FAIL=0 CLOSEOUT_OK=YES RECOVERY_SEAL_OK=YES RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T090905Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T090905Z DIRTY_COUNT=0 REPORT=/var/lib/homelab-independent-external-runner/no-deploy-seal-latest.md
1127:/var/lib/homelab-health/node-exporter-restore.txt:1:STATUS=OK TS=2026-07-09T08:13:40Z TYPE=node-exporter-restore JOB=homelab-pve01-app-node-exporter-20260709T081340Z RESTORE=/tmp/node-exporter-restore-20260709T081340Z RESTORE_CHECK=PACKAGE_OK PVE01_TEXTFILE_SCRAPE_ERROR_AFTER=0 NOTE=service-readiness-restore-proof
1128:/var/lib/homelab-health/cluster-admin-stage2-apply.txt:1:STATUS=OK TS=2026-07-13T01:38:37Z TYPE=cluster-admin-stage2-apply BAD=0 RESULT=APPLIED PRODUCTION_CHANGED=CANONICAL_SYNC_FUNCTION_COLLECTOR_MIGRATION_ARTIFACTS_AND_DESIRED_STATE PRODUCTION_SCHEMA_CHANGED=PLANNED_FUNCTION_AND_MIGRATION_REGISTRY_ONLY INCIDENT_ENGINE_RESTARTED=NO COLLECTOR_ONESHOT_RUN=YES COLLECTOR_TIMER_PAUSED_AND_RESUMED=YES CLOUD_WRITE_RUN=NO MIGRATION_ID=002_canonical_foundation_sync FUNCTION=cluster_admin_sync_foundation PREFLIGHT_NATURAL_KEY_DRIFT_ALLOWED=YES PREFLIGHT_NATURAL_KEY_MISSING_BEFORE=1 PREFLIGHT_NATURAL_KEY_STALE_BEFORE=1 PREFLIGHT_NATURAL_KEY_MISSING_AFTER=0 PREFLIGHT_NATURAL_KEY_STALE_AFTER=0 PREFLIGHT_EPHEMERAL_ID_DRIFT_ALLOWED=YES PREFLIGHT_EPHEMERAL_ID_DRIFT_BEFORE=25 PREFLIGHT_EPHEMERAL_ID_DRIFT_AFTER=0 PREFLIGHT_TRANSACTION_ROLLBACK=OK COMMITTED_SYNC=YES PARTIAL_ROLLBACK_AFTER_COMMIT=NOT_EXECUTED APPLIED_STATE_PRESERVED_POLICY=ACTIVE SAME_TRANSACTION=YES FIXED_TOTAL_ENTITIES_REQUIRED=NO FIXED_TOTAL_ALIASES_REQUIRED=NO FOUNDATION_COUNTS=85|21|2|229|26|0 DYNAMIC_INVARIANTS=0|0|21|21|0|0|0|0|0|0|0 ACTIVE_CURRENT_SET_CONVERGENCE=OK NATURAL_KEY_CONVERGENCE=OK EPHEMERAL_ID_REFRESH=OK RETIRED_PLACEHOLDERS_ALLOWED=YES ORPHANS=0 LIVE_COLLECTOR_RUN_ID=2992 LIVE_COLLECTOR_STATUS=OK SOURCE_INSTANCE_UPDATE=OK OBSERVER=OK FULL_OBSERVER=OK RESTRICTED_PROBES=OK FINAL_READINESS=OK CLUSTER_PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK API_HEALTH=OK VM180_FAILED_UNITS=0 PVE01_FAILED_UNITS=0 ROLLBACK_POINT=VERIFIED LIVE_MATCH_DESIRED_STATE=YES OLD_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 NEW_HEAD=8f888e5f2063e49da918b31584c9ea4bcf53af3a READY_FOR_POSTAPPLY_SEAL=YES STAGE2_CLOSED=NO CHANGE_SET=/var/lib/homelab-change-sets/stage2-production-apply/20260713T013837Z
1129:/var/lib/homelab-health/cluster-admin-node-online-green-dot-fix.txt:1:STATUS=OK TS=2026-07-14T01:47:22Z TYPE=cluster-admin-node-online-green-dot-fix BAD=0 WARN=0 ROOT_CAUSE=STATUS_KIND_MISSING_ONLINE_AND_OFFLINE PRE_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 POST_SHA256=a068c1824943af182acbaebcd3ff89e84203a3fde17ccb8051a4680e37d9f5e5 NODE_STATUS=online NODE_COUNT=3 ONLINE_CLASSIFICATION=ok ONLINE_BADGE_CLASS=state-ok OFFLINE_CLASSIFICATION=bad OFFLINE_BADGE_CLASS=state-bad DATABASE_CHANGED=NO COLLECTOR_CHANGED=NO SERVICE_RESULT=success EXEC_MAIN_STATUS=0 API_HEALTH=OK BACKUP_JOB=homelab-pve01-app-cluster-admin-incident-engine-20260714T014645Z BACKUP_STATUS=success BACKUP_TWO_CLOUDS=OK RESTORE_CHECK=OK_BOTH_CLOUDS REMOTE_ROLLBACK=/var/lib/cluster-admin-incident-engine/change-sets/node-online-green-dot-20260714T014640Z/app.py.before EVIDENCE_ROOT=/var/lib/homelab-change-sets/cluster-admin-node-online-green-dot-fix/20260714T014640Z
1130:/var/lib/homelab-health/crypto-portfolio-ui-preapply-backup.txt:1:STATUS=OK TS=2026-07-13T07:42:53Z TYPE=crypto-portfolio-ui-preapply-backup BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=INLINE_PYTHON_QUOTE_ERROR_IN_NONESSENTIAL_HELPER_PROOF BACKUP_SERVICE_RERUN=NO MANUAL_REHYDRATE_RERUN=NO CANONICAL_RESTORE_CHECK_RERUN=YES_READ_ONLY PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CONTAINER_RECREATED=NO CLOUD_WRITE_RUN=NO_READ_ONLY_RESTORE_CHECK TARGET_HOST=core-apps.gram1.ru INPUT_INDEX_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e CANDIDATE_INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T071127Z BACKUP_STATUS=success BACKUP_SCHEMA=homelab-app-backup-manifest-v2-flat BACKUP_LAYOUT=flat-v1 BACKUP_PATH_REL=hb/a/cp-260713-071127 BACKUP_DATABASE_INTEGRITY=ok BACKUP_FILE_COUNT=19 BACKUP_ARCHIVE_SIZE_BYTES=136632 BACKUP_ARCHIVE_SHA256=b29eb55820ec46c474f51e995e92e371251b6df717ac98182d807d1dbf8473f5 BACKUP_LOCAL_STAGING_DELETED=YES CANONICAL_TWO_CLOUD_RESTORE_CHECK=OK APPBACKUPCTL_RESTORE_CHECK=OK_BOTH_CLOUDS ROLLBACK_PATH=/var/lib/homelab-incidents/crypto-ui-center-alert-currency-prechange-20260713T071125Z ROLLBACK_MANIFEST_SHA256=0ef56c8f021f549777131ffe878cd5a263bdfb6f1dfb8bb080cefdd554573aca ROLLBACK_INDEX_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e PRODUCTION_UNCHANGED=YES PRODUCTION_HEALTH=200 VM170_FAILED_UNITS=0 PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES CHART_HEALTH_TIMER=ACTIVE BACKUP_TIMER=ACTIVE DESIRED_STATE_HEAD=80763a6c62b94cde4f3293f3b529980f76919d25 DESIRED_STATE_CLEAN=YES DESIRED_STATE_REMOTE_COUNT=0 READY_FOR_UI_PRODUCTION_APPLY=YES PREAPPLY_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-ui-preapply-backup-canonical-close/20260713T074253Z
1131:/var/lib/homelab-health/router-hard-power.txt:1:STATUS=ERROR TS=2026-07-16T06:20:34Z MODE=cycle REMOTE=[PRIVATE_IP] RC=74
1132:/var/lib/homelab-health/operator-handoff-pack.txt:1:STATUS=OK TS=2026-07-08T09:42:39Z TYPE=operator-handoff-pack BAD=0 WARN=0 MODE=FINAL_OPERATOR_HANDOFF_PACK HANDOFF_OK=YES PACK_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T093758Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T093758Z RECOVERY_CARD=YES FREEZE_PACK=YES FINAL_CLOSEOUT=YES INDEPENDENT_RUNNER_NO_DEPLOY=YES DESIRED_STATE_REMOTE_GIT=YES GIT_REMOTE_CHANGED=NO GIT_PUSH_DONE=NO DIRTY_COUNT=0 PACK=/var/lib/homelab-operator-handoff/operator-handoff-pack-20260708T094239Z.tar.gz README=/var/lib/homelab-operator-handoff/HANDOFF-latest.md STATUS_TSV=/var/lib/homelab-operator-handoff/handoff-status-latest.tsv NEXT=/var/lib/homelab-operator-handoff/NEXT-ACTIONS-latest.md
1133:/var/lib/homelab-health/pve01-failed-units.txt:1:STATUS=OK TS=20260624T180727Z TYPE=pve01-failed-units FAILED_COUNT=0 ACTION=RESET_FAILED_ONLY SECRET_PRINTED=REDACTED
1134:/var/lib/homelab-health/final-global-green-audit.txt:1:STATUS=OK TS=2026-07-08T07:20:01Z TYPE=final-global-green-audit BAD=0 WARN=0 MODE=SELF_REFERENCE_CLOSED_FINAL_GLOBAL_GREEN_SEAL AUDIT_OK=YES PRE_STATUS=WARN SELF_REFERENCE_CLOSED=yes LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 EXACT_WARN_REVIEW_FILES=0 LOCAL_OVERALL_OK=1 LAN_OVERALL_OK=1 PUBLIC_OVERALL_OK=1 EXTERNAL_PROBE_STATUS=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T064444Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T064444Z DIRTY_COUNT=0 REPORT=/var/lib/homelab-final-global-green-audit/final-seal-latest.md
1135:/var/lib/homelab-health/remote-failed-unit-audit.txt:1:STATUS=OK TS=2026-07-08T05:20:16Z TYPE=remote-failed-unit-audit BAD=0 WARN=1 MODE=READ_ONLY_REMOTE_FAILED_UNIT_AUDIT LOCAL_FAILED_COUNT=1 REMOTE_FAILED_COUNT=8 CONNECT_ERRORS=0 JOURNAL_FAIL_HINTS=194 OVERALL_BAD=0 FINAL_FAIL=0 DUTY_V2_GATED_NONOK=0 RESET_FAILED_DONE=no REPORT=/var/lib/homelab-remote-failed-unit-cleanup/audit-latest.md
1136:/var/lib/homelab-health/legacy-monitoring-backups-migrated.txt:1:STATUS=OK TS=2026-07-10T20:00:03Z TYPE=legacy-monitoring-backups-migrated BAD=0 APPS=6 RESTORE_FAILURES=0 ACTIVE_LEGACY_STATES=0 OFFHOST=OK RESTORE=OK
1137:/var/lib/homelab-health/cluster-admin-stage4h-collector-patch-candidate.txt:1:STATUS=OK TS=2026-07-14T17:41:32Z TYPE=cluster-admin-stage4h-collector-patch-candidate FAILED_ATTEMPT_TAIL=CLOSED CANDIDATE_SUBSTEP=CLOSED STAGE4H_STATUS=OPEN PRODUCTION_CHANGE=NO UNRESOLVED_TAIL_COUNT=0 BLOCKER_COUNT=0 NEXT_ACTION=TEMPORARY_DATABASE_MIGRATION_AND_COLLECTOR_ACCEPTANCE_HARNESS CANDIDATE_SHA256=4204fe568d46fec84f643af974c82770cb5d5b08d3d8ceb4aad96cca57abc909 REPORT_SHA256=ecfdaab5438e1fc3bc7a90e08ec68c71d06a6c6a91a430dc1ea7b6a74b7699a9
1138:/var/lib/homelab-health/cluster-admin-stage2-final-seal.txt:1:STATUS=OK TS=2026-07-13T01:45:29Z TYPE=cluster-admin-stage2-final-seal BAD=0 STAGE1_CLOSED=YES STAGE2_APPLY=OK PRODUCTION_CHANGED=BACKUP_HEALTH_AND_SEAL_ARTIFACTS_ONLY PRODUCTION_SCHEMA_CHANGED=NO_ADDITIONAL_CHANGE_DURING_SEAL INCIDENT_ENGINE_RESTARTED=NO MIGRATION_ID=002_canonical_foundation_sync FUNCTION=cluster_admin_sync_foundation COMMITTED_SYNC=YES MULTICYCLE_COLLECTOR_TEST=OK MULTICYCLE_NEW_RUNS=2 MULTICYCLE_GOOD_RUNS=2 ACTIVE_CURRENT_SET_CONVERGENCE=OK NATURAL_KEY_CONVERGENCE=OK EPHEMERAL_ID_REFRESH=OK RETIRED_PLACEHOLDERS_ALLOWED=YES FOUNDATION_COUNTS=85|21|2|229|26|0 DYNAMIC_INVARIANTS=0|0|21|21|0|0|0|0|0|0|0 ORPHANS=0 POSTAPPLY_DIRECT_SNAPSHOT=VERIFIED POSTAPPLY_RESTORE_TEST=OK POSTAPPLY_APPBACKUP=VERIFIED POSTAPPLY_APPBACKUP_JOB=homelab-pve01-app-cluster-admin-incident-engine-20260713T014757Z LIVE_MATCH_DESIRED_STATE=YES DESIRED_STATE_HEAD=8f888e5f2063e49da918b31584c9ea4bcf53af3a OBSERVER=OK FULL_OBSERVER=OK RESTRICTED_PROBES=OK FINAL_READINESS=OK CLUSTER_PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK VM180_FAILED_UNITS=0 PVE01_FAILED_UNITS=0 CURRENT_HEALTH_RECHECKED=YES STAGE2_CLOSED=YES REPORT=/var/lib/homelab-cluster-admin-stage2/stage2-final-seal-latest.txt REPORT_SHA256=32d4d37d82aa25194d789083d0e7ad2e9052ee38a72df5312b711226898cf1a6 BLOCKERS=/var/lib/homelab-cluster-admin-stage2/20260713T014529Z/blockers.txt
1139:/var/lib/homelab-health/remote-failed-unit-plan.txt:1:STATUS=OK TS=2026-07-08T05:22:47Z TYPE=remote-failed-unit-plan BAD=0 WARN=1 MODE=READ_ONLY_CLEANUP_PLAN LOCAL_FAILED_COUNT=1 REMOTE_FAILED_COUNT=7 LOCAL_NOOP_COUNT=4 REMOTE_RESET_ONLY_COUNT=1 REMOTE_NOOP_COUNT=5 REMOTE_DEFER_COUNT=1 OVERALL_BAD=0 FINAL_FAIL=0 RESET_FAILED_DONE=no PLAN=/var/lib/homelab-remote-failed-unit-cleanup/cleanup-plan-latest.tsv REPORT=/var/lib/homelab-remote-failed-unit-cleanup/plan-latest.md
1140:/var/lib/homelab-health/review-surface-registry-apply.txt:1:STATUS=OK TS=2026-07-08T06:36:39Z TYPE=review-surface-registry-apply BAD=0 WARN=0 MODE=POLICY_AWARE_DUTY_ADMIN_REPAIR_SELF_CLOSED PRECHECK=YES APPLY_OK=YES REPAIR_OK=YES UPDATED=27 ARCHIVED=27 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW_WAS=1 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_BAD=0 SELF_REFERENCE_CLOSED=YES REPORT=/var/lib/homelab-review-surface-registry/seal-latest.md
1141:/var/lib/homelab-health/private-vpn-hosts.txt.bak-ok-wrapper-20260624T132705Z:16:STATUS=ERROR
1142:/var/lib/homelab-health/cluster-admin-strict-final-seal.txt:1:STATUS=OK TS=2026-07-09T04:49:23Z TYPE=cluster-admin-strict-final-seal BAD=0 VMID=180 IP=[PRIVATE_IP] MODE=DEDICATED_CLUSTER_ADMIN_HARDGATED OBSERVER=OK RESTRICTED_PROBES=OK FULL_OBSERVER=OK OVERALL_HARDGATE=YES FINAL_READINESS_HARDGATE=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-ideal-cluster/cluster-admin-strict-final-seal-latest.md
1143:/var/lib/homelab-health/paperless-restore.txt:1:STATUS=STALE
1144:/var/lib/homelab-health/paperless-restore.txt:20:FRESHNESS=STALE
1145:/var/lib/homelab-health/ideal-cluster-noise-seal.txt:1:STATUS=OK TS=2026-07-08T17:41:49Z TYPE=ideal-cluster-noise-seal BAD=0 MODE=POLICY_AWARE_NON_STATUS_ARTIFACT_SEAL_SELF_REFERENCE_CLOSED SELF_REFERENCE_CLOSED=YES NON_STATUS_TOTAL=8 NON_STATUS_OK_INFO=8 NON_STATUS_UNKNOWN=0 WARNBAD_ACCEPTED_POLICY_INFO=13 WARNBAD_NEEDS_REVIEW=0 TOP_GATES_OK=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-ideal-cluster/noise-seal-latest.md
1146:/var/lib/homelab-health/crypto-portfolio-chart-health.txt:1:STATUS=OK TS=2026-07-21T13:18:29Z TYPE=crypto-portfolio-chart-health BAD=0 LIBRARY_OK=true DISK_CACHE_PRESENT=true SYMBOLS=2 STALE_COUNT=2
1147:/var/lib/homelab-health/duty-admin-v2.txt:1:STATUS=OK TS=2026-07-21T13:16:33Z TYPE=duty-admin-v2 BAD=7 WARN=0 MODE=LOCAL_LONG_RUNNING_ANALYZER_POLICY_AWARE SEPARATE_VM=NO TOTAL_HEALTH_INPUTS=497 GATED_OK=45 GATED_NONOK=7 UNGATED_ERROR_REVIEW=2 UNGATED_WARN_REVIEW=3 UNGATED_REVIEW_ITEM=0 LEGACY_OR_PHASE_REVIEW=0 REPORT=/var/lib/homelab-duty-admin-v2/latest.md
1148:/var/lib/homelab-health/cluster-admin-stage3-final-seal.txt:1:STATUS=OK TS=2026-07-13T03:46:30Z TYPE=cluster-admin-stage3-final-seal BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=HEALTH_REFRESH_AND_SEAL_ARTIFACTS_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO LIVE_APPLICATION_CODE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO STAGE2_CLOSED=YES STAGE3_APPLY=OK AUTHENTICATED_PRODUCTION_VERIFY=OK AUTHENTICATED_PRODUCTION_TESTS=25 APP_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 LIVE_MATCH_DESIRED_STATE=YES DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 NEW_GET_ROUTES=7 DATABASE_TRANSACTION_MODE=BEGIN_READ_ONLY OPENAPI_DOCS_EXPOSURE=PRESERVED_DISABLED POSTAPPLY_APPBACKUP=VERIFIED POSTAPPLY_APPBACKUP_JOB=homelab-pve01-app-cluster-admin-incident-engine-20260713T034400Z POSTAPPLY_APPBACKUP_RESTORE_CHECK=OK COLLECTOR_STABILITY=OK LATEST_COLLECTOR_RUN_ID=3114 CANONICAL_CONSISTENCY=0|0|0|0|0 OBSERVER=OK FULL_OBSERVER=OK RESTRICTED_PROBES=OK FINAL_READINESS=OK CLUSTER_PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK VM180_FAILED_UNITS=0 PVE01_FAILED_UNITS=0 CURRENT_HEALTH_RECHECKED=YES STAGE3_CLOSED=YES REPORT=/var/lib/homelab-cluster-admin-stage3/stage3-final-seal-latest.txt REPORT_SHA256=6f67545cd7f0a75b87146c1df74c61c3b88e142f9e53a96b083ec27209e6d265
1149:/var/lib/homelab-health/hapusya-site-production-browser.txt:1:STATUS=OK TS=2026-07-13T11:48:15Z TYPE=hapusya-site-production-browser BAD=0 URL=https://hapusya.ru/ BROWSER_RUNS=5 BROWSER_ERRORS=0 DESKTOP_VIDEO=OK MOBILE_VIDEO=OK STATIC_FALLBACK=OK EVIDENCE_ROOT=/var/lib/homelab-change-sets/hapusya-site-production-browser-v2/20260713T114810Z MANIFEST_SHA256=9f3cdf3063f8724d1d1c1ad619f47d6f98d434c3422a74903585146858a8dfac ROUTE_SHA256=02d65cb0082a04cae282126cc829a002f171e863291c18ba04d788dc2d6282ca UNIT_SHA256=b3b07137b25ac0a0b4bdadadad654cf209387ade86a2ef2cdca58e8a7c0cc212 RELATIVE_TREE_SHA256=9dadf21da041bc4d6ee4a42f5b3ac7719c16bcc39052de32af895c6cd28286cb PRODUCTION_CHANGED=YES CLOSURE_PENDING=POSTCHANGE_BACKUP_RETENTION_DESIRED_STATE_FINAL_SEAL
1150:/var/lib/homelab-health/backup-matrix-authoritative.txt:1:STATUS=FAIL
1151:/var/lib/homelab-health/backup-matrix-desired-state-persistence.txt:1:STATUS=OK TS=2026-07-13T17:40:37Z TYPE=backup-matrix-desired-state-persistence BAD=0 GENERATED_HEAD=af26ec3c4829b1330d49656205ac580ce1706263 GENERATED_PARENT=25568fab24c0594418b3676dc511c8234b4e1ae5 GENERATOR_SHA256=19da7f049b4f7f7286faf292c5419c37cc1e1768c2d8fe36b257ab0cf9c18874 MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 DESIRED_MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 MATRIX_STATUS=OK JOBS_BAD=0 FAILED_UNITS=0 BACKUP_RUN=NO RESTORE_RUN=NO CLOUD_WRITE_RUN=NO RETENTION_RUN=NO ROLLBACK=/var/lib/homelab-incidents/backup-matrix-desired-state-persistence-20260713T174036Z
1152:/var/lib/homelab-health/duty-admin-residual-audit.txt:1:STATUS=OK TS=2026-07-08T05:40:37Z TYPE=duty-admin-residual-audit BAD=0 WARN=3 MODE=READ_ONLY_RESIDUAL_NOISE_AUDIT LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=1 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 OVERALL_STATUS=OK OVERALL_BAD=0 FINAL_FAIL=0 WARN_REVIEW_FILES=19 HARD_GATE_WARN_MATCHES=0 REPORT=/var/lib/homelab-duty-admin-residual-cleanup/audit-latest.md
1153:/var/lib/homelab-health/ungated-health-backlog.txt:1:STATUS=OK TS=2026-07-07T07:45:01Z TYPE=ungated-health-backlog BAD=0 WARN=1 MODE=POST_PVE03_ROOT_CLEANUP_TRIAGE TOTAL=8 SUPERSEDED=4 STALE=2 POLICY_GAP_NEXT_STAGE=1 ACTIONABLE_CAPACITY=1 PVE03_ROOT_USAGE_PERCENT=31 PVE03_STAGING_USAGE_PERCENT=83 REPORT=/var/lib/homelab-ungated-health-backlog/latest.md
1154:/var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-final-seal.txt:1:STATUS=OK TS=2026-07-13T08:37:54Z TYPE=crypto-portfolio-ui-center-alert-currency-final-seal BAD=0 DANGLING_IMAGE_REFERENCE_GATE=OK DESIRED_STATE_SYNC_ARTIFACT_GATE=OK APP=crypto-portfolio PRODUCTION_CHANGED=NO_FINAL_SEAL_ONLY APP_SHA256=44df2b25183b7c5d87ff1516ec2d9cd1d169acde3031f7c2aa1fc4e0c7813cd2 WORKER_SHA256=174a14b86014535980d200a76d0057110c7af102c66f16c2e21fc3e74bb2e49e INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb LIBRARY_SHA256=c0992580867c4912cc9385b3c2728315bcc1a76c7f1087dca908430fccdf31d7 ACTIVE_COINS=HBAR,POL GRT_REMOVED=YES GRT_ROWS=0|0|0 GRT_SOURCE_REFERENCES=0 GRT_CACHE_ENTRIES=0 SHARP_MOVES_CENTERED=YES ALERT_CURRENCY_USD_RUB=OK BROWSER_SMOKE=OK BROWSER_RUNS=2 BROWSER_ERRORS=0 BROWSER_RESULT_SHA256=b566f492cd80c8f99bc6b24cbef6d5f326c04f4d7168359d05eb1f189589e733 SCREENSHOT_SHA256=00b2853bee9c880146d0dd75fac60feb7e46d4c129ee06754047ef30d90bd4ba BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T075801Z TWO_CLOUDS=OK RESTORE_CHECK=OK DATABASE_INTEGRITY=ok RETENTION=OK ROLLBACK_COUNT=3 DANGLING_IMAGES=2 REFERENCED_DANGLING_IMAGES=2 UNREFERENCED_DANGLING_IMAGES=0 PRUNED_UNREFERENCED_DANGLING_IMAGES=0 CHART_HEALTH=OK PUBLIC=OK MONITOR_TIMER=active BACKUP_TIMER=active CLOSURE_DESIRED_STATE_HEAD=2999910549eaff9b9431b9c656d318340eb95cbd DESIRED_STATE=OK PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES ALL_CRYPTO_PORTFOLIO_UI_TAILS_CLOSED=YES
1155:/var/lib/homelab-health/crypto-portfolio-remove-grt-runtime-verifier-prepared.txt:1:STATUS=OK TS=2026-07-13T04:35:29Z TYPE=crypto-portfolio-remove-grt-runtime-verifier-prepared BAD=0 COMMAND_PHYSICAL_LINES=1 VERIFIER_REVISION=V3 DEEP_ANALYSIS_RESULT=DIRECT_ROUTE_JSONRESPONSE_CONTRACT_CORRECTED FAILURE_CLASS=VERIFIER_ONLY PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO CANDIDATE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CONTAINER_STARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 INPUT_APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd INPUT_WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INPUT_INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 CANDIDATE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-candidate-v3/20260713T042046Z RUNTIME_POLICY=CONTAINER_PYTHON_ONLY PYTHONPATH_APP_INJECTED=YES ROUTE_RETURN_DECODER=DICT_OR_RESPONSE_BODY STATE_RETURN_DECODED=YES CHART_HEALTH_RETURN_DECODED=YES FETCH_KLINES_RETURN_DECODED=YES PROTECTED_KLINES_CODES=401_OR_403 LIVE_PRODUCTION_DATA_MOUNTED=NO ISOLATED_PORT=18030 FAILED_RUN_CLEANUP=VERIFIED REMOTE_RUNTIME_SCRIPT_SHA256=b60290b08f6bdb9801f4ccadbeadc4268399426996fd271527830fd399055448 VERIFIER_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-runtime-verifier-v3/20260713T043529Z READY_FOR_GRT_ISOLATED_RUNTIME_DRY_RUN_RUN=YES
1156:/var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-candidate.txt:1:STATUS=OK TS=2026-07-13T06:11:23Z TYPE=crypto-portfolio-ui-center-alert-currency-candidate BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=EXPECTED_DIFF_RC_1_TRIGGERED_SET_E_PIPEFAIL PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=80763a6c62b94cde4f3293f3b529980f76919d25 INPUT_INDEX_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e CANDIDATE_INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb BUILDER_DOUBLE_CHECK=OK CANDIDATE_DOUBLE_CHECK=OK SHARP_MOVES_POSITION=CENTERED SHARP_MOVES_MAX_WIDTH_PX=860 SHARP_MOVES_COLUMNS_DESKTOP=2 SHARP_MOVES_COLUMNS_MOBILE=1 ALERT_RENDERER=V5 ALERT_CURRENCY_SOURCE=TOP_USD_RUB_SWITCH ALERT_EXCHANGE_RATE_SOURCE=STATE_MARKET_USD_RUB ALERT_PRICE_THRESHOLD_CONVERTED=YES ALERT_CAP_THRESHOLD_CONVERTED=YES ALERT_SALE_VALUE_CONVERTED=YES CURRENCY_SWITCH_REDRAW=YES PUMP_DUMP_PERCENT_UNCHANGED=YES FIXED_USD_ALERT_FORMATTER=ABSENT RENDER_MOVES_COUNT=1 ALERT_RENDERER_MARKER_COUNT=1 GRT_REFERENCES=0 PRODUCTION_INDEX_UNCHANGED=YES PRODUCTION_HEALTH=200 READY_FOR_UI_ISOLATED_BROWSER_SMOKE=YES CANDIDATE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-ui-center-alert-currency-candidate-v4/20260713T061123Z
1157:/var/lib/homelab-health/cluster-admin-stage2-design.txt:1:STATUS=OK TS=2026-07-13T00:35:14Z TYPE=cluster-admin-stage2-design BAD=0 STAGE1_CLOSED=YES CAPTURE_REUSED=YES PRODUCTION_CHANGED=EVIDENCE_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO SERVICE_RESTARTED=NO FOUNDATION_STATE=9|7|1|1 EXPECTED_ENTITIES=62 MANAGED_ENTITIES=62 PLACEHOLDERS=21 MISSING_ENTITIES=0 STALE_ENTITIES=0 LEGACY_DEPENDENCIES=26 TYPED_DEPENDENCIES=26 DEPENDENCY_ID_MATCHES=0 DEPENDENCY_NATURAL_KEY_MATCHES=26 DEPENDENCY_NATURAL_KEY_MISSING=0 DEPENDENCY_NATURAL_KEY_STALE=0 EPHEMERAL_ID_MISSING=26 EPHEMERAL_ID_STALE=26 ROOT_CAUSE=LEGACY_DEPENDENCIES_DELETE_REINSERT_EACH_RUN CANONICAL_DEPENDENCY_IDENTITY=PARENT_CHILD_RELATION MIGRATION_ID=002_canonical_foundation_sync DATABASE_FUNCTION=cluster_admin_sync_foundation COLLECTOR_CALL_REQUIRED=YES SAME_TRANSACTION_REQUIRED=YES LEGACY_TABLES_PRESERVED=YES AUTOMATIC_REMEDIATION_ACTIONS=NO STAGE2_READY_FOR_DRY_RUN=YES DESIRED_STATE_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 CHANGE_SET=/var/lib/homelab-change-sets/stage2-design/20260713T003514Z CONTRACT=/var/lib/homelab-change-sets/stage2-design/20260713T003514Z/stage2-contract.json REPORT=/var/lib/homelab-change-sets/stage2-design/20260713T003514Z/stage2-design-report.txt
1158:/var/lib/homelab-health/final-green-freeze-seal.txt:1:STATUS=OK TS=2026-07-08T08:13:08Z TYPE=final-green-freeze-seal BAD=0 WARN=0 MODE=FINAL_FREEZE_EVIDENCE_SEAL FREEZE_SEAL_OK=YES AUDIT_STATUS=OK AUDIT_FREEZE_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 EXACT_WARN_REVIEW_FILES=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T072711Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T072711Z DIRTY_COUNT=0 EVIDENCE=/var/lib/homelab-final-green-freeze/freeze-evidence-latest.tsv REPORT=/var/lib/homelab-final-green-freeze/final-seal-latest.md
1159:/var/lib/homelab-health/cluster-admin-stage2-dry-run.txt:1:STATUS=OK TS=2026-07-13T00:39:40Z TYPE=cluster-admin-stage2-dry-run BAD=0 PRODUCTION_CHANGED=TEST_DATABASE_AND_CANDIDATE_ARTIFACTS_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO MIGRATION_ID=002_canonical_foundation_sync DATABASE_FUNCTION=cluster_admin_sync_foundation MIGRATION_APPLY_TEST=OK MIGRATION_IDEMPOTENCE_TEST=OK NATURAL_KEY_CONVERGENCE=OK EPHEMERAL_ID_REFRESH=OK ENTITY_ADD_SYNC=OK ENTITY_RETIRE_SYNC=OK PLACEHOLDER_ADD_SYNC=OK PLACEHOLDER_RETIRE_SYNC=OK STALE_TYPED_EDGE_DELETE=OK SOURCE_INSTANCE_UPDATE=OK COLLECTOR_CANDIDATE_SYNTAX=OK COLLECTOR_CALL_BEFORE_COMMIT=OK ROLLBACK_TEST=OK TEST_DATABASE_DROPPED=YES PRODUCTION_UNCHANGED=YES READY_FOR_PREAPPLY_BACKUP=YES DESIRED_STATE_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 CHANGE_SET=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z MIGRATION=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z/002_canonical_foundation_sync.sql ROLLBACK=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z/002_canonical_foundation_sync.rollback.sql COLLECTOR=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z/collector.py.candidate PLAN=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z/stage2-migration-plan.json RESULT=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z/dry-run-result.txt
1160:/var/lib/homelab-health/cluster-admin-stage3-production-verify.txt:1:STATUS=OK TS=2026-07-13T03:28:52Z TYPE=cluster-admin-stage3-production-verify BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=VERIFICATION_EVIDENCE_ONLY PRODUCTION_DATABASE_CHANGED=NO LIVE_APPLICATION_CODE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO PREVIOUS_PERMISSION_FAILURE=CORRECTED VERIFIER_OWNER=CLUSTERADMIN VERIFIER_MODE=0700 LIVE_APP_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 AUTHENTICATED_PRODUCTION_TESTS=25 AUTHENTICATED_PRODUCTION_FAILED=0 LEGACY_AUTHENTICATED_SUMMARY=OK NEW_GET_ROUTES_AUTHENTICATED=OK NEW_GET_ROUTES_UNAUTHENTICATED_401=OK ENTITY_LIST=OK ENTITY_DETAIL=OK ENTITY_DEPENDENCIES=OK DEPENDENCIES=OK SOURCES=OK POLICIES=OK COLLECTOR_RUNS=OK VALIDATION_ERRORS=OK FORBIDDEN_RESPONSE_FIELDS=ABSENT OPENAPI_DOCS_EXPOSURE_CHANGED=NO SECURITY_HEADERS=OK PRODUCTION_RUNTIME_HEALTH=OK REMOTE_VERIFIER_CLEANUP=OK READY_FOR_STAGE3_POSTAPPLY_SEAL=YES STAGE3_CLOSED=NO RESULT=/var/lib/homelab-change-sets/stage3-api-production-verify-run/20260713T032852Z/production-result.json CHANGE_SET=/var/lib/homelab-change-sets/stage3-api-production-verify-run/20260713T032852Z
1161:/var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-browser-smoke.txt:1:STATUS=OK TS=2026-07-13T07:04:45Z TYPE=crypto-portfolio-ui-center-alert-currency-browser-smoke BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=UNPRIVILEGED_REMOTE_RM_AFTER_SUDO_HARNESS PRODUCTION_CHANGED=NO_BROWSER_EVIDENCE_FINALIZATION_ONLY PRODUCTION_DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru CANDIDATE_INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb CANDIDATE_STATIC_GRT_DOUBLE_CHECK=OK BROWSER_ENGINE=CHROMIUM BROWSER_RUNS=2 BROWSER_RUNS_IDENTICAL=YES MOVES=2 SHARP_MOVES_CENTERED=YES SHARP_MOVES_MAX_WIDTH_OK=YES SHARP_MOVES_COLUMNS=2 ALERT_ROWS=2 USD_ALERT_RENDER=OK RUB_ALERT_RENDER=OK TOP_CURRENCY_SWITCH_REDRAW=OK RUB_PRICE_THRESHOLD=OK RUB_CAP_THRESHOLD=OK RUB_SALE_VALUE=OK ALERT_USD_RESIDUE_IN_RUB_MODE=ABSENT CHART_CARDS=2 CHART_CANVASES=14 CHARTS_BOTTOM=YES BROWSER_ERRORS=0 GRT_VISIBLE=NO RESULT_SHA256=b566f492cd80c8f99bc6b24cbef6d5f326c04f4d7168359d05eb1f189589e733 SCREENSHOT_SHA256=68a9a68d15673ef64c248a4fe886ac36f94923ec917daa2772fc9e7c1b5f4c16 ROOT_OWNED_STAGE_ENTRIES=11 ROOT_OWNED_STAGE_DIRECTORIES=2 REMOTE_STAGE_CLEANED=YES PRODUCTION_INDEX_UNCHANGED=YES PRODUCTION_HEALTH=200 VM170_FAILED_UNITS=0 PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES READY_FOR_UI_PREAPPLY_BACKUP_AND_PRODUCTION_APPLY=YES BROWSER_SMOKE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-ui-center-alert-currency-browser-smoke-v2/20260713T070125Z
1162:/var/lib/homelab-health/crypto-portfolio-remove-grt-surface.txt:1:STATUS=OK TS=2026-07-13T04:05:42Z TYPE=crypto-portfolio-remove-grt-surface BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PRODUCTION_CHANGED=NO DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru TARGET_STACK=/opt/stacks/crypto-portfolio DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 DESIRED_STATE_CLEAN=YES APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 APP_GRT_REFERENCES=5 WORKER_GRT_REFERENCES=2 INDEX_GRT_REFERENCES=2 LOCAL_GRT_REFERENCES=3 SQLITE_SCHEMA_CAPTURED=YES READY_FOR_GRT_CANDIDATE_PREPARE=YES SURFACE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-surface/20260713T040542Z
1163:/var/lib/homelab-health/cluster-admin-stage4d-adapter-candidates-in-isolation.txt:1:STATUS=OK TS=2026-07-14T06:14:58Z TYPE=cluster-admin-stage4d-adapter-candidates-in-isolation-final-seal BAD=0 STAGE4D_CLOSED=YES NEXT_STAGE=4E_RESTRICTED_PROBE_AGENT_DESIGN PRODUCTION_CHANGED=NO DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO APPLICATION_CHANGED=NO ENVELOPE_TOTAL=48 PROMETHEUS_ENVELOPES=5 BACKUP_MATRIX_ENVELOPES=40 HEALTH_ENVELOPES=3 LOCAL_REMOTE_OUTPUT_MATCH=OK PRODUCTION_STATE=0|0|OK DESIRED_STATE_HEAD=e77fb37a0fa0033fe46558045e31dd92b6d1238a DESIRED_STATE_CLEAN=YES PVE01_FAILED_UNITS=0 REPORT=/var/lib/homelab-cluster-admin-stage4/20260714T061458Z/stage4d-adapter-candidates-in-isolation-seal.txt REPORT_SHA256=da6244c0a8a458b508fd58a0fec9cb1e6fa8da98a36f44d1368cbe947f4b5835
1164:/var/lib/homelab-health/backup-matrix-cluster-desired-state-sync.txt:1:STATUS=OK TS=2026-07-13T17:51:31Z TYPE=backup-matrix-cluster-desired-state-sync BAD=0 PRE_SYNC_HEAD=af26ec3c4829b1330d49656205ac580ce1706263 GENERATED_SYNC_HEAD=597119391e7b265b85fd9a76644e4e33af47e469 GENERATED_SYNC_MODE=NEW_GENERATED_COMMIT GENERATED_SYNC_COMMIT_DELTA=1 MATRIX_STATUS=OK JOBS_TOTAL=35 JOBS_OK=31 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 CLUSTER_PASSPORT=OK FINAL_READINESS=OK OVERALL=OK CLUSTER_ADMIN=OK PROBE_OK=4 PROBE_BAD=0 FAILED_UNITS=0 DESIRED_STATE_DIRTY=0 MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 MATRIX_STATE_SHA256=14a31112aac5d577ae01df567921b093b2fedcf7189c4cc8f009216ee0749380 PRESEAL_SHA256=7ea1f710cd311bd51dba05a313b4c540f57d97407f008881e502fc213b4b3c99 BACKUP_RUN=NO RESTORE_RUN=NO CLOUD_WRITE_RUN=NO RETENTION_RUN=NO EVIDENCE_ROOT=/var/lib/homelab-change-sets/backup-matrix-cluster-final-seal/20260713T175131Z
1165:/var/lib/homelab-health/cluster-admin-stage2-recovery.txt:1:STATUS=OK TS=2026-07-13T01:04:59Z TYPE=cluster-admin-stage2-recovery BAD=0 RESULT=RECOVERED PRODUCTION_CHANGED=HEALTH_REGENERATION_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO INCIDENT_ENGINE_RESTARTED=NO DATABASE_ROLLBACK=OK COLLECTOR_ROLLBACK=OK CORE_DATABASE_STATE=0|0|9|7|1 LIVE_COLLECTOR_SHA256=a83abde2290d8392b1c631122671bcb680f9271d0b576fafacb8876d3c8034f1 OBSERVER=OK FULL_OBSERVER=OK RESTRICTED_PROBES=OK RESTRICTED_PROBES_AUTHORITY=CENTRAL_SYNTHESIZED_FROM_REMOTE_FULL_OBSERVER VERIFIER_BUG=REMOTE_RESTRICTED_PROBES_FILE_INCORRECTLY_REQUIRED FINAL_READINESS=OK CLUSTER_PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK VM180_FAILED_UNITS=0 PVE01_FAILED_UNITS=0 DESIRED_STATE_UNCHANGED=YES ROLLBACK_POINT=VERIFIED READY_FOR_STAGE2_RETRY=YES STAGE2_CLOSED=NO CHANGE_SET=/var/lib/homelab-change-sets/stage2-recovery-final/20260713T010459Z
1166:/var/lib/homelab-health/cluster-admin-stage3-api-runtime-dry-run.txt:1:STATUS=OK TS=2026-07-13T02:53:34Z TYPE=cluster-admin-stage3-api-runtime-dry-run BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=TEMPORARY_ISOLATED_RUNTIME_AND_EVIDENCE_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO LIVE_APPLICATION_CODE_CHANGED=NO DESIRED_STATE_CHANGED=NO PRODUCTION_SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO APP_PRECHANGE_SHA256=c2748b8f96df842f491b5d2790790594222549853d0e2336cf60ecd6d4de0020 APP_CANDIDATE_SHA256=672d6a2975dd0e093788e7eb81a76e7f901ea6374ca4092a25ba0091aba1cb50 ISOLATED_BIND=127.0.0.1:18081 TEMPORARY_DATABASE_RESTORED=YES TEMPORARY_DATABASE_DROPPED=YES TEMPORARY_DATABASE_UNCHANGED=YES RUNTIME_TESTS=41 RUNTIME_FAILED_TESTS=0 EXISTING_ROUTES_BACKWARD_COMPATIBLE=YES NEW_GET_ROUTES=7 UNAUTHENTICATED_NEW_ROUTES_401=YES AUTHENTICATED_NEW_ROUTES=OK PAGINATION=OK FILTER_VALIDATION=OK ENTITY_DETAIL=OK ENTITY_DEPENDENCIES=OK SOURCES=OK POLICIES=OK COLLECTOR_RUNS=OK DATABASE_TRANSACTION_MODE=BEGIN_READ_ONLY FORBIDDEN_RESPONSE_FIELDS=ABSENT OPENAPI_DOCS_EXPOSURE_CHANGED=NO SECURITY_HEADERS=OK ISOLATED_RUNTIME_STOPPED=YES ISOLATED_PORT_CLOSED=YES LIVE_APP_UNCHANGED=YES PRODUCTION_RUNTIME_HEALTH=OK READY_FOR_STAGE3_PREAPPLY_BACKUP=YES DESIRED_STATE_HEAD=8f888e5f2063e49da918b31584c9ea4bcf53af3a CHANGE_SET=/var/lib/homelab-change-sets/stage3-api-runtime-dry-run/20260713T025334Z RESULT=/var/lib/homelab-change-sets/stage3-api-runtime-dry-run/20260713T025334Z/runtime-result.json
1167:/var/lib/homelab-health/crypto-portfolio-remove-grt-failed-units-diagnostic.txt:1:STATUS=OK TS=2026-07-13T05:41:00Z TYPE=crypto-portfolio-remove-grt-failed-units-diagnostic BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=NO_DIAGNOSTIC_ONLY PRODUCTION_DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO FAILED_STATE_RESET=NO CLOUD_WRITE_RUN=NO PVE01_FAILED_UNIT_COUNT=2 PVE01_FAILED_UNIT_NAMES=homelab-backup-matrix-refresh.service,homelab-cluster-admin-pull.service CRYPTO_PORTFOLIO_FAILED_UNIT_COUNT=0 OTHER_FAILED_UNIT_COUNT=2 CURRENT_CRYPTO_PRECHANGE_HEALTH=YES UNIT_DETAILS_CAPTURED=YES READY_FOR_TARGETED_FAILED_UNIT_DECISION=YES DIAGNOSTIC_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-failed-units-diagnostic/20260713T054100Z
1168:/var/lib/homelab-health/crypto-portfolio-remove-grt-production-apply.txt:1:STATUS=OK TS=2026-07-13T05:47:54Z TYPE=crypto-portfolio-remove-grt-production-apply BAD=0 COMMAND_PHYSICAL_LINES=1 APPLY_REVISION=V3 FIX_UID_READONLY=YES FIX_READINESS_ENDPOINT=YES FIX_ROLLBACK_DATABASE_METADATA=YES PRODUCTION_CHANGED=YES_REMOVE_GRT DESIRED_STATE_CHANGED=NO_BY_THIS_COMMAND PRODUCTION_CONTAINER_RECREATED=YES CLOUD_WRITE_RUN=NO ROLLBACK_ON_FAILURE=ARMED_NOT_NEEDED TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=80763a6c62b94cde4f3293f3b529980f76919d25 PRECHANGE_BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T051015Z ROLLBACK_PATH=/var/lib/homelab-incidents/crypto-remove-grt-prechange-20260713T051013Z APP_SHA256=44df2b25183b7c5d87ff1516ec2d9cd1d169acde3031f7c2aa1fc4e0c7813cd2 WORKER_SHA256=174a14b86014535980d200a76d0057110c7af102c66f16c2e21fc3e74bb2e49e INDEX_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e CHART_HEALTH_SCRIPT_SHA256=c16473f16390eeeff4d4eb701db604093564dd78d0be2d30b4cc8ffeeb22364a ACTIVE_COINS=HBAR,POL GRT_SOURCE_REFERENCES=0 GRT_DATABASE_ROWS=0|0|0 GRT_MARKET_ENTRY=ABSENT GRT_CACHE_ENTRIES=ABSENT DATABASE_MODE=0644 DATABASE_UID=10001 DATABASE_GID=10001 DATABASE_INTEGRITY=ok WORKER_STATUS=OK WORKER_COIN_COUNT=2 PUBLIC_CODES=200|303|401|200|200|401 CHART_HEALTH=OK CHART_SYMBOLS=HBAR,POL CHART_HEALTH_TIMER=ACTIVE VM170_FAILED_UNITS=0 PVE01_UNRELATED_FAILED_UNITS=homelab-backup-matrix-refresh.service,homelab-cluster-admin-pull.service PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES REMOTE_STAGE_CLEANED=YES READY_FOR_GRT_POSTAPPLY_VERIFY_BACKUP_SYNC=YES APPLY_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-production-apply-v3/20260713T054754Z
1169:/var/lib/homelab-health/mail-cloud-restore-drill.txt:1:STATUS=STALE
1170:/var/lib/homelab-health/mail-cloud-restore-drill.txt:20:FRESHNESS=STALE
1171:/var/lib/homelab-health/edge-restore-proofs-offhost.txt:1:STATUS=OK TS=20260624T190040Z TYPE=edge-restore-proofs-offhost DEST_HOST=pve02 DEST_DIR=/mnt/staging/offhost/edge-restore-proofs-from-edge ITEMS=14 OK=14 ERROR=0 BYTES=226024287 SECRET_PRINTED=NO
1172:/var/lib/homelab-health/crypto-portfolio-runtime-final-seal.txt:1:STATUS=OK TS=2026-07-13T08:37:54Z TYPE=crypto-portfolio-ui-center-alert-currency-final-seal BAD=0 DANGLING_IMAGE_REFERENCE_GATE=OK DESIRED_STATE_SYNC_ARTIFACT_GATE=OK APP=crypto-portfolio PRODUCTION_CHANGED=NO_FINAL_SEAL_ONLY APP_SHA256=44df2b25183b7c5d87ff1516ec2d9cd1d169acde3031f7c2aa1fc4e0c7813cd2 WORKER_SHA256=174a14b86014535980d200a76d0057110c7af102c66f16c2e21fc3e74bb2e49e INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb LIBRARY_SHA256=c0992580867c4912cc9385b3c2728315bcc1a76c7f1087dca908430fccdf31d7 ACTIVE_COINS=HBAR,POL GRT_REMOVED=YES GRT_ROWS=0|0|0 GRT_SOURCE_REFERENCES=0 GRT_CACHE_ENTRIES=0 SHARP_MOVES_CENTERED=YES ALERT_CURRENCY_USD_RUB=OK BROWSER_SMOKE=OK BROWSER_RUNS=2 BROWSER_ERRORS=0 BROWSER_RESULT_SHA256=b566f492cd80c8f99bc6b24cbef6d5f326c04f4d7168359d05eb1f189589e733 SCREENSHOT_SHA256=00b2853bee9c880146d0dd75fac60feb7e46d4c129ee06754047ef30d90bd4ba BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T075801Z TWO_CLOUDS=OK RESTORE_CHECK=OK DATABASE_INTEGRITY=ok RETENTION=OK ROLLBACK_COUNT=3 DANGLING_IMAGES=2 REFERENCED_DANGLING_IMAGES=2 UNREFERENCED_DANGLING_IMAGES=0 PRUNED_UNREFERENCED_DANGLING_IMAGES=0 CHART_HEALTH=OK PUBLIC=OK MONITOR_TIMER=active BACKUP_TIMER=active CLOSURE_DESIRED_STATE_HEAD=2999910549eaff9b9431b9c656d318340eb95cbd DESIRED_STATE=OK PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES ALL_CRYPTO_PORTFOLIO_UI_TAILS_CLOSED=YES
1173:/var/lib/homelab-health/backup-matrix-cluster-preseal-convergence.txt:1:STATUS=OK TS=2026-07-13T17:48:14Z TYPE=backup-matrix-cluster-preseal-convergence BAD=0 MATRIX_STATUS=OK JOBS_TOTAL=35 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 CLUSTER_PASSPORT=OK FINAL_READINESS=OK OVERALL=OK CLUSTER_ADMIN=OK PROBE_OK=4 PROBE_BAD=0 WEBPANEL=OK FAILED_UNITS=0 DESIRED_STATE_HEAD=af26ec3c4829b1330d49656205ac580ce1706263 DESIRED_STATE_DIRTY=0 MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 MATRIX_STATE_SHA256=3791fa78cdfb56cd7027f89350dff245abcffc4f935adfc336ceeea00854ac89 CLUSTER_PASSPORT_SHA256=1b8b924ae0e5ce6872fe5671318094ce494806c2db7efe28299b5ff49adc0d0d FINAL_READINESS_SHA256=ac5159fa8c497afe021b5cb232f08cfec0e1a4d516d8aae966ba03f7034f1412 OVERALL_SHA256=6ae529bfdf5811f686d3b53978f453ce4f8757c74fc1be105de219b13a61ba46 BACKUP_RUN=NO RESTORE_RUN=NO CLOUD_WRITE_RUN=NO RETENTION_RUN=NO EVIDENCE_ROOT=/var/lib/homelab-change-sets/backup-matrix-cluster-preseal-resume/20260713T174748Z
1174:/var/lib/homelab-health/uptime-kuma-pending-health-monitors-fix.txt:8:NOTE=Paperless Restore may become red if its health file contains STATUS=ERROR; that is a real restore-health issue, not a Kuma pending issue.
1175:/var/lib/homelab-health/pve03-capacity-forensics.txt:1:STATUS=OK TS=2026-07-08T05:42:29Z TYPE=pve03-capacity-forensics MODE=SUPERSEDED_BY_PVE03_ROOT_STAGING_CLEANUP BAD=0 WARN=0 OLD_STATUS=WARN_ARCHIVED ROOT_CLEANUP=OK STAGING_RETENTION=OK STAGING_CLEANUP=OK FAILED_UNIT_CLEANUP=OK REPORT=/var/lib/homelab-duty-admin-residual-cleanup/pve03-capacity-forensics-closure-latest.md
1176:/var/lib/homelab-health/remote-failed-unit-apply.txt:1:STATUS=OK TS=2026-07-08T05:25:22Z TYPE=remote-failed-unit-apply BAD=0 WARN=0 MODE=APPLY_CLEANUP_PLAN_EXCEPT_PAPERLESS_CORE_GUARD PRECHECK=YES APPLY_OK=YES LOCAL_OVERRIDE_COUNT=4 PVE02_RESET=1 EDGE_APPLY=1 LOCAL_FAILED_AFTER=0 PVE02_FAILED_AFTER=0 EDGE_FAILED_AFTER=1 EDGE_DEFER_ONLY=YES PVE02_CODEVIPE_FAILED=inactive DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 OVERALL_STATUS=OK OVERALL_BAD=0 DEFERRED_UNIT=homelab-paperless-core-guard.service REPORT=/var/lib/homelab-remote-failed-unit-cleanup/apply-latest.md
1177:/var/lib/homelab-health/recovery-card.txt:1:STATUS=OK TS=2026-07-08T08:24:05Z TYPE=recovery-card BAD=0 WARN=0 MODE=READ_ONLY_RECOVERY_CARD_GENERATION RECOVERY_CARD_OK=YES FREEZE_STATUS=OK FREEZE_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T082034Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T082034Z EMERGENCY_JOB=homelab-pve01-emergency-pack-20260708T002739Z VM_DUMPS_JOB=homelab-pve01-vm-local-dumps-20260706T153606Z ENDPOINT_LOCAL_OK=1 ENDPOINT_LAN_OK=1 ENDPOINT_PUBLIC_OK=1 BACKUP_SLA=OK BACKUP_COVERAGE=OK EXTENDED_APPBACKUP=OK VM_BACKUP=OK EXTERNAL_PROBE=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 DIRTY_COUNT=0 CARD=/var/lib/homelab-recovery-card/recovery-card-latest.md SUMMARY=/var/lib/homelab-recovery-card/recovery-summary-latest.tsv REPORT=/var/lib/homelab-recovery-card/recovery-card-latest.md
1178:/var/lib/homelab-health/cluster-admin-stage0-convergence.txt:1:STATUS=WARN TS=2026-07-12T23:12:59Z TYPE=cluster-admin-stage0-targeted-convergence BAD=2 PROFILE_OK=NO CLUSTER_PASSPORT=FAIL FINAL_READINESS=FAIL OVERALL=FAIL DUTY_ADMIN=FAIL FAILED_UNITS=0 LIVE_MATCH_DESIRED_STATE=YES READY_FOR_FINAL_SEAL=NO REPORT_DIR=/var/lib/homelab-cluster-admin-stage0/readiness-profile-20260712T231259Z PRODUCTION_CHANGED=HEALTH_REGENERATION_ONLY PRODUCTION_SCHEMA_CHANGED=NO
1179:/var/lib/homelab-health/cluster-admin-stage4c-database-migration-dry-run.txt:1:STATUS=OK TS=2026-07-14T05:33:26Z TYPE=cluster-admin-stage4c-database-migration-dry-run-final-seal BAD=0 STAGE4C_CLOSED=YES NEXT_STAGE=4D_ADAPTER_CANDIDATES_IN_ISOLATION MIGRATION_SHA256=a0bd6a50c6ad9b90528c2e90e6d9cb4279b09811b559b2504c83d7527b19e578 FINAL_READINESS=OK OVERALL=OK PUBLIC_ENDPOINT=OK WEBPANEL_SYNC=OK PVE01_FAILED_UNITS=0 VM180_FAILED_UNITS=0 DESIRED_STATE_HEAD=df86e4d27a64613b60ae55253b6348866dd34fa9 DESIRED_STATE_CLEAN=YES REPORT=/var/lib/homelab-cluster-admin-stage4/20260714T053326Z/stage4c-database-migration-dry-run-seal.txt REPORT_SHA256=3368c302ca0ee76fab8fbc97caa65adf4cf57e61c5b5cdb9a1f3829a05af2e2a
1180:/var/lib/homelab-health/operator-command-size-guard.txt:1:STATUS=OK TS=2026-07-14T18:26:54Z TYPE=operator-command-size-guard RULE_ACTIVE=YES PARSE_ERROR_TAIL=CLOSED UNRESOLVED_TAIL_COUNT=0 BLOCKER_COUNT=0 REPORT_SHA256=2ed7949dde9d5f8346e46af362e27cb8982d65f00ec602546efab506509e47af
1181:/var/lib/homelab-health/node-loss-final-closure.txt:1:STATUS=OK TS=2026-07-08T16:35:56Z TYPE=node-loss-final-closure BAD=0 MODE=ALL_NODE_LOSS_DRILLS_SEALED PVE03_EDGE_RESTORE=OK PVE03_EDGE_FRESHNESS=OK PVE02_FORUM_RUNTIME=OK PVE01_NEXTCLOUD_DNS=OK PVE01_CORE_MONITORING_APPLEVEL=OK FULL_VM_GAP_VM170_VM171=ACKNOWLEDGED_APP_LEVEL_ONLY TEST_VMS_CLEANED=YES DNS_PAIR_OK=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-node-loss/final-closure-latest.md
1182:/var/lib/homelab-health/crypto-portfolio-remove-grt-browser-smoke.txt:1:STATUS=OK TS=2026-07-13T04:52:06Z TYPE=crypto-portfolio-remove-grt-browser-smoke BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=OPTIONAL_XXD_DEPENDENCY_MISSING_WITH_STDERR_SUPPRESSED BROWSER_RERUN=NO EVIDENCE_SALVAGED=YES PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO CANDIDATE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CONTAINER_STARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 INPUT_APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd INPUT_WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INPUT_INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 ACTIVE_COINS=HBAR,POL GRT_VISIBLE=NO BROWSER_CARDS=2 BROWSER_CANVASES=14 BROWSER_MOVES=2 BROWSER_COIN_CARDS=2 BROWSER_POSITION_ROWS=2 BROWSER_ALERT_OPTIONS=2 BROWSER_ACTIVE_COIN=HBAR BROWSER_CHART_SYMBOLS=HBAR,POL BROWSER_ERRORS=0 BROWSER_LOADING_TEXT=ABSENT BROWSER_ERROR_TEXT=ABSENT CHARTS_POSITION=BOTTOM PNG_VALIDATION=PYTHON_STDLIB PNG_SIGNATURE=89504e470d0a1a0a SCREENSHOT_SHA256=3d132b477a382487e34db2f2feaf7e4f43af8d30fd8e1530225782da5d7336b9 RESULT_SHA256=0deadbf38aeef86726035da66eb1991c9494f59172b1c1a5c502c50c2de95c60 LIVE_PRODUCTION_DATA_MOUNTED=NO NPM_INSTALL_RUN=NO DOCKER_PULL_RUN=NO PRODUCTION_SOURCE_HASHES_UNCHANGED=YES PRODUCTION_CONTAINER_IDS_AND_RESTARTS_UNCHANGED=YES PRODUCTION_DATABASE_INTEGRITY=ok PRODUCTION_GRT_STATE_PRESERVED=YES PUBLIC_PRODUCTION_STATE_PRESERVED=YES REMOTE_RESIDUE=NONE READY_FOR_GRT_PRECHANGE_BACKUP=YES BROWSER_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-browser-smoke-salvage/20260713T045206Z
1183:/var/lib/homelab-health/independent-external-runner-audit.txt:1:STATUS=OK TS=2026-07-08T09:13:25Z TYPE=independent-external-runner-audit BAD=0 WARN=0 MODE=READ_ONLY_INDEPENDENT_EXTERNAL_RUNNER_AUDIT AUDIT_OK=YES INDEPENDENT_RUNNER_READY=NO DEPLOY_DECISION=NO_DEPLOY CURRENT_INDEPENDENT_RUNNER_AVAILABLE=no TRUSTED_NETBIRD_IDENTITY=edge-vm PUBLIC_VPS_DECISION=REJECTED_HOSTKEY_MISMATCH PUBLIC_VPS_SSH_OK=0 PUBLIC_VPS_REJECTED_HOSTKEY=YES EDGE_LAN_SSH_OK=2 EDGE_NETBIRD_SSH_OK=2 EDGE_DNS_SSH_OK=1 PUBLIC_OVERALL_OK=1 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 OVERALL_STATUS=OK OVERALL_BAD=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 CLOSEOUT_OK=YES RECOVERY_SEAL_OK=YES DIRTY_COUNT=0 REPORT=/var/lib/homelab-independent-external-runner/audit-latest.md
1184:/var/lib/homelab-health/sops-edge-secret-coverage-closure.txt:1:STATUS=OK TS=2026-07-08T05:01:47Z TYPE=sops-edge-secret-coverage-closure BAD=0 WARN=0 MODE=POLICY_SKELETON_SUPERSEDED_NOOP PRECHECK=YES POLICY_MODE=POLICY_SKELETON_NOT_ACTIVE AUDIT_CLASS=UNGATED_FAILED_STATE_REVIEW IN_HARD_GATES=0 OVERRIDE_WRITTEN=1 FAILED_UNITS_AFTER=0 UNIT_FAILED_AFTER=inactive MARKER_STATUS=OK DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 DIRTY_COUNT=0 SOPS_CLOSURE_OK=YES REPORT=/var/lib/homelab-failed-unit-legacy-cleanup/sops-edge-secret-coverage-closure-latest.md
1185:/var/lib/homelab-health/mail-cloud-canary.txt:1:STATUS=OK TS=20260618T174444Z TYPE=mail-cloud-canary REMOTE_TOTAL=13 REMOTE_OK=13 REMOTE_FAIL=0 DEST=restore-test/p1-canary/20260618T174444Z SHA256=a6bf6a1493a2556b92c8e6547163500345d4e0e32688937370b942b881c47182 SECRET_PRINTED=no NOTES=none
1186:/var/lib/homelab-health/filebrowser-restore.txt:1:STATUS=STALE
1187:/var/lib/homelab-health/filebrowser-restore.txt:20:FRESHNESS=STALE
1188:/var/lib/homelab-health/evidence-seal/latest.env:6:TOTAL_FAILURES=0
1189:/var/lib/homelab-health/final-green-freeze-audit.txt:1:STATUS=OK TS=2026-07-08T07:45:35Z TYPE=final-green-freeze-audit BAD=0 WARN=0 MODE=READ_ONLY_FREEZE_EVIDENCE_AUDIT FREEZE_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 EXACT_WARN_REVIEW_FILES=0 LOCAL_OVERALL_OK=1 LAN_OVERALL_OK=1 PUBLIC_OVERALL_OK=1 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T072711Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T072711Z BACKUP_SLA=OK BACKUP_COVERAGE=OK EXTENDED_APPBACKUP=OK VM_BACKUP=OK SECRET_EXPOSURE=OK REVIEW_SURFACE_REGISTRY=YES FINAL_GLOBAL_GREEN=YES EXTERNAL_PROBE=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 DIRTY_COUNT=0 EVIDENCE=/var/lib/homelab-final-green-freeze/freeze-evidence-latest.tsv REPORT=/var/lib/homelab-final-green-freeze/freeze-audit-latest.md
1190:/var/lib/homelab-health/crypto-portfolio-remove-grt-runtime-dry-run.txt:1:STATUS=OK TS=2026-07-13T04:41:24Z TYPE=crypto-portfolio-remove-grt-runtime-dry-run BAD=0 COMMAND_PHYSICAL_LINES=1 VERIFIER_REVISION=V3 ANALYSIS_SOURCES=FULL_CHAT_CRYPTO_HANDOFF_CLUSTER_REFERENCE_ERROR_HISTORY PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO CANDIDATE_DATABASE_CHANGED=ISOLATED_COPY_RUNTIME_CHECK_ONLY DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO PRODUCTION_CONTAINER_RECREATED=NO ISOLATED_CONTAINER_STARTED=YES ISOLATED_CONTAINER_CLEANED=YES CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 INPUT_APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd INPUT_WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INPUT_INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 ACTIVE_COINS=HBAR,POL CANDIDATE_GRT_REFERENCES=0 DATABASE_GRT_AFTER=0|0|0 WORKER_RUNTIME_CHECK=OK WEB_RUNTIME_CHECK=OK HTTP_RUNTIME_CHECK=OK HTTP_CODES=200|303|401|200|200|401 CHART_HEALTH=OK CHART_SYMBOLS=HBAR,POL ROUTE_RETURN_DECODER=DICT_OR_RESPONSE_BODY LIVE_PRODUCTION_DATA_MOUNTED=NO PRODUCTION_SOURCE_HASHES_UNCHANGED=YES PRODUCTION_CONTAINER_IDS_AND_RESTARTS_UNCHANGED=YES PRODUCTION_DATABASE_INTEGRITY=ok PRODUCTION_GRT_STATE_PRESERVED=YES PUBLIC_PRODUCTION_STATE_PRESERVED=YES REMOTE_RESIDUE=NONE BROWSER_SMOKE=NOT_RUN READY_FOR_GRT_ISOLATED_BROWSER_SMOKE=YES RUNTIME_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-runtime-run-v3/20260713T044124Z
1191:/var/lib/homelab-health/sops-edge-secret-coverage-audit.txt:1:STATUS=OK TS=2026-07-08T04:59:58Z TYPE=sops-edge-secret-coverage-audit BAD=0 WARN=1 MODE=READ_ONLY_SOPS_EDGE_SECRET_COVERAGE_AUDIT UNIT_FAILED=failed CLASS=UNGATED_FAILED_STATE_REVIEW IN_HARD_GATES=no OVERALL_BAD=0 FINAL_FAIL=0 DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=1 RELATED_WARN_OR_ERROR_HEALTH=3 RESET_FAILED_DONE=no REPORT=/var/lib/homelab-failed-unit-legacy-cleanup/sops-edge-secret-coverage-audit-latest.md
1192:/var/lib/homelab-health/duty-admin-residual-cleanup-seal.txt:1:STATUS=OK TS=2026-07-08T05:44:22Z TYPE=duty-admin-residual-cleanup-seal BAD=0 WARN=0 MODE=FINAL_RESIDUAL_NOISE_SEAL RESIDUAL_SEAL_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 REVIEW_ITEMS_INTENTIONALLY_LEFT=yes LEGACY_PHASE_ITEMS_INTENTIONALLY_LEFT=yes FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 PVE03_CAPACITY_FORENSICS_STATUS=OK LOCAL_CLEANUP_OK=YES REMOTE_CLEANUP_OK=YES DIRTY_COUNT=0 REPORT=/var/lib/homelab-duty-admin-residual-cleanup/seal-latest.md
1193:/var/lib/homelab-health/prometheus-npmplus-kuma-supersede-repair-20260709T073019Z/npmplus-kuma-config-backup.txt.bak:1:STATUS=ERROR TS=20260709T072619Z TYPE=npmplus-kuma-config-backup NOTES=edge_backup_failed
1194---- FILESYSTEMS ---
1195-Filesystem Type Size Used Avail Use% Mounted on
1196-udev devtmpfs 15G 0 15G 0% /dev
--
1289- "pveversion": "pve-manager/9.2.3/d0fde103346cf89a"
1290-}
1291:--- FAILED_UNITS ---
1292---- TIMERS ---
1293-Tue 2026-07-21 16:26:07 MSK 4min 53s Tue 2026-07-21 16:21:07 MSK 6s ago homelab-smartctl-textfile.timer homelab-smartctl-textfile.service
1294-Tue 2026-07-21 16:27:30 MSK 6min Tue 2026-07-21 16:12:30 MSK 8min ago prometheus-node-exporter-smartmon.timer prometheus-node-exporter-smartmon.service
--
1309-2026-07-21T08:03:00.2596892130 770 /var/lib/homelab-health/vaultwarden-isolated-restore-drill.txt
1310-2026-07-21T16:21:07.1497113500 108 /var/lib/homelab-health/routerguard-ap.txt
1311:/var/lib/homelab-health/codevipe-cloud-backup-reset.txt:1:STATUS=OK TS=2026-07-08T05:25:22Z TYPE=remote-failed-unit-codevipe-reset BAD=0 WARN=0 MODE=RESET_STALE_FAILED_STATE UNIT=codevipe-cloud-backup.service REASON=last_journal_success_and_no_timer ACTION=reset_failed_only
1312---- FILESYSTEMS ---
1313-Filesystem Type Size Used Avail Use% Mounted on
1314-udev devtmpfs 6.8G 0 6.8G 0% /dev
--
1402- "pveversion": "pve-manager/9.2.3/d0fde103346cf89a"
1403-}
1404:--- FAILED_UNITS ---
1405:● homelab-immutable-offline-verify.service loaded failed failed Verify immutable offline backup image
1406---- TIMERS ---
1407-Tue 2026-07-21 16:21:41 MSK 25s Tue 2026-07-21 16:16:41 MSK 4min 34s ago homelab-smartctl-textfile.timer homelab-smartctl-textfile.service
1408-Tue 2026-07-21 16:28:31 MSK 7min Tue 2026-07-21 16:13:31 MSK 7min ago prometheus-node-exporter-smartmon.timer prometheus-node-exporter-smartmon.service
--
1470-2404 /var/lib/vz/dump/vzdump-qemu-190-2026_07_19-03_33_53.log
1471-2305 /var/lib/vz/dump/vzdump-qemu-190-2026_07_18-03_33_48.log
1472:=== RECENT_FAILED_COMMANDS ===
1473:2026-07-21T08:03:07Z GITEA-PUSH-AUTH-001 fcfc28bcc5acb6382e895d5ed7cf51d7d5a3e2f5d8f8332770c58b42039f8a94 1 FAIL gitea-runtime
1474:2026-07-21T08:13:32Z GITEA-TOKEN-CLI-HELP-003 f1d0f967eed6490ad822944a42108c82dc4b59c9fd2557d8c64624dcb6853cd4 1 FAIL gitea-runtime
1475:2026-07-21T08:14:52Z GITEA-CLI-CAPABILITIES-004 2cd6c4d28c8f646d8063105fca80b2f80d38c6a5db3a3eb76869de326b12d2d6 1 FAIL gitea-runtime
1476:2026-07-21T09:49:51Z RUNTIME-RECEIVER-STATE-027 0831002c78c26f35826cd7199995e335ceadcd39f08b9bac23882a0fbb59c8cc 127 FAIL gitea-runtime
1477:2026-07-21T10:26:18Z RUNTIME-RECEIVER-CLEANUP-036 65fbcfbce38a55723a92a8790429d6567389321f030f02cb4bf301cbe67e2531 1 FAIL gitea-runtime
1478:2026-07-21T11:35:23Z BACKUP-DUAL-CLOUD-PREFLIGHT-059 c6f4134d15a9f641ddaf67535760d0abede973dd08222a37af05612a6f580479 1 FAIL backup-audit
1479:2026-07-21T12:28:26Z BACKUP-VM-DUMP-PIPELINE-073 cd8291d0f14a37241e6062408603455d1ebd483091bebfa46c2b7a768f21308b 1 FAIL backup-audit
1480:2026-07-21T12:42:53Z BACKUP-VM-DUMP-CHANGESET-PREFLIGHT-075 a9a8a6046441de0c40a8b13bdfe96698c53a5156b4346b7ce28dc1f2362e8936 143 FAIL backup-audit
1481:2026-07-21T12:56:52Z BACKUP-DRAIN-LOCAL-ARCHIVES-079 531e1da7ce1681eed084b22ced7da87eb1ee8995f662dc7559f4f9dde3c0ac32 75 FAIL backup-remediation
1482:2026-07-21T13:04:48Z BACKUP-FINAL-PROOF-082 9e675a3cb621d0d1fcd8918b02278ab08c1a974f544ebef9a5639bf6578ee601 127 FAIL backup-audit
1483:2026-07-21T13:07:09Z CLUSTER-CLEANUP-INVENTORY-083 09d3f14ca0205936a910dc7af7991474030273a47d76fa033ea6a3185c083922 1 FAIL cluster-cleanup
1484:2026-07-21T13:14:27Z CLUSTER-BACKUP-POSTCLEAN-PROOF-086 601958c48614dbc25cc4592b111de4dec3306e53b756b275068b81a41d42b2da 127 FAIL backup-cleanup
1485:2026-07-21T13:15:21Z BACKUP-POLICY-CLOSEOUT-087 d79c1c881ebca24db09c5b4ed4b71b09a5ad338c30789c99b0644c8935072c52 1 FAIL backup-cleanup
1486:2026-07-21T13:17:20Z BACKUP-CLEANUP-FINAL-CLOSEOUT-088 5ba2893489b6f5831fc4599d3efa874d89c3fd41da7151f9c32945d11d1a23d6 1 FAIL backup-cleanup
1487-CRITICAL_TAILS_BASELINE=COMPLETE
=== P1_BOOTSTRAP_AND_COVERAGE ===
=== BOOTSTRAP ===
{
"schema_version": 2,
"status": "READY",
"generated_at": null,
"coverage": {
"status": "PARTIAL",
"missing": [
"automatic_refresh",
"per_node_freshness",
"structured_service_inventory"
]
},
"freshness": {
"max_age_seconds": 86400,
"stale_requires_refresh": true
},
"entrypoints": {
"full_context_url": "https://git.gram1.ru/homelab-admin/homelab-public-context/raw/commit/e23caf794cc68ce5fa6f8922efa7f2a35c8aa6e0/runtime/history/CONTEXT-SOURCES-FULL-002.txt",
"errors_index_url": "https://git.gram1.ru/homelab-admin/homelab-public-context/raw/commit/a615069de3d32564fc4de29baff949bdaf811caf/runtime/history/ERRORS-INDEX-004.json",
"workflow_contract_url": "https://git.gram1.ru/homelab-admin/homelab-public-context/raw/commit/2f37a4eb600528d03f5c570af8c5aa0e463b1798/runtime/history/WORKFLOW-CONTRACT-003.json",
"double_check_rule_url": "https://git.gram1.ru/homelab-admin/homelab-public-context/raw/commit/d1061e47e7955dba9c73a2841d4eacf02ccc1bf5/runtime/policy/double-check-rule.json"
}
}
=== CLUSTER_STATUS ===
=== P1_STORAGE_AND_CAPACITY ===
17-130:32. Ошибка: 153_PROMETHEUS_TARGETS_SETTLED_PROOF содержит SyntaxError из-за вложенного Python внутри SSH.
18-137:Такие выводы не являются инфраструктурным состоянием; считать невалидными только соответствующие failed command attempts.
19:150:Факт: terminal output around storage block creation showed possible paste/truncation artifact near smartctCK=FAIL.
20-154:37. Coverage gap: pve03_staging missing from disk-space health coverage.
21-159:Факт: terminal output around command 222 showed paste artifact near "FAIL; fi".
22-163:Факт: строка справочника с фразой No unresolved final REVIEW_REQUIRED... SyntaxError... была принята за BAD_MARKER.
--
405- 3 1 pve02
406-=== CLUSTER_RESOURCES ===
407:{"type":"lxc","node":"pve01","vmid":110,"name":"dns1","status":"running","uptime":448924,"maxcpu":1,"cpu":0.000299025656238528,"maxmem":536870912,"mem":90935296,"maxdisk":8350298112,"disk":1933914112,"storage":null}
408:{"type":"lxc","node":"pve02","vmid":111,"name":"dns2","status":"running","uptime":448922,"maxcpu":1,"cpu":0.00118032586667801,"maxmem":536870912,"mem":77062144,"maxdisk":8350298112,"disk":1435082752,"storage":null}
409:{"type":"lxc","node":"pve01","vmid":112,"name":"unbound1","status":"running","uptime":448927,"maxcpu":1,"cpu":0.0000996752187461762,"maxmem":536870912,"mem":41971712,"maxdisk":8350298112,"disk":757272576,"storage":null}
410:{"type":"lxc","node":"pve02","vmid":113,"name":"unbound2","status":"running","uptime":448925,"maxcpu":1,"cpu":0.0000983604888898342,"maxmem":536870912,"mem":31571968,"maxdisk":8350298112,"disk":750432256,"storage":null}
411:{"type":"qemu","node":"pve03","vmid":130,"name":"edge-vm","status":"running","uptime":448787,"maxcpu":4,"cpu":0.0296798201262478,"maxmem":12884901888,"mem":10835238912,"maxdisk":103079215104,"disk":0,"storage":null}
412:{"type":"qemu","node":"pve01","vmid":150,"name":"nextcloud","status":"running","uptime":448922,"maxcpu":4,"cpu":0.0224714397065076,"maxmem":8589934592,"mem":5494030336,"maxdisk":68719476736,"disk":0,"storage":null}
413:{"type":"qemu","node":"pve02","vmid":160,"name":"forum-prod","status":"running","uptime":448920,"maxcpu":4,"cpu":0.00147539847586977,"maxmem":8589934592,"mem":8104173568,"maxdisk":214748364800,"disk":0,"storage":null}
414:{"type":"qemu","node":"pve01","vmid":170,"name":"core-apps","status":"running","uptime":448918,"maxcpu":4,"cpu":0.0397571625576673,"maxmem":10737418240,"mem":10081083392,"maxdisk":128849018880,"disk":0,"storage":null}
415:{"type":"qemu","node":"pve01","vmid":171,"name":"monitoring","status":"running","uptime":448915,"maxcpu":4,"cpu":0.0190142951362757,"maxmem":8589934592,"mem":4098203648,"maxdisk":128849018880,"disk":0,"storage":null}
416:{"type":"qemu","node":"pve02","vmid":180,"name":"cluster-admin","status":"running","uptime":448917,"maxcpu":2,"cpu":0.0201637791702202,"maxmem":2147483648,"mem":1965608960,"maxdisk":25769803776,"disk":0,"storage":null}
417:{"type":"qemu","node":"pve03","vmid":190,"name":"homelab-ops-worker","status":"running","uptime":444150,"maxcpu":4,"cpu":0.00197865467508319,"maxmem":8589934592,"mem":2149752832,"maxdisk":85899345920,"disk":0,"storage":null}
418:{"type":"qemu","node":"pve02","vmid":9130,"name":"edge-cold-standby","status":"stopped","uptime":0,"maxcpu":4,"cpu":0,"maxmem":12884901888,"mem":0,"maxdisk":103079215104,"disk":0,"storage":null}
419:{"type":"node","node":"pve02","vmid":null,"name":null,"status":"online","uptime":448958,"maxcpu":4,"cpu":0.0202669303015324,"maxmem":16536748032,"mem":12247134208,"maxdisk":105089261568,"disk":27718709248,"storage":null}
420:{"type":"node","node":"pve03","vmid":null,"name":null,"status":"online","uptime":448804,"maxcpu":8,"cpu":0.0187554341075643,"maxmem":33553518592,"mem":14079365120,"maxdisk":105089261568,"disk":55840108544,"storage":null}
421:{"type":"node","node":"pve01","vmid":null,"name":null,"status":"online","uptime":448941,"maxcpu":24,"cpu":0.0113622226829774,"maxmem":33379028992,"mem":21412495360,"maxdisk":105089261568,"disk":42922749952,"storage":null}
422:{"type":"storage","node":"pve02","vmid":null,"name":null,"status":"available","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":846016282624,"disk":290521991453,"storage":"local-lvm"}
423:{"type":"storage","node":"pve03","vmid":null,"name":null,"status":"available","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":355660201984,"disk":155636904388,"storage":"local-lvm"}
424:{"type":"storage","node":"pve01","vmid":null,"name":null,"status":"available","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":869168840704,"disk":286912634316,"storage":"local-lvm"}
425:{"type":"storage","node":"pve02","vmid":null,"name":null,"status":"available","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":105089261568,"disk":27718709248,"storage":"local"}
426:{"type":"storage","node":"pve03","vmid":null,"name":null,"status":"available","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":105089261568,"disk":55840108544,"storage":"local"}
427:{"type":"storage","node":"pve01","vmid":null,"name":null,"status":"available","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":105089261568,"disk":42922749952,"storage":"local"}
428:{"type":"network","node":"pve02","vmid":null,"name":null,"status":"ok","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":null,"disk":null,"storage":null}
429:{"type":"network","node":"pve03","vmid":null,"name":null,"status":"ok","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":null,"disk":null,"storage":null}
430:{"type":"network","node":"pve01","vmid":null,"name":null,"status":"ok","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":null,"disk":null,"storage":null}
431:=== STORAGE ===
432-Name Type Status Total (KiB) Used (KiB) Available (KiB) %
433:local dir active 102626232 41916788 55450180 40.84%
434-local-lvm lvmthin active 848797696 280188119 568609576 33.01%
435-=== BACKUP_JOBS ===
436:[{"compress":"zstd","enabled":1,"id":"homelab-nightly-all","mailnotification":"failure","mode":"snapshot","next-run":1784680200,"prune-backups":{"keep-daily":"4"},"schedule":"03:30","storage":"local","type":"vzdump","vmid":"110,111,112,113,130,150,190"}]
437-=== BACKUP_MATRIX ===
438-STATUS=FAIL
439-TS=2026-07-21T07:02:41Z
--
528---- TIMERS ---
529-Tue 2026-07-21 16:23:54 MSK 2min 43s Tue 2026-07-21 16:18:29 MSK 2min 41s ago homelab-crypto-portfolio-chart-health.timer homelab-crypto-portfolio-chart-health.service
530:Tue 2026-07-21 16:24:14 MSK 3min 3s Tue 2026-07-21 16:19:14 MSK 1min 56s ago homelab-smartctl-textfile.timer homelab-smartctl-textfile.service
531-Tue 2026-07-21 16:24:14 MSK 3min 3s Tue 2026-07-21 16:19:14 MSK 1min 56s ago netbird-peers-health.timer netbird-peers-health.service
532-Tue 2026-07-21 16:30:41 MSK 9min Tue 2026-07-21 16:16:27 MSK 4min 43s ago homelab-external-probe-vps-health.timer homelab-external-probe-vps-health.service
533-Tue 2026-07-21 16:31:06 MSK 9min Tue 2026-07-21 16:16:25 MSK 4min 45s ago homelab-forum-snuffleupagus-health.timer homelab-forum-snuffleupagus-health.service
--
574-- - Tue 2026-07-21 16:21:10 MSK 161ms ago homelab-health-metrics.timer homelab-health-metrics.service
575-- - Tue 2026-07-21 16:21:10 MSK 161ms ago homelab-private-vpn-hosts-health.timer homelab-private-vpn-hosts-health.service
576:- - - - prometheus-node-exporter-smartmon.timer prometheus-node-exporter-smartmon.service
577---- HEALTH_FILES ---
578-2026-06-17T10:54:28.0506782690 357 /var/lib/homelab-health/netbird-vps-restore.txt
579-2026-06-17T22:53:44.6880508040 1257 /var/lib/homelab-health/app-stacks-batch-offhost.txt
--
998-2026-07-17T04:59:42.6476420400 242 /var/lib/homelab-health/netbird-vps-offhost.txt
999-2026-07-17T09:31:15.5873108310 443 /var/lib/homelab-health/paperless-offhost.txt
1000:2026-07-17T14:21:55.3567278930 272 /var/lib/homelab-health/cluster-storage-retention.txt
1001-2026-07-18T01:52:29.8931662090 684 /var/lib/homelab-health/paperless-backup.txt
1002-2026-07-18T01:54:49.4850009180 728 /var/lib/homelab-health/netbird-vps-trust.txt
1003-2026-07-18T02:54:28.2167554790 200 /var/lib/homelab-health/full-handoff-recovery.txt
--
1192-/var/lib/homelab-health/duty-admin-residual-cleanup-seal.txt:1:STATUS=OK TS=2026-07-08T05:44:22Z TYPE=duty-admin-residual-cleanup-seal BAD=0 WARN=0 MODE=FINAL_RESIDUAL_NOISE_SEAL RESIDUAL_SEAL_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 REVIEW_ITEMS_INTENTIONALLY_LEFT=yes LEGACY_PHASE_ITEMS_INTENTIONALLY_LEFT=yes FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 PVE03_CAPACITY_FORENSICS_STATUS=OK LOCAL_CLEANUP_OK=YES REMOTE_CLEANUP_OK=YES DIRTY_COUNT=0 REPORT=/var/lib/homelab-duty-admin-residual-cleanup/seal-latest.md
1193-/var/lib/homelab-health/prometheus-npmplus-kuma-supersede-repair-20260709T073019Z/npmplus-kuma-config-backup.txt.bak:1:STATUS=ERROR TS=20260709T072619Z TYPE=npmplus-kuma-config-backup NOTES=edge_backup_failed
1194:--- FILESYSTEMS ---
1195:Filesystem Type Size Used Avail Use% Mounted on
1196-udev devtmpfs 15G 0 15G 0% /dev
1197-tmpfs tmpfs 3.2G 3.8M 3.2G 1% /run
1198-/dev/mapper/pve-root ext4 98G 40G 53G 44% /
--
1202-tmpfs tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-journald.service
1203-tmpfs tmpfs 16G 47M 16G 1% /tmp
1204:/dev/nvme0n1p2 vfat 1022M 9.1M 1013M 1% /boot/efi
1205-/dev/mapper/pve-homelab--staging ext4 295G 148G 133G 53% /mnt/staging
1206-/dev/fuse fuse 128M 1.4M 127M 2% /etc/pve
1207:shm tmpfs 63M 0 63M 0% /var/lib/containers/storage/overlay-containers/fbb240ea5b547fa140d8edbc3b16daaf4b28a21c7ee424abedab068865a95d5b/userdata/shm
1208:overlay overlay 98G 40G 53G 44% /var/lib/containers/storage/overlay/4f68ccfa2802b35db2a4c6b396e477f756cb85aa42fa818dd111745ec503616b/merged
1209-tmpfs tmpfs 1.0M 0 1.0M 0% /run/credentials/getty@tty1.service
1210-tmpfs tmpfs 3.2G 4.0K 3.2G 1% /run/user/0
1211:Filesystem Inodes IUsed IFree IUse% Mounted on
1212-udev 3.6M 733 3.6M 1% /dev
1213-tmpfs 3.9M 1.4K 3.9M 1% /run
1214-/dev/mapper/pve-root 6.3M 128K 6.2M 2% /
--
1218-tmpfs 1.0K 1 1023 1% /run/credentials/systemd-journald.service
1219-tmpfs 1.0M 3.5K 1021K 1% /tmp
1220:/dev/nvme0n1p2 0 0 0 - /boot/efi
1221-/dev/mapper/pve-homelab--staging 19M 4.3K 19M 1% /mnt/staging
1222-/dev/fuse 256K 89 256K 1% /etc/pve
1223:shm 3.9M 1 3.9M 1% /var/lib/containers/storage/overlay-containers/fbb240ea5b547fa140d8edbc3b16daaf4b28a21c7ee424abedab068865a95d5b/userdata/shm
1224:overlay 6.3M 128K 6.2M 2% /var/lib/containers/storage/overlay/4f68ccfa2802b35db2a4c6b396e477f756cb85aa42fa818dd111745ec503616b/merged
1225-tmpfs 1.0K 1 1023 1% /run/credentials/getty@tty1.service
1226-tmpfs 796K 28 796K 1% /run/user/0
1227- total used free shared buff/cache available
--
1229-Swap: 9Gi 1.7Gi 8.3Gi
1230- 16:21:11 up 5 days, 4:42, 3 users, load average: 0.26, 0.37, 0.42
1231:--- STORAGE_HEALTH ---
1232-no pools available
1233-Personalities :
1234-unused devices: <none>
--
1291---- FAILED_UNITS ---
1292---- TIMERS ---
1293:Tue 2026-07-21 16:26:07 MSK 4min 53s Tue 2026-07-21 16:21:07 MSK 6s ago homelab-smartctl-textfile.timer homelab-smartctl-textfile.service
1294:Tue 2026-07-21 16:27:30 MSK 6min Tue 2026-07-21 16:12:30 MSK 8min ago prometheus-node-exporter-smartmon.timer prometheus-node-exporter-smartmon.service
1295-Tue 2026-07-21 18:41:03 MSK 2h 19min Tue 2026-07-21 12:41:22 MSK 3h 39min ago forum-prod-fresh5-cloud-backup.timer forum-prod-fresh5-cloud-backup.service
1296-Wed 2026-07-22 00:00:00 MSK 7h Tue 2026-07-21 00:00:22 MSK 16h ago dpkg-db-backup.timer dpkg-db-backup.service
1297-Wed 2026-07-22 04:20:00 MSK 11h Tue 2026-07-21 04:20:12 MSK 12h ago homelab-cluster-admin-vm-mail-cloud-backup.timer homelab-cluster-admin-vm-mail-cloud-backup.service
--
1310-2026-07-21T16:21:07.1497113500 108 /var/lib/homelab-health/routerguard-ap.txt
1311-/var/lib/homelab-health/codevipe-cloud-backup-reset.txt:1:STATUS=OK TS=2026-07-08T05:25:22Z TYPE=remote-failed-unit-codevipe-reset BAD=0 WARN=0 MODE=RESET_STALE_FAILED_STATE UNIT=codevipe-cloud-backup.service REASON=last_journal_success_and_no_timer ACTION=reset_failed_only
1312:--- FILESYSTEMS ---
1313:Filesystem Type Size Used Avail Use% Mounted on
1314-udev devtmpfs 6.8G 0 6.8G 0% /dev
1315-tmpfs tmpfs 1.6G 3.1M 1.6G 1% /run
1316-/dev/mapper/pve-root ext4 98G 26G 68G 28% /
--
1325-tmpfs tmpfs 1.0M 0 1.0M 0% /run/credentials/getty@tty1.service
1326-tmpfs tmpfs 1.6G 4.0K 1.6G 1% /run/user/0
1327:Filesystem Inodes IUsed IFree IUse% Mounted on
1328-udev 1.7M 653 1.7M 1% /dev
1329-tmpfs 2.0M 1.3K 2.0M 1% /run
1330-/dev/mapper/pve-root 6.3M 80K 6.2M 2% /
--
1343-Swap: 9Gi 1.8Gi 8.2Gi
1344- 16:21:13 up 5 days, 4:42, 1 user, load average: 0.01, 0.04, 0.00
1345:--- STORAGE_HEALTH ---
1346-no pools available
1347-Personalities :
1348-unused devices: <none>
--
1405-● homelab-immutable-offline-verify.service loaded failed failed Verify immutable offline backup image
1406---- TIMERS ---
1407:Tue 2026-07-21 16:21:41 MSK 25s Tue 2026-07-21 16:16:41 MSK 4min 34s ago homelab-smartctl-textfile.timer homelab-smartctl-textfile.service
1408:Tue 2026-07-21 16:28:31 MSK 7min Tue 2026-07-21 16:13:31 MSK 7min ago prometheus-node-exporter-smartmon.timer prometheus-node-exporter-smartmon.service
1409-Wed 2026-07-22 00:00:00 MSK 7h Tue 2026-07-21 00:00:10 MSK 16h ago dpkg-db-backup.timer dpkg-db-backup.service
1410-Sun 2026-07-26 03:10:18 MSK 4 days Sun 2026-07-19 03:10:41 MSK 2 days ago e2scrub_all.timer e2scrub_all.service
1411-Sun 2026-07-26 03:10:55 MSK 4 days Sun 2026-07-19 03:11:07 MSK 2 days ago xfs_scrub_all.timer xfs_scrub_all.service
1412-Mon 2026-07-27 00:08:17 MSK 5 days Mon 2026-07-20 00:35:49 MSK 1 day 15h ago fstrim.timer fstrim.service
1413---- HEALTH_FILES ---
1414:--- FILESYSTEMS ---
1415:Filesystem Type Size Used Avail Use% Mounted on
1416-udev devtmpfs 15G 0 15G 0% /dev
1417-tmpfs tmpfs 3.2G 3.0M 3.2G 1% /run
1418-/dev/mapper/pve-root ext4 98G 53G 41G 57% /
--
1427-tmpfs tmpfs 1.0M 0 1.0M 0% /run/credentials/getty@tty1.service
1428-tmpfs tmpfs 3.2G 4.0K 3.2G 1% /run/user/0
1429:Filesystem Inodes IUsed IFree IUse% Mounted on
1430-udev 3.6M 747 3.6M 1% /dev
1431-tmpfs 4.0M 1.3K 4.0M 1% /run
1432-/dev/mapper/pve-root 6.3M 78K 6.2M 2% /
--
1445-Swap: 9Gi 3.0Gi 7.0Gi
1446- 16:21:16 up 5 days, 4:40, 1 user, load average: 0.54, 0.48, 0.44
1447:--- STORAGE_HEALTH ---
1448-no pools available
1449-Personalities :
1450-unused devices: <none>
=== OPEN_ERROR_REGISTER_MARKERS ===
=== ERROR_REGISTER_OPEN_MARKERS ===
1:# HOMELAB ASSISTANT ERROR REGISTER
35:11. Ошибка: считать offhost OK после failed rsync.
77:21. Ошибка: nested PHP php -r дал Parse error на forum-prod.
83:Команда 108 дала PHP Parse error на smtpHost и пустой proof-файл.
88:Команда 111 не изменила smtpSsl: OLD_SMTPSSL=false, SQL ERROR 1064, NEW_SMTPSSL=false.
93:Нельзя использовать { echo RULE_CHECK_FAIL; exit 1; } в командах, выполняемых прямо в интерактивном root@pve01 shell.
94:При failed-check использовать безопасный шаблон: if grep ...; then echo RULE_CHECK_OK; ...; else echo RULE_CHECK_FAIL; fi.
100:Сначала проверять AUTH через openssl s_client -starttls smtp, без вывода секрета.
102:26. Ошибка: openssl -crlf вместе с ручным CRLF ломает SMTP AUTH checker.
109:Команда 138 пересобрала dashboard, но диагностический Python-блок упал с NameError из-за кавычек.
126:Перед любым действием команда обязана проверять /etc/pve/HOMELAB_ASSISTANT_ERROR_REGISTER.md и текущий 31_HOMELAB_REFERENCE.md.
127:Команда обязана печатать ERROR_REGISTER_CHECK=OK и REFERENCE_CHECK=OK до основного действия.
130:32. Ошибка: 153_PROMETHEUS_TARGETS_SETTLED_PROOF содержит SyntaxError из-за вложенного Python внутри SSH.
137:Такие выводы не являются инфраструктурным состоянием; считать невалидными только соответствующие failed command attempts.
150:Факт: terminal output around storage block creation showed possible paste/truncation artifact near smartctCK=FAIL.
154:37. Coverage gap: pve03_staging missing from disk-space health coverage.
159:Факт: terminal output around command 222 showed paste artifact near "FAIL; fi".
163:Факт: строка справочника с фразой No unresolved final REVIEW_REQUIRED... SyntaxError... была принята за BAD_MARKER.
166:40. Ошибка/вывод: VM150 Mail cloud single-file upload failed with 413.
171:Факт: first reconciliation wrote STATUS=ERROR because router-running-config-mail-cloud.txt is on pve01, not edge-vm.
180:## HOME_PORTAL_BASE64_APPLY_FAILURE_20260630
181:- Prior home-portal apply attempt 312 failed because a large base64 Python payload decoded as invalid UTF-8.
188:- Any Traceback, UnicodeDecodeError, SyntaxError, Permission denied, command-not-found, nonzero critical RC, or failed write marker must force REVIEW or FAILED, never OK.
191:- If an apply step fails, the next command must explicitly record the failed attempt and then run a corrected transparent apply command.
193:## HOME_PORTAL_API_ERROR_CLOSURE_RULE_20260630
194:- Rule: home portal closure requires current Homepage API-error logs to be zero after restart/reload, not only HTTP 200.
195:- If Homepage UI shows API error or logs contain current httpProxy/API timeout errors, the portal is REVIEW until the widget/API cause is disabled or fixed.
196:- External informational widgets such as weather/Open-Meteo must not block the service launcher portal.
198:## HOME_PORTAL_COMPLEX_QUOTING_FAILURE_20260630
199:- Previous home portal cleanup command failed in local bash before execution: syntax error near unexpected token `('.
211:- A portal card is openable only when DNS, reverse-proxy host mapping, and target content are all valid.
213:## ROUTER_CLI_STDIN_APPLY_FAILURE_20260630
216:- Failure: STDIN_SHOW_OK=no, APPLY_SKIPPED=stdin_show_failed, CONFIG_HAS_EDGE_5080=no, CONFIG_HAS_EDGE_5083=no.
219:## ROUTER_CLI_PROBE_STDIN_CONSUMPTION_ERROR_20260630
227:- Observed: CMD_TEXT=access-list ? returned Core::Configurator error execute denied [cli].
232:- Evidence: 395_ROUTER_CLI_ERROR_REGISTER_UPDATE_PROOF.txt has blank P393_COMMAND_COUNT/P394_COMMAND_COUNT/P394_ZERO_RC_COUNT fields.
243:- Failure: NODE_HTTPS_RESOLVE_ERROR=ERR_INVALID_IP_ADDRESS and NODE_HTTPS_RESOLVE_INSECURE_ERROR=ERR_INVALID_IP_ADDRESS.
254:## PY_COMPILE_PYC_PERMISSION_ERROR_20260701
257:- Actual impact: non-root syntax check attempted to create /usr/local/sbin/__pycache__ and failed with Permission denied.
273:## HOMEPAGE_ROUTER_MOSCOW_APPLY_SCRIPT_SYNTAX_ERROR_20260701
276:- Evidence: proof 408 showed SyntaxError in /tmp/homepage-router-moscow-apply-fixed.py.
284:- Rule: do not proceed with OS baseline until SSH failure is diagnosed; likely old known_hosts key or cloud-init/root-key issue.
307:- Additional failed approach: proof 480 ran xf-dev:rebuild-caches on CodeVipe and returned RC=1, not a valid fix.
313:- Issue: curl returned HTTP 403; unzip then failed because the ZIP was not downloaded.
318:## PVE01_UNZIP_MISSING_FOR_XF_REQUIREMENTS_VERIFY_20260701
327:- Note: proof 511 initially failed dsmods path detection; proof 512 fixed dsmods by locating public_html via src/XF.php marker.
330:## NPMPLUS_SQLITE_PASTE_FAILURE_20260701
332:- Issue: shell entered multiline prompt and produced syntax errors.
343:- Context: NPMplus API login attempts in proofs 526/527 failed.
349:- Context: NPMplus API login attempts failed even though the forum VM and edge-to-forum connectivity are healthy.
357:- Impact: certbot ran without token and all certificate attempts failed; DNS cutover must not proceed from proof 535.
360:## EDGE_CERTBOT_FAILED_DNS_CUTOVER_ABORTED_20260701
361:- Context: edge manual routes are working locally, but certbot DNS-01 certificate issue failed in proof 538.
368:- Evidence: proof 542 showed all five zones missing and DNS record create probes failed.
369:- Impact: certbot DNS-01 failed and DNS cutover was correctly aborted; public DNS still points to old hosting.
375:- Important corrections: initial NPMplus API route failed; manual Nginx routes were used. Initial Cloudflare token had no zone access; corrected token passed zone/DNS probe.
379:- Issue: Proxmox snapshot failed because generated snapshot name exceeded 40 characters.
388:- Impact: web routing is OK, but mail-related DNS may still contain stale provider data.
392:- Context: SPF cleanup dry-run command 563 had broken shell quoting and left the terminal at multiline prompt.
397:- Context: SPF cleanup command 565 attempted to replace stale SPF records.
398:- Issue: backup write permission problem caused record detection failure, so new v=spf1 -all records were created while old v=spf1 ip4:87.236.18.45 records remained.
402:## MSMTP_SECRET_SOURCE_PARSE_ERROR_576_20260701
403:- Context: proof 576 installed msmtp but sendmail auth test failed.
404:- Issue: [SENSITIVE_PATH] was sourced as shell, but SMTP password contained shell-special characters; source failed and auth used an invalid/truncated secret path.
408:## FORUM_MSMTP_MAIL_TRANSPORT_STILL_FAILING_20260701
410:- Issue: one or more sendmail/PHP mail tests still failed.
415:- Context: SMTP password was exposed in terminal output during failed msmtp setup.
416:- Cause: secret file with raw SMTP_PASS was later parsed/sourced by shell; shell error printed the raw assignment line.
426:## FORUM_SMTP_FAILED_CONFIG_DISABLED_AFTER_MAILCOW_BAN_20260701
429:- Action: remove forum-prod msmtp secret/config files so XenForo cannot keep retrying broken SMTP auth.
432:## FORUM_OLD_CODEVIPE_CLOUD_BACKUP_STALE_AFTER_FRESH5_20260701
435:- Evidence: latest service run failed with missing /root/backups/codevipe; remote inventory contains old codevipe-public/codevipe.sql files, not current five-forum archive.
445:## FORUM_XENFORO_MAIL_SMOKE_TEST_FAILED_20260701
450:- Context: custom XenForo mail smoke test 623 failed with Could not open input file /root/scripts/623_xenforo_mail_smoke.php.
456:## FORUM_NEXT_CHAT_KNOWN_ERRORS_AND_CLOSED_INCIDENTS_20260701
466: - Custom mail proof 623 failed with "Could not open input file".
471:- FORUM_MSMTP_MAIL_TRANSPORT_STILL_FAILING_20260701 / FORUM_SMTP_FAILED_CONFIG_DISABLED_AFTER_MAILCOW_BAN_20260701:
472: - Earlier SMTP attempts failed and triggered mailcow netfilter ban for forum public IP.
489: - Classified as external timeout noise, not runtime failure.
493:- Classification: deliverability/reputation/content filtering, not server failure.
509:## PARKED_DOMAINS_STAGE4_DNS01_PREFLIGHT_FAILED_20260701
512:- Observed: stage4 stopped safely with preflight_failed; CERTBOT_PRESENT=no, CERTBOT_DNS_CLOUDFLARE_PLUGIN=no, and CF_ZONE_ACCESS=no for newfi.ru, hapusya.ru and kingofwolk.ru.
513:- Impact: no parked-domain NPMplus route/certificate changes were applied by stage4; public HTTPS validation still failed certificate verification.
550:- Stage9 combined server_name block passed 5/6 but failed bare newfi.ru, so final HTTP-01 cutover must use separate exact per-host server blocks.
553:## PARKED_DOMAINS_STAGE11_LOW_ID_EXACT_ROUTE_FAILED_20260701
560:## PARKED_DOMAINS_SWITCH_TO_DNS01_AFTER_HTTP01_FAILURES_20260701
569:- Rule: for parked domains finalization, use existing DNS-01 certificates and validate from pve01 public HTTPS with content marker; do not use the failed local --resolve TLS check as the closure gate.
579:- Stage18 parked dry-run note: kingofwolk.ru dry-run hit Let's Encrypt rateLimited/service-busy after active certificate issuance and public HTTPS were already OK. Treat as transient external dry-run failure, not route/cert closure blocker.
587:## PVEPRO_STAGE21_LANDING_SERVICE_FAILED_20260701
593:- Context: Stage22 succeeded on edge-side landing, cert, NPMplus route and Cloudflare A upserts, but wrapper had a non-critical scp wildcard failure due __pycache__ and public validation used cached DNS for root pvepro.ru.
594:- Mistakes recorded: do not scp scripts/* after local py_compile because __pycache__ may be present; do not continue after critical scp failure in future apply stages; do not classify PVEPro landing as NetBird just because it contains a link text; use Cloudflare authoritative A records and --resolve validation when DNS propagation may lag.
613:- Rule: do not open router public admin or certificate deployment until WireGuard handshake is proven.
617:- Proof 666 is superseded because it showed PUBLIC_SSH_22_STILL_OPEN.
620:## TAFTAUTO_CERTBOT_CLOUDFLARE_PLUGIN_MISSING_20260702
621:- Certbot on edge rejected --dns-cloudflare-* arguments: dns-cloudflare plugin missing/not loaded.
622:- No taftauto certificate issued in failed run; no DNS A records or NPMplus routes changed.
634:- No certificate, DNS, NPMplus route, or deploy state changed by the failed dry-run.
637:- Proof 672 is superseded/partial: it printed STATUS=OK even though certbot dry-run failed.
638:- Failure reason: Let's Encrypt staging/service returned rateLimited / Service busy; retry later.
643:## HOMEPAGE_XENFORO_INFO_INLINE_PYTHON_SYNTAX_ERROR_20260702
645:- Failed before services.yaml write.
649:- Failed before write. Fix uses YAML-aware edit with exact names Router/Public Domain.
655:## 20260702_CF_TOKEN_AUDIT_BROKEN_COMMAND
657:- Факт: bash начал интерпретировать Python-код, появились syntax error и вторичный prompt >.
671:- Do not touch Cloudflare Homepage card when operator says it is green and opens correctly.
675:- When asked to fix one missing Uptime Kuma service, add only one monitor, then generate proposals separately.
678:- Do not touch Cloudflare when operator says it is green and opens correctly.
680:## LESSON_20260702_DOCKGE_STALE_STACKS_NOT_CONTAINERS
681:- Dockge inactive items after migration can be stale compose folders, not stopped containers.
684:- Do not delete containers to fix Dockge inactive UI; connect agents and archive only confirmed stale moved stack definitions.
686:## STAGE4B_MASKED_SQL_QUERY_FAILURE_20260714
688:- Empty blocks are query failures, not proof that metadata, triggers or functions are absent.
689:- Future PostgreSQL audits must use ON_ERROR_STOP, visible stderr, independent query RC and quote-safe dollar literals.
693:- PostgreSQL returned column id does not exist and ON_ERROR_STOP correctly stopped all following SQL checks.
698:- pg_dump failed only because fsync on /dev/null returned Invalid argument.
702:## STAGE4C_REMOTE_PREP_THIRD_SUBSTEP_FAILED_20260714
713:## STAGE4C_MIGRATION003_FINGERPRINT_DOLLAR_QUOTE_FAILURE_20260714
714:- Migration 003 isolated dry-run failed at fingerprint CHECK because the regex end anchor collided with the SQL dollar-quote delimiter and produced an unterminated dollar-quoted string.
727:- This false expectation caused diagnostic RC=32 without any infrastructure failure or production change.
737:- The six wrapper validation failures are cascading missing-marker checks and do not prove six independent infrastructure faults.
750:- The exact traceback and failing assertion must be captured by an isolated identical reproduction before changing the candidate.
753:- Adapter candidate passed Python 3.13 but failed Python 3.11 parsing at Path(row["path"]) inside a double-quoted f-string.
755:- The reproduction command captured the expected compile failure but incorrectly returned CAPTURE_BAD=1 because its validator required compile RC=0.
761:- The wrapper validation count is cascading missing-marker evidence and does not represent twenty-one independent faults.
767:- This caused UnicodeDecodeError before any design assertion failed.
772:## STAGE4E_V2_VALIDATOR_FAILED_STATIC_COMPILE_20260714
773:- Stage4E design candidate v2 failed static compilation before local design validation started.
774:- The reported validation count is cascading missing-marker evidence, not twenty-five independent design defects.
775:- The generated validator must be inspected at the exact SyntaxError line before another candidate is created.
787:- The executed backup script does not write that health file; therefore the stale health timestamp was not evidence of backup failure.
792:- Failure class: переход к следующей задаче при наличии незакрытого хвоста.
794:- Closure gate: UNRESOLVED_TAIL_COUNT=0; BLOCKER_COUNT=0; TEMPORARY_ARTIFACT_COUNT=0; ROLLBACK_VERIFIED=YES; HEALTH=OK; PROOF_UPDATED=YES; REFERENCE_UPDATED=YES.
795:- Если закрытие невозможно из-за внешней зависимости, опасной неоднозначности или обязательного решения пользователя, фиксировать BLOCKED/OPEN и не заявлять CLOSED.
798:## ERROR_20260714_DEPENDENCY_FACT_QUERY_GLOBAL_LIMIT
806:## ERROR_20260714_COLLECTOR_PATCH_REQUIREMENT_PATTERN_MISMATCH
812:- Status: CLOSED; PRIOR_TAIL_UNRESOLVED_COUNT=0.
814:## ERROR_20260714_PRIVILEGED_FILE_REDIRECTION_BEFORE_SUDO
816:- Root cause: `< collector.py` was opened by the unprivileged caller shell before sudo executed wc.
822:## ERROR_20260714_COLLECTOR_RUN_COLUMN_ASSUMPTION
823:- Symptom: SQL failed because completed_at did not exist.
824:- Root cause: collector_runs columns were assumed instead of read from information_schema; actual names are finished_at and error_text.
828:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
830:## ERROR_20260714_REMOTE_STDIN_ARGUMENT_SHIFT
836:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
838:## ERROR_20260714_OVERSIZED_COMMAND_PARSE_FAILURE
839:- Symptom: Bash получил одиночный фрагмент и сообщил syntax error near unexpected token `(`.
845:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
847:## ERROR_20260714_HOMELAB_ADMIN_HELP_RC_ASSUMPTION
848:- Symptom: template audit завершился HOMELAB_ADMIN_HELP_FAILED.
854:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
856:## ERROR_20260715_EXPECTED_NEGATIVE_RC_TRAP_CLASSIFICATION
859:- Correction: expected failures are evaluated only through an explicit if/else branch with immediate RC capture.
862:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
864:## ERROR_20260715_DUPLICATE_MACHINE_MARKER_COUNT
865:- Symptom: error-register candidate construction stopped with RC1 before applying the candidate.
872:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
875:## ERROR_20260715_OUTER_WRAPPER_SHELL_SYNTAX
876:- Symptom: the operator shell rejected the generated one-line wrapper with syntax error near the final subshell parenthesis.
878:- Impact: parsing failed before the start marker, guards and homelab-admin invocation; no task phase or infrastructure action executed.
881:- Anti-regression: after any parser-level failure, treat the attempted phase as not executed and verify package and run-state before retrying.
885:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
888:## ERROR_20260715_INLINE_REMOTE_SHELL_IN_APPLY_PHASE
899:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
902:## ERROR_20260715_OPTIONAL_RUN_DIRECTORY_FIND_UNDER_PIPEFAIL
904:- Root cause: find was executed against an optional directory that did not exist; under pipefail the assignment returned RC1.
906:ANTI_REGRESSION_TOKEN=MISSING_OPTIONAL_RUN_DIRECTORY_TREATED_AS_ZERO
907:- Anti-regression: optional paths must have an explicit existence branch before find under pipefail.
909:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
912:## ERROR_20260715_PG_CONSTRAINT_CONTYPE_CHAR_CONCAT
913:- Symptom: isolated Stage4H acceptance failed during schema baseline capture.
922:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
925:## ERROR_20260715_LIVE_COLLECTOR_VERIFIER_WRONG_PATH
933:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
936:AUTOREFRESH_INLINE_FAILURE_20260721
940:AUTOREFRESH_WRAPPER_PUBLICATION_FAILURE_20260721
948:BACKUP-AUDIT-CONTROLPLANE-054 was successfully published at commit 2b966acfe553b4d9f8d4a62bf9533478bdd1a073, but was incorrectly reported missing because a stale directory listing was treated as authoritative.
951:BACKUP_UNSAFE_SOURCE_CAPTURE_071_PARSE_ERROR_20260721
952:BACKUP-UNSAFE-SOURCE-CAPTURE-071 failed before execution: shell syntax error near unexpected token (.
=== REFERENCE_TAIL ===
=== RECENT_FAILED_COMMANDS ===
=== RECENT_FAILED_COMMANDS ===
2026-07-21T08:03:07Z GITEA-PUSH-AUTH-001 fcfc28bcc5acb6382e895d5ed7cf51d7d5a3e2f5d8f8332770c58b42039f8a94 1 FAIL gitea-runtime
2026-07-21T08:13:32Z GITEA-TOKEN-CLI-HELP-003 f1d0f967eed6490ad822944a42108c82dc4b59c9fd2557d8c64624dcb6853cd4 1 FAIL gitea-runtime
2026-07-21T08:14:52Z GITEA-CLI-CAPABILITIES-004 2cd6c4d28c8f646d8063105fca80b2f80d38c6a5db3a3eb76869de326b12d2d6 1 FAIL gitea-runtime
2026-07-21T09:49:51Z RUNTIME-RECEIVER-STATE-027 0831002c78c26f35826cd7199995e335ceadcd39f08b9bac23882a0fbb59c8cc 127 FAIL gitea-runtime
2026-07-21T10:26:18Z RUNTIME-RECEIVER-CLEANUP-036 65fbcfbce38a55723a92a8790429d6567389321f030f02cb4bf301cbe67e2531 1 FAIL gitea-runtime
2026-07-21T11:35:23Z BACKUP-DUAL-CLOUD-PREFLIGHT-059 c6f4134d15a9f641ddaf67535760d0abede973dd08222a37af05612a6f580479 1 FAIL backup-audit
2026-07-21T12:28:26Z BACKUP-VM-DUMP-PIPELINE-073 cd8291d0f14a37241e6062408603455d1ebd483091bebfa46c2b7a768f21308b 1 FAIL backup-audit
2026-07-21T12:42:53Z BACKUP-VM-DUMP-CHANGESET-PREFLIGHT-075 a9a8a6046441de0c40a8b13bdfe96698c53a5156b4346b7ce28dc1f2362e8936 143 FAIL backup-audit
2026-07-21T12:56:52Z BACKUP-DRAIN-LOCAL-ARCHIVES-079 531e1da7ce1681eed084b22ced7da87eb1ee8995f662dc7559f4f9dde3c0ac32 75 FAIL backup-remediation
2026-07-21T13:04:48Z BACKUP-FINAL-PROOF-082 9e675a3cb621d0d1fcd8918b02278ab08c1a974f544ebef9a5639bf6578ee601 127 FAIL backup-audit
2026-07-21T13:07:09Z CLUSTER-CLEANUP-INVENTORY-083 09d3f14ca0205936a910dc7af7991474030273a47d76fa033ea6a3185c083922 1 FAIL cluster-cleanup
2026-07-21T13:14:27Z CLUSTER-BACKUP-POSTCLEAN-PROOF-086 601958c48614dbc25cc4592b111de4dec3306e53b756b275068b81a41d42b2da 127 FAIL backup-cleanup
2026-07-21T13:15:21Z BACKUP-POLICY-CLOSEOUT-087 d79c1c881ebca24db09c5b4ed4b71b09a5ad338c30789c99b0644c8935072c52 1 FAIL backup-cleanup
2026-07-21T13:17:20Z BACKUP-CLEANUP-FINAL-CLOSEOUT-088 5ba2893489b6f5831fc4599d3efa874d89c3fd41da7151f9c32945d11d1a23d6 1 FAIL backup-cleanup
CRITICAL_TAILS_BASELINE=COMPLETE
=== BASELINE_COMPLETION_MARKERS ===
1132:/var/lib/homelab-health/operator-handoff-pack.txt:1:STATUS=OK TS=2026-07-08T09:42:39Z TYPE=operator-handoff-pack BAD=0 WARN=0 MODE=FINAL_OPERATOR_HANDOFF_PACK HANDOFF_OK=YES PACK_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T093758Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T093758Z RECOVERY_CARD=YES FREEZE_PACK=YES FINAL_CLOSEOUT=YES INDEPENDENT_RUNNER_NO_DEPLOY=YES DESIRED_STATE_REMOTE_GIT=YES GIT_REMOTE_CHANGED=NO GIT_PUSH_DONE=NO DIRTY_COUNT=0 PACK=/var/lib/homelab-operator-handoff/operator-handoff-pack-20260708T094239Z.tar.gz README=/var/lib/homelab-operator-handoff/HANDOFF-latest.md STATUS_TSV=/var/lib/homelab-operator-handoff/handoff-status-latest.tsv NEXT=/var/lib/homelab-operator-handoff/NEXT-ACTIONS-latest.md
1133:/var/lib/homelab-health/pve01-failed-units.txt:1:STATUS=OK TS=20260624T180727Z TYPE=pve01-failed-units FAILED_COUNT=0 ACTION=RESET_FAILED_ONLY SECRET_PRINTED=REDACTED
1134:/var/lib/homelab-health/final-global-green-audit.txt:1:STATUS=OK TS=2026-07-08T07:20:01Z TYPE=final-global-green-audit BAD=0 WARN=0 MODE=SELF_REFERENCE_CLOSED_FINAL_GLOBAL_GREEN_SEAL AUDIT_OK=YES PRE_STATUS=WARN SELF_REFERENCE_CLOSED=yes LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 EXACT_WARN_REVIEW_FILES=0 LOCAL_OVERALL_OK=1 LAN_OVERALL_OK=1 PUBLIC_OVERALL_OK=1 EXTERNAL_PROBE_STATUS=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T064444Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T064444Z DIRTY_COUNT=0 REPORT=/var/lib/homelab-final-global-green-audit/final-seal-latest.md
1135:/var/lib/homelab-health/remote-failed-unit-audit.txt:1:STATUS=OK TS=2026-07-08T05:20:16Z TYPE=remote-failed-unit-audit BAD=0 WARN=1 MODE=READ_ONLY_REMOTE_FAILED_UNIT_AUDIT LOCAL_FAILED_COUNT=1 REMOTE_FAILED_COUNT=8 CONNECT_ERRORS=0 JOURNAL_FAIL_HINTS=194 OVERALL_BAD=0 FINAL_FAIL=0 DUTY_V2_GATED_NONOK=0 RESET_FAILED_DONE=no REPORT=/var/lib/homelab-remote-failed-unit-cleanup/audit-latest.md
1136:/var/lib/homelab-health/legacy-monitoring-backups-migrated.txt:1:STATUS=OK TS=2026-07-10T20:00:03Z TYPE=legacy-monitoring-backups-migrated BAD=0 APPS=6 RESTORE_FAILURES=0 ACTIVE_LEGACY_STATES=0 OFFHOST=OK RESTORE=OK
1137:/var/lib/homelab-health/cluster-admin-stage4h-collector-patch-candidate.txt:1:STATUS=OK TS=2026-07-14T17:41:32Z TYPE=cluster-admin-stage4h-collector-patch-candidate FAILED_ATTEMPT_TAIL=CLOSED CANDIDATE_SUBSTEP=CLOSED STAGE4H_STATUS=OPEN PRODUCTION_CHANGE=NO UNRESOLVED_TAIL_COUNT=0 BLOCKER_COUNT=0 NEXT_ACTION=TEMPORARY_DATABASE_MIGRATION_AND_COLLECTOR_ACCEPTANCE_HARNESS CANDIDATE_SHA256=4204fe568d46fec84f643af974c82770cb5d5b08d3d8ceb4aad96cca57abc909 REPORT_SHA256=ecfdaab5438e1fc3bc7a90e08ec68c71d06a6c6a91a430dc1ea7b6a74b7699a9
1138:/var/lib/homelab-health/cluster-admin-stage2-final-seal.txt:1:STATUS=OK TS=2026-07-13T01:45:29Z TYPE=cluster-admin-stage2-final-seal BAD=0 STAGE1_CLOSED=YES STAGE2_APPLY=OK PRODUCTION_CHANGED=BACKUP_HEALTH_AND_SEAL_ARTIFACTS_ONLY PRODUCTION_SCHEMA_CHANGED=NO_ADDITIONAL_CHANGE_DURING_SEAL INCIDENT_ENGINE_RESTARTED=NO MIGRATION_ID=002_canonical_foundation_sync FUNCTION=cluster_admin_sync_foundation COMMITTED_SYNC=YES MULTICYCLE_COLLECTOR_TEST=OK MULTICYCLE_NEW_RUNS=2 MULTICYCLE_GOOD_RUNS=2 ACTIVE_CURRENT_SET_CONVERGENCE=OK NATURAL_KEY_CONVERGENCE=OK EPHEMERAL_ID_REFRESH=OK RETIRED_PLACEHOLDERS_ALLOWED=YES FOUNDATION_COUNTS=85|21|2|229|26|0 DYNAMIC_INVARIANTS=0|0|21|21|0|0|0|0|0|0|0 ORPHANS=0 POSTAPPLY_DIRECT_SNAPSHOT=VERIFIED POSTAPPLY_RESTORE_TEST=OK POSTAPPLY_APPBACKUP=VERIFIED POSTAPPLY_APPBACKUP_JOB=homelab-pve01-app-cluster-admin-incident-engine-20260713T014757Z LIVE_MATCH_DESIRED_STATE=YES DESIRED_STATE_HEAD=8f888e5f2063e49da918b31584c9ea4bcf53af3a OBSERVER=OK FULL_OBSERVER=OK RESTRICTED_PROBES=OK FINAL_READINESS=OK CLUSTER_PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK VM180_FAILED_UNITS=0 PVE01_FAILED_UNITS=0 CURRENT_HEALTH_RECHECKED=YES STAGE2_CLOSED=YES REPORT=/var/lib/homelab-cluster-admin-stage2/stage2-final-seal-latest.txt REPORT_SHA256=32d4d37d82aa25194d789083d0e7ad2e9052ee38a72df5312b711226898cf1a6 BLOCKERS=/var/lib/homelab-cluster-admin-stage2/20260713T014529Z/blockers.txt
1139:/var/lib/homelab-health/remote-failed-unit-plan.txt:1:STATUS=OK TS=2026-07-08T05:22:47Z TYPE=remote-failed-unit-plan BAD=0 WARN=1 MODE=READ_ONLY_CLEANUP_PLAN LOCAL_FAILED_COUNT=1 REMOTE_FAILED_COUNT=7 LOCAL_NOOP_COUNT=4 REMOTE_RESET_ONLY_COUNT=1 REMOTE_NOOP_COUNT=5 REMOTE_DEFER_COUNT=1 OVERALL_BAD=0 FINAL_FAIL=0 RESET_FAILED_DONE=no PLAN=/var/lib/homelab-remote-failed-unit-cleanup/cleanup-plan-latest.tsv REPORT=/var/lib/homelab-remote-failed-unit-cleanup/plan-latest.md
1140:/var/lib/homelab-health/review-surface-registry-apply.txt:1:STATUS=OK TS=2026-07-08T06:36:39Z TYPE=review-surface-registry-apply BAD=0 WARN=0 MODE=POLICY_AWARE_DUTY_ADMIN_REPAIR_SELF_CLOSED PRECHECK=YES APPLY_OK=YES REPAIR_OK=YES UPDATED=27 ARCHIVED=27 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW_WAS=1 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_BAD=0 SELF_REFERENCE_CLOSED=YES REPORT=/var/lib/homelab-review-surface-registry/seal-latest.md
1141:/var/lib/homelab-health/private-vpn-hosts.txt.bak-ok-wrapper-20260624T132705Z:16:STATUS=ERROR
1142:/var/lib/homelab-health/cluster-admin-strict-final-seal.txt:1:STATUS=OK TS=2026-07-09T04:49:23Z TYPE=cluster-admin-strict-final-seal BAD=0 VMID=180 IP=[PRIVATE_IP] MODE=DEDICATED_CLUSTER_ADMIN_HARDGATED OBSERVER=OK RESTRICTED_PROBES=OK FULL_OBSERVER=OK OVERALL_HARDGATE=YES FINAL_READINESS_HARDGATE=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-ideal-cluster/cluster-admin-strict-final-seal-latest.md
1143:/var/lib/homelab-health/paperless-restore.txt:1:STATUS=STALE
1145:/var/lib/homelab-health/ideal-cluster-noise-seal.txt:1:STATUS=OK TS=2026-07-08T17:41:49Z TYPE=ideal-cluster-noise-seal BAD=0 MODE=POLICY_AWARE_NON_STATUS_ARTIFACT_SEAL_SELF_REFERENCE_CLOSED SELF_REFERENCE_CLOSED=YES NON_STATUS_TOTAL=8 NON_STATUS_OK_INFO=8 NON_STATUS_UNKNOWN=0 WARNBAD_ACCEPTED_POLICY_INFO=13 WARNBAD_NEEDS_REVIEW=0 TOP_GATES_OK=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-ideal-cluster/noise-seal-latest.md
1146:/var/lib/homelab-health/crypto-portfolio-chart-health.txt:1:STATUS=OK TS=2026-07-21T13:18:29Z TYPE=crypto-portfolio-chart-health BAD=0 LIBRARY_OK=true DISK_CACHE_PRESENT=true SYMBOLS=2 STALE_COUNT=2
1147:/var/lib/homelab-health/duty-admin-v2.txt:1:STATUS=OK TS=2026-07-21T13:16:33Z TYPE=duty-admin-v2 BAD=7 WARN=0 MODE=LOCAL_LONG_RUNNING_ANALYZER_POLICY_AWARE SEPARATE_VM=NO TOTAL_HEALTH_INPUTS=497 GATED_OK=45 GATED_NONOK=7 UNGATED_ERROR_REVIEW=2 UNGATED_WARN_REVIEW=3 UNGATED_REVIEW_ITEM=0 LEGACY_OR_PHASE_REVIEW=0 REPORT=/var/lib/homelab-duty-admin-v2/latest.md
1148:/var/lib/homelab-health/cluster-admin-stage3-final-seal.txt:1:STATUS=OK TS=2026-07-13T03:46:30Z TYPE=cluster-admin-stage3-final-seal BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=HEALTH_REFRESH_AND_SEAL_ARTIFACTS_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO LIVE_APPLICATION_CODE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO STAGE2_CLOSED=YES STAGE3_APPLY=OK AUTHENTICATED_PRODUCTION_VERIFY=OK AUTHENTICATED_PRODUCTION_TESTS=25 APP_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 LIVE_MATCH_DESIRED_STATE=YES DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 NEW_GET_ROUTES=7 DATABASE_TRANSACTION_MODE=BEGIN_READ_ONLY OPENAPI_DOCS_EXPOSURE=PRESERVED_DISABLED POSTAPPLY_APPBACKUP=VERIFIED POSTAPPLY_APPBACKUP_JOB=homelab-pve01-app-cluster-admin-incident-engine-20260713T034400Z POSTAPPLY_APPBACKUP_RESTORE_CHECK=OK COLLECTOR_STABILITY=OK LATEST_COLLECTOR_RUN_ID=3114 CANONICAL_CONSISTENCY=0|0|0|0|0 OBSERVER=OK FULL_OBSERVER=OK RESTRICTED_PROBES=OK FINAL_READINESS=OK CLUSTER_PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK VM180_FAILED_UNITS=0 PVE01_FAILED_UNITS=0 CURRENT_HEALTH_RECHECKED=YES STAGE3_CLOSED=YES REPORT=/var/lib/homelab-cluster-admin-stage3/stage3-final-seal-latest.txt REPORT_SHA256=6f67545cd7f0a75b87146c1df74c61c3b88e142f9e53a96b083ec27209e6d265
1149:/var/lib/homelab-health/hapusya-site-production-browser.txt:1:STATUS=OK TS=2026-07-13T11:48:15Z TYPE=hapusya-site-production-browser BAD=0 URL=https://hapusya.ru/ BROWSER_RUNS=5 BROWSER_ERRORS=0 DESKTOP_VIDEO=OK MOBILE_VIDEO=OK STATIC_FALLBACK=OK EVIDENCE_ROOT=/var/lib/homelab-change-sets/hapusya-site-production-browser-v2/20260713T114810Z MANIFEST_SHA256=9f3cdf3063f8724d1d1c1ad619f47d6f98d434c3422a74903585146858a8dfac ROUTE_SHA256=02d65cb0082a04cae282126cc829a002f171e863291c18ba04d788dc2d6282ca UNIT_SHA256=b3b07137b25ac0a0b4bdadadad654cf209387ade86a2ef2cdca58e8a7c0cc212 RELATIVE_TREE_SHA256=9dadf21da041bc4d6ee4a42f5b3ac7719c16bcc39052de32af895c6cd28286cb PRODUCTION_CHANGED=YES CLOSURE_PENDING=POSTCHANGE_BACKUP_RETENTION_DESIRED_STATE_FINAL_SEAL
1150:/var/lib/homelab-health/backup-matrix-authoritative.txt:1:STATUS=FAIL
1151:/var/lib/homelab-health/backup-matrix-desired-state-persistence.txt:1:STATUS=OK TS=2026-07-13T17:40:37Z TYPE=backup-matrix-desired-state-persistence BAD=0 GENERATED_HEAD=af26ec3c4829b1330d49656205ac580ce1706263 GENERATED_PARENT=25568fab24c0594418b3676dc511c8234b4e1ae5 GENERATOR_SHA256=19da7f049b4f7f7286faf292c5419c37cc1e1768c2d8fe36b257ab0cf9c18874 MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 DESIRED_MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 MATRIX_STATUS=OK JOBS_BAD=0 FAILED_UNITS=0 BACKUP_RUN=NO RESTORE_RUN=NO CLOUD_WRITE_RUN=NO RETENTION_RUN=NO ROLLBACK=/var/lib/homelab-incidents/backup-matrix-desired-state-persistence-20260713T174036Z
1152:/var/lib/homelab-health/duty-admin-residual-audit.txt:1:STATUS=OK TS=2026-07-08T05:40:37Z TYPE=duty-admin-residual-audit BAD=0 WARN=3 MODE=READ_ONLY_RESIDUAL_NOISE_AUDIT LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=1 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 OVERALL_STATUS=OK OVERALL_BAD=0 FINAL_FAIL=0 WARN_REVIEW_FILES=19 HARD_GATE_WARN_MATCHES=0 REPORT=/var/lib/homelab-duty-admin-residual-cleanup/audit-latest.md
1153:/var/lib/homelab-health/ungated-health-backlog.txt:1:STATUS=OK TS=2026-07-07T07:45:01Z TYPE=ungated-health-backlog BAD=0 WARN=1 MODE=POST_PVE03_ROOT_CLEANUP_TRIAGE TOTAL=8 SUPERSEDED=4 STALE=2 POLICY_GAP_NEXT_STAGE=1 ACTIONABLE_CAPACITY=1 PVE03_ROOT_USAGE_PERCENT=31 PVE03_STAGING_USAGE_PERCENT=83 REPORT=/var/lib/homelab-ungated-health-backlog/latest.md
1154:/var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-final-seal.txt:1:STATUS=OK TS=2026-07-13T08:37:54Z TYPE=crypto-portfolio-ui-center-alert-currency-final-seal BAD=0 DANGLING_IMAGE_REFERENCE_GATE=OK DESIRED_STATE_SYNC_ARTIFACT_GATE=OK APP=crypto-portfolio PRODUCTION_CHANGED=NO_FINAL_SEAL_ONLY APP_SHA256=44df2b25183b7c5d87ff1516ec2d9cd1d169acde3031f7c2aa1fc4e0c7813cd2 WORKER_SHA256=174a14b86014535980d200a76d0057110c7af102c66f16c2e21fc3e74bb2e49e INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb LIBRARY_SHA256=c0992580867c4912cc9385b3c2728315bcc1a76c7f1087dca908430fccdf31d7 ACTIVE_COINS=HBAR,POL GRT_REMOVED=YES GRT_ROWS=0|0|0 GRT_SOURCE_REFERENCES=0 GRT_CACHE_ENTRIES=0 SHARP_MOVES_CENTERED=YES ALERT_CURRENCY_USD_RUB=OK BROWSER_SMOKE=OK BROWSER_RUNS=2 BROWSER_ERRORS=0 BROWSER_RESULT_SHA256=b566f492cd80c8f99bc6b24cbef6d5f326c04f4d7168359d05eb1f189589e733 SCREENSHOT_SHA256=00b2853bee9c880146d0dd75fac60feb7e46d4c129ee06754047ef30d90bd4ba BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T075801Z TWO_CLOUDS=OK RESTORE_CHECK=OK DATABASE_INTEGRITY=ok RETENTION=OK ROLLBACK_COUNT=3 DANGLING_IMAGES=2 REFERENCED_DANGLING_IMAGES=2 UNREFERENCED_DANGLING_IMAGES=0 PRUNED_UNREFERENCED_DANGLING_IMAGES=0 CHART_HEALTH=OK PUBLIC=OK MONITOR_TIMER=active BACKUP_TIMER=active CLOSURE_DESIRED_STATE_HEAD=2999910549eaff9b9431b9c656d318340eb95cbd DESIRED_STATE=OK PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES ALL_CRYPTO_PORTFOLIO_UI_TAILS_CLOSED=YES
1155:/var/lib/homelab-health/crypto-portfolio-remove-grt-runtime-verifier-prepared.txt:1:STATUS=OK TS=2026-07-13T04:35:29Z TYPE=crypto-portfolio-remove-grt-runtime-verifier-prepared BAD=0 COMMAND_PHYSICAL_LINES=1 VERIFIER_REVISION=V3 DEEP_ANALYSIS_RESULT=DIRECT_ROUTE_JSONRESPONSE_CONTRACT_CORRECTED FAILURE_CLASS=VERIFIER_ONLY PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO CANDIDATE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CONTAINER_STARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 INPUT_APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd INPUT_WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INPUT_INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 CANDIDATE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-candidate-v3/20260713T042046Z RUNTIME_POLICY=CONTAINER_PYTHON_ONLY PYTHONPATH_APP_INJECTED=YES ROUTE_RETURN_DECODER=DICT_OR_RESPONSE_BODY STATE_RETURN_DECODED=YES CHART_HEALTH_RETURN_DECODED=YES FETCH_KLINES_RETURN_DECODED=YES PROTECTED_KLINES_CODES=401_OR_403 LIVE_PRODUCTION_DATA_MOUNTED=NO ISOLATED_PORT=18030 FAILED_RUN_CLEANUP=VERIFIED REMOTE_RUNTIME_SCRIPT_SHA256=b60290b08f6bdb9801f4ccadbeadc4268399426996fd271527830fd399055448 VERIFIER_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-runtime-verifier-v3/20260713T043529Z READY_FOR_GRT_ISOLATED_RUNTIME_DRY_RUN_RUN=YES
1156:/var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-candidate.txt:1:STATUS=OK TS=2026-07-13T06:11:23Z TYPE=crypto-portfolio-ui-center-alert-currency-candidate BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=EXPECTED_DIFF_RC_1_TRIGGERED_SET_E_PIPEFAIL PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=80763a6c62b94cde4f3293f3b529980f76919d25 INPUT_INDEX_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e CANDIDATE_INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb BUILDER_DOUBLE_CHECK=OK CANDIDATE_DOUBLE_CHECK=OK SHARP_MOVES_POSITION=CENTERED SHARP_MOVES_MAX_WIDTH_PX=860 SHARP_MOVES_COLUMNS_DESKTOP=2 SHARP_MOVES_COLUMNS_MOBILE=1 ALERT_RENDERER=V5 ALERT_CURRENCY_SOURCE=TOP_USD_RUB_SWITCH ALERT_EXCHANGE_RATE_SOURCE=STATE_MARKET_USD_RUB ALERT_PRICE_THRESHOLD_CONVERTED=YES ALERT_CAP_THRESHOLD_CONVERTED=YES ALERT_SALE_VALUE_CONVERTED=YES CURRENCY_SWITCH_REDRAW=YES PUMP_DUMP_PERCENT_UNCHANGED=YES FIXED_USD_ALERT_FORMATTER=ABSENT RENDER_MOVES_COUNT=1 ALERT_RENDERER_MARKER_COUNT=1 GRT_REFERENCES=0 PRODUCTION_INDEX_UNCHANGED=YES PRODUCTION_HEALTH=200 READY_FOR_UI_ISOLATED_BROWSER_SMOKE=YES CANDIDATE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-ui-center-alert-currency-candidate-v4/20260713T061123Z
1157:/var/lib/homelab-health/cluster-admin-stage2-design.txt:1:STATUS=OK TS=2026-07-13T00:35:14Z TYPE=cluster-admin-stage2-design BAD=0 STAGE1_CLOSED=YES CAPTURE_REUSED=YES PRODUCTION_CHANGED=EVIDENCE_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO SERVICE_RESTARTED=NO FOUNDATION_STATE=9|7|1|1 EXPECTED_ENTITIES=62 MANAGED_ENTITIES=62 PLACEHOLDERS=21 MISSING_ENTITIES=0 STALE_ENTITIES=0 LEGACY_DEPENDENCIES=26 TYPED_DEPENDENCIES=26 DEPENDENCY_ID_MATCHES=0 DEPENDENCY_NATURAL_KEY_MATCHES=26 DEPENDENCY_NATURAL_KEY_MISSING=0 DEPENDENCY_NATURAL_KEY_STALE=0 EPHEMERAL_ID_MISSING=26 EPHEMERAL_ID_STALE=26 ROOT_CAUSE=LEGACY_DEPENDENCIES_DELETE_REINSERT_EACH_RUN CANONICAL_DEPENDENCY_IDENTITY=PARENT_CHILD_RELATION MIGRATION_ID=002_canonical_foundation_sync DATABASE_FUNCTION=cluster_admin_sync_foundation COLLECTOR_CALL_REQUIRED=YES SAME_TRANSACTION_REQUIRED=YES LEGACY_TABLES_PRESERVED=YES AUTOMATIC_REMEDIATION_ACTIONS=NO STAGE2_READY_FOR_DRY_RUN=YES DESIRED_STATE_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 CHANGE_SET=/var/lib/homelab-change-sets/stage2-design/20260713T003514Z CONTRACT=/var/lib/homelab-change-sets/stage2-design/20260713T003514Z/stage2-contract.json REPORT=/var/lib/homelab-change-sets/stage2-design/20260713T003514Z/stage2-design-report.txt
1158:/var/lib/homelab-health/final-green-freeze-seal.txt:1:STATUS=OK TS=2026-07-08T08:13:08Z TYPE=final-green-freeze-seal BAD=0 WARN=0 MODE=FINAL_FREEZE_EVIDENCE_SEAL FREEZE_SEAL_OK=YES AUDIT_STATUS=OK AUDIT_FREEZE_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 EXACT_WARN_REVIEW_FILES=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T072711Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T072711Z DIRTY_COUNT=0 EVIDENCE=/var/lib/homelab-final-green-freeze/freeze-evidence-latest.tsv REPORT=/var/lib/homelab-final-green-freeze/final-seal-latest.md
1159:/var/lib/homelab-health/cluster-admin-stage2-dry-run.txt:1:STATUS=OK TS=2026-07-13T00:39:40Z TYPE=cluster-admin-stage2-dry-run BAD=0 PRODUCTION_CHANGED=TEST_DATABASE_AND_CANDIDATE_ARTIFACTS_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO MIGRATION_ID=002_canonical_foundation_sync DATABASE_FUNCTION=cluster_admin_sync_foundation MIGRATION_APPLY_TEST=OK MIGRATION_IDEMPOTENCE_TEST=OK NATURAL_KEY_CONVERGENCE=OK EPHEMERAL_ID_REFRESH=OK ENTITY_ADD_SYNC=OK ENTITY_RETIRE_SYNC=OK PLACEHOLDER_ADD_SYNC=OK PLACEHOLDER_RETIRE_SYNC=OK STALE_TYPED_EDGE_DELETE=OK SOURCE_INSTANCE_UPDATE=OK COLLECTOR_CANDIDATE_SYNTAX=OK COLLECTOR_CALL_BEFORE_COMMIT=OK ROLLBACK_TEST=OK TEST_DATABASE_DROPPED=YES PRODUCTION_UNCHANGED=YES READY_FOR_PREAPPLY_BACKUP=YES DESIRED_STATE_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 CHANGE_SET=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z MIGRATION=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z/002_canonical_foundation_sync.sql ROLLBACK=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z/002_canonical_foundation_sync.rollback.sql COLLECTOR=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z/collector.py.candidate PLAN=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z/stage2-migration-plan.json RESULT=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z/dry-run-result.txt
1160:/var/lib/homelab-health/cluster-admin-stage3-production-verify.txt:1:STATUS=OK TS=2026-07-13T03:28:52Z TYPE=cluster-admin-stage3-production-verify BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=VERIFICATION_EVIDENCE_ONLY PRODUCTION_DATABASE_CHANGED=NO LIVE_APPLICATION_CODE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO PREVIOUS_PERMISSION_FAILURE=CORRECTED VERIFIER_OWNER=CLUSTERADMIN VERIFIER_MODE=0700 LIVE_APP_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 AUTHENTICATED_PRODUCTION_TESTS=25 AUTHENTICATED_PRODUCTION_FAILED=0 LEGACY_AUTHENTICATED_SUMMARY=OK NEW_GET_ROUTES_AUTHENTICATED=OK NEW_GET_ROUTES_UNAUTHENTICATED_401=OK ENTITY_LIST=OK ENTITY_DETAIL=OK ENTITY_DEPENDENCIES=OK DEPENDENCIES=OK SOURCES=OK POLICIES=OK COLLECTOR_RUNS=OK VALIDATION_ERRORS=OK FORBIDDEN_RESPONSE_FIELDS=ABSENT OPENAPI_DOCS_EXPOSURE_CHANGED=NO SECURITY_HEADERS=OK PRODUCTION_RUNTIME_HEALTH=OK REMOTE_VERIFIER_CLEANUP=OK READY_FOR_STAGE3_POSTAPPLY_SEAL=YES STAGE3_CLOSED=NO RESULT=/var/lib/homelab-change-sets/stage3-api-production-verify-run/20260713T032852Z/production-result.json CHANGE_SET=/var/lib/homelab-change-sets/stage3-api-production-verify-run/20260713T032852Z
1161:/var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-browser-smoke.txt:1:STATUS=OK TS=2026-07-13T07:04:45Z TYPE=crypto-portfolio-ui-center-alert-currency-browser-smoke BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=UNPRIVILEGED_REMOTE_RM_AFTER_SUDO_HARNESS PRODUCTION_CHANGED=NO_BROWSER_EVIDENCE_FINALIZATION_ONLY PRODUCTION_DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru CANDIDATE_INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb CANDIDATE_STATIC_GRT_DOUBLE_CHECK=OK BROWSER_ENGINE=CHROMIUM BROWSER_RUNS=2 BROWSER_RUNS_IDENTICAL=YES MOVES=2 SHARP_MOVES_CENTERED=YES SHARP_MOVES_MAX_WIDTH_OK=YES SHARP_MOVES_COLUMNS=2 ALERT_ROWS=2 USD_ALERT_RENDER=OK RUB_ALERT_RENDER=OK TOP_CURRENCY_SWITCH_REDRAW=OK RUB_PRICE_THRESHOLD=OK RUB_CAP_THRESHOLD=OK RUB_SALE_VALUE=OK ALERT_USD_RESIDUE_IN_RUB_MODE=ABSENT CHART_CARDS=2 CHART_CANVASES=14 CHARTS_BOTTOM=YES BROWSER_ERRORS=0 GRT_VISIBLE=NO RESULT_SHA256=b566f492cd80c8f99bc6b24cbef6d5f326c04f4d7168359d05eb1f189589e733 SCREENSHOT_SHA256=68a9a68d15673ef64c248a4fe886ac36f94923ec917daa2772fc9e7c1b5f4c16 ROOT_OWNED_STAGE_ENTRIES=11 ROOT_OWNED_STAGE_DIRECTORIES=2 REMOTE_STAGE_CLEANED=YES PRODUCTION_INDEX_UNCHANGED=YES PRODUCTION_HEALTH=200 VM170_FAILED_UNITS=0 PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES READY_FOR_UI_PREAPPLY_BACKUP_AND_PRODUCTION_APPLY=YES BROWSER_SMOKE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-ui-center-alert-currency-browser-smoke-v2/20260713T070125Z
1162:/var/lib/homelab-health/crypto-portfolio-remove-grt-surface.txt:1:STATUS=OK TS=2026-07-13T04:05:42Z TYPE=crypto-portfolio-remove-grt-surface BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PRODUCTION_CHANGED=NO DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru TARGET_STACK=/opt/stacks/crypto-portfolio DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 DESIRED_STATE_CLEAN=YES APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 APP_GRT_REFERENCES=5 WORKER_GRT_REFERENCES=2 INDEX_GRT_REFERENCES=2 LOCAL_GRT_REFERENCES=3 SQLITE_SCHEMA_CAPTURED=YES READY_FOR_GRT_CANDIDATE_PREPARE=YES SURFACE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-surface/20260713T040542Z
1163:/var/lib/homelab-health/cluster-admin-stage4d-adapter-candidates-in-isolation.txt:1:STATUS=OK TS=2026-07-14T06:14:58Z TYPE=cluster-admin-stage4d-adapter-candidates-in-isolation-final-seal BAD=0 STAGE4D_CLOSED=YES NEXT_STAGE=4E_RESTRICTED_PROBE_AGENT_DESIGN PRODUCTION_CHANGED=NO DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO APPLICATION_CHANGED=NO ENVELOPE_TOTAL=48 PROMETHEUS_ENVELOPES=5 BACKUP_MATRIX_ENVELOPES=40 HEALTH_ENVELOPES=3 LOCAL_REMOTE_OUTPUT_MATCH=OK PRODUCTION_STATE=0|0|OK DESIRED_STATE_HEAD=e77fb37a0fa0033fe46558045e31dd92b6d1238a DESIRED_STATE_CLEAN=YES PVE01_FAILED_UNITS=0 REPORT=/var/lib/homelab-cluster-admin-stage4/20260714T061458Z/stage4d-adapter-candidates-in-isolation-seal.txt REPORT_SHA256=da6244c0a8a458b508fd58a0fec9cb1e6fa8da98a36f44d1368cbe947f4b5835
1164:/var/lib/homelab-health/backup-matrix-cluster-desired-state-sync.txt:1:STATUS=OK TS=2026-07-13T17:51:31Z TYPE=backup-matrix-cluster-desired-state-sync BAD=0 PRE_SYNC_HEAD=af26ec3c4829b1330d49656205ac580ce1706263 GENERATED_SYNC_HEAD=597119391e7b265b85fd9a76644e4e33af47e469 GENERATED_SYNC_MODE=NEW_GENERATED_COMMIT GENERATED_SYNC_COMMIT_DELTA=1 MATRIX_STATUS=OK JOBS_TOTAL=35 JOBS_OK=31 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 CLUSTER_PASSPORT=OK FINAL_READINESS=OK OVERALL=OK CLUSTER_ADMIN=OK PROBE_OK=4 PROBE_BAD=0 FAILED_UNITS=0 DESIRED_STATE_DIRTY=0 MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 MATRIX_STATE_SHA256=14a31112aac5d577ae01df567921b093b2fedcf7189c4cc8f009216ee0749380 PRESEAL_SHA256=7ea1f710cd311bd51dba05a313b4c540f57d97407f008881e502fc213b4b3c99 BACKUP_RUN=NO RESTORE_RUN=NO CLOUD_WRITE_RUN=NO RETENTION_RUN=NO EVIDENCE_ROOT=/var/lib/homelab-change-sets/backup-matrix-cluster-final-seal/20260713T175131Z
1165:/var/lib/homelab-health/cluster-admin-stage2-recovery.txt:1:STATUS=OK TS=2026-07-13T01:04:59Z TYPE=cluster-admin-stage2-recovery BAD=0 RESULT=RECOVERED PRODUCTION_CHANGED=HEALTH_REGENERATION_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO INCIDENT_ENGINE_RESTARTED=NO DATABASE_ROLLBACK=OK COLLECTOR_ROLLBACK=OK CORE_DATABASE_STATE=0|0|9|7|1 LIVE_COLLECTOR_SHA256=a83abde2290d8392b1c631122671bcb680f9271d0b576fafacb8876d3c8034f1 OBSERVER=OK FULL_OBSERVER=OK RESTRICTED_PROBES=OK RESTRICTED_PROBES_AUTHORITY=CENTRAL_SYNTHESIZED_FROM_REMOTE_FULL_OBSERVER VERIFIER_BUG=REMOTE_RESTRICTED_PROBES_FILE_INCORRECTLY_REQUIRED FINAL_READINESS=OK CLUSTER_PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK VM180_FAILED_UNITS=0 PVE01_FAILED_UNITS=0 DESIRED_STATE_UNCHANGED=YES ROLLBACK_POINT=VERIFIED READY_FOR_STAGE2_RETRY=YES STAGE2_CLOSED=NO CHANGE_SET=/var/lib/homelab-change-sets/stage2-recovery-final/20260713T010459Z
1166:/var/lib/homelab-health/cluster-admin-stage3-api-runtime-dry-run.txt:1:STATUS=OK TS=2026-07-13T02:53:34Z TYPE=cluster-admin-stage3-api-runtime-dry-run BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=TEMPORARY_ISOLATED_RUNTIME_AND_EVIDENCE_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO LIVE_APPLICATION_CODE_CHANGED=NO DESIRED_STATE_CHANGED=NO PRODUCTION_SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO APP_PRECHANGE_SHA256=c2748b8f96df842f491b5d2790790594222549853d0e2336cf60ecd6d4de0020 APP_CANDIDATE_SHA256=672d6a2975dd0e093788e7eb81a76e7f901ea6374ca4092a25ba0091aba1cb50 ISOLATED_BIND=127.0.0.1:18081 TEMPORARY_DATABASE_RESTORED=YES TEMPORARY_DATABASE_DROPPED=YES TEMPORARY_DATABASE_UNCHANGED=YES RUNTIME_TESTS=41 RUNTIME_FAILED_TESTS=0 EXISTING_ROUTES_BACKWARD_COMPATIBLE=YES NEW_GET_ROUTES=7 UNAUTHENTICATED_NEW_ROUTES_401=YES AUTHENTICATED_NEW_ROUTES=OK PAGINATION=OK FILTER_VALIDATION=OK ENTITY_DETAIL=OK ENTITY_DEPENDENCIES=OK SOURCES=OK POLICIES=OK COLLECTOR_RUNS=OK DATABASE_TRANSACTION_MODE=BEGIN_READ_ONLY FORBIDDEN_RESPONSE_FIELDS=ABSENT OPENAPI_DOCS_EXPOSURE_CHANGED=NO SECURITY_HEADERS=OK ISOLATED_RUNTIME_STOPPED=YES ISOLATED_PORT_CLOSED=YES LIVE_APP_UNCHANGED=YES PRODUCTION_RUNTIME_HEALTH=OK READY_FOR_STAGE3_PREAPPLY_BACKUP=YES DESIRED_STATE_HEAD=8f888e5f2063e49da918b31584c9ea4bcf53af3a CHANGE_SET=/var/lib/homelab-change-sets/stage3-api-runtime-dry-run/20260713T025334Z RESULT=/var/lib/homelab-change-sets/stage3-api-runtime-dry-run/20260713T025334Z/runtime-result.json
1167:/var/lib/homelab-health/crypto-portfolio-remove-grt-failed-units-diagnostic.txt:1:STATUS=OK TS=2026-07-13T05:41:00Z TYPE=crypto-portfolio-remove-grt-failed-units-diagnostic BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=NO_DIAGNOSTIC_ONLY PRODUCTION_DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO FAILED_STATE_RESET=NO CLOUD_WRITE_RUN=NO PVE01_FAILED_UNIT_COUNT=2 PVE01_FAILED_UNIT_NAMES=homelab-backup-matrix-refresh.service,homelab-cluster-admin-pull.service CRYPTO_PORTFOLIO_FAILED_UNIT_COUNT=0 OTHER_FAILED_UNIT_COUNT=2 CURRENT_CRYPTO_PRECHANGE_HEALTH=YES UNIT_DETAILS_CAPTURED=YES READY_FOR_TARGETED_FAILED_UNIT_DECISION=YES DIAGNOSTIC_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-failed-units-diagnostic/20260713T054100Z
1168:/var/lib/homelab-health/crypto-portfolio-remove-grt-production-apply.txt:1:STATUS=OK TS=2026-07-13T05:47:54Z TYPE=crypto-portfolio-remove-grt-production-apply BAD=0 COMMAND_PHYSICAL_LINES=1 APPLY_REVISION=V3 FIX_UID_READONLY=YES FIX_READINESS_ENDPOINT=YES FIX_ROLLBACK_DATABASE_METADATA=YES PRODUCTION_CHANGED=YES_REMOVE_GRT DESIRED_STATE_CHANGED=NO_BY_THIS_COMMAND PRODUCTION_CONTAINER_RECREATED=YES CLOUD_WRITE_RUN=NO ROLLBACK_ON_FAILURE=ARMED_NOT_NEEDED TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=80763a6c62b94cde4f3293f3b529980f76919d25 PRECHANGE_BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T051015Z ROLLBACK_PATH=/var/lib/homelab-incidents/crypto-remove-grt-prechange-20260713T051013Z APP_SHA256=44df2b25183b7c5d87ff1516ec2d9cd1d169acde3031f7c2aa1fc4e0c7813cd2 WORKER_SHA256=174a14b86014535980d200a76d0057110c7af102c66f16c2e21fc3e74bb2e49e INDEX_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e CHART_HEALTH_SCRIPT_SHA256=c16473f16390eeeff4d4eb701db604093564dd78d0be2d30b4cc8ffeeb22364a ACTIVE_COINS=HBAR,POL GRT_SOURCE_REFERENCES=0 GRT_DATABASE_ROWS=0|0|0 GRT_MARKET_ENTRY=ABSENT GRT_CACHE_ENTRIES=ABSENT DATABASE_MODE=0644 DATABASE_UID=10001 DATABASE_GID=10001 DATABASE_INTEGRITY=ok WORKER_STATUS=OK WORKER_COIN_COUNT=2 PUBLIC_CODES=200|303|401|200|200|401 CHART_HEALTH=OK CHART_SYMBOLS=HBAR,POL CHART_HEALTH_TIMER=ACTIVE VM170_FAILED_UNITS=0 PVE01_UNRELATED_FAILED_UNITS=homelab-backup-matrix-refresh.service,homelab-cluster-admin-pull.service PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES REMOTE_STAGE_CLEANED=YES READY_FOR_GRT_POSTAPPLY_VERIFY_BACKUP_SYNC=YES APPLY_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-production-apply-v3/20260713T054754Z
1169:/var/lib/homelab-health/mail-cloud-restore-drill.txt:1:STATUS=STALE
1171:/var/lib/homelab-health/edge-restore-proofs-offhost.txt:1:STATUS=OK TS=20260624T190040Z TYPE=edge-restore-proofs-offhost DEST_HOST=pve02 DEST_DIR=/mnt/staging/offhost/edge-restore-proofs-from-edge ITEMS=14 OK=14 ERROR=0 BYTES=226024287 SECRET_PRINTED=NO
1172:/var/lib/homelab-health/crypto-portfolio-runtime-final-seal.txt:1:STATUS=OK TS=2026-07-13T08:37:54Z TYPE=crypto-portfolio-ui-center-alert-currency-final-seal BAD=0 DANGLING_IMAGE_REFERENCE_GATE=OK DESIRED_STATE_SYNC_ARTIFACT_GATE=OK APP=crypto-portfolio PRODUCTION_CHANGED=NO_FINAL_SEAL_ONLY APP_SHA256=44df2b25183b7c5d87ff1516ec2d9cd1d169acde3031f7c2aa1fc4e0c7813cd2 WORKER_SHA256=174a14b86014535980d200a76d0057110c7af102c66f16c2e21fc3e74bb2e49e INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb LIBRARY_SHA256=c0992580867c4912cc9385b3c2728315bcc1a76c7f1087dca908430fccdf31d7 ACTIVE_COINS=HBAR,POL GRT_REMOVED=YES GRT_ROWS=0|0|0 GRT_SOURCE_REFERENCES=0 GRT_CACHE_ENTRIES=0 SHARP_MOVES_CENTERED=YES ALERT_CURRENCY_USD_RUB=OK BROWSER_SMOKE=OK BROWSER_RUNS=2 BROWSER_ERRORS=0 BROWSER_RESULT_SHA256=b566f492cd80c8f99bc6b24cbef6d5f326c04f4d7168359d05eb1f189589e733 SCREENSHOT_SHA256=00b2853bee9c880146d0dd75fac60feb7e46d4c129ee06754047ef30d90bd4ba BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T075801Z TWO_CLOUDS=OK RESTORE_CHECK=OK DATABASE_INTEGRITY=ok RETENTION=OK ROLLBACK_COUNT=3 DANGLING_IMAGES=2 REFERENCED_DANGLING_IMAGES=2 UNREFERENCED_DANGLING_IMAGES=0 PRUNED_UNREFERENCED_DANGLING_IMAGES=0 CHART_HEALTH=OK PUBLIC=OK MONITOR_TIMER=active BACKUP_TIMER=active CLOSURE_DESIRED_STATE_HEAD=2999910549eaff9b9431b9c656d318340eb95cbd DESIRED_STATE=OK PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES ALL_CRYPTO_PORTFOLIO_UI_TAILS_CLOSED=YES
1173:/var/lib/homelab-health/backup-matrix-cluster-preseal-convergence.txt:1:STATUS=OK TS=2026-07-13T17:48:14Z TYPE=backup-matrix-cluster-preseal-convergence BAD=0 MATRIX_STATUS=OK JOBS_TOTAL=35 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 CLUSTER_PASSPORT=OK FINAL_READINESS=OK OVERALL=OK CLUSTER_ADMIN=OK PROBE_OK=4 PROBE_BAD=0 WEBPANEL=OK FAILED_UNITS=0 DESIRED_STATE_HEAD=af26ec3c4829b1330d49656205ac580ce1706263 DESIRED_STATE_DIRTY=0 MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 MATRIX_STATE_SHA256=3791fa78cdfb56cd7027f89350dff245abcffc4f935adfc336ceeea00854ac89 CLUSTER_PASSPORT_SHA256=1b8b924ae0e5ce6872fe5671318094ce494806c2db7efe28299b5ff49adc0d0d FINAL_READINESS_SHA256=ac5159fa8c497afe021b5cb232f08cfec0e1a4d516d8aae966ba03f7034f1412 OVERALL_SHA256=6ae529bfdf5811f686d3b53978f453ce4f8757c74fc1be105de219b13a61ba46 BACKUP_RUN=NO RESTORE_RUN=NO CLOUD_WRITE_RUN=NO RETENTION_RUN=NO EVIDENCE_ROOT=/var/lib/homelab-change-sets/backup-matrix-cluster-preseal-resume/20260713T174748Z
1174:/var/lib/homelab-health/uptime-kuma-pending-health-monitors-fix.txt:8:NOTE=Paperless Restore may become red if its health file contains STATUS=ERROR; that is a real restore-health issue, not a Kuma pending issue.
1175:/var/lib/homelab-health/pve03-capacity-forensics.txt:1:STATUS=OK TS=2026-07-08T05:42:29Z TYPE=pve03-capacity-forensics MODE=SUPERSEDED_BY_PVE03_ROOT_STAGING_CLEANUP BAD=0 WARN=0 OLD_STATUS=WARN_ARCHIVED ROOT_CLEANUP=OK STAGING_RETENTION=OK STAGING_CLEANUP=OK FAILED_UNIT_CLEANUP=OK REPORT=/var/lib/homelab-duty-admin-residual-cleanup/pve03-capacity-forensics-closure-latest.md
1176:/var/lib/homelab-health/remote-failed-unit-apply.txt:1:STATUS=OK TS=2026-07-08T05:25:22Z TYPE=remote-failed-unit-apply BAD=0 WARN=0 MODE=APPLY_CLEANUP_PLAN_EXCEPT_PAPERLESS_CORE_GUARD PRECHECK=YES APPLY_OK=YES LOCAL_OVERRIDE_COUNT=4 PVE02_RESET=1 EDGE_APPLY=1 LOCAL_FAILED_AFTER=0 PVE02_FAILED_AFTER=0 EDGE_FAILED_AFTER=1 EDGE_DEFER_ONLY=YES PVE02_CODEVIPE_FAILED=inactive DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 OVERALL_STATUS=OK OVERALL_BAD=0 DEFERRED_UNIT=homelab-paperless-core-guard.service REPORT=/var/lib/homelab-remote-failed-unit-cleanup/apply-latest.md
1177:/var/lib/homelab-health/recovery-card.txt:1:STATUS=OK TS=2026-07-08T08:24:05Z TYPE=recovery-card BAD=0 WARN=0 MODE=READ_ONLY_RECOVERY_CARD_GENERATION RECOVERY_CARD_OK=YES FREEZE_STATUS=OK FREEZE_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T082034Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T082034Z EMERGENCY_JOB=homelab-pve01-emergency-pack-20260708T002739Z VM_DUMPS_JOB=homelab-pve01-vm-local-dumps-20260706T153606Z ENDPOINT_LOCAL_OK=1 ENDPOINT_LAN_OK=1 ENDPOINT_PUBLIC_OK=1 BACKUP_SLA=OK BACKUP_COVERAGE=OK EXTENDED_APPBACKUP=OK VM_BACKUP=OK EXTERNAL_PROBE=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 DIRTY_COUNT=0 CARD=/var/lib/homelab-recovery-card/recovery-card-latest.md SUMMARY=/var/lib/homelab-recovery-card/recovery-summary-latest.tsv REPORT=/var/lib/homelab-recovery-card/recovery-card-latest.md
1178:/var/lib/homelab-health/cluster-admin-stage0-convergence.txt:1:STATUS=WARN TS=2026-07-12T23:12:59Z TYPE=cluster-admin-stage0-targeted-convergence BAD=2 PROFILE_OK=NO CLUSTER_PASSPORT=FAIL FINAL_READINESS=FAIL OVERALL=FAIL DUTY_ADMIN=FAIL FAILED_UNITS=0 LIVE_MATCH_DESIRED_STATE=YES READY_FOR_FINAL_SEAL=NO REPORT_DIR=/var/lib/homelab-cluster-admin-stage0/readiness-profile-20260712T231259Z PRODUCTION_CHANGED=HEALTH_REGENERATION_ONLY PRODUCTION_SCHEMA_CHANGED=NO
1179:/var/lib/homelab-health/cluster-admin-stage4c-database-migration-dry-run.txt:1:STATUS=OK TS=2026-07-14T05:33:26Z TYPE=cluster-admin-stage4c-database-migration-dry-run-final-seal BAD=0 STAGE4C_CLOSED=YES NEXT_STAGE=4D_ADAPTER_CANDIDATES_IN_ISOLATION MIGRATION_SHA256=a0bd6a50c6ad9b90528c2e90e6d9cb4279b09811b559b2504c83d7527b19e578 FINAL_READINESS=OK OVERALL=OK PUBLIC_ENDPOINT=OK WEBPANEL_SYNC=OK PVE01_FAILED_UNITS=0 VM180_FAILED_UNITS=0 DESIRED_STATE_HEAD=df86e4d27a64613b60ae55253b6348866dd34fa9 DESIRED_STATE_CLEAN=YES REPORT=/var/lib/homelab-cluster-admin-stage4/20260714T053326Z/stage4c-database-migration-dry-run-seal.txt REPORT_SHA256=3368c302ca0ee76fab8fbc97caa65adf4cf57e61c5b5cdb9a1f3829a05af2e2a
1180:/var/lib/homelab-health/operator-command-size-guard.txt:1:STATUS=OK TS=2026-07-14T18:26:54Z TYPE=operator-command-size-guard RULE_ACTIVE=YES PARSE_ERROR_TAIL=CLOSED UNRESOLVED_TAIL_COUNT=0 BLOCKER_COUNT=0 REPORT_SHA256=2ed7949dde9d5f8346e46af362e27cb8982d65f00ec602546efab506509e47af
1181:/var/lib/homelab-health/node-loss-final-closure.txt:1:STATUS=OK TS=2026-07-08T16:35:56Z TYPE=node-loss-final-closure BAD=0 MODE=ALL_NODE_LOSS_DRILLS_SEALED PVE03_EDGE_RESTORE=OK PVE03_EDGE_FRESHNESS=OK PVE02_FORUM_RUNTIME=OK PVE01_NEXTCLOUD_DNS=OK PVE01_CORE_MONITORING_APPLEVEL=OK FULL_VM_GAP_VM170_VM171=ACKNOWLEDGED_APP_LEVEL_ONLY TEST_VMS_CLEANED=YES DNS_PAIR_OK=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-node-loss/final-closure-latest.md
1182:/var/lib/homelab-health/crypto-portfolio-remove-grt-browser-smoke.txt:1:STATUS=OK TS=2026-07-13T04:52:06Z TYPE=crypto-portfolio-remove-grt-browser-smoke BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=OPTIONAL_XXD_DEPENDENCY_MISSING_WITH_STDERR_SUPPRESSED BROWSER_RERUN=NO EVIDENCE_SALVAGED=YES PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO CANDIDATE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CONTAINER_STARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 INPUT_APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd INPUT_WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INPUT_INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 ACTIVE_COINS=HBAR,POL GRT_VISIBLE=NO BROWSER_CARDS=2 BROWSER_CANVASES=14 BROWSER_MOVES=2 BROWSER_COIN_CARDS=2 BROWSER_POSITION_ROWS=2 BROWSER_ALERT_OPTIONS=2 BROWSER_ACTIVE_COIN=HBAR BROWSER_CHART_SYMBOLS=HBAR,POL BROWSER_ERRORS=0 BROWSER_LOADING_TEXT=ABSENT BROWSER_ERROR_TEXT=ABSENT CHARTS_POSITION=BOTTOM PNG_VALIDATION=PYTHON_STDLIB PNG_SIGNATURE=89504e470d0a1a0a SCREENSHOT_SHA256=3d132b477a382487e34db2f2feaf7e4f43af8d30fd8e1530225782da5d7336b9 RESULT_SHA256=0deadbf38aeef86726035da66eb1991c9494f59172b1c1a5c502c50c2de95c60 LIVE_PRODUCTION_DATA_MOUNTED=NO NPM_INSTALL_RUN=NO DOCKER_PULL_RUN=NO PRODUCTION_SOURCE_HASHES_UNCHANGED=YES PRODUCTION_CONTAINER_IDS_AND_RESTARTS_UNCHANGED=YES PRODUCTION_DATABASE_INTEGRITY=ok PRODUCTION_GRT_STATE_PRESERVED=YES PUBLIC_PRODUCTION_STATE_PRESERVED=YES REMOTE_RESIDUE=NONE READY_FOR_GRT_PRECHANGE_BACKUP=YES BROWSER_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-browser-smoke-salvage/20260713T045206Z
1183:/var/lib/homelab-health/independent-external-runner-audit.txt:1:STATUS=OK TS=2026-07-08T09:13:25Z TYPE=independent-external-runner-audit BAD=0 WARN=0 MODE=READ_ONLY_INDEPENDENT_EXTERNAL_RUNNER_AUDIT AUDIT_OK=YES INDEPENDENT_RUNNER_READY=NO DEPLOY_DECISION=NO_DEPLOY CURRENT_INDEPENDENT_RUNNER_AVAILABLE=no TRUSTED_NETBIRD_IDENTITY=edge-vm PUBLIC_VPS_DECISION=REJECTED_HOSTKEY_MISMATCH PUBLIC_VPS_SSH_OK=0 PUBLIC_VPS_REJECTED_HOSTKEY=YES EDGE_LAN_SSH_OK=2 EDGE_NETBIRD_SSH_OK=2 EDGE_DNS_SSH_OK=1 PUBLIC_OVERALL_OK=1 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 OVERALL_STATUS=OK OVERALL_BAD=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 CLOSEOUT_OK=YES RECOVERY_SEAL_OK=YES DIRTY_COUNT=0 REPORT=/var/lib/homelab-independent-external-runner/audit-latest.md
1184:/var/lib/homelab-health/sops-edge-secret-coverage-closure.txt:1:STATUS=OK TS=2026-07-08T05:01:47Z TYPE=sops-edge-secret-coverage-closure BAD=0 WARN=0 MODE=POLICY_SKELETON_SUPERSEDED_NOOP PRECHECK=YES POLICY_MODE=POLICY_SKELETON_NOT_ACTIVE AUDIT_CLASS=UNGATED_FAILED_STATE_REVIEW IN_HARD_GATES=0 OVERRIDE_WRITTEN=1 FAILED_UNITS_AFTER=0 UNIT_FAILED_AFTER=inactive MARKER_STATUS=OK DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 DIRTY_COUNT=0 SOPS_CLOSURE_OK=YES REPORT=/var/lib/homelab-failed-unit-legacy-cleanup/sops-edge-secret-coverage-closure-latest.md
1185:/var/lib/homelab-health/mail-cloud-canary.txt:1:STATUS=OK TS=20260618T174444Z TYPE=mail-cloud-canary REMOTE_TOTAL=13 REMOTE_OK=13 REMOTE_FAIL=0 DEST=restore-test/p1-canary/20260618T174444Z SHA256=a6bf6a1493a2556b92c8e6547163500345d4e0e32688937370b942b881c47182 SECRET_PRINTED=no NOTES=none
1186:/var/lib/homelab-health/filebrowser-restore.txt:1:STATUS=STALE
1189:/var/lib/homelab-health/final-green-freeze-audit.txt:1:STATUS=OK TS=2026-07-08T07:45:35Z TYPE=final-green-freeze-audit BAD=0 WARN=0 MODE=READ_ONLY_FREEZE_EVIDENCE_AUDIT FREEZE_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 EXACT_WARN_REVIEW_FILES=0 LOCAL_OVERALL_OK=1 LAN_OVERALL_OK=1 PUBLIC_OVERALL_OK=1 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T072711Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T072711Z BACKUP_SLA=OK BACKUP_COVERAGE=OK EXTENDED_APPBACKUP=OK VM_BACKUP=OK SECRET_EXPOSURE=OK REVIEW_SURFACE_REGISTRY=YES FINAL_GLOBAL_GREEN=YES EXTERNAL_PROBE=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 DIRTY_COUNT=0 EVIDENCE=/var/lib/homelab-final-green-freeze/freeze-evidence-latest.tsv REPORT=/var/lib/homelab-final-green-freeze/freeze-audit-latest.md
1190:/var/lib/homelab-health/crypto-portfolio-remove-grt-runtime-dry-run.txt:1:STATUS=OK TS=2026-07-13T04:41:24Z TYPE=crypto-portfolio-remove-grt-runtime-dry-run BAD=0 COMMAND_PHYSICAL_LINES=1 VERIFIER_REVISION=V3 ANALYSIS_SOURCES=FULL_CHAT_CRYPTO_HANDOFF_CLUSTER_REFERENCE_ERROR_HISTORY PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO CANDIDATE_DATABASE_CHANGED=ISOLATED_COPY_RUNTIME_CHECK_ONLY DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO PRODUCTION_CONTAINER_RECREATED=NO ISOLATED_CONTAINER_STARTED=YES ISOLATED_CONTAINER_CLEANED=YES CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 INPUT_APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd INPUT_WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INPUT_INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 ACTIVE_COINS=HBAR,POL CANDIDATE_GRT_REFERENCES=0 DATABASE_GRT_AFTER=0|0|0 WORKER_RUNTIME_CHECK=OK WEB_RUNTIME_CHECK=OK HTTP_RUNTIME_CHECK=OK HTTP_CODES=200|303|401|200|200|401 CHART_HEALTH=OK CHART_SYMBOLS=HBAR,POL ROUTE_RETURN_DECODER=DICT_OR_RESPONSE_BODY LIVE_PRODUCTION_DATA_MOUNTED=NO PRODUCTION_SOURCE_HASHES_UNCHANGED=YES PRODUCTION_CONTAINER_IDS_AND_RESTARTS_UNCHANGED=YES PRODUCTION_DATABASE_INTEGRITY=ok PRODUCTION_GRT_STATE_PRESERVED=YES PUBLIC_PRODUCTION_STATE_PRESERVED=YES REMOTE_RESIDUE=NONE BROWSER_SMOKE=NOT_RUN READY_FOR_GRT_ISOLATED_BROWSER_SMOKE=YES RUNTIME_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-runtime-run-v3/20260713T044124Z
1191:/var/lib/homelab-health/sops-edge-secret-coverage-audit.txt:1:STATUS=OK TS=2026-07-08T04:59:58Z TYPE=sops-edge-secret-coverage-audit BAD=0 WARN=1 MODE=READ_ONLY_SOPS_EDGE_SECRET_COVERAGE_AUDIT UNIT_FAILED=failed CLASS=UNGATED_FAILED_STATE_REVIEW IN_HARD_GATES=no OVERALL_BAD=0 FINAL_FAIL=0 DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=1 RELATED_WARN_OR_ERROR_HEALTH=3 RESET_FAILED_DONE=no REPORT=/var/lib/homelab-failed-unit-legacy-cleanup/sops-edge-secret-coverage-audit-latest.md
1192:/var/lib/homelab-health/duty-admin-residual-cleanup-seal.txt:1:STATUS=OK TS=2026-07-08T05:44:22Z TYPE=duty-admin-residual-cleanup-seal BAD=0 WARN=0 MODE=FINAL_RESIDUAL_NOISE_SEAL RESIDUAL_SEAL_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 REVIEW_ITEMS_INTENTIONALLY_LEFT=yes LEGACY_PHASE_ITEMS_INTENTIONALLY_LEFT=yes FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 PVE03_CAPACITY_FORENSICS_STATUS=OK LOCAL_CLEANUP_OK=YES REMOTE_CLEANUP_OK=YES DIRTY_COUNT=0 REPORT=/var/lib/homelab-duty-admin-residual-cleanup/seal-latest.md
1193:/var/lib/homelab-health/prometheus-npmplus-kuma-supersede-repair-20260709T073019Z/npmplus-kuma-config-backup.txt.bak:1:STATUS=ERROR TS=20260709T072619Z TYPE=npmplus-kuma-config-backup NOTES=edge_backup_failed
1311:/var/lib/homelab-health/codevipe-cloud-backup-reset.txt:1:STATUS=OK TS=2026-07-08T05:25:22Z TYPE=remote-failed-unit-codevipe-reset BAD=0 WARN=0 MODE=RESET_STALE_FAILED_STATE UNIT=codevipe-cloud-backup.service REASON=last_journal_success_and_no_timer ACTION=reset_failed_only
1487:CRITICAL_TAILS_BASELINE=COMPLETE
CRITICAL_TAILS_TRIAGE=COMPLETE
OUTPUT_END
CHAT_OUTPUT_END
View Git Blame Copy Permalink