homelab-admin/homelab-public-context
1
0
Fork 0
Code Activity
Files
d501d96f03a4728841b2f5380e4090d4fb0920f2
homelab-public-context/runtime/latest.txt
T
homelab-runtime-publisher d501d96f03 runtime: publish CRITICAL-TAILS-BASELINE-089
2026-07-21 13:21:17 +00:00

1512 lines
196 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
CHAT_OUTPUT_BEGIN
COMMAND_ID=CRITICAL-TAILS-BASELINE-089
STATUS=OK
RC=0
HOST=pve01
MODE=read-only
COMPONENT=cluster-audit
REFERENCE_REGISTER_CHECK=OK
REFERENCE_SHA256=5763f2f8edc75fcf6f3951f8c95896d1cc112ceccb6cf5b64a383f540a4deb8e
ERROR_REGISTER_CHECK=OK
ERROR_REGISTER_SHA256=08b3a769b8e38eaa1acec915352fb328180d70caa7ff6cd9d1ec0da426262752
COMMAND_SHA256=02690ef89c3e99e487c86a030be4e00fdf04c7a769159d168d209a482871d0eb
DUPLICATE_FAILED_COMMAND_BLOCKED=false
EXECUTION_STARTED=true
CHANGES_MADE=false
SANITIZED=yes
SECRETS_INCLUDED=no
PRIVATE_ADDRESSES_INCLUDED=no
RAW_EVIDENCE_SHA256=656c197290e6571b3d71813586dcbd77828c98ff7b6aa51585a16ca35b84b2d5
SANITIZED_OUTPUT_SHA256=f2a6f26d33ed50c9cf3d98d6236f2800cf9e57cb3d60a7b690eb745cfa896099
OUTPUT_BEGIN
=== CONTROL_FILES ===
5763f2f8edc75fcf6f3951f8c95896d1cc112ceccb6cf5b64a383f540a4deb8e /etc/pve/31_HOMELAB_REFERENCE.md
08b3a769b8e38eaa1acec915352fb328180d70caa7ff6cd9d1ec0da426262752 /etc/pve/HOMELAB_ASSISTANT_ERROR_REGISTER.md
=== ERROR_REGISTER_OPEN_MARKERS ===
1:# HOMELAB ASSISTANT ERROR REGISTER
35:11. Ошибка: считать offhost OK после failed rsync.
77:21. Ошибка: nested PHP php -r дал Parse error на forum-prod.
83:Команда 108 дала PHP Parse error на smtpHost и пустой proof-файл.
88:Команда 111 не изменила smtpSsl: OLD_SMTPSSL=false, SQL ERROR 1064, NEW_SMTPSSL=false.
93:Нельзя использовать { echo RULE_CHECK_FAIL; exit 1; } в командах, выполняемых прямо в интерактивном root@pve01 shell.
94:При failed-check использовать безопасный шаблон: if grep ...; then echo RULE_CHECK_OK; ...; else echo RULE_CHECK_FAIL; fi.
100:Сначала проверять AUTH через openssl s_client -starttls smtp, без вывода секрета.
102:26. Ошибка: openssl -crlf вместе с ручным CRLF ломает SMTP AUTH checker.
109:Команда 138 пересобрала dashboard, но диагностический Python-блок упал с NameError из-за кавычек.
126:Перед любым действием команда обязана проверять /etc/pve/HOMELAB_ASSISTANT_ERROR_REGISTER.md и текущий 31_HOMELAB_REFERENCE.md.
127:Команда обязана печатать ERROR_REGISTER_CHECK=OK и REFERENCE_CHECK=OK до основного действия.
130:32. Ошибка: 153_PROMETHEUS_TARGETS_SETTLED_PROOF содержит SyntaxError из-за вложенного Python внутри SSH.
137:Такие выводы не являются инфраструктурным состоянием; считать невалидными только соответствующие failed command attempts.
150:Факт: terminal output around storage block creation showed possible paste/truncation artifact near smartctCK=FAIL.
154:37. Coverage gap: pve03_staging missing from disk-space health coverage.
159:Факт: terminal output around command 222 showed paste artifact near "FAIL; fi".
163:Факт: строка справочника с фразой No unresolved final REVIEW_REQUIRED... SyntaxError... была принята за BAD_MARKER.
166:40. Ошибка/вывод: VM150 Mail cloud single-file upload failed with 413.
171:Факт: first reconciliation wrote STATUS=ERROR because router-running-config-mail-cloud.txt is on pve01, not edge-vm.
180:## HOME_PORTAL_BASE64_APPLY_FAILURE_20260630
181:- Prior home-portal apply attempt 312 failed because a large base64 Python payload decoded as invalid UTF-8.
188:- Any Traceback, UnicodeDecodeError, SyntaxError, Permission denied, command-not-found, nonzero critical RC, or failed write marker must force REVIEW or FAILED, never OK.
191:- If an apply step fails, the next command must explicitly record the failed attempt and then run a corrected transparent apply command.
193:## HOME_PORTAL_API_ERROR_CLOSURE_RULE_20260630
194:- Rule: home portal closure requires current Homepage API-error logs to be zero after restart/reload, not only HTTP 200.
195:- If Homepage UI shows API error or logs contain current httpProxy/API timeout errors, the portal is REVIEW until the widget/API cause is disabled or fixed.
196:- External informational widgets such as weather/Open-Meteo must not block the service launcher portal.
198:## HOME_PORTAL_COMPLEX_QUOTING_FAILURE_20260630
199:- Previous home portal cleanup command failed in local bash before execution: syntax error near unexpected token `('.
211:- A portal card is openable only when DNS, reverse-proxy host mapping, and target content are all valid.
213:## ROUTER_CLI_STDIN_APPLY_FAILURE_20260630
216:- Failure: STDIN_SHOW_OK=no, APPLY_SKIPPED=stdin_show_failed, CONFIG_HAS_EDGE_5080=no, CONFIG_HAS_EDGE_5083=no.
219:## ROUTER_CLI_PROBE_STDIN_CONSUMPTION_ERROR_20260630
227:- Observed: CMD_TEXT=access-list ? returned Core::Configurator error execute denied [cli].
232:- Evidence: 395_ROUTER_CLI_ERROR_REGISTER_UPDATE_PROOF.txt has blank P393_COMMAND_COUNT/P394_COMMAND_COUNT/P394_ZERO_RC_COUNT fields.
243:- Failure: NODE_HTTPS_RESOLVE_ERROR=ERR_INVALID_IP_ADDRESS and NODE_HTTPS_RESOLVE_INSECURE_ERROR=ERR_INVALID_IP_ADDRESS.
254:## PY_COMPILE_PYC_PERMISSION_ERROR_20260701
257:- Actual impact: non-root syntax check attempted to create /usr/local/sbin/__pycache__ and failed with Permission denied.
273:## HOMEPAGE_ROUTER_MOSCOW_APPLY_SCRIPT_SYNTAX_ERROR_20260701
276:- Evidence: proof 408 showed SyntaxError in /tmp/homepage-router-moscow-apply-fixed.py.
284:- Rule: do not proceed with OS baseline until SSH failure is diagnosed; likely old known_hosts key or cloud-init/root-key issue.
307:- Additional failed approach: proof 480 ran xf-dev:rebuild-caches on CodeVipe and returned RC=1, not a valid fix.
313:- Issue: curl returned HTTP 403; unzip then failed because the ZIP was not downloaded.
318:## PVE01_UNZIP_MISSING_FOR_XF_REQUIREMENTS_VERIFY_20260701
327:- Note: proof 511 initially failed dsmods path detection; proof 512 fixed dsmods by locating public_html via src/XF.php marker.
330:## NPMPLUS_SQLITE_PASTE_FAILURE_20260701
332:- Issue: shell entered multiline prompt and produced syntax errors.
343:- Context: NPMplus API login attempts in proofs 526/527 failed.
349:- Context: NPMplus API login attempts failed even though the forum VM and edge-to-forum connectivity are healthy.
357:- Impact: certbot ran without token and all certificate attempts failed; DNS cutover must not proceed from proof 535.
360:## EDGE_CERTBOT_FAILED_DNS_CUTOVER_ABORTED_20260701
361:- Context: edge manual routes are working locally, but certbot DNS-01 certificate issue failed in proof 538.
368:- Evidence: proof 542 showed all five zones missing and DNS record create probes failed.
369:- Impact: certbot DNS-01 failed and DNS cutover was correctly aborted; public DNS still points to old hosting.
375:- Important corrections: initial NPMplus API route failed; manual Nginx routes were used. Initial Cloudflare token had no zone access; corrected token passed zone/DNS probe.
379:- Issue: Proxmox snapshot failed because generated snapshot name exceeded 40 characters.
388:- Impact: web routing is OK, but mail-related DNS may still contain stale provider data.
392:- Context: SPF cleanup dry-run command 563 had broken shell quoting and left the terminal at multiline prompt.
397:- Context: SPF cleanup command 565 attempted to replace stale SPF records.
398:- Issue: backup write permission problem caused record detection failure, so new v=spf1 -all records were created while old v=spf1 ip4:87.236.18.45 records remained.
402:## MSMTP_SECRET_SOURCE_PARSE_ERROR_576_20260701
403:- Context: proof 576 installed msmtp but sendmail auth test failed.
404:- Issue: [SENSITIVE_PATH] was sourced as shell, but SMTP password contained shell-special characters; source failed and auth used an invalid/truncated secret path.
408:## FORUM_MSMTP_MAIL_TRANSPORT_STILL_FAILING_20260701
410:- Issue: one or more sendmail/PHP mail tests still failed.
415:- Context: SMTP password was exposed in terminal output during failed msmtp setup.
416:- Cause: secret file with raw SMTP_PASS was later parsed/sourced by shell; shell error printed the raw assignment line.
426:## FORUM_SMTP_FAILED_CONFIG_DISABLED_AFTER_MAILCOW_BAN_20260701
429:- Action: remove forum-prod msmtp secret/config files so XenForo cannot keep retrying broken SMTP auth.
432:## FORUM_OLD_CODEVIPE_CLOUD_BACKUP_STALE_AFTER_FRESH5_20260701
435:- Evidence: latest service run failed with missing /root/backups/codevipe; remote inventory contains old codevipe-public/codevipe.sql files, not current five-forum archive.
445:## FORUM_XENFORO_MAIL_SMOKE_TEST_FAILED_20260701
450:- Context: custom XenForo mail smoke test 623 failed with Could not open input file /root/scripts/623_xenforo_mail_smoke.php.
456:## FORUM_NEXT_CHAT_KNOWN_ERRORS_AND_CLOSED_INCIDENTS_20260701
466: - Custom mail proof 623 failed with "Could not open input file".
471:- FORUM_MSMTP_MAIL_TRANSPORT_STILL_FAILING_20260701 / FORUM_SMTP_FAILED_CONFIG_DISABLED_AFTER_MAILCOW_BAN_20260701:
472: - Earlier SMTP attempts failed and triggered mailcow netfilter ban for forum public IP.
489: - Classified as external timeout noise, not runtime failure.
493:- Classification: deliverability/reputation/content filtering, not server failure.
509:## PARKED_DOMAINS_STAGE4_DNS01_PREFLIGHT_FAILED_20260701
512:- Observed: stage4 stopped safely with preflight_failed; CERTBOT_PRESENT=no, CERTBOT_DNS_CLOUDFLARE_PLUGIN=no, and CF_ZONE_ACCESS=no for newfi.ru, hapusya.ru and kingofwolk.ru.
513:- Impact: no parked-domain NPMplus route/certificate changes were applied by stage4; public HTTPS validation still failed certificate verification.
550:- Stage9 combined server_name block passed 5/6 but failed bare newfi.ru, so final HTTP-01 cutover must use separate exact per-host server blocks.
553:## PARKED_DOMAINS_STAGE11_LOW_ID_EXACT_ROUTE_FAILED_20260701
560:## PARKED_DOMAINS_SWITCH_TO_DNS01_AFTER_HTTP01_FAILURES_20260701
569:- Rule: for parked domains finalization, use existing DNS-01 certificates and validate from pve01 public HTTPS with content marker; do not use the failed local --resolve TLS check as the closure gate.
579:- Stage18 parked dry-run note: kingofwolk.ru dry-run hit Let's Encrypt rateLimited/service-busy after active certificate issuance and public HTTPS were already OK. Treat as transient external dry-run failure, not route/cert closure blocker.
587:## PVEPRO_STAGE21_LANDING_SERVICE_FAILED_20260701
593:- Context: Stage22 succeeded on edge-side landing, cert, NPMplus route and Cloudflare A upserts, but wrapper had a non-critical scp wildcard failure due __pycache__ and public validation used cached DNS for root pvepro.ru.
594:- Mistakes recorded: do not scp scripts/* after local py_compile because __pycache__ may be present; do not continue after critical scp failure in future apply stages; do not classify PVEPro landing as NetBird just because it contains a link text; use Cloudflare authoritative A records and --resolve validation when DNS propagation may lag.
613:- Rule: do not open router public admin or certificate deployment until WireGuard handshake is proven.
617:- Proof 666 is superseded because it showed PUBLIC_SSH_22_STILL_OPEN.
620:## TAFTAUTO_CERTBOT_CLOUDFLARE_PLUGIN_MISSING_20260702
621:- Certbot on edge rejected --dns-cloudflare-* arguments: dns-cloudflare plugin missing/not loaded.
622:- No taftauto certificate issued in failed run; no DNS A records or NPMplus routes changed.
634:- No certificate, DNS, NPMplus route, or deploy state changed by the failed dry-run.
637:- Proof 672 is superseded/partial: it printed STATUS=OK even though certbot dry-run failed.
638:- Failure reason: Let's Encrypt staging/service returned rateLimited / Service busy; retry later.
643:## HOMEPAGE_XENFORO_INFO_INLINE_PYTHON_SYNTAX_ERROR_20260702
645:- Failed before services.yaml write.
649:- Failed before write. Fix uses YAML-aware edit with exact names Router/Public Domain.
655:## 20260702_CF_TOKEN_AUDIT_BROKEN_COMMAND
657:- Факт: bash начал интерпретировать Python-код, появились syntax error и вторичный prompt >.
671:- Do not touch Cloudflare Homepage card when operator says it is green and opens correctly.
675:- When asked to fix one missing Uptime Kuma service, add only one monitor, then generate proposals separately.
678:- Do not touch Cloudflare when operator says it is green and opens correctly.
680:## LESSON_20260702_DOCKGE_STALE_STACKS_NOT_CONTAINERS
681:- Dockge inactive items after migration can be stale compose folders, not stopped containers.
684:- Do not delete containers to fix Dockge inactive UI; connect agents and archive only confirmed stale moved stack definitions.
686:## STAGE4B_MASKED_SQL_QUERY_FAILURE_20260714
688:- Empty blocks are query failures, not proof that metadata, triggers or functions are absent.
689:- Future PostgreSQL audits must use ON_ERROR_STOP, visible stderr, independent query RC and quote-safe dollar literals.
693:- PostgreSQL returned column id does not exist and ON_ERROR_STOP correctly stopped all following SQL checks.
698:- pg_dump failed only because fsync on /dev/null returned Invalid argument.
702:## STAGE4C_REMOTE_PREP_THIRD_SUBSTEP_FAILED_20260714
713:## STAGE4C_MIGRATION003_FINGERPRINT_DOLLAR_QUOTE_FAILURE_20260714
714:- Migration 003 isolated dry-run failed at fingerprint CHECK because the regex end anchor collided with the SQL dollar-quote delimiter and produced an unterminated dollar-quoted string.
727:- This false expectation caused diagnostic RC=32 without any infrastructure failure or production change.
737:- The six wrapper validation failures are cascading missing-marker checks and do not prove six independent infrastructure faults.
750:- The exact traceback and failing assertion must be captured by an isolated identical reproduction before changing the candidate.
753:- Adapter candidate passed Python 3.13 but failed Python 3.11 parsing at Path(row["path"]) inside a double-quoted f-string.
755:- The reproduction command captured the expected compile failure but incorrectly returned CAPTURE_BAD=1 because its validator required compile RC=0.
761:- The wrapper validation count is cascading missing-marker evidence and does not represent twenty-one independent faults.
767:- This caused UnicodeDecodeError before any design assertion failed.
772:## STAGE4E_V2_VALIDATOR_FAILED_STATIC_COMPILE_20260714
773:- Stage4E design candidate v2 failed static compilation before local design validation started.
774:- The reported validation count is cascading missing-marker evidence, not twenty-five independent design defects.
775:- The generated validator must be inspected at the exact SyntaxError line before another candidate is created.
787:- The executed backup script does not write that health file; therefore the stale health timestamp was not evidence of backup failure.
792:- Failure class: переход к следующей задаче при наличии незакрытого хвоста.
794:- Closure gate: UNRESOLVED_TAIL_COUNT=0; BLOCKER_COUNT=0; TEMPORARY_ARTIFACT_COUNT=0; ROLLBACK_VERIFIED=YES; HEALTH=OK; PROOF_UPDATED=YES; REFERENCE_UPDATED=YES.
795:- Если закрытие невозможно из-за внешней зависимости, опасной неоднозначности или обязательного решения пользователя, фиксировать BLOCKED/OPEN и не заявлять CLOSED.
798:## ERROR_20260714_DEPENDENCY_FACT_QUERY_GLOBAL_LIMIT
806:## ERROR_20260714_COLLECTOR_PATCH_REQUIREMENT_PATTERN_MISMATCH
812:- Status: CLOSED; PRIOR_TAIL_UNRESOLVED_COUNT=0.
814:## ERROR_20260714_PRIVILEGED_FILE_REDIRECTION_BEFORE_SUDO
816:- Root cause: `< collector.py` was opened by the unprivileged caller shell before sudo executed wc.
822:## ERROR_20260714_COLLECTOR_RUN_COLUMN_ASSUMPTION
823:- Symptom: SQL failed because completed_at did not exist.
824:- Root cause: collector_runs columns were assumed instead of read from information_schema; actual names are finished_at and error_text.
828:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
830:## ERROR_20260714_REMOTE_STDIN_ARGUMENT_SHIFT
836:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
838:## ERROR_20260714_OVERSIZED_COMMAND_PARSE_FAILURE
839:- Symptom: Bash получил одиночный фрагмент и сообщил syntax error near unexpected token `(`.
845:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
847:## ERROR_20260714_HOMELAB_ADMIN_HELP_RC_ASSUMPTION
848:- Symptom: template audit завершился HOMELAB_ADMIN_HELP_FAILED.
854:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
856:## ERROR_20260715_EXPECTED_NEGATIVE_RC_TRAP_CLASSIFICATION
859:- Correction: expected failures are evaluated only through an explicit if/else branch with immediate RC capture.
862:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
864:## ERROR_20260715_DUPLICATE_MACHINE_MARKER_COUNT
865:- Symptom: error-register candidate construction stopped with RC1 before applying the candidate.
872:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
875:## ERROR_20260715_OUTER_WRAPPER_SHELL_SYNTAX
876:- Symptom: the operator shell rejected the generated one-line wrapper with syntax error near the final subshell parenthesis.
878:- Impact: parsing failed before the start marker, guards and homelab-admin invocation; no task phase or infrastructure action executed.
881:- Anti-regression: after any parser-level failure, treat the attempted phase as not executed and verify package and run-state before retrying.
885:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
888:## ERROR_20260715_INLINE_REMOTE_SHELL_IN_APPLY_PHASE
899:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
902:## ERROR_20260715_OPTIONAL_RUN_DIRECTORY_FIND_UNDER_PIPEFAIL
904:- Root cause: find was executed against an optional directory that did not exist; under pipefail the assignment returned RC1.
906:ANTI_REGRESSION_TOKEN=MISSING_OPTIONAL_RUN_DIRECTORY_TREATED_AS_ZERO
907:- Anti-regression: optional paths must have an explicit existence branch before find under pipefail.
909:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
912:## ERROR_20260715_PG_CONSTRAINT_CONTYPE_CHAR_CONCAT
913:- Symptom: isolated Stage4H acceptance failed during schema baseline capture.
922:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
925:## ERROR_20260715_LIVE_COLLECTOR_VERIFIER_WRONG_PATH
933:- Status: CLOSED; UNRESOLVED_TAIL_COUNT=0.
936:AUTOREFRESH_INLINE_FAILURE_20260721
940:AUTOREFRESH_WRAPPER_PUBLICATION_FAILURE_20260721
948:BACKUP-AUDIT-CONTROLPLANE-054 was successfully published at commit 2b966acfe553b4d9f8d4a62bf9533478bdd1a073, but was incorrectly reported missing because a stale directory listing was treated as authoritative.
951:BACKUP_UNSAFE_SOURCE_CAPTURE_071_PARSE_ERROR_20260721
952:BACKUP-UNSAFE-SOURCE-CAPTURE-071 failed before execution: shell syntax error near unexpected token (.
=== REFERENCE_TAIL ===
- Existing NPMplus Admin monitor was repaired to http://[PRIVATE_IP]:18091/npmplus, not deleted.
- Edge health wrapper /usr/local/sbin/router-moscow-health provides safe endpoints on [PRIVATE_IP]:18091 for router-moscow, npmplus, loki, alloy, cadvisor, registry-cache, socket-proxy, diun, kopia and crowdsec.
- Cloudflare was not touched by explicit operator request.
- Uptime Kuma DB backups are under /root/uptime-kuma-manual-backups/773d-approved-batch-* on monitoring VM before the DB mutation.
## UPTIME_KUMA_PENDING_HEALTH_MONITORS_FIX_20260702
- Four pending Uptime Kuma keyword monitors were repaired by using local pve01 health endpoint paths instead of public backup.gram1.ru paths:
- Backup Restore Dashboard -> http://[PRIVATE_IP]:9101/backup-restore-dashboard.txt
- Restore Drill Index -> http://[PRIVATE_IP]:9101/restore-drill-index.txt
- Paperless Restore -> http://[PRIVATE_IP]:9101/paperless-restore.txt
- AdGuard Rewrite Sync -> http://[PRIVATE_IP]:9101/adguard-rewrite-sync.txt
- Cloudflare was not touched.
- Monitors were repaired, not deleted.
## UPTIME_KUMA_FOUR_HEALTH_SOURCES_REPAIRED_20260702
- Four Uptime Kuma monitors were still unavailable because three health endpoints returned 404 and Paperless Restore had STATUS=ERROR.
- Repaired pve01 health source files under /var/lib/homelab-health:
- backup-restore-dashboard.txt
- restore-drill-index.txt
- paperless-restore.txt
- adguard-rewrite-sync.txt
- Kuma monitors point to local pve01 health endpoints on http://[PRIVATE_IP]:9101/.
- Cloudflare was not touched.
- Monitors were repaired, not deleted.
## DOCKGE_DISCONNECTED_HOSTS_AND_STALE_STACKS_20260702
- Dockge is currently present only on edge-vm.
- core-apps and monitoring run Docker containers but have no Dockge/Dockge-agent detected.
- Dockge "2 not connected" should be interpreted as two disconnected Docker hosts unless later evidence shows otherwise.
- No stopped containers were found across edge-vm, core-apps and monitoring during analysis.
- Edge Dockge has stale inactive stack definitions after services moved to core-apps and monitoring.
- Do not delete containers. First connect remote hosts or archive stale stack definitions after classification.
## DOCKGE_REMOTE_AGENTS_INSTALLED_20260702
- Dockge main instance runs on edge-vm [PRIVATE_IP].
- Remote Dockge agents were installed on:
- core-apps [PRIVATE_IP]:5001
- monitoring [PRIVATE_IP]:5001
- Edge Dockge agent table stores the two agent URLs with username/password; password must never be printed.
- No runtime containers were deleted.
- Stale edge stack definitions were not archived yet; verify Dockge UI connected state first.
## DOCKGE_REMOTE_AGENTS_AND_STALE_ARCHIVE_FINAL_20260702
- Dockge main instance: edge-vm [PRIVATE_IP].
- Remote agents connected: core-apps [PRIVATE_IP]:5001, monitoring [PRIVATE_IP]:5001.
- Stale moved edge stack folders archived under /opt/dockge-stale-archive/20260702T230442Z on edge-vm; no runtime containers were deleted.
- External remote stacks exposed to Dockge using bind mounts: /opt/vaultwarden-compose, /opt/gotify-compose, /opt/uptime-kuma-compose into /opt/stacks.
- Final target: remote not_visible count must be 0 and edge stale remaining matches must be 0.
## DOCKGE_FINAL_MANAGED_STACKS_20260702
- Dockge main stack, npmplus and remote dockge-agent stacks were recreated from /opt/stacks so Dockge can manage them.
- Edge Dockge listens on 127.0.0.1:5001; LAN probe to [PRIVATE_IP]:5001 may return 000 and is not the correct health proof.
- Remote agents listen on [PRIVATE_IP]:5001 and [PRIVATE_IP]:5001.
- Stale moved stacks were archived, not deleted.
## SERVICE_SETUP_ROADMAP_DEEP_20260702
- Configuration order: secrets, DNS/proxy/TLS, backup/restore, monitoring, SSO, docs, core data apps, sensitive apps, productivity, media, automation, utilities.
- Do not configure data-heavy or automation services before backup and monitoring are proven.
- Per-service done criteria: URL, TLS, admin, SSO decision, volumes, backup, restore test, Kuma, Homepage, docs.
## PHASE00_FREEZE_CLOSED_20260702
- Service inventory normalized: no manual containers and no compose projects outside /opt/stacks remain.
- Registry cache is managed by Dockge/Compose: project registry-cache, compose /opt/stacks/registry-cache/compose.yaml.
- Old systemd unit homelab-registry-cache.service was disabled and masked because it recreated the container without Compose labels.
- Correct registry health URL is http://[PRIVATE_IP]:5000/v2/.
- Next phase: Vaultwarden and break-glass secret baseline.
## PHASE00_FREEZE_CLOSED_20260702
- Service inventory normalized: no manual containers and no compose projects outside /opt/stacks remain.
- Registry cache is managed by Dockge/Compose: project registry-cache, compose /opt/stacks/registry-cache/compose.yaml.
- Old systemd unit homelab-registry-cache.service was archived and masked because it recreated the container without Compose labels.
- Correct registry health URL is http://[PRIVATE_IP]:5000/v2/.
- Next phase: Vaultwarden and break-glass secret baseline.
## PHASE00_FREEZE_CLOSED_20260702
- Service inventory normalized: no manual containers and no compose projects outside /opt/stacks remain.
- Registry cache is managed by Dockge/Compose: project registry-cache, compose /opt/stacks/registry-cache/compose.yaml.
- Old systemd unit homelab-registry-cache.service was archived and masked with /dev/null because it recreated the container without Compose labels.
- Correct registry health URL is http://[PRIVATE_IP]:5000/v2/.
- Next phase: Vaultwarden and break-glass secret baseline.
## OPERATOR_RULE_CLOSE_TAILS_IMMEDIATELY_20260714
- Жёсткое правило: хвосты не оставлять. Любая проблема, временный артефакт, неудачная проверка, блокер или анти-регрессия, обнаруженные в текущем scope, должны быть устранены и закрыты до перехода к следующей задаче.
- После любой неуспешной команды текущая задача остаётся активной до установления точной причины, исправления, проверки исправления, добавления анти-регрессии, очистки временных артефактов и выпуска закрывающего proof.
- Этап разрешено помечать CLOSED только при UNRESOLVED_TAIL_COUNT=0, BLOCKER_COUNT=0, TEMPORARY_ARTIFACT_COUNT=0, проверенном rollback, зелёном health, обновлённых proof и reference.
- Запрещено откладывать выявленный хвост только ради удобства или перехода к следующему этапу.
- Если технически необходим отдельный подэтап, он должен быть ограничен точным scope, выполнен и закрыт немедленно до возврата к основной работе.
- Исключения: внешняя зависимость, опасная неоднозначность, риск раскрытия секрета, отсутствующий обязательный файл или явное решение пользователя. В таком случае статус должен быть BLOCKED или OPEN с точным блокером; статус CLOSED запрещён.
- Следующий этап не начинается, пока текущий хвост не закрыт.
- Proof: /var/lib/homelab-change-sets/operator-rule-close-tails-immediately-v1/20260714T171631Z/report.txt
- Secret scan: /var/lib/homelab-change-sets/operator-rule-close-tails-immediately-v1/20260714T171631Z/strict-secret-scan.txt
## CLUSTER_ADMIN_STAGE4H_SCOPE_DEFINED_20260714
- Stage4G остаётся закрытым и стабильно работает в режиме observe-notify.
- Закрыты два хвоста dependency-аудита: глобальный LIMIT всего UNION ALL и узкий шаблон collector_patch_required.
- Migration plan содержит collector_patch_required=true; dynamic contract содержит collector_patch_required_after_stage4c=true. Это два подтверждения одного обязательного требования.
- Migration 003 не применена, 12 новых столбцов отсутствуют.
- Migration выполняет полный UPDATE observations, четыре SET NOT NULL, пять VALIDATE CONSTRAINT и пять неконкурентных CREATE INDEX.
- Текущий collector не формирует обязательный provenance-набор; готовый collector patch отсутствует.
- Production adapters и production apply phase отсутствуют.
- Отдельный apply migration 003 без collector patch запрещён.
- Безопасный порядок: collector patch candidate → временная БД и acceptance/rollback → контролируемый migration+collector cutover → adapter integration.
- Текущий этап открыт: 4H_COLLECTOR_PROVENANCE_PATCH_AND_MIGRATION003_PREAPPLY_READINESS.
- Предыдущие хвосты закрыты; Stage4H не считается CLOSED до реализации и полного seal.
- Proof: /var/lib/homelab-cluster-admin-stage4/stage4h-collector-provenance-readiness-scope-v1/report.txt
- Secret scan: /var/lib/homelab-cluster-admin-stage4/stage4h-collector-provenance-readiness-scope-v1/strict-secret-scan.txt
## CLUSTER_ADMIN_STAGE4H_COLLECTOR_BASELINE_AUDIT_CLOSED_20260714
- Исправленный baseline-аудит collector завершён.
- collector.py: 113 строк, SHA256 подтверждён.
- Привилегированные файлы нельзя читать через caller-side `< file` перед sudo; применён sudo wc без внешнего перенаправления.
- Фактические поля collector_runs: finished_at и error_text; completed_at и error отсутствуют.
- Найдены один canonical source instance и один collector_runs_foundation_enrich_trg.
- Trigger обогащает collector_runs полями source_instance_id и run_key.
- Текущий collector ещё не пишет provenance-поля observations; migration 003 не применена.
- Production осталась 0|0|OK, timer активен, failed units отсутствуют.
- Baseline-аудит закрыт без хвостов; Stage4H остаётся OPEN для isolated collector provenance patch candidate.
- Proof: /var/lib/homelab-cluster-admin-stage4/stage4h-collector-baseline-audit-final-v1/report.txt
- Secret scan: /var/lib/homelab-cluster-admin-stage4/stage4h-collector-baseline-audit-final-v1/strict-secret-scan.txt
## CLUSTER_ADMIN_STAGE4H_COLLECTOR_PATCH_CANDIDATE_CLOSED_20260714
- Из точного live collector создан isolated provenance patch candidate; production collector не изменён.
- Candidate пишет все 12 migration003 provenance-полей и применяет SHA256 fallback-дедупликацию.
- Metadata разрешает только instance, job, mountpoint и device, ограничена по типам и размеру.
- Candidate скомпилирован локально и проверен на VM180 Python 3.11.2 от clusteradmin.
- SSH stdin harness исправлен: bash -s получает ровно path, SHA256 и bytes; первый аргумент дополнительно проверяется по безопасному шаблону.
- Self-test и отрицательный CLI-тест RC64 прошли; временный remote-файл удалён.
- Production осталась 0|0|OK; live collector и timer не изменены.
- Неудачная попытка закрыта без хвостов; Stage4H остаётся OPEN до temporary-DB acceptance и controlled cutover.
- Candidate root: /var/lib/homelab-change-sets/stage4h-collector-provenance-patch-candidate-v1/20260714T174131Z
- Proof: /var/lib/homelab-change-sets/stage4h-collector-provenance-patch-candidate-v1/20260714T174131Z/report.txt
- Secret scan: /var/lib/homelab-change-sets/stage4h-collector-provenance-patch-candidate-v1/20260714T174131Z/strict-secret-scan.txt
## OPERATOR_RULE_NO_OVERSIZED_MONOLITH_COMMANDS_20260714
- Запрещены чрезмерно длинные интерактивные однострочные команды с глубокой вложенностью SSH, SQL, Python, awk и кавычек.
- Рекомендуемый предел интерактивной команды: 8000 байт. Превышение допускается только для простого текста без вложенных языков.
- Сложная операция оформляется как отдельный versioned task-скрипт с contract, syntax-check, dry-run, manifest, rollback и proof.
- Создание task-скрипта и его запуск выполняются разными короткими командами.
- Remote stdout/stderr всегда сохраняются до проверки RC; запрещено терять вывод из-за errexit-sensitive command substitution.
- Перед запуском проверяются SHA256, размер, владелец, режим и синтаксис task-скрипта.
- Обрезанная, повреждённая или частично вставленная команда считается невыполненной; сначала проводится side-effect audit.
- Хвост parse-ошибки закрывается только после доказательства отсутствия изменений и временных артефактов.
- Proof: /var/lib/homelab-change-sets/operator-command-size-guard-v1/20260714T182653Z/report.txt
## HOMELAB_ADMIN_CLI_CONTRACT_20260714
- Поддерживаемые команды: version, status, lint TASK_DIR, validate TASK_DIR, run TASK_DIR PHASE.
- Отдельная команда --help отсутствует. Неизвестный аргумент печатает usage и штатно возвращает RC=64.
- Нельзя трактовать RC=64 от --help как неисправность runner.
- Для проверки доступности runner использовать homelab-admin version; для структуры CLI использовать зафиксированную usage-строку.
- Proof: /var/lib/homelab-change-sets/homelab-admin-cli-contract-fix-v1/20260714T183338Z/report.txt
AUTOMATIC_CLUSTER_CONTEXT_REFRESH_20260721
Collector: /usr/local/sbin/homelab-context-collect
Scheduler: hourly at minute 17 via /etc/cron.d/homelab-context-refresh
Destination: immutable Gitea runtime history with CONTEXT-AUTO UTC command IDs.
AUTOMATIC_CLUSTER_CONTEXT_REFRESH_DIRECT_20260721
Publisher: /usr/local/sbin/homelab-context-refresh-direct
Schedule: hourly at minute 17.
Destination: immutable assistant-runtime history with CONTEXT-AUTO UTC IDs.
=== BOOTSTRAP ===
{
"schema_version": 2,
"status": "READY",
"generated_at": null,
"coverage": {
"status": "PARTIAL",
"missing": [
"automatic_refresh",
"per_node_freshness",
"structured_service_inventory"
]
},
"freshness": {
"max_age_seconds": 86400,
"stale_requires_refresh": true
},
"entrypoints": {
"full_context_url": "https://git.gram1.ru/homelab-admin/homelab-public-context/raw/commit/e23caf794cc68ce5fa6f8922efa7f2a35c8aa6e0/runtime/history/CONTEXT-SOURCES-FULL-002.txt",
"errors_index_url": "https://git.gram1.ru/homelab-admin/homelab-public-context/raw/commit/a615069de3d32564fc4de29baff949bdaf811caf/runtime/history/ERRORS-INDEX-004.json",
"workflow_contract_url": "https://git.gram1.ru/homelab-admin/homelab-public-context/raw/commit/2f37a4eb600528d03f5c570af8c5aa0e463b1798/runtime/history/WORKFLOW-CONTRACT-003.json",
"double_check_rule_url": "https://git.gram1.ru/homelab-admin/homelab-public-context/raw/commit/d1061e47e7955dba9c73a2841d4eacf02ccc1bf5/runtime/policy/double-check-rule.json"
}
}
=== CLUSTER_STATUS ===
Cluster information
-------------------
Name: homelab
Config Version: 3
Transport: knet
Secure auth: on
Quorum information
------------------
Date: Tue Jul 21 16:21:08 2026
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000001
Ring ID: 1.9d
Quorate: Yes
Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 [PRIVATE_IP] (local)
0x00000002 1 [PRIVATE_IP]
0x00000003 1 [PRIVATE_IP]
Membership information
----------------------
Nodeid Votes Name
1 1 pve01 (local)
2 1 pve03
3 1 pve02
=== CLUSTER_RESOURCES ===
{"type":"lxc","node":"pve01","vmid":110,"name":"dns1","status":"running","uptime":448924,"maxcpu":1,"cpu":0.000299025656238528,"maxmem":536870912,"mem":90935296,"maxdisk":8350298112,"disk":1933914112,"storage":null}
{"type":"lxc","node":"pve02","vmid":111,"name":"dns2","status":"running","uptime":448922,"maxcpu":1,"cpu":0.00118032586667801,"maxmem":536870912,"mem":77062144,"maxdisk":8350298112,"disk":1435082752,"storage":null}
{"type":"lxc","node":"pve01","vmid":112,"name":"unbound1","status":"running","uptime":448927,"maxcpu":1,"cpu":0.0000996752187461762,"maxmem":536870912,"mem":41971712,"maxdisk":8350298112,"disk":757272576,"storage":null}
{"type":"lxc","node":"pve02","vmid":113,"name":"unbound2","status":"running","uptime":448925,"maxcpu":1,"cpu":0.0000983604888898342,"maxmem":536870912,"mem":31571968,"maxdisk":8350298112,"disk":750432256,"storage":null}
{"type":"qemu","node":"pve03","vmid":130,"name":"edge-vm","status":"running","uptime":448787,"maxcpu":4,"cpu":0.0296798201262478,"maxmem":12884901888,"mem":10835238912,"maxdisk":103079215104,"disk":0,"storage":null}
{"type":"qemu","node":"pve01","vmid":150,"name":"nextcloud","status":"running","uptime":448922,"maxcpu":4,"cpu":0.0224714397065076,"maxmem":8589934592,"mem":5494030336,"maxdisk":68719476736,"disk":0,"storage":null}
{"type":"qemu","node":"pve02","vmid":160,"name":"forum-prod","status":"running","uptime":448920,"maxcpu":4,"cpu":0.00147539847586977,"maxmem":8589934592,"mem":8104173568,"maxdisk":214748364800,"disk":0,"storage":null}
{"type":"qemu","node":"pve01","vmid":170,"name":"core-apps","status":"running","uptime":448918,"maxcpu":4,"cpu":0.0397571625576673,"maxmem":10737418240,"mem":10081083392,"maxdisk":128849018880,"disk":0,"storage":null}
{"type":"qemu","node":"pve01","vmid":171,"name":"monitoring","status":"running","uptime":448915,"maxcpu":4,"cpu":0.0190142951362757,"maxmem":8589934592,"mem":4098203648,"maxdisk":128849018880,"disk":0,"storage":null}
{"type":"qemu","node":"pve02","vmid":180,"name":"cluster-admin","status":"running","uptime":448917,"maxcpu":2,"cpu":0.0201637791702202,"maxmem":2147483648,"mem":1965608960,"maxdisk":25769803776,"disk":0,"storage":null}
{"type":"qemu","node":"pve03","vmid":190,"name":"homelab-ops-worker","status":"running","uptime":444150,"maxcpu":4,"cpu":0.00197865467508319,"maxmem":8589934592,"mem":2149752832,"maxdisk":85899345920,"disk":0,"storage":null}
{"type":"qemu","node":"pve02","vmid":9130,"name":"edge-cold-standby","status":"stopped","uptime":0,"maxcpu":4,"cpu":0,"maxmem":12884901888,"mem":0,"maxdisk":103079215104,"disk":0,"storage":null}
{"type":"node","node":"pve02","vmid":null,"name":null,"status":"online","uptime":448958,"maxcpu":4,"cpu":0.0202669303015324,"maxmem":16536748032,"mem":12247134208,"maxdisk":105089261568,"disk":27718709248,"storage":null}
{"type":"node","node":"pve03","vmid":null,"name":null,"status":"online","uptime":448804,"maxcpu":8,"cpu":0.0187554341075643,"maxmem":33553518592,"mem":14079365120,"maxdisk":105089261568,"disk":55840108544,"storage":null}
{"type":"node","node":"pve01","vmid":null,"name":null,"status":"online","uptime":448941,"maxcpu":24,"cpu":0.0113622226829774,"maxmem":33379028992,"mem":21412495360,"maxdisk":105089261568,"disk":42922749952,"storage":null}
{"type":"storage","node":"pve02","vmid":null,"name":null,"status":"available","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":846016282624,"disk":290521991453,"storage":"local-lvm"}
{"type":"storage","node":"pve03","vmid":null,"name":null,"status":"available","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":355660201984,"disk":155636904388,"storage":"local-lvm"}
{"type":"storage","node":"pve01","vmid":null,"name":null,"status":"available","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":869168840704,"disk":286912634316,"storage":"local-lvm"}
{"type":"storage","node":"pve02","vmid":null,"name":null,"status":"available","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":105089261568,"disk":27718709248,"storage":"local"}
{"type":"storage","node":"pve03","vmid":null,"name":null,"status":"available","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":105089261568,"disk":55840108544,"storage":"local"}
{"type":"storage","node":"pve01","vmid":null,"name":null,"status":"available","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":105089261568,"disk":42922749952,"storage":"local"}
{"type":"network","node":"pve02","vmid":null,"name":null,"status":"ok","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":null,"disk":null,"storage":null}
{"type":"network","node":"pve03","vmid":null,"name":null,"status":"ok","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":null,"disk":null,"storage":null}
{"type":"network","node":"pve01","vmid":null,"name":null,"status":"ok","uptime":null,"maxcpu":null,"cpu":null,"maxmem":null,"mem":null,"maxdisk":null,"disk":null,"storage":null}
=== STORAGE ===
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
local dir active 102626232 41916788 55450180 40.84%
local-lvm lvmthin active 848797696 280188119 568609576 33.01%
=== BACKUP_JOBS ===
[{"compress":"zstd","enabled":1,"id":"homelab-nightly-all","mailnotification":"failure","mode":"snapshot","next-run":1784680200,"prune-backups":{"keep-daily":"4"},"schedule":"03:30","storage":"local","type":"vzdump","vmid":"110,111,112,113,130,150,190"}]
=== BACKUP_MATRIX ===
STATUS=FAIL
TS=2026-07-21T07:02:41Z
TYPE=unified-backup-matrix
MATRIX_MODE=authoritative
JOBS_TOTAL=41
JOBS_OK=30
JOBS_BAD=6
JOBS_UNMAPPED=0
JOBS_DISABLED=5
STATE_JSON=/var/lib/homelab-backup-matrix/state.json
MATRIX_JSON=/etc/homelab-backup-matrix/matrix.json
AUTHORITY=canonical
COMPAT_OUTPUTS=29
COMPAT_MAP=/var/lib/homelab-backup-matrix/compat-map.json
JOB=dpkg-db-backup STATUS=OK EVIDENCE=execution_only AGE=36147 MAX_AGE=172800
JOB=filebrowser-backup STATUS=OK EVIDENCE=artifact_or_health AGE=14443 MAX_AGE=259200
JOB=filebrowser-restore-validate STATUS=STALE EVIDENCE=artifact_or_health AGE=1016424 MAX_AGE=864000
JOB=homelab-appbackup-dockge-stacks STATUS=OK EVIDENCE=artifact_or_health AGE=22248 MAX_AGE=172800
JOB=homelab-appbackup-mariadb-batch STATUS=OK EVIDENCE=execution_only AGE=15387 MAX_AGE=172800
JOB=homelab-appbackup-npmplus-edge STATUS=OK EVIDENCE=artifact_or_health AGE=20573 MAX_AGE=172800
JOB=homelab-appbackup-postgres-batch STATUS=OK EVIDENCE=execution_only AGE=16416 MAX_AGE=172800
JOB=homelab-appbackup-sqlite-batch STATUS=OK EVIDENCE=execution_only AGE=18327 MAX_AGE=172800
JOB=homelab-authentik-backup STATUS=DISABLED EVIDENCE=artifact_or_health AGE=987790 MAX_AGE=172800
JOB=homelab-auto-backup STATUS=OK EVIDENCE=execution_only AGE=195424 MAX_AGE=864000
JOB=homelab-backup-coverage-matrix STATUS=OK EVIDENCE=artifact_or_health AGE=4346 MAX_AGE=172800
JOB=homelab-backup-freshness STATUS=OK EVIDENCE=execution_only AGE=35589 MAX_AGE=172800
JOB=homelab-backup-healthcheck STATUS=OK EVIDENCE=execution_only AGE=10631 MAX_AGE=172800
JOB=homelab-backup-sla-health STATUS=FAIL EVIDENCE=artifact_or_health AGE=144 MAX_AGE=172800
JOB=homelab-edge-vm-offhost-freshness STATUS=OK EVIDENCE=artifact_or_health AGE=18426 MAX_AGE=172800
JOB=homelab-emergency-backup STATUS=OK EVIDENCE=artifact_or_health AGE=24539 MAX_AGE=172800
JOB=homelab-extended-appbackup STATUS=OK EVIDENCE=artifact_or_health AGE=14156 MAX_AGE=172800
JOB=homelab-gitea-backup STATUS=DISABLED EVIDENCE=artifact_or_health AGE=987785 MAX_AGE=172800
JOB=homelab-golden-state-index STATUS=OK EVIDENCE=artifact_or_health AGE=4112 MAX_AGE=172800
JOB=homelab-mail-cloud-critical-upload STATUS=OK EVIDENCE=artifact_or_health AGE=11997 MAX_AGE=172800
JOB=homelab-mail-cloud-edge-vm-upload STATUS=OK EVIDENCE=artifact_or_health AGE=174045 MAX_AGE=864000
JOB=homelab-mail-cloud-nextcloud-vm-upload STATUS=OK EVIDENCE=artifact_or_health AGE=7575 MAX_AGE=864000
JOB=homelab-mail-cloud-restore-drill STATUS=STALE EVIDENCE=artifact_or_health AGE=988269 MAX_AGE=864000
JOB=homelab-npmplus-kuma-config-backup STATUS=DISABLED EVIDENCE=artifact_or_health AGE=60071 MAX_AGE=172800
JOB=homelab-router-running-config-mail-cloud STATUS=OK EVIDENCE=artifact_or_health AGE=17076 MAX_AGE=172800
JOB=homelab-vaultwarden-backup STATUS=DISABLED EVIDENCE=artifact_or_health AGE=987768 MAX_AGE=172800
JOB=homelab-verified-backup-health STATUS=OK EVIDENCE=artifact_or_health AGE=11845 MAX_AGE=172800
JOB=homelab-vm-backup-health STATUS=OK EVIDENCE=artifact_or_health AGE=8742 MAX_AGE=172800
JOB=homelab-vm-backup-policy STATUS=OK EVIDENCE=artifact_or_health AGE=5548 MAX_AGE=172800
JOB=homelab-vm-full-backup STATUS=OK EVIDENCE=artifact_or_health AGE=24778 MAX_AGE=172800
JOB=homelab-vm-full-restore-drill STATUS=OK EVIDENCE=artifact_or_health AGE=523085 MAX_AGE=10368000
JOB=homelab-vm-local-dumps-2cloud STATUS=OK EVIDENCE=artifact_or_health AGE=172141 MAX_AGE=864000
JOB=homelab-vm-local-dumps-retention STATUS=OK EVIDENCE=artifact_or_health AGE=168839 MAX_AGE=864000
JOB=immich-media-offhost-sync STATUS=OK EVIDENCE=artifact_or_health AGE=4286 MAX_AGE=259200
JOB=memos-backup STATUS=OK EVIDENCE=artifact_or_health AGE=14547 MAX_AGE=259200
JOB=netbird-vps-backup STATUS=FAIL EVIDENCE=artifact_or_health AGE=1747896 MAX_AGE=864000
JOB=paperless-backup STATUS=DISABLED EVIDENCE=artifact_or_health AGE=16347 MAX_AGE=259200
JOB=paperless-restore-dry-run STATUS=STALE EVIDENCE=artifact_or_health AGE=1001986 MAX_AGE=864000
JOB=skladchik-reports-monitor-backup STATUS=OK EVIDENCE=artifact_or_health AGE=19639 MAX_AGE=172800
JOB=skladchik-reports-monitor-backup-restore-check STATUS=OK EVIDENCE=artifact_or_health AGE=190273 MAX_AGE=864000
JOB=vm160-remote-backup STATUS=FAIL EVIDENCE=artifact_or_health AGE=26607 MAX_AGE=172800
========== NODE=pve01 ==========
--- PVE_STATUS ---
{
"status": null,
"uptime": 448946,
"cpu": 0,
"loadavg": [
"0.26",
"0.37",
"0.42"
],
"memory": {
"available": 11658358784,
"free": 9646227456,
"total": 33379028992,
"used": 21720670208
},
"swap": {
"free": 8883740672,
"total": 10737414144,
"used": 1853673472
},
"rootfs": {
"avail": 56780976128,
"free": 62166462464,
"total": 105089261568,
"used": 42922799104
},
"kversion": "Linux 7.0.12-1-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.12-1 (2026-06-09T21:07Z)",
"pveversion": "pve-manager/9.2.3/d0fde103346cf89a"
}
--- FAILED_UNITS ---
● homelab-backup-matrix-refresh.service loaded failed failed Generate unified homelab backup matrix state
● homelab-cluster-admin-pull.service loaded failed failed Pull cluster-admin observer health into pve01
● homelab-vm160-remote-backup.service loaded failed failed VM160 full-image backup streamed from pve02 to pve03
--- TIMERS ---
Tue 2026-07-21 16:23:54 MSK 2min 43s Tue 2026-07-21 16:18:29 MSK 2min 41s ago homelab-crypto-portfolio-chart-health.timer homelab-crypto-portfolio-chart-health.service
Tue 2026-07-21 16:24:14 MSK 3min 3s Tue 2026-07-21 16:19:14 MSK 1min 56s ago homelab-smartctl-textfile.timer homelab-smartctl-textfile.service
Tue 2026-07-21 16:24:14 MSK 3min 3s Tue 2026-07-21 16:19:14 MSK 1min 56s ago netbird-peers-health.timer netbird-peers-health.service
Tue 2026-07-21 16:30:41 MSK 9min Tue 2026-07-21 16:16:27 MSK 4min 43s ago homelab-external-probe-vps-health.timer homelab-external-probe-vps-health.service
Tue 2026-07-21 16:31:06 MSK 9min Tue 2026-07-21 16:16:25 MSK 4min 45s ago homelab-forum-snuffleupagus-health.timer homelab-forum-snuffleupagus-health.service
Tue 2026-07-21 16:31:30 MSK 10min Tue 2026-07-21 16:16:30 MSK 4min 40s ago homelab-disk-space-health.timer homelab-disk-space-health.service
Tue 2026-07-21 16:34:30 MSK 13min Tue 2026-07-21 16:19:30 MSK 1min 40s ago homelab-pve03-staging-capacity-health.timer homelab-pve03-staging-capacity-health.service
Wed 2026-07-22 00:00:00 MSK 7h Tue 2026-07-21 00:00:14 MSK 16h ago dpkg-db-backup.timer dpkg-db-backup.service
Wed 2026-07-22 00:09:43 MSK 7h Tue 2026-07-21 00:09:32 MSK 16h ago homelab-backup-freshness.timer homelab-backup-freshness.service
Wed 2026-07-22 00:10:12 MSK 7h Tue 2026-07-21 00:16:14 MSK 16h ago homelab-docker-health.timer homelab-docker-health.service
Wed 2026-07-22 02:15:00 MSK 9h Tue 2026-07-21 02:15:05 MSK 14h ago homelab-vm-full-backup.timer homelab-vm-full-backup.service
Wed 2026-07-22 03:10:02 MSK 10h Tue 2026-07-21 03:13:14 MSK 13h ago homelab-emergency-backup.timer homelab-emergency-backup.service
Wed 2026-07-22 03:35:00 MSK 11h Tue 2026-07-21 03:35:14 MSK 12h ago filebrowser-backup.timer filebrowser-backup.service
Wed 2026-07-22 03:55:02 MSK 11h Tue 2026-07-21 03:51:25 MSK 12h ago homelab-appbackup-dockge-stacks.timer homelab-appbackup-dockge-stacks.service
Wed 2026-07-22 04:10:00 MSK 11h Tue 2026-07-21 04:10:07 MSK 12h ago filebrowser-restore-validate.timer filebrowser-restore-validate.service
Wed 2026-07-22 04:17:29 MSK 11h Tue 2026-07-21 04:19:17 MSK 12h ago homelab-appbackup-npmplus-edge.timer homelab-appbackup-npmplus-edge.service
Wed 2026-07-22 04:26:54 MSK 12h Tue 2026-07-21 04:28:14 MSK 11h ago homelab-gitea-secondary-backup.timer homelab-gitea-secondary-backup.service
Wed 2026-07-22 04:38:04 MSK 12h Tue 2026-07-21 04:34:48 MSK 11h ago skladchik-reports-monitor-backup.timer skladchik-reports-monitor-backup.service
Wed 2026-07-22 05:01:44 MSK 12h Tue 2026-07-21 04:57:14 MSK 11h ago homelab-appbackup-sqlite-batch.timer homelab-appbackup-sqlite-batch.service
Wed 2026-07-22 05:27:13 MSK 13h Tue 2026-07-21 05:29:05 MSK 10h ago homelab-appbackup-postgres-batch.timer homelab-appbackup-postgres-batch.service
Wed 2026-07-22 05:47:19 MSK 13h Tue 2026-07-21 05:46:14 MSK 10h ago homelab-appbackup-mariadb-batch.timer homelab-appbackup-mariadb-batch.service
Wed 2026-07-22 06:01:19 MSK 13h Tue 2026-07-21 05:57:30 MSK 10h ago homelab-extended-appbackup.timer homelab-extended-appbackup.service
Wed 2026-07-22 06:27:43 MSK 14h Tue 2026-07-21 06:32:25 MSK 9h ago homelab-crypto-portfolio-backup.timer homelab-crypto-portfolio-backup.service
Wed 2026-07-22 06:38:51 MSK 14h Tue 2026-07-21 06:43:54 MSK 9h ago homelab-verified-backup-health.timer homelab-verified-backup-health.service
Wed 2026-07-22 06:39:22 MSK 14h Tue 2026-07-21 06:40:25 MSK 9h ago homelab-alertmanager-backup.timer homelab-alertmanager-backup.service
Wed 2026-07-22 06:52:06 MSK 14h Tue 2026-07-21 06:47:47 MSK 9h ago homelab-beszel-backup.timer homelab-beszel-backup.service
Wed 2026-07-22 06:58:58 MSK 14h Tue 2026-07-21 06:57:30 MSK 9h ago homelab-blackbox-exporter-backup.timer homelab-blackbox-exporter-backup.service
Wed 2026-07-22 07:02:27 MSK 14h Tue 2026-07-21 07:05:30 MSK 9h ago homelab-backup-healthcheck.timer homelab-backup-healthcheck.service
Wed 2026-07-22 07:07:14 MSK 14h Tue 2026-07-21 07:18:55 MSK 9h ago homelab-legacy-monitoring-backups.timer homelab-legacy-monitoring-backups.service
Wed 2026-07-22 07:35:06 MSK 15h Tue 2026-07-21 07:36:59 MSK 8h ago homelab-vm-backup-health.timer homelab-vm-backup-health.service
Wed 2026-07-22 07:38:21 MSK 15h Tue 2026-07-21 07:36:14 MSK 8h ago homelab-cluster-admin-incident-engine-backup.timer homelab-cluster-admin-incident-engine-backup.service
Wed 2026-07-22 07:53:13 MSK 15h Tue 2026-07-21 07:51:14 MSK 8h ago homelab-alerting-health.timer homelab-alerting-health.service
Wed 2026-07-22 08:10:37 MSK 15h Tue 2026-07-21 08:07:25 MSK 8h ago homelab-backup-sla-health.timer homelab-backup-sla-health.service
Wed 2026-07-22 08:18:43 MSK 15h Tue 2026-07-21 08:01:14 MSK 8h ago homelab-overall-health.timer homelab-overall-health.service
Wed 2026-07-22 08:37:02 MSK 16h Tue 2026-07-21 08:30:07 MSK 7h ago homelab-vm-backup-policy.timer homelab-vm-backup-policy.service
Wed 2026-07-22 08:40:13 MSK 16h Tue 2026-07-21 08:50:14 MSK 7h ago homelab-backup-coverage-matrix.timer homelab-backup-coverage-matrix.service
Wed 2026-07-22 08:53:36 MSK 16h Tue 2026-07-21 08:41:14 MSK 7h ago memos-backup.timer memos-backup.service
Wed 2026-07-22 10:10:47 MSK 17h Tue 2026-07-21 10:02:41 MSK 6h ago paperless-restore-dry-run.timer paperless-restore-dry-run.service
Sun 2026-07-26 03:10:21 MSK 4 days Sun 2026-07-19 03:10:29 MSK 2 days ago e2scrub_all.timer e2scrub_all.service
Sun 2026-07-26 03:10:52 MSK 4 days Sun 2026-07-19 03:10:14 MSK 2 days ago xfs_scrub_all.timer xfs_scrub_all.service
Sun 2026-07-26 03:34:47 MSK 4 days Sun 2026-07-19 03:45:37 MSK 2 days ago homelab-auto-backup.timer homelab-auto-backup.service
Sun 2026-07-26 05:17:43 MSK 4 days Sun 2026-07-19 05:11:25 MSK 2 days ago skladchik-reports-monitor-backup-restore-check.timer skladchik-reports-monitor-backup-restore-check.service
Mon 2026-07-27 01:39:15 MSK 5 days Mon 2026-07-20 00:51:14 MSK 1 day 15h ago fstrim.timer fstrim.service
Mon 2026-07-27 07:34:51 MSK 5 days Mon 2026-07-20 07:25:14 MSK 1 day 8h ago homelab-mail-cloud-restore-drill.timer homelab-mail-cloud-restore-drill.service
Thu 2026-10-15 06:00:00 MSK 2 months 24 days Wed 2026-07-15 06:00:00 MSK - homelab-vm-full-restore-drill.timer homelab-vm-full-restore-drill.service
- - Tue 2026-07-21 16:21:10 MSK 161ms ago homelab-health-metrics.timer homelab-health-metrics.service
- - Tue 2026-07-21 16:21:10 MSK 161ms ago homelab-private-vpn-hosts-health.timer homelab-private-vpn-hosts-health.service
- - - - prometheus-node-exporter-smartmon.timer prometheus-node-exporter-smartmon.service
--- HEALTH_FILES ---
2026-06-17T10:54:28.0506782690 357 /var/lib/homelab-health/netbird-vps-restore.txt
2026-06-17T22:53:44.6880508040 1257 /var/lib/homelab-health/app-stacks-batch-offhost.txt
2026-06-17T22:55:09.3892995820 4258 /var/lib/homelab-health/app-stacks-batch-restore.txt
2026-06-17T23:02:13.9525590360 2679 /var/lib/homelab-health/homelab-backup-baseline.txt
2026-06-18T19:16:23.9155288070 305 /var/lib/homelab-health/immich-media-restore.txt
2026-06-18T20:45:46.5765458970 242 /var/lib/homelab-health/mail-cloud-canary.txt
2026-06-24T15:19:34.8276009230 232 /var/lib/homelab-health/npmplus-kuma-config-backup.txt.bak-20260624T121934Z
2026-06-24T16:02:38.7930804000 145 /var/lib/homelab-health/vps-identity-peer-discovery.txt
2026-06-24T16:23:03.5382758840 2155 /var/lib/homelab-health/private-vpn-hosts.txt.bak-ok-wrapper-20260624T132705Z
2026-06-24T21:07:27.7414593140 118 /var/lib/homelab-health/pve01-failed-units.txt
2026-06-24T22:00:51.3972551230 196 /var/lib/homelab-health/edge-restore-proofs-offhost.txt
2026-06-24T22:03:31.7226417460 263 /var/lib/homelab-health/nextcloud-restore-proof-offhost.txt
2026-07-01T04:31:05.4336469050 312 /var/lib/homelab-health/netbird-vps-backup.txt
2026-07-02T11:54:39.6295399090 373 /var/lib/homelab-health/homelab-mailru-architecture-smoke.txt
2026-07-02T22:30:48.2601436500 644 /var/lib/homelab-health/crowdsec-capi-netbird-final.txt
2026-07-03T00:01:18.6769837810 398 /var/lib/homelab-health/homepage-final-status.txt
2026-07-03T00:21:03.3988362700 548 /var/lib/homelab-health/uptime-kuma-final-status.txt
2026-07-03T00:24:41.6771270360 304 /var/lib/homelab-health/uptime-kuma-npmplus-admin-fix.txt
2026-07-03T00:32:26.6061363170 249 /var/lib/homelab-health/uptime-kuma-approved-monitors-install.txt
2026-07-03T00:40:23.7603348270 535 /var/lib/homelab-health/uptime-kuma-pending-health-monitors-fix.txt
2026-07-03T00:45:30.9409011290 255 /var/lib/homelab-health/adguard-rewrite-sync.txt
2026-07-03T00:45:30.9411592050 672 /var/lib/homelab-health/restore-drill-index.txt
2026-07-03T00:45:30.9413791610 847 /var/lib/homelab-health/backup-restore-dashboard.txt
2026-07-03T00:48:36.3605124240 379 /var/lib/homelab-health/uptime-kuma-four-health-sources-repair.txt
2026-07-03T01:20:07.8445806940 325 /var/lib/homelab-health/uptime-kuma-four-heartbeat-final.txt
2026-07-03T01:47:00.7509517560 345 /var/lib/homelab-health/dockge-agent-install-register.txt
2026-07-03T02:09:14.1454067110 332 /var/lib/homelab-health/dockge-final-closure.txt
2026-07-03T02:20:50.3059631970 231 /var/lib/homelab-health/dockge-final-managed-labels.txt
2026-07-03T02:29:13.5615634610 17950 /var/lib/homelab-health/service-inventory-matrix-phase00.tsv
2026-07-03T13:44:54.6056260110 49 /var/lib/homelab-health/health-url.env
2026-07-07T01:09:37.3328266250 198 /var/lib/homelab-health/p1-safe-data-backup.txt
2026-07-07T01:22:38.5488080160 214 /var/lib/homelab-health/p1-remaining-tranche2.txt
2026-07-07T03:35:26.7019788500 196 /var/lib/homelab-health/p1-final-tranche.txt
2026-07-07T03:57:33.0001246560 269 /var/lib/homelab-health/p2-media-backup.txt
2026-07-07T05:40:35.0716067090 191 /var/lib/homelab-health/residual-review.txt
2026-07-07T07:11:06.5481479160 405 /var/lib/homelab-health/vm170-vm171-full-strategy.txt
2026-07-07T08:36:26.0853658590 286 /var/lib/homelab-health/node-loss-readiness.txt
2026-07-07T08:37:36.0224235940 200 /var/lib/homelab-health/node-loss-runbooks.txt
2026-07-07T09:18:31.3375599320 345 /var/lib/homelab-health/node-loss-closure.txt
2026-07-07T09:21:33.8143735330 371 /var/lib/homelab-health/power-loss-readiness.txt
2026-07-07T09:22:45.8234102790 282 /var/lib/homelab-health/power-loss-runbook.txt
2026-07-07T10:27:44.2204118760 347 /var/lib/homelab-health/power-loss-closure.txt
2026-07-07T10:40:01.2285714610 375 /var/lib/homelab-health/pve03-root-cleanup.txt
2026-07-07T10:45:01.8591246400 302 /var/lib/homelab-health/ungated-health-backlog.txt
2026-07-07T11:24:07.6285928400 409 /var/lib/homelab-health/pve03-staging-retention-proof.txt
2026-07-07T11:25:41.5290134290 437 /var/lib/homelab-health/pve03-staging-retention-cleanup.txt
2026-07-07T11:29:49.2086910530 369 /var/lib/homelab-health/pve03-staging-retention.txt
2026-07-07T15:21:23.8337500480 448 /var/lib/homelab-health/pve03-staging-retention-closure.txt
2026-07-07T15:25:57.4458619660 291 /var/lib/homelab-health/pve03-failed-unit-cleanup.txt
2026-07-07T15:52:21.5095968040 398 /var/lib/homelab-health/pve03-failed-unit-cleanup-closure.txt
2026-07-07T15:59:30.7733163830 343 /var/lib/homelab-health/remote-git-sops-age-readiness.txt
2026-07-07T16:01:07.1647092920 369 /var/lib/homelab-health/remote-git-sops-age-policy.txt
2026-07-07T16:42:29.4222211440 485 /var/lib/homelab-health/remote-git-sops-age-closure.txt
2026-07-07T16:51:17.2367953280 336 /var/lib/homelab-health/desired-state-remote-target-trace.txt
2026-07-07T16:53:30.3894596920 382 /var/lib/homelab-health/desired-state-remote-target.txt
2026-07-07T17:33:08.7816787090 565 /var/lib/homelab-health/desired-state-remote-target-closure.txt
2026-07-07T18:29:49.1668160110 408 /var/lib/homelab-health/netbird-vps-trust-closure.txt
2026-07-07T18:29:49.1695821150 403 /var/lib/homelab-health/external-probe-runner-decision.txt
2026-07-07T18:55:43.9106665130 433 /var/lib/homelab-health/netbird-vps-trust-clean-seal.txt
2026-07-07T23:23:46.4871597760 635 /var/lib/homelab-health/final-cycle-stability-seal.txt
2026-07-08T07:58:18.9509332240 299 /var/lib/homelab-health/mail-cloud-restore-drill.txt.bak-refresh-20260709T203132Z
2026-07-08T07:59:58.6953604310 426 /var/lib/homelab-health/sops-edge-secret-coverage-audit.txt
2026-07-08T08:01:49.3036518230 547 /var/lib/homelab-health/sops-edge-secret-coverage-closure.txt
2026-07-08T08:20:29.6041867030 337 /var/lib/homelab-health/remote-failed-unit-audit.txt
2026-07-08T08:23:09.5176054980 420 /var/lib/homelab-health/remote-failed-unit-plan.txt
2026-07-08T08:25:25.7376658940 508 /var/lib/homelab-health/remote-failed-unit-apply.txt
2026-07-08T08:27:02.9011355390 426 /var/lib/homelab-health/paperless-core-guard-audit.txt
2026-07-08T08:38:32.4205675490 531 /var/lib/homelab-health/remote-failed-unit-cleanup-seal.txt
2026-07-08T08:40:40.2615024670 497 /var/lib/homelab-health/duty-admin-residual-audit.txt
2026-07-08T08:42:30.3881692640 322 /var/lib/homelab-health/pve03-capacity-forensics.txt
2026-07-08T08:42:31.7994319160 591 /var/lib/homelab-health/duty-admin-residual-cleanup.txt
2026-07-08T08:58:45.2482918440 691 /var/lib/homelab-health/duty-admin-residual-cleanup-seal.txt
2026-07-08T09:08:11.4945075240 618 /var/lib/homelab-health/post-cleanup-owner-timer-replay.txt
2026-07-08T09:17:26.8701684910 819 /var/lib/homelab-health/post-cleanup-final-seal.txt
2026-07-08T09:20:13.6454507410 645 /var/lib/homelab-health/review-surface-registry-audit.txt
2026-07-08T09:26:36.7402046120 390 /var/lib/homelab-health/dockge-agent-capability-analysis.txt
2026-07-08T09:26:36.7402750460 390 /var/lib/homelab-health/service-inventory-leftovers-phase00.txt
2026-07-08T09:26:36.7403618890 390 /var/lib/homelab-health/dockge-stale-stack-classification-after-agents.txt
2026-07-08T09:26:36.7403838590 390 /var/lib/homelab-health/dockge-remediation-plan.txt
2026-07-08T09:26:36.7404284930 386 /var/lib/homelab-health/failed-unit-legacy-inventory.txt
2026-07-08T09:26:36.7404732790 390 /var/lib/homelab-health/service-setup-backlog-exact-order.txt
2026-07-08T09:26:36.7405361960 386 /var/lib/homelab-health/ungated-health-backlog-closure.txt
2026-07-08T09:26:36.7406026680 390 /var/lib/homelab-health/homepage-missing-card-proposals.txt
2026-07-08T09:26:36.7406281280 390 /var/lib/homelab-health/dockge-agent-runtime-method.txt
2026-07-08T09:26:36.7406657810 390 /var/lib/homelab-health/homepage-card-recommendations.txt
2026-07-08T09:26:36.7408259230 390 /var/lib/homelab-health/homepage-missing-card-proposals-final.txt
2026-07-08T09:26:36.7408607340 390 /var/lib/homelab-health/service-inventory-summary-phase00.txt
2026-07-08T09:26:36.7409298280 386 /var/lib/homelab-health/failed-unit-legacy-classification.txt
2026-07-08T09:26:36.7409559880 381 /var/lib/homelab-health/failed-unit-legacy-local-cleanup.txt
2026-07-08T09:26:36.7409816770 381 /var/lib/homelab-health/phase00-freeze-final.txt
2026-07-08T09:26:36.7410552890 389 /var/lib/homelab-health/dockge-stale-archive-last.txt
2026-07-08T09:26:36.7410994270 390 /var/lib/homelab-health/dockge-agent-schema-and-register-method.txt
2026-07-08T09:26:36.7411245420 390 /var/lib/homelab-health/dockge-disconnected-containers-analysis.txt
2026-07-08T09:26:36.7411439090 386 /var/lib/homelab-health/final-readiness-cycle-break.txt
2026-07-08T09:26:36.7411631530 390 /var/lib/homelab-health/dockge-user-hash-agent-install-readiness.txt
2026-07-08T09:26:36.7411808970 390 /var/lib/homelab-health/service-setup-order-analysis.txt
2026-07-08T09:26:36.7412497980 390 /var/lib/homelab-health/dockge-exact-agent-table-inspect.txt
2026-07-08T09:26:36.7413054120 381 /var/lib/homelab-health/failed-unit-legacy-cleanup-seal.txt
2026-07-08T09:26:36.7413803250 390 /var/lib/homelab-health/service-setup-roadmap-deep.txt
2026-07-08T09:26:36.7414375920 390 /var/lib/homelab-health/uptime-kuma-monitoring-proposals.txt
2026-07-08T09:26:36.7414750460 381 /var/lib/homelab-health/kuma-legacy-monitor-quarantine.txt
2026-07-08T09:26:36.7415177720 390 /var/lib/homelab-health/dockge-agent-users-login-fix.txt
2026-07-08T09:36:40.2533945430 464 /var/lib/homelab-health/review-surface-registry-apply.txt
2026-07-08T09:44:44.0027202780 639 /var/lib/homelab-health/review-surface-registry-seal.txt
2026-07-08T10:27:09.8992163430 918 /var/lib/homelab-health/final-global-green-audit.txt
2026-07-08T10:52:21.5920659640 1001 /var/lib/homelab-health/final-green-freeze-audit.txt
2026-07-08T11:20:32.9478120810 855 /var/lib/homelab-health/final-green-freeze-seal.txt
2026-07-08T11:31:18.9846514570 1108 /var/lib/homelab-health/recovery-card.txt
2026-07-08T11:41:46.1321730550 1204 /var/lib/homelab-health/recovery-card-seal.txt
2026-07-08T12:09:03.7459627090 1164 /var/lib/homelab-health/final-closeout.txt
2026-07-08T12:13:28.4829706890 762 /var/lib/homelab-health/independent-external-runner-audit.txt
2026-07-08T12:24:03.1785809940 1066 /var/lib/homelab-health/independent-external-runner.txt
2026-07-08T12:27:26.7886643440 643 /var/lib/homelab-health/desired-state-remote-git-audit.txt
2026-07-08T12:37:57.1142092610 1051 /var/lib/homelab-health/desired-state-remote-git.txt
2026-07-08T12:50:05.9132437970 1034 /var/lib/homelab-health/operator-handoff-pack.txt
2026-07-08T13:48:38.7184424100 260 /var/lib/homelab-health/live-tail-audit.txt
2026-07-08T15:04:03.6928977300 292 /var/lib/homelab-health/pve03-edge-vm-restore-drill.txt
2026-07-08T15:23:29.3375349030 277 /var/lib/homelab-health/pve03-edge-vm-offhost-freshness-seal.txt
2026-07-08T15:53:39.8149292590 369 /var/lib/homelab-health/pve02-forum-prod-restore-drill.txt
2026-07-08T17:10:02.1861577390 536 /var/lib/homelab-health/pve02-forum-prod-runtime-restore-drill.txt
2026-07-08T18:27:57.6728793210 469 /var/lib/homelab-health/pve01-nextcloud-dns-restore-drill.txt
2026-07-08T18:57:27.4506461580 287 /var/lib/homelab-health/pve01-core-monitoring-app-restorecheck.txt
2026-07-08T19:07:22.5726501490 344 /var/lib/homelab-health/pve01-core-monitoring-app-level-drill.txt
2026-07-08T19:35:56.6356041270 326 /var/lib/homelab-health/pve03-edge-vm-offhost-freshness.txt
2026-07-08T19:43:08.4121538910 433 /var/lib/homelab-health/node-loss-final-closure.txt
2026-07-08T20:49:00.3922878570 411 /var/lib/homelab-health/ideal-cluster-noise-seal.txt
2026-07-09T00:12:15.6152895130 340 /var/lib/homelab-health/ideal-cluster-final-seal.txt
2026-07-09T07:57:13.0808629150 380 /var/lib/homelab-health/cluster-admin-strict-final-seal.txt
2026-07-09T08:07:47.2134520460 217 /var/lib/homelab-health/cluster-admin-vm-mail-cloud-backup.txt
2026-07-09T08:24:54.0720153050 386 /var/lib/homelab-health/cluster-admin-backup-webpanel-final-seal.txt
2026-07-09T09:02:55.0056277600 147 /var/lib/homelab-health/cluster-admin-homepage-card.txt
2026-07-09T09:05:58.6994207660 249 /var/lib/homelab-health/cluster-admin-service-readiness.txt
2026-07-09T09:14:02.6367788630 298 /var/lib/homelab-health/service-readiness-cluster-admin-webpanel.txt
2026-07-09T09:22:58.3799240000 270 /var/lib/homelab-health/service-readiness-vaultwarden.txt
2026-07-09T09:29:21.9787558270 333 /var/lib/homelab-health/service-readiness-authentik.txt
2026-07-09T09:37:47.4434368570 290 /var/lib/homelab-health/service-readiness-gitea.txt
2026-07-09T09:41:40.0979703420 262 /var/lib/homelab-health/service-readiness-backup-health.txt
2026-07-09T09:47:55.5636728080 331 /var/lib/homelab-health/service-readiness-uptime-kuma.txt
2026-07-09T09:52:06.0994778650 261 /var/lib/homelab-health/service-readiness-gotify.txt
2026-07-09T10:22:35.5302603170 150 /var/lib/homelab-health/app-stacks-batch-backup.txt
2026-07-09T10:22:35.8882657530 229 /var/lib/homelab-health/router-startup-config-mail-cloud.txt
2026-07-09T10:40:55.9699739050 363 /var/lib/homelab-health/service-readiness-prometheus.txt
2026-07-09T10:46:00.9056083920 258 /var/lib/homelab-health/grafana-offhost.txt
2026-07-09T10:46:00.9056083920 261 /var/lib/homelab-health/grafana-restore.txt
2026-07-09T10:46:00.9056083920 294 /var/lib/homelab-health/grafana-backup.txt
2026-07-09T10:47:19.2097991550 326 /var/lib/homelab-health/service-readiness-grafana.txt
2026-07-09T10:50:17.8267126930 216 /var/lib/homelab-health/alertmanager-restore.txt
2026-07-09T10:50:17.8267126930 278 /var/lib/homelab-health/alertmanager-offhost.txt
2026-07-09T10:50:17.8267126930 282 /var/lib/homelab-health/alertmanager-backup.txt
2026-07-09T10:51:33.5136663000 407 /var/lib/homelab-health/service-readiness-alertmanager.txt
2026-07-09T11:00:42.7700187710 262 /var/lib/homelab-health/service-readiness-ntfy.txt
2026-07-09T11:04:19.9613236750 247 /var/lib/homelab-health/blackbox-exporter-restore.txt
2026-07-09T11:04:19.9613236750 296 /var/lib/homelab-health/blackbox-exporter-backup.txt
2026-07-09T11:04:19.9613236750 298 /var/lib/homelab-health/blackbox-exporter-offhost.txt
2026-07-09T11:05:57.9168142140 396 /var/lib/homelab-health/service-readiness-blackbox-exporter.txt
2026-07-09T11:14:05.4352326180 264 /var/lib/homelab-health/node-exporter-restore.txt
2026-07-09T11:14:05.4352326180 282 /var/lib/homelab-health/node-exporter-offhost.txt
2026-07-09T11:14:05.4352326180 331 /var/lib/homelab-health/node-exporter-backup.txt
2026-07-09T11:17:45.6165830230 412 /var/lib/homelab-health/service-readiness-node-exporter.txt
2026-07-09T11:25:57.2660677240 228 /var/lib/homelab-health/cadvisor-restore.txt
2026-07-09T11:25:57.2660677240 262 /var/lib/homelab-health/cadvisor-offhost.txt
2026-07-09T11:25:57.2660677240 275 /var/lib/homelab-health/cadvisor-backup.txt
2026-07-09T11:28:23.8062985970 394 /var/lib/homelab-health/service-readiness-cadvisor.txt
2026-07-09T11:32:11.0687583470 206 /var/lib/homelab-health/dozzle-restore.txt
2026-07-09T11:32:11.0687583470 254 /var/lib/homelab-health/dozzle-offhost.txt
2026-07-09T11:32:11.0687583470 270 /var/lib/homelab-health/dozzle-backup.txt
2026-07-09T11:33:54.7913373940 277 /var/lib/homelab-health/service-readiness-dozzle.txt
2026-07-09T11:38:36.1186130800 215 /var/lib/homelab-health/beszel-restore.txt
2026-07-09T11:38:36.1186130800 254 /var/lib/homelab-health/beszel-offhost.txt
2026-07-09T11:38:36.1186130800 269 /var/lib/homelab-health/beszel-backup.txt
2026-07-09T11:44:52.1683257450 267 /var/lib/homelab-health/service-readiness-beszel.txt
2026-07-09T11:51:31.1083861380 205 /var/lib/homelab-health/healthchecks-offhost.txt
2026-07-09T11:51:31.1083861380 211 /var/lib/homelab-health/healthchecks-restore.txt
2026-07-09T11:51:31.1083861380 265 /var/lib/homelab-health/healthchecks-backup.txt
2026-07-09T12:39:46.4043274480 278 /var/lib/homelab-health/searxng-backup.txt
2026-07-09T12:39:46.4053274630 218 /var/lib/homelab-health/searxng-restore.txt
2026-07-09T12:39:46.4053274630 258 /var/lib/homelab-health/searxng-offhost.txt
2026-07-09T12:58:00.5629333130 306 /var/lib/homelab-health/service-readiness-searxng.txt
2026-07-09T13:02:56.5714256630 205 /var/lib/homelab-health/npmplus-edge-offhost.txt
2026-07-09T13:02:56.5714256630 211 /var/lib/homelab-health/npmplus-edge-restore.txt
2026-07-09T13:02:56.5714256630 267 /var/lib/homelab-health/npmplus-edge-backup.txt
2026-07-09T13:07:32.0966068350 430 /var/lib/homelab-health/service-readiness-npmplus.txt
2026-07-09T13:12:46.6093796530 207 /var/lib/homelab-health/dockge-stacks-offhost.txt
2026-07-09T13:12:46.6093796530 213 /var/lib/homelab-health/dockge-stacks-restore.txt
2026-07-09T13:12:46.6093796530 218 /var/lib/homelab-health/dockge-offhost.txt
2026-07-09T13:12:46.6093796530 224 /var/lib/homelab-health/dockge-restore.txt
2026-07-09T13:12:46.6093796530 264 /var/lib/homelab-health/dockge-backup.txt
2026-07-09T13:12:46.6093796530 271 /var/lib/homelab-health/dockge-stacks-backup.txt
2026-07-09T13:16:48.1830455990 369 /var/lib/homelab-health/service-readiness-dockge.txt
2026-07-09T13:23:55.6935309890 193 /var/lib/homelab-health/netbox-offhost.txt
2026-07-09T13:23:55.6935309890 199 /var/lib/homelab-health/netbox-restore.txt
2026-07-09T13:23:55.6935309890 252 /var/lib/homelab-health/netbox-backup.txt
2026-07-09T13:40:18.9044441000 419 /var/lib/homelab-health/service-readiness-netbox.txt
2026-07-09T15:13:16.8761117060 191 /var/lib/homelab-health/minio-console-offhost.txt
2026-07-09T15:13:16.8761117060 191 /var/lib/homelab-health/minio-offhost.txt
2026-07-09T15:13:16.8761117060 197 /var/lib/homelab-health/minio-console-restore.txt
2026-07-09T15:13:16.8761117060 197 /var/lib/homelab-health/minio-restore.txt
2026-07-09T15:13:16.8761117060 245 /var/lib/homelab-health/minio-backup.txt
2026-07-09T15:13:16.8761117060 245 /var/lib/homelab-health/minio-console-backup.txt
2026-07-09T15:18:29.6618645950 283 /var/lib/homelab-health/service-readiness-minio.txt
2026-07-09T15:29:41.9851382980 203 /var/lib/homelab-health/proxmox-pve01-restore.txt
2026-07-09T15:29:41.9851382980 203 /var/lib/homelab-health/pve01-restore.txt
2026-07-09T15:29:41.9851382980 236 /var/lib/homelab-health/proxmox-pve01-offhost.txt
2026-07-09T15:29:41.9851382980 236 /var/lib/homelab-health/pve01-offhost.txt
2026-07-09T15:29:41.9851382980 276 /var/lib/homelab-health/proxmox-pve01-backup.txt
2026-07-09T15:29:41.9851382980 276 /var/lib/homelab-health/pve01-backup.txt
2026-07-09T15:33:52.8128905890 397 /var/lib/homelab-health/service-readiness-proxmox-pve01.txt
2026-07-09T15:42:17.7925620460 203 /var/lib/homelab-health/filebrowser-offhost.txt
2026-07-09T15:47:18.8311350910 276 /var/lib/homelab-health/service-readiness-filebrowser.txt
2026-07-09T15:53:45.2480051850 207 /var/lib/homelab-health/actual-budget-offhost.txt
2026-07-09T15:53:45.2480051850 207 /var/lib/homelab-health/actual-offhost.txt
2026-07-09T15:53:45.2480051850 213 /var/lib/homelab-health/actual-budget-restore.txt
2026-07-09T15:53:45.2480051850 213 /var/lib/homelab-health/actual-restore.txt
2026-07-09T15:53:45.2480051850 269 /var/lib/homelab-health/actual-backup.txt
2026-07-09T15:53:45.2480051850 269 /var/lib/homelab-health/actual-budget-backup.txt
2026-07-09T16:04:50.0111067090 281 /var/lib/homelab-health/service-readiness-actual-budget.txt
2026-07-09T19:01:20.7901933280 195 /var/lib/homelab-health/homebox-offhost.txt
2026-07-09T19:01:20.7901933280 201 /var/lib/homelab-health/homebox-restore.txt
2026-07-09T19:01:20.7901933280 251 /var/lib/homelab-health/homebox-backup.txt
2026-07-09T19:06:43.1941052730 268 /var/lib/homelab-health/service-readiness-homebox.txt
2026-07-09T19:17:35.0170448070 193 /var/lib/homelab-health/mealie-offhost.txt
2026-07-09T19:17:35.0170448070 199 /var/lib/homelab-health/mealie-restore.txt
2026-07-09T19:17:35.0170448070 250 /var/lib/homelab-health/mealie-backup.txt
2026-07-09T19:23:39.3486047240 283 /var/lib/homelab-health/service-readiness-mealie.txt
2026-07-09T19:46:33.3515780290 299 /var/lib/homelab-health/service-readiness-paperless.txt
2026-07-09T19:49:28.4202504550 212 /var/lib/homelab-health/memos-restore.txt
2026-07-09T19:50:22.5980774800 261 /var/lib/homelab-health/service-readiness-memos.txt
2026-07-09T20:00:19.8381943590 214 /var/lib/homelab-health/linkding-backup.txt
2026-07-09T20:00:19.8381943590 227 /var/lib/homelab-health/linkding-offhost.txt
2026-07-09T20:00:19.8381943590 227 /var/lib/homelab-health/linkding-restore.txt
2026-07-09T20:40:58.8843463730 270 /var/lib/homelab-health/service-readiness-linkding.txt
2026-07-09T20:47:41.7774675140 301 /var/lib/homelab-health/healthchecks-canonical-domain-repair.txt
2026-07-09T20:47:41.7774675140 340 /var/lib/homelab-health/service-readiness-healthchecks.txt
2026-07-09T20:51:20.8507958870 210 /var/lib/homelab-health/vikunja-backup.txt
2026-07-09T20:51:20.8507958870 222 /var/lib/homelab-health/vikunja-offhost.txt
2026-07-09T20:51:20.8507958870 222 /var/lib/homelab-health/vikunja-restore.txt
2026-07-09T20:55:48.6988688880 269 /var/lib/homelab-health/service-readiness-vikunja.txt
2026-07-09T20:58:33.5284046710 218 /var/lib/homelab-health/bookstack-backup.txt
2026-07-09T20:58:33.5284046710 232 /var/lib/homelab-health/bookstack-offhost.txt
2026-07-09T20:58:33.5284046710 232 /var/lib/homelab-health/bookstack-restore.txt
2026-07-09T21:01:03.7247153370 297 /var/lib/homelab-health/service-readiness-bookstack.txt
2026-07-09T21:08:55.5329737450 243 /var/lib/homelab-health/stirling-pdf-backup.txt
2026-07-09T21:08:55.5329737450 247 /var/lib/homelab-health/stirling-pdf-offhost.txt
2026-07-09T21:08:55.5329737450 247 /var/lib/homelab-health/stirling-pdf-restore.txt
2026-07-09T21:13:22.9720909840 269 /var/lib/homelab-health/service-readiness-stirling-pdf.txt
2026-07-09T21:16:17.6627935760 214 /var/lib/homelab-health/jellyfin-backup.txt
2026-07-09T21:16:17.6627935760 227 /var/lib/homelab-health/jellyfin-offhost.txt
2026-07-09T21:16:17.6627935760 227 /var/lib/homelab-health/jellyfin-restore.txt
2026-07-09T21:19:11.7814873230 270 /var/lib/homelab-health/service-readiness-jellyfin.txt
2026-07-09T21:22:25.6704869300 238 /var/lib/homelab-health/audiobookshelf-backup.txt
2026-07-09T21:22:25.6704869300 257 /var/lib/homelab-health/audiobookshelf-offhost.txt
2026-07-09T21:22:25.6704869300 257 /var/lib/homelab-health/audiobookshelf-restore.txt
2026-07-09T21:25:22.5602235460 279 /var/lib/homelab-health/service-readiness-audiobookshelf.txt
2026-07-09T21:46:20.9338292150 225 /var/lib/homelab-health/calibre-web-backup.txt
2026-07-09T21:46:20.9338292150 242 /var/lib/homelab-health/calibre-web-offhost.txt
2026-07-09T21:46:20.9338292150 242 /var/lib/homelab-health/calibre-web-restore.txt
2026-07-09T21:57:20.3030989240 271 /var/lib/homelab-health/service-readiness-calibre-web.txt
2026-07-09T22:04:04.0733864250 236 /var/lib/homelab-health/homeassistant-backup.txt
2026-07-09T22:04:04.0733864250 252 /var/lib/homelab-health/homeassistant-offhost.txt
2026-07-09T22:04:04.0733864250 252 /var/lib/homelab-health/homeassistant-restore.txt
2026-07-09T22:07:41.2227931800 281 /var/lib/homelab-health/service-readiness-homeassistant.txt
2026-07-09T22:13:14.6980274280 225 /var/lib/homelab-health/it-tools-backup.txt
2026-07-09T22:13:14.6980274280 227 /var/lib/homelab-health/it-tools-offhost.txt
2026-07-09T22:13:14.6980274280 227 /var/lib/homelab-health/it-tools-restore.txt
2026-07-09T22:23:57.3350983150 271 /var/lib/homelab-health/service-readiness-it-tools.txt
2026-07-09T22:28:35.5724369860 214 /var/lib/homelab-health/karakeep-backup.txt
2026-07-09T22:28:35.5724369860 227 /var/lib/homelab-health/karakeep-offhost.txt
2026-07-09T22:28:35.5724369860 227 /var/lib/homelab-health/karakeep-restore.txt
2026-07-09T22:32:36.7481977520 327 /var/lib/homelab-health/service-readiness-karakeep.txt
2026-07-09T22:39:24.2375519320 194 /var/lib/homelab-health/n8n-backup.txt
2026-07-09T22:39:24.2375519320 202 /var/lib/homelab-health/n8n-offhost.txt
2026-07-09T22:39:24.2375519320 202 /var/lib/homelab-health/n8n-restore.txt
2026-07-09T22:42:27.8124145040 261 /var/lib/homelab-health/service-readiness-n8n.txt
2026-07-09T22:46:45.7324363630 214 /var/lib/homelab-health/node-red-backup.txt
2026-07-09T22:46:45.7324363630 227 /var/lib/homelab-health/node-red-offhost.txt
2026-07-09T22:46:45.7324363630 227 /var/lib/homelab-health/node-red-restore.txt
2026-07-09T22:50:18.7317577520 270 /var/lib/homelab-health/service-readiness-node-red.txt
2026-07-09T22:55:13.5023542440 217 /var/lib/homelab-health/syncthing-backup.txt
2026-07-09T22:55:13.5023542440 232 /var/lib/homelab-health/syncthing-offhost.txt
2026-07-09T22:55:13.5023542440 232 /var/lib/homelab-health/syncthing-restore.txt
2026-07-09T22:59:10.6320324800 273 /var/lib/homelab-health/service-readiness-syncthing.txt
2026-07-09T23:09:25.0745502100 254 /var/lib/homelab-health/immich-restore.txt
2026-07-09T23:09:25.0745502100 278 /var/lib/homelab-health/immich-backup.txt
2026-07-09T23:09:25.0745502100 287 /var/lib/homelab-health/immich-offhost.txt
2026-07-09T23:12:55.3968081120 349 /var/lib/homelab-health/service-readiness-immich.txt
2026-07-09T23:39:37.9114415990 206 /var/lib/homelab-health/gotify-backup.txt
2026-07-09T23:39:37.9124416140 217 /var/lib/homelab-health/gotify-offhost.txt
2026-07-09T23:39:37.9124416140 217 /var/lib/homelab-health/gotify-restore.txt
2026-07-09T23:39:41.3274938750 198 /var/lib/homelab-health/ntfy-backup.txt
2026-07-09T23:39:41.3274938750 207 /var/lib/homelab-health/ntfy-offhost.txt
2026-07-09T23:39:41.3274938750 207 /var/lib/homelab-health/ntfy-restore.txt
2026-07-09T23:39:50.5946356910 229 /var/lib/homelab-health/uptime-kuma-backup.txt
2026-07-09T23:39:50.5946356910 242 /var/lib/homelab-health/uptime-kuma-offhost.txt
2026-07-09T23:39:50.5946356910 242 /var/lib/homelab-health/uptime-kuma-restore.txt
2026-07-09T23:45:51.3911568860 311 /var/lib/homelab-health/service-readiness-nextcloud.txt
2026-07-09T23:55:38.2600376160 277 /var/lib/homelab-health/service-readiness-dns1.txt
2026-07-09T23:55:38.2600376160 277 /var/lib/homelab-health/service-readiness-dns2.txt
2026-07-09T23:55:38.2600376160 284 /var/lib/homelab-health/service-readiness-homepage.txt
2026-07-09T23:58:29.2076174390 405 /var/lib/homelab-health/external-link-aleisaevn-netcraze-5083.txt
2026-07-10T00:12:45.3264837560 292 /var/lib/homelab-health/deep-service-backup-audit.txt
2026-07-10T21:46:36.9609141470 200 /var/lib/homelab-health/alertmanager-two-cloud-backup.txt
2026-07-10T22:05:25.8570406280 189 /var/lib/homelab-health/beszel-two-cloud-backup.txt
2026-07-10T22:20:26.1397323180 209 /var/lib/homelab-health/blackbox-exporter-two-cloud-backup.txt
2026-07-10T23:00:03.1168349530 151 /var/lib/homelab-health/legacy-monitoring-backups-migrated.txt
2026-07-10T23:25:23.2350187140 422 /var/lib/homelab-health/drift-baseline-acceptance.txt
2026-07-11T00:52:36.4697413270 151 /var/lib/homelab-health/cluster-admin-inventory-sync.txt
2026-07-11T02:25:25.1542644770 188 /var/lib/homelab-health/cluster-admin-public-route.txt
2026-07-11T02:26:01.7218294760 176 /var/lib/homelab-health/cluster-admin-prometheus-integration.txt
2026-07-11T02:30:49.6252637080 249 /var/lib/homelab-health/cluster-admin-incident-engine-backup.txt
2026-07-11T04:07:23.3393000650 361 /var/lib/homelab-health/service-readiness-cluster-admin-incident-engine.txt
2026-07-11T04:39:28.9976540970 212 /var/lib/homelab-health/cluster-admin-webpanel.txt
2026-07-11T06:23:40.9114023280 361 /var/lib/homelab-health/cluster-mail-cloud-cleanup.txt
2026-07-13T01:15:40.0109283350 386 /var/lib/homelab-health/cluster-admin-stage0-remediation.txt
2026-07-13T02:08:25.4011133770 436 /var/lib/homelab-health/cluster-admin-stage0-desired-state-sync.txt
2026-07-13T02:14:59.9458929520 400 /var/lib/homelab-health/cluster-admin-stage0-convergence.txt
2026-07-13T02:22:57.9862108430 523 /var/lib/homelab-health/cluster-admin-stage0-readiness-fix.txt
2026-07-13T02:26:02.8616913120 587 /var/lib/homelab-health/cluster-admin-stage0-baseline.txt
2026-07-13T02:32:35.0280552640 515 /var/lib/homelab-health/cluster-admin-stage1-capture.txt
2026-07-13T02:35:09.6424265770 727 /var/lib/homelab-health/cluster-admin-stage1-design-inputs.txt
2026-07-13T02:46:43.8820757430 698 /var/lib/homelab-health/cluster-admin-stage1-data-shape.txt
2026-07-13T03:11:06.6805116280 1065 /var/lib/homelab-health/cluster-admin-stage1-migration-dry-run.txt
2026-07-13T03:11:06.6805116280 406 /var/lib/homelab-health/cluster-admin-stage1-migration-compatibility.txt
2026-07-13T03:11:06.7005119340 926 /var/lib/homelab-health/cluster-admin-stage1-preapply-backup.txt
2026-07-13T03:22:22.4838707540 898 /var/lib/homelab-health/cluster-admin-stage1-foundation-apply.txt
2026-07-13T03:26:26.3016083230 956 /var/lib/homelab-health/cluster-admin-stage1-final-seal.txt
2026-07-13T03:29:40.0025776360 821 /var/lib/homelab-health/appbackupctl-routing.txt
2026-07-13T03:35:17.3427496850 1215 /var/lib/homelab-health/cluster-admin-stage2-design.txt
2026-07-13T03:39:46.1788887390 1361 /var/lib/homelab-health/cluster-admin-stage2-dry-run.txt
2026-07-13T04:05:34.7337002360 854 /var/lib/homelab-health/cluster-admin-stage2-recovery.txt
2026-07-13T04:15:16.8986427760 1035 /var/lib/homelab-health/cluster-admin-stage2-placeholder-diagnosis.txt
2026-07-13T04:20:17.9572677960 1318 /var/lib/homelab-health/cluster-admin-stage2-dynamic-redryrun.txt
2026-07-13T04:24:04.9957556580 1727 /var/lib/homelab-health/cluster-admin-stage2-preapply-backup.txt
2026-07-13T04:39:27.6419256780 1705 /var/lib/homelab-health/cluster-admin-stage2-apply.txt
2026-07-13T04:49:49.9254840660 1338 /var/lib/homelab-health/cluster-admin-stage2-final-seal.txt
2026-07-13T05:04:04.9066160570 1349 /var/lib/homelab-health/cluster-admin-stage3-api-design.txt
2026-07-13T05:36:13.0711962680 854 /var/lib/homelab-health/cluster-admin-stage3-api-auth-diagnosis.txt
2026-07-13T05:45:21.0465781570 1724 /var/lib/homelab-health/cluster-admin-stage3-api-dry-run.txt
2026-07-13T05:53:43.0632562670 1433 /var/lib/homelab-health/cluster-admin-stage3-api-runtime-dry-run.txt
2026-07-13T06:05:42.4882482970 728 /var/lib/homelab-health/cluster-admin-stage3-preapply-checkpoint.txt
2026-07-13T06:15:56.2396234100 1118 /var/lib/homelab-health/cluster-admin-stage3-preapply-backup.txt
2026-07-13T06:20:26.7327543700 1191 /var/lib/homelab-health/cluster-admin-stage3-api-candidate-normalized.txt
2026-07-13T06:22:40.5977987490 1106 /var/lib/homelab-health/cluster-admin-stage3-api-apply.txt
2026-07-13T06:25:40.2405422500 732 /var/lib/homelab-health/cluster-admin-stage3-production-verifier-prepared.txt
2026-07-13T06:28:56.1285342090 1207 /var/lib/homelab-health/cluster-admin-stage3-production-verify.txt
2026-07-13T06:44:42.7609957740 972 /var/lib/homelab-health/cluster-admin-stage3-postapply-backup.txt
2026-07-13T06:47:15.5353299170 1281 /var/lib/homelab-health/cluster-admin-stage3-final-seal.txt
2026-07-13T06:50:50.2356102460 507 /var/lib/homelab-health/crypto-portfolio-remove-grt-discovery.txt
2026-07-13T06:52:07.9807980800 488 /var/lib/homelab-health/crypto-portfolio-remove-grt-target.txt
2026-07-13T07:00:02.5170483350 822 /var/lib/homelab-health/crypto-portfolio-remove-grt-current.txt
2026-07-13T07:05:43.6132589850 879 /var/lib/homelab-health/crypto-portfolio-remove-grt-surface.txt
2026-07-13T07:20:46.9160557760 1741 /var/lib/homelab-health/crypto-portfolio-remove-grt-candidate.txt
2026-07-13T07:35:30.8895464510 1403 /var/lib/homelab-health/crypto-portfolio-remove-grt-runtime-verifier-prepared.txt
2026-07-13T07:41:31.9430520860 1497 /var/lib/homelab-health/crypto-portfolio-remove-grt-runtime-dry-run.txt
2026-07-13T07:52:08.1187504330 1724 /var/lib/homelab-health/crypto-portfolio-remove-grt-browser-smoke.txt
2026-07-13T08:05:15.1507543740 762 /var/lib/homelab-health/crypto-portfolio-remove-grt-desired-state-diagnostic.txt
2026-07-13T08:07:21.2856783820 1184 /var/lib/homelab-health/crypto-portfolio-remove-grt-desired-state-model.txt
2026-07-13T08:10:44.7277816120 1647 /var/lib/homelab-health/crypto-portfolio-remove-grt-prechange-backup.txt
2026-07-13T08:39:10.8547948980 330 /var/lib/homelab-health/crypto-portfolio-remove-grt-recovery-audit.txt
2026-07-13T08:41:01.3664795150 684 /var/lib/homelab-health/crypto-portfolio-remove-grt-failed-units-diagnostic.txt
2026-07-13T08:48:13.2600633440 1640 /var/lib/homelab-health/crypto-portfolio-remove-grt-production-apply.txt
2026-07-13T09:11:24.1542650490 1347 /var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-candidate.txt
2026-07-13T10:04:47.2300377080 1490 /var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-browser-smoke.txt
2026-07-13T10:42:57.1487691660 1851 /var/lib/homelab-health/crypto-portfolio-ui-preapply-backup.txt
2026-07-13T11:12:13.3023976030 192 /var/lib/homelab-health/crypto-portfolio-restore.txt
2026-07-13T11:12:13.3023976030 266 /var/lib/homelab-health/crypto-portfolio-offhost.txt
2026-07-13T11:12:13.3023976030 325 /var/lib/homelab-health/service-readiness-crypto-portfolio.txt
2026-07-13T11:12:13.3023976030 353 /var/lib/homelab-health/crypto-portfolio-backup.txt
2026-07-13T11:12:14.9294222730 757 /var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-production-closure.txt
2026-07-13T11:37:59.5159486840 1407 /var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-final-seal.txt
2026-07-13T11:37:59.5171037700 1407 /var/lib/homelab-health/crypto-portfolio-runtime-final-seal.txt
2026-07-13T14:03:06.0152234300 388 /var/lib/homelab-health/hapusya-site-preapply-backup.txt
2026-07-13T14:34:59.2083653510 533 /var/lib/homelab-health/hapusya-site-v2-preapply-backup.txt
2026-07-13T14:45:27.4649272800 793 /var/lib/homelab-health/hapusya-site-production-apply.txt
2026-07-13T14:48:15.6134864600 694 /var/lib/homelab-health/hapusya-site-production-browser.txt
2026-07-13T15:10:58.1372215280 156 /var/lib/homelab-health/hapusya-site-offhost.txt
2026-07-13T15:10:58.1372215280 185 /var/lib/homelab-health/hapusya-site-retention.txt
2026-07-13T15:10:58.1372215280 194 /var/lib/homelab-health/hapusya-site-restore.txt
2026-07-13T15:10:58.1372215280 499 /var/lib/homelab-health/hapusya-site-backup.txt
2026-07-13T15:10:58.1382215430 424 /var/lib/homelab-health/hapusya-site-backup-path-mapping.txt
2026-07-13T15:26:19.5872390600 651 /var/lib/homelab-health/hapusya-site-desired-state-sync.txt
2026-07-13T15:33:08.6444630100 1520 /var/lib/homelab-health/hapusya-site-final-seal.txt
2026-07-13T20:29:02.8967807310 628 /var/lib/homelab-health/backup-matrix-vm-health-feedback-fix.txt
2026-07-13T20:40:37.0653547720 648 /var/lib/homelab-health/backup-matrix-desired-state-persistence.txt
2026-07-13T20:48:14.6313253050 943 /var/lib/homelab-health/backup-matrix-cluster-preseal-convergence.txt
2026-07-13T20:51:32.4163384480 887 /var/lib/homelab-health/backup-matrix-cluster-desired-state-sync.txt
2026-07-13T20:51:32.4223385400 1445 /var/lib/homelab-health/backup-matrix-cluster-final-seal.txt
2026-07-13T21:13:01.1129724320 556 /var/lib/homelab-health/mail-cloud-edge-vm-runtime-contract-fix.txt
2026-07-13T21:15:43.9664537840 670 /var/lib/homelab-health/mail-cloud-edge-vm-readonly-probe.txt
2026-07-13T21:27:22.2230931620 630 /var/lib/homelab-health/mail-cloud-edge-vm-controlled-no-retention-mode.txt
2026-07-13T21:39:22.4540680390 719 /var/lib/homelab-health/mail-cloud-edge-vm-controlled-upload.txt
2026-07-13T21:40:21.6099694610 408 /var/lib/homelab-health/mail-cloud-edge-vm-service-convergence.txt
2026-07-13T21:56:30.9657631360 881 /var/lib/homelab-health/mail-cloud-edge-vm-matrix-registration.txt
2026-07-14T04:02:58.3995631560 1245 /var/lib/homelab-health/mail-cloud-edge-vm-final-seal.txt
2026-07-14T04:12:54.7747139510 707 /var/lib/homelab-health/remaining-special-families-matrix-registration.txt
2026-07-14T04:15:48.2773761880 1115 /var/lib/homelab-health/special-vm-emergency-non-app-final-seal.txt
2026-07-14T04:47:22.1524259850 902 /var/lib/homelab-health/cluster-admin-node-online-green-dot-fix.txt
2026-07-14T04:54:55.1333752320 1021 /var/lib/homelab-health/cluster-admin-collapsible-health-sections.txt
2026-07-14T08:33:26.7124814180 629 /var/lib/homelab-health/cluster-admin-stage4c-database-migration-dry-run.txt
2026-07-14T08:48:04.5210170900 482 /var/lib/homelab-health/homelab-admin-runner.txt
2026-07-14T09:14:58.6397106130 702 /var/lib/homelab-health/cluster-admin-stage4d-adapter-candidates-in-isolation.txt
2026-07-14T09:39:00.0749801060 610 /var/lib/homelab-health/cluster-admin-stage4e-restricted-probe-agent-design.txt
2026-07-14T10:39:53.2848761800 600 /var/lib/homelab-health/homelab-admin-guardrails.txt
2026-07-14T10:54:50.6815482590 692 /var/lib/homelab-health/cluster-admin-stage4f-preapply-backup.txt
2026-07-14T20:02:48.4191051180 266 /var/lib/homelab-health/cluster-admin-stage4g-production-apply-read-only.txt
2026-07-14T20:16:32.3610333130 213 /var/lib/homelab-health/operator-rule-close-tails-immediately.txt
2026-07-14T20:23:46.5434583710 328 /var/lib/homelab-health/cluster-admin-stage4h-scope-definition.txt
2026-07-14T20:29:41.0475984260 268 /var/lib/homelab-health/cluster-admin-stage4h-collector-baseline-audit.txt
2026-07-14T20:41:34.5353531810 455 /var/lib/homelab-health/cluster-admin-stage4h-collector-patch-candidate.txt
2026-07-14T21:26:54.1724254520 226 /var/lib/homelab-health/operator-command-size-guard.txt
2026-07-14T21:29:30.0818406610 312 /var/lib/homelab-health/cluster-admin-stage4h-final-acceptance-contract.txt
2026-07-14T21:33:40.0697208200 209 /var/lib/homelab-health/homelab-admin-cli-contract.txt
2026-07-14T21:53:16.5230291610 357 /var/lib/homelab-health/cluster-admin-stage4h-task-package-build.txt
2026-07-15T10:10:19.9159742040 1732 /var/lib/homelab-health/cluster-admin-stage4h-final-acceptance.txt
2026-07-15T10:17:53.0215082300 921 /var/lib/homelab-health/cluster-admin-stage4i-scope-definition.txt
2026-07-16T01:32:11.0876121490 237 /var/lib/homelab-health/homelab-ops-bootstrap.txt
2026-07-16T09:20:34.5941094330 75 /var/lib/homelab-health/router-hard-power.txt
2026-07-16T12:14:29.4893285610 202 /var/lib/homelab-health/xenforo-branding-rollout.txt
2026-07-16T12:41:28.8422409190 192 /var/lib/homelab-health/xenforo-branding-header-visibility.txt
2026-07-16T12:48:31.4566023130 185 /var/lib/homelab-health/xenforo-branding-header-order.txt
2026-07-16T13:01:16.2035467580 250 /var/lib/homelab-health/homelab-ops-worker-vm190.txt
2026-07-17T04:59:42.6476420400 242 /var/lib/homelab-health/netbird-vps-offhost.txt
2026-07-17T09:31:15.5873108310 443 /var/lib/homelab-health/paperless-offhost.txt
2026-07-17T14:21:55.3567278930 272 /var/lib/homelab-health/cluster-storage-retention.txt
2026-07-18T01:52:29.8931662090 684 /var/lib/homelab-health/paperless-backup.txt
2026-07-18T01:54:49.4850009180 728 /var/lib/homelab-health/netbird-vps-trust.txt
2026-07-18T02:54:28.2167554790 200 /var/lib/homelab-health/full-handoff-recovery.txt
2026-07-19T09:41:56.0870397860 437 /var/lib/homelab-health/mail-cloud-edge-vm.txt
2026-07-21T02:37:04.7969419110 195 /var/lib/homelab-health/cluster-admin-observer.txt
2026-07-21T02:37:04.7969419110 200 /var/lib/homelab-health/cluster-admin-restricted-probes.txt
2026-07-21T02:37:04.7969419110 219 /var/lib/homelab-health/cluster-admin-full-observer.txt
2026-07-21T02:39:14.3120870600 153 /var/lib/homelab-health/vm160-remote-backup.txt
2026-07-21T04:55:35.3129057980 316 /var/lib/homelab-health/edge-vm-vzdump-offhost.txt
2026-07-21T04:55:35.3129057980 334 /var/lib/homelab-health/edge-vm-vzdump-restore.txt
2026-07-21T04:55:35.3129057980 409 /var/lib/homelab-health/edge-vm-offhost-freshness.txt
2026-07-21T04:55:35.3408368720 314 /var/lib/homelab-health/sops-edge-secret-coverage.txt
2026-07-21T05:18:05.9056051500 633 /var/lib/homelab-health/router-running-config-mail-cloud.txt
2026-07-21T06:42:44.4264139010 339 /var/lib/homelab-health/mail-cloud-critical.txt
2026-07-21T07:05:43.0783306490 1583 /var/lib/homelab-health/backup-last-run.log
2026-07-21T07:05:43.0870410520 217 /var/lib/homelab-health/backup.txt
2026-07-21T07:05:43.0879592500 274 /var/lib/homelab-health/backup.json
2026-07-21T07:15:46.8155363020 232 /var/lib/homelab-health/drift-check.txt
2026-07-21T07:36:59.5343449800 170 /var/lib/homelab-health/vm-backup.txt
2026-07-21T07:49:15.8201883610 136 /var/lib/homelab-health/desired-state.txt
2026-07-21T07:56:14.9677282530 146 /var/lib/homelab-health/runbooks.txt
2026-07-21T07:56:26.4127609750 475 /var/lib/homelab-health/mail-cloud-nextcloud-vm.txt
2026-07-21T08:01:15.0701844650 1522 /var/lib/homelab-health/overall.txt
2026-07-21T08:10:47.5199010040 117 /var/lib/homelab-health/duty-admin.txt
2026-07-21T08:29:16.9008054160 178 /var/lib/homelab-health/capacity-risk.txt
2026-07-21T08:31:47.8421084810 128 /var/lib/homelab-health/final-readiness.txt
2026-07-21T08:35:41.8646792150 147 /var/lib/homelab-health/secret-exposure.txt
2026-07-21T08:51:15.1419524730 115 /var/lib/homelab-health/cluster-passport.txt
2026-07-21T08:51:15.4733058530 323 /var/lib/homelab-health/immich-media-offhost.txt
2026-07-21T08:54:09.3706207380 509 /var/lib/homelab-health/golden-state.txt
2026-07-21T09:15:30.8626761250 193 /var/lib/homelab-health/memos-offhost.txt
2026-07-21T10:02:41.8774944390 698 /var/lib/homelab-health/filebrowser-backup.txt
2026-07-21T10:02:41.8983445980 796 /var/lib/homelab-health/filebrowser-restore.txt
2026-07-21T10:02:41.9637054430 811 /var/lib/homelab-health/npmplus-kuma-config-backup.txt
2026-07-21T10:02:41.9638165500 812 /var/lib/homelab-health/npmplus-kuma-config-offhost.txt
2026-07-21T10:02:41.9638965790 812 /var/lib/homelab-health/npmplus-kuma-config-restore.txt
2026-07-21T10:02:41.9880922600 754 /var/lib/homelab-health/authentik-backup.txt
2026-07-21T10:02:41.9883026710 755 /var/lib/homelab-health/authentik-offhost.txt
2026-07-21T10:02:41.9885226120 755 /var/lib/homelab-health/authentik-restore.txt
2026-07-21T10:02:42.0123817100 738 /var/lib/homelab-health/gitea-backup.txt
2026-07-21T10:02:42.0126132350 739 /var/lib/homelab-health/gitea-offhost.txt
2026-07-21T10:02:42.0128119330 739 /var/lib/homelab-health/gitea-restore.txt
2026-07-21T10:02:42.0130029950 744 /var/lib/homelab-health/vaultwarden-backup.txt
2026-07-21T10:02:42.0132144910 745 /var/lib/homelab-health/vaultwarden-offhost.txt
2026-07-21T10:02:42.0134265770 745 /var/lib/homelab-health/vaultwarden-restore.txt
2026-07-21T10:02:42.0850718970 808 /var/lib/homelab-health/backup-coverage-matrix.txt
2026-07-21T10:02:42.1723598770 769 /var/lib/homelab-health/edge-vm-vzdump-backup.txt
2026-07-21T10:02:42.2144238040 777 /var/lib/homelab-health/extended-appbackup.txt
2026-07-21T10:02:42.3608260970 829 /var/lib/homelab-health/mail-cloud-restore-drill.txt
2026-07-21T10:02:42.4582078560 803 /var/lib/homelab-health/backup-framework.txt
2026-07-21T10:02:42.5028324600 760 /var/lib/homelab-health/vm-backup-policy.txt
2026-07-21T10:02:42.5224863440 763 /var/lib/homelab-health/vm170-vm171-full-backup.txt
2026-07-21T10:02:42.5469238210 822 /var/lib/homelab-health/vm170-vm171-full-restore-drill.txt
2026-07-21T10:02:42.5676022180 800 /var/lib/homelab-health/vm-local-dumps-cloud.txt
2026-07-21T10:02:42.5851669640 826 /var/lib/homelab-health/vm-local-dumps-retention.txt
2026-07-21T10:02:42.6060015540 717 /var/lib/homelab-health/immich-media-backup.txt
2026-07-21T10:02:42.6265610130 656 /var/lib/homelab-health/memos-backup.txt
2026-07-21T10:02:42.6979119670 774 /var/lib/homelab-health/paperless-restore.txt
2026-07-21T11:42:40.5821734030 236 /var/lib/homelab-health/vps-identity-audit.txt
2026-07-21T16:05:25.1099029070 170 /var/lib/homelab-health/mkdocs-auto-refresh.txt
2026-07-21T16:15:14.9576243870 142 /var/lib/homelab-health/incident-journal.txt
2026-07-21T16:15:16.1573180790 295 /var/lib/homelab-health/alerting.txt
2026-07-21T16:15:16.6026862220 327 /var/lib/homelab-health/kuma-monitor-policy.txt
2026-07-21T16:15:17.1666948120 154 /var/lib/homelab-health/backup-sla.txt
2026-07-21T16:15:17.1835326620 141 /var/lib/homelab-health/service-registry.txt
2026-07-21T16:15:17.2000146280 128 /var/lib/homelab-health/dependency-map.txt
2026-07-21T16:15:17.2016953450 142 /var/lib/homelab-health/safe-autoheal.txt
2026-07-21T16:16:27.0167586760 233 /var/lib/homelab-health/forum-snuffleupagus.txt
2026-07-21T16:16:27.1277603660 539 /var/lib/homelab-health/external-probe-vps.txt
2026-07-21T16:16:30.7472906570 489 /var/lib/homelab-health/disk-space.txt
2026-07-21T16:16:33.8154650810 317 /var/lib/homelab-health/duty-admin-v2.txt
2026-07-21T16:18:29.0889120900 139 /var/lib/homelab-health/crypto-portfolio-chart-health.txt
2026-07-21T16:19:17.1103481560 477 /var/lib/homelab-health/netbird-peers.txt
2026-07-21T16:19:30.2695482230 0 /var/lib/homelab-health/pve03-staging-capacity.txt.err
2026-07-21T16:19:30.6108705160 207 /var/lib/homelab-health/pve03-staging-capacity.txt
2026-07-21T16:20:06.2880407250 215 /var/lib/homelab-health/cluster-admin-webpanel-sync.txt
2026-07-21T16:20:07.3121113750 227 /var/lib/homelab-health/router-watchdog.txt
2026-07-21T16:20:08.5291874140 641 /var/lib/homelab-health/private-vpn-hosts.txt
2026-07-21T16:21:10.8330770850 0 /var/lib/homelab-health/private-vpn-hosts.txt.tmp
/var/lib/homelab-health/final-closeout.txt:1:STATUS=OK TS=2026-07-08T09:02:10Z TYPE=final-closeout BAD=0 WARN=0 MODE=FINAL_OPERATOR_CLOSEOUT CLOSEOUT_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 EXACT_WARN_REVIEW_FILES=0 ENDPOINT_LOCAL_OK=1 ENDPOINT_LAN_OK=1 ENDPOINT_PUBLIC_OK=1 RECOVERY_SEAL=YES FREEZE_SEAL=YES FINAL_GLOBAL_GREEN=YES REVIEW_SURFACE_REGISTRY=YES POST_CLEANUP=YES REMOTE_CLEANUP=YES BACKUP_SLA=OK BACKUP_COVERAGE=OK EXTENDED_APPBACKUP=OK VM_BACKUP=OK SECRET_EXPOSURE=OK EXTERNAL_PROBE=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T084146Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T084146Z EMERGENCY_JOB=homelab-pve01-emergency-pack-20260708T002739Z VM_DUMPS_JOB=homelab-pve01-vm-local-dumps-20260706T153606Z DIRTY_COUNT=0 REPORT=/var/lib/homelab-final-closeout/final-closeout-latest.md
/var/lib/homelab-health/backup-matrix-cluster-final-seal.txt:1:STATUS=OK TS=2026-07-13T17:51:31Z TYPE=backup-matrix-cluster-final-seal BAD=0 CURRENT_STAGE_COMPLETE=YES MATRIX_STATUS=OK JOBS_TOTAL=35 JOBS_OK=31 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 READ_WRITE_OVERLAP=0 VM_BACKUP_HEALTH=OK VM_FULL_BACKUP=OK EDGE_VM_CANONICAL_BACKUP=homelab-edge-vm-offhost-freshness.timer OLD_EDGE_TIMER_REENABLED=NO CLUSTER_PASSPORT=OK FINAL_READINESS=OK FINAL_READINESS_PASS=53 FINAL_READINESS_FAIL=0 OVERALL=OK CLUSTER_ADMIN=OK PROBE_OK=4 PROBE_BAD=0 WEBPANEL=OK FAILED_UNITS=0 CLUSTER_NODES=3 QUORATE=YES GENERATED_SYNC_HEAD=597119391e7b265b85fd9a76644e4e33af47e469 DESIRED_STATE_DIRTY=0 MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 MATRIX_STATE_SHA256=14a31112aac5d577ae01df567921b093b2fedcf7189c4cc8f009216ee0749380 CLUSTER_PASSPORT_SHA256=1b8b924ae0e5ce6872fe5671318094ce494806c2db7efe28299b5ff49adc0d0d FINAL_READINESS_SHA256=ac5159fa8c497afe021b5cb232f08cfec0e1a4d516d8aae966ba03f7034f1412 OVERALL_SHA256=6ae529bfdf5811f686d3b53978f453ce4f8757c74fc1be105de219b13a61ba46 PRESEAL_SHA256=7ea1f710cd311bd51dba05a313b4c540f57d97407f008881e502fc213b4b3c99 EVIDENCE_MANIFEST_SHA256=b4c68312026e1328b208a74090891f4809ab31ec69069e94a3d6902429bf3566 BACKUP_RUN=NO RESTORE_RUN=NO CLOUD_WRITE_RUN=NO RETENTION_RUN=NO NEXT_ACTION=AUDIT_AND_MIGRATE_SPECIAL_VM_EMERGENCY_AND_NON_APP_BACKUP_FAMILIES EVIDENCE_ROOT=/var/lib/homelab-change-sets/backup-matrix-cluster-final-seal/20260713T175131Z
/var/lib/homelab-health/final-cycle-stability-seal.txt:1:STATUS=OK TS=2026-07-07T20:08:34Z TYPE=final-cycle-stability-seal BAD=0 WARN=0 MODE=FORCED_REPLAY_AFTER_CYCLE_BREAK STABLE=YES EXTERNAL_PROBE_STATUS=OK EXTERNAL_PROBE_WARN=0 EXTERNAL_PROBE_SMOKE_BAD=0 FINAL_STATUS=OK FINAL_BAD=0 FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 OVERALL_STATUS=OK OVERALL_BAD=0 OLD_EXTERNAL_PROBE_LIVE_MARKERS=0 OLD_EXTERNAL_PROBE_SOURCE_MARKERS=0 FINAL_CLUSTER_CHECKS=0 FINAL_LEGACY_DUTY_CHECKS=0 OVERALL_LEGACY_DUTY_GATE=0 CLUSTER_LEGACY_DUTY_GATE=0 SOURCE_CLUSTER_LEGACY_DUTY_GATE=0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-final-cycle-break/stability-seal-latest.md
/var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-production-closure.txt:1:STATUS=OK TS=2026-07-13T08:12:14Z TYPE=crypto-portfolio-ui-center-alert-currency-production-closure BAD=0 PRODUCTION_CHANGED=YES_UI_ONLY INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb SHARP_MOVES_CENTERED=YES ALERT_CURRENCY_USD_RUB=OK BROWSER_RUNS=2 BROWSER_ERRORS=0 GRT_ROWS=0|0|0 GRT_SOURCE_REFERENCES=0 POSTCHANGE_BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T075801Z RESTORE_CHECK=OK_BOTH_CLOUDS RETENTION_ROLLBACK_COUNT=3 RETENTION_OPTIONAL_OTHER_CANDIDATES=1 RETENTION_REMOVED_ROLLBACK_COUNT=0 DANGLING_IMAGES_REMAINING=2 UNREFERENCED_DANGLING_IMAGES=0 DESIRED_STATE_HEAD=2999910549eaff9b9431b9c656d318340eb95cbd DESIRED_STATE_CLEAN=YES PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES READY_FOR_UI_FINAL_SEAL=YES
/var/lib/homelab-health/desired-state-remote-git-audit.txt:1:STATUS=OK TS=2026-07-08T09:27:24Z TYPE=desired-state-remote-git-audit BAD=0 WARN=0 MODE=READ_ONLY_DESIRED_STATE_REMOTE_GIT_AUDIT AUDIT_OK=YES REMOTE_READY=NO ADD_REMOTE_DECISION=NO_CHANGE REPO_PRESENT=1 DIRTY_COUNT=0 REMOTE_COUNT=0 URL_CREDENTIAL_HINT=0 HTTPS_GITEA_OK=1 HTTPS_API_OK=0 BACKEND_OK=1 BACKEND_API_OK=0 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 OVERALL_STATUS=OK OVERALL_BAD=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 CLOSEOUT_OK=YES RECOVERY_SEAL_OK=YES INDEPENDENT_RUNNER_NO_DEPLOY=YES REPORT=/var/lib/homelab-desired-state-remote-git/audit-latest.md
/var/lib/homelab-health/cluster-admin-stage3-postapply-backup.txt:1:STATUS=OK TS=2026-07-13T03:43:59Z TYPE=cluster-admin-stage3-postapply-backup BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=POSTAPPLY_BACKUP_AND_CONTROLLED_COLLECTOR_RUN PRODUCTION_DATABASE_CHANGED=COLLECTOR_NORMAL_OPERATION_ONLY PRODUCTION_SCHEMA_CHANGED=NO LIVE_APPLICATION_CODE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=YES_CONTROLLED_POSTAPPLY_BACKUP LIVE_APP_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 APPBACKUP=VERIFIED APPBACKUP_JOB=homelab-pve01-app-cluster-admin-incident-engine-20260713T034400Z APPBACKUP_SIZE_BYTES=1791603 APPBACKUP_RESTORE_CHECK=OK APPBACKUP_ROUTING=VERIFIED COLLECTOR_CONTROL_RUN=OK COLLECTOR_RUN_ID=3113 CANONICAL_CONSISTENCY=0|0|0|0|0 INCIDENT_ENGINE=ACTIVE COLLECTOR_TIMER=ACTIVE VM180_FAILED_UNITS=0 READY_FOR_STAGE3_FINAL_SEAL=YES STAGE3_CLOSED=NO POSTAPPLY_ROOT=/var/lib/homelab-stage3-postapply/20260713T034359Z
/var/lib/homelab-health/cluster-admin-webpanel.txt:1:STATUS=OK TS=2026-07-11T01:39:28Z TYPE=cluster-admin-webpanel BAD=0 URL=http://[PRIVATE_IP]:8080/ STATUS_URL=http://[PRIVATE_IP]:8080/status.txt SELF_FAILED_UNITS=0 QGA=active BACKUP=MAIL_CLOUD_2COPY_VERIFY_OK
/var/lib/homelab-health/post-cleanup-final-seal.txt:1:STATUS=OK TS=2026-07-08T06:10:30Z TYPE=post-cleanup-final-seal BAD=0 WARN=0 MODE=POST_CLEANUP_OWNER_TIMER_FINAL_SEAL FINAL_SEAL_OK=YES REPLAY_STATUS=OK REPLAY_OK=YES LOCAL_START_BAD=0 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 REVIEW_ITEMS_INTENTIONALLY_LEFT=yes LEGACY_PHASE_ITEMS_INTENTIONALLY_LEFT=yes FINAL_STATUS=OK FINAL_BAD=0 FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 LOCAL_CLEANUP_OK=YES REMOTE_CLEANUP_OK=YES RESIDUAL_SEAL_OK=YES SOPS_STATUS=OK EXTERNAL_PROBE_STATUS=OK EXTERNAL_PROBE_WARN=0 EXTERNAL_PROBE_SMOKE=0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-post-cleanup-replay/final-seal-latest.md
/var/lib/homelab-health/pve01-core-monitoring-app-restorecheck.txt:1:STATUS=OK TS=2026-07-08T15:57:27Z TYPE=pve01-core-monitoring-app-restorecheck BAD=0 MODE=APP_LEVEL_RESTORE_CHECKS_TWO_CLOUDS FULL_VM_RESTORE_AVAILABLE=NO TOTAL_APPS=30 OK_APPS=30 FAIL_LIST=none REPORT=/var/lib/homelab-node-loss/pve01-core-monitoring-app-level/app-restorecheck-latest.md
/var/lib/homelab-health/restore-drill-index.txt:5:MISSING_OR_NOT_OK=none
/var/lib/homelab-health/mail-cloud-edge-vm-final-seal.txt:1:STATUS=OK TS=2026-07-14T01:02:58Z TYPE=mail-cloud-edge-vm-final-seal BAD=0 WARN=0 VMID=130 SOURCE=/mnt/staging/offhost/edge-vm-vzdump-from-pve03/vzdump-qemu-130-2026_07_13-03_30_01.vma.zst SIZE=10942350973 SHA256=6979558850813d99881b20c559600ec1c8748cc4f59f18ef9875eee6e40c6757 CLOUD_FOUND_DIR=20260713T183122Z MATRIX_JOB_ID=homelab_mail_cloud_edge_vm_upload MATRIX_STATUS=OK JOBS_TOTAL=36 JOBS_OK=32 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 UPLOAD_VERIFIED=YES DOWNLOAD_VERIFY=YES SHA_MATCH=YES SIZE_MATCH=YES PART_COUNT_MATCH=YES CONTROLLED_UPLOAD_MODE=--upload-no-retention RETENTION_MODE=disabled RETENTION_RUN=NO RETENTION_DELETE_COUNT=0 CLOUD_PURGE_RUN=NO SOURCE_DELETE_RUN=NO EDGE_SERVICE_RESULT=success EDGE_EXEC_MAIN_STATUS=0 EDGE_TIMER_ENABLED=enabled EDGE_TIMER_ACTIVE=active CLUSTER_ADMIN_STATUS=OK PROBE_OK=4 PROBE_BAD=0 CLUSTER_PASSPORT=OK DUTY_ADMIN=OK OVERALL=OK OVERALL_BAD=0 FINAL_READINESS=OK FINAL_PASS=53 FINAL_FAIL=0 WEBPANEL=OK WEBPANEL_SYNC=OK PUBLIC_OVERALL=OK FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DESIRED_STATE_PRE_SEAL_HEAD=98dfaeb35683e18cd4e986d0121bde60daac1e7c CURRENT_EDGE_VM_CLOUD_SUBSTAGE_COMPLETE=YES EVIDENCE_ROOT=/var/lib/homelab-change-sets/mail-cloud-edge-vm-final-seal-idle-safe/20260714T010226Z
/var/lib/homelab-health/cluster-admin-stage2-dynamic-redryrun.txt:1:STATUS=OK TS=2026-07-13T01:20:06Z TYPE=cluster-admin-stage2-dynamic-redryrun BAD=0 PRODUCTION_CHANGED=TEMPORARY_DATABASE_AND_EVIDENCE_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO INCIDENT_ENGINE_RESTARTED=NO MIGRATION_ID=002_canonical_foundation_sync MIGRATION_SHA256=92a7f87ed6bf5215c81ec5f70ab1dc3c8339ecffa0b0f9fcab4eb6322651dc68 MIGRATION_PATCH_REQUIRED=NO COLLECTOR_PATCH_REQUIRED=NO APPLY_VERIFIER_PATCH_REQUIRED=YES FIXED_TOTAL_ENTITIES_REQUIRED=NO FIXED_TOTAL_ALIASES_REQUIRED=NO ACTIVE_CURRENT_SET_CONVERGENCE=OK NATURAL_KEY_CONVERGENCE=OK EPHEMERAL_ID_REFRESH=OK DYNAMIC_PLACEHOLDER_LIFECYCLE=OK RETIRED_PLACEHOLDERS_ALLOWED=YES ORPHANS=0 PREFLIGHT_TRANSACTION_ROLLBACK=OK DDL_ROLLBACK_TEST=OK ROLLBACK_AFTER_COMMITTED_SYNC=DO_NOT_PARTIAL_DDL_ROLLBACK PRESERVE_APPLIED_STATE_ON_POSTSYNC_VERIFIER_FAILURE=YES TEST_DATABASE_DROPPED=YES PRODUCTION_UNCHANGED=YES FRESH_PREAPPLY_BACKUP_REQUIRED=YES READY_FOR_FRESH_PREAPPLY_BACKUP=YES DESIRED_STATE_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 CONTRACT=/var/lib/homelab-change-sets/stage2-dynamic-redryrun/20260713T012006Z/dynamic-acceptance-contract.json RESULT=/var/lib/homelab-change-sets/stage2-dynamic-redryrun/20260713T012006Z/dynamic-redryrun-result.txt CHANGE_SET=/var/lib/homelab-change-sets/stage2-dynamic-redryrun/20260713T012006Z
/var/lib/homelab-health/desired-state-remote-git.txt:1:STATUS=OK TS=2026-07-08T09:31:25Z TYPE=desired-state-remote-git BAD=0 WARN=0 MODE=NO_REMOTE_ADD_NO_PUSH_SEAL NO_REMOTE_SEAL_OK=YES AUDIT_STATUS=OK AUDIT_OK=YES REMOTE_READY=NO ADD_REMOTE_DECISION=NO_CHANGE GIT_REMOTE_CHANGED=NO GIT_PUSH_DONE=NO REPO_PRESENT=1 REPO_TOP=/srv/homelab-desired-state BRANCH=master DIRTY_COUNT=0 REMOTE_COUNT=0 URL_CREDENTIAL_HINT=0 HTTPS_GITEA_OK=1 HTTPS_API_OK=0 BACKEND_OK=1 BACKEND_API_OK=0 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 OVERALL_STATUS=OK OVERALL_BAD=0 FINAL_STATUS=OK FINAL_FAIL=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 CLOSEOUT_OK=YES RECOVERY_SEAL_OK=YES INDEPENDENT_RUNNER_NO_DEPLOY=YES RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T092404Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T092404Z REPORT=/var/lib/homelab-desired-state-remote-git/no-change-seal-latest.md
/var/lib/homelab-health/service-registry.txt:1:STATUS=OK TS=2026-07-21T13:15:17Z TYPE=service-registry SERVICES=12 MISSING=0 REGISTRY=/var/lib/homelab-registry/service-registry-latest.tsv
/var/lib/homelab-health/crypto-portfolio-remove-grt-discovery.txt:1:STATUS=OK TS=2026-07-13T03:50:49Z TYPE=crypto-portfolio-remove-grt-discovery BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=NO DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO PREVIOUS_FAILURE_ROOT_CAUSE=UNVERIFIED_STACK_PATH_ASSUMPTION CANDIDATE_DIRECTORIES=4 CANDIDATE_DATABASES=0 CANDIDATE_CONTAINERS=0 CANDIDATE_REFERENCE_FILES=8 READY_FOR_GRT_TARGETED_CAPTURE=YES DISCOVERY_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-discovery/20260713T035049Z
/var/lib/homelab-health/post-cleanup-owner-timer-replay.txt:1:STATUS=OK TS=2026-07-08T06:01:13Z TYPE=post-cleanup-owner-timer-replay BAD=0 WARN=0 MODE=FORCED_REPLAY_CLEANUP_OWNERS REPLAY_OK=YES LOCAL_START_OK=10 LOCAL_START_BAD=0 EDGE_RC=0 EDGE_HEALTH=OK LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 EXTERNAL_PROBE_STATUS=OK EXTERNAL_PROBE_WARN=0 EXTERNAL_PROBE_SMOKE=0 SOPS_STATUS=OK DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-post-cleanup-replay/owner-timer-replay-latest.md
/var/lib/homelab-health/paperless-core-guard-audit.txt:1:STATUS=OK TS=2026-07-08T05:26:59Z TYPE=paperless-core-guard-audit BAD=0 WARN=1 MODE=READ_ONLY_PAPERLESS_CORE_GUARD_AUDIT CLASS=REMOTE_DEFERRED_GUARD_NOT_HARD_GATE CONNECT_ERRORS=0 FAILED_ONLY_GUARD=YES IN_HARD_GATES=no OVERALL_BAD=0 FINAL_FAIL=0 DUTY_V2_GATED_NONOK=0 PAPERLESS_CONTAINER_LINES=0 CORE_RELATED_LINES=1 RESET_FAILED_DONE=no REPORT=/var/lib/homelab-remote-failed-unit-cleanup/paperless-core-guard-audit-latest.md
/var/lib/homelab-health/hapusya-site-final-seal.txt:1:STATUS=OK TS=2026-07-13T12:33:08Z TYPE=hapusya-site-final-seal BAD=0 APP=hapusya-site URL=https://hapusya.ru/ PRESEAL_DESIRED_STATE_HEAD=69966eb83fc065141df3c46df0d6c0f282275046 ROUTE_SHA256=02d65cb0082a04cae282126cc829a002f171e863291c18ba04d788dc2d6282ca UNIT_SHA256=b3b07137b25ac0a0b4bdadadad654cf209387ade86a2ef2cdca58e8a7c0cc212 RELATIVE_TREE_SHA256=9dadf21da041bc4d6ee4a42f5b3ac7719c16bcc39052de32af895c6cd28286cb APPLY_MANIFEST_SHA256=f44992e3e0cc443b372779cb35d9e899d904a9960a9de03c14559886fdb1fc75 PRODUCTION_BROWSER_MANIFEST_SHA256=9f3cdf3063f8724d1d1c1ad619f47d6f98d434c3422a74903585146858a8dfac BROWSER_SMOKE=OK BROWSER_RUNS=5 BROWSER_ERRORS=0 DESKTOP_VIDEO=OK MOBILE_VIDEO=OK STATIC_FALLBACK=OK BACKUP_JOB=homelab-pve01-app-hapusya-site-20260713T115259Z ARCHIVE_SHA256=bf284292974600384c43c47fc99be6fedf0ebd68f722b402a7260f1df680d167 TWO_CLOUDS=OK RESTORE_CHECK=OK_BOTH_CLOUDS RETENTION=OK DESIRED_STATE_SYNC=OK GENERATOR_SHA256=507bbbc2691f1f08cb1dd3660ed76cf13a8c36c23833d5fdec40cf6e07486cb8 MAP_SHA256=815b52dcedf7e3e92e07939bade32c7e8ed6acc0f9e064b65db6daec9f559c63 POLICY_SHA256=4bf996ff451ae3ec56eb2fcd42842606cd97295b4d5141aedb135c956395e4b5 EVIDENCE_ROOT=/var/lib/homelab-change-sets/hapusya-site-final-seal/20260713T123308Z EVIDENCE_MANIFEST_SHA256=7eb993a17ac18a1f7c9ae0b52d745d3778833d15626ed34c82496f74916eb26a PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES PRODUCTION_CHANGED=NO_FINAL_SEAL_ONLY DESIRED_STATE_CHANGED=YES_FINAL_SEAL_COMMIT_ONLY ALL_HAPUSYA_SITE_TAILS_CLOSED=YES TASK_COMPLETE=YES
/var/lib/homelab-health/router-watchdog.txt:1:STATUS=OK TS=2026-07-21T13:20:07Z TYPE=router-watchdog ACTION=none REASON=router_wan_and_outbound_healthy SSH_OK=1 WAN_OK=1 OUTBOUND_OK=2 FAIL_COUNT=0 THRESHOLD=5 LAST_REBOOT_EPOCH=0 REBOOTS_TODAY=0 HARD_POWER_HELPER_READY=yes
/var/lib/homelab-health/pve03-failed-unit-cleanup-closure.txt:1:STATUS=OK TS=2026-07-07T12:52:21Z TYPE=pve03-failed-unit-cleanup-closure BAD=0 WARN=0 MODE=READ_ONLY_CLOSURE LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260707T124031Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260707T124031Z GOLDEN_CP_MATCH=YES RESTORE_CHECK=YES PVE03_FAILED_UNITS=0 REPORT=/var/lib/homelab-pve03-failed-unit-cleanup/closure-latest.md
/var/lib/homelab-health/review-surface-registry-audit.txt:1:STATUS=OK TS=2026-07-08T06:20:10Z TYPE=review-surface-registry-audit BAD=0 WARN=1 MODE=READ_ONLY_REVIEW_SURFACE_REGISTRY_AUDIT LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 OVERALL_STATUS=OK OVERALL_BAD=0 REVIEW_COUNT=22 LEGACY_COUNT=5 HARD_GATE_COUNT=0 CONVERT_PLANNING=19 CONVERT_CLOSURE=4 ARCHIVE_LEGACY=4 FILES_CHANGED=no REGISTRY=/var/lib/homelab-review-surface-registry/review-surfaces-latest.tsv REPORT=/var/lib/homelab-review-surface-registry/audit-latest.md
/var/lib/homelab-health/nextcloud-restore-proof-offhost.txt:1:STATUS=OK TS=20260624T190330Z TYPE=nextcloud-restore-proof-offhost DEST_HOST=pve02 DEST_DIR=/mnt/staging/offhost/edge-restore-proofs-from-edge/nextcloud-aio/nextcloud-aio-restore-proof/20260624T120344Z ITEMS=1 OK=1 ERROR=0 BYTES=2135684 FILES=7 SECRET_PRINTED=NO
/var/lib/homelab-health/recovery-card-seal.txt:1:STATUS=OK TS=2026-07-08T08:34:39Z TYPE=recovery-card-seal BAD=0 WARN=0 MODE=RECOVERY_CARD_FINAL_SEAL RECOVERY_SEAL_OK=YES RECOVERY_CARD_STATUS=OK RECOVERY_CARD_OK=YES FREEZE_STATUS=OK FREEZE_OK=YES FINAL_GLOBAL_GREEN=YES REVIEW_SURFACE_REGISTRY=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T082034Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T082034Z EMERGENCY_JOB=homelab-pve01-emergency-pack-20260708T002739Z VM_DUMPS_JOB=homelab-pve01-vm-local-dumps-20260706T153606Z ENDPOINT_LOCAL_OK=1 ENDPOINT_LAN_OK=1 ENDPOINT_PUBLIC_OK=1 BACKUP_SLA=OK BACKUP_COVERAGE=OK EXTENDED_APPBACKUP=OK VM_BACKUP=OK EXTERNAL_PROBE=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 DIRTY_COUNT=0 CARD=/var/lib/homelab-recovery-card/recovery-card-latest.md SUMMARY=/var/lib/homelab-recovery-card/recovery-summary-latest.tsv REPORT=/var/lib/homelab-recovery-card/recovery-card-seal-latest.md
/var/lib/homelab-health/pve03-failed-unit-cleanup.txt:1:STATUS=OK TS=2026-07-07T12:25:56Z TYPE=pve03-failed-unit-cleanup BAD=0 WARN=0 MODE=RETRY_MANDB_AFTER_SPACE_RECOVERY START_RC=0 MAN_DB_FAILED_LEFT=0 AFTER_FAILED_COUNT=0 ROOT_USAGE_PERCENT=31 STAGING_USAGE_PERCENT=27 REPORT=/var/lib/homelab-pve03-failed-unit-cleanup/man-db-cleanup-latest.md
/var/lib/homelab-health/crypto-portfolio-remove-grt-candidate.txt:1:STATUS=OK TS=2026-07-13T04:20:46Z TYPE=crypto-portfolio-remove-grt-candidate BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=KLINE_CACHE_FILTER_APPLIED_TO_TOP_LEVEL_INSTEAD_OF_ENTRIES EXCLUSIVE_ROOT_CAUSE_PROOF=OK PRODUCTION_CHANGED=NO DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 INPUT_APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd INPUT_WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INPUT_INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 APP_CANDIDATE_SHA256=44df2b25183b7c5d87ff1516ec2d9cd1d169acde3031f7c2aa1fc4e0c7813cd2 WORKER_CANDIDATE_SHA256=174a14b86014535980d200a76d0057110c7af102c66f16c2e21fc3e74bb2e49e INDEX_CANDIDATE_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e MARKET_CANDIDATE_SHA256=eb6c21667d4a42a7a4df8a5d6c3a6808bf69dc5670390e5ee7bfb95687b25074 KLINE_CACHE_CANDIDATE_SHA256=f1a2fe26bedbe457ced26f4efdc605a16aaa1e26e0d5c3b5135533be6292d583 DATABASE_CANDIDATE_SHA256=ae04bf3e7a73568d1fe41813cf1ef224b324276e19a80da1d4b1ca65afcb3d87 CHART_HEALTH_CANDIDATE_SHA256=c16473f16390eeeff4d4eb701db604093564dd78d0be2d30b4cc8ffeeb22364a DATABASE_GRT_AFTER=0|0|0 DATABASE_INTEGRITY=ok CANDIDATE_GRT_REFERENCES=0 KLINE_CACHE_SCHEMA=ENTRIES_OBJECT KLINE_CACHE_GRT_ENTRIES_REMOVED=2 KLINE_CACHE_GRT_ENTRIES_AFTER=0 ACTIVE_COINS=HBAR,POL MARKET_COIN_COUNT=2 EXACT_COINS_SELECTOR_PATCH=OK MOVES_GRID_UNCHANGED=YES READY_FOR_GRT_ISOLATED_DRY_RUN=YES CANDIDATE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-candidate-v3/20260713T042046Z
/var/lib/homelab-health/cluster-admin-stage0-readiness-fix.txt:1:STATUS=OK TS=2026-07-12T23:22:37Z TYPE=cluster-admin-stage0-readiness-fix BAD=0 RESULT=APPLIED FINAL_READINESS=OK FINAL_READINESS_DURATION_SECONDS=17 PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK FAILED_UNITS=0 LIVE_MATCH_DESIRED_STATE=YES READY_FOR_FINAL_SEAL=YES OLD_HEAD=a6e09a8af3604e47d7cc78be4176e6d567e8dbfb NEW_HEAD=1379b4fb4e809e56e9bc16f0e0b19b0a509192d5 CHANGE_SET=/var/lib/homelab-change-sets/stage0-readiness/20260712T232237Z PRODUCTION_CHANGED=READINESS_GATE_LOGIC_ONLY PRODUCTION_SCHEMA_CHANGED=NO CLOUD_WRITE_RUN=NO
/var/lib/homelab-health/crypto-portfolio-remove-grt-recovery-audit.txt:1:STATUS=ERROR TS=2026-07-13T05:39:07Z TYPE=crypto-portfolio-remove-grt-recovery-audit BAD=1 COMMAND_PHYSICAL_LINES=1 ROOT_CAUSE=READINESS_ENDPOINT_MISMATCH RECOVERY_VERIFIED=NO TASK_COMPLETE=NO READY_FOR_GRT_APPLY_SCRIPT_FIX=NO AUDIT_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-recovery-audit-v2/20260713T053907Z
/var/lib/homelab-health/cluster-admin-backup-webpanel-final-seal.txt:1:STATUS=OK TS=2026-07-09T05:24:54Z TYPE=cluster-admin-backup-webpanel-final-seal BAD=0 VMID=180 URL=http://[PRIVATE_IP]:8080/ BACKUP=MAIL_CLOUD_2COPY_VERIFY_OK WEBPANEL=OK WEBPANEL_SYNC=OK OVERALL_HARDGATE=YES FINAL_READINESS_HARDGATE=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-ideal-cluster/cluster-admin-backup-webpanel-final-seal-latest.md
/var/lib/homelab-health/review-surface-registry-seal.txt:1:STATUS=OK TS=2026-07-08T06:36:39Z TYPE=review-surface-registry-seal BAD=0 WARN=0 MODE=REGISTERED_REVIEW_SURFACES_POLICY_AWARE_SEAL PRECHECK=YES REGISTRY_SEAL_OK=YES UPDATED=27 ARCHIVED=27 REGISTERED_COUNT=27 EXACT_WARN_REVIEW_FILES=0 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 FINAL_STATUS=OK FINAL_BAD=0 FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-review-surface-registry/seal-latest.md
/var/lib/homelab-health/mail-cloud-critical.txt:1:STATUS=OK TS=20260721T034025Z TYPE=mail-cloud-critical MODE=recurring REMOTES=2 REMOTE_OK=2 REMOTE_FAIL=0 DEST=app/p0-critical-baseline/20260721T034025Z BUNDLE=p0-critical-baseline-20260721T034025Z.tar.gz SIZE=973275719 SHA256=e1681f82099b2430a46d47c3734063dce8322fd7cec37d4614b3d99ee293b45a RETENTION_KEEP=14 SECRET_PRINTED=no NOTES=none
/var/lib/homelab-health/duty-admin-residual-cleanup.txt:1:STATUS=OK TS=2026-07-08T05:42:29Z TYPE=duty-admin-residual-cleanup BAD=0 WARN=0 MODE=SUPERSEDE_STALE_PVE03_CAPACITY_FORENSICS PRECHECK=YES CLEANUP_OK=YES NEW_PVE03_CAPACITY_STATUS=OK FAILED_LOCAL=0 FAILED_PVE02=0 FAILED_PVE03=0 FAILED_EDGE=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 OVERALL_STATUS=OK OVERALL_BAD=0 WARN_REVIEW_FILES=19 REVIEW_ITEMS_INTENTIONALLY_LEFT=yes REPORT=/var/lib/homelab-duty-admin-residual-cleanup/pve03-capacity-forensics-closure-latest.md
/var/lib/homelab-health/appbackupctl-routing.txt:1:STATUS=OK TS=2026-07-13T00:29:36Z TYPE=appbackupctl-routing BAD=0 RESULT=APPLIED PRODUCTION_CHANGED=BACKUP_CLI_ROUTING_AND_DESIRED_STATE_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO DEFAULT_ROUTE=dockge-stacks CLUSTER_ADMIN_ROUTE=homelab-cluster-admin-incident-engine-backup.service UNSUPPORTED_TARGET_POLICY=REFUSE_RC_2 ROUTE_TEST_DOCKGE=OK ROUTE_TEST_CLUSTER_ADMIN=OK ROUTE_TEST_DEFAULT=OK ROUTE_TEST_UNSUPPORTED=OK FRAMEWORK_CHECK=OK UNSUPPORTED_ROUTE_BACKUP_CREATED=NO LIVE_MATCH_DESIRED_STATE=YES PVE01_FAILED_UNITS=0 OVERALL=OK DUTY_ADMIN=OK STAGE1_CLOSED=YES STAGE2_READY=YES OLD_HEAD=bd5eb856a8d61a5fb83c2bcaf74cb8ac98f075c5 NEW_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 CHANGE_SET=/var/lib/homelab-change-sets/appbackupctl-routing/20260713T002936Z
/var/lib/homelab-health/final-readiness.txt:1:STATUS=WARN TS=2026-07-21T05:31:30Z TYPE=final-readiness BAD=5 PASS=48 FAIL=5 REPORT=/var/lib/homelab-final-readiness/latest.md
/var/lib/homelab-health/cluster-admin-stage3-api-apply.txt:1:STATUS=OK TS=2026-07-13T03:22:35Z TYPE=cluster-admin-stage3-api-apply BAD=0 COMMAND_PHYSICAL_LINES=1 RESULT=APPLIED PRODUCTION_CHANGED=LIVE_APP_AND_DESIRED_STATE PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO SERVICE_RESTARTED=YES CLOUD_WRITE_RUN=NO_REUSE_VERIFIED_PREAPPLY_BACKUP APP_PRECHANGE_SHA256=c2748b8f96df842f491b5d2790790594222549853d0e2336cf60ecd6d4de0020 APP_CANDIDATE_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 LIVE_APP_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 LIVE_MATCH_DESIRED_STATE=YES PREAPPLY_BACKUP_REUSED=YES AST_IDENTICAL_NORMALIZATION=YES EXISTING_ROUTES_BASELINE=OK NEW_GET_ROUTES=7 UNAUTHENTICATED_NEW_ROUTES_401=YES OPENAPI_DOCS_EXPOSURE_CHANGED=NO CORE_DATABASE_STATE=1|1|9|7|1 INCIDENT_ENGINE=ACTIVE VM180_FAILED_UNITS=0 ROLLBACK_CHECKPOINT=VERIFIED OLD_HEAD=8f888e5f2063e49da918b31584c9ea4bcf53af3a NEW_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 READY_FOR_STAGE3_AUTHENTICATED_PRODUCTION_VERIFY=YES STAGE3_CLOSED=NO CHANGE_SET=/var/lib/homelab-change-sets/stage3-api-production-apply/20260713T032235Z
/var/lib/homelab-health/cluster-admin-stage1-foundation-apply.txt:1:STATUS=OK TS=2026-07-13T00:21:21Z TYPE=cluster-admin-stage1-foundation-apply BAD=0 RESULT=APPLIED PRODUCTION_CHANGED=DATABASE_FOUNDATION_AND_DESIRED_STATE PRODUCTION_SCHEMA_CHANGED=PLANNED_ADDITIVE_ONLY SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO MIGRATION_ID=001_entity_source_policy_foundation NEW_TABLES=9 COLLECTOR_RUN_COLUMNS=7 TRIGGER=collector_runs_foundation_enrich_trg FUNCTION=cluster_admin_enrich_collector_run ENTITIES=83 PLACEHOLDERS=21 ENTITY_DEPENDENCIES=26 ENTITY_ALIASES=225 ALIAS_COLLISIONS=3 SOURCES=1 SOURCE_INSTANCES=1 SOURCE_EXPECTATIONS=1 POLICIES=1 POLICY_VERSIONS=1 ORPHANS=0 LIVE_COLLECTOR_COMPATIBILITY=OK API_HEALTH=OK VM180_FAILED_UNITS=0 ROLLBACK_POINT=VERIFIED OLD_HEAD=1379b4fb4e809e56e9bc16f0e0b19b0a509192d5 NEW_HEAD=bd5eb856a8d61a5fb83c2bcaf74cb8ac98f075c5 LIVE_MATCH_DESIRED_STATE=YES CHANGE_SET=/var/lib/homelab-change-sets/stage1-production-apply/20260713T002121Z
/var/lib/homelab-health/homelab-admin-guardrails.txt:1:STATUS=OK TS=2026-07-14T07:39:53Z TYPE=homelab-admin-machine-guardrails BAD=0 RUNNER_VERSION=0.2.0-guardrails LINTER_HOTFIX=2.1 RULES=13 FAIL_FAST=YES ERR_TRAP_REQUIRED=YES UNGUARDED_GREP_COUNT_BLOCKED=YES RUNNER_SHA256=bc76e629e434cc6406ae2f3e7886034a0cedbca43795b6f4197a1ee517c1f322 LINTER_SHA256=1d0cdc2c31adfb7446d9f253539b4004621c6d29a5ea6d1c99bbd9aed34957a2 POLICY_SHA256=ce6d46a01dfe14adc8b2a35a683f2771e4324b04fea611dd4f4bd0458b09d1af DESIRED_STATE_HEAD=69be75caeecaa242567d2d06c735ac5db3a9c048 EVIDENCE_ROOT=/var/lib/homelab-change-sets/homelab-admin-guardrails-v21-install/20260714T073953Z
/var/lib/homelab-health/cluster-admin-stage4i-scope-definition.txt:12:TASK_MODEL=VERSIONED_SCHEMA_V2_FAIL_FAST
/var/lib/homelab-health/cluster-admin-stage4h-final-acceptance.txt:15:ERROR_REGISTER_SHA256=ab8308549db90d235b591af0581580c3c7ba1b287b129cc6111c2eef0a0f9ecc
/var/lib/homelab-health/cluster-admin-stage4h-final-acceptance.txt:35:VM180_FAILED_UNITS=0
/var/lib/homelab-health/cluster-admin-stage4h-final-acceptance.txt:36:PVE01_FAILED_UNITS=0
/var/lib/homelab-health/ideal-cluster-final-seal.txt:1:STATUS=OK TS=2026-07-08T19:49:09Z TYPE=ideal-cluster-final-seal BAD=0 MODE=FINAL_GREEN_SEAL TOP_GATES_OK=YES NODE_LOSS_OK=YES NOISE_SEAL_OK=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 TEST_VMS_CLEANED=YES PROD_VMS_RUNNING=YES DNS_PAIR_OK=YES RESTORE_CHECK_OK=YES DIRTY_COUNT=0 REPORT=/var/lib/homelab-ideal-cluster/final-seal-latest.md
/var/lib/homelab-health/vm160-remote-backup.txt:1:STATUS=FAIL
/var/lib/homelab-health/dockge-final-closure.txt:7:EDGE_STALE_REMAINING_MATCHES=0
/var/lib/homelab-health/disk-space.txt:10:ERRORS=none
/var/lib/homelab-health/remote-failed-unit-cleanup-seal.txt:1:STATUS=OK TS=2026-07-08T05:30:11Z TYPE=remote-failed-unit-cleanup-seal BAD=0 WARN=0 MODE=PAPERLESS_CORE_GUARD_SUPERSEDED_FINAL_SEAL PRECHECK=YES EDGE_APPLY=1 REMOTE_CLEANUP_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 EDGE_GUARD_STATE=inactive EDGE_MARKER_STATUS=OK DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-remote-failed-unit-cleanup/seal-latest.md
/var/lib/homelab-health/special-vm-emergency-non-app-final-seal.txt:1:STATUS=OK TS=2026-07-14T01:15:48Z TYPE=special-vm-emergency-non-app-final-seal BAD=0 WARN=0 FAMILY_COVERAGE=11/11 MATRIX_STATUS=OK JOBS_TOTAL=40 JOBS_OK=36 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 COMPATIBILITY_OUTPUTS=31 REQUIRED_GREEN_JOBS=14 VM_FULL=OK VM_LOCAL_DUMPS=OK EDGE_VM_SOURCE=OK EDGE_VM_CLOUD=OK NEXTCLOUD_VM_SOURCE=homelab_vm_local_dumps_2cloud NEXTCLOUD_VM_CLOUD=OK EMERGENCY_PACK=OK SKLADCHIK_REPORTS_MONITOR=OK GOLDEN_STATE=OK ROUTER_RUNNING_CONFIG=OK MAIL_CLOUD_CRITICAL=OK MAIL_CLOUD_EDGE=OK MAIL_CLOUD_NEXTCLOUD=OK WRITE_MODE=legacy READ_MODE=dual MIGRATE_EXISTING=no SOURCE_BACKUP_RUN=NO CLOUD_OPERATIONS_RUN=NO RETENTION_RUN=NO CLOUD_PURGE_RUN=NO CLUSTER_ADMIN=OK PROBE_OK=4 PROBE_BAD=0 CLUSTER_PASSPORT=OK DUTY_ADMIN=OK OVERALL=OK OVERALL_BAD=0 FINAL_READINESS=OK FINAL_PASS=53 FINAL_FAIL=0 WEBPANEL=OK PUBLIC_OVERALL=OK FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DESIRED_STATE_PRE_SEAL_HEAD=b26f498b55de5d80bcbb4b9fb3a9bccded9f99b4 CURRENT_SPECIAL_VM_EMERGENCY_NON_APP_STAGE_COMPLETE=YES EVIDENCE_ROOT=/var/lib/homelab-change-sets/special-vm-emergency-non-app-final-seal/20260714T011514Z
/var/lib/homelab-health/independent-external-runner.txt:1:STATUS=OK TS=2026-07-08T09:17:06Z TYPE=independent-external-runner BAD=0 WARN=0 MODE=NO_DEPLOY_SEAL NO_DEPLOY_SEAL_OK=YES AUDIT_STATUS=OK AUDIT_OK=YES INDEPENDENT_RUNNER_READY=NO DEPLOY_DECISION=NO_DEPLOY CURRENT_INDEPENDENT_RUNNER_AVAILABLE=no TRUSTED_NETBIRD_IDENTITY=edge-vm PUBLIC_VPS_DECISION=REJECTED_HOSTKEY_MISMATCH PUBLIC_VPS_SSH_OK=0 PUBLIC_VPS_REJECTED_HOSTKEY=YES EDGE_LAN_SSH_OK=2 EDGE_NETBIRD_SSH_OK=2 EDGE_DNS_SSH_OK=1 PUBLIC_OVERALL_OK=1 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 FINAL_STATUS=OK FINAL_FAIL=0 CLOSEOUT_OK=YES RECOVERY_SEAL_OK=YES RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T090905Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T090905Z DIRTY_COUNT=0 REPORT=/var/lib/homelab-independent-external-runner/no-deploy-seal-latest.md
/var/lib/homelab-health/node-exporter-restore.txt:1:STATUS=OK TS=2026-07-09T08:13:40Z TYPE=node-exporter-restore JOB=homelab-pve01-app-node-exporter-20260709T081340Z RESTORE=/tmp/node-exporter-restore-20260709T081340Z RESTORE_CHECK=PACKAGE_OK PVE01_TEXTFILE_SCRAPE_ERROR_AFTER=0 NOTE=service-readiness-restore-proof
/var/lib/homelab-health/cluster-admin-stage2-apply.txt:1:STATUS=OK TS=2026-07-13T01:38:37Z TYPE=cluster-admin-stage2-apply BAD=0 RESULT=APPLIED PRODUCTION_CHANGED=CANONICAL_SYNC_FUNCTION_COLLECTOR_MIGRATION_ARTIFACTS_AND_DESIRED_STATE PRODUCTION_SCHEMA_CHANGED=PLANNED_FUNCTION_AND_MIGRATION_REGISTRY_ONLY INCIDENT_ENGINE_RESTARTED=NO COLLECTOR_ONESHOT_RUN=YES COLLECTOR_TIMER_PAUSED_AND_RESUMED=YES CLOUD_WRITE_RUN=NO MIGRATION_ID=002_canonical_foundation_sync FUNCTION=cluster_admin_sync_foundation PREFLIGHT_NATURAL_KEY_DRIFT_ALLOWED=YES PREFLIGHT_NATURAL_KEY_MISSING_BEFORE=1 PREFLIGHT_NATURAL_KEY_STALE_BEFORE=1 PREFLIGHT_NATURAL_KEY_MISSING_AFTER=0 PREFLIGHT_NATURAL_KEY_STALE_AFTER=0 PREFLIGHT_EPHEMERAL_ID_DRIFT_ALLOWED=YES PREFLIGHT_EPHEMERAL_ID_DRIFT_BEFORE=25 PREFLIGHT_EPHEMERAL_ID_DRIFT_AFTER=0 PREFLIGHT_TRANSACTION_ROLLBACK=OK COMMITTED_SYNC=YES PARTIAL_ROLLBACK_AFTER_COMMIT=NOT_EXECUTED APPLIED_STATE_PRESERVED_POLICY=ACTIVE SAME_TRANSACTION=YES FIXED_TOTAL_ENTITIES_REQUIRED=NO FIXED_TOTAL_ALIASES_REQUIRED=NO FOUNDATION_COUNTS=85|21|2|229|26|0 DYNAMIC_INVARIANTS=0|0|21|21|0|0|0|0|0|0|0 ACTIVE_CURRENT_SET_CONVERGENCE=OK NATURAL_KEY_CONVERGENCE=OK EPHEMERAL_ID_REFRESH=OK RETIRED_PLACEHOLDERS_ALLOWED=YES ORPHANS=0 LIVE_COLLECTOR_RUN_ID=2992 LIVE_COLLECTOR_STATUS=OK SOURCE_INSTANCE_UPDATE=OK OBSERVER=OK FULL_OBSERVER=OK RESTRICTED_PROBES=OK FINAL_READINESS=OK CLUSTER_PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK API_HEALTH=OK VM180_FAILED_UNITS=0 PVE01_FAILED_UNITS=0 ROLLBACK_POINT=VERIFIED LIVE_MATCH_DESIRED_STATE=YES OLD_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 NEW_HEAD=8f888e5f2063e49da918b31584c9ea4bcf53af3a READY_FOR_POSTAPPLY_SEAL=YES STAGE2_CLOSED=NO CHANGE_SET=/var/lib/homelab-change-sets/stage2-production-apply/20260713T013837Z
/var/lib/homelab-health/cluster-admin-node-online-green-dot-fix.txt:1:STATUS=OK TS=2026-07-14T01:47:22Z TYPE=cluster-admin-node-online-green-dot-fix BAD=0 WARN=0 ROOT_CAUSE=STATUS_KIND_MISSING_ONLINE_AND_OFFLINE PRE_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 POST_SHA256=a068c1824943af182acbaebcd3ff89e84203a3fde17ccb8051a4680e37d9f5e5 NODE_STATUS=online NODE_COUNT=3 ONLINE_CLASSIFICATION=ok ONLINE_BADGE_CLASS=state-ok OFFLINE_CLASSIFICATION=bad OFFLINE_BADGE_CLASS=state-bad DATABASE_CHANGED=NO COLLECTOR_CHANGED=NO SERVICE_RESULT=success EXEC_MAIN_STATUS=0 API_HEALTH=OK BACKUP_JOB=homelab-pve01-app-cluster-admin-incident-engine-20260714T014645Z BACKUP_STATUS=success BACKUP_TWO_CLOUDS=OK RESTORE_CHECK=OK_BOTH_CLOUDS REMOTE_ROLLBACK=/var/lib/cluster-admin-incident-engine/change-sets/node-online-green-dot-20260714T014640Z/app.py.before EVIDENCE_ROOT=/var/lib/homelab-change-sets/cluster-admin-node-online-green-dot-fix/20260714T014640Z
/var/lib/homelab-health/crypto-portfolio-ui-preapply-backup.txt:1:STATUS=OK TS=2026-07-13T07:42:53Z TYPE=crypto-portfolio-ui-preapply-backup BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=INLINE_PYTHON_QUOTE_ERROR_IN_NONESSENTIAL_HELPER_PROOF BACKUP_SERVICE_RERUN=NO MANUAL_REHYDRATE_RERUN=NO CANONICAL_RESTORE_CHECK_RERUN=YES_READ_ONLY PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CONTAINER_RECREATED=NO CLOUD_WRITE_RUN=NO_READ_ONLY_RESTORE_CHECK TARGET_HOST=core-apps.gram1.ru INPUT_INDEX_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e CANDIDATE_INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T071127Z BACKUP_STATUS=success BACKUP_SCHEMA=homelab-app-backup-manifest-v2-flat BACKUP_LAYOUT=flat-v1 BACKUP_PATH_REL=hb/a/cp-260713-071127 BACKUP_DATABASE_INTEGRITY=ok BACKUP_FILE_COUNT=19 BACKUP_ARCHIVE_SIZE_BYTES=136632 BACKUP_ARCHIVE_SHA256=b29eb55820ec46c474f51e995e92e371251b6df717ac98182d807d1dbf8473f5 BACKUP_LOCAL_STAGING_DELETED=YES CANONICAL_TWO_CLOUD_RESTORE_CHECK=OK APPBACKUPCTL_RESTORE_CHECK=OK_BOTH_CLOUDS ROLLBACK_PATH=/var/lib/homelab-incidents/crypto-ui-center-alert-currency-prechange-20260713T071125Z ROLLBACK_MANIFEST_SHA256=0ef56c8f021f549777131ffe878cd5a263bdfb6f1dfb8bb080cefdd554573aca ROLLBACK_INDEX_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e PRODUCTION_UNCHANGED=YES PRODUCTION_HEALTH=200 VM170_FAILED_UNITS=0 PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES CHART_HEALTH_TIMER=ACTIVE BACKUP_TIMER=ACTIVE DESIRED_STATE_HEAD=80763a6c62b94cde4f3293f3b529980f76919d25 DESIRED_STATE_CLEAN=YES DESIRED_STATE_REMOTE_COUNT=0 READY_FOR_UI_PRODUCTION_APPLY=YES PREAPPLY_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-ui-preapply-backup-canonical-close/20260713T074253Z
/var/lib/homelab-health/router-hard-power.txt:1:STATUS=ERROR TS=2026-07-16T06:20:34Z MODE=cycle REMOTE=[PRIVATE_IP] RC=74
/var/lib/homelab-health/operator-handoff-pack.txt:1:STATUS=OK TS=2026-07-08T09:42:39Z TYPE=operator-handoff-pack BAD=0 WARN=0 MODE=FINAL_OPERATOR_HANDOFF_PACK HANDOFF_OK=YES PACK_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T093758Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T093758Z RECOVERY_CARD=YES FREEZE_PACK=YES FINAL_CLOSEOUT=YES INDEPENDENT_RUNNER_NO_DEPLOY=YES DESIRED_STATE_REMOTE_GIT=YES GIT_REMOTE_CHANGED=NO GIT_PUSH_DONE=NO DIRTY_COUNT=0 PACK=/var/lib/homelab-operator-handoff/operator-handoff-pack-20260708T094239Z.tar.gz README=/var/lib/homelab-operator-handoff/HANDOFF-latest.md STATUS_TSV=/var/lib/homelab-operator-handoff/handoff-status-latest.tsv NEXT=/var/lib/homelab-operator-handoff/NEXT-ACTIONS-latest.md
/var/lib/homelab-health/pve01-failed-units.txt:1:STATUS=OK TS=20260624T180727Z TYPE=pve01-failed-units FAILED_COUNT=0 ACTION=RESET_FAILED_ONLY SECRET_PRINTED=REDACTED
/var/lib/homelab-health/final-global-green-audit.txt:1:STATUS=OK TS=2026-07-08T07:20:01Z TYPE=final-global-green-audit BAD=0 WARN=0 MODE=SELF_REFERENCE_CLOSED_FINAL_GLOBAL_GREEN_SEAL AUDIT_OK=YES PRE_STATUS=WARN SELF_REFERENCE_CLOSED=yes LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 EXACT_WARN_REVIEW_FILES=0 LOCAL_OVERALL_OK=1 LAN_OVERALL_OK=1 PUBLIC_OVERALL_OK=1 EXTERNAL_PROBE_STATUS=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T064444Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T064444Z DIRTY_COUNT=0 REPORT=/var/lib/homelab-final-global-green-audit/final-seal-latest.md
/var/lib/homelab-health/remote-failed-unit-audit.txt:1:STATUS=OK TS=2026-07-08T05:20:16Z TYPE=remote-failed-unit-audit BAD=0 WARN=1 MODE=READ_ONLY_REMOTE_FAILED_UNIT_AUDIT LOCAL_FAILED_COUNT=1 REMOTE_FAILED_COUNT=8 CONNECT_ERRORS=0 JOURNAL_FAIL_HINTS=194 OVERALL_BAD=0 FINAL_FAIL=0 DUTY_V2_GATED_NONOK=0 RESET_FAILED_DONE=no REPORT=/var/lib/homelab-remote-failed-unit-cleanup/audit-latest.md
/var/lib/homelab-health/legacy-monitoring-backups-migrated.txt:1:STATUS=OK TS=2026-07-10T20:00:03Z TYPE=legacy-monitoring-backups-migrated BAD=0 APPS=6 RESTORE_FAILURES=0 ACTIVE_LEGACY_STATES=0 OFFHOST=OK RESTORE=OK
/var/lib/homelab-health/cluster-admin-stage4h-collector-patch-candidate.txt:1:STATUS=OK TS=2026-07-14T17:41:32Z TYPE=cluster-admin-stage4h-collector-patch-candidate FAILED_ATTEMPT_TAIL=CLOSED CANDIDATE_SUBSTEP=CLOSED STAGE4H_STATUS=OPEN PRODUCTION_CHANGE=NO UNRESOLVED_TAIL_COUNT=0 BLOCKER_COUNT=0 NEXT_ACTION=TEMPORARY_DATABASE_MIGRATION_AND_COLLECTOR_ACCEPTANCE_HARNESS CANDIDATE_SHA256=4204fe568d46fec84f643af974c82770cb5d5b08d3d8ceb4aad96cca57abc909 REPORT_SHA256=ecfdaab5438e1fc3bc7a90e08ec68c71d06a6c6a91a430dc1ea7b6a74b7699a9
/var/lib/homelab-health/cluster-admin-stage2-final-seal.txt:1:STATUS=OK TS=2026-07-13T01:45:29Z TYPE=cluster-admin-stage2-final-seal BAD=0 STAGE1_CLOSED=YES STAGE2_APPLY=OK PRODUCTION_CHANGED=BACKUP_HEALTH_AND_SEAL_ARTIFACTS_ONLY PRODUCTION_SCHEMA_CHANGED=NO_ADDITIONAL_CHANGE_DURING_SEAL INCIDENT_ENGINE_RESTARTED=NO MIGRATION_ID=002_canonical_foundation_sync FUNCTION=cluster_admin_sync_foundation COMMITTED_SYNC=YES MULTICYCLE_COLLECTOR_TEST=OK MULTICYCLE_NEW_RUNS=2 MULTICYCLE_GOOD_RUNS=2 ACTIVE_CURRENT_SET_CONVERGENCE=OK NATURAL_KEY_CONVERGENCE=OK EPHEMERAL_ID_REFRESH=OK RETIRED_PLACEHOLDERS_ALLOWED=YES FOUNDATION_COUNTS=85|21|2|229|26|0 DYNAMIC_INVARIANTS=0|0|21|21|0|0|0|0|0|0|0 ORPHANS=0 POSTAPPLY_DIRECT_SNAPSHOT=VERIFIED POSTAPPLY_RESTORE_TEST=OK POSTAPPLY_APPBACKUP=VERIFIED POSTAPPLY_APPBACKUP_JOB=homelab-pve01-app-cluster-admin-incident-engine-20260713T014757Z LIVE_MATCH_DESIRED_STATE=YES DESIRED_STATE_HEAD=8f888e5f2063e49da918b31584c9ea4bcf53af3a OBSERVER=OK FULL_OBSERVER=OK RESTRICTED_PROBES=OK FINAL_READINESS=OK CLUSTER_PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK VM180_FAILED_UNITS=0 PVE01_FAILED_UNITS=0 CURRENT_HEALTH_RECHECKED=YES STAGE2_CLOSED=YES REPORT=/var/lib/homelab-cluster-admin-stage2/stage2-final-seal-latest.txt REPORT_SHA256=32d4d37d82aa25194d789083d0e7ad2e9052ee38a72df5312b711226898cf1a6 BLOCKERS=/var/lib/homelab-cluster-admin-stage2/20260713T014529Z/blockers.txt
/var/lib/homelab-health/remote-failed-unit-plan.txt:1:STATUS=OK TS=2026-07-08T05:22:47Z TYPE=remote-failed-unit-plan BAD=0 WARN=1 MODE=READ_ONLY_CLEANUP_PLAN LOCAL_FAILED_COUNT=1 REMOTE_FAILED_COUNT=7 LOCAL_NOOP_COUNT=4 REMOTE_RESET_ONLY_COUNT=1 REMOTE_NOOP_COUNT=5 REMOTE_DEFER_COUNT=1 OVERALL_BAD=0 FINAL_FAIL=0 RESET_FAILED_DONE=no PLAN=/var/lib/homelab-remote-failed-unit-cleanup/cleanup-plan-latest.tsv REPORT=/var/lib/homelab-remote-failed-unit-cleanup/plan-latest.md
/var/lib/homelab-health/review-surface-registry-apply.txt:1:STATUS=OK TS=2026-07-08T06:36:39Z TYPE=review-surface-registry-apply BAD=0 WARN=0 MODE=POLICY_AWARE_DUTY_ADMIN_REPAIR_SELF_CLOSED PRECHECK=YES APPLY_OK=YES REPAIR_OK=YES UPDATED=27 ARCHIVED=27 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW_WAS=1 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_BAD=0 SELF_REFERENCE_CLOSED=YES REPORT=/var/lib/homelab-review-surface-registry/seal-latest.md
/var/lib/homelab-health/private-vpn-hosts.txt.bak-ok-wrapper-20260624T132705Z:16:STATUS=ERROR
/var/lib/homelab-health/cluster-admin-strict-final-seal.txt:1:STATUS=OK TS=2026-07-09T04:49:23Z TYPE=cluster-admin-strict-final-seal BAD=0 VMID=180 IP=[PRIVATE_IP] MODE=DEDICATED_CLUSTER_ADMIN_HARDGATED OBSERVER=OK RESTRICTED_PROBES=OK FULL_OBSERVER=OK OVERALL_HARDGATE=YES FINAL_READINESS_HARDGATE=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-ideal-cluster/cluster-admin-strict-final-seal-latest.md
/var/lib/homelab-health/paperless-restore.txt:1:STATUS=STALE
/var/lib/homelab-health/paperless-restore.txt:20:FRESHNESS=STALE
/var/lib/homelab-health/ideal-cluster-noise-seal.txt:1:STATUS=OK TS=2026-07-08T17:41:49Z TYPE=ideal-cluster-noise-seal BAD=0 MODE=POLICY_AWARE_NON_STATUS_ARTIFACT_SEAL_SELF_REFERENCE_CLOSED SELF_REFERENCE_CLOSED=YES NON_STATUS_TOTAL=8 NON_STATUS_OK_INFO=8 NON_STATUS_UNKNOWN=0 WARNBAD_ACCEPTED_POLICY_INFO=13 WARNBAD_NEEDS_REVIEW=0 TOP_GATES_OK=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-ideal-cluster/noise-seal-latest.md
/var/lib/homelab-health/crypto-portfolio-chart-health.txt:1:STATUS=OK TS=2026-07-21T13:18:29Z TYPE=crypto-portfolio-chart-health BAD=0 LIBRARY_OK=true DISK_CACHE_PRESENT=true SYMBOLS=2 STALE_COUNT=2
/var/lib/homelab-health/duty-admin-v2.txt:1:STATUS=OK TS=2026-07-21T13:16:33Z TYPE=duty-admin-v2 BAD=7 WARN=0 MODE=LOCAL_LONG_RUNNING_ANALYZER_POLICY_AWARE SEPARATE_VM=NO TOTAL_HEALTH_INPUTS=497 GATED_OK=45 GATED_NONOK=7 UNGATED_ERROR_REVIEW=2 UNGATED_WARN_REVIEW=3 UNGATED_REVIEW_ITEM=0 LEGACY_OR_PHASE_REVIEW=0 REPORT=/var/lib/homelab-duty-admin-v2/latest.md
/var/lib/homelab-health/cluster-admin-stage3-final-seal.txt:1:STATUS=OK TS=2026-07-13T03:46:30Z TYPE=cluster-admin-stage3-final-seal BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=HEALTH_REFRESH_AND_SEAL_ARTIFACTS_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO LIVE_APPLICATION_CODE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO STAGE2_CLOSED=YES STAGE3_APPLY=OK AUTHENTICATED_PRODUCTION_VERIFY=OK AUTHENTICATED_PRODUCTION_TESTS=25 APP_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 LIVE_MATCH_DESIRED_STATE=YES DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 NEW_GET_ROUTES=7 DATABASE_TRANSACTION_MODE=BEGIN_READ_ONLY OPENAPI_DOCS_EXPOSURE=PRESERVED_DISABLED POSTAPPLY_APPBACKUP=VERIFIED POSTAPPLY_APPBACKUP_JOB=homelab-pve01-app-cluster-admin-incident-engine-20260713T034400Z POSTAPPLY_APPBACKUP_RESTORE_CHECK=OK COLLECTOR_STABILITY=OK LATEST_COLLECTOR_RUN_ID=3114 CANONICAL_CONSISTENCY=0|0|0|0|0 OBSERVER=OK FULL_OBSERVER=OK RESTRICTED_PROBES=OK FINAL_READINESS=OK CLUSTER_PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK VM180_FAILED_UNITS=0 PVE01_FAILED_UNITS=0 CURRENT_HEALTH_RECHECKED=YES STAGE3_CLOSED=YES REPORT=/var/lib/homelab-cluster-admin-stage3/stage3-final-seal-latest.txt REPORT_SHA256=6f67545cd7f0a75b87146c1df74c61c3b88e142f9e53a96b083ec27209e6d265
/var/lib/homelab-health/hapusya-site-production-browser.txt:1:STATUS=OK TS=2026-07-13T11:48:15Z TYPE=hapusya-site-production-browser BAD=0 URL=https://hapusya.ru/ BROWSER_RUNS=5 BROWSER_ERRORS=0 DESKTOP_VIDEO=OK MOBILE_VIDEO=OK STATIC_FALLBACK=OK EVIDENCE_ROOT=/var/lib/homelab-change-sets/hapusya-site-production-browser-v2/20260713T114810Z MANIFEST_SHA256=9f3cdf3063f8724d1d1c1ad619f47d6f98d434c3422a74903585146858a8dfac ROUTE_SHA256=02d65cb0082a04cae282126cc829a002f171e863291c18ba04d788dc2d6282ca UNIT_SHA256=b3b07137b25ac0a0b4bdadadad654cf209387ade86a2ef2cdca58e8a7c0cc212 RELATIVE_TREE_SHA256=9dadf21da041bc4d6ee4a42f5b3ac7719c16bcc39052de32af895c6cd28286cb PRODUCTION_CHANGED=YES CLOSURE_PENDING=POSTCHANGE_BACKUP_RETENTION_DESIRED_STATE_FINAL_SEAL
/var/lib/homelab-health/backup-matrix-authoritative.txt:1:STATUS=FAIL
/var/lib/homelab-health/backup-matrix-desired-state-persistence.txt:1:STATUS=OK TS=2026-07-13T17:40:37Z TYPE=backup-matrix-desired-state-persistence BAD=0 GENERATED_HEAD=af26ec3c4829b1330d49656205ac580ce1706263 GENERATED_PARENT=25568fab24c0594418b3676dc511c8234b4e1ae5 GENERATOR_SHA256=19da7f049b4f7f7286faf292c5419c37cc1e1768c2d8fe36b257ab0cf9c18874 MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 DESIRED_MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 MATRIX_STATUS=OK JOBS_BAD=0 FAILED_UNITS=0 BACKUP_RUN=NO RESTORE_RUN=NO CLOUD_WRITE_RUN=NO RETENTION_RUN=NO ROLLBACK=/var/lib/homelab-incidents/backup-matrix-desired-state-persistence-20260713T174036Z
/var/lib/homelab-health/duty-admin-residual-audit.txt:1:STATUS=OK TS=2026-07-08T05:40:37Z TYPE=duty-admin-residual-audit BAD=0 WARN=3 MODE=READ_ONLY_RESIDUAL_NOISE_AUDIT LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=1 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 OVERALL_STATUS=OK OVERALL_BAD=0 FINAL_FAIL=0 WARN_REVIEW_FILES=19 HARD_GATE_WARN_MATCHES=0 REPORT=/var/lib/homelab-duty-admin-residual-cleanup/audit-latest.md
/var/lib/homelab-health/ungated-health-backlog.txt:1:STATUS=OK TS=2026-07-07T07:45:01Z TYPE=ungated-health-backlog BAD=0 WARN=1 MODE=POST_PVE03_ROOT_CLEANUP_TRIAGE TOTAL=8 SUPERSEDED=4 STALE=2 POLICY_GAP_NEXT_STAGE=1 ACTIONABLE_CAPACITY=1 PVE03_ROOT_USAGE_PERCENT=31 PVE03_STAGING_USAGE_PERCENT=83 REPORT=/var/lib/homelab-ungated-health-backlog/latest.md
/var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-final-seal.txt:1:STATUS=OK TS=2026-07-13T08:37:54Z TYPE=crypto-portfolio-ui-center-alert-currency-final-seal BAD=0 DANGLING_IMAGE_REFERENCE_GATE=OK DESIRED_STATE_SYNC_ARTIFACT_GATE=OK APP=crypto-portfolio PRODUCTION_CHANGED=NO_FINAL_SEAL_ONLY APP_SHA256=44df2b25183b7c5d87ff1516ec2d9cd1d169acde3031f7c2aa1fc4e0c7813cd2 WORKER_SHA256=174a14b86014535980d200a76d0057110c7af102c66f16c2e21fc3e74bb2e49e INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb LIBRARY_SHA256=c0992580867c4912cc9385b3c2728315bcc1a76c7f1087dca908430fccdf31d7 ACTIVE_COINS=HBAR,POL GRT_REMOVED=YES GRT_ROWS=0|0|0 GRT_SOURCE_REFERENCES=0 GRT_CACHE_ENTRIES=0 SHARP_MOVES_CENTERED=YES ALERT_CURRENCY_USD_RUB=OK BROWSER_SMOKE=OK BROWSER_RUNS=2 BROWSER_ERRORS=0 BROWSER_RESULT_SHA256=b566f492cd80c8f99bc6b24cbef6d5f326c04f4d7168359d05eb1f189589e733 SCREENSHOT_SHA256=00b2853bee9c880146d0dd75fac60feb7e46d4c129ee06754047ef30d90bd4ba BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T075801Z TWO_CLOUDS=OK RESTORE_CHECK=OK DATABASE_INTEGRITY=ok RETENTION=OK ROLLBACK_COUNT=3 DANGLING_IMAGES=2 REFERENCED_DANGLING_IMAGES=2 UNREFERENCED_DANGLING_IMAGES=0 PRUNED_UNREFERENCED_DANGLING_IMAGES=0 CHART_HEALTH=OK PUBLIC=OK MONITOR_TIMER=active BACKUP_TIMER=active CLOSURE_DESIRED_STATE_HEAD=2999910549eaff9b9431b9c656d318340eb95cbd DESIRED_STATE=OK PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES ALL_CRYPTO_PORTFOLIO_UI_TAILS_CLOSED=YES
/var/lib/homelab-health/crypto-portfolio-remove-grt-runtime-verifier-prepared.txt:1:STATUS=OK TS=2026-07-13T04:35:29Z TYPE=crypto-portfolio-remove-grt-runtime-verifier-prepared BAD=0 COMMAND_PHYSICAL_LINES=1 VERIFIER_REVISION=V3 DEEP_ANALYSIS_RESULT=DIRECT_ROUTE_JSONRESPONSE_CONTRACT_CORRECTED FAILURE_CLASS=VERIFIER_ONLY PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO CANDIDATE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CONTAINER_STARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 INPUT_APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd INPUT_WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INPUT_INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 CANDIDATE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-candidate-v3/20260713T042046Z RUNTIME_POLICY=CONTAINER_PYTHON_ONLY PYTHONPATH_APP_INJECTED=YES ROUTE_RETURN_DECODER=DICT_OR_RESPONSE_BODY STATE_RETURN_DECODED=YES CHART_HEALTH_RETURN_DECODED=YES FETCH_KLINES_RETURN_DECODED=YES PROTECTED_KLINES_CODES=401_OR_403 LIVE_PRODUCTION_DATA_MOUNTED=NO ISOLATED_PORT=18030 FAILED_RUN_CLEANUP=VERIFIED REMOTE_RUNTIME_SCRIPT_SHA256=b60290b08f6bdb9801f4ccadbeadc4268399426996fd271527830fd399055448 VERIFIER_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-runtime-verifier-v3/20260713T043529Z READY_FOR_GRT_ISOLATED_RUNTIME_DRY_RUN_RUN=YES
/var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-candidate.txt:1:STATUS=OK TS=2026-07-13T06:11:23Z TYPE=crypto-portfolio-ui-center-alert-currency-candidate BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=EXPECTED_DIFF_RC_1_TRIGGERED_SET_E_PIPEFAIL PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=80763a6c62b94cde4f3293f3b529980f76919d25 INPUT_INDEX_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e CANDIDATE_INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb BUILDER_DOUBLE_CHECK=OK CANDIDATE_DOUBLE_CHECK=OK SHARP_MOVES_POSITION=CENTERED SHARP_MOVES_MAX_WIDTH_PX=860 SHARP_MOVES_COLUMNS_DESKTOP=2 SHARP_MOVES_COLUMNS_MOBILE=1 ALERT_RENDERER=V5 ALERT_CURRENCY_SOURCE=TOP_USD_RUB_SWITCH ALERT_EXCHANGE_RATE_SOURCE=STATE_MARKET_USD_RUB ALERT_PRICE_THRESHOLD_CONVERTED=YES ALERT_CAP_THRESHOLD_CONVERTED=YES ALERT_SALE_VALUE_CONVERTED=YES CURRENCY_SWITCH_REDRAW=YES PUMP_DUMP_PERCENT_UNCHANGED=YES FIXED_USD_ALERT_FORMATTER=ABSENT RENDER_MOVES_COUNT=1 ALERT_RENDERER_MARKER_COUNT=1 GRT_REFERENCES=0 PRODUCTION_INDEX_UNCHANGED=YES PRODUCTION_HEALTH=200 READY_FOR_UI_ISOLATED_BROWSER_SMOKE=YES CANDIDATE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-ui-center-alert-currency-candidate-v4/20260713T061123Z
/var/lib/homelab-health/cluster-admin-stage2-design.txt:1:STATUS=OK TS=2026-07-13T00:35:14Z TYPE=cluster-admin-stage2-design BAD=0 STAGE1_CLOSED=YES CAPTURE_REUSED=YES PRODUCTION_CHANGED=EVIDENCE_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO SERVICE_RESTARTED=NO FOUNDATION_STATE=9|7|1|1 EXPECTED_ENTITIES=62 MANAGED_ENTITIES=62 PLACEHOLDERS=21 MISSING_ENTITIES=0 STALE_ENTITIES=0 LEGACY_DEPENDENCIES=26 TYPED_DEPENDENCIES=26 DEPENDENCY_ID_MATCHES=0 DEPENDENCY_NATURAL_KEY_MATCHES=26 DEPENDENCY_NATURAL_KEY_MISSING=0 DEPENDENCY_NATURAL_KEY_STALE=0 EPHEMERAL_ID_MISSING=26 EPHEMERAL_ID_STALE=26 ROOT_CAUSE=LEGACY_DEPENDENCIES_DELETE_REINSERT_EACH_RUN CANONICAL_DEPENDENCY_IDENTITY=PARENT_CHILD_RELATION MIGRATION_ID=002_canonical_foundation_sync DATABASE_FUNCTION=cluster_admin_sync_foundation COLLECTOR_CALL_REQUIRED=YES SAME_TRANSACTION_REQUIRED=YES LEGACY_TABLES_PRESERVED=YES AUTOMATIC_REMEDIATION_ACTIONS=NO STAGE2_READY_FOR_DRY_RUN=YES DESIRED_STATE_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 CHANGE_SET=/var/lib/homelab-change-sets/stage2-design/20260713T003514Z CONTRACT=/var/lib/homelab-change-sets/stage2-design/20260713T003514Z/stage2-contract.json REPORT=/var/lib/homelab-change-sets/stage2-design/20260713T003514Z/stage2-design-report.txt
/var/lib/homelab-health/final-green-freeze-seal.txt:1:STATUS=OK TS=2026-07-08T08:13:08Z TYPE=final-green-freeze-seal BAD=0 WARN=0 MODE=FINAL_FREEZE_EVIDENCE_SEAL FREEZE_SEAL_OK=YES AUDIT_STATUS=OK AUDIT_FREEZE_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 EXACT_WARN_REVIEW_FILES=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T072711Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T072711Z DIRTY_COUNT=0 EVIDENCE=/var/lib/homelab-final-green-freeze/freeze-evidence-latest.tsv REPORT=/var/lib/homelab-final-green-freeze/final-seal-latest.md
/var/lib/homelab-health/cluster-admin-stage2-dry-run.txt:1:STATUS=OK TS=2026-07-13T00:39:40Z TYPE=cluster-admin-stage2-dry-run BAD=0 PRODUCTION_CHANGED=TEST_DATABASE_AND_CANDIDATE_ARTIFACTS_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO MIGRATION_ID=002_canonical_foundation_sync DATABASE_FUNCTION=cluster_admin_sync_foundation MIGRATION_APPLY_TEST=OK MIGRATION_IDEMPOTENCE_TEST=OK NATURAL_KEY_CONVERGENCE=OK EPHEMERAL_ID_REFRESH=OK ENTITY_ADD_SYNC=OK ENTITY_RETIRE_SYNC=OK PLACEHOLDER_ADD_SYNC=OK PLACEHOLDER_RETIRE_SYNC=OK STALE_TYPED_EDGE_DELETE=OK SOURCE_INSTANCE_UPDATE=OK COLLECTOR_CANDIDATE_SYNTAX=OK COLLECTOR_CALL_BEFORE_COMMIT=OK ROLLBACK_TEST=OK TEST_DATABASE_DROPPED=YES PRODUCTION_UNCHANGED=YES READY_FOR_PREAPPLY_BACKUP=YES DESIRED_STATE_HEAD=4d0d03395d2baf4024ad7c9d5870f21bdaee9f99 CHANGE_SET=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z MIGRATION=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z/002_canonical_foundation_sync.sql ROLLBACK=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z/002_canonical_foundation_sync.rollback.sql COLLECTOR=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z/collector.py.candidate PLAN=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z/stage2-migration-plan.json RESULT=/var/lib/homelab-change-sets/stage2-dry-run/20260713T003940Z/dry-run-result.txt
/var/lib/homelab-health/cluster-admin-stage3-production-verify.txt:1:STATUS=OK TS=2026-07-13T03:28:52Z TYPE=cluster-admin-stage3-production-verify BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=VERIFICATION_EVIDENCE_ONLY PRODUCTION_DATABASE_CHANGED=NO LIVE_APPLICATION_CODE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO PREVIOUS_PERMISSION_FAILURE=CORRECTED VERIFIER_OWNER=CLUSTERADMIN VERIFIER_MODE=0700 LIVE_APP_SHA256=a330a2d33ee5a8a57778e3541179e3faebcc0531600a504df744435e80a93b34 DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 AUTHENTICATED_PRODUCTION_TESTS=25 AUTHENTICATED_PRODUCTION_FAILED=0 LEGACY_AUTHENTICATED_SUMMARY=OK NEW_GET_ROUTES_AUTHENTICATED=OK NEW_GET_ROUTES_UNAUTHENTICATED_401=OK ENTITY_LIST=OK ENTITY_DETAIL=OK ENTITY_DEPENDENCIES=OK DEPENDENCIES=OK SOURCES=OK POLICIES=OK COLLECTOR_RUNS=OK VALIDATION_ERRORS=OK FORBIDDEN_RESPONSE_FIELDS=ABSENT OPENAPI_DOCS_EXPOSURE_CHANGED=NO SECURITY_HEADERS=OK PRODUCTION_RUNTIME_HEALTH=OK REMOTE_VERIFIER_CLEANUP=OK READY_FOR_STAGE3_POSTAPPLY_SEAL=YES STAGE3_CLOSED=NO RESULT=/var/lib/homelab-change-sets/stage3-api-production-verify-run/20260713T032852Z/production-result.json CHANGE_SET=/var/lib/homelab-change-sets/stage3-api-production-verify-run/20260713T032852Z
/var/lib/homelab-health/crypto-portfolio-ui-center-alert-currency-browser-smoke.txt:1:STATUS=OK TS=2026-07-13T07:04:45Z TYPE=crypto-portfolio-ui-center-alert-currency-browser-smoke BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=UNPRIVILEGED_REMOTE_RM_AFTER_SUDO_HARNESS PRODUCTION_CHANGED=NO_BROWSER_EVIDENCE_FINALIZATION_ONLY PRODUCTION_DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru CANDIDATE_INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb CANDIDATE_STATIC_GRT_DOUBLE_CHECK=OK BROWSER_ENGINE=CHROMIUM BROWSER_RUNS=2 BROWSER_RUNS_IDENTICAL=YES MOVES=2 SHARP_MOVES_CENTERED=YES SHARP_MOVES_MAX_WIDTH_OK=YES SHARP_MOVES_COLUMNS=2 ALERT_ROWS=2 USD_ALERT_RENDER=OK RUB_ALERT_RENDER=OK TOP_CURRENCY_SWITCH_REDRAW=OK RUB_PRICE_THRESHOLD=OK RUB_CAP_THRESHOLD=OK RUB_SALE_VALUE=OK ALERT_USD_RESIDUE_IN_RUB_MODE=ABSENT CHART_CARDS=2 CHART_CANVASES=14 CHARTS_BOTTOM=YES BROWSER_ERRORS=0 GRT_VISIBLE=NO RESULT_SHA256=b566f492cd80c8f99bc6b24cbef6d5f326c04f4d7168359d05eb1f189589e733 SCREENSHOT_SHA256=68a9a68d15673ef64c248a4fe886ac36f94923ec917daa2772fc9e7c1b5f4c16 ROOT_OWNED_STAGE_ENTRIES=11 ROOT_OWNED_STAGE_DIRECTORIES=2 REMOTE_STAGE_CLEANED=YES PRODUCTION_INDEX_UNCHANGED=YES PRODUCTION_HEALTH=200 VM170_FAILED_UNITS=0 PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES READY_FOR_UI_PREAPPLY_BACKUP_AND_PRODUCTION_APPLY=YES BROWSER_SMOKE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-ui-center-alert-currency-browser-smoke-v2/20260713T070125Z
/var/lib/homelab-health/crypto-portfolio-remove-grt-surface.txt:1:STATUS=OK TS=2026-07-13T04:05:42Z TYPE=crypto-portfolio-remove-grt-surface BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PRODUCTION_CHANGED=NO DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru TARGET_STACK=/opt/stacks/crypto-portfolio DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 DESIRED_STATE_CLEAN=YES APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 APP_GRT_REFERENCES=5 WORKER_GRT_REFERENCES=2 INDEX_GRT_REFERENCES=2 LOCAL_GRT_REFERENCES=3 SQLITE_SCHEMA_CAPTURED=YES READY_FOR_GRT_CANDIDATE_PREPARE=YES SURFACE_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-surface/20260713T040542Z
/var/lib/homelab-health/cluster-admin-stage4d-adapter-candidates-in-isolation.txt:1:STATUS=OK TS=2026-07-14T06:14:58Z TYPE=cluster-admin-stage4d-adapter-candidates-in-isolation-final-seal BAD=0 STAGE4D_CLOSED=YES NEXT_STAGE=4E_RESTRICTED_PROBE_AGENT_DESIGN PRODUCTION_CHANGED=NO DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO APPLICATION_CHANGED=NO ENVELOPE_TOTAL=48 PROMETHEUS_ENVELOPES=5 BACKUP_MATRIX_ENVELOPES=40 HEALTH_ENVELOPES=3 LOCAL_REMOTE_OUTPUT_MATCH=OK PRODUCTION_STATE=0|0|OK DESIRED_STATE_HEAD=e77fb37a0fa0033fe46558045e31dd92b6d1238a DESIRED_STATE_CLEAN=YES PVE01_FAILED_UNITS=0 REPORT=/var/lib/homelab-cluster-admin-stage4/20260714T061458Z/stage4d-adapter-candidates-in-isolation-seal.txt REPORT_SHA256=da6244c0a8a458b508fd58a0fec9cb1e6fa8da98a36f44d1368cbe947f4b5835
/var/lib/homelab-health/backup-matrix-cluster-desired-state-sync.txt:1:STATUS=OK TS=2026-07-13T17:51:31Z TYPE=backup-matrix-cluster-desired-state-sync BAD=0 PRE_SYNC_HEAD=af26ec3c4829b1330d49656205ac580ce1706263 GENERATED_SYNC_HEAD=597119391e7b265b85fd9a76644e4e33af47e469 GENERATED_SYNC_MODE=NEW_GENERATED_COMMIT GENERATED_SYNC_COMMIT_DELTA=1 MATRIX_STATUS=OK JOBS_TOTAL=35 JOBS_OK=31 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 CLUSTER_PASSPORT=OK FINAL_READINESS=OK OVERALL=OK CLUSTER_ADMIN=OK PROBE_OK=4 PROBE_BAD=0 FAILED_UNITS=0 DESIRED_STATE_DIRTY=0 MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 MATRIX_STATE_SHA256=14a31112aac5d577ae01df567921b093b2fedcf7189c4cc8f009216ee0749380 PRESEAL_SHA256=7ea1f710cd311bd51dba05a313b4c540f57d97407f008881e502fc213b4b3c99 BACKUP_RUN=NO RESTORE_RUN=NO CLOUD_WRITE_RUN=NO RETENTION_RUN=NO EVIDENCE_ROOT=/var/lib/homelab-change-sets/backup-matrix-cluster-final-seal/20260713T175131Z
/var/lib/homelab-health/cluster-admin-stage2-recovery.txt:1:STATUS=OK TS=2026-07-13T01:04:59Z TYPE=cluster-admin-stage2-recovery BAD=0 RESULT=RECOVERED PRODUCTION_CHANGED=HEALTH_REGENERATION_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO INCIDENT_ENGINE_RESTARTED=NO DATABASE_ROLLBACK=OK COLLECTOR_ROLLBACK=OK CORE_DATABASE_STATE=0|0|9|7|1 LIVE_COLLECTOR_SHA256=a83abde2290d8392b1c631122671bcb680f9271d0b576fafacb8876d3c8034f1 OBSERVER=OK FULL_OBSERVER=OK RESTRICTED_PROBES=OK RESTRICTED_PROBES_AUTHORITY=CENTRAL_SYNTHESIZED_FROM_REMOTE_FULL_OBSERVER VERIFIER_BUG=REMOTE_RESTRICTED_PROBES_FILE_INCORRECTLY_REQUIRED FINAL_READINESS=OK CLUSTER_PASSPORT=OK OVERALL=OK DUTY_ADMIN=OK VM180_FAILED_UNITS=0 PVE01_FAILED_UNITS=0 DESIRED_STATE_UNCHANGED=YES ROLLBACK_POINT=VERIFIED READY_FOR_STAGE2_RETRY=YES STAGE2_CLOSED=NO CHANGE_SET=/var/lib/homelab-change-sets/stage2-recovery-final/20260713T010459Z
/var/lib/homelab-health/cluster-admin-stage3-api-runtime-dry-run.txt:1:STATUS=OK TS=2026-07-13T02:53:34Z TYPE=cluster-admin-stage3-api-runtime-dry-run BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=TEMPORARY_ISOLATED_RUNTIME_AND_EVIDENCE_ONLY PRODUCTION_DATABASE_CHANGED=NO PRODUCTION_SCHEMA_CHANGED=NO LIVE_APPLICATION_CODE_CHANGED=NO DESIRED_STATE_CHANGED=NO PRODUCTION_SERVICE_RESTARTED=NO CLOUD_WRITE_RUN=NO APP_PRECHANGE_SHA256=c2748b8f96df842f491b5d2790790594222549853d0e2336cf60ecd6d4de0020 APP_CANDIDATE_SHA256=672d6a2975dd0e093788e7eb81a76e7f901ea6374ca4092a25ba0091aba1cb50 ISOLATED_BIND=127.0.0.1:18081 TEMPORARY_DATABASE_RESTORED=YES TEMPORARY_DATABASE_DROPPED=YES TEMPORARY_DATABASE_UNCHANGED=YES RUNTIME_TESTS=41 RUNTIME_FAILED_TESTS=0 EXISTING_ROUTES_BACKWARD_COMPATIBLE=YES NEW_GET_ROUTES=7 UNAUTHENTICATED_NEW_ROUTES_401=YES AUTHENTICATED_NEW_ROUTES=OK PAGINATION=OK FILTER_VALIDATION=OK ENTITY_DETAIL=OK ENTITY_DEPENDENCIES=OK SOURCES=OK POLICIES=OK COLLECTOR_RUNS=OK DATABASE_TRANSACTION_MODE=BEGIN_READ_ONLY FORBIDDEN_RESPONSE_FIELDS=ABSENT OPENAPI_DOCS_EXPOSURE_CHANGED=NO SECURITY_HEADERS=OK ISOLATED_RUNTIME_STOPPED=YES ISOLATED_PORT_CLOSED=YES LIVE_APP_UNCHANGED=YES PRODUCTION_RUNTIME_HEALTH=OK READY_FOR_STAGE3_PREAPPLY_BACKUP=YES DESIRED_STATE_HEAD=8f888e5f2063e49da918b31584c9ea4bcf53af3a CHANGE_SET=/var/lib/homelab-change-sets/stage3-api-runtime-dry-run/20260713T025334Z RESULT=/var/lib/homelab-change-sets/stage3-api-runtime-dry-run/20260713T025334Z/runtime-result.json
/var/lib/homelab-health/crypto-portfolio-remove-grt-failed-units-diagnostic.txt:1:STATUS=OK TS=2026-07-13T05:41:00Z TYPE=crypto-portfolio-remove-grt-failed-units-diagnostic BAD=0 COMMAND_PHYSICAL_LINES=1 PRODUCTION_CHANGED=NO_DIAGNOSTIC_ONLY PRODUCTION_DATABASE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO FAILED_STATE_RESET=NO CLOUD_WRITE_RUN=NO PVE01_FAILED_UNIT_COUNT=2 PVE01_FAILED_UNIT_NAMES=homelab-backup-matrix-refresh.service,homelab-cluster-admin-pull.service CRYPTO_PORTFOLIO_FAILED_UNIT_COUNT=0 OTHER_FAILED_UNIT_COUNT=2 CURRENT_CRYPTO_PRECHANGE_HEALTH=YES UNIT_DETAILS_CAPTURED=YES READY_FOR_TARGETED_FAILED_UNIT_DECISION=YES DIAGNOSTIC_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-failed-units-diagnostic/20260713T054100Z
/var/lib/homelab-health/crypto-portfolio-remove-grt-production-apply.txt:1:STATUS=OK TS=2026-07-13T05:47:54Z TYPE=crypto-portfolio-remove-grt-production-apply BAD=0 COMMAND_PHYSICAL_LINES=1 APPLY_REVISION=V3 FIX_UID_READONLY=YES FIX_READINESS_ENDPOINT=YES FIX_ROLLBACK_DATABASE_METADATA=YES PRODUCTION_CHANGED=YES_REMOVE_GRT DESIRED_STATE_CHANGED=NO_BY_THIS_COMMAND PRODUCTION_CONTAINER_RECREATED=YES CLOUD_WRITE_RUN=NO ROLLBACK_ON_FAILURE=ARMED_NOT_NEEDED TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=80763a6c62b94cde4f3293f3b529980f76919d25 PRECHANGE_BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T051015Z ROLLBACK_PATH=/var/lib/homelab-incidents/crypto-remove-grt-prechange-20260713T051013Z APP_SHA256=44df2b25183b7c5d87ff1516ec2d9cd1d169acde3031f7c2aa1fc4e0c7813cd2 WORKER_SHA256=174a14b86014535980d200a76d0057110c7af102c66f16c2e21fc3e74bb2e49e INDEX_SHA256=8bbe88f5f420fe429d479c598e1f3f904521eb166d4707e32e579851e84c205e CHART_HEALTH_SCRIPT_SHA256=c16473f16390eeeff4d4eb701db604093564dd78d0be2d30b4cc8ffeeb22364a ACTIVE_COINS=HBAR,POL GRT_SOURCE_REFERENCES=0 GRT_DATABASE_ROWS=0|0|0 GRT_MARKET_ENTRY=ABSENT GRT_CACHE_ENTRIES=ABSENT DATABASE_MODE=0644 DATABASE_UID=10001 DATABASE_GID=10001 DATABASE_INTEGRITY=ok WORKER_STATUS=OK WORKER_COIN_COUNT=2 PUBLIC_CODES=200|303|401|200|200|401 CHART_HEALTH=OK CHART_SYMBOLS=HBAR,POL CHART_HEALTH_TIMER=ACTIVE VM170_FAILED_UNITS=0 PVE01_UNRELATED_FAILED_UNITS=homelab-backup-matrix-refresh.service,homelab-cluster-admin-pull.service PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES REMOTE_STAGE_CLEANED=YES READY_FOR_GRT_POSTAPPLY_VERIFY_BACKUP_SYNC=YES APPLY_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-production-apply-v3/20260713T054754Z
/var/lib/homelab-health/mail-cloud-restore-drill.txt:1:STATUS=STALE
/var/lib/homelab-health/mail-cloud-restore-drill.txt:20:FRESHNESS=STALE
/var/lib/homelab-health/edge-restore-proofs-offhost.txt:1:STATUS=OK TS=20260624T190040Z TYPE=edge-restore-proofs-offhost DEST_HOST=pve02 DEST_DIR=/mnt/staging/offhost/edge-restore-proofs-from-edge ITEMS=14 OK=14 ERROR=0 BYTES=226024287 SECRET_PRINTED=NO
/var/lib/homelab-health/crypto-portfolio-runtime-final-seal.txt:1:STATUS=OK TS=2026-07-13T08:37:54Z TYPE=crypto-portfolio-ui-center-alert-currency-final-seal BAD=0 DANGLING_IMAGE_REFERENCE_GATE=OK DESIRED_STATE_SYNC_ARTIFACT_GATE=OK APP=crypto-portfolio PRODUCTION_CHANGED=NO_FINAL_SEAL_ONLY APP_SHA256=44df2b25183b7c5d87ff1516ec2d9cd1d169acde3031f7c2aa1fc4e0c7813cd2 WORKER_SHA256=174a14b86014535980d200a76d0057110c7af102c66f16c2e21fc3e74bb2e49e INDEX_SHA256=e497a36d8549654c6d451360a11a2ad714e19c23e6d384330a5436f7d32267cb LIBRARY_SHA256=c0992580867c4912cc9385b3c2728315bcc1a76c7f1087dca908430fccdf31d7 ACTIVE_COINS=HBAR,POL GRT_REMOVED=YES GRT_ROWS=0|0|0 GRT_SOURCE_REFERENCES=0 GRT_CACHE_ENTRIES=0 SHARP_MOVES_CENTERED=YES ALERT_CURRENCY_USD_RUB=OK BROWSER_SMOKE=OK BROWSER_RUNS=2 BROWSER_ERRORS=0 BROWSER_RESULT_SHA256=b566f492cd80c8f99bc6b24cbef6d5f326c04f4d7168359d05eb1f189589e733 SCREENSHOT_SHA256=00b2853bee9c880146d0dd75fac60feb7e46d4c129ee06754047ef30d90bd4ba BACKUP_JOB=homelab-pve01-app-crypto-portfolio-20260713T075801Z TWO_CLOUDS=OK RESTORE_CHECK=OK DATABASE_INTEGRITY=ok RETENTION=OK ROLLBACK_COUNT=3 DANGLING_IMAGES=2 REFERENCED_DANGLING_IMAGES=2 UNREFERENCED_DANGLING_IMAGES=0 PRUNED_UNREFERENCED_DANGLING_IMAGES=0 CHART_HEALTH=OK PUBLIC=OK MONITOR_TIMER=active BACKUP_TIMER=active CLOSURE_DESIRED_STATE_HEAD=2999910549eaff9b9431b9c656d318340eb95cbd DESIRED_STATE=OK PVE01_UNRELATED_FAILED_BASELINE_UNCHANGED=YES ALL_CRYPTO_PORTFOLIO_UI_TAILS_CLOSED=YES
/var/lib/homelab-health/backup-matrix-cluster-preseal-convergence.txt:1:STATUS=OK TS=2026-07-13T17:48:14Z TYPE=backup-matrix-cluster-preseal-convergence BAD=0 MATRIX_STATUS=OK JOBS_TOTAL=35 JOBS_BAD=0 JOBS_UNMAPPED=0 JOBS_DISABLED=4 CLUSTER_PASSPORT=OK FINAL_READINESS=OK OVERALL=OK CLUSTER_ADMIN=OK PROBE_OK=4 PROBE_BAD=0 WEBPANEL=OK FAILED_UNITS=0 DESIRED_STATE_HEAD=af26ec3c4829b1330d49656205ac580ce1706263 DESIRED_STATE_DIRTY=0 MATRIX_SHA256=7613bcb25d32e61fac114da13af81e38354ec22a012a6deeaa168ee4cc454f87 MATRIX_STATE_SHA256=3791fa78cdfb56cd7027f89350dff245abcffc4f935adfc336ceeea00854ac89 CLUSTER_PASSPORT_SHA256=1b8b924ae0e5ce6872fe5671318094ce494806c2db7efe28299b5ff49adc0d0d FINAL_READINESS_SHA256=ac5159fa8c497afe021b5cb232f08cfec0e1a4d516d8aae966ba03f7034f1412 OVERALL_SHA256=6ae529bfdf5811f686d3b53978f453ce4f8757c74fc1be105de219b13a61ba46 BACKUP_RUN=NO RESTORE_RUN=NO CLOUD_WRITE_RUN=NO RETENTION_RUN=NO EVIDENCE_ROOT=/var/lib/homelab-change-sets/backup-matrix-cluster-preseal-resume/20260713T174748Z
/var/lib/homelab-health/uptime-kuma-pending-health-monitors-fix.txt:8:NOTE=Paperless Restore may become red if its health file contains STATUS=ERROR; that is a real restore-health issue, not a Kuma pending issue.
/var/lib/homelab-health/pve03-capacity-forensics.txt:1:STATUS=OK TS=2026-07-08T05:42:29Z TYPE=pve03-capacity-forensics MODE=SUPERSEDED_BY_PVE03_ROOT_STAGING_CLEANUP BAD=0 WARN=0 OLD_STATUS=WARN_ARCHIVED ROOT_CLEANUP=OK STAGING_RETENTION=OK STAGING_CLEANUP=OK FAILED_UNIT_CLEANUP=OK REPORT=/var/lib/homelab-duty-admin-residual-cleanup/pve03-capacity-forensics-closure-latest.md
/var/lib/homelab-health/remote-failed-unit-apply.txt:1:STATUS=OK TS=2026-07-08T05:25:22Z TYPE=remote-failed-unit-apply BAD=0 WARN=0 MODE=APPLY_CLEANUP_PLAN_EXCEPT_PAPERLESS_CORE_GUARD PRECHECK=YES APPLY_OK=YES LOCAL_OVERRIDE_COUNT=4 PVE02_RESET=1 EDGE_APPLY=1 LOCAL_FAILED_AFTER=0 PVE02_FAILED_AFTER=0 EDGE_FAILED_AFTER=1 EDGE_DEFER_ONLY=YES PVE02_CODEVIPE_FAILED=inactive DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 OVERALL_STATUS=OK OVERALL_BAD=0 DEFERRED_UNIT=homelab-paperless-core-guard.service REPORT=/var/lib/homelab-remote-failed-unit-cleanup/apply-latest.md
/var/lib/homelab-health/recovery-card.txt:1:STATUS=OK TS=2026-07-08T08:24:05Z TYPE=recovery-card BAD=0 WARN=0 MODE=READ_ONLY_RECOVERY_CARD_GENERATION RECOVERY_CARD_OK=YES FREEZE_STATUS=OK FREEZE_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T082034Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T082034Z EMERGENCY_JOB=homelab-pve01-emergency-pack-20260708T002739Z VM_DUMPS_JOB=homelab-pve01-vm-local-dumps-20260706T153606Z ENDPOINT_LOCAL_OK=1 ENDPOINT_LAN_OK=1 ENDPOINT_PUBLIC_OK=1 BACKUP_SLA=OK BACKUP_COVERAGE=OK EXTENDED_APPBACKUP=OK VM_BACKUP=OK EXTERNAL_PROBE=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 DIRTY_COUNT=0 CARD=/var/lib/homelab-recovery-card/recovery-card-latest.md SUMMARY=/var/lib/homelab-recovery-card/recovery-summary-latest.tsv REPORT=/var/lib/homelab-recovery-card/recovery-card-latest.md
/var/lib/homelab-health/cluster-admin-stage0-convergence.txt:1:STATUS=WARN TS=2026-07-12T23:12:59Z TYPE=cluster-admin-stage0-targeted-convergence BAD=2 PROFILE_OK=NO CLUSTER_PASSPORT=FAIL FINAL_READINESS=FAIL OVERALL=FAIL DUTY_ADMIN=FAIL FAILED_UNITS=0 LIVE_MATCH_DESIRED_STATE=YES READY_FOR_FINAL_SEAL=NO REPORT_DIR=/var/lib/homelab-cluster-admin-stage0/readiness-profile-20260712T231259Z PRODUCTION_CHANGED=HEALTH_REGENERATION_ONLY PRODUCTION_SCHEMA_CHANGED=NO
/var/lib/homelab-health/cluster-admin-stage4c-database-migration-dry-run.txt:1:STATUS=OK TS=2026-07-14T05:33:26Z TYPE=cluster-admin-stage4c-database-migration-dry-run-final-seal BAD=0 STAGE4C_CLOSED=YES NEXT_STAGE=4D_ADAPTER_CANDIDATES_IN_ISOLATION MIGRATION_SHA256=a0bd6a50c6ad9b90528c2e90e6d9cb4279b09811b559b2504c83d7527b19e578 FINAL_READINESS=OK OVERALL=OK PUBLIC_ENDPOINT=OK WEBPANEL_SYNC=OK PVE01_FAILED_UNITS=0 VM180_FAILED_UNITS=0 DESIRED_STATE_HEAD=df86e4d27a64613b60ae55253b6348866dd34fa9 DESIRED_STATE_CLEAN=YES REPORT=/var/lib/homelab-cluster-admin-stage4/20260714T053326Z/stage4c-database-migration-dry-run-seal.txt REPORT_SHA256=3368c302ca0ee76fab8fbc97caa65adf4cf57e61c5b5cdb9a1f3829a05af2e2a
/var/lib/homelab-health/operator-command-size-guard.txt:1:STATUS=OK TS=2026-07-14T18:26:54Z TYPE=operator-command-size-guard RULE_ACTIVE=YES PARSE_ERROR_TAIL=CLOSED UNRESOLVED_TAIL_COUNT=0 BLOCKER_COUNT=0 REPORT_SHA256=2ed7949dde9d5f8346e46af362e27cb8982d65f00ec602546efab506509e47af
/var/lib/homelab-health/node-loss-final-closure.txt:1:STATUS=OK TS=2026-07-08T16:35:56Z TYPE=node-loss-final-closure BAD=0 MODE=ALL_NODE_LOSS_DRILLS_SEALED PVE03_EDGE_RESTORE=OK PVE03_EDGE_FRESHNESS=OK PVE02_FORUM_RUNTIME=OK PVE01_NEXTCLOUD_DNS=OK PVE01_CORE_MONITORING_APPLEVEL=OK FULL_VM_GAP_VM170_VM171=ACKNOWLEDGED_APP_LEVEL_ONLY TEST_VMS_CLEANED=YES DNS_PAIR_OK=YES FAILED_UNITS=pve01:0,pve02:0,pve03:0,edge:0 DIRTY_COUNT=0 REPORT=/var/lib/homelab-node-loss/final-closure-latest.md
/var/lib/homelab-health/crypto-portfolio-remove-grt-browser-smoke.txt:1:STATUS=OK TS=2026-07-13T04:52:06Z TYPE=crypto-portfolio-remove-grt-browser-smoke BAD=0 COMMAND_PHYSICAL_LINES=1 PREVIOUS_FAILURE=CORRECTED PREVIOUS_FAILURE_ROOT_CAUSE=OPTIONAL_XXD_DEPENDENCY_MISSING_WITH_STDERR_SUPPRESSED BROWSER_RERUN=NO EVIDENCE_SALVAGED=YES PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO CANDIDATE_CHANGED=NO DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO CONTAINER_STARTED=NO CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 INPUT_APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd INPUT_WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INPUT_INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 ACTIVE_COINS=HBAR,POL GRT_VISIBLE=NO BROWSER_CARDS=2 BROWSER_CANVASES=14 BROWSER_MOVES=2 BROWSER_COIN_CARDS=2 BROWSER_POSITION_ROWS=2 BROWSER_ALERT_OPTIONS=2 BROWSER_ACTIVE_COIN=HBAR BROWSER_CHART_SYMBOLS=HBAR,POL BROWSER_ERRORS=0 BROWSER_LOADING_TEXT=ABSENT BROWSER_ERROR_TEXT=ABSENT CHARTS_POSITION=BOTTOM PNG_VALIDATION=PYTHON_STDLIB PNG_SIGNATURE=89504e470d0a1a0a SCREENSHOT_SHA256=3d132b477a382487e34db2f2feaf7e4f43af8d30fd8e1530225782da5d7336b9 RESULT_SHA256=0deadbf38aeef86726035da66eb1991c9494f59172b1c1a5c502c50c2de95c60 LIVE_PRODUCTION_DATA_MOUNTED=NO NPM_INSTALL_RUN=NO DOCKER_PULL_RUN=NO PRODUCTION_SOURCE_HASHES_UNCHANGED=YES PRODUCTION_CONTAINER_IDS_AND_RESTARTS_UNCHANGED=YES PRODUCTION_DATABASE_INTEGRITY=ok PRODUCTION_GRT_STATE_PRESERVED=YES PUBLIC_PRODUCTION_STATE_PRESERVED=YES REMOTE_RESIDUE=NONE READY_FOR_GRT_PRECHANGE_BACKUP=YES BROWSER_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-browser-smoke-salvage/20260713T045206Z
/var/lib/homelab-health/independent-external-runner-audit.txt:1:STATUS=OK TS=2026-07-08T09:13:25Z TYPE=independent-external-runner-audit BAD=0 WARN=0 MODE=READ_ONLY_INDEPENDENT_EXTERNAL_RUNNER_AUDIT AUDIT_OK=YES INDEPENDENT_RUNNER_READY=NO DEPLOY_DECISION=NO_DEPLOY CURRENT_INDEPENDENT_RUNNER_AVAILABLE=no TRUSTED_NETBIRD_IDENTITY=edge-vm PUBLIC_VPS_DECISION=REJECTED_HOSTKEY_MISMATCH PUBLIC_VPS_SSH_OK=0 PUBLIC_VPS_REJECTED_HOSTKEY=YES EDGE_LAN_SSH_OK=2 EDGE_NETBIRD_SSH_OK=2 EDGE_DNS_SSH_OK=1 PUBLIC_OVERALL_OK=1 LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 OVERALL_STATUS=OK OVERALL_BAD=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 CLOSEOUT_OK=YES RECOVERY_SEAL_OK=YES DIRTY_COUNT=0 REPORT=/var/lib/homelab-independent-external-runner/audit-latest.md
/var/lib/homelab-health/sops-edge-secret-coverage-closure.txt:1:STATUS=OK TS=2026-07-08T05:01:47Z TYPE=sops-edge-secret-coverage-closure BAD=0 WARN=0 MODE=POLICY_SKELETON_SUPERSEDED_NOOP PRECHECK=YES POLICY_MODE=POLICY_SKELETON_NOT_ACTIVE AUDIT_CLASS=UNGATED_FAILED_STATE_REVIEW IN_HARD_GATES=0 OVERRIDE_WRITTEN=1 FAILED_UNITS_AFTER=0 UNIT_FAILED_AFTER=inactive MARKER_STATUS=OK DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 DIRTY_COUNT=0 SOPS_CLOSURE_OK=YES REPORT=/var/lib/homelab-failed-unit-legacy-cleanup/sops-edge-secret-coverage-closure-latest.md
/var/lib/homelab-health/mail-cloud-canary.txt:1:STATUS=OK TS=20260618T174444Z TYPE=mail-cloud-canary REMOTE_TOTAL=13 REMOTE_OK=13 REMOTE_FAIL=0 DEST=restore-test/p1-canary/20260618T174444Z SHA256=a6bf6a1493a2556b92c8e6547163500345d4e0e32688937370b942b881c47182 SECRET_PRINTED=no NOTES=none
/var/lib/homelab-health/filebrowser-restore.txt:1:STATUS=STALE
/var/lib/homelab-health/filebrowser-restore.txt:20:FRESHNESS=STALE
/var/lib/homelab-health/evidence-seal/latest.env:6:TOTAL_FAILURES=0
/var/lib/homelab-health/final-green-freeze-audit.txt:1:STATUS=OK TS=2026-07-08T07:45:35Z TYPE=final-green-freeze-audit BAD=0 WARN=0 MODE=READ_ONLY_FREEZE_EVIDENCE_AUDIT FREEZE_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=0 DUTY_V2_LEGACY_OR_PHASE_REVIEW=0 OVERALL_STATUS=OK OVERALL_BAD=0 EXACT_WARN_REVIEW_FILES=0 LOCAL_OVERALL_OK=1 LAN_OVERALL_OK=1 PUBLIC_OVERALL_OK=1 RESTORE_CHECK_OK=YES GOLDEN_MATCH=YES LATEST_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T072711Z GOLDEN_CONTROL_PLANE=homelab-pve01-app-safety-net-control-plane-20260708T072711Z BACKUP_SLA=OK BACKUP_COVERAGE=OK EXTENDED_APPBACKUP=OK VM_BACKUP=OK SECRET_EXPOSURE=OK REVIEW_SURFACE_REGISTRY=YES FINAL_GLOBAL_GREEN=YES EXTERNAL_PROBE=OK EXTERNAL_PUBLIC_SMOKE_BAD=0 DIRTY_COUNT=0 EVIDENCE=/var/lib/homelab-final-green-freeze/freeze-evidence-latest.tsv REPORT=/var/lib/homelab-final-green-freeze/freeze-audit-latest.md
/var/lib/homelab-health/crypto-portfolio-remove-grt-runtime-dry-run.txt:1:STATUS=OK TS=2026-07-13T04:41:24Z TYPE=crypto-portfolio-remove-grt-runtime-dry-run BAD=0 COMMAND_PHYSICAL_LINES=1 VERIFIER_REVISION=V3 ANALYSIS_SOURCES=FULL_CHAT_CRYPTO_HANDOFF_CLUSTER_REFERENCE_ERROR_HISTORY PRODUCTION_CHANGED=NO PRODUCTION_DATABASE_CHANGED=NO CANDIDATE_DATABASE_CHANGED=ISOLATED_COPY_RUNTIME_CHECK_ONLY DESIRED_STATE_CHANGED=NO SERVICE_RESTARTED=NO PRODUCTION_CONTAINER_RECREATED=NO ISOLATED_CONTAINER_STARTED=YES ISOLATED_CONTAINER_CLEANED=YES CLOUD_WRITE_RUN=NO TARGET_HOST=core-apps.gram1.ru DESIRED_STATE_HEAD=b974057b536ece861299d0b230c06cc0e6702c39 INPUT_APP_SHA256=684ad884119554503db0ed36252aeb017c0c04b54729f30c51e66a7ab293f8bd INPUT_WORKER_SHA256=40a1c81111d7097120125b140683a8604abcb4051db5073ab8af708a11854038 INPUT_INDEX_SHA256=2b1ab459cabf3050af1db4a7b293fff7c2a9ac18bf4a14faa1efc70a3a3f63e3 ACTIVE_COINS=HBAR,POL CANDIDATE_GRT_REFERENCES=0 DATABASE_GRT_AFTER=0|0|0 WORKER_RUNTIME_CHECK=OK WEB_RUNTIME_CHECK=OK HTTP_RUNTIME_CHECK=OK HTTP_CODES=200|303|401|200|200|401 CHART_HEALTH=OK CHART_SYMBOLS=HBAR,POL ROUTE_RETURN_DECODER=DICT_OR_RESPONSE_BODY LIVE_PRODUCTION_DATA_MOUNTED=NO PRODUCTION_SOURCE_HASHES_UNCHANGED=YES PRODUCTION_CONTAINER_IDS_AND_RESTARTS_UNCHANGED=YES PRODUCTION_DATABASE_INTEGRITY=ok PRODUCTION_GRT_STATE_PRESERVED=YES PUBLIC_PRODUCTION_STATE_PRESERVED=YES REMOTE_RESIDUE=NONE BROWSER_SMOKE=NOT_RUN READY_FOR_GRT_ISOLATED_BROWSER_SMOKE=YES RUNTIME_ROOT=/var/lib/homelab-change-sets/crypto-portfolio-remove-grt-runtime-run-v3/20260713T044124Z
/var/lib/homelab-health/sops-edge-secret-coverage-audit.txt:1:STATUS=OK TS=2026-07-08T04:59:58Z TYPE=sops-edge-secret-coverage-audit BAD=0 WARN=1 MODE=READ_ONLY_SOPS_EDGE_SECRET_COVERAGE_AUDIT UNIT_FAILED=failed CLASS=UNGATED_FAILED_STATE_REVIEW IN_HARD_GATES=no OVERALL_BAD=0 FINAL_FAIL=0 DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=1 RELATED_WARN_OR_ERROR_HEALTH=3 RESET_FAILED_DONE=no REPORT=/var/lib/homelab-failed-unit-legacy-cleanup/sops-edge-secret-coverage-audit-latest.md
/var/lib/homelab-health/duty-admin-residual-cleanup-seal.txt:1:STATUS=OK TS=2026-07-08T05:44:22Z TYPE=duty-admin-residual-cleanup-seal BAD=0 WARN=0 MODE=FINAL_RESIDUAL_NOISE_SEAL RESIDUAL_SEAL_OK=YES LOCAL_FAILED=0 PVE02_FAILED=0 PVE03_FAILED=0 EDGE_FAILED=0 DUTY_V2_STATUS=OK DUTY_V2_GATED_NONOK=0 DUTY_V2_UNGATED_ERROR_REVIEW=0 DUTY_V2_UNGATED_WARN_REVIEW=0 DUTY_V2_UNGATED_REVIEW_ITEM=22 DUTY_V2_LEGACY_OR_PHASE_REVIEW=5 REVIEW_ITEMS_INTENTIONALLY_LEFT=yes LEGACY_PHASE_ITEMS_INTENTIONALLY_LEFT=yes FINAL_STATUS=OK FINAL_FAIL=0 CLUSTER_STATUS=OK CLUSTER_BAD=0 OVERALL_STATUS=OK OVERALL_BAD=0 PVE03_CAPACITY_FORENSICS_STATUS=OK LOCAL_CLEANUP_OK=YES REMOTE_CLEANUP_OK=YES DIRTY_COUNT=0 REPORT=/var/lib/homelab-duty-admin-residual-cleanup/seal-latest.md
/var/lib/homelab-health/prometheus-npmplus-kuma-supersede-repair-20260709T073019Z/npmplus-kuma-config-backup.txt.bak:1:STATUS=ERROR TS=20260709T072619Z TYPE=npmplus-kuma-config-backup NOTES=edge_backup_failed
--- FILESYSTEMS ---
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 15G 0 15G 0% /dev
tmpfs tmpfs 3.2G 3.8M 3.2G 1% /run
/dev/mapper/pve-root ext4 98G 40G 53G 44% /
tmpfs tmpfs 16G 66M 16G 1% /dev/shm
efivarfs efivarfs 192K 135K 53K 72% /sys/firmware/efi/efivars
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-journald.service
tmpfs tmpfs 16G 47M 16G 1% /tmp
/dev/nvme0n1p2 vfat 1022M 9.1M 1013M 1% /boot/efi
/dev/mapper/pve-homelab--staging ext4 295G 148G 133G 53% /mnt/staging
/dev/fuse fuse 128M 1.4M 127M 2% /etc/pve
shm tmpfs 63M 0 63M 0% /var/lib/containers/storage/overlay-containers/fbb240ea5b547fa140d8edbc3b16daaf4b28a21c7ee424abedab068865a95d5b/userdata/shm
overlay overlay 98G 40G 53G 44% /var/lib/containers/storage/overlay/4f68ccfa2802b35db2a4c6b396e477f756cb85aa42fa818dd111745ec503616b/merged
tmpfs tmpfs 1.0M 0 1.0M 0% /run/credentials/getty@tty1.service
tmpfs tmpfs 3.2G 4.0K 3.2G 1% /run/user/0
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 3.6M 733 3.6M 1% /dev
tmpfs 3.9M 1.4K 3.9M 1% /run
/dev/mapper/pve-root 6.3M 128K 6.2M 2% /
tmpfs 3.9M 137 3.9M 1% /dev/shm
efivarfs 0 0 0 - /sys/firmware/efi/efivars
tmpfs 3.9M 32 3.9M 1% /run/lock
tmpfs 1.0K 1 1023 1% /run/credentials/systemd-journald.service
tmpfs 1.0M 3.5K 1021K 1% /tmp
/dev/nvme0n1p2 0 0 0 - /boot/efi
/dev/mapper/pve-homelab--staging 19M 4.3K 19M 1% /mnt/staging
/dev/fuse 256K 89 256K 1% /etc/pve
shm 3.9M 1 3.9M 1% /var/lib/containers/storage/overlay-containers/fbb240ea5b547fa140d8edbc3b16daaf4b28a21c7ee424abedab068865a95d5b/userdata/shm
overlay 6.3M 128K 6.2M 2% /var/lib/containers/storage/overlay/4f68ccfa2802b35db2a4c6b396e477f756cb85aa42fa818dd111745ec503616b/merged
tmpfs 1.0K 1 1023 1% /run/credentials/getty@tty1.service
tmpfs 796K 28 796K 1% /run/user/0
total used free shared buff/cache available
Mem: 31Gi 20Gi 9.1Gi 106Mi 2.4Gi 10Gi
Swap: 9Gi 1.7Gi 8.3Gi
16:21:11 up 5 days, 4:42, 3 users, load average: 0.26, 0.37, 0.42
--- STORAGE_HEALTH ---
no pools available
Personalities :
unused devices: <none>
--- LOCAL_BACKUP_RECHECK ---
7022386770 /var/lib/vz/dump/vzdump-qemu-150-2026_07_18-03_30_13.vma.zst
7021952392 /var/lib/vz/dump/vzdump-qemu-150-2026_07_21-03_30_16.vma.zst
7021297699 /var/lib/vz/dump/vzdump-qemu-150-2026_07_19-03_30_16.vma.zst
7016275481 /var/lib/vz/dump/vzdump-qemu-150-2026_07_20-03_30_14.vma.zst
329541843 /var/lib/vz/dump/vzdump-lxc-110-2026_07_21-03_30_02.tar.zst
326601118 /var/lib/vz/dump/vzdump-lxc-110-2026_07_20-03_30_00.tar.zst
323668231 /var/lib/vz/dump/vzdump-lxc-110-2026_07_19-03_45_38.tar.zst
321121649 /var/lib/vz/dump/vzdump-lxc-110-2026_07_18-03_30_00.tar.zst
222777732 /var/lib/vz/dump/vzdump-lxc-112-2026_07_21-03_30_10.tar.zst
222761273 /var/lib/vz/dump/vzdump-lxc-112-2026_07_20-03_30_08.tar.zst
222745965 /var/lib/vz/dump/vzdump-lxc-112-2026_07_19-03_45_47.tar.zst
222727760 /var/lib/vz/dump/vzdump-lxc-112-2026_07_18-03_30_08.tar.zst
3021 /var/lib/vz/dump/vzdump-qemu-150-2026_07_19-03_30_16.log
3020 /var/lib/vz/dump/vzdump-qemu-150-2026_07_21-03_30_16.log
3018 /var/lib/vz/dump/vzdump-qemu-150-2026_07_18-03_30_13.log
3017 /var/lib/vz/dump/vzdump-qemu-150-2026_07_20-03_30_14.log
1089 /var/lib/vz/dump/vzdump-lxc-112-2026_07_20-03_30_08.log
1086 /var/lib/vz/dump/vzdump-lxc-110-2026_07_20-03_30_00.log
989 /var/lib/vz/dump/vzdump-lxc-112-2026_07_21-03_30_10.log
989 /var/lib/vz/dump/vzdump-lxc-112-2026_07_18-03_30_08.log
986 /var/lib/vz/dump/vzdump-lxc-110-2026_07_21-03_30_02.log
986 /var/lib/vz/dump/vzdump-lxc-110-2026_07_18-03_30_00.log
731 /var/lib/vz/dump/vzdump-lxc-112-2026_07_19-03_45_47.log
729 /var/lib/vz/dump/vzdump-lxc-110-2026_07_19-03_45_38.log
========== NODE=pve02 ==========
--- PVE_STATUS ---
{
"status": null,
"uptime": 448968,
"cpu": 0,
"loadavg": [
"0.01",
"0.04",
"0.00"
],
"memory": {
"available": 4139548672,
"free": 2858442752,
"total": 16536748032,
"used": 12397199360
},
"swap": {
"free": 8850722816,
"total": 10737414144,
"used": 1886691328
},
"rootfs": {
"avail": 71985065984,
"free": 77370552320,
"total": 105089261568,
"used": 27718709248
},
"kversion": "Linux 7.0.12-1-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.12-1 (2026-06-09T21:07Z)",
"pveversion": "pve-manager/9.2.3/d0fde103346cf89a"
}
--- FAILED_UNITS ---
--- TIMERS ---
Tue 2026-07-21 16:26:07 MSK 4min 53s Tue 2026-07-21 16:21:07 MSK 6s ago homelab-smartctl-textfile.timer homelab-smartctl-textfile.service
Tue 2026-07-21 16:27:30 MSK 6min Tue 2026-07-21 16:12:30 MSK 8min ago prometheus-node-exporter-smartmon.timer prometheus-node-exporter-smartmon.service
Tue 2026-07-21 18:41:03 MSK 2h 19min Tue 2026-07-21 12:41:22 MSK 3h 39min ago forum-prod-fresh5-cloud-backup.timer forum-prod-fresh5-cloud-backup.service
Wed 2026-07-22 00:00:00 MSK 7h Tue 2026-07-21 00:00:22 MSK 16h ago dpkg-db-backup.timer dpkg-db-backup.service
Wed 2026-07-22 04:20:00 MSK 11h Tue 2026-07-21 04:20:12 MSK 12h ago homelab-cluster-admin-vm-mail-cloud-backup.timer homelab-cluster-admin-vm-mail-cloud-backup.service
Sun 2026-07-26 03:10:05 MSK 4 days Sun 2026-07-19 03:10:22 MSK 2 days ago xfs_scrub_all.timer xfs_scrub_all.service
Sun 2026-07-26 03:10:26 MSK 4 days Sun 2026-07-19 03:10:42 MSK 2 days ago e2scrub_all.timer e2scrub_all.service
Mon 2026-07-27 01:01:11 MSK 5 days Mon 2026-07-20 00:30:07 MSK 1 day 15h ago fstrim.timer fstrim.service
Tue 2026-07-28 08:02:43 MSK 6 days Tue 2026-07-21 08:02:30 MSK 8h ago homelab-vaultwarden-isolated-restore-drill.timer homelab-vaultwarden-isolated-restore-drill.service
--- HEALTH_FILES ---
2026-07-08T08:25:22.7370010450 217 /var/lib/homelab-health/codevipe-cloud-backup-reset.txt
2026-07-12T01:29:08.8140509520 108 /var/lib/homelab-health/routerguard-hard-power-cycle.txt
2026-07-14T06:42:03.4801032090 108 /var/lib/homelab-health/routerguard-ap.txt.tmp.3936294
2026-07-19T02:59:29.0591803690 108 /var/lib/homelab-health/routerguard-ap.txt.tmp.1662197
2026-07-19T02:59:46.2495844810 108 /var/lib/homelab-health/routerguard-ap.txt.tmp.1662387
2026-07-21T04:21:29.8655959860 370 /var/lib/homelab-health/cluster-admin-vm-mail-cloud-backup-local.txt
2026-07-21T08:03:00.2596892130 770 /var/lib/homelab-health/vaultwarden-isolated-restore-drill.txt
2026-07-21T16:21:07.1497113500 108 /var/lib/homelab-health/routerguard-ap.txt
/var/lib/homelab-health/codevipe-cloud-backup-reset.txt:1:STATUS=OK TS=2026-07-08T05:25:22Z TYPE=remote-failed-unit-codevipe-reset BAD=0 WARN=0 MODE=RESET_STALE_FAILED_STATE UNIT=codevipe-cloud-backup.service REASON=last_journal_success_and_no_timer ACTION=reset_failed_only
--- FILESYSTEMS ---
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 6.8G 0 6.8G 0% /dev
tmpfs tmpfs 1.6G 3.1M 1.6G 1% /run
/dev/mapper/pve-root ext4 98G 26G 68G 28% /
tmpfs tmpfs 7.8G 66M 7.7G 1% /dev/shm
efivarfs efivarfs 192K 93K 95K 50% /sys/firmware/efi/efivars
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-journald.service
/dev/sda2 vfat 1022M 9.1M 1013M 1% /boot/efi
tmpfs tmpfs 7.8G 36K 7.8G 1% /tmp
/dev/mapper/pve-homelab--staging ext4 147G 103G 37G 74% /mnt/staging
/dev/fuse fuse 128M 1.4M 127M 2% /etc/pve
tmpfs tmpfs 1.0M 0 1.0M 0% /run/credentials/getty@tty1.service
tmpfs tmpfs 1.6G 4.0K 1.6G 1% /run/user/0
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 1.7M 653 1.7M 1% /dev
tmpfs 2.0M 1.3K 2.0M 1% /run
/dev/mapper/pve-root 6.3M 80K 6.2M 2% /
tmpfs 2.0M 136 2.0M 1% /dev/shm
efivarfs 0 0 0 - /sys/firmware/efi/efivars
tmpfs 2.0M 22 2.0M 1% /run/lock
tmpfs 1.0K 1 1023 1% /run/credentials/systemd-journald.service
/dev/sda2 0 0 0 - /boot/efi
tmpfs 1.0M 326 1.0M 1% /tmp
/dev/mapper/pve-homelab--staging 9.4M 1.4K 9.4M 1% /mnt/staging
/dev/fuse 256K 89 256K 1% /etc/pve
tmpfs 1.0K 1 1023 1% /run/credentials/getty@tty1.service
tmpfs 395K 26 395K 1% /run/user/0
total used free shared buff/cache available
Mem: 15Gi 11Gi 2.8Gi 57Mi 1.6Gi 4.0Gi
Swap: 9Gi 1.8Gi 8.2Gi
16:21:13 up 5 days, 4:42, 1 user, load average: 0.01, 0.04, 0.00
--- STORAGE_HEALTH ---
no pools available
Personalities :
unused devices: <none>
--- LOCAL_BACKUP_RECHECK ---
997503422 /var/lib/vz/dump/vzdump-qemu-160-2026_07_01-03_30_27.vma.zst
868048369 /var/lib/vz/dump/vzdump-qemu-160-2026_06_30-03_30_25.vma.zst
288349157 /var/lib/vz/dump/vzdump-lxc-111-2026_07_21-03_30_04.tar.zst
286538526 /var/lib/vz/dump/vzdump-lxc-111-2026_07_20-03_30_02.tar.zst
284443608 /var/lib/vz/dump/vzdump-lxc-111-2026_07_19-03_47_26.tar.zst
283022034 /var/lib/vz/dump/vzdump-lxc-111-2026_07_18-03_30_03.tar.zst
220080970 /var/lib/vz/dump/vzdump-lxc-113-2026_07_21-03_30_21.tar.zst
220065825 /var/lib/vz/dump/vzdump-lxc-113-2026_07_20-03_30_18.tar.zst
220049041 /var/lib/vz/dump/vzdump-lxc-113-2026_07_19-03_47_43.tar.zst
220033534 /var/lib/vz/dump/vzdump-lxc-113-2026_07_18-03_30_20.tar.zst
7547 /var/lib/vz/dump/vzdump-qemu-160-2026_07_18-01_02_44.log
7443 /var/lib/vz/dump/vzdump-qemu-160-2026_07_18-01_38_20.log
5796 /var/lib/vz/dump/vzdump-qemu-160-2026_07_17-21_30_20.log
2685 /var/lib/vz/dump/vzdump-qemu-160-2026_06_30-03_30_25.log
2590 /var/lib/vz/dump/vzdump-qemu-160-2026_07_01-03_30_27.log
1088 /var/lib/vz/dump/vzdump-lxc-113-2026_07_20-03_30_18.log
1085 /var/lib/vz/dump/vzdump-lxc-111-2026_07_20-03_30_02.log
988 /var/lib/vz/dump/vzdump-lxc-113-2026_07_21-03_30_21.log
988 /var/lib/vz/dump/vzdump-lxc-113-2026_07_18-03_30_20.log
985 /var/lib/vz/dump/vzdump-lxc-111-2026_07_21-03_30_04.log
985 /var/lib/vz/dump/vzdump-lxc-111-2026_07_18-03_30_03.log
731 /var/lib/vz/dump/vzdump-lxc-113-2026_07_19-03_47_43.log
728 /var/lib/vz/dump/vzdump-lxc-111-2026_07_19-03_47_26.log
========== NODE=pve03 ==========
--- PVE_STATUS ---
{
"status": null,
"uptime": 448818,
"cpu": 0,
"loadavg": [
"0.54",
"0.48",
"0.44"
],
"memory": {
"available": 19315478528,
"free": 6366478336,
"total": 33553518592,
"used": 14238040064
},
"swap": {
"free": 7508238336,
"total": 10737414144,
"used": 3229175808
},
"rootfs": {
"avail": 43863662592,
"free": 49249148928,
"total": 105089261568,
"used": 55840112640
},
"kversion": "Linux 7.0.12-1-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.12-1 (2026-06-09T21:07Z)",
"pveversion": "pve-manager/9.2.3/d0fde103346cf89a"
}
--- FAILED_UNITS ---
● homelab-immutable-offline-verify.service loaded failed failed Verify immutable offline backup image
--- TIMERS ---
Tue 2026-07-21 16:21:41 MSK 25s Tue 2026-07-21 16:16:41 MSK 4min 34s ago homelab-smartctl-textfile.timer homelab-smartctl-textfile.service
Tue 2026-07-21 16:28:31 MSK 7min Tue 2026-07-21 16:13:31 MSK 7min ago prometheus-node-exporter-smartmon.timer prometheus-node-exporter-smartmon.service
Wed 2026-07-22 00:00:00 MSK 7h Tue 2026-07-21 00:00:10 MSK 16h ago dpkg-db-backup.timer dpkg-db-backup.service
Sun 2026-07-26 03:10:18 MSK 4 days Sun 2026-07-19 03:10:41 MSK 2 days ago e2scrub_all.timer e2scrub_all.service
Sun 2026-07-26 03:10:55 MSK 4 days Sun 2026-07-19 03:11:07 MSK 2 days ago xfs_scrub_all.timer xfs_scrub_all.service
Mon 2026-07-27 00:08:17 MSK 5 days Mon 2026-07-20 00:35:49 MSK 1 day 15h ago fstrim.timer fstrim.service
--- HEALTH_FILES ---
--- FILESYSTEMS ---
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 15G 0 15G 0% /dev
tmpfs tmpfs 3.2G 3.0M 3.2G 1% /run
/dev/mapper/pve-root ext4 98G 53G 41G 57% /
tmpfs tmpfs 16G 66M 16G 1% /dev/shm
efivarfs efivarfs 128K 125K 0 100% /sys/firmware/efi/efivars
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-journald.service
tmpfs tmpfs 16G 32K 16G 1% /tmp
/dev/sda2 vfat 1022M 9.1M 1013M 1% /boot/efi
/dev/mapper/pve-homelab--staging ext4 196G 100G 87G 54% /mnt/staging
/dev/fuse fuse 128M 1.4M 127M 2% /etc/pve
tmpfs tmpfs 1.0M 0 1.0M 0% /run/credentials/getty@tty1.service
tmpfs tmpfs 3.2G 4.0K 3.2G 1% /run/user/0
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 3.6M 747 3.6M 1% /dev
tmpfs 4.0M 1.3K 4.0M 1% /run
/dev/mapper/pve-root 6.3M 78K 6.2M 2% /
tmpfs 4.0M 136 4.0M 1% /dev/shm
efivarfs 0 0 0 - /sys/firmware/efi/efivars
tmpfs 4.0M 20 4.0M 1% /run/lock
tmpfs 1.0K 1 1023 1% /run/credentials/systemd-journald.service
tmpfs 1.0M 15 1.0M 1% /tmp
/dev/sda2 0 0 0 - /boot/efi
/dev/mapper/pve-homelab--staging 13M 33 13M 1% /mnt/staging
/dev/fuse 256K 89 256K 1% /etc/pve
tmpfs 1.0K 1 1023 1% /run/credentials/getty@tty1.service
tmpfs 800K 26 800K 1% /run/user/0
total used free shared buff/cache available
Mem: 31Gi 13Gi 6.0Gi 57Mi 12Gi 18Gi
Swap: 9Gi 3.0Gi 7.0Gi
16:21:16 up 5 days, 4:40, 1 user, load average: 0.54, 0.48, 0.44
--- STORAGE_HEALTH ---
no pools available
Personalities :
unused devices: <none>
--- LOCAL_BACKUP_RECHECK ---
11116472459 /var/lib/vz/dump/vzdump-qemu-130-2026_07_20-03_30_00.vma.zst
11043834894 /var/lib/vz/dump/vzdump-qemu-130-2026_07_21-03_30_06.vma.zst
11013723404 /var/lib/vz/dump/vzdump-qemu-130-2026_07_19-03_30_03.vma.zst
10994820121 /var/lib/vz/dump/vzdump-qemu-130-2026_07_18-03_30_02.vma.zst
1137072350 /var/lib/vz/dump/vzdump-qemu-190-2026_07_21-03_33_55.vma.zst
838951976 /var/lib/vz/dump/vzdump-qemu-190-2026_07_19-03_33_53.vma.zst
838817925 /var/lib/vz/dump/vzdump-qemu-190-2026_07_20-03_33_54.vma.zst
785249297 /var/lib/vz/dump/vzdump-qemu-190-2026_07_18-03_33_48.vma.zst
7106 /var/lib/vz/dump/vzdump-qemu-130-2026_07_02-03_30_02.log
7102 /var/lib/vz/dump/vzdump-qemu-130-2026_07_03-03_30_05.log
6564 /var/lib/vz/dump/vzdump-qemu-130-2026_07_18-03_30_02.log
6560 /var/lib/vz/dump/vzdump-qemu-130-2026_07_20-03_30_00.log
6559 /var/lib/vz/dump/vzdump-qemu-130-2026_07_21-03_30_06.log
6459 /var/lib/vz/dump/vzdump-qemu-130-2026_07_19-03_30_03.log
6246 /var/lib/vz/dump/vzdump-qemu-130-2026_07_04-03_30_02.log
4107 /var/lib/vz/dump/vzdump-qemu-130-2026_07_07-03_30_01.log
2642 /var/lib/vz/dump/vzdump-qemu-190-2026_07_21-03_33_55.log
2500 /var/lib/vz/dump/vzdump-qemu-190-2026_07_20-03_33_54.log
2404 /var/lib/vz/dump/vzdump-qemu-190-2026_07_19-03_33_53.log
2305 /var/lib/vz/dump/vzdump-qemu-190-2026_07_18-03_33_48.log
=== RECENT_FAILED_COMMANDS ===
2026-07-21T08:03:07Z GITEA-PUSH-AUTH-001 fcfc28bcc5acb6382e895d5ed7cf51d7d5a3e2f5d8f8332770c58b42039f8a94 1 FAIL gitea-runtime
2026-07-21T08:13:32Z GITEA-TOKEN-CLI-HELP-003 f1d0f967eed6490ad822944a42108c82dc4b59c9fd2557d8c64624dcb6853cd4 1 FAIL gitea-runtime
2026-07-21T08:14:52Z GITEA-CLI-CAPABILITIES-004 2cd6c4d28c8f646d8063105fca80b2f80d38c6a5db3a3eb76869de326b12d2d6 1 FAIL gitea-runtime
2026-07-21T09:49:51Z RUNTIME-RECEIVER-STATE-027 0831002c78c26f35826cd7199995e335ceadcd39f08b9bac23882a0fbb59c8cc 127 FAIL gitea-runtime
2026-07-21T10:26:18Z RUNTIME-RECEIVER-CLEANUP-036 65fbcfbce38a55723a92a8790429d6567389321f030f02cb4bf301cbe67e2531 1 FAIL gitea-runtime
2026-07-21T11:35:23Z BACKUP-DUAL-CLOUD-PREFLIGHT-059 c6f4134d15a9f641ddaf67535760d0abede973dd08222a37af05612a6f580479 1 FAIL backup-audit
2026-07-21T12:28:26Z BACKUP-VM-DUMP-PIPELINE-073 cd8291d0f14a37241e6062408603455d1ebd483091bebfa46c2b7a768f21308b 1 FAIL backup-audit
2026-07-21T12:42:53Z BACKUP-VM-DUMP-CHANGESET-PREFLIGHT-075 a9a8a6046441de0c40a8b13bdfe96698c53a5156b4346b7ce28dc1f2362e8936 143 FAIL backup-audit
2026-07-21T12:56:52Z BACKUP-DRAIN-LOCAL-ARCHIVES-079 531e1da7ce1681eed084b22ced7da87eb1ee8995f662dc7559f4f9dde3c0ac32 75 FAIL backup-remediation
2026-07-21T13:04:48Z BACKUP-FINAL-PROOF-082 9e675a3cb621d0d1fcd8918b02278ab08c1a974f544ebef9a5639bf6578ee601 127 FAIL backup-audit
2026-07-21T13:07:09Z CLUSTER-CLEANUP-INVENTORY-083 09d3f14ca0205936a910dc7af7991474030273a47d76fa033ea6a3185c083922 1 FAIL cluster-cleanup
2026-07-21T13:14:27Z CLUSTER-BACKUP-POSTCLEAN-PROOF-086 601958c48614dbc25cc4592b111de4dec3306e53b756b275068b81a41d42b2da 127 FAIL backup-cleanup
2026-07-21T13:15:21Z BACKUP-POLICY-CLOSEOUT-087 d79c1c881ebca24db09c5b4ed4b71b09a5ad338c30789c99b0644c8935072c52 1 FAIL backup-cleanup
2026-07-21T13:17:20Z BACKUP-CLEANUP-FINAL-CLOSEOUT-088 5ba2893489b6f5831fc4599d3efa874d89c3fd41da7151f9c32945d11d1a23d6 1 FAIL backup-cleanup
CRITICAL_TAILS_BASELINE=COMPLETE
OUTPUT_END
CHAT_OUTPUT_END
View Git Blame Copy Permalink