Instance Reporting

Audience: System Admin, Deployment Administrator, Support Related: Audit And Logging · Environment And Secrets

Privacy-conscious instance reporting provides deployment identity, lifecycle, version, and safe operational metadata for configured reporting workflows. It is separate from end-user activity and must avoid transmitting recipes, credentials, private content, or unrestricted logs.

Runtime branding and host-aware behavior should use the deployment’s configured public identity rather than hard-coded production values.

Blackcap automatically registers each installation with the Microwave Pie Instance Tracker and sends a privacy-conscious heartbeat approximately every six hours. Reporting is part of the normal Flask/Gunicorn application lifecycle: starting inky_admin.service starts reporting, and stopping that service stops it.

Runtime architecture

• inky_admin/services/instance_tracker_service.py owns deployment/environment detection, payload construction, registration, heartbeat timing, retry/backoff, revocation, and the application-owned reporting loop.
• inky_admin/services/instance_tracker_db_service.py provides durable identity/state storage and aggregate database health through Blackcap's centralized provider-aware database services.
• inky_admin/repositories/instance_tracker_repository.py is the only repository that reads or writes the private internal state row.
• inky_admin/inky_admin_app.py schedules the reporting loop after normal application database initialization.
• A process-lifetime file lock elects one local Gunicorn worker. Other workers exit their startup timer immediately.
• A short database lease prevents duplicate traffic if multiple Blackcap application hosts point at the same database.
• There is no separate tracker systemd service, timer, cron entry, Admin setting, or user-facing disable switch.

Configuration

BLACKCAP_ENVIRONMENT="production"

Supported normalized values are production, staging, development, test, demo, and unknown. Blank values default to Production. Instance registration and heartbeat traffic always uses the built-in tracker URL https://tracking.microwavepie.com; the URL is not configurable through Blackcap settings or environment variables.

Durable identity and state

The stable installation identity, permanent token, retry state, heartbeat timestamps, boot marker, and current lifecycle state are stored in Blackcap's application database. SQLite and PostgreSQL use the same provider-aware repository path.

Permanent revocation also writes ${BLACKCAP_RUNTIME_DATA_DIR}/instance-tracker/revoked.json. This credential-free tombstone is created atomically with 0600 permissions inside a 0700 directory. Its presence always overrides the database, so restoring an older database or switching providers cannot reactivate a known revoked installation through supported application paths.

The backing internal_service_state table is application-internal. It is intentionally:

• absent from the Database Admin resource registry;
• absent from SQL Console schema/autocomplete/export metadata;
• blocked from direct SQL Console and CSV-export queries;
• omitted from detailed database parity schema output; and
• not exposed through an Admin route or setting.

Because identity now follows the application database, restoring or replacing the database also restores or replaces the tracker identity. A database migration between SQLite and PostgreSQL should therefore include this internal row when the installation is intended to keep the same tracker identity.

Data sent

Blackcap sends its application name/version/build, normalized environment, deployment type and detection source, Python/OS/architecture, SQLite or PostgreSQL provider, Blackcap schema version when available, aggregate user/recipe/organization/display counts, and sanitized database availability. Regression-test organizations, diagnostic-clone organizations, and their organization-scoped users, recipes, and displays are excluded from these counts. Reports include the metadata marker aggregateCountsPolicy=exclude_regression_and_diagnostic_organizations_v1 so newly filtered snapshots can be distinguished from older heartbeat history. Blackcap never sends customer content or Blackcap application credentials.

Scheduling and duplicate protection

One elected Gunicorn worker performs a due check shortly after application startup and then every 15 minutes. Database state enforces the normal six-hour heartbeat interval and tracker-provided interval bounds. A new system boot allows one early heartbeat.

Network and tracker failures use persisted bounded exponential backoff with jitter: approximately 15 minutes, 30 minutes, 1 hour, 2 hours, 4 hours, then at most 6 hours. Reporting failures never fail an HTTP request, display refresh, scheduled job, or Blackcap background-job record.

An explicit instance_paused response changes the entire Blackcap installation to a locally enforced paused state. Admin, Mobile, API Tester, API-token requests, display/receiver APIs, voice commands, scheduled operations, and Blackcap background workers are blocked or skipped. The health endpoint remains available and reports the lifecycle state. While paused, the login page and any blocked browser landing page perform a short authenticated GET /api/v1/instances/status check; when the tracker reports active, Blackcap clears the temporary pause and normal activity resumes. If the tracker is unavailable, the installation remains paused. The paused and revoked landing pages direct users to blackcap@microwavepie.com when they believe the status is incorrect. Standalone Blackcap scripts never call the tracker: they read the durable local lifecycle state and exit successfully before performing work when the installation is paused or revoked.

An explicit instance_revoked response permanently locks the entire installation, writes the independent revocation tombstone, stops tracker traffic, and prevents Blackcap from generating a replacement identity. Login never attempts to clear a revoked state.

Lifecycle changes are normally learned during the next due heartbeat. A System Admin may force immediate detection while the installation is still active through API Tester → System → Trigger Instance Report.

Deployment detection order

1. Raspberry Pi model information.
2. Docker marker or container cgroup evidence.
3. Google Compute Engine metadata using the required header and sub-second timeout.
4. Google DMI values.
5. Generic Linux fallback.
6. Unknown platform fallback.

Operations and troubleshooting

Reporting is supervised by the existing application service:

sudo systemctl status inky_admin.service --no-pager -l
sudo journalctl -u inky_admin.service -n 200 --no-pager | grep -i "instance heartbeat\|instance registration\|instance reporting"
sudo systemctl restart inky_admin.service

A System Admin can also open API Tester → System → Trigger Instance Report. The action accepts no request data: Blackcap builds the same production registration/heartbeat payload locally, forces one attempt, and returns only masked identity state plus sanitized connectivity, deployment-detection, database-provider, retry, and revocation diagnostics. The internal trigger is CSRF-protected, requires the API Tester mutating confirmation, and is not exported to Postman.

The scheduled-operations installer removes obsolete blackcap_instance_tracker.service and blackcap_instance_tracker.timer units from earlier builds.

Lifecycle enforcement

• Local active plus tracker/network outage: Blackcap continues operating.
• Local paused plus tracker/network outage: Blackcap remains locked.
• Local revoked: Blackcap remains permanently locked.
• /health remains HTTP 200 and includes lifecycle_status plus activity_allowed.
• Browser requests receive a branded pause/revocation page; API-like requests receive instance_paused or instance_revoked.
• Cron operations write a successful skipped status and exit 0 while locked. Long-running watchers poll local lifecycle state and perform no display/client work while locked.