Performance and Job Status

Audience: System Admin, Support, Developer Related: Regression Testing · Background Jobs · Index

Performance presents operational trends, duration and queue metrics, resource snapshots, display/render activity, email and backup timing, slow requests, and regression comparisons. Job Status presents queued, running, failed, stale, ignored, reviewed, retryable, and stoppable background work.

Use the cross-links between these pages to move from a metric anomaly to the relevant job family or from a failed job back to its trend context.

Performance metrics

Blackcap includes a lightweight Performance screen for System Admin users. The goal is to make operational health visible without turning a Raspberry Pi install into a monitoring server.

The screen is designed for both self-hosted Raspberry Pi installations and current GCP/PostgreSQL deployments. It focuses on inexpensive operational signals first: background job durations, queue wait time, recipe cache categorization, display/render timing, display push requests, email/backup/purge timing, slow request sampling, compatible trend graphs, and a current host/resource snapshot.

Access and navigation

The screen is available from the Admin sidebar under Administration → Performance at /admin/performance.

Access is controlled by:

• system.performance.view for viewing the page.
• system.performance.configure for future metric collection settings.

System Admin roles receive full dashboard access. Org Admin users receive scoped view access: the Metrics Organization filter is locked to the active organization context, all-organization views are hidden, and host/resource snapshot details remain System Admin-only.

The page header's default-display card follows the active Admin organization context, not the metrics organization filter. Changing the Metrics Organization dropdown filters metric data only; use the normal Admin organization context selector when you want the header to show a different organization's default display status.

Page layout

The Performance page has shared filters above the tabs:

• Date / Time Window controls the metrics and graphs time range.
• Metrics Organization filters metric data only for System Admin users. For Org Admin users it is fixed to the active organization context and cannot be changed from the page.
• Pressing Apply keeps the currently selected tab, so changing the window or organization on 📋 Metrics stays on Metrics and changing it on 📈 Graphs stays on Graphs.

The page has two tabs:

• 📋 Metrics is the default table/card view.
• 📈 Graphs is a trend workspace for comparing compatible metric families over time.

The main metric panels also include 📈 View Trend actions. These open a movable in-page graph pop-up rather than forcing a tab switch. The pop-up can be dragged by its header and includes compact actions for 📈 Open Graphs Tab and ✕ Close.

Metrics model

The metrics implementation uses two tables:

• performance_events: detailed event rows for recent diagnostics.
• performance_aggregates: daily rollups for lower-cost trend and history retention.

Detailed events include nullable organization, display, and background job references so platform-level metrics and organization-specific metrics can live in the same model. Events also include dimensions such as category, operation name, method, provider, display type, render phase, status, duration, queue wait, sanitized metadata, and error category.

Daily aggregates are updated as events are recorded. The aggregate row is keyed by bucket, organization, category, operation, method, provider, display type, render phase, and status. This keeps the dashboard responsive and creates a path for longer retention without storing every detail forever.

Metric writes are best-effort. A metrics failure should be logged at debug level and should not break recipe rendering, display updates, backup, email, login, purge workflows, or any other business operation.

Metrics captured

The implementation captures the safe, high-value metrics already exposed by existing Blackcap workflows:

• Background job completion/failure duration.
• Background job queue wait time when queued_at and started_at are available.
• Background job status, attempts, entity/resource references, and progress message.
• Background job queue health from durable background_jobs rows: queued, running, failed, stuck, oldest queued job age, and queued/running breakdown by job type.
• Recipe cache job category and source/method heuristics.
• Explicit recipe extraction markers emitted by the recipe renderer where available, including JSON-LD, microdata, DOM/meta fallback, Chrome capture/manual/source categories, and Playwright-assisted variants.
• Playwright/browser-fetch duration, success/failure, timeout/error category, and safe domain-only metadata for recipe rendering.
• Recipe preview image-save timing and local e-ink hardware-write timing as separate display phases.
• Menu render-preparation timing and local e-ink hardware-write timing as separate display phases.
• Virtual/mock/web display updates recorded separately from local e-ink hardware writes.
• Home/Admin display content push requests recorded separately from the background render/write worker.
• Display content push render/write timing per target display when the display_content_push worker processes the batch.
• SMTP send duration, success/failure, and source classification for recipe shares, grouped recipe shares, plain/file emails, support requests, and meal plan review emails.
• Manual and scheduled purge duration for selected purge operations, including dry-run versus real purge metadata.
• Backup and restore phase metrics for organization export, cloud upload, restore download/staging, restore apply, and platform backup/restore totals.
• Backup provider grouping for local, Dropbox, and Google Drive where the service boundary exposes the provider.
• Shopping-list generation and rebuild timing from the Admin/Recipe Cart flows.
• Let's Cook full-render timing and low-noise timer/display refresh metrics; quiet polling is intentionally not recorded.
• Slow Admin/API/mobile/kiosk/extension route timing above the configured threshold.
• HTTP 500+ route timing even if the request is faster than the slow threshold.
• Host/resource snapshot: CPU utilization, CPU temperature when the host exposes it, load average, RAM usage/availability, swap usage, disk space, database/WAL size, host type, platform, Python version, and process RSS when available.

Metrics tab

The 📋 Metrics tab uses Blackcap-style cards and tables rather than heavy chart dependencies:

• Summary cards for successful jobs, failed jobs, recipe cache average, slowest recent operation, queue backlog, display average, backup status, and email status.
• Job Performance table with runs, success rate, average/min/max/p95 duration, last run, last failure, and average queue wait.
• Recipe Cache Breakdown table with extraction/source method percentage, duration, queue wait, and the domain of the slowest observed cache build where available. Full source URLs are intentionally not shown; domain-only metadata such as foodnetwork.com or delish.com is enough to identify slow source sites without exposing query strings or private URLs.
• Display Performance table split by operation, display type, and render phase. It excludes Home/Admin content-push request rows so software/image-save/hardware-write timings stay readable.
• Display Content Pushes table for Home/Admin show-now requests and render/write completions.
• Email Performance table by message type and SMTP source.
• Backup / Restore Performance table by provider and phase.
• External List Performance table by provider and sync/check-off operation.
• Operational Workflows table for lower-volume workflows such as shopping-list and Let's Cook operations.
• Playwright / Browser Rendering table.
• Browser Domains table with safe domain-only summaries.
• Queue Health summary table.
• Recent Slow Operations with friendly labels plus raw diagnostic codes when useful.
• Failure Reasons, including each reason's share of all failures in the selected window. Shares over 50% are shown as red; all other failure-reason shares are shown as yellow.
• Slow Requests.
• Resource Snapshot.

The default time window is Last 24 hours, with options for last hour, last 7 days, last 30 days, last 90 days, last 180 days, and last 365 days. Longer windows only show data if performance retention has kept the needed detailed events.

Friendly labels and diagnostic codes

The dashboard displays friendly labels first, such as Permission Denied, JSON-LD + Playwright, or Recipe E-Ink Hardware Write. Raw internal codes remain available underneath when useful for debugging, such as:

• permission_denied
• json_ld_playwright
• recipe_eink_write

This keeps the page readable while preserving enough detail for troubleshooting.

Recipe cache and extraction method interpretation

Recipe cache builds are categorized from the recipe record, cache job payload/result, and BLACKCAP_PERF markers emitted by the recipe renderer. Typical categories include:

• json_ld and json_ld_playwright
• microdata and microdata_playwright
• dom_fallback and dom_fallback_playwright
• dom_fallback_meta and dom_fallback_meta_playwright
• manual_recipe
• chrome_extension_capture
• editable_content
• photo_or_ocr
• recipe_discovery
• web_unknown
• unknown

Playwright/browser metrics should avoid storing page content. Domain-level summaries can be useful when already safe, but detailed page URLs, page HTML, recipe text, secrets, tokens, and request headers should not be stored in performance metadata.

Display render versus hardware write timing

The dashboard schema separates category, display_type, and render_phase so render preparation can be measured separately from display output.

Current display instrumentation records phases such as:

• push_request for Home/Admin display content push requests.
• render_and_write for the display content push worker's per-display operation.
• render_preparation for menu image processing/OCR/footer composition.
• preview_image_save for saving the app preview/current-display image artifacts.
• hardware_write for the actual local e-ink hardware refresh/output.
• virtual_update for mock/web/non-attached display updates.

This distinction is important for e-ink hardware. A local e-ink refresh can be slow because of the display technology, not because Blackcap rendering is slow. Dashboard copy and tables should avoid mixing software render time and hardware refresh/write time without a phase or display-type label.

Display content pushes

Display content pushes are tracked in two parts:

1. Display Content Push Request records the Home/Admin show-now request immediately. This makes quick back-to-back pushes, such as Recipe followed by Meal Plan, visible even before the worker finishes.
2. Display render/write timing records what the background worker actually does for each target display.

These are intentionally shown separately. The push request answers “what did the user/app ask Blackcap to show?” while render/write timing answers “how long did it take to prepare and output?”

Graphs tab

The 📈 Graphs tab intentionally limits comparisons to compatible metric families. This prevents confusing charts such as comparing Playwright recipe-cache duration against mock display content pushes.

Current metric families are:

• Background Jobs
• Recipe Cache
• Display Rendering / Writes
• Display Content Pushes
• Email
• Backup / Restore
• Operational Workflows
• Playwright / Browser Rendering
• Slow Requests

Graph controls include:

• Metric family / graph type.
• Point frequency: Auto, Hourly, Daily, Weekly, or Monthly.
• Measure: Average Duration, p95 Duration, Max Duration, Run Count, Failure Rate, or Average Queue Wait.
• Status filter.
• Series selector.
• 🔄 Update Graph action.

Changing the graph type or status refreshes the visible series selection immediately in the browser, so users can choose compatible series before pressing 🔄 Update Graph. Pressing 🔄 Update Graph redraws the graph based on the selected compatible series.

Graph series selection

The series selector uses compact responsive tiles rather than a full-width table. The layout adapts to the available width and label lengths, usually allowing 2–4 columns when the labels fit without excessive wrapping. Event counts stay on one line.

The series selector includes:

• Individual series checkboxes.
• Event counts.
• ✅ Select All.
• ⬜ Deselect All.

The selected series determine which series are initially available/rendered in the graph. Deselecting all and pressing 🔄 Update Graph keeps the graph empty rather than falling back to a default selection.

Graphs are capped at 8 visible series for readability and color mapping. If more than 8 series are selected, the UI shows a warning and the graph draws only the first 8 selected series.

Graph legend behavior

The chart legend and the summary table under the chart both use the same color dots as the graph lines. This keeps the visual chart and tabular values associated without relying on line order alone.

Legend pills are content-width and can be clicked to hide or show a rendered series immediately without a full page refresh. Clicking a legend pill updates both:

• the graph line/points
• the matching summary table row

This legend toggling only affects the currently rendered graph view. Use the series selector and 🔄 Update Graph when you want to change which series are included in the graph query/render.

View Trend pop-up

Each major Metrics panel has a 📈 View Trend action. It opens a movable graph pop-up on the same Metrics tab rather than switching tabs.

The pop-up:

• fetches the relevant graph in-place
• can be dragged by its header
• has compact same-height actions for 📈 Open Graphs Tab and ✕ Close
• includes the same graph legend behavior as the Graphs tab
• falls back to normal navigation if JavaScript is unavailable

Use 📈 Open Graphs Tab when a trend needs deeper configuration, such as changing the graph family, selected series, point frequency, or measure.

Point frequency and trend windows

Auto point frequency resolves based on the selected window:

• Last hour: short buckets when available.
• Last 24 hours: hourly points.
• Last 7 days and Last 30 days: daily points.
• Last 90/180 days: weekly points.
• Last 365 days: monthly points.

Graph values are calculated from retained detailed events for the selected window. If month-over-month history is desired, increase the detailed performance metrics retention period accordingly or add a future aggregate-backed graph query.

Future graph improvements could include:

• aggregate-backed graph queries for long-term daily/weekly/monthly trends after detailed events have been purged
• comparison overlays such as previous period, previous week, or previous month
• scheduled resource snapshots if resource trends become important

Resource Snapshot

The Resource Snapshot panel helps diagnose Raspberry Pi/self-hosted resource pressure and provides a lightweight view of host status.

It includes:

• CPU utilization.
• CPU temperature when the host exposes it.
• Load average.
• RAM used and available.
• Swap used and available.
• Disk usage/free space.
• Database and WAL size.
• Host/platform details.
• Python version.
• App process RSS when available.

CPU/RAM/swap/temperature values refresh live every 10 seconds while the Performance Metrics tab is visible. These live refreshes are read-only, are not written to performance_events, and intentionally avoid expensive folder-size/database-size recalculation.

The Load avg field includes a tooltip explaining that load average is the average number of runnable or waiting processes over the 1/5/15-minute windows.

Resource metrics vary by host. Raspberry Pi/Linux metrics are richer than generic Docker/GCP environments unless additional platform signals are provided. CPU temperature may be unavailable on non-Pi hosts.

Slow request sampling

Route/page timing is intentionally lightweight:

• Static asset requests are skipped.
• Performance page requests are skipped to avoid self-noise.
• Only requests slower than performance.slow_request_threshold_ms are recorded.
• Requests returning HTTP 500+ are recorded even when they are faster than the threshold.

The default threshold is 1500 milliseconds.

Retention and purge

Default retention is:

• Detailed performance events: 30 days.
• Daily performance aggregates: 90 days.

These defaults are DB-backed platform settings:

• purge.performance_metrics.enabled
• purge.performance_metrics.detail_after_days
• purge.performance_metrics.aggregate_after_days

The scheduled data purge service trims both detailed events and aggregates. Retention is platform-scoped because this is a System Admin-only diagnostics feature.

For longer trends:

• increase detailed retention if the current graph query depends on detailed rows
• increase aggregate retention for future aggregate-backed trend history
• avoid recording noisy high-frequency live resource refreshes as detailed events

Storage/schema summary

performance_events is the detailed event table. Key fields:

• event_id
• organization_id
• display_id
• job_id
• event_type
• category
• operation_name
• status
• duration_ms
• queue_wait_ms
• method
• provider
• display_type
• render_phase
• started_at
• ended_at
• metadata_json
• error_category
• created_at

performance_aggregates is the daily rollup table. Key fields:

• aggregate_id
• bucket_grain
• bucket_start
• organization_id
• category
• operation_name
• method
• provider
• display_type
• render_phase
• status
• count
• success_count
• failure_count
• min_duration_ms
• max_duration_ms
• avg_duration_ms
• total_duration_ms
• min_queue_wait_ms
• max_queue_wait_ms
• avg_queue_wait_ms
• total_queue_wait_ms
• metadata_json

Percentiles are calculated from the retained detailed event window for now. If this becomes expensive at SaaS scale, add hourly aggregates or an approximate percentile sketch/table.

Service architecture

The implementation is intentionally modular:

• inky_admin/blueprints/performance.py: route, permission gate, graph/resource JSON endpoints, and lightweight request timer hook.
• inky_admin/services/performance_metrics_service.py: recording helpers, context manager, dashboard aggregation, graph data building, route timing, resource snapshot, and purge helper.
• inky_admin/services/background_job_service.py: emits one terminal performance event per background job.
• inky_admin/services/recipe_service.py: adds a best-effort recipe extraction/source method to cache-build job results.
• inky_admin/services/display_content_push_service.py: records display content push requests and worker render/write timing.
• inky_admin/services/cloud_backup_job_service.py: records backup/restore phase timing where service boundaries expose it.
• inky_admin/services/recipe_email_service.py, support_email_service.py, and meal_planner_review_service.py: record email send timing without body contents or secrets.
• inky_admin/services/data_purge_service.py: purges detailed and aggregate performance metrics according to platform retention settings.

Database work goes through the centralized database service transaction helpers. Do not add direct SQLite connection handling for performance metrics.

Collection boundaries and safety

Performance collection is intentionally lightweight and best-effort. It records operational categories, duration, queue wait, status, safe domain-only metadata, resource identifiers, and sanitized failure classifications where those signals are useful. It does not store page HTML, recipe text, email bodies, credentials, API Tokens, provider secrets, request headers, or full private URLs.

Request timing remains threshold-based rather than recording every poll. Resource snapshot richness varies by host: Raspberry Pi/Linux can expose CPU temperature and local database/WAL sizes, while GCP or container hosts may expose a different subset. Detailed events support recent diagnostics; daily aggregates support longer lower-cost retention.

Troubleshooting and interpretation

The page or a section is empty

Metrics are created only when instrumented workflows run. A new install or recently added metric family may show empty tables/graphs until matching jobs, display pushes, recipe cache builds, emails, backups, or slow requests occur.

Try widening the time window from Last 24 hours to Last 7 days or Last 30 days.

External List Performance

The External List Performance section summarizes Household List browser-bridge activity recorded through the Chrome extension:

• External List Sync / Browser Bridge Import runs;
• External List Check-Off / Browser Bridge Completion runs;
• provider-level success/failure counts;
• average/min/max/p95 duration;
• last run time.

These metrics complement the Job Status External Lists tab. Performance is for trends and timing; Job Status is for reviewing individual sync/check-off outcomes. Provider automation still requires an active Chrome extension session signed in to the provider account, so Performance links to Job Status rather than launching provider actions directly.

A graph has fewer series than expected

The series selector only shows compatible series with matching data in the selected time window/status filter. Change the graph family, status, time window, or metrics organization to find other series.

If more than 8 series are selected, only the first 8 are rendered and a warning is shown.

CPU temperature is missing

CPU temperature is shown only when the host exposes a readable temperature source, such as a Raspberry Pi thermal zone. Generic Linux hosts, containers, or hosted environments may not provide it.

E-ink writes look slow

Local e-ink hardware writes are expected to be slower than mock/web/preview updates. Use the display phase labels to distinguish hardware refresh time from render preparation or preview image save time.

Graph trends do not go back far enough

Graph values currently use retained detailed performance events. Increase detailed retention for longer graph windows, or add aggregate-backed graph queries for longer-term history.

Current interpretation limits

• Some legacy recipe-cache paths expose only a conservative extraction/source category rather than an exact parser phase.
• Display timing is separated into request, render/prepare, transport, and physical-write phases where the owning path reports them; not every display type exposes every phase.
• Live resource readings are host-dependent and are not stored every 10 seconds merely because the page refreshes them.
• Graphs depend on retained matching data; a missing series can mean no event was recorded in that window, not that the feature is unavailable.
• Percentiles are calculated from retained detail and should be interpreted with the displayed sample count.

Job Status integration

Performance and Job Status are designed to work together.

Use Performance for trends, duration, queue-wait, failure-rate, and resource signals. Use Job Status for current queue state, active failures, stale running jobs, and safe admin actions.

The Performance page now includes Job Status links from:

• the top operational link row
• Job Performance
• Recipe Cache Breakdown
• Display Performance
• Display Content Pushes
• Email Performance
• Backup / Restore Performance
• External List Performance
• Operational Workflows

These links preserve the selected Metrics Organization and map the selected time window to the closest Job Status window. Job Status supports 24h, 7d, 30d, 90d, and all.

The reverse links appear in Job Status summary family cards and job-family headers as 📈 View Performance, opening the matching Performance Graphs family where practical.

Performance metric panels may also include direct Job Status links. Those links should navigate normally to /admin/job-status; they are intentionally separate from 📈 View Trend controls, which open the movable trend pop-up on the Performance page.

SQLite slow-operation diagnostics

Raspberry Pi deployments log threshold-based SQLite diagnostics from the centralized database adapter. The diagnostics are inactive for normal fast statements and do not log bound parameter values.

Default thresholds:

• BLACKCAP_SQLITE_SLOW_QUERY_MS=750 logs slow statements, connection opens, batches, scripts, successful BEGIN IMMEDIATE waits, and WAL checkpoints.
• BLACKCAP_SQLITE_SLOW_TRANSACTION_MS=1000 logs transactions whose total lifetime or final commit/rollback exceeds the threshold.

Slow entries include the event type, elapsed time, connection ID, process and thread, request method/path when available, background-job ID when available, and current database/WAL file sizes. SQLite busy or locked failures are always logged, even when the configured threshold is higher. SQL string literals are redacted and bound parameter values are never included.

Examples of event types include:

• begin_immediate_lock_wait
• write_statement_possible_lock_wait
• busy_error
• context_commit
• context_rollback
• wal_checkpoint

These measurements distinguish a slow SQL statement from a long-lived transaction and make 30-second busy_timeout stalls visible without lowering the timeout and converting contention into application failures.

Regression performance panel query shape

The Regression Tests performance panel uses two query phases:

1. Aggregate count, average, best, worst, latest, and previous timing for all test identities across the selected completed runs.
2. Load historical trend points only for the test rows that will actually be displayed.

This prevents the Pi from constructing point dictionaries and sparklines for every historical test when the page shows only a limited number of rows. Trend-point retrieval is chunked to remain below SQLite parameter limits, and the same provider-neutral SQL path supports PostgreSQL.

Recipe cache status polling

Single-recipe and visible-multi-recipe cache polling load only the requested organization-scoped recipe rows. The status APIs no longer scan the complete recipe library or re-read ingredients and editable recipe content merely to report cache progress. Requests that intentionally omit recipe IDs retain the existing all-recipes compatibility behavior.

Job Status operations

The Job Status screen is an operational page for reviewing Blackcap background work.

Navigation:

Administration → Job Status
/admin/job-status

The older /admin/background-operations URL remains as a redirect for compatibility, but user-facing documentation and navigation should call this page Job Status.

The screen complements the Performance page. Performance answers how operations are trending over time. Job Status answers what is happening right now and what failed jobs need attention.

Access

Access is controlled through these permissions:

system.background_operations.view
system.background_operations.manage
system.background_operations.retry
system.background_operations.cancel

System Admin users can view and act across the selected/all organization filter. Org Admin users receive view access only and the page is locked to their current organization context; they must switch organization context to review another organization.

Header and organization context

The page header follows the same visible description/status pattern as other Admin operational pages. It shows the current Admin organization context default display status using the same Meal Plan / Recipe / Menu content status convention as the Recipe Library and Performance pages.

Important distinction for System Admin users:

• The Job Organization filter controls which background jobs are shown.
• The default-display status in the description panel follows the active Admin organization context selector.
• Changing the Job Organization filter does not change the current organization context or the display status card.

Tabs

The page uses real tabs in the same style as the Performance page:

📊 Summary
🍽️ Recipe Cache
📺 Display
☁️ Backup / Restore
✉️ Email / Support
🧹 Maintenance
🧩 Other
⏱️ Scheduled Operations (System Admin only)

The Summary tab is the default. It groups current and recent background work by job family so a System Admin can quickly see backlog patterns, such as:

21 recipe cache jobs queued across 6 organizations

Family tabs show the queued, running, failed, and completed jobs for that specific job family.

Scheduled Operations

The System Admin-only Scheduled Operations tab is host-scoped rather than organization-scoped. It does not use the Job Organization or Time Window filters.

It reads:

• /etc/cron.d/blackcap for installed cron expressions and deployment mode
• per-operation status snapshots written by tools/run_scheduled_task.py
• the installed GeoIP systemd timer
• the Let’s Cook watcher: local-eink full-refresh scope on a full Pi server, or remote-client preview scope on GCP

The tab shows the last recorded start/result/duration and the next calculated cron execution. Newly installed tracking displays Not recorded yet until each operation runs at least once. Cron output remains available through journald; the snapshot stores only the latest compact result summary.

The managed file is system-wide, so it is not displayed by crontab -e. Use the commands below. A direct edit is temporary because the deployment installer can regenerate the file:

sudo cat /etc/cron.d/blackcap
sudo nano /etc/cron.d/blackcap
sudo systemctl status cron.service --no-pager

Shared filters

The page has shared filters above the tabs:

• Job Organization
• Time Window

Applying filters stays on the currently selected tab.

Current queued and running jobs are included even when they were queued before the selected time window. Completed and failed job history respects the selected time window.

For Org Admin users, the organization selector is replaced with the locked current-organization label and direct route parameters cannot switch the page to another org or to all organizations.

When there are active failed jobs outside the selected time window, Job Status shows a warning rather than automatically mixing older failures into the displayed tables. Use Show All Time from that warning to review or ignore those older failures.

Summary cards

The summary cards show:

• queued jobs
• running jobs
• failed jobs needing attention
• safe retryable jobs
• completed jobs in the selected window

Ignored failures are retained for history but do not inflate active failure counts.

Failed job actions

Failed jobs can be actioned from the family tabs. The Summary tab is intentionally a drill-in overview and does not show job/error detail tables.

Available actions:

• Retry — only shown when the job type has an explicit safe retry policy.
• Reviewed — marks the failure as reviewed.
• Ignore — keeps the failed job in history but removes it from active failure counts.

Retry creates a new background job and preserves the original failed record. It does not mutate the failed job back to queued.

Retry safety

The retry model is intentionally conservative. Jobs are not replayed blindly.

Safe initial retry examples:

• recipe cache jobs
• recipe search rebuild jobs
• ingredient search rebuild jobs
• recipe discovery jobs
• emoji/noun cache jobs when they appear in the queue

Display content jobs, purge jobs, restore jobs, and email jobs are not one-click retried by default because they can be stale, destructive, or duplicate external side effects. Those workflows should be retried from the owning page unless a dedicated safe retry handler is added later.

Job family behavior

Recipe Cache

Shows recipe cache builds, search index rebuilds, recipe imports, and recipe discovery jobs.

This is the most important family for diagnosing queue backlog on Raspberry Pi installs because recipe cache generation can be slow and can involve browser rendering.

Display

Shows display content pushes, display rendering, and scheduled display work when those items are represented as background jobs.

Display retries are intentionally conservative. A failed display job can be superseded by a newer Recipe, Meal Plan, or Menu push. Retrying an old display job blindly could put stale content on a display.

Backup / Restore

Shows backup, restore, export, and upload jobs when queued as background work.

Backup retries may be added with confirmation-required behavior. Restore retries should remain manual from the restore workflow.

Email / Support

Shows background email/support jobs when queued.

Email retry is not enabled by default because it may send duplicate email.

Maintenance

Shows purge, cleanup, emoji/noun cache, index maintenance, and System-page audit work. The Direct SQL Audit and Database Parity Audit appear here while queued, running, completed, or failed. The separate Safe Database Parity Repair also appears here, but is not automatically retryable because it performs explicitly confirmed additive database changes.

Purge and repair jobs should generally be reviewed from their owning page rather than blindly retried.

Other

Shows durable background jobs that do not yet map to a known family. This helps catch new job types that should be classified later.

Data retention

Background jobs continue to follow the existing background job purge settings. The Job Status page does not create a permanent log of every job beyond the existing retention behavior.

Action metadata is stored in background_job_actions so review/ignore/retry actions have a lightweight history.

Implementation notes

The page is intentionally modular. The user-facing page name is Job Status; the underlying blueprint/service names still use background_operations for compatibility with the first implementation:

blueprint → background_operations_service → background_job_repository / DB service

No large logic belongs in inky_admin_app.py, and route handlers stay thin.

The initial schema support adds lightweight fields to background_jobs for review/ignore/retry tracking and a background_job_actions table for admin actions.

Troubleshooting

The page shows no jobs

Background jobs are purged according to the existing retention settings. Widen the time window or trigger a workflow that creates background jobs, such as recipe cache building.

A failed job has no Retry button

Only explicitly safe retry types show a Retry button. Jobs that can duplicate email, restore data, purge data, or show stale display content are intentionally not one-click retryable.

The default display status does not match the Job Organization filter

That is expected. The Job Organization filter controls job rows. The display status card follows the current Admin organization context selector.

Expandable detail sections

Family tabs use expandable sections so the page stays usable when there are several job tables on the same screen.

Default behavior:

• Failed sections are expanded because they contain review/retry/ignore actions.
• Queued and Running sections expand when they contain current work.
• Completed sections are collapsed by default because they are mainly for verification.
• Ignored Failures sections are collapsed by default because those failures no longer count as active failures.

Mass actions

Failed-job tables include row checkboxes plus a select/deselect-all checkbox in the table header.

The mass-action toolbar only shows actions that apply to the currently selected rows:

• Retry appears only for selected failed jobs with a safe retry policy.
• Review appears only for selected failed jobs that have not already been reviewed.
• Ignore appears only for selected failed jobs that have not already been ignored.

When a mixed set of rows is selected, each mass-action button applies only to the selected rows that are eligible for that specific action. For example, if five rows are selected but only two are safe to retry, the Retry button applies to those two jobs and leaves the others unchanged.

Database health cross-check

The Summary tab includes all-time database health cross-check cards for background jobs:

• Failed Jobs Check
• Stuck Running Check

These are intentionally separate from the selected time-window summary. They are designed to tie out with the command-line database tools:

python3 tools/verify_database.py
python3 tools/check_jobs.py --status failed

The failed-job cross-check counts active failed rows that have not been ignored, which matches tools/check_jobs.py --status failed and tools/verify_database.py. The regular Summary failed count remains focused on active, non-ignored failures in the selected time window. If older active failures exist outside that window, a warning appears with a Show All Time link instead of silently adding old jobs to the visible tables.

The stuck-running cross-check uses the shared background-job integrity helper. A running job is considered stuck when its latest heartbeat/start/update/queue timestamp is older than the configured stale-running threshold, currently 30 minutes.

Stopping stale running jobs

A running job that has been active for at least 30 minutes is treated as stale enough for manual intervention from Job Status. These jobs show additional actions in the Running section:

• 🛑 Mark Failed stops tracking the job as running and records it as a failed job. This keeps the failure visible so it can still be reviewed, retried where supported, or investigated.
• 🚫 Mark Failed + Ignore stops tracking the job as running, records it as failed, and immediately moves it to the ignored-failure bucket. Use this when the job is known to be stale or no longer actionable.

Both actions preserve the job record and write a Job Status action entry. They do not attempt to terminate an operating-system process; they correct the durable job state when a worker has disappeared, restarted, or stopped updating the job heartbeat.

The database verification health check and Job Status health cards intentionally ignore failed jobs that have been ignored. Running-job health checks only count jobs still in running state with no recent heartbeat for the stale-running threshold.

tools/check_jobs.py --status failed also hides ignored failed jobs by default so the command-line count ties out with tools/verify_database.py and the Job Status active-failure cards.

External Lists tab

Job Status includes an External Lists tab for Household List browser-bridge work. This tab is intentionally separate from normal background-job rows because External Shopping List sync and check-off are attended browser/extension operations, not server-side workers.

The tab shows:

• Household List Browser Imports from Google Keep, Amazon/Alexa, or custom provider pages;
• External Shopping Sync Runs such as provider check-off/update results after shopping;
• provider, organization, status, item counts, message/error text, and sanitized detail summaries;
• failures where provider check-off failed and items were put back on the Household List.

The tab is grouped like the other Job Status families: Queued, Running, Failed, Completed, and Ignored where applicable. Completed records are collapsed by default because they are mainly verification history. Failed and active sections stay open so provider/login/selector problems are easy to see.

The tab provides Open Connections as the safe action. It does not expose cross-org Sync Again or Update External List actions because those operations require the active desktop Chrome extension and a browser session signed in to the organization’s external provider account. A System Admin may review another organization’s run history, but cannot safely launch provider automation for that organization unless their browser is actually signed in to that organization’s provider list.

Performance page integration

Job Status and Performance are linked as complementary operational views.

• Performance links to Job Status from the main operational link row and from relevant metric panels such as Job Performance, Recipe Cache, Display, Backup / Restore, Email, and External Lists.
• Job Status links back to Performance from the Summary family cards and from each job-family tab header using 📈 View Performance.
• Shared filters are preserved where possible. Job Status supports 24h, 7d, 30d, 90d, and all; Performance windows outside that set are mapped to the closest practical Job Status view.

Use Performance when you want trends and timing. Use Job Status when you want current queue state, active failures, stale running jobs, or admin actions.

Link behavior

Performance-to-Job Status links should land on the closest useful Job Status view rather than opening a Performance trend pop-up. Trend-pop-up controls on Performance are separate from Job Status links.

Job Status-to-Performance links should preserve the selected organization and map the job family to the closest graph family:

• Recipe Cache → Recipe Cache graphs
• Display → Display Content Pushes or Display Rendering / Writes graphs
• Backup / Restore → Backup / Restore graphs
• Email / Support → Email graphs
• External Lists → External Lists graphs
• Maintenance / Other → Background Jobs or Operational Workflows graphs

If a direct job-to-metric match is unavailable, link to the family-level Performance view instead of hiding the link.

Route names

The canonical route is now:

/admin/job-status

The older route remains as a redirect for backward compatibility:

/admin/background-operations

Navigation and new links should use the canonical Job Status route via the existing background_operations.admin_background_operations_page endpoint.

Job detail modal

Each visible job row includes an information button that opens a lightweight detail modal. The modal summarizes:

• job id and job type
• job family and organization
• status, queue/start/finish/failure timestamps
• duration, worker, attempts, and retry policy
• review/ignore state
• related entity/resource
• sanitized failure text
• sanitized payload/result summaries when available

The modal is meant for troubleshooting without requiring tools/check_jobs.py for every lookup. Raw payload data is summarized and obvious secrets/tokens are redacted.

Developer requirements

• Add metric instrumentation at service boundaries, not in templates or oversized route handlers.
• Metric writes must be best-effort and must not fail the business operation.
• Use stable category, operation, provider, method, display-type, render-phase, and status values so tables and graphs remain compatible.
• Store sanitized metadata only; never record secrets, full provider payloads, email bodies, recipe content, or private query strings.
• Register new background job types with a Job Status family and an explicit retry policy.
• Treat destructive, stale, or externally side-effecting work as non-retryable unless a dedicated safe retry handler exists.
• Keep SQLite and PostgreSQL query shapes provider-neutral and test both backends.
• Update Data Retention and Purge when adding retained detail or aggregate records.