Background Job Architecture

Audience: Developer, System Admin, Deployment Administrator Related: Performance And Job Status · Background Jobs And Schedules · Testing

Background work is represented in the database with organization and requesting-user context, queue/family metadata, status, progress, heartbeat, result, and failure details. Jobs must be idempotent or safely retryable where the UI exposes retry.

Queue priority and process niceness protect interactive display and web behavior on constrained Raspberry Pi hardware. Persistent watchers are systemd services and are separate from database-backed one-shot jobs.

Background-job design

Blackcap Pi uses durable SQLite background-job rows for asynchronous work such as backups, restores, recipe cache builds, search index rebuilds, shopping list generation, display refresh, and deep-clean maintenance.

The current implementation still executes work in in-process worker threads, which keeps the Raspberry Pi deployment simple, but job ownership and progress are now modeled centrally through inky_admin/services/background_job_service.py.

Job ownership model

Every tracked job is expected to carry:

Legacy entity_type, entity_id, and created_by_user_id columns remain for compatibility. New code should prefer resource_type, resource_id, and requested_by_user_id.

Isolation rules

• Organization users must never see jobs from another organization.
• Manual user-triggered jobs are visible only to the requesting user, plus org/system jobs visible in that same organization.
• Scheduled or org-wide jobs set is_system_job = 1 and are visible only inside their organization.
• Hardware/display state remains global because there is one physical e-ink display and one hardware lock.

Status endpoints

Use scoped status endpoints instead of polling one overloaded global endpoint:

The legacy /status route remains intentionally lightweight for existing UI compatibility and should not become a dumping ground for backup/cache/job state.

Supported job types

Canonical job types are:

• backup
• restore
• recipe_cache
• recipe_render
• recipe_search_rebuild
• ingredient_search_rebuild
• shopping_list_generation
• display_refresh
• deep_clean

The service still maps older names such as recipe_cache_build and recipe_search_index_rebuild to the canonical names so existing history remains understandable.

Recipe cache jobs are explicit work. Normal recipe viewing or artifact lookup must not queue rebuilds merely because a newer render variant is missing. A successful recipe-cache job must verify the canonical Recipe Library PDF exists and is non-empty before reporting cached / completed.

Recipe cache queue requests are stored in durable background_jobs rows, while the lightweight worker queue itself lives in process memory. On app startup Blackcap now performs a short delayed recovery pass: queued or previously running recipe_cache jobs are reset to queued, rehydrated into the in-process queue, and processed asynchronously. This prevents recipes from remaining queued forever after a systemd/Gunicorn restart.

For normal Recipe Library cache builds, the render profile is based on the organization's default display resolution, color/image/rich-layout support, and Preferred Recipe Units. Recipe share PDF portrait/landscape orientation is intentionally excluded because it only applies to share-by-email PDFs.

Scheduled display runners and priority locking

Two recurring scripts coordinate display content outside the normal request/response flow:

The deployment-managed invocations live in /etc/cron.d/blackcap, not a personal user crontab. tools/run_scheduled_task.py records a compact last-run snapshot for the System Admin-only Job Status → Scheduled Operations tab while continuing to send command output to journald.

These runners use inky_admin/services/scheduler_lock_service.py so they do not overlap. Menu refresh has priority because it is heavier and less frequent: a top-level inky_menu.py run creates /tmp/blackcap_menu_priority.request, waits for /tmp/blackcap_scheduler.lock, and clears the request in a finally path. The display-content refresh runner checks that priority request before starting and at safe checkpoints between work units. Bound child menu workers launched by an already-locked parent run with BLACKCAP_SCHEDULER_LOCK_HELD=1.

The scheduler lock is separate from /tmp/inky_menu_display.lock, which remains the hardware/display lock used around actual local e-ink operations.

Performance and safety

Migration 025_background_job_queue_isolation.sql adds ownership/progress columns and indexes for scoped polling:

• (organization_id, requested_by_user_id, status, updated_at DESC)
• (organization_id, job_type, status, updated_at DESC)
• (organization_id, job_type, resource_type, resource_id, status, updated_at DESC)
• (organization_id, is_system_job, status, updated_at DESC)
• updated_at DESC

Polling code should use short-lived SQLite connections, small payloads, and org/user filters. Do not scan all jobs or return global queue snapshots to normal UI pages.

Recipe Cache Profile Audit

The Organization page includes a Recipe Cache Status panel for the active organization. System Admins see the same panel from Organizations → Detail for a selected organization. To keep those organization pages fast, the panel loads its current audit data asynchronously and can refresh itself without a full page reload. The manual Refresh button always requests current server/cache/job information. Optional auto refresh is disabled by default, stored only for the browser session, and refreshes every 30 seconds when enabled.

A normal Recipe Library cache build records the profile that was used to render the cached PDF:

• Default Display render profile key/hash
• Default Display id/type
• width and height
• color mode
• image support
• rich-layout support
• attached local e-ink flag
• Preferred Recipe Units
• recipe renderer version
• cache profile recorded timestamp

The Share PDF portrait/landscape setting is intentionally not part of this profile. It only applies to share-by-email PDFs and should not make normal recipe caches look stale.

The Recipe Cache Status panel compares each active recipe's cached profile to the organization's current Default Display profile and Preferred Recipe Units. It also flags recipes with missing PDFs, missing profile metadata, renderer-version changes, layout/source changes, or edits newer than the cached file.

Actions in this panel never rebuild synchronously. The single-row ↻ action, Queue Selected, and Queue All Out-of-Date only add eligible recipes to the asynchronous recipe cache build queue. Edited recipes continue to render from the saved editable recipe content instead of re-fetching the source website. The panel refresh endpoint recomputes from current database/cache/job state rather than guessing from already-rendered page text.

Recipe cache artifact filenames

Recipe cache builds use one canonical filename helper for the renderer, Recipe Library, mobile UI, cache audit, and PDF routes. The helper preserves recipe-id hyphens, so a recipe id such as cannoli-pie writes and reads cannoli-pie.pdf consistently instead of mixing hyphen and underscore variants.

Process priority guidance

Admin UI service

Add to: /etc/systemd/system/inky_admin.service

[Service]
Nice=-5

Then:

sudo systemctl daemon-reload
sudo systemctl restart inky_admin.service

Render / cache jobs

Recommended shell wrapper:

ionice -c2 -n7 nice -n10 /home/pi/inky_env/bin/python3 render_recipe_mode.py

Recommended Python subprocess pattern:

subprocess.Popen(
    cmd,
    preexec_fn=lambda: os.nice(10)
)