Blackcap includes an administrative regression harness for immediate, scheduled, resumable, and diagnostic-clone runs. Test activity is organization-attributed, results are stored in the database, and the UI supports filtering, details, exports, performance comparison, cleanup, and completion email behavior.

Regression documentation is intentionally detailed because the harness is both a product feature and the primary end-to-end verification mechanism. Keep test names and expected behavior synchronized with the code.

Blackcap includes a System Admin-only regression harness for creating isolated test organizations, accepting generated invites without real email delivery, logging in as generated users, exercising key Blackcap routes/API-token behavior, collecting structured step results, and optionally cleaning up through the normal account-delete / delete-last-user-in-organization path.

Design Summary

The harness is intentionally modular and keeps the main Flask app small:

inky_admin/blueprints/regression_tests.py owns the System Admin page and actions.
inky_admin/services/regression_harness_service.py creates run records and page context.
inky_admin/services/regression_runner_service.py runs the background test lifecycle.
inky_admin/services/regression_http_client_service.py wraps a real Flask HTTP session/client and records timings.
inky_admin/services/regression_cleanup_service.py performs cleanup through normal user account deletion behavior.
inky_admin/services/regression_schedule_service.py stores quiet-hour recurring schedules and queues due runs.
inky_admin/services/regression_report_service.py stores sanitized HTML/JSON/JSONL-style report payloads in the database.
inky_admin/services/regression_test_org_service.py marks/list test organizations separately from normal organizations.

The harness uses the existing app DB transaction/service patterns, the normal organization service, invite service, API-token service, audit service, background job service, and account lifecycle service.

Test Lifecycle

A manual or scheduled run follows these phases:

System Admin starts or schedules a run from /admin/regression-tests.
A background job is created with created_via=regression_harness.
A unique organization name is generated, such as Test-Org-20260623-083015-a7f3.
The organization is created through the normal organization creation service.
The generated organization is marked in regression_test_organizations so it is hidden from normal organization lists.
An Org Admin invite is created through the normal invite service.
The invite token returned by the service is accepted without requiring a real mailbox.
A generated password is set for the Org Admin test user.
The harness logs in through a real HTTP session and records request durations/results.
A test API token is created through the normal API-token service.
Extension API-token checks run with Authorization: Bearer <test token>.
If configured, one secondary user is invited, accepted, logged in, and permission-denial checks are recorded.
Unsupported first-pass areas are not attempted by the first-pass harness.
Full cleanup mode deletes the secondary user first, then the Org Admin user, through the normal account lifecycle service.
Leave-open mode keeps the org/users active and shows generated credentials on the regression page.
Sanitized HTML/JSON/JSONL-style/performance report payloads are stored on the regression run record in the database.

New Admin Page

Route: /admin/regression-tests

Access rules:

System Admins with system.regression_tests.view can view the page.
System Admins with system.regression_tests.manage can start runs, save schedules, switch into test org context, and trigger cleanup.
Org Admins, normal users, and unauthenticated users cannot access the page.

The page includes:

Start New Run
Schedule Runs
Active Runs
Test Performance
Test Organizations
Historical Runs
Run Details, including generated credentials while applicable
Collapsible panels so long regression history, performance, organization, and schedule sections can be opened only when needed
A generated-user reference table with copy buttons and hidden passwords by default
Step-level pass/fail results with expanded failure request/response detail
User-friendly labels for modes/statuses and Blackcap-formatted timestamps with seconds
Sticky table headers, pinned action columns, and contained horizontal scrollbars for long run/history/performance tables
Cleanup actions from the Test Organizations list only

Scheduled Runs

Scheduled runs are stored in regression_test_schedules. Saved test-data defaults are stored in regression_test_defaults and are merged into scheduled runs so recurring quiet-hour runs use the latest saved fake recipe/API input values.

Supported fields:

enabled / disabled
daily or weekly
weekly day of week
preferred time of day
time zone selected from the common Blackcap time-zone list
schedule-specific run mode
schedule-specific secondary user creation/tests/role
schedule-specific keep-org-on-failure behavior
schedule-specific optional backup restore test
schedule-specific Code & Schema Verification option, enabled by default, which runs and waits for the System page Direct SQL Audit and read-only Database Parity/Schema Validation jobs
report retention count
next and last run timestamps

Production scheduling should call run_regression_tasks.py from a cron/systemd timer using the same pattern as run_daily_purge.py; the Admin page does not need a manual due-schedule trigger when that background path is active. Use the Start New Run panel to save shared recipe/API test-data defaults; schedules intentionally keep their own run mode and run-option checkboxes while referencing the shared fake recipe/API input values.

Diagnostic Clone Runs

Diagnostic Clone extends the regression harness for organization-specific troubleshooting. A System Admin starts it from the normal Organizations list or organization details page, and the resulting run is tracked on the Regression Tests page with the rest of the harness history and generated credentials.

Diagnostic Clone creates exactly one regression/test organization and always leaves it open for inspection. The source organization is associated to the run for attribution, but the run history remains platform/Default-owned. Regression/test organizations are not valid diagnostic clone sources; the Default organization can be used as a source if needed. Only one active diagnostic clone is allowed per source organization; while one is active, the organization actions open that clone/run instead of creating another.

The first implementation is configuration-focused:

Copies safe organization-scoped configuration, including timezone/locale fields, scoped settings, meal planner settings and slots, Noun Project footer/rule configuration, grocery departments/category mappings, kitchen inventory storage locations, organization AI Seed entitlement/billing configuration, and other non-secret org preferences available through the service layer. A finite diagnostic organization starts with the full current monthly allotment unless current billing-period AI usage is explicitly copied.
Clones display rows for diagnostic use, but does not copy display clients, receiver tokens, kiosk tokens, physical hardware write targets, or other display secrets.
Creates a safe mock display when no source displays are available.
Creates generated regression users through the same invite/acceptance flow as normal regression runs.
Runs a single-organization diagnostic regression profile when selected. This profile skips A/B import-export, cross-organization sharing, and backup-restore tests, but includes the deterministic mocked AI recipe-image generation/save workflow, Org Admin AI Usage scope and provider-diagnostic privacy checks, the dedicated AI Usage page, and the read-only organization and global personal AI Seed summary endpoints. The mocked image test spends from the generated user's personal bucket so it does not alter the copied organization balance. If the regression option is not selected, the run only creates the diagnostic org, accepts the generated Org Admin invite, copies configuration/displays, records the setup steps, and leaves the org open.
Restores temporary configuration changes made by diagnostic/config checks, such as the recipe unit preference, before the diagnostic run is finalized.
Adds a collapsed Current Operational Data copy panel on every diagnostic clone admin page directly under the regression context banner. This panel is for source org current-state snapshots that may change over time.
Adds page-specific targeted copy panels for diagnostic clones: Copy Recipes from Source Organization on Recipe Library, Copy Shopping Lists from Source Organization on Shopping Lists, and Copy Meal Plans from Source Organization on Meal Planner. Recipe copy supports source recipe search by title or recipe ID. Shopping-list copy brings the selected list, the current recipe cart, and any related recipes. Meal-plan copy shows only weeks within 8 weeks before/after the source org's current planner week that actually contain data. Already-copied source recipes, shopping lists, and meal-plan weeks show a green check indicator instead of a selectable checkbox. Copied recipes include metadata, editable/converted content, ingredients, tags, source-facing paths, ready recipe cache artifacts, thumbnails/images, related recipe cache metadata, and recipe-scoped AI image status. The latest/success/failure AI jobs are remapped to the generated diagnostic Org Admin, and candidate image files are copied when present, so a source image failure is visible and can be retried under normal retry rules. This recipe AI status copy is independent of the optional AI Usage copy. Cache profile metadata is remapped to the diagnostic clone's regenerated display IDs so the Recipe Cache Status panel mirrors the source state instead of falsely reporting that the default display render profile changed.
Adds an optional Current operational data copy mode at clone creation and as a later manual action from the diagnostic clone. This syncs a minimal issue-reproduction snapshot: current display assignments, display content settings, selected/displayed recipe references, current display preview assets, the source organization's current planner week and associated recipes, meal-plan entries and recipes referenced by displays showing day/week/month meal plans, active Let’s Cook sessions and their recipe/progress/timer data, only the single current active standard shopping list with related recipe data, the local household list, the current recipe cart, and kitchen inventory items including both on-hand and not-on-hand records. External provider sessions/mappings remain excluded. Current-state recipe/display artifacts are copied into diagnostic storage when they are part of the active operational slice, so there is no user-facing cache rebuild option for diagnostic data copy. Operational data can be copied multiple times as source state changes; later copies sync the diagnostic slice instead of duplicating meal-plan slots, shopping lists, household list data, Let’s Cook sessions, or inventory items. Copied diagnostic data keeps the source organization’s user-facing names/titles, including shopping list names, recipe titles, display names, meal-plan labels, Let’s Cook labels, and inventory item names, while only internal IDs/UUIDs are regenerated for isolation. The regression context banner shows an Operational Data Copied - N pill after the first copy.
Adds an independent optional Current billing-period AI usage and organization balance selection at clone creation and in the later Current Operational Data panel. It copies only usage events whose timestamps fall within the source organization's active AI billing period, their diagnostic-safe job context, the current monthly bucket's initial/spent/reserved amounts, and an aggregate of active additional available/reserved organization AI Seeds. Older periods, reservations, ledger history, and unrelated AI jobs are not copied. Re-running the copy replaces the prior imported current-period snapshot. If this selection is not used, the clone retains the full monthly allotment initialized during configuration copy.
Diagnostic regression tests can be run again, or for the first time, after the clone exists from the Run Details panel. These tests append new steps to the same diagnostic clone run instead of creating a second run, so step numbering continues from the existing run history. Diagnostic clone total time is shown as active step duration so idle time between later diagnostic test passes is excluded. The action is hidden after the diagnostic clone has been cleaned up.

Diagnostic Clone does not copy users, source invites, API tokens, Chrome Extension tokens, OAuth tokens, Dropbox/Google Drive credentials, SMTP secrets, external shopping provider sessions, browser profiles, backups, restore staging, support requests, audit history, unrelated job history, older AI usage periods, or performance history. Configuration-only clone creation does not copy operational cache/display artifacts; selected recipe copy and current operational-data sync copy only the artifacts and recipe-linked AI job state needed for the copied diagnostic slice into diagnostic-org storage. Provider/display shell configuration may be copied only in a disconnected diagnostic-safe form.

Diagnostic clone organizations appear in the Regression Tests Test Organizations section with source organization metadata. They remain hidden from the normal Organizations list like other regression/test organizations and are cleaned up with the existing regression cleanup path. When any associated test organization still exists, the Run Details header shows a View Organization button for a single-org run or distinct View Organization A/B/C buttons for paired-org runs.

The Regression Tests page is platform-only. It is visible to System Admins in the Default organization context, but not to Org Admins and not while a System Admin is viewing another organization as an org-admin support context. The diagnostic clone banner's View Clone Run action first returns to Default context, then opens the run detail.

Suite 2026.06.25.10 fixes the page-specific Shopping Lists and Meal Planner copy panels so they resolve the active organization through the same organization-context service used by the rest of Admin. It also refreshes stale browser-session recipe-cart state from the DB when diagnostic copy actions update the cart in app_state, so copied recipe cart data appears immediately in the diagnostic clone.

Suite 2026.06.26.1 adds Household List pin/unpin regression coverage. The shopping-list section now creates a Household List item, pins it, verifies session-only completion leaves the persistent Household item unchecked, unpins it, and confirms anchor/session metadata is cleared.

Suite 2026.07.30.8 adds heartbeat-aware recipe-cache waiting. The normal 120-second regression timeout remains the base limit, but a cache job that is still running with a recent durable heartbeat may continue up to a 360-second hard cap. The production editable renderer already has a 300-second subprocess limit, so the regression waiter now remains long enough to observe either successful completion or the renderer's real terminal timeout. Queued jobs without progress, stale heartbeats, and explicit terminal failures are not granted the extension. The regression result records whether the active-job extension was used.

Suite 2026.07.30.9 fixes diagnostic-clone AI reporting and balance fidelity. When a System Admin is actively viewing a live diagnostic clone, the AI Usage page includes that clone as the current authorized organization and defaults to it instead of falling back to All Organizations. Current-period usage copying remains explicitly scoped to the source organization. A copied current-period monthly bucket is also treated as authoritative when its boundaries reflect a transitional source billing period, preventing normal allotment maintenance from creating a second full bucket. Refreshing the current-period snapshot removes only empty overlapping diagnostic buckets created by older behavior and refuses to remove any overlapping bucket that has spending or reservations. Suite 2026.07.30.10 aligns the diagnostic-clone simulated Chrome Extension recipe with the normal regression harness. Diagnostic runs now select Blackcap’s deterministic hosted quick-add fixture through the shared _extension_recipe_test_url() path and no longer contain a diagnostic-only example.test fallback. This keeps the cache build fully offline and prevents DNS-dependent failures while preserving the normal web-backed recipe extraction path.

Suite 2026.07.30.7 keeps the current-period AI usage/balance snapshot, recipe-scoped AI image failure/status copy, and diagnostic-safe AI regression coverage introduced in 2026.07.30.6. On diagnostic clone creation, the current-period AI usage option is now grouped under Current operational data so all optional data-copy selections share one hierarchy. Recipe ingredient copying also normalizes legacy nullable notes/preparation_notes values before insert, preventing older Pi data from failing clone creation against current non-null recipe ingredient columns. The PostgreSQL dependency-safe AI backup restore order introduced in 2026.07.30.5 remains in place; full regression backup restore remains outside the diagnostic single-org profile.

Voice Assistant regression coverage

Suite 2026.07.01.1 adds basic voice-provider regression coverage without creating real Alexa or Google Home OAuth tokens. The harness intentionally does not call live Smart Home discovery/fulfillment routes because those require provider-issued bearer tokens. Instead, it verifies the shared provider-neutral command layer used by ask Blackcap, Alexa Smart Home, and Google Home after provider validation.

Implemented provider-neutral smoke tests

The run now includes a Voice Assistant group that:

loads /api/voice/alexa/status;
loads /api/voice/google-home/status;
verifies /api/voice/test-command rejects direct calls without the API Tester header;
starts a dedicated Let’s Cook session on the regression display;
runs NEXT_STEP and PREVIOUS_STEP through the API Tester-only provider-neutral command path;
starts a direct Let’s Cook timer and then runs PAUSE_TIMER, RESUME_TIMER, ADD_TIMER_MINUTES, SUBTRACT_TIMER_MINUTES, and CANCEL_TIMER;
verifies unsupported timer amounts such as minutes=2 are rejected;
ends the dedicated voice-test Let’s Cook session with END_LETS_COOK.

These checks use provider labels such as alexa and google_home only as internal adapter markers. They do not mint provider tokens, do not depend on Alexa Lambda, and do not depend on Google Home Cloud-to-cloud account linking.

Remaining manual / future provider-adapter coverage

Alexa Custom Skill

Use signed provider tests manually, but regression should at least verify the internal mapping artifacts:

account-linking OAuth status endpoint is configured;
previous-step and subtract-timer commands are present in the provider-neutral command set;
the selected/default display mapping is preserved for the ask Blackcap connection;
rich custom-skill speech remains available from shared command results.

Alexa Smart Home

When a fixture connection can be created without real Amazon tokens, regression should validate the Blackcap adapter directly:

discovery returns one microwave-style endpoint per enabled display, not one endpoint per action;
endpoints include cooking, time, hold/pause-resume, power, fallback mode, and endpoint-health capabilities;
ReportState reflects the current Let’s Cook on/off state for the resolved display;
native directives map correctly: turn on/start cooking, turn off/end cooking, hold/pause timer, resume/start-or-resume timer, adjust cook time, and fallback mode actions;
old SceneController endpoints are still tolerated only as legacy compatibility.

Google Home

Regression should validate the Cloud-to-cloud fulfillment adapter with synthetic OAuth-linked connection rows:

SYNC returns one microwave device per enabled display;
QUERY reports current cooking/start-stop/timer state even when the session was changed outside Google Home;
EXECUTE native traits map to the shared service: OnOff/StartStop/Cook, TransportControl, Timer, and Modes fallback;
timer play/resume resumes a paused timer and starts the current-step timer when nothing is paused;
DISCONNECT disables only the Google Home connection.

Link emails

Regression should exercise the voice-link email service with SMTP disabled and with a mocked SMTP sender:

linking/relinking queues the correct template for Alexa Custom Skill, Alexa Smart Home, or Google Home;
recipients include the linked user plus active org admins;
SMTP disabled skips quietly;
the Connections page resend action reports success, skipped, and failure cases.

Regression Maintenance and Scheduling

Regression runs are still tracked in the database as durable rows. Manual runs may execute in the Admin app process for immediate convenience, but the durable maintenance path is now centralized in inky_admin/services/regression_worker_service.py and can also be run from run_regression_tasks.py.

The regression maintenance pass performs two jobs:

Recover queued or interrupted regression runs.
Start due scheduled regression runs.

It does not purge regression history. Historical run/report retention is handled by the normal data purge flow.

The Admin app schedules a one-shot startup recovery pass so a queued or interrupted regression run can resume after the app restarts. A file lock and short marker window prevent multiple Gunicorn workers from duplicating the recovery pass. Scheduled quiet-hour runs should be started by the timed run_regression_tasks.py process.

For recurring scheduled runs, use the same systemd/cron style already used by Blackcap scheduled purge jobs. A typical systemd timer or cron entry can invoke:

cd /home/pi/Blackcap-Pi && /home/pi/inky_env/bin/python3 run_regression_tasks.py

Running it every 15 minutes is usually enough for quiet-hour schedules. The script executes scheduled and recovered runs synchronously so the process does not exit before a regression run finishes.

Historical regression run/report retention is controlled by platform purge settings:

purge.regression_test_runs.enabled, default true
purge.regression_test_runs.after_days, default 30

This purge removes historical run rows, step rows, and DB-backed report/request/performance payloads only when the run is old enough and no associated test environment is still available. If any regression_test_organizations marker still joins to an organizations row, the run details are protected so the Test Organizations panel can continue to link back to the run that created or cloned that environment. Test organizations continue to be cleaned up through the regression cleanup action and the normal organization lifecycle purge.

Test Organization Model

Test organizations are not marked by changing the core organizations table. Instead, they are tracked in regression_test_organizations:

organization id/name
run id/public id
org admin user/email
secondary user/email
cleanup status
metadata JSON

Normal organization listing functions exclude rows that appear in this marker table by default. The regression page lists live generated organizations separately.

Regression run details are retained while at least one associated environment remains available. In practice, the regression history purge skips any old run that still has a regression_test_organizations row joined to a current organizations row, including environments that are cleaned up but not yet fully lifecycle-purged. Once the normal organization lifecycle purge removes the generated organization row, the historical run/details become eligible for the configured regression history retention window. The Test Organizations dashboard panel joins back to the live organizations row, so a fully purged organization disappears from that operational list.

Invite Acceptance Strategy

Setup uses the normal invite service and the token returned by create_invite() / organization creation. No real email inbox is required. The harness does not expose invite tokens in the normal UI.

Credential Handling

Generated passwords are stored only in transient run fields so they can be displayed for active or leave-open runs. The regression page exposes a generated-user reference table with copy buttons for generated Org Admin, secondary user, and paired-org admin emails/passwords when values are still available. Passwords are hidden by default in the UI. Reports redact credentials. Cleanup clears transient password fields after successful cleanup.

Never log or report raw values for:

generated passwords
API tokens
session cookies
CSRF tokens
SMTP secrets
OAuth secrets
display tokens

Secondary User / Role Testing

A run can create at most one secondary generated user. The Org Admin is created first, then the secondary user invite is created and accepted. Lower-role permission denials are recorded as passing assertions when Blackcap correctly blocks the action.

The first-pass secondary suite includes login and a known admin-only route denial. It also records an ownership-sensitive recipe edit/delete denial scaffold for future deeper route coverage.

Run Modes

Full Run with Cleanup

Runs setup/tests, then deletes generated users through the normal account lifecycle behavior. If the Org Admin is the last user in the generated org, the existing delete-last-user-in-organization path schedules the organization for purge.

Run and Leave Open

Runs setup/tests and leaves the generated organization active. Credentials remain visible on the regression page for manual inspection. Cleanup can be triggered later from the page.

Organization time-format coverage

The regression suite saves preferred_time_format=24h through the Org Admin Organization route, verifies the setting persists on the generated test organization, and then runs page-load smoke tests with that organization context. This gives practical coverage for org-scoped pages such as Backups, Configuration, Meal Planner, and Mobile views under a 24-hour organization preference. Meal Planner also updates a slot to canonical 15:00/16:30 values and restores the original window so the API contract remains protected while the UI uses native time controls.

Rapid Shop a List checkoff coverage

The regression suite includes a dedicated Shopping Lists check for the rapid checkoff API contract. It creates a manual shopping list with twelve rows, sends a multi-item checked-state update through /shopping-list/{list_id}/toggle, reloads the list from storage, and verifies all twelve rows stayed checked.

The same block also verifies mixed check/uncheck final intent and duplicate-label safety. Duplicate-label rows must be keyed by stable item id first, with section/index retained only as a legacy fallback. Checking one regression duplicate eggs row should not check the other row unless both item ids are included.

These tests primarily protect the backend/API side of the mobile single-flight queue. Browser-level behavior should still be spot-checked manually when changing inky_admin/static/mobile_shopping.js: rapidly check 12 items, wait for the saved status, refresh, and confirm all 12 remain checked.

Account page smoke coverage

The page-load smoke suite includes My Account, My API Tokens, and Password pages. These checks are intentionally simple GET requests that fail the run if shared account templates, role labels, access-location decoration, token rows, or password-page rendering introduce a 500.

Interactive Let’s Cook display coverage

The Let’s Cook suite now creates a kiosk display and a hosted web receiver, starts a Let’s Cook session on each display, reads the display-specific state endpoint, checks off an ingredient and instruction, starts a 10-minute timer, pauses the timer, stops the timer, and ends the session with default-content restore requested. The hosted web receiver uses the /api/displays/{display_id}/lets-cook/... companion routes; the kiosk display uses the /api/kiosk/{display_id}/lets-cook/... routes.

Test Coverage Plan

Coverage Matrix

The harness is intended to be deterministic and safe to run repeatedly against generated organizations. Manual and scheduled runs use the same suite runner; the scheduled configuration keeps its own run mode and run-option checkboxes while sharing the saved fake recipe/API input defaults.

Legend: ✅ covered by default, ⚙️ covered when the related option is enabled, ◐ partial/smoke coverage, — intentionally not attempted.

Area	Manual Run	Scheduled Run	Backup Restore Option	Secondary User Option	Notes
Organization setup and invite acceptance	✅	✅	✅	✅	Uses generated orgs and generated invites without real email delivery.
Admin, account, and mobile smoke routes	✅	✅	—	◐	Lower-role routes may pass by being denied correctly. Account smoke coverage includes account, token, and password pages.
API token creation, use, and revocation	✅	✅	—	—	Includes negative add-recipe assertion after revocation.
Simulated Chrome Extension add recipe	✅	✅	—	—	Uses the API-token extension endpoint, not browser DOM extraction.
Manual recipe creation and cache readiness	✅	✅	✅	✅	Dependent flows use a recipe with verified editable content.
Recipe Library select/edit/metadata/cart flows	✅	✅	✅	◐	Secondary-user coverage focuses on safe role/permission behavior.
Paprika import/export and paired-org import	✅	✅	—	—	Uses local deterministic fixtures.
Managed Blackcap recipe sharing	✅	✅	—	—	Runs between generated paired organizations.
Shopping lists, recipe cart, and grouping views	✅	✅	✅	◐	Shopping-list data is DB-backed; no filesystem shopping-list state is expected.
Household-list include/review behavior	✅	✅	—	—	External provider sync is not attempted.
Meal planner create/move/push/list generation	✅	✅	—	◐	Uses current-day/current-week anchors for stability.
Safe display assignment and mock display pushes	✅	✅	—	—	Uses safe/mock-style targets rather than real e-ink writes.
Hosted receiver/kiosk-safe display route coverage	✅	✅	—	—	Smoke/assignment coverage plus interactive Let’s Cook companion-route coverage.
Let’s Cook session controls	✅	✅	—	—	Starts, updates, and ends normal, kiosk-display, and hosted-receiver sessions, including timer start/pause/stop and display-session end/restore.
Kitchen inventory create/search/location flows	✅	✅	—	◐	Secondary-user checks focus on allowed/blocked behavior.
Org-level settings/unit/time-format changes	✅	✅	—	—	Settings are changed through service-backed routes. Generated orgs run smoke coverage under the 24-hour time-format setting.
Backup/restore mutation and usability checks	⚙️	⚙️	✅	—	Creates sentinel data, mutates it after backup, restores, and validates usability/cache readiness.
Cleanup and lifecycle purge handoff	✅	✅	—	—	Uses normal account-delete/delete-last-user behavior.
Regression reporting, timing, and redaction	✅	✅	✅	✅	Run history is Default/platform-owned; generated org ids are attribution only.
Google Keep/Amazon external shopping-list sync	—	—	—	—	Intentionally excluded until isolated fixtures/stubs exist.
Shop With provider website automation	—	—	—	—	Intentionally excluded from the deterministic suite.
OAuth provider callbacks and real email delivery	—	—	—	—	Not attempted by the harness.
Real e-ink display writes	—	—	—	—	Avoided so scheduled runs are safe and non-disruptive.

Implemented Coverage Summary

Current implemented coverage includes:

create unique generated test organizations and hide them from normal organization lists
create, accept, and log in generated Org Admin users through normal invite/session flows
optionally create one generated secondary user and record allowed/blocked lower-role behavior
open key Admin and Mobile UI routes
create, use, and revoke an API token through normal services
simulate Chrome Extension add-recipe payloads through the API-token endpoint
wait for recipe cache readiness through the normal cache-status endpoint
verify editable recipe content readiness when cache succeeds
create manual editable recipes with deterministic cache behavior
exercise Recipe Library selection, edit, metadata, cart, import/export, share, and display-related flows
exercise shopping list creation, completion/reopen behavior, manual items, cart-derived lists, grouping views, household-list behavior, and inventory hydration
exercise organization time-format updates plus meal planner settings/events, canonical slot time updates, recipe/non-recipe entries, day/week/month display pushes, and meal-plan shopping-list generation
exercise Kitchen Inventory, Let’s Cook, safe display assignment, hosted/kiosk-safe display routes, local backup jobs, and selected purge behavior
optionally run backup/restore sentinel mutation and post-restore usability/cache-readiness validation
run normal cleanup through the account lifecycle path and preserve Default-owned regression history
store sanitized run reports, request logs, timing data, and step details in the database

Coverage Boundaries

The harness intentionally excludes tests that depend on real third-party sites, real mail delivery, OAuth callback ownership, or physical e-ink display writes. Those areas should remain outside the deterministic default suite unless they get isolated fixtures, stubs, or an explicitly separate external-integration test mode.

Recipe Cache and Editable Content Handling

The extension add-recipe test remains an API-token/Chrome Extension endpoint test. Because it creates a web-backed recipe whose cache build depends on a source URL, the default harness path now uses Blackcap's own temporary fixture route at /regression-fixtures/quick-add-recipe/<run_id> when [auth] external_base_url is configured. That route serves a tiny deterministic Recipe schema page with ingredients and instructions, so default regression runs do not depend on Delish, Food Network, or another third-party site that may be slow, blocked, redesigned, or learned into the slow-site list. The fixture route intentionally bypasses normal browser-login auth so the normal recipe importer can fetch it like an external recipe page, but the route-level fixture guard only returns content while the referenced regression run is actively running. Once the run finishes, its status closes the fixture. If a run is interrupted and never reaches a final status, the route stops serving the fixture two hours after the run start time. Unauthenticated access outside that active window returns 404.

The manual run form still allows a custom Extension recipe URL for deliberate external-site testing. Leaving that field blank is an explicit request to use the Blackcap-hosted fixture, even if an older saved default used a third-party URL. If a custom URL is known to slow-site policy, the harness also replaces it with the Blackcap-hosted fixture and records that choice in the step expectation. No fixture-specific database rows are created, so there is nothing separate to purge; historical regression run rows remain governed by the normal regression-history purge policy. If external_base_url is not configured and no usable custom URL is supplied, the harness cannot build the hosted fixture URL; configure External Base URL before relying on the default quick-add cache/editable-content coverage.

The harness waits on /api/recipes/<recipe_id>/cache-status and records that wait as a timed regression step. When the extension recipe cache reaches ready status, the harness verifies editable recipe content through the normal prepare-edit endpoint.

The harness also creates a manual editable recipe and waits for its cache build. Manual recipes are deterministic because editable content is written before render/cache work starts. Shopping-list and meal-plan tests use the extension recipe when its editable content is verified; otherwise they use the manual editable recipe and record a warning explaining the fallback. This keeps dependent shopping/meal-plan coverage meaningful while still surfacing extension recipe cache failures separately.

External Integration Limitations

The first pass does not attempt these areas and does not create step records for them:

Google Keep / Amazon external shopping-list sync
Shop With provider website automation
Chrome Extension DOM extraction
OAuth provider callbacks
real email delivery
support request creation/email delivery
real e-ink display writes

Cleanup Strategy

Cleanup is intentionally not a special purge path. It calls the same account lifecycle service used by normal user self-delete behavior:

Delete secondary generated user first, if present.
Delete generated Org Admin last.
Let the existing delete-last-user-in-organization behavior schedule the org for purge.
Verify organization status and purge status.
Clear transient generated passwords after cleanup.

Cleanup refuses to run unless the organization is marked as a regression test organization.

Performance and Reporting

Each HTTP/API step records:

method and path
acting user type
auth mode
duration
response size
status
sanitized request/response summaries

Reports are stored in regression_test_runs columns:

report_html
report_json
request_log_jsonl
performance_json

The stored report payloads redact credentials and tokens. The regression page renders details from regression_test_runs and regression_test_steps, so normal review no longer depends on files under regression_runs/.

Test Performance View

The Regression Tests page includes a Test Performance panel for comparing automated regression step timing across historical runs. It is a read-only analysis view derived from existing regression_test_runs and regression_test_steps rows; it does not create a separate performance-summary table.

The view is intended to answer which repeatable tests are getting slower, which tests are most variable, and whether total run time changed because of one slow step or broad suite drift. It excludes manual diagnostic/test activity and diagnostic copy activity by default because those records are useful run documentation but are not repeatable automated test cases.

Performance rows are matched by stable test identity rather than visible step number. The identity is based on the repeatable test metadata, such as group/phase, request path, and test name/description. This keeps comparisons meaningful when new tests are inserted, diagnostic activity rows are recorded without formal step numbers, or the visible numbering changes between suite versions.

The panel includes:

Summary cards for latest total run time, average latest step time, slowest latest test, and biggest increase.
Filters for search, group/phase, status, run type, and the maximum number of tracked tests returned.
A Max tests selector so installations with more than the default query limit can review 150, 200, 300, or 500 tracked automated tests without changing platform-wide query settings.
A table showing latest duration, previous duration, change, average, worst duration, run count, status, status basis, and a small sparkline.
A sticky trend chart and sticky column headers while scrolling the performance table.
A selected-test pill state; the first visible test is selected automatically when the panel loads and its trend is shown immediately.
A no-data state that clears the graph when filters, such as a group with no matching automated tests, produce no rows.

Status is based on the performance comparison basis shown in the table, not only the raw seconds shown in the change column. The basis text explains whether a row was marked stable, slower, faster, or variable because of percent change, absolute duration change, or variability.

Job Status / Performance Isolation

Normal Performance and Job Status views exclude organizations marked in regression_test_organizations by default. Regression request metrics are labeled with source=regression_harness metadata and the generated test organization id. The regression details page remains the place to inspect harness results.

Audit Events

The harness writes audit activity with source regression_harness, including:

run started
test organization created
invite accepted
API token created
cleanup started/completed/failed
run completed/failed

Secrets are never included in audit metadata.

Database Page Integration

The platform Database page includes these system-only resources:

Regression Test Runs
Regression Test Steps
Regression Test Schedules
Regression Test Defaults
Regression Test Organizations

Generated password fields are masked in the Database page.

Dashboard Summary

The Regression Tests page shows a tabular summary strip between the page description and the run controls. It uses proper-cased labels and compact values so the page can be scanned quickly before starting or resuming work. The summary includes:

Active Runs: currently queued or running regression runs.
Open Envs: generated test environments still present, with a Fully Open / Cleaned Up breakout.
Historical Runs: retained completed runs, with Passed / With Failures breakout.
Last Run: the most recent non-diagnostic regression run, including duration, local date/time, compact source label such as Manual or Sched, and run id.
Active Diagnostic Clones: diagnostic clone organizations still available for investigation.
Enabled Schedule: a check mark when the single scheduled regression run is enabled and an X when it is disabled. This is intentionally boolean rather than a count because Blackcap supports only one scheduled regression configuration.

Summary cells are navigation links. Active Runs jumps to the Active Runs panel, Open Envs and Active Diagnostic Clones jump to Test Organizations, Historical Runs jumps to and expands Historical Runs, Last Run opens the selected Run Details panel, and Enabled Schedule jumps to Scheduled Runs.

When a Run Details panel is selected, a compact arrow-only top button appears while the panel is visible but its header has scrolled off-screen. Pressing it returns to the top of the selected Run Details panel. The button uses the tooltip text: Return to the top of the Run Details panel.

The Test Performance panel uses the same compact floating arrow behavior. Its button appears only while the Test Performance panel is visible, the panel heading has scrolled off-screen, and the bottom of the panel has not yet moved above the bottom of the viewport. Pressing it returns to the top of the Test Performance panel. This keeps the helper visible while the user is deep in the panel but removes it once the panel has effectively been scrolled past.

Migration Steps

The regression harness schema migrations are:

database/migrations/161_regression_testing_harness.sql
database/migrations/162_regression_test_defaults.sql
database/migrations/163_regression_reports_db_storage.sql
database/migrations/164_regression_history_purge_and_worker.sql
database/migrations/165_regression_suite_version.sql
database/migrations/166_regression_step_org_attribution_and_emoji_save_validation.sql
database/migrations/167_regression_schedule_backup_restore_option.sql
...
database/migrations/202_regression_code_schema_verification_option.py

Migration 202 adds the schedule-level Code & Schema Verification flag idempotently for SQLite and PostgreSQL. On restart, Blackcap’s normal DB initialization/migration flow creates or updates the regression tables.

Restart Commands

On the Pi:

sudo systemctl restart inky_admin.service
sudo journalctl -u inky_admin.service -n 80 --no-pager

For manual local testing:

python -m py_compile \
  inky_admin/blueprints/regression_tests.py \
  inky_admin/services/regression_harness_service.py \
  inky_admin/services/regression_runner_service.py \
  inky_admin/services/regression_http_client_service.py \
  inky_admin/services/regression_cleanup_service.py \
  inky_admin/services/regression_report_service.py \
  inky_admin/services/regression_schedule_service.py \
  inky_admin/services/regression_test_org_service.py

Testing Checklist

Page Access

System Admin can access /admin/regression-tests.
Org Admin cannot access it.
Unauthenticated user redirects to login.
Page shows active runs, history, schedules, and test organizations.

Scheduling

Schedule can be enabled/disabled.
Next run time is shown.
Due schedules can be queued.
Scheduled runs appear in Active Runs.

Run Setup

Manual run queues a background job.
Generated test org name is unique.
Test org is hidden from normal Organizations list.
Test org appears in the Regression Test Organizations section.
Org Admin invite is created and accepted.
Org Admin generated credentials are visible only on the regression page while active/leave-open.
Optional secondary user invite is created and accepted.

Cleanup and leave-open run modes

Full cleanup mode runs tests then triggers normal cleanup.
Leave-open mode leaves test org active.
Manual cleanup can be triggered later.

Role Tests

Org Admin login succeeds.
Secondary lower-role login succeeds when enabled.
Unauthorized lower-role access is recorded as an expected permission-denial pass.

Functional Tests

Admin home opens.
My Account, My API Tokens, and Password pages open without a 500.
Recipe Library opens.
Meal Planner opens.
Shopping Lists opens.
Job Status opens.
API-token ping works.
Simulated Chrome Extension recipe payload posts.
Extension recipe cache wait records cache build duration and readiness.
Manual editable recipe creation succeeds.
Manual recipe cache wait records cache build duration and readiness.
Shopping-list and meal-plan dependent tests use a recipe whose editable content has been verified.

Unsupported Tests Not Attempted

External shopping list providers are not attempted in the first pass.
Shop With provider websites are not attempted.
OAuth callbacks are not attempted.
Real e-ink writes are not attempted.
Support requests are not attempted.
Real email delivery is not attempted.

Reporting

Step results are visible.
Counts update.
HTML-style report content is stored in the database.
JSON report content is stored in the database.
Request JSONL-style content is stored in the database.
Performance summary is written.
Test Performance shows historical automated-step timing trends.
Test Performance filters, row limit, selected-test chart, and no-data state work.
Secrets are redacted.

Isolation

Regression org data is excluded from normal Organizations list.
Regression org jobs are excluded from normal Job Status by default.
Regression org metrics are excluded from normal Performance by default.

Cleanup

Secondary user is deleted first.
Org Admin is deleted last.
Existing delete-last-user behavior schedules org purge.
Cleanup is retryable/idempotent enough for already-inactive generated users.
Cleanup refuses non-test organizations.

Regression

Admin UI still loads.
Mobile UI still loads.
API Tester still loads.
Chrome Extension endpoints still work.
Existing purge behavior is unchanged.

Regression task process versus purge

run_regression_tasks.py should not purge regression history. It only looks for due scheduled runs and queued/interrupted runs, then executes them in that process. Regression historical run/report retention is controlled by the existing Data Purge page/settings and defaults to 30 days.

The Admin UI can still queue/manual-start runs for immediate feedback, but the task script is the durable scheduled/recovery worker. If the app stops during a run, the next startup recovery or timed task pass can mark the run queued and continue it.

CSRF-sensitive session JSON requests use the same X-CSRF-Token header used by Blackcap browser JavaScript.

Error Export

Run details provide a Copy errors action whenever failed steps are present. The action copies failed steps as tab-separated text so the output can be pasted directly into a spreadsheet, document, or support/debugging conversation. The export includes run ID, step number, timestamp, group, test name, acting user, status, duration, request method/path, expected result, actual/error summary, failure details, sanitized request, and sanitized response.

Human-facing byte counts in regression summaries are formatted with thousands separators for readability.

2026-06-24 Regression Suite Expansion

The regression harness now records a suite_version and suite_label on each run. This keeps run duration comparisons meaningful as coverage grows; a short early harness run should not be compared directly with a later suite that exercises more screens and flows.

The run history shows total run time, and Run Details displays a larger total-time pill at the top of the detail panel.

Additional coverage added in this pass:

Shopping list item completion, unchecking, and re-completion.
Manual item completion on a recipe-derived shopping list.
Recipe-derived shopping list view validation.
Let’s Cook context retrieval.
Start a local Let’s Cook session using the verified dependent recipe.
Retrieve Let’s Cook session state.
Complete one Let’s Cook step.
Mark one Let’s Cook ingredient used when an ingredient is available.
Create and exercise a kiosk-display Let’s Cook session with step/ingredient checkoff, timer start/pause/stop, and end/restore.
Create and exercise a hosted-web-receiver Let’s Cook session with step/ingredient checkoff, timer start/pause/stop, and end/restore.
List Let’s Cook history.
End the Let’s Cook session.
Open Let’s Cook history detail for the completed session.

Manual cleanup of a leave-open regression organization now changes the run out of Left Open status. Once cleanup completes, the run is marked as Passed when there were no failures, otherwise Completed with Failures, matching the status it would have had if cleanup happened as part of the original run.

2026-06-24.3 regression harness expansion

Suite 2026.06.24.3 expands the default harness beyond the initial 49-step suite. The historical run table now keeps Total Time and Counts near the left side of the table, immediately after Status, so run duration and pass/fail count can be compared without horizontal scrolling.

The Started timestamp is the execution start time, not the moment the System Admin presses Start. The run row is created in a queued state first; started_at is set when the runner actually begins executing. Queue timing can still be inspected from the created/updated timestamps in the platform Database page if needed.

The page intentionally does not auto-refresh. Each major section has a lightweight Refresh link that reloads the page back to the same section anchor so the System Admin can update the data without losing their place or adding background polling load.

Additional default tests include:

broader Admin/Mobile page-load smoke tests that treat permission denial/redirects as acceptable as long as the route does not 500
recipe selection after cache build
editable recipe content and metadata save through the Recipe Library editable-content API
recipe display assignment from Home to the mock/default display
meal-plan display assignment from the Displays API
hosted web receiver creation and recipe assignment
Kitchen Inventory add/search/on-hand/not-on-hand flow
inventory hydration check from a shopping list
safe org-level unit setting changes through As Written, US, and Metric values
local backup job trigger
selected completed background-job purge through the centralized data purge service
API token revocation followed by a negative add-recipe assertion
emoji picker payload, unsupported emoji validation, and picker emoji validation
expanded secondary-user smoke, recipe, shopping-cart, delete/permission, and inventory checks when a secondary user is configured

Recipe Discovery web search is intentionally not part of the default suite because Google-backed discovery can be affected by bot checks and external network/provider behavior. The default harness may test local discovery support such as inventory ingredient suggestions, but external recipe discovery search should remain an optional slow/external test mode with dedicated fixtures.

2026-06-24.4 Paprika import/export and paired-org coverage

Suite 2026.06.24.4 adds deterministic Paprika fixture coverage using local files under regression_fixtures/paprika/.

New coverage includes:

Preview a multi-recipe Paprika fixture.
Import the fixture into the primary regression org.
Re-import the same fixture as a different recipe type to verify URL-based duplicate handling updates the existing recipes instead of creating duplicates.
Export an imported recipe back to Paprika format.
Create a paired regression test organization for the same run using the -A / -B naming pattern.
Preview the exported Paprika file in the paired org.
Import the exported recipe into the paired org as a different recipe type.
Wait for paired-org imported recipe cache build timing.

Historical run cleanup is available for multi-org runs when active test organizations still exist. That cleanup action cleans up every regression test organization associated with the run. Each org still appears independently in the Test Organizations list and can also be cleaned up from that list.

The section Refresh links now force a real page reload with a cache-busting query parameter before returning to the requested anchor. View Details links also jump directly to #run-detail after loading the selected run.

The background-job purge probe now records a compact summary instead of the full purge result payload.

Emoji unsupported validation now chooses an unsupported probe dynamically from a list of uncommon emojis. This avoids false failures on installations where a previously unsupported emoji, such as 🎠, was added to the custom emoji registry.

Suite 2026.06.24.5 adds organization-attributed regression step records. This lets the Test Organizations table summarize only the steps executed against each generated organization in multi-org runs, instead of showing the full run summary for both the -A and -B organizations.

Additional coverage includes:

Rebuild a recipe cache after changing each org-level recipe units setting.
Create a deterministic Spaghetti & Meatballs recipe and an org-level Noun Project footer rule for spaghetti/meatballs.
Add that recipe to the meal plan, push the meal plan to the mock display, and verify the meal-plan footer detection and Noun Project icon artifact.
Use an imported Paprika fixture recipe for the Let’s Cook session when available, rather than relying only on synthetic generated recipes.
Create a managed Blackcap recipe share from the primary regression org to the paired -B org.
Open the managed share manifest as the paired org admin.
Import the managed share into the paired org.
Verify the imported recipe carries shared-import metadata used by the Recipe Library shared badge.
Return to Default from a regression org context now redirects directly to the Regression Tests page #test-organizations section.

Emoji validation is now tested through the real recipe edit save path. The harness attempts to save unsupported emoji metadata on an editable recipe and expects the server to reject the save. Server-side edit save validation now rejects unsupported Blackcap emojis and reports unsupported_emojis so invalid emoji handling is enforced even when a client bypasses the browser-side emoji picker validation.

2026-06-24 Regression Harness Follow-up

Run detail rows now keep the pass/fail status immediately next to the timestamp and show the attributed organization immediately after the request path.
Regression steps without an explicit organization now default to the primary -A test organization when available.
The Noun Project footer check treats the display preview path as diagnostic only; current-day footer detection plus a cached spaghetti icon artifact is considered a pass.
Managed Blackcap recipe share imports now preserve as-written core recipe metadata such as total time, servings/yield, raw time metadata, nutrition metadata, and cleanup metadata. They do not copy translated render-unit values.
The paired-org recipe sharing coverage now verifies that shared import metadata for total time/core metadata is preserved in the recipient organization.

2026-06-24.8 user-flow expansion

Suite 2026.06.24.8 expands default coverage around regular organization-user workflows:

Mobile smoke checks now load the main mobile page plus mobile add recipe, recipe discovery, meal planner, Let’s Cook, shopping, kitchen inventory, and the shared About/status endpoint.
Admin smoke checks include the About/System surface where the acting user can reach it; redirects/403 responses are accepted as long as the route does not 500.
Shopping list coverage now completes and reopens both a fully checked list and a not-fully-checked list.
Shopping list coverage now removes a recipe from the recipe cart, re-adds it, creates a recipe-backed shopping list, moves a manual item to a department, and loads Recipe, Department, and None grouping views.
Meal planner coverage keeps recipe entry creation, movement, and shopping-list generation anchored to the current day/current week so the suite does not fail near the end of a planner week.
Meal planner coverage now creates a manual non-recipe meal-plan item and includes it in the generated shopping list.
Display coverage now pushes week, day, and month meal-plan views to a safe display using the current day as the anchor date.
Kitchen Inventory coverage now creates storage locations and moves an item between storage locations in addition to on-hand/not-on-hand checks.
Let’s Cook coverage now opens Recipe Library before starting the session and attempts to mark every detected step and ingredient complete/used before ending the session.

Generated test user references

Run Details shows generated test users in a reference table. Multi-organization runs show the primary organization administrator, the secondary test user when created, and paired organization administrators such as Org B Admin. Generated passwords remain transient, are hidden by default in the table, and are cleared after cleanup.

Cleanup organization attribution

Manual cleanup records cleanup steps against the relevant regression test organization. For paired import/export runs, cleanup steps for Org A and Org B are attributed independently so the Test Organizations summary can count those steps correctly.

2026-06-24.12 optional backup restore mutation and usability validation

Suite 2026.06.24.12 keeps the optional Test backup restore checkbox on the Start New Run panel. Current Blackcap organization backups restore in-place to the same organization; the restore service rejects backups whose embedded source organization differs from the restore target. Because of that, the regression harness validates same-organization restore against Org A rather than creating an Org C restore target.

When enabled, the restore validation still runs at the end of the main suite, after API token revocation has been tested. The harness now makes the restore probe destructive in a controlled way:

creates a dedicated sentinel recipe and sentinel shopping list,
captures a baseline of the sentinel recipe/list before backup,
captures the compact pre-restore summary of core user-facing org data counts,
creates a local organization backup through the existing organization backup service,
archives the sentinel recipe after the backup,
checks and completes the sentinel shopping list after the backup,
restores that backup back into Org A,
compares core user-facing org counts,
verifies that the archived recipe is active again and the completed shopping list is back to the pre-backup active/unchecked state, and
runs post-restore usability checks against the restored sentinel data: recipe detail payload, recipe selection, recipe cart add, shopping-list view, shopping-list completion, persisted completion status, reopen, and reopened active/unchecked state.

Regression harness tables are intentionally excluded from organization backup exports and restore clearing. This prevents an in-progress regression restore probe from deleting or rewinding its own run report, step records, or generated organization metadata.

The Run Details generated-user area is now a table with user type down the left side and Organization, Email, Password, and User ID across the top. Passwords are hidden by default, remain copyable, and can be temporarily revealed when needed.

2026-06-24.13 regression option reconciliation

Suite 2026.06.24.13 reconciles the manual and scheduled regression controls. Manual and scheduled runs now expose the same meaningful run-option checkboxes:

create secondary user,
run secondary-user tests,
keep org on failure, and
test backup restore.

The retired slow-test and skip-external-provider checkboxes were removed from the page because the current deterministic suite does not branch on those flags. External shopping-list provider automation and Google-backed recipe discovery remain intentionally outside the default deterministic harness until dedicated fixtures or isolated provider stubs exist.

Secondary-user tests now imply secondary-user creation in both manual and scheduled runs. The UI keeps those two checkboxes in sync, and the server-side config parsing enforces the same rule so submitted forms and scheduled rows stay valid even without JavaScript.

Scheduled runs now store their own backup-restore option in regression_test_schedules.include_backup_restore_test. They still merge in the shared fake recipe/API defaults from regression_test_defaults, but schedule run mode and schedule run options are independent from the Start New Run panel. Weekly schedules now expose a day-of-week selector, and the schedule time zone is selected from the common Blackcap time-zone list instead of being a fixed text value.

The dashboard panel order is now Active Runs, Test Organizations, Historical Runs, then Run Details. Historical Runs hides the verbose Current step column to make the history table easier to scan; the final/current step remains available as a Phase tooltip and in Run Details.

2026-06-24.14 restore cache readiness validation

Suite 2026.06.24.14 tightens the optional backup-restore checks around recipe cache behavior. The restore sentinel recipe now waits for its normal Recipe Library cache build to reach Cached before the organization backup ZIP is created. That ensures the backup contains both the restored recipe row/cache metadata and the org-scoped cache files for the sentinel recipe.

After restore, the post-restore recipe usability checks first call the normal recipe cache status endpoint and expect the restored sentinel recipe to already be ready. This validates that restore brought back cache assets and the metadata links to those assets, rather than depending on a new rebuild after restore. If that immediate readiness check fails, the harness records that failure and then waits for the cache as a fallback before attempting recipe selection, so the run continues with useful downstream diagnostics instead of failing only with a /select-recipe 409.

2026-06-24.15 regression history ownership and purged-org hiding

Suite 2026.06.24.15 clarifies the split between regression history ownership and test-organization association. Regression run history is platform/Default organization data; the generated organization id on steps and marker rows is an attribution value showing where each test actually ran, not ownership that makes those history rows part of the generated org's lifecycle purge.

The Test Organizations panel now shows only generated organizations that still have a live organizations row. When organization lifecycle purge permanently removes the generated org, the marker row is retained for historical association and marked purged, but it no longer appears in the operational Test Organizations list and can no longer be opened as an organization context. Historical Runs and Run Details remain available until the separate regression history retention policy removes them.

2026-06-25 documentation coverage matrix update

The regression testing document now includes a coverage matrix showing which areas are covered by manual runs, scheduled runs, the optional backup-restore flow, and the optional secondary-user flow. The matrix also calls out deterministic suite boundaries such as external shopping-list provider automation, Shop With provider website automation, OAuth callbacks, real email delivery, and real e-ink display writes.

This is a documentation-only update; it does not change the regression suite version or the number of executed test steps.

2026-06-26 test performance view

The Regression Tests page now includes a collapsible Test Performance panel. It analyzes existing regression run/step records to show how automated test duration changes from run to run without adding a new persistence layer. Manual diagnostic/test activities and diagnostic copy actions remain visible in Run Details and exports, but they are excluded from performance trends because they are not deterministic automated steps.

The performance table is keyed by stable test identity instead of visible step number. The page exposes search, group, status, run type, and max-test filters; the max-test selector lets System Admins raise the returned test count beyond the default 100-record query limit for larger suites. Selecting a test updates the sticky trend chart, and the selected pill remains highlighted, including the first auto-selected test when the panel loads.

This is a UI/reporting update only. It does not change the regression suite version, the number of executed test steps, normal Performance/Job Status isolation, or regression history purge behavior.

2026-07-02 Let’s Cook workflow and matcher regression coverage

Additional lightweight pytest regression coverage was added for the Let’s Cook matching and setup services because these behaviors are highly text-sensitive and easy to regress while tuning recipes:

Worcestershire sauce uses the distinctive term and does not overmatch unrelated generic sauce steps.
bell peppers does not match a salt and pepper seasoning step.
generic fallback still handles ingredients where directions use only cheese, salt, or flour.
collective instructions such as sauce ingredients can gap-fill otherwise-unmentioned sauce components without matching unrelated white rice serving steps.
generated Setup/Step 0 uses scaled quantities, ignores non-prep measurements and packaged product forms such as canned diced tomatoes, includes useful drain/rinse prep, and avoids numeric sub-step prefixes.

A small API/database visibility test also guards that the API Tester registry and Database Admin registry continue to expose the current-step, Step 0, full-recipe review, recipe units, and voice access-location fields introduced during this work.

2026-07-07 Database Admin diagnostics coverage

Suite 2026.07.08.3 includes the Database Admin diagnostics coverage first added in 2026.07.07.3 for the new saved-query and column-preference resources, async column preference save/reset, async saved query save/run/delete, on-demand Data Problems loading, the standalone Data Problems tab/fragment endpoint, and recipe purge regression coverage for archived recipes that still have meal-plan references. Run creation now reads the suite version and label from the executable runner metadata so newly queued runs show the same current suite version that the worker uses during execution.

Database Admin / SQL Console coverage

The regression suite includes System Admin SQL Console checks in addition to the normal Org Admin page-load smoke tests. These checks are run with the user that started the regression run, pinned to the Default organization context so platform-only SQL Console permissions are available.

Coverage includes:

SQL Console tab fragment loads without a full Database page reload and still shows the inline Export Schema button and rows-per-page control.
Schema metadata loads from /admin/database/api/sql-console/schema and includes protected-field tables such as display_content_assignments while omitting protected columns.
Table sample values are lazy-loaded from /admin/database/api/sql-console/schema/{table_name}/samples rather than being pulled during the initial schema request.
Schema export downloads Markdown that identifies the active SQLite/PostgreSQL provider and SQL dialect, then includes SQL Console limitations, display-assignment guidance, production-query guidance, and regression/diagnostic clone organization filters.
Explicit safe-column queries against protected-field tables are allowed.
alias.* / SELECT * against protected-field tables is blocked with a clear protected-field message.
WITH / CTE queries are blocked with a clear SQL Console limitation message.

These tests intentionally validate the guardrails and export text rather than trying to prove every possible SQL editor autocomplete interaction. Browser-only behaviors such as field-list scrolling, resource-click SQL insertion, and anchored horizontal result scrolling remain manual UI checks unless they are later moved into a browser automation layer.

Run detail performance

Run details are display-only diagnostics, so they are loaded independently from the main Regression Tests dashboard. Clicking a run's detail action now fetches only the run-detail panel instead of rebuilding the whole page, performance panel, and historical tables. Direct links with ?run=<public_id> still work: the page renders a lightweight placeholder and then loads the detail panel asynchronously.

The on-screen detail query avoids loading large sanitized request/response payload columns for passing rows. Failure and warning rows still include request/response detail so the expanded failure diagnostics and Copy issues action remain useful. Full CSV export intentionally reloads the complete payload set because that action is explicit and download-oriented.

Migration 183_regression_detail_indexes.sql adds indexes for the detail display path:

regression_test_steps(run_id, created_at, id) for run-detail step ordering
regression_test_steps(run_id, status, id) for issue payload lookup
regression_test_organizations(run_id, created_at, id) for generated-org detail lookups

Dashboard load performance

The Regression Tests dashboard now defers the Test Performance panel. The initial page load fetches active runs, recent history, schedules, and test organizations, then loads performance timing history asynchronously when the panel is viewed or explicitly loaded. This avoids making the main Regression Tests page wait on the heavier step-history trend query.

The Test Organizations panel no longer runs multiple correlated step-history subqueries for every displayed organization. It loads the visible marker rows first and aggregates all matching step counts/groups in a single indexed pass.

Migration 184_regression_dashboard_cleanup_performance.sql adds indexes for this path:

regression_test_runs(status, created_at, finished_at, updated_at)
regression_test_runs(public_id)
regression_test_steps(run_id, organization_id, status, sequence, id)
regression_test_organizations(run_public_id)
regression_test_organizations(run_id, organization_id)

Run-detail tables remain horizontally scrollable inside the run-detail card. The detail panel and slot are constrained to the page width so a wide step table does not create a page-level horizontal scrollbar.

Additional dashboard performance tuning keeps the initial Regression Tests page from doing per-run organization-count and diagnostic-duration queries. Recent active/history rows are annotated from grouped aggregate queries, and the async performance endpoint is now represented in the API Tester registry and regression endpoint coverage.

Regression run history can also be manually purged from Data Cleanup after the associated environment has been purged or removed. Manual purge can happen before the configured retention age with explicit confirmation, but scheduled purge still waits for purge.regression_test_runs.after_days.

Latest purge coverage

Regression suite 2026.07.08.3 includes coverage for manually purging organization-scoped audit rows plus the extended operational history purge groups: display assignment batches, setting audit rows, Data Cleanup run history, voice assistant command logs, and recipe share audit rows.

Home dashboard and display regression coverage

The regression suite now includes coverage for the Home/System display work:

Home dashboard smoke check for Recipe Quick Actions, Today & Tomorrow Meal Plan, Shopping Lists, and Kitchen Inventory panels.
Displays page check for Assign Content, Return to Default, and Today & Tomorrow options.
Display Details check for header Return to Default and Today & Tomorrow assignment option.
Display assignment endpoint check for the Today & Tomorrow meal-plan view.
Return-to-default endpoint check that first configures a renderable recipe default, temporarily assigns non-default meal-plan content, and then restores the configured default. This avoids coupling the test to whether a last rendered menu image exists in the test organization.
Here Let’s Cook start/end check through the same API used by the Home/Displays Assign Content flow.
API Tester registry check for Assign Display Content, Return Displays to Default, and Start Let’s Cook Session.

These tests intentionally exercise service/API behavior and page markup with the Flask test client. They do not simulate browser drag-and-drop JavaScript; the browser-specific Home interactions should still be validated during manual UI passes when the meal planner drag behavior changes.

Restart recovery boundary and warning summary

Application-start recovery derives the last contiguous fully completed regression result. Passed, warning, and skipped rows are complete; failed or non-terminal rows are retry boundaries. Resume markers and diagnostic/manual activity rows are excluded from the boundary calculation.

The runner uses persisted completed-step keys to control the resumed path rather than merely reporting a checkpoint. Generated identifiers needed by the next action, including manual and secondary-user recipe IDs, are saved with completed results and recovered after restart. Transient login and API-token state can be recreated silently.

Suite 2026.07.27.1 adds phase-aware resume sentinels for the major mutation-heavy sections. A completed Paprika, display, inventory, settings, shopping, meal-planner, voice, Let’s Cook, backup/purge, or backup-restore phase is skipped as a unit instead of being replayed from its first HTTP request. The secondary-user account setup is separate from secondary-user test execution, so finding an existing generated user no longer skips unfinished tests. Within that phase, completed page checks and recipe creations are skipped individually. An interruption after Create manual recipe: Secondary Regression Recipe 2 therefore resumes with Wait for secondary recipe 2 cache build rather than replaying Paprika tests or recreating the recipe.

The resume activity marker uses the retry-boundary sequence without advancing the formal test counter or warning totals. The first substantive resumed test consequently retains the expected next sequence. A newly-created transient API token is still revoked quietly when the original token-revocation phase had already completed.

Before any regression Let’s Cook start, the harness checks the exact target and ends only an orphan session explicitly owned by the regression harness. Non-regression user sessions are preserved. Cleanup is recorded as a warning.

Run Details displays Success, Warnings, and Failures summary tiles in that order, followed by Cleanup.

Suite 2026.07.13.2 adds focused coverage for Recipe Library add-to-plan slot/multiplier options, Home Other date enforcement, transactional display deletion safeguards, editable-content-preserving mass cache refresh, and mass-delete meal-plan reference handling. Restart recovery itself remains covered by focused service tests and real restart validation rather than by a regression test that restarts its own runner.

Suite 2026.07.13.3 corrects the editable-content refresh test so it submits the same string-based ingredients and instructions payload as the Recipe Library editor. It also validates the corrected mass-delete dependency path after the PostgreSQL-safe shopping-list dependency query was fixed. Recipe deletion now fails closed if dependency verification cannot complete, so an inspection error can no longer silently archive a recipe.

2026-07-14 Remote display-client lifecycle coverage

Suite 2026.07.14.1 adds a server-side Remote Display Client lifecycle check using a temporary remote_pi_client display. It generates a short-lived pairing code, exchanges it for a client token, reads state with bearer authentication, downloads a generated PNG preview, records a heartbeat containing four-level grayscale physical-panel acknowledgement metadata, explicitly unlinks, and verifies that the revoked token can no longer read state. The temporary display is removed during the test flow.

Focused Python tests—not the standard regression runner—cover attached-client failure and timing behavior: unchanged previews do not refresh the panel again, invalid/partial downloads preserve the last valid file, network and hardware errors preserve the physical image and report Client Error, a later successful update clears the error, Client mode suspends local schedules, expired windows are not replayed, and applicable schedules become eligible after local ownership returns.

The suite does not attempt to restart its own regression runner. Restart-checkpoint behavior remains covered by focused service tests and real restart validation.

Remote Let’s Cook client cadence and restore coverage

Suite 2026.07.14.2 extends the Remote Display Client lifecycle test. After pairing and validating preview/heartbeat behavior, the test starts a Let’s Cook session on the temporary remote_pi_client display and verifies that the state API advertises a 5-second pull interval. It then ends the session through the normal authenticated API and confirms that the display returns to non-Let’s-Cook default content and the client returns to its normal configured pull interval. The temporary client token and display are still revoked/deleted during cleanup.

Suite 2026.07.14.3 — Remote watcher PostgreSQL and client unlink fixes

The remote Let’s Cook refresh watcher now uses a PostgreSQL-safe timer-session lookup without SQLite-only relaxed GROUP BY behavior. The Remote Display Client lifecycle also verifies self-unlink through the durable client token. /api/display-client/unlink is handled by the remote-client token service rather than the normal Blackcap user/API-token middleware, and the revoked credential must then receive HTTP 401 from the state endpoint.

Code & Schema Verification option

The Start New Run and Schedule Runs panels include Code & schema verification. It is checked by default. When selected, the regression harness starts the same tracked jobs exposed on the System page and waits for each terminal result:

Direct SQL Audit;
read-only Database Parity and Schema Validation.

Each job receives its own regression step with the tracked job ID and concise result counts. Unchecking the option omits both jobs without changing the SQL Console smoke tests or other database regression coverage. The option is part of saved shared defaults and has a schedule-specific persisted value.

Scheduled recipe end timing regression

The display regression uses a two-phase real-time probe on a dedicated hosted receiver:

It configures Weekly Meal Plan as the display default, pushes a recipe immediately with an end instant five minutes in the future, verifies the recipe assignment/render state, invokes the scheduled-end path once, and confirms that the first pass leaves the recipe active.
A much later suite step reloads the saved assignment/display timestamps. If needed, it polls until the five-minute end instant, runs the established display refresh path, and records the actual default-restoration timestamp. The step fails if the display reverted before the scheduled end, if it reverted within the separate two-minute early-reversion guard, if the transition was not performed by display_assignment_expiration, or if the restored default is not the configured Weekly Meal Plan. Reaching this later verification after the nominal five-minute window is informational and does not turn a correct return-to-default into a false failure.

The probe uses a dedicated display so unrelated Let’s Cook, deletion, and default-display tests do not overwrite its state. It polls durable assignment state rather than using a single fragile sleep as the assertion.

Regression completion email

Migration 203_regression_completion_email_tracking.py adds idempotent delivery tracking to regression_test_runs. A terminal run atomically claims its completion notification so worker retries and restart recovery cannot send the same completion email twice.

Recipient rules are:

manual run: the initiating user and the primary organization admin, deduplicated by email;
scheduled/auto run: the primary organization admin only;
diagnostic clone: the same rule, using the source organization rather than the temporary clone organization.

The email records the suite, final status, start/end timestamps, duration, totals, success/warning/failure/skip counts, environment-left-open state, and an absolute Run Details URL. The external URL uses the normal [auth] external_base_url / External Base URL setting (with the deployment environment value as a fallback) before considering the local Admin bind host and port. This prevents background emails from linking to localhost:8080 when Blackcap is hosted through nginx, stunnel, or external port forwarding. A CSV issue export is attached only when the run contains an actual failed/warning step, cleanup failure/incomplete state, run-level failure, or a genuinely left-open environment. A completed cleanup is authoritative even if the generated organization remains queryable briefly while deletion or purge scheduling settles. SMTP failure is shown in Run Details and activity logging but never changes the regression run result.

The Run Details panel displays completion-email status, delivery time, recipients, or a safe failure summary. Normal application startup applies migration 203 on both SQLite and PostgreSQL.

Meal Planner guidance and Kitchen Inventory archive coverage

Regression suite 2026.07.29.1 strengthens Kitchen Inventory lifecycle coverage with an explicitly on-hand item. The flow marks the item on hand, deletes/archives it, verifies that it leaves the On hand view and appears in Recently Deleted, opens the Database Admin lifecycle safe edit, changes Lifecycle Status to Active / Restored, and verifies that the same item returns on hand. It then archives the item again through the Database safe edit and restores it through the normal Kitchen Inventory route, confirming the same public ID and pre-delete on-hand state are preserved. The secondary-user role phase still validates that System Admin, Org Admin, and Data Admin are allowed to archive, while Operator and Contributor are denied.

The emoji regression phase was updated because the former page-level emoji action moved beside each eligible field. HTTP regression now checks the stable data-blackcap-emoji-field targets on Recipe Library Add Recipe, Manual Recipe, and Edit Content surfaces. The actual 🙂 buttons are injected by the shared JavaScript decorator, so focused contract tests verify the stable data-blackcap-open-emoji-picker selector, field association, non-submit button type, cursor insertion behavior, and dynamic-field MutationObserver. Tests should not depend on the button's visual position in the DOM.

Focused tests separately cover displayed-week shopping-list fallback, field-specific emoji contracts, Home/Meal Planner PDF links, archive/restore uniqueness, the 14-day organization-scoped purge threshold, Database lifecycle safe-edit routing, API Tester lifecycle metadata, migration idempotence, and legacy endpoint aliases.

AI recipe-image regression coverage

Regression suite 2026.07.30.4 adds an AI Recipe Images phase after the manually entered recipe has verified editable content. The phase registers a temporary in-process regression provider and generates a deterministic PNG through the same provider-independent contract used by OpenAI and Gemini. Because AI image profiles are editable platform configuration, the runtime regression does not assume the original medium/high seed names or prices remain unchanged. It selects the least-expensive enabled positive-cost profile to minimize temporary test grants while still exercising reservation and settlement; an intentionally free-only configuration is also supported. The phase verifies the settled AI Seed terminology and fruit explanation, organization and global personal balances, transactional reservation and one-time settlement at the selected profile's authoritative cost, provider usage recording, and protected candidate preview access.

Image selection follows the current two-stage page workflow. The POST /api/ai/jobs/<job_id>/accept action means Use This Image: it returns pending_save=true and leaves both job and candidate in awaiting_review. The regression then reloads the recipe's current editable content and submits ai_image_candidate_id through POST /save-edit-recipe-content/<recipe_id>. That normal recipe save promotes the candidate to accepted, updates the recipe image, and queues one combined cache rebuild without creating another usage event or AI Seed charge. This distinction prevents the regression harness from mistaking a successful pending selection for a failed acceptance.

Migration/unit tests continue to verify the original seeded defaults separately. The regression adapter implements the same submission callback contract as production adapters, including provider request and response identifiers, so changes to the shared provider lifecycle are exercised by the deterministic phase. The adapter never calls an external provider and cannot incur OpenAI or Gemini charges. The temporary provider connection is encrypted while present and removed after the phase. The accepted image and accounting rows remain part of the test organization and are removed by normal regression cleanup. Restart resume uses the completed recipe-save acceptance step as the phase sentinel so a completed AI generation is not replayed. Failed generation steps include the persisted job error category and message to make provider-contract regressions diagnosable from the exported regression CSV.

AI administration, API Tester, and Database coverage

Regression suite 2026.07.30.4 extends the deterministic AI Recipe Images phase and System Admin tool checks without calling OpenAI or Gemini.

The suite now verifies:

the organization-scoped AI Usage JSON API returns the generated usage event;
Org Admin usage responses omit provider request identifiers and provider input/output/image token counts;
the dedicated AI Usage administration page renders the filters, server-side paging controls, and privacy-appropriate columns;
a System Admin can open the all-organization AI Usage JSON report and receives paging and summary metadata;
the API Tester spec exposes the AI Usage report, reconciliation CSV, organization balance, on-demand adjustment history, and personal balance endpoints;
Database Admin lists all 12 AI configuration/accounting/job/candidate/usage/state resources;
the AI Usage Events Database resource loads through normal server-paged resource handling;
protected provider usage and metadata JSON columns stay hidden;
the existing deterministic provider flow still reserves, settles once, records provider usage, exposes the protected candidate, and accepts the image through the normal recipe/cache workflow.

Focused tests remain responsible for exhaustive filter combinations, SQLite/PostgreSQL query contracts, organization balance facets, on-demand adjustment-dialog behavior, System Admin-only provider diagnostics, global-personal privacy, and the immediate billing-anchor overlap/double-grant fix. The immediate anchor tests assert that the current bucket is retimed in place, spent/reserved amounts are preserved, a repeated save does not add another allotment, and an existing overlap rolls back with a useful error.

The automated harness now uses the normal Chrome Extension API-token routes to verify that:

a 19-digit TikTok video ID remains exact through deterministic preview and resolves complete supplied evidence without AI
an explicitly unmatched Pinterest active pin is rejected instead of importing stale metadata from another pin
Database Admin exposes social import and AI media diagnostics while protected payload/hash/path fields remain hidden
the social recipe capture resource loads on both SQLite and PostgreSQL when the integer schema_version filter options are derived
API Tester publishes the complete social recipe preview, capture, status, AI preflight/reuse/options/job/import/cancel contract

Browser-only pixel behavior remains covered by focused extension contract tests for exact-pin DOM scoping, editor-state clearing, rendered video-content geometry, narrow portrait video handling, thumbnail display, and suppression of blank/manual/import/retry actions until automatic resolution is exhausted.

On this page

🧭 Start Here

🚀 Deploy Blackcap

🛠️ Administer Blackcap

🍽️ Use Recipes

📅 Plan Meals

🛒 Use Shopping Lists

🧺 Manage Kitchen Inventory

🖥️ Use Displays

🧑‍🍳 Cook with Let’s Cook

🤖 Use and Administer AI

🧩 Use the Chrome Extension

🎮 Play Games

🔌 Integrations

⚙️ Develop Blackcap

⚠️ Troubleshoot Blackcap

Regression Testing