Data Retention and Purge

Audience: System Admin, Org Admin, Support, Developer
Related: Configuration · Database Administration · Performance and Job Status · Background Jobs and Schedules

Blackcap uses a domain-aware cleanup framework rather than generic table deletion. Each cleanup group understands its own organization scope, dependencies, restore window, file artifacts, retention policy, audit behavior, and safety restrictions. The same behavior supports SQLite on Raspberry Pi and PostgreSQL on GCP through the centralized database service and provider adapters.

Why cleanup is domain-aware

Recipes, shopping lists, meal plans, Let’s Cook sessions, shares, search indexes, generated assets, display assignments, access activity, and background jobs are related. Permanently removing a recipe, for example, may require Blackcap to detach historical meal-plan rows, remove or detach shopping-list references, end or protect Let’s Cook dependencies, remove search and edit rows, and delete only safe cache artifacts.

The implementation follows the normal service boundary:

Admin route or scheduled runner
→ feature/domain purge service
→ centralized database service and repositories
→ active SQLite or PostgreSQL provider
→ safe filesystem cleanup where the resource owns artifacts

Routes remain thin. Purge services use explicit-column queries, organization-scoped transactions, and idempotent candidate evaluation. Cleanup must never be implemented as an unreviewed DELETE FROM ... route.

Two cleanup paths

Manual Data Cleanup

Manual cleanup is available from:

Administration → Data Cleanup

The page lists eligible records by content group, shows whether each item is already Due under the configured retention policy or is Not due yet, supports a preview, and requires typed confirmation such as:

PURGE 3 RECORDS

An authorized administrator may deliberately purge a supported item before its scheduled retention date after preview and confirmation. Manual cleanup still enforces the feature’s dependency and organization-scope rules.

Normal organization scoping applies:

System Admin in Default Organization context → Default/platform-visible groups
System Admin switched to another organization → that organization’s scoped groups
Org Admin → their Active Organization only and only permitted groups

Platform lifecycle groups such as self-deleted accounts, disabled/deleted organizations, and completed regression environments have additional System Admin and Default Organization restrictions.

Scheduled purge

Scheduled purge is a platform policy configured at:

Administration → Configuration → Platform → Purge

The managed schedule runs daily at 2:00 AM America/Chicago from /etc/cron.d/blackcap:

0 2 * * * <service-user> /usr/bin/python3 <app-dir>/tools/run_scheduled_task.py blackcap-daily-purge ...

Do not add a duplicate job with crontab -e; /etc/cron.d/blackcap is a system-wide file and includes the execution user. The installer owns that file.

The runner can also be invoked explicitly:

# Raspberry Pi examples
/home/pi/inky_env/bin/python3 /home/pi/Blackcap-Pi/run_daily_purge.py --dry-run
/home/pi/inky_env/bin/python3 /home/pi/Blackcap-Pi/run_daily_purge.py --delete

# GCP examples
/opt/blackcap_env/bin/python3 /opt/Blackcap/run_daily_purge.py --dry-run
/opt/blackcap_env/bin/python3 /opt/Blackcap/run_daily_purge.py --delete

Scheduled purge starts conservatively: purge.enabled defaults to false and purge.dry_run defaults to true. Enabling the schedule and disabling dry-run are separate decisions.

Complete platform purge policy

The Configuration page is the authoritative UI for these platform-scoped settings. Defaults below are the repository defaults; existing installations can have different database-backed values.

Cleanup group	Settings and defaults	What is eligible	What is protected
Scheduled worker	purge.enabled=false; purge.dry_run=true	Controls whether the daily worker evaluates or deletes	Manual preview remains available independently
Self-deleted accounts	purge.account_lifecycle.enabled=true; purge.account_lifecycle.self_delete_delay_hours=24	Accounts that confirmed self-delete and completed the minimum safety delay	Delay values below 24 hours are treated as 24; dependency and last-user rules still apply
Disabled/deleted organizations	purge.organizations.enabled=true	Non-default organizations whose lifecycle purge date has arrived and that pass lifecycle checks	Default Organization and organizations with blocking active users/dependencies
Shopping Lists	enabled; completed 5 days; archived/deleted 5 days	Completed, archived, or deleted list graphs	Active lists and another organization’s lists
Recipes	enabled; archived 30 days; deleted 30 days	Recipes already archived or deleted	Active recipes and recipes blocked by active/paused Let’s Cook sessions
Unused AI assets	enabled; global 7 days; use-case override off	Generated candidates not accepted into durable content, using use-case retention unless globally overridden	Accepted recipe images, active job assets, recovery-required assets, and other organizations’ assets
Kitchen Inventory	enabled; archived 14 days	Archived inventory items after the restore window	Active items and other organizations’ items
Background jobs	enabled; successful 15 days; failed/cancelled 30 days	Terminal durable job rows	Queued/running jobs and retained Job Status action requirements
Performance metrics	enabled; details 30 days; aggregates 90 days	Detailed performance events and older daily aggregates	Newer details and aggregates; live resource snapshot is not a retained event unless explicitly recorded
Regression test runs	enabled; 30 days	Completed run details, reports, request logs, and steps after generated environments are gone	Runs still linked to a live diagnostic/regression organization
Audit log	enabled; 30 days	Organization-scoped audit rows	A new cleanup-completion audit event is written after old rows are removed
Display assignment batches	enabled; 7 days	Terminal assignment-batch tracking rows	Current/scheduled assignments and queued/running batches
Setting audit log	enabled; 30 days	Scoped configuration change history	Rows newer than retention; platform rows remain Default-context managed
Data Cleanup history	enabled; 7 days	Completed data_purge_runs and data_purge_run_items	Running cleanup records; the cleanup operation creates a new history record
Voice assistant command log	enabled; 7 days	Sanitized Alexa/Google Home request/response troubleshooting history	Provider credentials, account links, tokens, and active connections
Recipe share audit	enabled; 7 days	Share audit events	Active share, recipient, and consumption records
Revoked API Tokens	enabled; 7 days	Revoked token records	Active API Tokens
Access IP/location activity	enabled; 90 days	Raw access-source rows, location detail, and GeoIP cache detail that reached retention	Current access summaries needed by the UI; credentials are never stored in this data
Completed user invites	enabled; 60 days	Accepted, expired, and revoked invites	Pending invites
Backup run history	enabled; 30 days; keep last 10	Terminal backup run/status metadata	Newest configured rows, latest successful records, and backup ZIP files
Cloud backup state history	enabled; 5 days	Old embedded status/history entries	Current state and provider-token rows
Restore staging	enabled; 7 days; keep last 3	Temporary extraction/staging directories and files	Newest configured entries and active restore work
Let’s Cook operational data	enabled; 7 days	Completed/cancelled/failed session graph: sessions, recipes, steps, ingredients, timers, display state, and deferrals	Active/paused sessions
Let’s Cook compact history	enabled; 30 days	Summary/history rows	Recent cooking history within retention

The policy loader also retains backward-compatible reads for older Let’s Cook setting names, but canonical documentation and new configuration should use purge.lets_cook_session_data.* and purge.lets_cook_session_history.*.

Cleanup groups and dependency behavior

Shopping Lists

A Shopping List is purged as a list graph, including its eligible items and source relationships. Scheduled cleanup processes Shopping Lists before Recipes so list/source links are detached before a recipe parent can be removed. Active lists are not selected.

See Shopping Lists.

Recipes and cache artifacts

Recipes must already be archived or deleted. Before deleting the recipe parent, cleanup:

• preserves historical meal-plan rows by detaching the recipe and converting them to manual entries where required;
• removes or detaches shopping-list item/source references;
• removes ended Let’s Cook operational graphs that reference the recipe;
• blocks deletion when an active or paused Let’s Cook session still references it;
• removes edit content, ingredients, tags, search-index rows, share metadata that belongs to the recipe lifecycle, and cache-artifact metadata;
• deletes only repository-controlled recipe/cache files that pass safe-path checks.

The operation must be transactional for database state and idempotent when retried after a partial filesystem failure.

Kitchen Inventory

Deleting a Kitchen Inventory item first archives it. Archived items stay restorable for the configured window, 14 days by default. Manual Data Cleanup can permanently remove selected archived items earlier after preview. Scheduled purge selects only status='archived' rows at or before the cutoff and remains organization-scoped.

See Kitchen Inventory.

Unused AI assets and temporary media

AI candidate assets use the retention configured by their registered AI use case unless the platform-wide override is enabled. The global value is also the fallback when an old use case no longer exists. Presentation labels come from use-case/job metadata rather than blindly inferring the asset type from a filename extension.

Protected assets include accepted recipe images, candidates attached to active jobs, recovery-required files, and assets owned by another organization. The scheduled maintenance pass also removes expired temporary social-recipe media through the social AI service. Monthly AI allotment maintenance runs in the same daily maintenance flow but is not presented as destructive content purge.

See AI Seeds and Usage and AI Recipe Image Generation.

Background jobs and Job Status

Terminal job rows are retained long enough for operational review: successful jobs default to 15 days and failed/cancelled jobs to 30 days. Queued and running jobs are never selected by the terminal-job policy. Review, ignore, retry, and stale-state actions remain available only while the underlying job/action rows are retained.

See Performance and Job Status.

Performance metrics

Performance cleanup is split intentionally:

• performance_events contains recent detailed events used for diagnostics and current graphs;
• performance_aggregates contains daily rollups retained longer for lower-cost trends.

The default windows are 30 days for detail and 90 days for aggregates. Cleanup records its own timing through a low-noise performance operation without making the business cleanup fail if metrics recording fails.

Access activity, GeoIP, displays, login, API, and voice

Access activity cleanup covers retained raw IP/location source rows and GeoIP cache detail. It does not delete authentication records, display records, API Tokens, or voice provider connections. Compact display and voice country/locale rollups are sanitized and use their own expiry/purge fields where present.

A server-observed IP can represent a proxy or provider edge rather than the human user. Trusted-proxy configuration and the GeoIP guide explain how to interpret it. Alexa-observed source IPs in particular may represent provider infrastructure.

See GeoIP and Access Location.

Revoked API Tokens and completed invites

Revoked API Tokens remain briefly visible so an administrator can verify revocation and review recent activity. Active tokens are never selected. Accepted, expired, and revoked invite rows remain for lifecycle review; pending invites are protected.

Account lifecycle

A user who confirms self-delete enters the normal account lifecycle process. Permanent removal does not occur until the configured safety delay has elapsed, with a hard minimum of 24 hours. Cleanup uses the account lifecycle service so ownership transfer, last-user, organization, membership, audit, and related-resource rules are evaluated rather than bypassed.

Organization lifecycle

Disable and Delete schedule a non-default organization for later permanent cleanup according to the organization lifecycle model. The purge service verifies eligibility and active-user/dependency safeguards. The Default Organization is never eligible. Diagnostic and regression organizations use the same lifecycle services rather than a special raw-delete path.

See Organizations and Diagnostic Clones.

Regression and diagnostic history

A regression run becomes eligible only after its generated environment has been removed or fully lifecycle-purged. If any regression_test_organizations marker still resolves to a live organization, the run is protected so Test Organizations can continue linking to its source run. Purging a run removes its steps and DB-backed report/request/performance payloads; it does not delete a live organization.

Audit and operational history

Separate groups retain separate authority:

• Audit Log: removes old organization-scoped audit rows; other cleanup groups record their action but do not erase audit history incidentally.
• Display Assignment Batches: removes terminal batch tracking, not active/default/scheduled display assignments.
• Setting Audit Log: removes old platform or organization setting-change rows according to scope.
• Data Cleanup History: removes completed cleanup run/item records, never a running cleanup.
• Voice Assistant Command Log: removes sanitized command diagnostics, not linked accounts or credentials.
• Recipe Share Audit: removes audit events while preserving active share lifecycle records.

Backup metadata, provider state, and restore staging

Backup cleanup deliberately distinguishes metadata from backup artifacts:

• old organization_backup_runs metadata is eligible while the newest configured records and latest successful records remain;
• embedded cloud-backup status history can be trimmed without deleting the current provider state;
• provider OAuth token rows are removed only by disconnect flows, not retention;
• restore staging directories/files are temporary and can be removed after retention while keeping the newest configured entries;
• local, Dropbox, and Google Drive backup ZIP retention is controlled by Backup/provider policy, not the backup-run-history purge rule;
• GCP PostgreSQL data is protected through Cloud SQL backup policy rather than embedding the live database inside a platform ZIP.

See Backups.

Let’s Cook data versus history

Operational session data and compact history are separate groups. The operational graph includes the session, session recipes, steps, ingredients, timers, display state, and scheduled-content deferrals. Children are removed before the parent. Compact history survives longer and no longer depends on the operational parent row.

See Let’s Cook.

Preview, execution, and records

Every manual or scheduled content purge records a run in:

• data_purge_runs
• data_purge_run_items

The run captures scope, operation, source, dry-run/delete mode, status, counts, timing, a safe summary, and an error message when appropriate. Activity/audit surfaces link to cleanup details without retaining a copy of every deleted row or secret-bearing payload.

Dry-run uses the same candidate and dependency logic but does not delete database rows or files. Repeated evaluation should return the same candidates until state changes. A failed cleanup should leave enough run/error information for Job Status, Performance, logs, or the owning feature page to explain what happened.

Permissions and scope

• System Admins manage platform policy and can operate in the Default or switched organization context.
• Org Admin access is limited to the Active Organization and granted cleanup groups.
• Platform rows must not appear in an arbitrary organization context.
• A selected public ID is resolved again inside the transaction; the server does not trust browser-provided organization ownership.
• Cross-organization cleanup is rejected even when an ID exists.

Monitoring and troubleshooting

Use these surfaces together:

1. Data Cleanup — candidates, preview, typed confirmation, and run details.
2. Job Status → Maintenance — queued/running/failed maintenance jobs and Scheduled Operations status.
3. Performance — purge duration, failure category, and resource/queue trends.
4. Activity/Audit — who initiated cleanup and what content group/count was affected.
5. Database Admin — provider/schema health and data problems; do not use arbitrary SQL as a substitute for the domain purge path.
6. Journald/cron status — deployment-level failures before the runner can create a database run record.

Useful checks include:

sudo cat /etc/cron.d/blackcap
sudo systemctl status cron.service --no-pager
sudo journalctl -u cron.service -n 200 --no-pager

On GCP, also confirm inky_admin.service and cloud-sql-proxy.service; on Pi, confirm the Pi application service and filesystem permissions for any artifact-owning cleanup group.

Developer requirements

When adding a purgeable feature:

1. Add a stable content group and friendly label.
2. Define platform settings and conservative defaults when scheduled retention is appropriate.
3. Implement candidate selection through repositories/database service with explicit columns and organization scope.
4. Implement preview and deletion from the same dependency model.
5. Protect active/in-use resources and cross-organization records.
6. Remove owned files only through safe, rooted path checks.
7. Write audit and cleanup-run summaries without secrets or full deleted payloads.
8. Add SQLite and PostgreSQL tests, including repeat execution and rollback/failure behavior.
9. Add the setting to Configuration and update this document.
10. Add Performance/Job Status visibility when the operation can be slow or asynchronous.

Do not create a new raw cleanup route, duplicate cron entry, or feature-specific retention implementation that bypasses the centralized policy and audit model.