Social Recipe Import

Social Recipe Import supports URL submission and extension-assisted capture for YouTube, TikTok, Instagram, Pinterest, and recognized related URL forms. The pipeline normalizes URLs, discovers available written sources, captions, transcripts, page metadata, and extension evidence, then applies deterministic extraction before optional AI enhancement.

Imports may contain one or more recipe candidates. Incomplete ingredients or instructions, uncertain source evidence, or conflicting candidates are surfaced for review. Users can edit the result, build cache artifacts, and refresh from the original source without treating AI output as authoritative source evidence.

Scope

Blackcap now treats supported social-media links as an extension of the existing Add Recipe by URL experience. The user pastes one URL and presses the existing Save action. Ordinary recipe webpages continue through the existing synchronous recipe URL path. Supported YouTube, TikTok, Instagram, Facebook Reel, and Pinterest URLs branch after Save into an organization-scoped background import.

Once a social recipe is accepted, it is an ordinary Blackcap recipe. It appears in the normal Recipe Library, uses the normal editable-content model and editor, builds the normal cache artifacts, and remains available to displays, Let’s Cook, meal planning, shopping lists, kitchen inventory, and sharing.

User flow

Normal recipe webpage

The user pastes a URL into the existing Add Recipe form or Chrome Extension.
URL classification returns normal_webpage.
Blackcap continues through the existing normal recipe path.

The user pastes a social URL into the same field and presses Save, or captures the active page with the Chrome Extension.
Blackcap normalizes the provider URL and content ID.
Blackcap creates or reuses an organization-scoped social import and durable background job.
The request returns HTTP 202 with a status-page URL instead of pretending a recipe already exists.
The worker collects public metadata and/or normalized extension evidence, follows explicit written-recipe links through the safe-fetch service, and assembles candidates.
One high-confidence candidate becomes a normal editable recipe automatically.
Multiple candidates wait for selection.
Incomplete evidence requests a Chrome Extension recapture or displays a review warning.

Source refresh versus cache rebuild

These are deliberately separate operations:

Rebuild from saved recipe content uses the current editable Blackcap recipe and does not contact the social source.
Refresh from social source creates a temporary candidate. The saved recipe is unchanged until the user explicitly selects one candidate and confirms replacement.
Keep my current recipe closes the source-refresh draft without changing recipe content.

Bulk Refresh Cache rebuilds social recipes from saved editable content. It does not silently re-import the social source.

Architecture

Admin/mobile URL form ─┐
                       ├─ URL classifier ─ normal webpage ─ existing recipe URL flow
Chrome quick URL ──────┤
                       └─ supported social source
Chrome page capture ────────────────┐
                                    v
                         Social import service
                         ├─ organization-scoped repository
                         ├─ background job
                         ├─ provider adapter registry
                         ├─ safe external fetch
                         ├─ written-source extraction
                         ├─ caption/candidate normalization
                         ├─ evidence and review issues
                         └─ normal recipe capture service
                                    v
                         Existing editable recipe + cache

Modular boundaries

social_recipe_url_service.py: deterministic provider/page detection and canonical content IDs.
social_recipe_provider_adapter_service.py: provider registry and legitimate public metadata collection.
social_recipe_capture_security_service.py: versioned capture-payload allowlisting, credential removal, sensitive-query stripping, and byte limits.
social_recipe_link_service.py: deterministic ranking of explicit recipe/newsletter/source links ahead of generic navigation links.
social_recipe_caption_service.py: deterministic parsing of labeled and common inline ingredient/instruction caption formats.
safe_external_fetch_service.py: all social discovery and linked-image retrieval through bounded public-network fetches.
social_recipe_written_source_service.py: Recipe JSON-LD, recipe microdata, and bounded written HTML extraction.
social_recipe_capability_service.py: provider-independent registration points for optional OCR, transcription, frame extraction, AI normalization, ranking, and supplemental source discovery.
social_recipe_import_service.py: organization-scoped state machine, jobs, candidates, normal recipe creation, refresh review, restart recovery, and retention.
social_recipe_import_repository.py: centralized explicit-column SQL only.
social_recipe_imports.py: browser-session status and action APIs.

Supported providers

Provider	Supported content	URL-only behavior	Extension behavior
YouTube	Videos and Shorts	Public page metadata and oEmbed; follows explicit public links when available	Captures visible description, links, title, creator metadata, and thumbnail evidence
TikTok	Video pages and supported shared URLs	Public metadata where available; often requires extension evidence	Expanded caption, public caption/transcript tracks when exposed, creator, links, images, and video presence
Instagram	Public Reels and posts	Public metadata where available; often requires extension evidence	Preferred path for expanded caption, creator/handle, profile/link-in-bio candidates, images, and visible links
Pinterest	Pins and supported short links	Prefers the explicit outbound source and public Rich Pin/page metadata	Captures title, description, images, outbound source links, and visible Pin evidence

Provider adapters use only public HTML, public metadata, public oEmbed, configured official mechanisms, and explicit source relationships. The implementation does not use copied cookies, social session tokens, private endpoints, or undocumented internal APIs.

URL normalization and deduplication

Equivalent URLs are normalized to a provider content ID before the import is created. Examples include YouTube watch/Short/share forms, Instagram Reel/post routes, TikTok video/share forms, Pinterest Pin/share forms, mobile host variants, and tracking parameters.

Deduplication is organization-scoped. Blackcap never reuses another organization’s import, candidate, capture, media, or recipe. A matching organization can reopen a completed import, continue an active import, retry a failed import, or add extension evidence to an existing URL-only import. A completed import is reusable only while its created recipe still exists and is active; archiving or deleting that recipe allows a fresh import instead of returning a stale recipe reference.

Provider isolation and capture diagnostics

Pinterest capture keeps evidence scoped to the identity-matched active Pin. It ignores stale Open Graph or canonical metadata and recommendation tiles, clears prior recipe-editor values before rendering evidence from a new source, and performs bounded readiness retries before accepting the capture.

TikTok capture preserves the exact content identity, including 19-digit video IDs. Thumbnail extraction uses the rendered video-content rectangle rather than the full player or page rectangle, including narrow portrait video and CSS object-fit and object-position behavior. Successful deterministic and AI conversions use the automatic thumbnail; blank manual fields and outer Import or Retry actions remain hidden until automatic resolution paths are exhausted.

Regression and administration coverage includes Pinterest exact-pin isolation, exact 19-digit TikTok identity preservation, rendered video-content geometry, social import state hierarchy, protected Database Admin resources for imports, captures, candidates, review issues, and AI media artifacts, plus the complete workflow in API Tester. Raw browser payloads, recipe or evidence JSON, storage paths, and hashes remain protected.

The social-recipe capture Database Admin resource uses PostgreSQL-safe distinct-filter discovery for the integer schema_version column. Optional dropdown metadata fails independently so a provider-specific filter query cannot prevent the protected resource itself from loading.

Import state machine

Primary states/stages are:

queued
processing / collecting_metadata
discovering_sources
awaiting_selection
review_required
extension_capture_required
completed
partial
failed
cancelled

Background jobs use the existing Blackcap job service. Workers run outside the original Save request, use bounded attempts, and are recovered after application restart. Redis and Celery are not required.

Candidate and evidence behavior

Candidate extraction priority is:

Explicit linked written recipe with Recipe JSON-LD.
Explicit linked written recipe using recipe microdata.
Bounded written HTML recipe sections.
Complete labeled recipe evidence supplied by the Chrome Extension.
Labeled caption/description recipe sections.
Browser-exposed caption/transcript tracks.
Explicitly confirmed AI video review for Instagram or TikTok, when configured and affordable.

The deterministic parser always runs first and does not consume AI Seeds. Neither deterministic extraction nor AI review may invent unsupported ingredients, quantities, units, temperatures, durations, servings, or instructions. Missing and conflicting fields remain visible in the review draft. Candidate records contain confidence, missing fields, conflicts, normalized recipe payload, compact evidence, and AI provenance when applicable.

A partial, transcript-derived, AI-preparation, or otherwise review-required candidate remains import evidence only. Blackcap does not create a recipe-library record until the user explicitly saves the reviewed recipe. Captures already reviewed in the extension may be created by that explicit Save action. When upgrading from an older build that prematurely created a draft, the final AI save updates that import-owned recipe in place instead of creating a duplicate.

A written source containing multiple Recipe JSON-LD objects produces multiple candidates. The user selects one, several, or all; each selected candidate becomes a separate normal Blackcap recipe.

Chrome Extension capture schema

Social page capture is versioned (payload_schema_version: 1) and includes only normalized useful evidence:

canonical and viewed URLs;
provider, content ID, and content type;
page/content title;
creator name and handle;
expanded visible caption/description where accessible;
recipe-relevant visible text;
Open Graph metadata;
bounded image and thumbnail candidates;
bounded outbound/profile/link-in-bio links;
video/image presence;
publication and capture timestamps;
extension version.

The extension does not capture or submit:

document.cookie;
social session tokens;
authorization headers;
social access tokens;
local/session storage;
browsing history;
the full page DOM;
unrelated page content.

The extension submits normalized text evidence quickly and polls the organization-scoped status API. When the user explicitly confirms an AI review, extension 1.7.0 can also capture a bounded recording of only the visible Instagram/TikTok/Facebook Reel video region, that tab's audio, and a bounded number of cropped frames. It validates the current social content ID before capture and sends media only to the configured Blackcap server. Provider credentials and AI calls remain server-side.

Retry from scratch

The Chrome Extension can start a fresh capture for the same social content even when a previous import or AI review succeeded. Retry from Scratch clears browser-side AI restoration state and submits control metadata that instructs the server to create a new import record. The flag is removed before capture evidence is stored. Existing recipes and historical imports remain unchanged until the user explicitly imports the newly reviewed candidate.

AI-assisted Instagram, TikTok, and Facebook Reel review

AI review is a paid fallback, not the default extraction path:

Deterministic extraction finishes first.
Blackcap requests enabled social_recipe_extraction profiles and current Organization/Personal AI Seed balances.
The user chooses a configured quality profile and one funding source, sees the exact cost and projected balance, and confirms the charge.
Blackcap reserves the selected AI Seeds before media capture.
The extension captures bounded media from the exact confirmed Instagram/TikTok post.
Gemini receives bounded video plus sampled frames; OpenAI transcribes the captured audio and analyzes the transcript with sampled frames.
Blackcap validates a strict provider-neutral recipe schema and creates a review-required candidate.
The user edits all recipe fields before import. The reviewed payload is authoritative.
Blackcap settles the reservation on success or releases it when capture/provider processing fails before a usable draft is produced.

The extension never calls Gemini or OpenAI directly. This keeps provider keys, model configuration, organization scoping, AI Seed accounting, audit history, retention, and provider usage reporting centralized in Blackcap.

Data model and migrations 211, 216, and 217

Migration 211_social_recipe_import_foundation.py adds:

social_recipe_imports
social_recipe_import_sources
social_recipe_import_captures
social_recipe_import_candidates
social_recipe_import_review_issues

All rows are organization-scoped and use public-safe IDs. Child records cascade from the import. Capture payloads are content-hash deduplicated. Migrations are idempotent and preserve migration history.

Migration 216_ai_social_recipe_extraction.py generalizes ai_jobs to recipe and non-recipe entities, links social imports/candidates to their AI job, and adds organization-scoped ai_media_artifacts for temporary bounded video and frame files. It also seeds disabled-by-use-case Economy, Standard, and Enhanced social-extraction profiles. A deployment must explicitly enable the use case and assign compatible provider connections/models before AI can be offered.

Migration 217_facebook_reels_and_social_retry.py enables Facebook Reel imports in installations that still use the original default provider list and updates the AI social-review description. It does not alter existing recipes, imports, AI jobs, or AI Seed ledger records.

Normal recipes are not stored in a separate social recipe table. The accepted deterministic or AI candidate is passed to the existing recipe capture/editable/cache services, with compact social and AI provenance retained in normal editable recipe metadata.

API contracts

Browser-session APIs

GET /admin/social-recipe-imports/<import_id> — compact status/review page.
GET /api/social-recipe-imports — active-organization import list.
GET /api/social-recipe-imports/<import_id> — active-organization status.
POST /api/social-recipe-imports/<import_id>/candidates/select — create selected recipes or explicitly replace a source-refresh target.
POST /api/social-recipe-imports/<import_id>/keep-current — resolve source refresh without changing the recipe.
POST /api/social-recipe-imports/<import_id>/cancel — cancel a non-completed import.
POST /api/social-recipe-imports/<import_id>/retry — retry a retryable terminal import.

Browser routes use existing recipes.view, recipes.create, and recipes.refresh_cache permissions.

Chrome Extension APIs

POST /api/recipes/add — ordinary URL quick add; branches to social import when the URL is supported.
POST /api/extension/social-recipe-imports/capture — versioned normalized active-page evidence.
GET /api/extension/social-recipe-imports/<import_id> — token-bound organization status.
POST /api/extension/social-recipe-imports/ai/options — compatible configured quality profiles, costs, balances, and capture limits.
POST /api/extension/social-recipe-imports/ai/jobs — final deterministic check, explicit profile/funding confirmation, job creation, and AI Seed reservation.
POST /api/extension/social-recipe-imports/ai/jobs/<job_id>/media — bounded multipart video/frame upload for the confirmed job.
GET /api/extension/social-recipe-imports/ai/jobs/<job_id> — safe job progress and review candidate.
POST /api/extension/social-recipe-imports/ai/jobs/<job_id>/import — import the exact user-reviewed candidate.
POST /api/extension/social-recipe-imports/ai/jobs/<job_id>/cancel — cancel before provider submission and release the reservation.

Extension endpoints require an authorized personal API token and bind to that token’s active organization context before reading or writing import data. Job reads and writes also enforce the creating user unless System Admin organization context authorizes otherwise.

Security

All server-side social and linked-source retrieval goes through safe_external_fetch_service.py. It enforces:

HTTP/HTTPS only;
no URL credentials;
standard public HTTP/HTTPS ports only;
blocklists for localhost/internal/link-local/metadata-style hostnames;
DNS resolution checks rejecting private, loopback, link-local, multicast, reserved, or unspecified addresses;
environment proxy disabling;
redirect limits and revalidation of every redirect target;
verification of the actual connected peer address to reduce DNS-rebinding risk;
response and decompressed-byte limits;
MIME checks;
connect/read timeouts;
no cookies or authentication headers;
response/session cleanup;
bounded HTML/JSON-LD parsing and script removal.

Captured thumbnail URLs now use the same safe-fetch service instead of an unrestricted direct request.

Configuration

The following DB-backed platform settings appear in Configuration under Recipe Capture:

social_recipe_imports.enabled
social_recipe_imports.enabled_providers
social_recipe_imports.url_only_enabled
social_recipe_imports.extension_capture_enabled
social_recipe_imports.max_capture_payload_bytes
social_recipe_imports.evidence_retention_days
social_recipe_imports.terminal_import_retention_days
social_recipe_imports.enable_ocr
social_recipe_imports.enable_transcription
social_recipe_imports.enable_frame_extraction
social_recipe_imports.enable_ai_normalization
social_recipe_imports.enable_candidate_ranking
social_recipe_imports.enable_optional_source_discovery

Legacy optional processor flags remain available for deterministic/deployment-specific extensions. The implemented paid AI fallback is configured through Configuration → Platform → AI using the social_recipe_extraction use case and the same quality-profile pattern used by recipe image generation. Each enabled profile selects a provider connection, compatible model, friendly quality label, AI Seed cost, duration/upload/frame limits, timeout, and provider-specific video/transcription settings. Only compatible models are shown.

The use case is disabled by default. Enabling it does not expose credentials to organizations or the Chrome Extension. The user still must confirm every charge and choose Organization or Personal AI Seeds. Blackcap never silently switches funding sources when the selected bucket is insufficient.

BLACKCAP_ENABLE_SOCIAL_RECIPE_IMPORTS remains an optional deployment-level emergency override for the base social-import feature. When present, it takes precedence over the DB-backed enabled setting.

Retention and purge behavior

Normalized extension capture evidence is removed after the configured evidence-retention period once the import is no longer active.
Old failed, cancelled, partial, or extension-required imports that are not linked to a created recipe are removed after the terminal-import retention period.
Compact provenance required by a created recipe is retained so source details and safe source refresh continue to work.
Organization purge explicitly removes social review issues, candidates, captures, sources, and imports in foreign-key-safe order on SQLite and PostgreSQL.
User purge detaches creator references from durable social import and capture rows.
AI media artifacts are size-, duration-, hash-, organization-, user-, and job-scoped.
Temporary video/frame files are removed immediately after processing, failure, or cancellation and are also covered by scheduled expiration cleanup.
Temporary media is not stored in normal recipe rows or ordinary backup payloads; durable AI usage/accounting records retain only safe metadata.

Pi and GCP behavior

Both SQLite/Pi and PostgreSQL/GCP use the same service, repository, migration, and AI Seed contracts. Basic deterministic extraction remains available without AI. When AI review is enabled, the Blackcap host temporarily accepts the bounded extension upload and calls the configured external provider; the Raspberry Pi does not run a local video model.

Profiles can be disabled independently by deployment. A Pi deployment may leave social AI review disabled while retaining deterministic social imports. GCP or another deployment may enable Gemini, OpenAI, or both without changing extension contracts or recipe storage.

Activity auditing

Safe audit events record the organization, user where available, provider, provider content ID, entry point, duplicate reuse, capture supplementation, cancellation, kept-current decisions, selected recipe IDs, and final action. Captions, cookies, tokens, raw transcripts, and full private page content are not written to activity metadata.

Deployment

Copy the changed files into the Blackcap deployment.
Update the existing Python environment from requirements.txt or requirements-cloud.txt if provider SDK versions changed.
Restart the application through the normal deployment process. Startup applies migration 216 and schedules recovery for both social imports and recoverable AI jobs.
In Configuration → Platform → AI, configure/test Gemini and/or OpenAI connections, assign them to enabled Social Recipe Extraction profiles, set AI Seed costs and media limits, then enable the use case.
Grant Organization or Personal AI Seeds as appropriate.
Reload the unpacked Chrome Extension or publish version 1.7.0. Chrome 116 or later is required for the offscreen capture workflow.
Validate one deterministic TikTok/Instagram import and one explicitly confirmed AI fallback for each enabled provider.
Run the focused tests and the normal Blackcap regression suite in the target environment.

No new systemd unit, cron entry, Redis instance, or Celery worker is required. AI review does require at least one configured external Gemini or OpenAI provider connection.

Manual provider validation

Use a separate test organization and confirm active organization context before each test.

YouTube

The Chrome Extension reads the expanded description and YouTube's public page metadata, unwraps explicit description redirect links, and asks Blackcap to preview the strongest linked written recipe before showing the review fields. A successful preview populates the recipe title, ingredients, instructions, times, yield, image, and written-source attribution while preserving the YouTube URL as the original social source.

Paste a normal watch URL and a Shorts URL into Add Recipe.
Confirm the same Save action returns a social import status page.
Test a video with an explicit written recipe link.
Test a video with no accessible recipe evidence and confirm extension capture is requested.
Capture the active YouTube page in the extension after expanding the description.

TikTok

Test a canonical /@creator/video/<id> URL and a supported shared URL.
Confirm URL-only import degrades to extension capture when public metadata is insufficient.
Expand the caption, capture the active page, and verify the existing import is supplemented rather than duplicated.

Transcript evidence states are deliberately explicit:

available: Chrome captured public caption segments. Blackcap attempts deterministic extraction and creates a review-required candidate only when it finds supported ingredient and instruction evidence.
unreadable: TikTok exposed a caption track, but Chrome could not read it. The deterministic failure explains the limitation and, when configured, offers an explicitly confirmed AI video review.
not_available: the video did not expose a usable caption track. Blackcap reports whether the written caption or any likely recipe link was found.
not_captured: the capture came from an older extension or omitted transcript evidence; the user is asked to reload the current extension and recapture.

A transcript-derived recipe never fills missing quantities, units, temperatures, durations, servings, or directions by inference. Partial evidence remains partial and is retained for review. Comments are not treated as a canonical recipe source.

Instagram

Test a public Reel and a public post.
Expand the caption before extension capture.
Test a caption recipe, a recipe in bio/newsletter link, and a post representing multiple recipes.
Confirm private/auth-only content fails safely without capturing the social login session.

Test a normal Pin, Rich Pin, and supported short Pin URL.
Confirm an explicit external recipe source is preferred.
Confirm a native image/video Pin without sufficient written evidence requests review/extension evidence.

Shared behavior

Verify a second submission in the same organization opens/reuses the import.
Verify another organization cannot read the import ID.
Edit the created recipe, rebuild its cache, and confirm edits remain.
Refresh from source and verify the recipe remains unchanged until explicit replacement.
Verify Let’s Cook, display rendering, meal planning, shopping-list generation, sharing, archive, and delete behavior use the normal recipe.

Known limitations and deferred functionality

Live provider pages were not contacted by the deterministic test suite; provider markup and access behavior must be validated in the deployment browser/network.
TikTok, Instagram, and Facebook commonly restrict useful URL-only metadata, so extension-assisted capture is expected to be the reliable path.
YouTube oEmbed supplies title/creator/thumbnail but not the full video description. Explicit links available in public page markup are used; otherwise the extension is requested.
AI provider credentials are never bundled; a System Admin must configure Gemini/OpenAI connections and decide which profiles and prices are available.
No real provider request can be exercised without configured keys, and live Instagram/TikTok capture remains dependent on current browser/provider page behavior.
The first implementation provides a detailed editable candidate rather than a field-by-field evidence merge editor.
The normal editor shows compact source/confidence details; a dedicated timestamped evidence viewer is deferred.
Media capture is intentionally bounded and user-confirmed; Blackcap does not crawl or download arbitrary social videos server-side.
Combined personal-plus-organization funding and automatic precedence are deferred. One selected source must cover the full displayed cost.

Completeness, partial AI enhancement, and duplicate protection

Social candidates are now evaluated by one shared completeness service rather than by non-empty fields alone. Ingredients and instructions are assessed independently. The evaluator returns explainable reasons such as ingredients_missing, instructions_missing, ingredients_low_confidence, and instructions_likely_misclassified. AI enhancement is offered whenever either essential recipe half is unusable; it is no longer necessary for both halves to be absent.

When deterministic extraction produced useful ingredients or useful instructions, that content remains the preferred evidence. AI fills or reorganizes the deficient portion, and conflicts are retained on the candidate for review instead of silently replacing stronger deterministic content.

Extension submissions enter a disabled working state immediately. Server-side protection remains authoritative: organization/provider content identity selects the reusable import, and a semantic capture hash prevents the same evidence from creating or re-queuing duplicate work. The network request key is deliberately excluded from that evidence hash, so retries with the same recipe content converge while genuinely richer capture evidence can still resume a waiting import.

AI-enhanced image lifecycle

Text extraction and image selection remain independent. AI enhancement carries forward deterministic image evidence including canonical recipe images, social images, video thumbnails, Pinterest images, and extension-captured data URLs. The selected candidate is passed through the normal recipe capture service, which remains responsible for safe retrieval, validation, managed storage, thumbnail creation, and normal PDF/PNG/cache generation. No paid AI image is generated as an implicit fallback.

Extension completion behavior

The extension uses Close only after a successful terminal state. While processing, review, or candidate selection remains active, the original action retains its working/cancel meaning. Pinterest and other coordinated extension windows use an extension-owned session relationship so Close can remove both owned surfaces without closing the user's original social tab or a newly opened Blackcap recipe tab.

Instruction quantity cleanup

AI social-recipe prompts tell providers to keep supported quantities in the ingredient list and omit them from steps when a step uses the full listed amount. Blackcap also applies a conservative deterministic cleanup after deterministic and AI evidence are merged. It removes an amount only when the exact ingredient-list amount appears once in the same instruction clause as that ingredient. Quantities remain when the ingredient is divided, reserved, used in portions, staged, or repeated. Temperatures, durations, pan sizes, speeds, and other non-ingredient numbers are never targeted. The cleanup summary is stored in raw_extraction_metadata.instruction_amount_cleanup_summary for diagnostics.

AI-assisted review

Purpose

Blackcap uses AI only as an explicitly confirmed fallback when deterministic Instagram or TikTok extraction cannot find both ingredients and instructions. Captions, visible text, browser-exposed transcript tracks, and legitimate linked recipe pages are always attempted first and do not spend AI Seeds.

The feature converts English social-video evidence into an editable Blackcap recipe draft. It does not automatically publish a recipe and does not silently infer unsupported quantities, units, temperatures, durations, servings, or safety guidance.

Why AI calls go through Blackcap

The Chrome Extension captures evidence but never calls Gemini or OpenAI directly. Blackcap owns provider credentials, compatible model selection, organization/user authorization, AI Seed accounting, provider usage tracking, temporary-media retention, strict response validation, and recipe creation. This also gives Admin, mobile, and extension workflows one accounting and provenance model.

Paid fallback workflow

Capture an Instagram Reel/post or TikTok video with the extension.
Blackcap runs deterministic extraction.
If a complete recipe is found, continue without AI.
If incomplete, Blackcap returns only enabled compatible Social Recipe Extraction profiles.
The user chooses a quality tier and Personal or Organization AI Seeds.
The popup shows provider, model, exact AI Seed price, available balance, projected balance, duration/upload/frame limits, and a privacy explanation.
The user confirms the charge.
Blackcap rechecks deterministic candidates, creates the AI job, and reserves the selected source.
The extension verifies that the original provider/content ID is still active, rewinds the visible video, records a bounded crop of the video element with the tab audio, and samples bounded cropped frames.
The extension uploads temporary media to Blackcap.
Blackcap calls the configured provider.
A strict recipe candidate is returned to the extension/Admin view.
The user reviews and edits every field, then explicitly imports the recipe.
The reviewed payload becomes authoritative and follows normal Blackcap editable-content/cache behavior.

Provider paths

Gemini

Gemini receives the bounded video plus sampled frames and the deterministic text evidence. The provider returns the shared strict JSON recipe schema. Blackcap deletes the uploaded Gemini file in a finally path.

OpenAI

OpenAI receives the captured WebM through the configured transcription model. Blackcap then submits the transcript, deterministic text evidence, and sampled frames to the configured structured-output model. The total upload limit is capped at 25 MB for this path.

Both providers return the same Blackcap candidate structure and always require review.

Quality profiles

Social Recipe Extraction profiles reuse the image-generation configuration pattern:

friendly quality name and description;
enabled/default/order;
AI Seed price;
provider connection;
compatible provider model;
maximum duration;
maximum upload size;
maximum frame count;
provider timeout;
Gemini media resolution;
OpenAI transcription model and frame detail.

Migration 216 seeds Economy, Standard, and Enhanced profiles at 2, 5, and 10 AI Seeds. The use case remains disabled until a System Admin assigns provider connections/models and enables it. Prices and limits are configuration, not hard-coded product promises.

Funding and accounting

The user selects exactly one source for each request:

Organization AI Seeds; or
Personal AI Seeds.

The selected source must cover the full cost. Blackcap does not silently switch sources. Combined funding and precedence are deferred.

The server reserves the displayed cost before media capture. It releases the reservation after cancellation, upload failure, missing media, provider failure, or another failure that occurs before a usable provider result is returned. Once a provider has returned a usable structured recipe, the request remains chargeable even if Blackcap must recover the editable candidate after an internal interruption. Saved results are recovered without submitting a second provider request. Provider token/usage measurements are stored separately from the Blackcap AI Seed price.

Privacy and media lifecycle

Capture is always initiated by the user after the price confirmation.
The active social provider and content ID must still match.
Only the visible video element is recorded; tab audio is included.
The microphone and other tabs are not captured.
Cookies, social access/session tokens, authorization headers, local/session storage, browsing history, and the complete DOM are excluded.
Media is organization-, user-, and job-scoped and stored under the runtime data directory.
Byte, duration, frame, MIME, path, and hash limits are enforced.
Temporary files are deleted after processing, failure, cancellation, or expiration.
Media is not copied into the recipe row or ordinary backup payloads.
Durable accounting retains safe provider/job metadata, not the video.

Data model

Migration 216_ai_social_recipe_extraction.py:

generalizes ai_jobs with entity type/id and safe input/result manifests;
adds latest_ai_job_id to social imports;
adds ai_job_id to social candidates;
creates ai_media_artifacts with job/organization/user scope and retention indexes;
seeds Social Recipe Extraction quality profiles;
preserves SQLite/PostgreSQL compatibility and migration history.

API summary

The extension uses token-bound organization endpoints for options, job creation, media upload, status, reviewed import, and cancellation. Every endpoint verifies organization and user ownership, and media upload has route-specific multipart limits without raising the global request limit for ordinary Blackcap routes.

Enabling AI-assisted review

Copy the updated files.
Update the existing Python environment from the normal requirements file when needed.
Restart Blackcap so migration 216 applies.
Configure/test Gemini and/or OpenAI under Configuration → Platform → AI.
Assign provider connections/models to Social Recipe Extraction profiles, set prices/limits, and enable the use case.
Grant Personal or Organization AI Seeds.
Reload extension 1.7.0 in Chrome 116 or later.
Validate deterministic-first behavior and one paid fallback for every enabled provider in a test organization.

No new systemd service, Redis instance, or Celery worker is required.

Current limitations

English social videos are assumed for this phase.
Combined Personal + Organization funding is deferred.
AI output uses a detailed editable candidate, not a timestamped field-by-field evidence merge UI.
Live provider quality depends on provider model behavior and current Instagram/TikTok browser markup.
Blackcap intentionally does not fetch arbitrary social video URLs server-side; media must come from the user-confirmed active tab.

On this page

🧭 Start Here

🚀 Deploy Blackcap

🛠️ Administer Blackcap

🍽️ Use Recipes

📅 Plan Meals

🛒 Use Shopping Lists

🧺 Manage Kitchen Inventory

🖥️ Use Displays

🧑‍🍳 Cook with Let’s Cook

🤖 Use and Administer AI

🧩 Use the Chrome Extension

🎮 Play Games

🔌 Integrations

⚙️ Develop Blackcap

⚠️ Troubleshoot Blackcap

Social import architecture and workflow