Raspberry Pi Deployment

Audience: Deployment Administrator Related: Application Updates · Reverse Proxy And Tls · Background Jobs And Schedules · Remote Pi Client · Deployment

A Raspberry Pi can run the Blackcap server, directly drive one local e-ink panel, act as a Remote Pi Client, or combine server and local-client responsibilities. The standard server path uses /home/pi/Blackcap-Pi, /home/pi/inky_env, SQLite by default, Gunicorn managed by inky_admin.service, and stunnel as the recommended Pi TLS front end.

The deployment script asks whether the Pi directly drives an e-ink display, whether hourly Menu refresh is needed, and whether the remote Let’s Cook watcher should run. Preserve those deployment choices when rerunning the scheduled-operations installer.

Hardware baseline

Blackcap is designed to run on low-power Raspberry Pi hardware. The established attached-panel path is a Raspberry Pi Zero 2 W or more capable Pi with the Waveshare 13.3-inch Type K e-ink display (epd13in3k, 960×680, four grayscale levels). A 16 GB or larger microSD card is a practical minimum for the application, database, cache artifacts, backups, logs, and operating system.

Other Raspberry Pi models can run the server or client, but attached-display support must use an implemented and tested display profile/driver. Do not assume that an arbitrary e-ink panel can be driven by the Type K path.

Standard installation

cd /home/pi/Blackcap-Pi
sudo ./tools/deploy_pi.sh

The script installs the Python dependencies from requirements.txt, creates or updates /home/<service-user>/inky_env, writes the application systemd unit, optionally installs the local e-ink client and Let’s Cook watchers, configures stunnel when selected, installs scheduled operations, and starts the service.

Service checks

sudo systemctl status inky_admin.service --no-pager
sudo systemctl status stunnel4 --no-pager
sudo cat /etc/cron.d/blackcap
sudo systemctl status blackcap_geoip_update.timer --no-pager

The attached-panel watcher is blackcap_lets_cook_timer_refresh.service. The optional remote-client watcher is blackcap_lets_cook_remote_client_refresh.service.

Pi TLS front end

For Raspberry Pi deployments, Blackcap should continue to use stunnel rather than moving to Nginx by default.

Browser / LAN / Alexa → HTTPS → stunnel on the Pi → Gunicorn 127.0.0.1:8080 → Blackcap

Decision: stay with stunnel on the Pi

Nginx is the preferred GCP front end, but stunnel remains the better default for the Pi because it is lighter, matches the existing Blackcap Pi deployment pattern, and works with Gunicorn's PROXY protocol mode so Blackcap can preserve real client IPs. Nginx is still a valid advanced option, but it should not be the standard Pi path unless a future Pi deployment needs more reverse-proxy features.

Script behavior

tools/deploy_pi.sh installs stunnel4 and openssl when the base Linux package step is accepted.

The script asks:

Use Gunicorn PROXY protocol for local stunnel/TLS forwarding? [Y]
Configure stunnel as the Pi TLS front-end? Recommended over Nginx for the Pi [Y]
Public stunnel HTTPS port [8443]
stunnel combined PEM certificate path [/etc/stunnel/blackcap.pem]

When stunnel is enabled, Gunicorn is bound to:

127.0.0.1:8080

The script writes:

/etc/stunnel/blackcap.conf

and enables/restarts stunnel4.

Certificate behavior

If the configured combined PEM certificate does not exist, the script creates a self-signed certificate. This is acceptable for private LAN testing. For public HTTPS, replace /etc/stunnel/blackcap.pem with a trusted certificate and private key combined into one PEM file.

A combined PEM contains the private key and certificate in the order required by stunnel. Keep the file root-protected and never place its contents in documentation or logs.

Expected stunnel config

The generated config is equivalent to:

[blackcap]
accept = 0.0.0.0:8443
connect = 127.0.0.1:8080
cert = /etc/stunnel/blackcap.pem
TIMEOUTclose = 0
protocol = proxy

The protocol = proxy line matches the Gunicorn service's PROXY protocol mode. If you disable PROXY protocol in the deploy script, do not use the generated stunnel PROXY protocol line.

Verification

sudo systemctl status inky_admin.service --no-pager
sudo systemctl status stunnel4 --no-pager
sudo ss -ltnp | egrep ':443|:8443|:8080'

Expected listener pattern:

0.0.0.0:8443      stunnel
127.0.0.1:8080    gunicorn

If using port 443 for Alexa account linking, the expected stunnel listener is 0.0.0.0:443 instead of 8443.

Gunicorn service operation

Blackcap Pi uses Gunicorn as the production WSGI server for the Admin UI.

Why Gunicorn?

The Flask built-in development server is intended only for development and testing.

Gunicorn provides:

• Better concurrency handling
• Improved stability
• Cleaner systemd integration
• Better production readiness
• Improved logging support

---

Install

/home/pi/inky_env/bin/pip install gunicorn

---

Restart the Service

sudo systemctl daemon-reload
sudo systemctl restart inky_admin.service

---

View Logs

journalctl -u inky_admin.service -f

---

Common Commands

Stop the service

sudo systemctl stop inky_admin.service

Start the service

sudo systemctl start inky_admin.service

Restart after a unit-file change

sudo systemctl restart inky_admin.service

View status

systemctl status inky_admin.service

---

Notes

Gunicorn logs may appear different from the Flask development server.

You will now see:

• Gunicorn startup logs
• HTTP access logs
• Application logs interleaved together

This is expected behavior.

Headless memory and reboot diagnostics

This runbook documents the recommended Raspberry Pi setup for a Blackcap device that runs with an attached e-ink display, no normal HDMI/DSI desktop display, and no camera. These changes reduce avoidable memory pressure on small Raspberry Pi models, especially the Raspberry Pi Zero 2 W, and make unexpected reboots diagnosable after the fact.

The short version:

• Boot to command line instead of the desktop.
• Disable the vc4-kms-v3d graphics overlay when the Pi has no normal display or camera dependency.
• Enable persistent journald storage so journalctl -b -1 works after a reboot.

These changes are particularly useful for Blackcap because the app's heavy work is mostly Python, Flask/Gunicorn, image rendering, recipe cache generation, and e-ink display updates. Those workloads need normal system RAM more than a large graphics/display CMA reservation.

When this applies

Use this guidance for a dedicated Blackcap Pi that:

• Uses a local e-ink display driven by Blackcap scripts.
• Does not need the Raspberry Pi desktop UI.
• Does not use HDMI/DSI display output for normal operation.
• Does not use the Raspberry Pi camera stack.
• Is memory constrained or has unexplained hangs/reboots under load.

Do not apply the vc4-kms-v3d change blindly to a Pi that needs a normal desktop display, DRM/KMS display output, camera/display overlays, or GPU/3D acceleration. For those devices, test carefully or leave the graphics overlay enabled.

Background: CMA and vc4-kms-v3d

CMA is the Linux Contiguous Memory Allocator. It reserves a physically contiguous memory pool during early boot so hardware drivers can allocate large contiguous buffers later. This matters for hardware paths such as display, camera, GPU/DRM, and DMA-style workloads.

On a small Pi, a large CMA pool can materially reduce the normal memory available to Blackcap. In one observed Raspberry Pi Zero 2 W case, the system booted with:

CmaTotal:         262144 kB
Memory: 139268K/458752K available ... 262144K cma-reserved

That means 256 MiB was reserved for CMA on a 512 MiB-class device.

The Raspberry Pi config line that caused this in the observed installation was:

dtoverlay=vc4-kms-v3d

vc4-kms-v3d enables the Raspberry Pi kernel graphics driver stack. That is useful for normal display/desktop graphics, but it is usually unnecessary for a headless Blackcap e-ink device.

After disabling that one overlay, the same class of device booted with:

CmaTotal:          65536 kB
Memory: 335316K/458752K available ... 65536K cma-reserved

That reduced CMA from 256 MiB to 64 MiB and made much more RAM available during boot.

CmaFree can be low and fluctuate. That is not automatically a problem. Linux can use parts of the CMA region for movable pages. The main thing to watch for this tuning is CmaTotal.

Step 1: Boot to command line instead of desktop

The Blackcap service runs under systemd and does not need the graphical desktop. Booting to command line removes desktop components such as LightDM, labwc, panel processes, file manager desktop integration, and related graphical services.

Option A: raspi-config

Run:

sudo raspi-config

Then choose the boot option that starts the system at the command line instead of the desktop. The exact menu labels can vary by Raspberry Pi OS version.

Option B: systemd commands

The same direction can be applied directly:

sudo systemctl set-default multi-user.target
sudo systemctl disable lightdm
sudo reboot

After reboot, verify:

systemctl get-default
ps aux | grep -Ei 'lightdm|labwc|wf-panel|pcmanfm|wayfire|wayland|Xorg' | grep -v grep
free -h
swapon --show

Expected:

multi-user.target

There should be no normal desktop processes such as lightdm, labwc, wf-panel-pi, or pcmanfm --desktop.

This change reduces active memory and swap use, but by itself it may not reduce the boot-time CMA reservation.

Step 2: Disable the unnecessary VC4 KMS graphics overlay

Back up the Raspberry Pi firmware config before editing it:

sudo cp /boot/firmware/config.txt /boot/firmware/config.txt.bak.$(date +%Y%m%d-%H%M%S)

Edit the config:

sudo nano /boot/firmware/config.txt

Find:

dtoverlay=vc4-kms-v3d

Comment it out:

## dtoverlay=vc4-kms-v3d

For this specific optimization, keep the change narrow. Do not change several display or camera options at the same time unless you are intentionally testing those separately.

Reboot:

sudo reboot

Verify CMA after reboot:

grep -Ei 'CmaTotal|CmaFree|MemTotal|MemAvailable|MemFree|SwapTotal|SwapFree' /proc/meminfo
free -h
dmesg | grep -i cma

A successful headless Blackcap tuning should show CmaTotal much lower than 256 MiB. On the observed Pi Zero 2 W setup, it dropped to 64 MiB:

CmaTotal:          65536 kB
[    0.000000] Reserved memory: created CMA memory pool ..., size 64 MiB

Also verify Blackcap and the e-ink workflow still work:

sudo systemctl status inky_admin.service --no-pager
sudo journalctl -u inky_admin.service -n 100 --no-pager

Then exercise the normal Blackcap display/render path from the Admin UI or the relevant command-line display script.

Rollback

If the local environment unexpectedly needs the graphics overlay, restore the backup:

sudo cp /boot/firmware/config.txt.bak.YYYYMMDD-HHMMSS /boot/firmware/config.txt
sudo reboot

Replace YYYYMMDD-HHMMSS with the timestamp from the backup file created earlier.

Step 3: Enable persistent logs for previous-boot diagnosis

Raspberry Pi OS may default systemd journal storage to volatile behavior. If the Pi unexpectedly reboots, volatile logs are lost and commands such as this will not help:

sudo journalctl -b -1 -e --no-pager

Enable persistent storage using a drop-in file under /etc:

sudo mkdir -p /etc/systemd/journald.conf.d
sudo mkdir -p /var/log/journal

sudo tee /etc/systemd/journald.conf.d/90-blackcap-persistent.conf >/dev/null <<'EOF_JOURNALD'
[Journal]
Storage=persistent
Compress=yes
SystemMaxUse=200M
SystemKeepFree=500M
RuntimeMaxUse=50M
MaxRetentionSec=14day
SyncIntervalSec=30s
EOF_JOURNALD

sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
sudo journalctl --flush
sudo sync

The limits above keep logs useful without letting them grow without bounds on the SD card.

Verify the configuration and storage location:

systemd-analyze cat-config systemd/journald.conf | sed -n '/^\[Journal\]/,/^\[/p'
ls -ld /var/log /var/log/journal
sudo find /var/log/journal -maxdepth 3 -type f -ls
findmnt /var/log /var/log/journal
df -h /var/log /var/log/journal
journalctl --disk-usage
journalctl --list-boots

The important signs are:

• Storage=persistent appears in the merged journald config.
• /var/log/journal exists.
• /var/log/journal is on the real root filesystem, not tmpfs.
• journalctl --disk-usage reports persistent journal usage.

Perform one intentional reboot to prove previous boot history works:

sudo reboot

After the Pi returns:

journalctl --list-boots
sudo journalctl -b -1 -n 50 --no-pager

Expected output should include both the previous and current boot:

 -1 previous-boot-id ...
  0 current-boot-id  ...

The previous boot should show a clean shutdown sequence for an intentional reboot, including lines similar to:

Reached target shutdown.target - System Shutdown.
Finished systemd-reboot.service - System Reboot.
Reached target reboot.target - System Reboot.
Journal stopped

That clean sequence becomes the comparison point for unexpected reboots.

After an unexpected reboot

Run these first:

journalctl --list-boots
sudo journalctl -b -1 -e --no-pager
sudo journalctl -b -1 -p warning..alert --no-pager

Then search the previous boot for common failure patterns:

sudo journalctl -b -1 -k --no-pager | grep -Ei 'watchdog|panic|oops|oom|killed process|under-voltage|undervoltage|thrott|thermal|mmc|ext4|i/o error|reset|hang|blocked for more than|soft lockup|hard lockup'

Also search all previous-boot messages for reboot and shutdown clues:

sudo journalctl -b -1 --no-pager | grep -Ei 'watchdog|systemd-reboot|systemd-shutdown|reboot.target|shutdown.target|Journal stopped|oom|killed process|blocked for more than|hung task'

Interpretation:

• A clean software reboot usually shows systemd-shutdown, systemd-reboot.service, reboot.target, and Journal stopped.
• A power loss, kernel hang, hard crash, or watchdog reset often leaves the previous boot log ending abruptly without the clean shutdown sequence.
• oom or Killed process points toward memory pressure.
• watchdog, hung task, soft lockup, hard lockup, or blocked for more than supports a hang/watchdog theory.
• mmc, ext4, or I/O error points toward SD card or filesystem/storage issues.
• under-voltage, throttled, or thermal messages point toward power or temperature problems.

For power and temperature state after the reboot:

vcgencmd get_throttled
vcgencmd measure_temp

throttled=0x0 means no undervoltage or throttling has been recorded since the current boot. It does not prove the previous boot had no power event, but it is still useful context.

Optional: inspect or tune the watchdog timeout

Blackcap Pi deployments may use the Raspberry Pi hardware watchdog through systemd. If the OS stops responding long enough, the watchdog can reset the device. That is helpful for unattended recovery, but an overly short timeout can hide the underlying hang before enough logs are written.

Inspect watchdog-related configuration:

grep -RniE 'RuntimeWatchdogSec|ShutdownWatchdogSec|WatchdogSec' \
  /etc/systemd/system.conf \
  /etc/systemd/*.conf \
  /etc/systemd/system \
  /lib/systemd/system 2>/dev/null

Inspect the current boot for watchdog initialization:

journalctl -b 0 --no-pager | grep -Ei 'watchdog|RuntimeWatchdog'

A 2-minute watchdog timeout is a reasonable starting point for a small Pi doing rendering/cache work. If this is changed, document the local setting and test that the device still recovers from real hangs.

Health check summary

After applying the headless tuning and persistent logging, this quick check is useful:

echo "Default target: $(systemctl get-default)"
grep -Ei 'CmaTotal|CmaFree|MemTotal|MemAvailable|SwapTotal|SwapFree' /proc/meminfo
free -h
swapon --show
vcgencmd get_throttled
vcgencmd measure_temp
journalctl --list-boots
ps aux | grep -Ei 'lightdm|labwc|wf-panel|pcmanfm|wayfire|wayland|Xorg' | grep -v grep || echo "No desktop processes found"

The ideal Blackcap e-ink-only Pi should show:

• multi-user.target as the default boot target.
• No desktop processes.
• CmaTotal around 64 MiB unless a local hardware need requires more.
• Swap use that is not constantly climbing under normal idle load.
• Persistent journal boot history with at least -1 and 0 after one intentional reboot.
• Normal Blackcap service status and successful e-ink render/display behavior.

References

• Raspberry Pi config.txt documentation: dtoverlay=vc4-kms-v3d loads the kernel graphics driver overlay.
• Linux kernel documentation: CMA reserves a pool during early boot for physically contiguous allocations, often used by display/DRM and DMA-style drivers.
• systemd journald.conf documentation: Storage=persistent stores journal data under /var/log/journal so previous-boot logs can be queried.