Reverse Proxy and TLS

Audience: Deployment Administrator Related: Raspberry Pi · Gcp · Environment And Secrets

Raspberry Pi and GCP deployments use different front ends. The Pi deployment script recommends stunnel because it is lightweight and compatible with the established Gunicorn PROXY-protocol configuration. GCP uses Nginx as the HTTPS-only reverse proxy and Certbot for browser-trusted certificates.

Keep Gunicorn bound to localhost when a local TLS front end is enabled. Validate the proxy configuration, the forwarded client address behavior, the external base URL, and certificate renewal after infrastructure changes.

Raspberry Pi stunnel

For Raspberry Pi deployments, Blackcap should continue to use stunnel rather than moving to Nginx by default.

Browser / LAN / Alexa → HTTPS → stunnel on the Pi → Gunicorn 127.0.0.1:8080 → Blackcap

Decision: stay with stunnel on the Pi

Nginx is the preferred GCP front end, but stunnel remains the better default for the Pi because it is lighter, matches the existing Blackcap Pi deployment pattern, and works with Gunicorn's PROXY protocol mode so Blackcap can preserve real client IPs. Nginx is still a valid advanced option, but it should not be the standard Pi path unless a future Pi deployment needs more reverse-proxy features.

Script behavior

tools/deploy_pi.sh installs stunnel4 and openssl when the base Linux package step is accepted.

The script asks:

Use Gunicorn PROXY protocol for local stunnel/TLS forwarding? [Y]
Configure stunnel as the Pi TLS front-end? Recommended over Nginx for the Pi [Y]
Public stunnel HTTPS port [8443]
stunnel combined PEM certificate path [/etc/stunnel/blackcap.pem]

When stunnel is enabled, Gunicorn is bound to:

127.0.0.1:8080

The script writes:

/etc/stunnel/blackcap.conf

and enables/restarts stunnel4.

Certificate behavior

If the configured combined PEM certificate does not exist, the script creates a self-signed certificate. This is acceptable for private LAN testing. For public HTTPS, replace /etc/stunnel/blackcap.pem with a trusted certificate and private key combined into one PEM file.

A combined PEM contains the private key and certificate in the order required by stunnel. Keep the file root-protected and never place its contents in documentation or logs.

Expected stunnel config

The generated config is equivalent to:

[blackcap]
accept = 0.0.0.0:8443
connect = 127.0.0.1:8080
cert = /etc/stunnel/blackcap.pem
TIMEOUTclose = 0
protocol = proxy

The protocol = proxy line matches the Gunicorn service's PROXY protocol mode. If you disable PROXY protocol in the deploy script, do not use the generated stunnel PROXY protocol line.

Verification

sudo systemctl status inky_admin.service --no-pager
sudo systemctl status stunnel4 --no-pager
sudo ss -ltnp | egrep ':443|:8443|:8080'

Expected listener pattern:

0.0.0.0:8443      stunnel
127.0.0.1:8080    gunicorn

If using port 443 for Alexa account linking, the expected stunnel listener is 0.0.0.0:443 instead of 8443.

GCP Nginx and Certbot

Blackcap's direct GCP deployment uses this trust boundary:

Internet client → Nginx :443 → Gunicorn 127.0.0.1:8080 → Blackcap

Gunicorn remains bound to loopback. The GCP firewall exposes TCP 443 only; TCP 80 is not required by this deployment pattern.

Why DNS-01 is used

Let's Encrypt HTTP-01 validation requires inbound TCP 80. Blackcap's HTTPS-only GCP setup instead uses automated DNS-01 validation. Certbot proves control of the hostname by creating and removing an _acme-challenge TXT record through the DNS provider's API.

The focused setup supports three automation modes:

• godaddy: native Blackcap hooks that use GoDaddy's production DNS API. This is the default for the current Blackcap deployment.
• google: Google Cloud DNS through Certbot's dns-google plugin.
• hooks: provider-specific executable authentication and cleanup hooks for other DNS providers.

Interactive manual DNS entry is not considered auto-renewal and is not used by this setup.

Configure an existing GCP instance

From the current checkout:

cd /opt/Blackcap
sudo ./tools/configure_gcp_frontend.sh

The script:

• installs Nginx and Certbot when missing;
• installs python3-certbot-dns-google when Google Cloud DNS mode is selected;
• installs root-owned GoDaddy DNS credentials and Certbot hook wrappers when GoDaddy mode is selected;
• obtains or converts the hostname certificate to unattended DNS-01 renewal;
• writes an Nginx site that listens on TCP 443 only;
• enables the package-provided certbot.timer, or installs a Blackcap fallback timer when the package has no timer;
• installs /etc/letsencrypt/renewal-hooks/deploy/blackcap-nginx-reload;
• writes trusted-local-proxy settings to /etc/blackcap/blackcap.env; and
• restarts Blackcap so host-level proxy settings take effect.

GCP firewall requirements

Allow inbound TCP:

• 443 for Blackcap HTTPS traffic.

Do not expose:

• 80; DNS-01 validation does not use it.
• 8080; Gunicorn listens only on loopback.
• 5432; the Cloud SQL Auth Proxy listens only on loopback.

Certbot needs outbound HTTPS access to the ACME service, and the selected DNS automation method must be able to update the authoritative DNS zone.

GoDaddy DNS automation

Choose godaddy when the hostname's authoritative DNS records are managed in GoDaddy. Before running the setup, create a production GoDaddy API key and secret for the same GoDaddy account that controls the domain. GoDaddy currently limits production Management and DNS API access to eligible accounts with at least one domain or an active Discount Domain Club Domain Pro plan.

The setup prompts for:

GoDaddy-managed root domain
Root-only GoDaddy credentials file
GoDaddy production API key
GoDaddy production API secret
DNS propagation wait in seconds

For a hostname such as blackcap.microwavepie.com, the GoDaddy-managed root domain is normally:

microwavepie.com

Credentials are stored outside the repository and outside Blackcap's application environment:

/etc/blackcap/godaddy-dns.ini

The file is owned by root and mode 0600. The API secret is never written to /etc/blackcap/blackcap.env, command-line arguments, Nginx configuration, or the PostgreSQL database.

Blackcap installs these Certbot hook commands:

/usr/local/sbin/blackcap-godaddy-dns-auth
/usr/local/sbin/blackcap-godaddy-dns-cleanup

The hooks use GoDaddy's production Domains API to read the existing TXT records, add only the current Let's Encrypt value, and remove only that same value after validation. This preserves unrelated TXT values at the same _acme-challenge record name. The hook waits for the configured propagation period before Certbot asks Let's Encrypt to validate the record.

Verify API access without changing DNS:

sudo /usr/bin/python3 /opt/Blackcap/tools/godaddy_dns_hook.py verify \
  --credentials /etc/blackcap/godaddy-dns.ini

Inspect the saved Certbot renewal hooks:

sudo grep -E \
  '^[[:space:]]*(authenticator|manual_auth_hook|manual_cleanup_hook)[[:space:]]*=' \
  /etc/letsencrypt/renewal/<hostname>.conf

Expected GoDaddy renewal configuration:

authenticator = manual
manual_auth_hook = /usr/local/sbin/blackcap-godaddy-dns-auth
manual_cleanup_hook = /usr/local/sbin/blackcap-godaddy-dns-cleanup

A manual authenticator is unattended when both saved hook commands are present. A plain interactive manual certificate without hooks is not renewable automatically.

Google Cloud DNS permissions

For google mode, Certbot uses Application Default Credentials by default. On a Compute Engine VM, that normally means the VM's attached service account.

The service account must be allowed to discover the relevant managed zone and create/delete challenge TXT records. Prefer zone-scoped, least-privilege roles. The setup also accepts a service-account JSON file, but the VM service account is preferable because it avoids a long-lived key file on disk.

The script prompts for:

Google Cloud project containing the public DNS zone
Google DNS service-account JSON path; leave blank to use the VM service account
DNS propagation wait in seconds

If the public zone is in another project, enter that DNS project rather than the Compute Engine project.

Non-Google DNS providers

Choose hooks mode and provide:

Executable DNS-01 authentication hook
Executable DNS-01 cleanup hook

The authentication hook must create the requested TXT record and wait until it is resolvable before returning successfully. The cleanup hook must remove the record. Certbot stores these hook paths in the renewal configuration, allowing the scheduled renewal command to run unattended.

Protect API tokens used by hook scripts. Keep credential files root-owned and mode 0600.

Client IP trust model

The Nginx site deliberately overwrites incoming forwarding headers:

proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header Forwarded "";

Using $remote_addr instead of $proxy_add_x_forwarded_for prevents an Internet client from injecting a fake first address into the forwarded chain. Blackcap accepts those headers only when the immediate socket peer is in:

BLACKCAP_TRUSTED_PROXY_HEADERS_ENABLED=1
BLACKCAP_TRUSTED_PROXY_CIDRS=127.0.0.1/32, ::1/128
BLACKCAP_ENFORCE_HTTPS_EXTERNAL=1
BLACKCAP_TRUST_EXTERNAL_BASE_URL_HTTPS=1

These environment settings are applied after DB-backed configuration because the proxy topology belongs to the host, not to the shared PostgreSQL database.

Renewal behavior

The Debian/Ubuntu Certbot package normally provides certbot.timer. Blackcap enables it when present. Successful renewals execute:

/usr/sbin/nginx -t
/bin/systemctl reload nginx

The reload happens only after Nginx configuration validation succeeds.

Check the schedule and certificate:

sudo systemctl list-timers --all | grep -E 'certbot|blackcap-certbot'
sudo certbot certificates
sudo certbot renew --dry-run

Inspect the renewal authenticator:

sudo grep -E '^[[:space:]]*(authenticator|manual_auth_hook|manual_cleanup_hook)[[:space:]]*=' \
  /etc/letsencrypt/renewal/<hostname>.conf

For GoDaddy, the authenticator is manual with both Blackcap GoDaddy hook paths saved. For Google Cloud DNS, the authenticator should be:

authenticator = dns-google

For custom hooks, it should be manual and include both saved hook paths. A plain interactive manual certificate without hooks is not unattended.

Verify HTTPS-only exposure

sudo nginx -t
sudo systemctl status nginx --no-pager
sudo ss -ltnp | egrep ':80|:443|:8080'

Expected:

• Nginx listens on 0.0.0.0:443 and/or [::]:443.
• Nothing listens publicly on port 80.
• Gunicorn listens on 127.0.0.1:8080.

The GCP firewall should also contain no inbound HTTP rule for this VM.

Validate the real client address

After an external browser request, use Blackcap's access activity or GeoIP diagnostics. The request should show the public client IP rather than 127.0.0.1.

You can also inspect the Nginx access log:

sudo tail -n 50 /var/log/nginx/access.log

Blackcap's security and access services use the validated client address for activity records, GeoIP lookup, rate-limit keys, and local-versus-external request decisions.

Troubleshooting DNS-01

If issuance or the renewal dry run fails:

1. Confirm the hostname's public DNS zone is managed by the provider selected in the script.

2. For GoDaddy, verify /etc/blackcap/godaddy-dns.ini is root-owned, mode 0600, contains a production API key, and names the correct root domain. Run the GoDaddy hook's verify command shown above.

3. For Google Cloud DNS, confirm the VM service account can list the managed zone and create/delete record changes.

4. Confirm the Certbot plugin is installed when using Google Cloud DNS:

   sudo certbot plugins | grep -E 'dns-google|dns_google'
   

5. Confirm the public authoritative nameservers can resolve TXT records under _acme-challenge.<hostname>.

6. Increase BLACKCAP_CERTBOT_DNS_PROPAGATION_SECONDS when the DNS provider publishes changes slowly.

7. Inspect Certbot logs:

   sudo journalctl -u certbot.service -n 100 --no-pager
   sudo tail -n 100 /var/log/letsencrypt/letsencrypt.log
   

8. Validate Nginx separately:

   sudo nginx -t
   sudo systemctl status nginx --no-pager
   

Real client IP inside Flask and Gunicorn logs

For the direct GCP topology, Nginx is the only proxy and Gunicorn remains bound to 127.0.0.1:8080. The deployment environment enables trusted proxy handling only for loopback CIDRs. Blackcap applies Werkzeug ProxyFix for exactly one proxy hop so Flask's standard request.remote_addr, scheme, host, and port use the values Nginx overwrites.

Gunicorn's default %(h)s access-log token intentionally reports its socket peer, which is local Nginx. The repository gunicorn.conf.py switches the GCP access log to the Nginx-provided X-Forwarded-For value when the loopback-only trusted proxy environment is enabled. No systemd ExecStart change is required; Gunicorn loads gunicorn.conf.py from /opt/Blackcap because that is the service working directory.

After updating these files, restart Blackcap:

sudo systemctl restart inky_admin.service

New access lines should begin with the client's public address rather than 127.0.0.1. Existing Access Activity rows are not rewritten.