← Back to blog

What I Learned Breaking My CI/CD Three Times

ci-cddevopsdockerself-hostinggithub

I already wrote about how I set up auto-deploys with Watchtower and GitHub Actions. Push to master, Watchtower polls ghcr.io, container restarts. Simple. Clean. It worked.

Then I got a new server.

The Problem

I was migrating everything to a new Proxmox setup — Ubuntu LXC, Docker Compose, no Portainer. Fresh start. I wanted to standardize how all my projects deployed. Same pattern, every project, no exceptions.

So I went to configure Watchtower for the new server and found it: Watchtower is deprecated.

Not broken. Not insecure. Just… nobody’s maintaining it anymore. That’s usually how things start falling apart — slowly, then all at once. I didn’t want to build my infrastructure on something heading toward abandonment, especially now that I had a clean slate.

Time to find something better.

Attempt One: GitHub Actions Self-Hosted Runners

My first instinct was GitHub Actions self-hosted runners. The idea is simple: instead of GitHub running your CI job on their servers, it runs on yours. Push to main → your server picks up the job → runs the deploy script.

I set it up. It worked, sort of.

push to main
     │
     ▼
GitHub Actions
     │
     ▼  (runner picks up job)
Your Server
     │
     ▼
deploy.sh → git pull → docker compose up --build -d

But the problems started stacking up.

The Permissions Problem

The runner needed to run Docker commands — which means it needed to be in the docker group. Being in the docker group is effectively root on that machine. There’s no way around it.

Then came the file ownership chaos. When the runner created a data/ directory for a container’s volumes, it was owned by the github-runner user. The container ran as a different user (UID 1000 inside the container). Result: the app couldn’t write to its own database. I added a chown step to the deploy script to fix it. Which felt like a hack, because it was.

The Public Repo Problem

Here’s the one that really killed it. GitHub shows you this warning when you try to add a self-hosted runner to a public repo:

“Self-hosted runners in public repositories can potentially run dangerous code on your machine by creating a pull request.”

Anyone can fork your public repo and open a pull request. If your workflow triggers on pull requests, that’s arbitrary code execution on your server. Even with protections, the attack surface is real.

Private repos don’t have this problem — but then you have a different problem.

The Private Repo Problem

To deploy a private repo from a GitHub-hosted runner, GitHub needs to reach your server. Options:

  • Expose SSH publicly (bad)
  • Open a port (bad)
  • Use a VPN (complex)

And for git pull on the server, you need credentials. You can’t just clone a private repo without some kind of authentication token or key.

Every solution created a new problem. That’s usually a sign you’re solving the wrong problem.

The Real Problem

I stepped back and looked at what I actually needed:

  • Works for public and private repos without different setups
  • No arbitrary code execution from outside
  • One pattern for all projects
  • Easy to add new projects (shouldn’t take more than 5 minutes)

Self-hosted runners failed points 1, 2, and kind of 3. Time for a different approach.

Webhooks: The Right Mental Model

The insight was flipping the direction.

With a self-hosted runner, GitHub reaches into your server and executes something. With webhooks, GitHub knocks on a door and your server decides what to do.

push to main
     │
     ▼
GitHub sends POST to https://webhooks.yourdomain.com/hooks/project
     │
     ▼
webhook service validates HMAC signature
     │
     ▼  (only if signature matches)
deploy.sh project → git pull → docker compose up --build -d

One webhook process on the server (installed via apt, running as a systemd service). One YAML file listing the hooks. One deploy script. That’s it.

Why This Is Actually Safer

When people hear “expose a public webhook endpoint,” they worry about security. But the security model here is solid.

HMAC-SHA256 signature validation. Every request GitHub sends is signed with a secret you generate. The webhook service checks the signature before running anything. If the signature doesn’t match — wrong secret, tampered payload, random internet scan — nothing happens. The endpoint is useless without the secret.

No arbitrary code execution. The webhook doesn’t run whatever GitHub sends. It runs a specific script (deploy.sh) with a specific argument (the project name). That’s it. There’s no way to escalate this into running arbitrary commands.

No persistent connection. With a runner, there’s a persistent process sitting on your server, waiting for jobs from GitHub. With webhooks, there’s no connection until GitHub sends one. Your server initiates nothing outbound to GitHub.

Identical setup for public and private repos. The webhook doesn’t care if the repo is public or private — it just runs deploy.sh. For private repos, SSH deploy keys (read-only) handle the git pull authentication. One key per repo, minimum permissions.

The risks that were keeping me up with self-hosted runners:

  • ❌ Fork PR executing code on my server → not possible (server runs a fixed script, not PR code)
  • ❌ Escalation to root → not possible (webhook runs one specific command)
  • ❌ Private repo exposure → solved (read-only SSH deploy keys)

Build on the Server, Not in the Cloud

Switching to webhooks also changed how I think about builds.

Before, my projects built Docker images on GitHub Actions, pushed them to ghcr.io, and Watchtower pulled them. That worked, but it meant:

  • Managing a container registry
  • Authenticating Watchtower with ghcr.io credentials
  • Dealing with image pull rate limits
  • Building on GitHub’s machines (which you don’t control)

Now deploy.sh just runs docker compose up --build -d. The server builds the image from source. No registry. No credentials. No Watchtower. The Dockerfile lives in the repo and the server uses it directly.

For a homelab running a handful of projects, building locally is faster and simpler than the roundtrip to a registry.

The Final Setup

/root/webhook/
├── hooks.yaml     ← one entry per project (id, secret, branch check)
└── deploy.sh      ← git pull + docker compose up --build -d

Adding a new project:

  1. Generate a secret: openssl rand -hex 32
  2. Add an entry to hooks.yaml
  3. systemctl restart webhook
  4. Add the webhook in GitHub → repo Settings → Webhooks
  5. Done.

Five minutes, no runner registration, no infrastructure changes.

What I Learned

The frustrating part of this whole journey wasn’t the technical problems — it was how each “solution” introduced a new problem. Runners fixed the Watchtower problem but created the permissions problem. Fixing permissions surfaced the public repo security problem. Trying to solve that for private repos introduced the authentication problem.

That’s how you know you’re going in the wrong direction: you’re solving a problem with another problem.

Webhooks didn’t feel like a clever trick. It felt like the obvious answer once I stopped trying to make GitHub reach into my server and started making my server listen for GitHub instead.

I still have a lot to learn about CI/CD. But I now have something that works, something I understand, and something I can explain to someone else — which means I actually learned it.

That’s probably the most important part.

The webhook service is adnanh/webhook, installed via apt on Ubuntu. One systemd service, one YAML file, handles everything.