One gloomy Tuesday evening, I SSHed into my CI server and ran htop as I had hundreds of times before. The machine is pinned running my jobs, but wait… Something is wrong.

htop showing bad things

I certainly wasn’t running any jobs involving Tensorflow, and I’m not in the habit of mining cryptocurrencies on my CI server.

This post will cover the series of events that lead to this happening, what happened, and the learnings I’ve taken away from it.

A series of unfortunate events ⛈

The first blow? I’ve been posting weekly updates on this blog which include a variety of links to my self-hosted Gitea server. As this blog is crawled by Google, my Gitea site got crawled too. That means that all of my code is now indexed by Google. Cool! What it also means is that people searching for Gitea servers to exploit can now easily find mine too.

Secondly, my self-hosted Gitea instance was setup to allow anyone with a valid email address to register. That is, they’d register and receive an address validation email, and completing that validation creates a full account. In the past this hasn’t been an issue, and has allowed friends/colleagues to create their own accounts without my intervention. However, soon after being indexed by Google, somebody took advantage.

Finally, my self-hosted CI server authenticates via my Gitea server. Therefore, anyone with an account on my Gitea server can start running CI jobs. This is of course useful for the same reasons as above, but combined means that anyone with a valid email address was allowed to use my CI server.

Crypto honeypots 💰🍯

While I originally felt like my space had been invaded, I realised after some thought that very little damage had been done. At the cost of a bit of CPU time, I’d been given a firm wake up call about holes in the security of my systems, rather than having an attacker poke around for something more juicy.

This situation immediately reminded me of the Mirage bitcoin pinata. The Mirage team were happy to give away some Bitcoin in exchange for learning about the flaws in their system. Still in the world of crypto, I gave away some CPU time to learn about gaps in my security.

The learnings 📚

What went wrong

  • Too trusting of the Internet. Until Tuesday, anyone with a valid email address could register on my Gitea server - intentionally! This is now locked down to a few friendly domains.

  • Forgetting unintended use cases. While undeniably a powerful tool, CI servers have literally the purpose of remote code execution. Containerisation protects the host, but not the network it lives in.

  • Internal firewalls are important. My external firewalls are pretty strong, but protection against internal threats was very weak. Banning my virtual machines from private network access, except where necessary, has reduced this threat vector significantly.

What went well

  • Constant vigilance worked out. Of the many times I’ve ran htop on this server I’ve been surprised only once, but I did catch it.

  • The attacker took something cheap. I lost some CPU time, rather than the attacker taking the time to look for something more interesting.

  • Containerisation limited the easy damage. Though there aren’t any resource limits on my CI jobs, they run on an single purpose virtual machine in containers. Though the processes in the earlier figure may appear to be running as root, there is very little damage they can do to the machine itself.

Final thoughts

This whole experience was a reminder to reconsider my threat model. A CI server is a machine built for remote code execution, and authorisation is often tightly coupled to a less sensitive source control system. This attack was certainly a surprise to me, particularly in the way it used my systems as intended but for a nefarious purpose.