A Field Guide to Container Abuse: What We Found Scanning a Multi-Tenant Kubernetes Cluster

When you give people a free tier on a Kubernetes-based platform, you learn very quickly what the internet does with free compute. The operators of the cluster I was working on had a name for it: "code-execution-as-a-service." It was a joke, but it was also accurate. SRE humor can get grim, especially on the security side of the operation. I'll provide some background to help understand why that is.

I spent months operating a ClamAV-based malware scanning system on a large, multi-tenant OpenShift cluster. It consisted of several hundred nodes, open to the public, with a free tier that let anyone deploy containerized workloads. The scanner inspected every container's writable layer on every node, continuously. What we found was a taxonomy of abuse that no build-time image scanner would have caught, because most of it arrived after deployment.

This is a field guide to what we observed, organized by category. All of it was running on production infrastructure, often for extended periods before detection.

Malware and Exploitation

Crypto miners were the most common finding. Containers would deploy what appeared to be a legitimate application, then pull down a mining binary, typically XMRig or a variant, via a post-startup script. Some were brazen about it, mining immediately at full CPU utilization. Others were more sophisticated, throttling CPU usage to avoid triggering resource alerts and only mining during off-peak hours. CPU-efficient crypto currencies like Monero grew in popularity, and became natural choices for the environment.

Trojans appeared in containers that looked normal from the outside. The container would serve its stated purpose, like a web application or an API, while simultaneously running a backdoor that allowed remote command execution. In several cases, the trojan was embedded in what appeared to be a standard application deployment. Often times the code was heavily obfuscated so it wouldn't be immediately clear to the application owner what was happening.

CVE exploitation attempts were time-correlated with public disclosure. When the steps to reproduce a vulnerability were published, we would see users on the platform attempting to replicate the exploit, sometimes within hours. Whether this was curiosity-driven security research or malicious intent, the effect was the same: someone was actively probing shared infrastructure for fresh weaknesses. Obviously not discouraged by the ToS.

Evasion Techniques

The more sophisticated abuse didn't just deploy malware, it also tried to hide its tracks.

Delayed payload download was a common pattern. The container would start with a clean image that would pass any build-time scan. Minutes or hours after startup, it would reach out to an external server and download the actual malicious payload. By the time the malware was running, the original image scan was long since passed and forgotten. At lot of the time, it came from a legitimate looking location like GitHub. And a lot of the time, it was in an encrypted archive. So to anyone who tried to inspect the calling application code (usually after deobfuscating it) would only see that it downloaded something, but not what.

Heavy obfuscation made static analysis difficult. We found PHP files where every string was constructed character by character using chr() calls and \x hex encoding. The code was functionally unreadable. You couldn't determine what it did without figuring out how to deobfuscate it, or at the very least base64-decode it. Layers of eval(base64_decode()) chains wrapped the actual payload in multiple levels of encoding.

Staged execution combined both techniques: a seemingly innocent container that downloaded an obfuscated payload, decoded it at runtime, and only then began its actual malicious operation. Each stage, examined in isolation, looked relatively benign.When all the pieces finally came together, only then was it possible to get the full picture of what the application was trying to accomplish.

Platform Abuse

Not everything malicious is malware in the traditional sense. Some of the most resource-intensive abuse was "legitimate" software deployed for illegitimate purposes.

Botnet storage nodes used the platform's free compute and bandwidth to host and distribute pornographic content. With the goal of NOT contributing to the moral corruption of humanity, the investigation revealed something actually interesting. The containers ran standard web servers, like nginx or Apache, which was serving content that had nothing to do with the platform's intended use. The operators were effectively getting free CDN hosting for their distribution network.

DDoS and brute-force tools turned the platform into an attack launchpad. We found containers running purpose-built tools for credential stuffing, login brute-forcing, and volumetric DDoS attacks against third-party targets. The platform's IP addresses would end up on blocklists, affecting legitimate users.

Social media automation tools like auto-likers, auto-followers, and engagement bots were deployed at scale. These targeted platforms like Facebook and Instagram, using the various IP spaces of the clusters to avoid rate limiting and detection. A single user might deploy dozens of containers, each running a bot instance with different credentials.

Spam and Fraud

PHPmailer spam bots were a persistent problem. Users would deploy containers running bulk email infrastructure, like modified PHPmailer instances configured to blast spam at scale. The platform's IP reputation would degrade as outbound email from its address space got flagged by spam filters.

SEO spam sites were purpose-built content farms. They would scrape articles from legitimate websites, rewrite them just enough to avoid duplicate content detection, and publish them on domains loaded with Amazon affiliate links. The containers ran WordPress or custom PHP sites, auto-generating hundreds of pages designed to capture search traffic and redirect it to affiliate purchases. Oftentimes they would publish word-for-word copies of whatever content would generate them ad revenue.

Dangerous and Harmful Content

Some discoveries went beyond terms-of-service violations into genuinely harmful territory.

Phishing applications impersonated legitimate banking websites. The containers ran convincing replicas of financial institution login pages, complete with SSL certificates, designed to harvest credentials from victims directed there via phishing emails. These were polished operations, making the fake sites often seem more responsive than the real ones.

Tor exit nodes were deployed on the platform, routing anonymous traffic through infrastructure that was never designed or authorized for that purpose. This exposed the platform to legal and abuse complaints for traffic it had no visibility into. A good portion of the time it would be used to facilitate login attempt spam, in an attempt to brute-force credentials or deliberately lock out legitimate users of external services.

Extremist content was the most disturbing finding. We discovered containers hosting recruitment propaganda for Middle Eastern terrorist organizations. The material was served through standard web servers, using the platform's infrastructure for hosting and distribution.

The Custom Signature Problem

The standard ClamAV signature database catches traditional malware pretty well. Things like trojans, known crypto miners, common web shells. But it was never designed to detect platform abuse.

There are no ClamAV signatures for "this container is running a Facebook auto-liker." And there is no virus definition for "SEO spam farm with affiliate links." The phishing kits were hand-crafted for specific banks; no generic signature would match them. The obfuscated PHP backdoors used novel encoding that evaded existing detection rules.

I ended up writing several dozen custom ClamAV signatures to fill these gaps. Hash-based signatures (.hdb files) for known-bad binaries like specific crypto miner builds and proxy tools. Logical signatures (.ldb files) with PCRE regular expressions to detect obfuscation patterns, like the chr() character-by-character construction, the nested eval(base64_decode()) chains, the PHP backdoor patterns that standard antivirus missed.

I also maintained a false-positive suppression list (.ign2 files) that grew over time as we learned which legitimate applications triggered our behavioral signatures. Tuning detection is as important as building it, a scanner that floods operators with false positives gets ignored, and an ignored scanner protects nothing. Especially when alerts had the potential to wake someone up at midnight on Christmas, you don't want to be known as the boy who cried wolf.

Lessons Learned

Three things became clear from this experience:

Build-time scanning is necessary but not sufficient. It catches known vulnerabilities in the base image. It catches nothing that arrives after deployment. And in a multi-tenant environment, most of the interesting threats arrive after deployment.

Generic antivirus signatures are necessary but not sufficient. They catch known malware. They don't catch platform abuse, custom phishing kits, or novel obfuscation. Any serious container scanning operation needs the ability to write and deploy custom detection rules.

Container "ephemeral nature" is a security assumption, not a security control. You cannot design your security posture around the hope that containers restart frequently. In practice, they often don't, and when they don't, malware has all the time in the world.

If you're operating a multi-tenant Kubernetes platform, or any platform where users deploy arbitrary code, and, you are not scanning container filesystems at runtime, you are flying blindfolded. The question is not whether abuse is happening. The question is whether you have any idea of where and when.