Microsoft has released a study describing an ongoing series of attacks on Kubeflow, a toolkit on top of Kubernetes clusters for running machine learning ( ML) operations. According to Microsoft, Kubernetes was at the forefront of a more recent large-scale XMRIG initiative. Misconfigured dashboards are at the center of a widespread campaign for the XMRIG Monero-mining. Closer inspection showed the image running a common open-source crypto-jacking malware that mines the virtual currency of Monero, known as XMRIG. The attacks have been ongoing since April 2020. The end goal of Microsoft is to deploy a cryptocurrency miner on K8s clusters running internet-exposed Kubeflow instances.
According to a security researcher, the business has found such kinds of attacks against ten of Kubernetes clusters running ML toolkit Kubeflow. Although the number of compromised clusters is small relative to previous attacks by Kubernetes (k8s). Server owners’ profits for crooks and financial losses are more likely than previously seen.
Nodes used to perform ML tasks are often fairly efficient, and often include GPUs. This makes Kubernetes clusters a great target for crypto-mining campaigns. They are also for ML tasks, which was the purpose of this attack.
It is an open-source project that integrates the most common resources of data science in one place. It’s an exciting toolkit with Kubernetes under the hood which can make life easier for Data Scientists and Data Engineers.
When you have a Kubeflow case going, though, you will get an integrated component. This shows that it will allow you to create machine learning models and bring them into production. Kubeflow is a project-based in the cloud. It can be implemented using various providers of cloud platforms: Azure, Google Cloud Platform (GCP), and Amazon Web Services ( AWS). It can also be installed on your local machine or on-prem infrastructure.
Firstly, Kubernetes is a system for running and coordinating containerized applications across a machine cluster. It is a platform to use methods that provide predictability, scalability, and high availability to manage the life cycle of container type applications and services. The data gets encrypted when writing into etcd. Any newly created or updated secret should be encrypted when stored until your Kube-API server has been restarted. To search, you can use the command line software etcdctl to get back the contents of your file. KMS provider uses gRPC to interact with a plugin unique to KMS. This is responsible for all remote KMS contact.
Popular attack vectors used to detect compromised Kubernetes clusters are mentioned below:
The method of mitigation to Anonymous Access is to turn on RBAC (Role-Based Access Control) that requires specific anonymous request authorization. Anonymous access is the most serious kind of misconfiguration. It allows both external and internal attackers to directly hit commands to various k8s components via API and compromise k8s cluster.
We can detect misconfiguration of Anonymous by verifying the user object for the system: anonymous username and system: unauthenticated group in the logs.
There are 2 types of accounts that exist in Kubernetes: regular users and service accounts. Therefore, implementing the least privilege principle for service accounts is important so that an attacker can’t use them to compromise the k8s cluster anymore.
While for service account misuse there is not any universal signature detection, there are many techniques like anomaly detection that can be to identify potential abuse. Service accounts are for a particular purpose by automating systems, their operation should be mention and consistent.
Widely and commonly used exec command is “/bin/bash”, opening the interactive shell on the Pod. Using that command, an attacker can get access to the file system of the Pod and its containers, possibly install a backdoor or search for the account service token of the Pod, which could have more rights than its current account.
The execution of Pod commands is a login by API-Server and displays API calls that target resource pod with the exec subresource. The following logic is to identify a user opening, interactive shell at a Pod: objectRef.resource=”pods” AND objectRef.subresource=”exec” AND verb=”create” AND (request URI contains “/bin/bash” OR “/bin/sh”)
However, underlying node hosting the Pods is a better alternative. There is an option to install a volume of the file system into the new Pod when building a Pod. Once an attacker gets access to create pod resources, they will be able to create a brand new Pod with a malicious container image configured to mount root directory of the node.
It would be highly suspicious for a Pod to build another Pod, as mostly Pod creations would come from the k8s controller. The call to the API may come from an account that normally does not have access to create Pods, such as a hacked service account, in that techniques of anomaly detection may also apply.
The secret may be installed as a number when an application or pod is required to access a secret object. Also, Secrets are not encrypted into repose by default in etcd storage.
Most of those accesses, as with the service account operation, should be routine and planned. It is possible to detect unauthorized secret access by tracking which secrets are usually accessed by account, which could be an intruder abusing the account to view secrets beyond reach.
These activities typically run in insecure containers, such as web applications, which exploit known vulnerabilities. Azure Security Center recently found a new campaign of crypto-mining that specifically targets Kubernetes environments. What separates this attack from other crypto-mining attacks is its scale: In just two hours, a malicious container will be deployed on tens of Kubernetes clusters. The containers were running a image in a public repository: kannix / monero-miner. Also the same actor who deployed the crypto mining containers identified the cluster resources including the secrets of Kubernetes. To be safe, it is necessary to carefully configure the Kubernetes environment to ensure that no container-focus surface doors are left open to attackers.cs, such as a privileged container detected.
Azure Security Center tracks and resists attacks of 1000 Kubernetes clusters that run on top of the Azure Kubernetes Service. Although, they have published a blog post related to large-scale campaign against k8s clusters that abused exposed dashboards to deploy cryptocurrency miners. Kubeflow, a Kubernetes ML workflow toolkit, found that this attack occurred on many of Kubernetes clusters.
Kubeflow, a free and open-source machine learning platform, started with Kubernetes as a project to run TensorFlow work. This fact however makes clusters of Kubernetes used for Machine Learning tasks. Kubeflow Dashboards are the exact focus for crypto-mining campaigns. In many different k8s clusters a suspicious image from a public repo was observed during April.
Kubeflow is a container type service. Few of the services include training model frameworks, notebook server Jupyter and Katib, and more. the different tasks in the cluster are running as containers. So if attackers acquire access Kubeflow. They have many doors to run malicious container images in the k8s cluster.
Once attackers acquire access to dashboard, they can upload backdoor container images in the k8s cluster using multiple methods. For example:
Since the past, Azure Security Center has identified several campaigns against K8s clusters which have a common vector of access: an exposed internet connection. This is however the first time we have detected an attack directly targeting Kubeflow environments. When deploying a service such as Kubeflow in a cluster it is essential to be aware of security aspects such as:
Moreover, the Kubernetes threat matrix includes techniques that attackers can use to target the Kubernetes cluster. The intruder used an exposed dashboard to obtain initial access to the cluster. The execution and durability of the cluster were achieved by establishing a container within the cluster that was installed inside the cluster.