This is the multi-page printable view of this section. Click here to print.
By default, Tetragon monitors process lifecycle, learn more about that in the dedicated use cases.
For more advanced use cases, Tetragon can observe tracepoints and arbitrary
kernel calls via kprobes. For that, Tetragon must be extended and configured
with custom resources objects named TracingPolicy.
It can then generates process_tracepoint and process_kprobes events.
Tetragon observes process creation and termination with default configuration
and generates process_exec and process_exit events:
process_exec events include useful information about the execution of
binaries and related process information. This includes the binary image that
was executed, command-line arguments, the UID context the process was
executed with, the process parent information, the capabilities that a
process had while executed, the process start time, the Kubernetes Pod,
labels and more.process_exit events, as the process_exec event shows how and when a
process started, indicate how and when a process is removed. The information
in the event includes the binary image that was executed, command-line
arguments, the UID context the process was executed with, process parent
information, process start time, the status codes and signals on process
exit. Understanding why a process exited and with what status code helps
understand the specifics of that exit.Both these events include Linux-level metadata (UID, parents, capabilities, start time, etc.) but also Kubernetes-level metadata (Kubernetes namespace, labels, name, etc.). This data make the connection between node-level concepts, the processes, and Kubernetes or container environments.
These events enable a full lifecycle view into a process that can aid an incident investigation, for example, we can determine if a suspicious process is still running in a particular environment. For concrete examples of such events, see the next use case on process execution.
process_exec and process_exitThis first use case is monitoring process execution, which can be observed with
the Tetragon process_exec and process_exit JSON events.
These events contain the full lifecycle of processes, from fork/exec to
exit, including metadata such as:
As a first step, let’s start monitoring the events from the xwing pod:
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact --namespace default --pod xwing
Then in another terminal, let’s kubectl exec into the xwing pod and execute
some example commands:
kubectl exec -it xwing -- /bin/bash
whoami
If you observe, the output in the first terminal should be:
🚀 process default/xwing /bin/bash
🚀 process default/xwing /usr/bin/whoami
💥 exit default/xwing /usr/bin/whoami 0
Here you can see the binary names along with its arguments, the pod info, and return codes in a compact one-line view of the events.
For more details use the raw JSON events to get detailed information, you can stop
the Tetragon CLI by Crl-C and parse the tetragon.log file by executing:
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | jq 'select(.process_exec.process.pod.name=="xwing" or .process_exit.process.pod.name=="xwing")'
Example process_exec and process_exit events can be:
{
"process_exec": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjExNDI4NjE1NjM2OTAxOjUxNTgz",
"pid": 51583,
"uid": 0,
"cwd": "/",
"binary": "/usr/bin/whoami",
"arguments": "--version",
"flags": "execve rootcwd clone",
"start_time": "2022-05-11T12:54:45.615Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2022-05-11T10:07:33Z",
"pid": 50
}
},
"docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI=",
"refcnt": 1
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI=",
"pid": 43872,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"flags": "execve rootcwd clone",
"start_time": "2022-05-11T12:15:36.225Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2022-05-11T10:07:33Z",
"pid": 43
}
},
"docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkxODU5NTMzOTk6NDM4NjE=",
"refcnt": 1
}
},
"node_name": "kind-control-plane",
"time": "2022-05-11T12:54:45.615Z"
}
{
"process_exit": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjExNDI4NjE1NjM2OTAxOjUxNTgz",
"pid": 51583,
"uid": 0,
"cwd": "/",
"binary": "/usr/bin/whoami",
"arguments": "--version",
"flags": "execve rootcwd clone",
"start_time": "2022-05-11T12:54:45.615Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2022-05-11T10:07:33Z",
"pid": 50
}
},
"docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI="
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkyMjU2MjMyNjk6NDM4NzI=",
"pid": 43872,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"flags": "execve rootcwd clone",
"start_time": "2022-05-11T12:15:36.225Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://1fb931d2f6e5e4cfdbaf30fdb8e2fdd81320bdb3047ded50120a4f82838209ce",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2022-05-11T10:07:33Z",
"pid": 43
}
},
"docker": "1fb931d2f6e5e4cfdbaf30fdb8e2fdd",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjkwNzkxODU5NTMzOTk6NDM4NjE="
}
},
"node_name": "kind-control-plane",
"time": "2022-05-11T12:54:45.616Z"
}
Advanced process execution can be performed by using Tracing Policies to monitor the execve system call path.
If we want to monitor execution of Executable and Linkable Format (ELF) or flat binaries before they are actually executed. Then the process-exec-elf-begin tracing policy is a good first choice.
The process-exec-elf-begin tracing policy, will not report the different binary format handlers or scripts being executed, but will report the final ELF or flat binary, like the shebang handler.
To report those another tracing policy can be used.
Before going forward, verify that all pods are up and running, ensure you deploy our Demo Application to explore the Security Observability Events:
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15.3/examples/minikube/http-sw-app.yaml
It might take several seconds for some pods until they satisfy all the dependencies:
kubectl get pods -A
The output should be similar to:
NAMESPACE NAME READY STATUS RESTARTS AGE
default deathstar-54bb8475cc-6c6lc 1/1 Running 0 2m54s
default deathstar-54bb8475cc-zmfkr 1/1 Running 0 2m54s
default tiefighter 1/1 Running 0 2m54s
default xwing 1/1 Running 0 2m54s
kube-system tetragon-sdwv6 2/2 Running 0 27m
Let’s apply the process-exec-elf-begin Tracing Policy.
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-exec/process-exec-elf-begin.yaml
Then start monitoring events with the tetra CLI:
kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents
In another terminal, kubectl exec into the xwing Pod:
kubectl exec -it xwing -- /bin/bash
And execute some commands:
id
The tetra CLI will generate the following ProcessKprobe events:
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjY0MDI4MTA4MzcxOjM2NDk5",
"pid": 36499,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"flags": "execve",
"start_time": "2023-08-02T11:58:53.618461573Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://775beeb1a25a95e10dc149d6eb166bf45dd5e6039e8af3b64e8fb4d29669f349",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-08-02T07:24:54Z",
"pid": 13
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "775beeb1a25a95e10dc149d6eb166bf",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjYyNzg3ODI1MTQ4OjM2NDkz",
"refcnt": 1,
"tid": 36499
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjYyNzg3ODI1MTQ4OjM2NDkz",
"pid": 36493,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"flags": "execve rootcwd clone",
"start_time": "2023-08-02T11:58:52.378178852Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://775beeb1a25a95e10dc149d6eb166bf45dd5e6039e8af3b64e8fb4d29669f349",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-08-02T07:24:54Z",
"pid": 13
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "775beeb1a25a95e10dc149d6eb166bf",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE2NjYyNzE2OTU0MjgzOjM2NDg0",
"tid": 36493
},
"function_name": "security_bprm_creds_from_file",
"args": [
{
"file_arg": {
"path": "/bin/busybox"
}
}
],
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-08-02T11:58:53.624096751Z"
}
In addition to the Kubernetes Identity and process metadata, ProcessKprobe events contain the binary being executed. In the above case they are:
function_name: where we are hooking into the kernel to read the binary that is being executed.file_arg: that includes the path being executed, and here it is /bin/busybox that is the real
binary being executed, since on the xwing pod the container is running busybox.
The binary /usr/bin/id -> /bin/busybox points to busybox.To disable the process-exec-elf-being Tracing Policy run:
kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-exec/process-exec-elf-begin.yaml
Tetragon also provides the ability to check process capabilities and kernel namespaces access.
This information would help us determine which process or Kubernetes pod has started or gained access to privileges or host namespaces that it should not have. This would help us answer questions like:
Which Kubernetes pods are running with
CAP_SYS_ADMINin my cluster?
Which Kubernetes pods have host network or pid namespace access in my cluster?
Edit the Tetragon configmap:
kubectl edit cm -n kube-system tetragon-config
Set the following flags from “false” to “true”:
# enable-process-cred: true
# enable-process-ns: true
Save your changes and exit.
Restart the Tetragon daemonset:
kubectl rollout restart -n kube-system ds/tetragon
Create a YAML file (e.g., privileged-nginx.yaml) with the following PodSpec:
apiVersion: v1
kind: Pod
metadata:
name: privileged-the-pod
spec:
hostPID: true
hostNetwork: true
containers:
- name: privileged-the-pod
image: nginx:latest
ports:
- containerPort: 80
securityContext:
privileged: true
Apply the configuration:
kubectl apply -f privileged-nginx.yaml
Start monitoring events from the privileged Nginx pod:
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents --namespace default --pod privileged-the-pod
You should observe Tetragon generating events similar to these, indicating the privileged container start:
🚀 process default/privileged-nginx /nginx -g daemon off; 🛑 CAP_SYS_ADMIN
This page shows how you can create a tracing policy to monitor filename access. For general information about tracing policies, see the tracing policy page.
There are two aspects of the tracing policy: (i) what hooks you can use to monitor specific types of access, and (ii) how you can filter at the kernel level for only specific events.
There are different ways applications can access and modify files, and for this tracing policy we focus in three different types.
The first is read and write accesses, which is the most common way that applications access files. Applications can perform this type of accesses with a variety of different system
calls: read and write, optimized system calls such as copy_file_range and sendfile, as well
as asynchronous I/O system call families such as the ones provided by aio and io_uring. Instead
of monitoring every system call, we opt to hook into the security_file_permission hook, which is a
common execution point for all the above system calls.
Applications can also access files by mapping them directly into their virtual address space. Since
it is difficult to catch the accesses themselves in this case, our policy will instead monitor the
point when the files are mapped into the application’s virtual memory. To do so, we use the
security_mmap_file hook.
Lastly, there is a family of system calls (e.g,. truncate) that allow to indirectly modify the
contents of the file by changing its size. To catch these types of access we will hook into
security_path_truncate.
Using the hooks above, you can monitor all accesses in the system. However, this will create a large number of events, and it is frequently the case that you are only interested in a specific subset those events. It is possible to filter the events after their generation, but this induces unnecessary overhead. Tetragon, using BPF, allows filtering these events directly in the kernel.
For example, the following snippet shows how you can limit the events from the
security_file_permission hook only for the /etc/passwd file. For this, you need to specify the
arguments of the function that you hooking into, as well as their type.
- call: "security_file_permission"
syscall: false
args:
- index: 0
type: "file" # (struct file *) used for getting the path
- index: 1
type: "int" # 0x04 is MAY_READ, 0x02 is MAY_WRITE
selectors:
- matchArgs:
- index: 0
operator: "Equal"
values:
- "/etc/passwd" # filter by filename (/etc/passwd)
- index: 1
operator: "Equal"
values:
- "2" # filter by type of access (MAY_WRITE)
The previous example uses the Equal operator. Similarly, you can use the Prefix operator to
filter events based on the prefix of a filename.
In this example, we monitor if a process inside a Kubernetes workload performs a read or write in
the /etc/ directory. The policy may be extended with additional directories or specific files if
needed.
As a first step, we apply the following policy that uses the three hooks mentioned previously as well as appropriate filtering:
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/filename_monitoring.yaml
Next, we deploy a file-access Pod with an interactive bash session:
kubectl run --rm -it file-access -n default --image=busybox --restart=Never
In another terminal, you can start monitoring the events from the file-access Pod:
kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact --namespace default --pod file-access
In the interactive bash session, edit the /etc/passwd file:
vi /etc/passwd
If you observe, the output in the second terminal should be:
🚀 process default/file-access /bin/sh
🚀 process default/file-access /bin/vi /etc/passwd
📚 read default/file-access /bin/vi /etc/passwd
📚 read default/file-access /bin/vi /etc/passwd
📚 read default/file-access /bin/vi /etc/passwd
📝 write default/file-access /bin/vi /etc/passwd
📝 truncate default/file-access /bin/vi /etc/passwd
💥 exit default/file-access /bin/vi /etc/passwd 0
Note that read and writes are only generated for /etc/ files based on BPF in-kernel filtering
specified in the policy. The default CRD additionally filters events associated with the pod init
process to filter init noise from pod start.
Similarly to the previous example, reviewing the JSON events provides additional data. An example
process_kprobe event observing a write can be:
{
"process_kprobe": {
"process": {
"exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6MTY4MTc3MDUwMTI1NDI6NjQ3NDY=",
"pid": 64746,
"uid": 0,
"cwd": "/",
"binary": "/bin/vi",
"arguments": "/etc/passwd",
"flags": "execve rootcwd clone",
"start_time": "2024-04-14T02:18:02.240856427Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "file-access",
"container": {
"id": "containerd://6b742e38ee3a212239e6d48b2954435a407af44b9a354bdf540db22f460ab40e",
"name": "file-access",
"image": {
"id": "docker.io/library/busybox@sha256:c3839dd800b9eb7603340509769c43e146a74c63dca3045a8e7dc8ee07e53966",
"name": "docker.io/library/busybox:latest"
},
"start_time": "2024-04-14T02:17:46Z",
"pid": 12
},
"pod_labels": {
"run": "file-access"
},
"workload": "file-access",
"workload_kind": "Pod"
},
"docker": "6b742e38ee3a212239e6d48b2954435",
"parent_exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6MTY4MDE3MDQ3OTQyOTg6NjQ2MTU=",
"refcnt": 1,
"tid": 64746
},
"parent": {
"exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6MTY4MDE3MDQ3OTQyOTg6NjQ2MTU=",
"pid": 64615,
"uid": 0,
"cwd": "/",
"binary": "/bin/sh",
"flags": "execve rootcwd clone",
"start_time": "2024-04-14T02:17:46.240638141Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "file-access",
"container": {
"id": "containerd://6b742e38ee3a212239e6d48b2954435a407af44b9a354bdf540db22f460ab40e",
"name": "file-access",
"image": {
"id": "docker.io/library/busybox@sha256:c3839dd800b9eb7603340509769c43e146a74c63dca3045a8e7dc8ee07e53966",
"name": "docker.io/library/busybox:latest"
},
"start_time": "2024-04-14T02:17:46Z",
"pid": 1
},
"pod_labels": {
"run": "file-access"
},
"workload": "file-access",
"workload_kind": "Pod"
},
"docker": "6b742e38ee3a212239e6d48b2954435",
"parent_exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6MTY3OTgyOTA2MDc3NTc6NjQ1NjQ=",
"tid": 64615
},
"function_name": "security_file_permission",
"args": [
{
"file_arg": {
"path": "/etc/passwd",
"permission": "-rw-r--r--"
}
},
{
"int_arg": 2
}
],
"return": {
"int_arg": 0
},
"action": "KPROBE_ACTION_POST",
"policy_name": "file-monitoring",
"return_action": "KPROBE_ACTION_POST"
},
"node_name": "tetragon-dev-control-plane",
"time": "2024-04-14T02:18:14.376304204Z"
}
In addition to the Kubernetes Identity
and process metadata from exec events, process_kprobe events contain
the arguments of the observed system call. In the above case they are
file_arg.path: the observed file pathint_arg: is the type of the operation (2 for a write and 4 for a read)return.int_arg: is 0 if the operation is allowedTo disable the TracingPolicy run:
kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/filename_monitoring.yaml
To delete the file-access Pod from the interactive bash session, type:
exit
Another example of a similar policy can be found in our examples folder.
Note that this policy has certain limitations because it matches on the filename that the application uses to access. If an application accesses the same file via a hard link or a different bind mount, no event will be generated.
To view TCP connect events, apply the example TCP connect TracingPolicy:
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/tcp-connect.yaml
To start monitoring events in the xwing pod run the Tetragon CLI:
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact --namespace default --pod xwing
In another terminal, start generate a TCP connection. Here we use curl.
kubectl exec -it xwing -- curl http://cilium.io
The output in the first terminal will capture the new connect and write,
🚀 process default/xwing /usr/bin/curl http://cilium.io
🔌 connect default/xwing /usr/bin/curl tcp 10.244.0.6:34965 -> 104.198.14.52:80
📤 sendmsg default/xwing /usr/bin/curl tcp 10.244.0.6:34965 -> 104.198.14.52:80 bytes 73
🧹 close default/xwing /usr/bin/curl tcp 10.244.0.6:34965 -> 104.198.14.52:80
💥 exit default/xwing /usr/bin/curl http://cilium.io 0
To disable the TracingPolicy run:
kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/tcp-connect.yaml
On Linux each process has various associated user, group IDs, capabilities, secure management flags, keyring, LSM security that are used part of the security checks upon acting on other objects. These are called the task privileges or process credentials.
Changing the process credentials is a standard operation to perform privileged actions or to execute commands as another user. The obvious example is sudo that allows to gain high privileges and run commands as root or another user. An other example is services or containers that can gain high privileges during execution to perform restricted operations.
execve(2).execve(2), especially when a binary is executed that will execute as
UID 0.These govern the way the UIDs/GIDs and capabilities are manipulated and
inherited over certain operations such as execve(2).
The LSM framework provides a mechanism for various security checks to be hooked by new kernel extensions. Tasks can have extra controls part of LSM on what operations they are allowed to perform.
Monitoring Linux process credentials is a good practice to idenfity programs
running with high privileges. Tetragon allows retrieving Linux process credentials
as a process_credentials object.
Changes to credentials can be monitored either in system calls or in internal kernel functions.
Generally it is better to monitor in internal kernel functions. For further details please read Advantages and disadvantages of kernel layer monitoring compared to the system call layer section.
Tetragon can hook at the system calls that directly manipulate the credentials. This allows us to determine which process is trying to change its credentials and the new credentials that could be applied by the kernel.
This answers the questions:
Which process or container is trying to change its UIDs/GIDs in my cluster?
Which process or container is trying to change its capabilities in my cluster?
Before going forward, verify that all pods are up and running, ensure you deploy our Demo Application to explore the Security Observability Events:
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15.3/examples/minikube/http-sw-app.yaml
It might take several seconds for some pods until they satisfy all the dependencies:
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default deathstar-54bb8475cc-6c6lc 1/1 Running 0 2m54s
default deathstar-54bb8475cc-zmfkr 1/1 Running 0 2m54s
default tiefighter 1/1 Running 0 2m54s
default xwing 1/1 Running 0 2m54s
kube-system tetragon-sdwv6 2/2 Running 0 27m
We use the
process.credentials.changes.at.syscalls
Tracing Policy that hooks the
setuid system calls
family:
setuidsetgidsetfsuidsetfsgidsetreuidsetregidsetresuidsetresgidLet’s apply the process.credentials.changes.at.syscalls Tracing Policy.
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/process.credentials.changes.at.syscalls.yaml
Then start monitoring events with the tetra CLI:
kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents
In another terminal, kubectl exec into the xwing Pod:
kubectl exec -it xwing -- /bin/bash
And execute su as this will call the
related setuid system calls:
su root
The tetra CLI will generate the following ProcessKprobe events:
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk4ODc2MDI2NTk4OjEyNTc5OA==",
"pid": 125798,
"uid": 0,
"cwd": "/",
"binary": "/bin/su",
"arguments": "root",
"flags": "execve rootcwd clone",
"start_time": "2023-07-05T19:14:30.918693157Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://55936e548de63f77ceb595d64966dd8e267b391ff0ef63b26c17eb8c2f6510be",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-05T18:45:16Z",
"pid": 19
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "55936e548de63f77ceb595d64966dd8",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjYyMDM3MzMyOjEyNTc5Mg==",
"refcnt": 1,
"tid": 125798
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjYyMDM3MzMyOjEyNTc5Mg==",
"pid": 125792,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"flags": "execve rootcwd clone",
"start_time": "2023-07-05T19:14:27.704703805Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://55936e548de63f77ceb595d64966dd8e267b391ff0ef63b26c17eb8c2f6510be",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-05T18:45:16Z",
"pid": 13
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "55936e548de63f77ceb595d64966dd8",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjE2MTU0NzA2OjEyNTc4Mw==",
"refcnt": 2,
"tid": 125792
},
"function_name": "__x64_sys_setgid",
"args": [
{
"int_arg": 0
}
],
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-07-05T19:14:30.918977160Z"
}
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk4ODc2MDI2NTk4OjEyNTc5OA==",
"pid": 125798,
"uid": 0,
"cwd": "/",
"binary": "/bin/su",
"arguments": "root",
"flags": "execve rootcwd clone",
"start_time": "2023-07-05T19:14:30.918693157Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://55936e548de63f77ceb595d64966dd8e267b391ff0ef63b26c17eb8c2f6510be",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-05T18:45:16Z",
"pid": 19
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "55936e548de63f77ceb595d64966dd8",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjYyMDM3MzMyOjEyNTc5Mg==",
"refcnt": 1,
"tid": 125798
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjYyMDM3MzMyOjEyNTc5Mg==",
"pid": 125792,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"flags": "execve rootcwd clone",
"start_time": "2023-07-05T19:14:27.704703805Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://55936e548de63f77ceb595d64966dd8e267b391ff0ef63b26c17eb8c2f6510be",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-05T18:45:16Z",
"pid": 13
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "55936e548de63f77ceb595d64966dd8",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQwNzk1NjE2MTU0NzA2OjEyNTc4Mw==",
"refcnt": 2,
"tid": 125792
},
"function_name": "__x64_sys_setuid",
"args": [
{
"int_arg": 0
}
],
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-07-05T19:14:30.918990583Z"
}
In addition to the Kubernetes Identity and process metadata from exec events, ProcessKprobe events contain the arguments of the observed system call. In the above case they are:
function_name: the system call, __x64_sys_setuid or
__x64_sys_setgidint_arg: the uid or gid to use, in our case it’s 0 which corresponds to
the root user.To disable the process.credentials.changes.at.syscalls Tracing Policy run:
kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/process.credentials.changes.at.syscalls.yaml
Monitoring Process Credentials changes at the kernel layer is also possible.
This allows to capture the new process_credentials that should be applied.
This process-creds-installed tracing policy can be used to answer the following questions:
Which process or container is trying to change its own UIDs/GIDs in the cluster?
Which process or container is trying to change its own capabilities in the cluster?
In which user namespace the credentials are being changed?
How to monitor
process_credentialschanges?
The main advantages of monitoring at the kernel layer compared to the system call layer:
Not vulnerable to user space arguments tampering.
Ability to display the full new credentials to be applied.
It is more reliable since it has full context on where and how the new credentials should be applied including the user namespace.
A catch all layer for all system calls, and every normal kernel path that manipulate credentials.
One potential disadvantage is that this approach may generate a lot of events, so appropriate filtering must be applied to reduce the noise.
First, verify that your k8s environment is all setup and that all pods are up and running, and deploy the Demo Application:
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15.3/examples/minikube/http-sw-app.yaml
It might take several seconds until all pods are Running:
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default deathstar-54bb8475cc-6c6lc 1/1 Running 0 2m54s
default deathstar-54bb8475cc-zmfkr 1/1 Running 0 2m54s
default tiefighter 1/1 Running 0 2m54s
default xwing 1/1 Running 0 2m54s
kube-system tetragon-sdwv6 2/2 Running 0 27m
We use the process-creds-installed Tracing Policy that hooks the kernel layer when credentials are being installed.
So let’s apply the process-creds-installed Tracing Policy.
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/process-creds-installed.yaml
Then we start monitoring for events with tetra cli:
kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents
In another terminal, inside a pod and as a non root user we will execute a setuid binary (suid):
/tmp/su -
The tetra cli will generate the following ProcessKprobe events:
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE0MDczMDQyODk3MTc6MjIzODY=",
"pid": 22386,
"uid": 11,
"cwd": "/",
"binary": "/tmp/su",
"arguments": "-",
"flags": "execve rootcwd clone",
"start_time": "2023-07-25T12:04:59.359333454Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://2e58c8357465961fd96f758e87d0269dfb5f97c536847485de9d7ec62be34a64",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-25T11:44:48Z",
"pid": 43
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "2e58c8357465961fd96f758e87d0269",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEzOTU0MDY5OTA1ODc6MjIzNTI=",
"refcnt": 1,
"tid": 22386
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEzOTU0MDY5OTA1ODc6MjIzNTI=",
"pid": 22352,
"uid": 11,
"cwd": "/",
"binary": "/bin/sh",
"flags": "execve rootcwd",
"start_time": "2023-07-25T12:04:47.462035587Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://2e58c8357465961fd96f758e87d0269dfb5f97c536847485de9d7ec62be34a64",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-25T11:44:48Z",
"pid": 41
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "2e58c8357465961fd96f758e87d0269",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEzOTU0MDQ3NzY5NzI6MjIzNTI=",
"refcnt": 2,
"tid": 22352
},
"function_name": "commit_creds",
"args": [
{
"process_credentials_arg": {
"uid": 0,
"gid": 0,
"euid": 0,
"egid": 0,
"suid": 0,
"sgid": 0,
"fsuid": 0,
"fsgid": 0,
"caps": {
"permitted": [
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_NET_BIND_SERVICE",
"CAP_NET_RAW",
"CAP_SYS_CHROOT",
"CAP_MKNOD",
"CAP_AUDIT_WRITE",
"CAP_SETFCAP"
],
"effective": [
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_NET_BIND_SERVICE",
"CAP_NET_RAW",
"CAP_SYS_CHROOT",
"CAP_MKNOD",
"CAP_AUDIT_WRITE",
"CAP_SETFCAP"
]
},
"user_ns": {
"level": 0,
"uid": 0,
"gid": 0,
"ns": {
"inum": 4026531837,
"is_host": true
}
}
}
}
],
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-07-25T12:05:01.410834172Z"
}
In addition to the Kubernetes Identity and process metadata from exec events, ProcessKprobe events contain the arguments of the observed system call. In the above case they are:
function_name: the kernel commit_creds() function to install new credentials.process_credentials_arg: the new process_credentials to be installed
on the current process. It includes the UIDs/GIDs, the capabilities and the target user namespace.Here we can clearly see that the suid binary is being executed by a user ID 11 in order to elevate its privileges to user ID 0 including capabilities.
To disable the process-creds-installed Tracing Policy run:
kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/process-creds-installed.yaml
Some pods need to change the host system or kernel parameters in order to perform administrative tasks, obvious examples are pods loading a kernel module to extend the operating system functionality, or pods managing the network.
However, there are also other cases where a compromised container may want to load a kernel module to hide its behaviour.
In this aspect, monitoring such host system changes helps to identify pods and containers that affect the host system.
A kernel module is a code that can be loaded into the kernel image at runtime, without rebooting. These modules, which can be loaded by pods and containers, can modify the host system. The Monitor Linux kernel modules guide will assist you in observing such events.
Monitoring kernel modules helps to identify processes that load kernel modules to add features, to the operating system, to alter host system functionality or even hide their behaviour. This can be used to answer the following questions:
Which process or container is changing the kernel?
Which process or container is loading or unloading kernel modules in the cuslter?
Which process or container requested a feature that triggered the kernel to automatically load a module?
Are the loaded kernel modules signed?
After deploying Tetragon, use the monitor-kernel-modules tracing policy which generates ProcessKprobe events to trace kernel module operations.
Apply the monitor-kernel-modules tracing policy:
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/host-changes/monitor-kernel-modules.yaml
Then start monitoring for events with tetra CLI:
kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents
When loading an out of tree module named kernel_module_hello.ko with the command insmod,
tetra CLI will generate the following ProcessKprobe events:
{
"process_kprobe": {
"process": {
"exec_id": "OjEzMTg4MTQwNDUwODkwOjgyMDIz",
"pid": 82023,
"uid": 0,
"cwd": "/home/tixxdz/tetragon",
"binary": "/usr/sbin/insmod",
"arguments": "contrib/tester-progs/kernel_module_hello.ko",
"flags": "execve clone",
"start_time": "2023-08-30T11:01:22.846516679Z",
"auid": 1000,
"parent_exec_id": "OjEzMTg4MTM4MjY2ODQyOjgyMDIy",
"refcnt": 1,
"tid": 82023
},
"parent": {
"exec_id": "OjEzMTg4MTM4MjY2ODQyOjgyMDIy",
"pid": 82022,
"uid": 1000,
"cwd": "/home/tixxdz/tetragon",
"binary": "/usr/bin/sudo",
"arguments": "insmod contrib/tester-progs/kernel_module_hello.ko",
"flags": "execve",
"start_time": "2023-08-30T11:01:22.844332959Z",
"auid": 1000,
"parent_exec_id": "OjEzMTg1NTE3MTgzNDM0OjgyMDIx",
"refcnt": 1,
"tid": 0
},
"function_name": "security_kernel_read_file",
"args": [
{
"file_arg": {
"path": "/home/tixxdz/tetragon/contrib/tester-progs/kernel_module_hello.ko"
}
},
{
"int_arg": 2
}
],
"return": {
"int_arg": 0
},
"action": "KPROBE_ACTION_POST"
},
"time": "2023-08-30T11:01:22.847554295Z"
}
In addition to the process metadata from exec events, ProcessKprobe events contain the arguments of the observed call. In the above case they are:
security_kernel_read_file: the kernel security hook when the kernel loads file specified by user space.file_arg: the full path of the kernel module on the file system.
{
"process_kprobe": {
"process": {
"exec_id": "OjEzMTg4MTQwNDUwODkwOjgyMDIz",
"pid": 82023,
"uid": 0,
"cwd": "/home/tixxdz/tetragon",
"binary": "/usr/sbin/insmod",
"arguments": "contrib/tester-progs/kernel_module_hello.ko",
"flags": "execve clone",
"start_time": "2023-08-30T11:01:22.846516679Z",
"auid": 1000,
"parent_exec_id": "OjEzMTg4MTM4MjY2ODQyOjgyMDIy",
"refcnt": 1,
"tid": 82023
},
"parent": {
"exec_id": "OjEzMTg4MTM4MjY2ODQyOjgyMDIy",
"pid": 82022,
"uid": 1000,
"cwd": "/home/tixxdz/tetragon",
"binary": "/usr/bin/sudo",
"arguments": "insmod contrib/tester-progs/kernel_module_hello.ko",
"flags": "execve",
"start_time": "2023-08-30T11:01:22.844332959Z",
"auid": 1000,
"parent_exec_id": "OjEzMTg1NTE3MTgzNDM0OjgyMDIx",
"refcnt": 1,
"tid": 0
},
"function_name": "do_init_module",
"args": [
{
"module_arg": {
"name": "kernel_module_hello",
"tainted": [
"TAINT_OUT_OF_TREE_MODULE",
"TAINT_UNSIGNED_MODULE"
]
}
}
],
"action": "KPROBE_ACTION_POST"
},
"time": "2023-08-30T11:01:22.847638990Z"
}
This ProcessKprobe event contains:
do_init_module: the function call where the module is finaly loaded.module_arg: the kernel module information, it contains:
name: the name of the kernel module as a string.tainted: the module tainted flags that will be applied on the kernel. In the example above, it indicates we are loading an out-of-tree module, that is unsigned module which may compromise the integrity of our system.Kernels compiled with CONFIG_MODULE_SIG option will check if the modules being loaded were cryptographically signed.
This allows to assert that:
If the module being loaded is signed, the kernel has its key and the signature verification succeeded.
The integrity of the system or the kernel was not compromised.
After deploying Tetragon, use the monitor-signed-kernel-modules tracing policy which generates ProcessKprobe events to identify if kernel modules are signed or not.
Apply the monitor-signed-kernel-modules tracing policy:
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/host-changes/monitor-signed-kernel-modules.yaml
Before going forward, deploy the test-pod into the demo-app namespace, which has its security context set to privileged.
This allows to run the demo by mountig an xfs file system inside the test-pod which requires privileges,
but will also trigger an automatic xfs module loading operation.
kubectl create namespace demo-app
kubectl apply -n demo-app -f https://raw.githubusercontent.com/cilium/tetragon/main/testdata/specs/testpod.yaml
Start monitoring for events with tetra CLI:
kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents
In another terminal, kubectl exec into the test-pod and run the following commands to create an xfs filesystem:
kubectl exec -it -n demo-app test-pod -- /bin/sh
apk update
dd if=/dev/zero of=loop.xfs bs=1 count=0 seek=32M
ls -lha loop.xfs
apk add xfsprogs
mkfs.xfs -q loop.xfs
mkdir /mnt/xfs.volume
mount -o loop -t xfs loop.xfs /mnt/xfs.volume/
losetup -a | grep xfs
Now the xfs filesystem should be mounted at /mnt/xfs.volume. To unmount it and release the loop device run:
umount /mnt/xfs.volume/
tetra CLI will generate the following events:
First the mount command will trigger an automatic operation to load the xfs kernel module.
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjc1NTA0OTk5NTcyOjEzMDg3Ng==",
"pid": 130876,
"uid": 0,
"cwd": "/",
"binary": "/bin/mount",
"arguments": "-o loop -t xfs loop.xfs /mnt/xfs.volume/",
"flags": "execve rootcwd clone",
"start_time": "2023-09-09T23:27:42.732039059Z",
"auid": 4294967295,
"pod": {
"namespace": "demo-app",
"name": "test-pod",
"container": {
"id": "containerd://1e910d5cc8d8d68c894934170b162ef93aea5652867ed6bd7c620c7e3f9a10f1",
"name": "test-pod",
"image": {
"id": "docker.io/cilium/starwars@sha256:f92c8cd25372bac56f55111469fe9862bf682385a4227645f5af155eee7f58d9",
"name": "docker.io/cilium/starwars:latest"
},
"start_time": "2023-09-09T22:46:09Z",
"pid": 45672
},
"workload": "test-pod"
},
"docker": "1e910d5cc8d8d68c894934170b162ef",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjYyOTc1MjI1MDkzOjEzMDgwOQ==",
"refcnt": 1,
"tid": 130876
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjYyOTc1MjI1MDkzOjEzMDgwOQ==",
"pid": 130809,
"uid": 0,
"cwd": "/",
"binary": "/bin/sh",
"flags": "execve rootcwd clone",
"start_time": "2023-09-09T23:27:30.202263472Z",
"auid": 4294967295,
"pod": {
"namespace": "demo-app",
"name": "test-pod",
"container": {
"id": "containerd://1e910d5cc8d8d68c894934170b162ef93aea5652867ed6bd7c620c7e3f9a10f1",
"name": "test-pod",
"image": {
"id": "docker.io/cilium/starwars@sha256:f92c8cd25372bac56f55111469fe9862bf682385a4227645f5af155eee7f58d9",
"name": "docker.io/cilium/starwars:latest"
},
"start_time": "2023-09-09T22:46:09Z",
"pid": 45612
},
"workload": "test-pod"
},
"docker": "1e910d5cc8d8d68c894934170b162ef",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjYyOTEwMjM3OTQ2OjEzMDgwMA==",
"tid": 130809
},
"function_name": "security_kernel_module_request",
"args": [
{
"string_arg": "fs-xfs"
}
],
"return": {
"int_arg": 0
},
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-09-09T23:27:42.751151233Z"
}
In addition to the process metadata from exec events, ProcessKprobe event contains the arguments of the observed call. In the above case they are:
security_kernel_module_request: the kernel security hook where modules are loaded on-demand.string_arg: the name of the kernel module. When modules are automatically loaded, for security reasons,
the kernel prefixes the module with the name of the subsystem that requested it. In our case, it’s requested
by the file system subsystem, hence the name is fs-xfs.
The kernel will then call user space modprobe to load the kernel module.
{
"process_exec": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjc1NTI0MjYzMjIxOjEzMDg3Nw==",
"pid": 130877,
"uid": 0,
"cwd": "/",
"binary": "/sbin/modprobe",
"arguments": "-q -- fs-xfs",
"flags": "execve rootcwd clone",
"start_time": "2023-09-09T23:27:42.751301124Z",
"auid": 4294967295,
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"tid": 130877
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"pid": 0,
"uid": 0,
"binary": "<kernel>",
"flags": "procFS",
"start_time": "2023-09-09T11:59:47.227037763Z",
"auid": 0,
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"tid": 0
}
},
"node_name": "kind-control-plane",
"time": "2023-09-09T23:27:42.751300984Z"
}
The ProcessExec event where modprobe tries to load the xfs module.
modprobe is started in the initial Linux host namespaces, outside of the container namespaces. When kernel
modules are loaded on-demand, the kernel will spawn a user space process modprobe that finds and load the appropriate
module from the host file system. This is done on behalf of the container and since its originate from the kernel then
the inherited Linux namespaces including the file system are eventually from the host.
modprobe will read the passed xfs kernel module from the host file system.
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjc1NTI0MjYzMjIxOjEzMDg3Nw==",
"pid": 130877,
"uid": 0,
"cwd": "/",
"binary": "/sbin/modprobe",
"arguments": "-q -- fs-xfs",
"flags": "execve rootcwd clone",
"start_time": "2023-09-09T23:27:42.751301124Z",
"auid": 4294967295,
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"refcnt": 1,
"tid": 130877
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"pid": 0,
"uid": 0,
"binary": "<kernel>",
"flags": "procFS",
"start_time": "2023-09-09T11:59:47.227037763Z",
"auid": 0,
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"tid": 0
},
"function_name": "security_kernel_read_file",
"args": [
{
"file_arg": {
"path": "/usr/lib/modules/6.2.0-32-generic/kernel/fs/xfs/xfs.ko"
}
},
{
"int_arg": 2
}
],
"return": {
"int_arg": 0
},
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-09-09T23:27:42.752425825Z"
}
This ProcessKprobe event contains:
security_kernel_read_file: the kernel security hook when the kernel loads file specified by user space.file_arg: the full path of the kernel module on the host file system.
The final event is when the kernel is parsing the module sections. If all succeed the module will be loaded.
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjQxMjc1NTI0MjYzMjIxOjEzMDg3Nw==",
"pid": 130877,
"uid": 0,
"cwd": "/",
"binary": "/sbin/modprobe",
"arguments": "-q -- fs-xfs",
"flags": "execve rootcwd clone",
"start_time": "2023-09-09T23:27:42.751301124Z",
"auid": 4294967295,
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"refcnt": 1,
"tid": 130877
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"pid": 0,
"uid": 0,
"binary": "<kernel>",
"flags": "procFS",
"start_time": "2023-09-09T11:59:47.227037763Z",
"auid": 0,
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE6MA==",
"tid": 0
},
"function_name": "find_module_sections",
"args": [
{
"module_arg": {
"name": "xfs",
"signature_ok": true
}
}
],
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-09-09T23:27:42.760880332Z"
}
This ProcessKprobe event contains the module argument.
find_module_sections: the function call where the kernel parses the module sections.module_arg: the kernel module information, it contains:
name: the name of the kernel module as a string.signature_ok: a boolean value, if set to true then module signature was successfully verified by the kernel. If it is false
or missing then the signature verification was not performed or probably failed. In all cases this means the integrity of the system has been compromised. Depends on kernels compiled with CONFIG_MODULE_SIG option.Using the same monitor-kernel-modules tracing policy allows to monitor unloading of kernel modules.
The following ProcessKprobe event will be generated:
{
"process_kprobe": {
"process": {
"exec_id": "OjMzNzQ4NzY1MDAyNDk5OjI0OTE3NQ==",
"pid": 249175,
"uid": 0,
"cwd": "/home/tixxdz/tetragon",
"binary": "/usr/sbin/rmmod",
"arguments": "kernel_module_hello",
"flags": "execve clone",
"start_time": "2023-08-30T16:44:03.471068355Z",
"auid": 1000,
"parent_exec_id": "OjMzNzQ4NzY0MjQ4MTY5OjI0OTE3NA==",
"refcnt": 1,
"tid": 249175
},
"parent": {
"exec_id": "OjMzNzQ4NzY0MjQ4MTY5OjI0OTE3NA==",
"pid": 249174,
"uid": 1000,
"cwd": "/home/tixxdz/tetragon",
"binary": "/usr/bin/sudo",
"arguments": "rmmod kernel_module_hello",
"flags": "execve",
"start_time": "2023-08-30T16:44:03.470314558Z",
"auid": 1000,
"parent_exec_id": "OjMzNzQ2MjA5OTUxODI4OjI0OTE3Mw==",
"refcnt": 1,
"tid": 0
},
"function_name": "free_module",
"args": [
{
"module_arg": {
"name": "kernel_module_hello",
"tainted": [
"TAINT_OUT_OF_TREE_MODULE",
"TAINT_UNSIGNED_MODULE"
]
}
}
],
"action": "KPROBE_ACTION_POST"
},
"time": "2023-08-30T16:44:03.471984676Z"
}
To disable the monitor-kernel-modules run:
kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/host-changes/monitor-kernel-modules.yaml
Tetragon is able to observe various security events and even enforce security policies.
The Record Linux Capabilities Usage guide shows how to monitor and record Capabilities checks conducted by the kernel on behalf of applications during privileged operations. This can be used to inspect and produce security profiles for pods and containers.
When the kernel needs to perform a privileged operation on behalf of a process, it checks the Capabilities of the process and issues a verdict to allow or deny the operation.
Tetragon is able to record these checks performed by the kernel. This can be used to answer the following questions:
What is the capabilities profile of pods or containters running in the cluster?
What capabilities to add or remove when configuring a security context for a pod or container?
First, verify that your k8s environment is set up and that all pods are up and running, and deploy the demo application:
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15.3/examples/minikube/http-sw-app.yaml
It might take several seconds until all pods are Running:
kubectl get pods -A
The output should be similar to:
NAMESPACE NAME READY STATUS RESTARTS AGE
default deathstar-54bb8475cc-6c6lc 1/1 Running 0 2m54s
default deathstar-54bb8475cc-zmfkr 1/1 Running 0 2m54s
default tiefighter 1/1 Running 0 2m54s
default xwing 1/1 Running 0 2m54s
kube-system tetragon-sdwv6 2/2 Running 0 27m
We use the creds-capability-usage tracing policy which generates ProcessKprobe events.
Tracing policies as the one used here, may emit a high number of events.
To reduce events, the creds-capability-usage rate limits events to 1 minute. More details about rate-limiting can be found in the tracing policy documentation.
Apply the creds-capability-usage policy:
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/creds-capability-usage.yaml
Start monitoring for events with tetra cli, but match only events of xwing pod:
kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents --namespaces default --pods xwing
In another terminal, kubectl exec into the xwing pod:
kubectl exec -it xwing -- /bin/bash
As an example execute dmesg to print the kernel ring buffer. This requires the special capability CAP_SYSLOG:
dmesg
The output should be similar to:
dmesg: klogctl: Operation not permitted
The tetra cli will generate the following ProcessKprobe events:
{
"process_kprobe": {
"process": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEyODQyNzgzMzUwNjg0OjczODYw",
"pid": 73860,
"uid": 0,
"cwd": "/",
"binary": "/bin/dmesg",
"flags": "execve rootcwd clone",
"start_time": "2023-07-06T10:13:33.834390020Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://cfb961400ff25811d22d139a10f6a62efef53c2ecc11af47bc911a7f9a2ac1f7",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-06T08:07:30Z",
"pid": 171
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "cfb961400ff25811d22d139a10f6a62",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEyODQyMTI3MTIwOTcyOjczODUw",
"refcnt": 1,
"ns": {
"uts": {
"inum": 4026534655
},
"ipc": {
"inum": 4026534656
},
"mnt": {
"inum": 4026534731
},
"pid": {
"inum": 4026534732
},
"pid_for_children": {
"inum": 4026534732
},
"net": {
"inum": 4026534512
},
"time": {
"inum": 4026531834,
"is_host": true
},
"time_for_children": {
"inum": 4026531834,
"is_host": true
},
"cgroup": {
"inum": 4026534733
},
"user": {
"inum": 4026531837,
"is_host": true
}
},
"tid": 73860
},
"parent": {
"exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEyODQyMTI3MTIwOTcyOjczODUw",
"pid": 73850,
"uid": 0,
"cwd": "/",
"binary": "/bin/bash",
"flags": "execve rootcwd clone",
"start_time": "2023-07-06T10:13:33.178160018Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "xwing",
"container": {
"id": "containerd://cfb961400ff25811d22d139a10f6a62efef53c2ecc11af47bc911a7f9a2ac1f7",
"name": "spaceship",
"image": {
"id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6",
"name": "docker.io/tgraf/netperf:latest"
},
"start_time": "2023-07-06T08:07:30Z",
"pid": 165
},
"pod_labels": {
"app.kubernetes.io/name": "xwing",
"class": "xwing",
"org": "alliance"
}
},
"docker": "cfb961400ff25811d22d139a10f6a62",
"parent_exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEyODQyMDgxNTA3MzUzOjczODQx",
"refcnt": 2,
"tid": 73850
},
"function_name": "cap_capable",
"args": [
{
"user_ns_arg": {
"level": 0,
"uid": 0,
"gid": 0,
"ns": {
"inum": 4026531837,
"is_host": true
}
}
},
{
"capability_arg": {
"value": 34,
"name": "CAP_SYSLOG"
}
}
],
"return": {
"int_arg": -1
},
"action": "KPROBE_ACTION_POST"
},
"node_name": "kind-control-plane",
"time": "2023-07-06T10:13:33.834882128Z"
}
In addition to the Kubernetes Identity and process metadata from exec events, ProcessKprobe events contain the arguments of the observed system call. In the above case they are:
function_name: that is the cap_capable kernel function.
user_ns_arg: is the user namespace where the capability is required.
level: is the nested level of the user namespace. Here it is zero which indicates the initial user namespace.uid: is the user ID of the owner of the user namespace.gid: is the group ID of the owner of the user namespace.ns: details the information about the namespace. is_host indicates that the target user namespace where the capability is required is the host namespace.capability_arg: is the capability required to perform the operation. In this example reading the kernel ring buffer.
value: is the integer number of the required capability.name: is the name of the required capability. Here it is the CAP_SYSLOG.return: indicates via the int_arg if the capability check succeeded or failed. 0 means it succeeded and the access was granted while -1 means it failed and the operation was denied.
To disable the creds-capability-usage run:
kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/creds-capability-usage.yaml