This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Enforcement

Documentation for Tetragon enforcement system

1: Persistent enforcement
2: Persistent gRPC policies

Tetragon allows enforcing events in the kernel inline with the operation itself. This document describes the types of enforcement provided by Tetragon and concerns policy implementors must be aware of.

There are two ways that Tetragon performs enforcement: overriding the return value of a function and sending a signal (e.g., SIGKILL) to the process.

Override return value

Override the return value of a call means that the function will never be executed and, instead, a value (typically an error) will be returned to the caller. Generally speaking, only system calls and security check functions allow to change their return value in this manner. Details about how users can configure tracing policies to override the return value can be found in the Override action documentation.

Signals

Another type of enforcement is signals. For example, users can write a TracingPolicy (details can be found in the Signal action documentation) that sends a SIGKILL to a process matching certain criteria and thus terminate it.

In contrast with overriding the return value, sending a SIGKILL signal does not always stop the operation being performed by the process that triggered the operation. For example, a SIGKILL sent in a write() system call does not guarantee that the data will not be written to the file. However, it does ensure that the process is terminated synchronously (and any threads will be stopped). In some cases it may be sufficient to ensure the process is stopped and the process does not handle the return of the call. To ensure the operation is not completed, though, the Signal action should be combined with the Override action.

1 - Persistent enforcement

How to configure persistent enforcement

This page shows you how to configure persistent enforcement.

Concept

The idea of persistent enforcement is to allow the enforcement policy to continue running even when its tetragon process is gone.

This is configured with the --keep-sensors-on-exit option.

When the tetragon process exits, the policy stays active because it’s pinned in sysfs bpf tree under /sys/fs/bpf/tetragon directory.

When a new tetragon process is started, it performs the following actions:

checks if there’s existing /sys/fs/bpf/tetragon and moves it to /sys/fs/bpf/tetragon_old directory;
sets up configured policy;
removes /sys/fs/bpf/tetragon_old directory.

Example

This example shows how the persistent enforcement works on simple tracing policy.

Consider the following enforcement tracing policy that kills any process that touches /tmp/tetragon file.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: "enforcement"
spec:
 kprobes:
 - call: "fd_install"
   syscall: false
   args:
   - index: 0
     type: int
   - index: 1
     type: "file"
   selectors:
   - matchArgs:
     - index: 1
       operator: "Equal"
       values:
       - "/tmp/tetragon"
     matchActions:
     - action: Sigkill

Spawn tetragon with the above policy and --keep-sensors-on-exit option.

tetragon --bpf-lib bpf/objs/ --keep-sensors-on-exit --tracing-policy enforcement.yaml

Verify that the enforcement policy is in place.
```
cat /tmp/tetragon
```
The output should be similar to
```
Killed
```

Kill tetragon with CTRL+C.

time="2024-07-26T14:47:45Z" level=info msg="Perf ring buffer size (bytes)" percpu=68K total=272K
time="2024-07-26T14:47:45Z" level=info msg="Perf ring buffer events queue size (events)" size=63K
time="2024-07-26T14:47:45Z" level=info msg="Listening for events..."
^C
time="2024-07-26T14:50:50Z" level=info msg="Received signal interrupt, shutting down..."
time="2024-07-26T14:50:50Z" level=info msg="Listening for events completed." error="context canceled"

Verify that the enforcement policy is STILL in place.
```
cat /tmp/tetragon
```
The output should be still similar to
```
Killed
```

Limitations

At the moment we are not able to receive any events during the tetragon down time, only the the enforcement is in place.

2 - Persistent gRPC policies

Persist and restore tracing policies installed through gRPC

Tetragon can persist tracing policies installed through the gRPC API and restore their desired state after the agent restarts. Combined with persistent enforcement, this allows the old BPF programs to remain active while Tetragon loads replacement programs from the stored policies.

This feature applies only to policies installed through gRPC, including policies added with tetra tracingpolicy add. Kubernetes and statically loaded policies continue to be managed by their respective policy sources.

Configuration

Start Tetragon with the following options:

sudo ./tetragon \
  --btf /sys/kernel/btf/vmlinux \
  --bpf-lib ./bpf/objs/ \
  --keep-sensors-on-exit \
  --release-pinned-bpf=false \
  --persist-grpc-policies \
  --persist-grpc-policies-dir /var/run/tetragon/grpc-policies

Use the same options each time Tetragon starts.

Caution --release-pinned-bpf defaults to true. Set it explicitly to false when continuous enforcement across an agent restart is required. Enabling only the policy store restores policies, but does not preserve the previous BPF tree during startup.

The options have the following roles:

--keep-sensors-on-exit leaves pinned sensors active when Tetragon exits.
--release-pinned-bpf=false prevents Tetragon from removing the previous BPF tree at the beginning of startup. Tetragon moves the tree aside and removes it only after replacement policies have loaded.
--persist-grpc-policies enables saving and restoring policies installed through gRPC.
--persist-grpc-policies-dir selects the policy store directory. Its default value is /var/run/tetragon/grpc-policies.

Restart sequence

On a normal restart, Tetragon performs the following handoff:

Rename /sys/fs/bpf/tetragon to /sys/fs/bpf/tetragon_old. The links pinned below this directory keep the previous BPF programs attached.
Create a new /sys/fs/bpf/tetragon tree and load the initial sensors.
Read and validate all records in the gRPC policy store.
Recreate each stored policy in the new BPF tree. Tetragon registers disabled policies without loading their BPF programs.
Remove /sys/fs/bpf/tetragon_old, which detaches the previous programs.

The policies are rebuilt from the persistent store; Tetragon does not reload them from tetragon_old. The old BPF tree exists only to maintain coverage during the handoff.

Note There is an interval during startup in which both the old and replacement programs are attached. This avoids an enforcement gap.

Manage persisted policies

Policies added through gRPC are persisted automatically after they load successfully:

sudo ./tetra tracingpolicy add policy.yaml

Deleting a policy removes it from both the running agent and the persistent store:

sudo ./tetra tracingpolicy delete policy-name

Enable and disable operations update the stored desired state:

sudo ./tetra tracingpolicy disable policy-name
sudo ./tetra tracingpolicy enable policy-name

Mode changes also update the stored policy YAML:

sudo ./tetra tracingpolicy set-mode policy-name monitor
sudo ./tetra tracingpolicy set-mode policy-name enforce

Stored desired state

The store uses one JSON record per policy. A policy is uniquely identified by its domain, namespace, and name.

Each record contains:

yaml: the complete policy, including its configured policy mode;
enabled: whether the policy is enabled or not.

The policy identity is encoded in the record filename as <name>:<namespace>:<domain>.json.

For example:

{
  "yaml": "apiVersion: cilium.io/v1alpha1\nkind: TracingPolicy\nmetadata:\n  name: lsm-file-open\nspec:\n  options:\n  - name: policy-mode\n    value: monitor\n  lsmhooks:\n  - hook: file_open\n",
  "enabled": true
}

Failure behavior

The running policy and its stored desired state are updated together:

If adding a new record to the store fails, Tetragon removes the newly loaded runtime policy and restores any previous store state.
Enable, disable, and mode changes write the desired state before changing the running policy. If the runtime change fails, Tetragon restores the previous record.
Deleting a policy removes its record before deleting the running policy. If the runtime deletion fails, Tetragon restores the record.

At startup, Tetragon validates every stored policy before loading any of them. If loading a policy fails, it rolls back policies restored earlier in the same startup attempt and reports the error. The old BPF tree is removed only after startup policy loading succeeds.