Troubleshooting

Learn how to troubleshoot Tetragon

Automatic log and state collection

Before you report a problem, make sure to retrieve the necessary information from your cluster.

Tetragon’s bugtool captures potentially useful information about your environment for debugging. The tool is meant to be used for debugging a single Tetragon agent node but can be run automatically in a cluster. Note that in the context of Kubernetes, the command needs to be run from inside the Tetragon Pod’s container.

Key information collected by bugtool:

Tetragon configuration
Network configuration
Kernel configuration
eBPF maps
Process traces (if tracing is enabled)

Automatic Kubernetes cluster sysdump

You can collect information in a Kubernetes cluster using the Cilium CLI:

cilium-cli sysdump

More details can be found in the Cilium docs. The Cilium CLI sysdump command will automatically run tetra bugtool on each nodes where Tetragon is running.

Manual single node sysdump

It’s also possible to run the bug collection tool manually with the scope of a single node using tetra bugtool.

Kubernetes installation

Identify the Tetragon Pod (<tetragon-namespace> is likely to be kube-system with the default install):
```
kubectl get pods -n <tetragon-namespace> -l app.kubernetes.io/name=tetragon
```

Execute tetra bugtool within the Pod:

kubectl exec -n <tetragon-namespace> <tetragon-pod-name> -c tetragon -- tetra bugtool

Retrieve the created archive from the Pod’s filesystem:

kubectl cp -c tetragon <tetragon-namespace>/<tetragon-pod-name>:tetragon-bugtool.tar.gz tetragon-bugtool.tar.gz

Container installation

Enter the Tetragon Container:

docker exec -it <tetragon-container-id> tetra bugtool

Retrieve the archive using docker cp:

docker cp <tetragon-container-id>:/tetragon-bugtool.tar.gz tetragon-bugtool.tar.gz

Systemd host installation

Execute tetra bugtool with Elevated Permissions:
```
sudo tetra bugtool
```

Enable debug log level

When debugging, it might be useful to change the log level. The default log level is controlled by the log-level option at startup:

Enable debug level with --log-level=debug
Enable trace level with --log-level=trace

Change log level on Kubernetes

Warning The Pods of the Tetragon DaemonSet will be restarted automatically after changing the debug Helm value.

It is possible to change the log level of Tetragon’s DaemonSet Pods by setting tetragon.debug to true.

Change log level dynamically

It is possible to change the log level dynamically by using the tetra loglevel sub-command. tetra needs access to Tetragon’s gRPC server endpoint which can be configured via --server-address.

Warning Keep in mind that Tetragon’s gRPC server is (by default) only exposed on localhost. Also, it’s important to understand that this only changes the log level of the single Tetragon instance targeted with --server-address and not all Tetragon instances when it’s, for example, running as DaemonSet in a Kubernetes environment.

Get the current log level:
```
tetra loglevel get
```
Dynamically change the log level. Allowed values are [trace|debug|info|warning|error|fatal|panic]:
```
tetra loglevel set debug
```