Image Assurance Troubleshooting

Verify the Agent Installation Status

Installation of the Inventory agent is a basic requirement to run Image Assurance.

• On the Image Assurance page, you can see the status for all nodes, while practically only the engine pod The smallest and simplest Kubernetes object. A pod represents a set of running containers on your cluster. A pod is typically set up to run a single primary container. It can also run optional sidecar containers that add supplementary features like logging. Pods are commonly managed by a Deployment. connects with CloudGuard

• Installed pods:

  • An Asset management agent (Inventory deployment) - A basic requirement for Image Assurance.

  • A minimum of one imagescan-engine pod.

  • For CloudGuard Helm Chart A package of pre-configured Kubernetes resources that can be managed with the Helm tool. Charts provide a reproducible way of creating and sharing Kubernetes applications. A single chart can be used to deploy something simple, like a memcached Pod, or something complex, like a full web app stack with HTTP servers, databases, caches, and so on. versions from 2.13.0, an imagescan-list pod.

  • Daemonset of imagescan-daemon pods - Based on the number of nodes. In addition, it runs on available control plane nodes.

    Example of a cluster with 2 nodes:

    ~$ kubectl -n checkpoint get pods NAME READY STATUS RESTARTS AGE asset-mgmt-imagescan-daemon-blzlj 2/2 Running 0 5m asset-mgmt-imagescan-daemon-kmftj 2/2 Running 0 5m asset-mgmt-imagescan-engine-767fdb686f-rqmlc 1/1 Running 0 5m asset-mgmt-imagescan-list-5b48574cbd-q7g4p 1/1 Running 0 5m asset-mgmt-inventory-agent-79764cc64f-dhrpk 1/1 Running 0 5m

• Get pod - Make sure that all the containers are ready:

  • imagescan-engine (central) - 1 pod - 1/1 ready

  • imagescan-list - 1 pod - 1/1 ready

  • imagescan-daemon - X pods (where X is the number of nodes) - 2/2 ready for each pod; for Docker Docker (specifically, Docker Engine) is a software technology providing operating-system-level virtualization also known as containers. container runtime, it is 1/1

• Get logs - Make sure that you do not see errors:

  • inventory-agent pod - Make sure that there are no errors in logs and that all the entities uploaded successfully.

  • imagescan-engine (central) - Make sure that there are no errors, and that scans are completed successfully.

  • imagescan-list - Make sure that there are no errors and that image lists are uploaded successfully.

  • imagescan-daemon - Make sure that there are no errors and that the imagescan-list and imagescan-engine pods can get image list and image content from these agents.

    Example:

    # kubectl -n checkpoint logs asset-mgmt-imagescan-engine-6f7bf8787b-tkw77 engine | grep -i error

• Basic use - When Image Assurance is deployed, user interaction is not necessary (only for advanced configuration settings). It scans running images first, from the smallest to the largest.

Central Agent Environment Variables

Name	Workloads	Default Value	Max Value	Comments
LOG_LEVEL	imagescan-engine imagescan-daemon imagescan-list	info	N/A	Possible values: debug, trace, warn, error The same for the imagescan-daemon agent
CP_IMAGESCAN_INTERNAL_PROTO	imagescan-engine imagescan-daemon imagescan-list	HTTPS	N/A	If you set it as HTTP, agents use HTTP and not HTTPS for communication in the cluster.
CP_IMAGESCAN_SCAN_TIMEOUT	imagescan-engine	N/A	24h	By default, the scan timeout is set in by the CloudGuard engine. It is possible to override this value. The timeout is in seconds.
IMAGE_TRANSFER_TIMEOUT_SECONDS	imagescan-engine imagescan-daemon	N/A	24h	Kubernetes Kubernetes, often abbreviated as “K8s”, orchestrates containerized applications to run on a cluster of hosts. image is transferred between imagescan-daemon and imagescan-engine pods to be scanned by the imagescan-engine. This variable configures the timeout for the transfer operation. The timeout is in seconds. Important - Define the environment variable for two workloads and use the same value for each of them.

To configure the environment variables:

1. Edit the applicable imagescan workload and add one or more applicable variables with valid values. Use one of these methods:

   • Edit a running deployment, for example:

     kubectl edit deployment asset-mgmt-imagescan-image -n checkpoint

   • Set the environment variables with the Helm A Kubernetes deployment tool for automating creation, packaging, configuration, and deployment of applications and services to Kubernetes clusters. commands.

     Examples

     • To set CP_IMAGESCAN_INTERNAL_PROTO to HTTP, run:

       --set-string addons.imageScan.daemon.env[0].name=CP_IMAGESCAN_INTERNAL_PROTO,addons.imageScan.daemon.env[0].value="HTTP" \ --set-string addons.imageScan.engine.env[0].name=CP_IMAGESCAN_INTERNAL_PROTO,addons.imageScan.engine.env[0].value="HTTP" \ --set-string addons.imageScan.list.env[0].name=CP_IMAGESCAN_INTERNAL_PROTO,addons.imageScan.list.env[0].value="HTTP"

     • To set IMAGE_TRANSFER_TIMEOUT_SECONDS to 1 hour and CP_IMAGESCAN_SCAN_TIMEOUT to 2 hours, run:

       --set-string addons.imageScan.daemon.env[0].name=IMAGE_TRANSFER_TIMEOUT_SECONDS,addons.imageScan.daemon.env[0].value=3600 \ --set-string addons.imageScan.engine.env[0].name=IMAGE_TRANSFER_TIMEOUT_SECONDS,addons.imageScan.engine.env[0].value=3600 \ --set-string addons.imageScan.engine.env[1].name=CP_IMAGESCAN_SCAN_TIMEOUT,addons.imageScan.engine.env[1].value=7200

Note - When you set manually more than one environment variable, increase accordingly the index of the variable (0 and 1 in the above).

Istio

As Istio adds HTTPS proxies which break mutual TLS between ImageScan engine and ImageScan daemon agents. Change the protocol they use to connect to HTTP. It is done through environment variables passed to ImageScan Engine deployment and ImageScan daemon DaemonSet: CP_IMAGESCAN_INTERNAL_PROTO=HTTP. For this, append the below lines to the Helm installation or upgrade command (index 0 is used assuming no other environment variables are changed):

--set-string addons.imageScan.daemon.env[0].name=CP_IMAGESCAN_INTERNAL_PROTO,addons.imageScan.daemon.env[0].value="HTTP" \

--set-string addons.imageScan.engine.env[0].name=CP_IMAGESCAN_INTERNAL_PROTO,addons.imageScan.engine.env[0].value="HTTP" \

If the list deployment exists, add this line:

--set-string addons.imageScan.list.env[0].name=CP_IMAGESCAN_INTERNAL_PROTO,addons.imageScan.list.env[0].value="HTTP" \

Low Rate of Image Scan

By default, there is only one image scan engine that scans images sequentially. You can increase the rate of image scanning. For this, deploy more image scan engines on your cluster.

• Add this parameter to the Helm command:

  --set addons.imageScan.engine.replicaCount=<number-of-scanners>

Common Errors

1. When the engine and daemon pods cannot connect and the ImageScan engine reports a timeout error, it can be because of the cluster configuration.

   handleImageListTask returned error: get image list failed from agent at 172.17.0.3 (ubuntu2004): from https://consec1-imagescan-daemon:8443/imagelist: Get "https://172.17.0.3:8443/imagelist: dial tcp 172.17.0.3:8443: i/o timeout

2. When the engine and daemon pods cannot connect and the ImageScan engine reports a certificate validation error, it can occur one time (for each connection with the ImageScan daemon pod) after upgrading Helm. This occurs because pods are restarted one after the other, and some can use certificates from before for internal communication, and some possibly already use the new ones.

   Example

   get image failed from agent at 192.168.50.164 (ip-192-168-29-125.ec2.internal): from https://asset-mgmt-imagescan-daemon:8443/getimage/sha256:ee597f5bb5bc95c01d79a04ed053a388d05836c96ae3aed117df5b2fea81f6aa: Get "https://192.168.50.164:8443/getimage/sha256:ee597f5bb5bc95c01d79a04ed053a388d05836c96ae3aed117df5b2fea81f6aa": x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate" while trying to verify candidate authority certificate "asset-mgmt-imagescan-daemon"

3. Sometimes the Image Assurance Agent status shows the error message: "No images found, please set containerRuntime to be docker", although the cluster worked before. It is possible that your cluster has recently changed its containerRuntime.

   Solution: Run the Helm upgrade command again to solve the issue.