Skyline Configuration on Check Point Servers that run Gaia OS - Prometheus with Grafana

Notes:

  • This section provides the steps for the Prometheus Server and the Grafana Server. For other monitoring tools, see Skyline Configuration on Check Point Servers that run Gaia OS - Other Monitoring Tools.

  • By default, Skyline on a Check Point Server uses a proxy server configured in one of these:

    • SmartConsole > Global properties > Proxy.

    • SmartConsole > the object of the Check Point Server > Network Management > Proxy.

    • The proxy server configured in the Gaia operating system on this Check Point Server.

    If this Check Point Server must connect to a specific destination directly (to bypass the configured proxy server), then configure an exception:

    1. Connect to the command line on this Check Point Server.

    2. Log in to the Expert mode.

    3. Edit this file:

      vi /opt/CPotelcol/no_proxy

      Skyline uses this file to configure the standard environment variable named "NO_PROXY".

    4. Add a comma-separated list of domains, IP addresses, or networks to exclude from the configured proxy server.

      Example:

      localhost,my.collector.host

    5. Save the changes in the file and exit the editor.

This section applies to these Check Point Servers:

  • Security Gateways / VSX Gateways.

  • ClusterXL Members.

    In a Cluster, you must configure all the Cluster Members in the same way.

  • Security Groups on Scalable Platforms (ElasticXL Cluster, Maestro, and Scalable Chassis).

  • Security Management Servers.

  • Multi-Domain Security Management Servers.

  • Multi-Domain Log Servers.

  • Log Servers.

  • SmartEvent Servers.

  • Endpoint Security Management Servers.

  • Endpoint Policy Servers.

Step 1 - Install the Prometheus Server

Note - Skip this step if you have already installed the Prometheus Server

Installing a Prometheus Server

To install a Prometheus Server on the external server, refer to the Prometheus installation instructions for the various platforms.

Prometheus Server Default URL

http://localhost:9090

Configuring a Prometheus Server

  1. Mandatory: On the Prometheus Server, enable the flag "--web.enable-remote-write-receiver" to get metrics data from the Check Point Servers. Refer to these Prometheus instructions.
    Note - The "--web.enable-remote-write-receiver" is a command-line flag used when starting a Prometheus server. It enables the Prometheus instance to act as a receiver for remote write requests.

  2. Optional: Use TLS Encryption and Basic authentication to secure the connection between the Prometheus Server and the OpenTelemetry Collector.

    The Prometheus Server and OpenTelemetry Collector support Transport Layer Security (TLS) encryption for their connection. Refer to these Prometheus instructions.

    Check Point also requires you to enable basic authentication to make the security bi-directional, refer to these Prometheus instructions.

    TLS configuration has two main components:

    • A pair of a Key and a Certificate, used to encrypt your communication.

    • (Optional) Certificate Authority (CA) that you trust, used to verify and trust the certificate of the other endpoint with which you communicate. If the certificate of the other endpoint is unknown to the CA, the communication is rejected.

    You can create these certificates:

    • CA-signed certificates: You create a key and a certificate request, which is then signed by the CA.

    • Self-signed certificates: You create a key and a certificate that is signed by the user.

    The steps below describe a self-signed certificate.

    To configure TLS, you must create two pairs of a key and a certificate on the Prometheus Server:

    1. Create a self-signed certificate and a private key on the Prometheus Server:

      1. Create the file called openssl.conf with the template below.

        Enter the applicable information in the "[ dn ]" and "[ alt_names ]" sections.

        Important - The Prometheus Server and the OpenTelemetry Collector must have different hostnames and IP addresses.

        Template:

        [ req ]
default_bits       = 4096
default_md         = sha256
req_extensions     = v3_req
distinguished_name = dn
prompt = no

[ v3_req ]
subjectAltName = @alt_names

[ dn ]
C = <Country Name>
ST = <State Or Province Name>
L = <Locality>
O = <Organization>
OU = <Organization Unit>
CN = <Common Name>

[ alt_names ]
DNS = <HOSTNAME>
IP = <IP ADDRESS>

      2. Generate the key and certificate pair on any Gaia OS server (in the Expert mode):

        cpopenssl req -x509 -newkey rsa:4096 -nodes -config openssl.conf -keyout mykey.key -out mycert.crt -extensions v3_req

        This command creates two files in the current working directory:

        • mykey.key

        • mycert.crt

        Move these two files from the Gaia OS server to the Prometheus Server.

    2. Use the newly generated key and certificate file to configure TLS on the Prometheus Server in the web-config.yaml configuration file (you may need to create this file).

      Example of a web-config.yaml file:

      tls_server_config:
   key_file: /home/admin/mykey.key
   cert_file: /home/admin/mycert.crt

Step 2 - Install the Grafana Server

Note - Skip this step if you have already installed the Grafana Server.

Installing a Grafana Server

To install a Grafana Server on the external server, refer to the Grafana installation instructions.

Note - You can install the Grafana Server on the same server that contains your Prometheus Server instance.

Grafana Server Default URL

http://localhost:3000

Configuring a Grafana Server (connecting Data-source and Dashboards)

  1. Log in to Grafana and add a new Prometheus Data Source with the Prometheus Server you created.

    On the side panel, click Home > Connections > click the Add data source button > select the Prometheus data source.

    Make sure to mark the Prometheus data source as Default.

    Example:

  2. Import the Check PointGrafana dashboard to start monitoring your environment.

    On the side panel, click Create > Import > upload the applicable JSON file for a dashboard.

    Note - For the JSON packages, refer to sk178566.

    Example:

  3. You can modify the existing dashboards or create new dashboards according to your needs.

    To see a full description of all the data exposed by Skyline, see Skyline Metrics Repository.

    Example:

Step 3 - Install the OpenTelemetry Agent and OpenTelemetry Collector on the Check Point Server

On an online Check Point Server:

The OpenTelemetry Agent (OtlpAgent) and the OpenTelemetry Collector (CPotelcol) packages are installed automatically on all applicable Check Point Servers, if you allowed the Automatic Update downloads as described in sk94508.

On an offline Check Point Server:

The minimum required OpenTelemetry Agent and OpenTelemetry Collector packages are also installed as part of the Jumbo Hotfix Accumulator installation.

See sk178566 > section "Requirements".

Related offline update packages:

Step 4 - Configure the OpenTelemetry Collector on the Check Point Server to work with the Prometheus Server

Important:

  • Do not cut and paste the commands below. Enter them manually.

  • In a Cluster, you must configure all the Cluster Members in the same way.

  • In Management High Availability, we recommend to configure all the Management Servers in the same way.

  • On a Maestro Orchestrator (MHO):

    You can run the CLI command "sklnctl" (in the Expert mode) or run the Gaia REST API command (requires Gaia API v1.7 or higher).

    The CLI command runs on the Orchestrator and configures only the Orchestrator.

  • In a Scalable Platform Security Group:

    You can run the CLI command "g_all sklnctl" (in the Expert mode) or run the Gaia REST API command (requires Gaia API v1.7 or higher).

  • For the applicable Gaia REST API commands, refer to the Check Point Gaia API Reference (at the top, select the correct version) (v1.7 and higher):

    • In Gaia API v1.8 and higher: Section "Diagnostics" > Section "OpenTelemetry.

    • In Gaia API v1.7: Section "OpenTelemetry".

This step provides two different procedures - for an on-premises Prometheus Server, and for a Prometheus Server in AWS.

  1. Prepare the required payload for the command:

    Notes:

    • Download the sample payload files from sk178566 > section "Downloads".

      Replace the placeholder strings "<...>" in the sample files with your actual strings.

    • In the "basic" section, configure the "username" and "password" attributes to your monitoring server's username / password.

      The "password" attribute supports only these characters:

      • uppercase letters (A-Z)
      • lowercase letters (a-z)
      • digits (0-9)
      • underscore (_)
      • comma (,)
      • period (.)
      • backslash (\)
      • slash (/)
      • hyphen (-)

    • In the "ca-public-key" section, configure the TLS settings with the CA certificate of your monitoring server (as PEM X509), or use the self-signed certificate you generated previously.

      Replace "<CERTIFICATE>" with the CA certificate of the monitoring server (PEX X509) - paste the entire string:

      -----BEGIN CERTIFICATE-----<BASE64_TEXT>-----END CERTIFICATE-----

    • In the "url" attribute, configure your monitoring server's IP address / URL.

      If you do not wish to use TLS encryption in labs or test environments, then make these changes in the payload:

      1. In the "url" attribute, change the URL protocol from "https://" to "http://".

      2. Remove the "client-auth" and "server-auth" attributes.

        Note - When you apply the payload, this warning appears: "it is recommended to have both client and server authentication").

    Example JSON payloads:

    {
    "enabled": true,
    "export-targets": {
        "add": [
            {
                "client-auth": {
                    "basic": {
                        "username": "<USERNAME>",
                        "password": "<PASSWORD>"
                    }
                },
                "enabled": true,
                "server-auth": {
                    "ca-public-key": {
                        "type": "PEM-X509",
                        "value": "<CERTIFICATE>"
                    }
                },
                "type": "prometheus-remote-write",
                "url": "https://<EXTERNAL_PROMETHEUS_IP_ADDRESS>:9090/api/v1/write"
            }
        ]
    }
}
    		 
    {
    "enabled": true,
    "export-targets": {
        "add": [
            {
                "enabled": true,
                "type": "prometheus-remote-write",
                "url": "http://<EXTERNAL_PROMETHEUS_IP_ADDRESS>:9090/api/v1/write"
            }
        ]
    }
}

  2. Run the configuration command to apply the payload - either the CLI command (in the Expert mode), or the Gaia REST API command:

    • Method 1 - Run the CLI command "sklnctl":

      1. Save the JSON payload in a file (for example, /home/admin/payload.json).

      2. On a Scalable Platform Security Group, copy the JSON payload file to all Security Group Members:

        asg_cp2blades /home/admin/payload.json

      3. Run this command:

        • On a Security Gateway / each Cluster Member / Management Server / Log Server / SmartEvent Server:

          sklnctl export --set "$(cat /home/admin/payload.json)"

        • On a Scalable Platform Security Group:

          g_all sklnctl export --set "$(cat /home/admin/payload.json)"

    • Method 2 - Run the Gaia REST API command "set-open-telemetry" (requires Gaia API v1.7 and higher):

      POST{server}/v1.7/set-open-telemetry
Content-Type: application/json
X-chkp-sid: {{session}}
{
   <JSON Payload>
}

      Note - To disable Skyline completely:

      1. Change the value of the "enabled" attribute in the JSON payload file:

        from "enabled": true

        to "enabled": false

      2. Run the API command again

  1. Prepare the required payload for the command:

    Notes:

    • Refer to the AWS official documentation for detailed information about how to generate the keys.

    • The Session Token is optional.

    To work with a single Export Target over HTTPS:

    {
    "enabled": true,
    "export-targets": {
        "add": [
            {
                "server-auth": {
                    "sigv4auth": {
                        "region": "<Region>",
                        "aws-access-key-id": "<Access Key ID>",
                        "aws-secret-access-key": "<Access Key>",
                        "session-token": "<Session Token>"
                    }
                },
                "enabled": true,
                "type": "prometheus-remote-write",
                "url": "https://<PROMETHEUS_IP_ADDRESS>:9090/api/v1/write"
            }
        ]
    }
}

    To work with multiple Export Targets over HTTPS (this example shows two):

    {
    "enabled": true,
    "export-targets": {
        "add": [
            {
                "server-auth": {
                    "sigv4auth": {
                        "region": "<Region>",
                        "aws-access-key-id": "<Access Key ID>",
                        "aws-secret-access-key": "<Access Key>",
                        "session-token": "<Session Token>"
                    }
                },
                "enabled": true,
                "type": "prometheus-remote-write-1",
                "url": "https://<PROMETHEUS_IP_ADDRESS_#1>:9090/api/v1/write",
                "name": "My-Prometheus-Target-1"
            },
            {
                "enabled": true,
                "type": "prometheus-remote-write-2",
                "url": "https://<PROMETHEUS_IP_ADDRESS_#2>:9090/api/v1/write",
                "name": "My-Prometheus-Target-2"
            }
        ]
    }
}

    To work with multiple Export Targets over HTTPS and over HTTP (this example shows two):

    {
    "enabled": true,
    "export-targets": {
        "add": [
            {
                "server-auth": {
                    "sigv4auth": {
                        "region": "<Region>",
                        "aws-access-key-id": "<Access Key ID>",
                        "aws-secret-access-key": "<Access Key>",
                        "session-token": "<Session Token>",
                        "enabled": true,
                        "type": "prometheus-remote-write-1",
                        "url": "https://<PROMETHEUS_IP_ADDRESS_#1>:9090/api/v1/write",
                        "name": "My-Prometheus-Target-1"
                    }
                }
            },
            {
                "enabled": true,
                "type": "prometheus-remote-write-2",
                "url": "http://<PROMETHEUS_IP_ADDRESS_#2>:9090/api/v1/write",
                "name": "My-Prometheus-Target-2"
            }
        ]
    }
}

  2. Run the configuration command to apply the payload - either the CLI command (in the Expert mode), or the Gaia REST API command:

    • Method 1 - Run the CLI command "sklnctl":

      1. Save the JSON payload in a file (for example, /home/admin/payload_AWS.json).

      2. On a Scalable Platform Security Group, copy the JSON payload file to all Security Group Members:

        asg_cp2blades /home/admin/payload_AWS.json

      3. Run this command:

        • On a Security Gateway / each Cluster Member / Management Server / Log Server / SmartEvent Server:

          sklnctl export --set "$(cat /home/admin/payload_AWS.json)"

        • On a Scalable Platform Security Group:

          g_all sklnctl export --set "$(cat /home/admin/payload_AWS.json)"

    • Method 2 - Run the Gaia REST API command "set-open-telemetry" (requires Gaia API v1.7 and higher):

      POST{server}/v1.7/set-open-telemetry
Content-Type: application/json
X-chkp-sid: {{session}}
{
   <JSON AWS Payload>
}

Step 5 - Configure the filter for the OpenTelemetry Collector exported metrics

The sklnctl tool configures the OpenTelemetry Collector.

The OpenTelemetry Collector filter works on the allow-list basis.

Notes:

  • For the available commands in the Expert mode, see CLI Syntax for 'sklnctl'.

  • On a Scalable Platform Security Group, run the "g_all sklnctl <options>" command.

Example workflow (in the Expert mode):

  1. Show the currently exported metrics:

    sklnctl otelcol metrics --show > /var/log/metrics.txt

  2. Edit the file to keep only the desired metrics:

    vi /var/log/metrics.txt

    See Skyline Metrics Repository.

  3. On a Scalable Platform Security Group, copy the file to all Security Group Members:

    asg_cp2blades /var/log/metrics.txt

  4. Add the desired metrics to the allow-list:

    • On a Security Gateway / each Cluster Member / Management Server / Log Server / SmartEvent Server:

      sklnctl otelcol metrics --add $(cat /var/log/metrics.txt | tr '\n' ' ')

    • On a Scalable Platform Security Group:

      g_all sklnctl otelcol metrics --add $(cat /var/log/metrics.txt | tr '\n' ' ')

Step 6 - Configure Access Control Policy

If you configured Skyline on a Security Gateway, ClusterXL, or Scalable Platform Security Group, then you must make sure your Access Control Policy allows the connection to the Prometheus Server to send the exported metrics.

You must configure the required rule on the Management Server (in SmartConsole or with Management API) and install the policy.

See the: