Exploring Red Hat Insights integration with PagerDuty

Red Hat Insights is a managed service included in every Red Hat subscription. It continuously analyzes platforms and applications to help enterprises manage hybrid cloud environments. It also triggers events through its Notifications service. Each account configures how and who can receive these events, with the ability to perform actions.

A common example is to forward new recommendations found on Red Hat Enterprise Linux (RHEL) system configuration to specific teams by email and/or create a new ticket in an IT Service Management (ITSM) solution. Through its Integrations service, Insights provides endpoint integrations to Splunk, ServiceNow, Slack and any third-party application with its generic webhooks integration (HTTP POST requests with optional token authentication). Once configured, these integrations can forward Insights events to third-party applications and include Insights findings in the organization's operational workflow.

PagerDuty is a software-as-a-service (SaaS) incident response platform offering management of operations. The platform collects signals from different sources and applies analytics and intelligence to take appropriate actions. These include routing requests to correct teams, running automation, or forwarding to third-party applications.

This article covers configuring integration between Insights and PagerDuty. This consists of exposing an endpoint on PagerDuty that accepts JSON-formatted events from Insights. We also provide a Custom Event Transformer (CET) for parsing incoming events and triggering PagerDuty alerts and incidents.

PagerDuty's Services and Integrations functionality

PagerDuty offers an Event Integration feature and associated events API to handle incoming events from third-party applications. Received data can be turned into alerts and actionable incidents in PagerDuty. This feature is commonly used to integrate infrastructure monitoring tools (e.g., Nagios, SignalFX, Datadog), application performance monitoring tools (e.g., New Relic, AppDynamics) and external site checks (e.g., Pingdom, Wormly). In the rest of this article, we use this feature to handle incoming events received from Insights.

Configuring an event integration in PagerDuty is straightforward and documented. It consists of creating a new service and adding an events integration. In this example, select and add a Custom Event Transformer (CET) integration. This allows for inspecting incoming events and extracting relevant data (using an editable JavaScript script) for the alert and incident creation. Details and documentation about this functionality can be found on the PagerDuty Developer Platform.

The new CET integration provides an endpoint URL that can be used by third-party applications to post events. The URL respects the following format, where <integration key> is a unique identifier for your integration:

https://events.pagerduty.com/integration/<integration key>/enqueue

This endpoint URL must be conserved preciously as you'll use it in Insights when configuring Integrations & Notifications (see following section).

The following is an example of the JavaScript script for the CET. The source code is available on GitHub on the redhat-insights-pagerduty repository. Its basic functionality consists of extracting common fields from the JSON formatted Insights events and looping through the data to deduplicate events received from Insights.

var event = JSON.parse(PD.inputRequest.rawBody);
var contexts = [{
  "bundle": event.bundle,
  "application": event.application,
  "event_type": event.event_type,
  "org_id": event.org_id,
  "hostname": event.context.hostname,
  "inventory_id": event.context.inventory_id
}];

// Filtering incoming events (optional)
if (event.bundle != "rhel") {
  PD.fail("Event ignored: (" + event.bundle + ") " + event.application + "/" + event.event_type)
}

var application = event.source && event.source.application && event.source.application.display_name ? event.source.application.display_name : event.application;
var event_type = event.source && event.source.event_type && event.source.event_type.display_name ? event.source.event_type.display_name : event.event_type;
var hostname = event.context && event.context.display_name ? event.context.display_name : event.context.hostname;
var client_url = event.context && event.context.host_url ? event.context.host_url : "https://console.redhat.com";

// Trigger one alert per recommendation (PagerDuty limits to 40)

var normalized_event = [];

for (var i=1; i<event.events.length+1; i++) {
    var evt = {
      event_type: PD.Trigger,
      description: application + "/" + event_type + " on " + hostname + " (" + i + "/" + event.events.length + ")",
      details: event.events[i-1],
      client: "Red Hat Insights",
      client_url: client_url,
      contexts: contexts
    };
    normalized_event.push(evt);
}

PD.emitGenericEvents(normalized_event);

With this Custom Event Transformer in place, PagerDuty can receive and handle events from Insights and forward them for alerts and incident creation. Please note that the supplied integration is an open source contribution and is not supported by Red Hat. The script can be modified according to your organization's needs and/or for handling incoming events data differently. 

Lastly, we recommend using the intelligent grouping functionality from PagerDuty set with a window of five minutes to automatically group all related alerts in one incident. This is handy for dealing with different events generated from the same RHEL system as part of one PagerDuty incident.

Configuring Red Hat Insights for forwarding events

Insights forwards HTTP POST requests to a receiver endpoint every time an application triggers an event. The request contains a JSON payload body that includes all relevant Insights data. The data varies depending on the source application that triggers the event.

For the integration with PagerDuty, use Insights' webhook integration. Add a new integration by navigating to Settings > Integrations. The integration type must be Webhook and the endpoint URL corresponds to the PagerDuty endpoint URL previously generated in the Custom Event Transformer configuration. Finally, enable SSL verification for security reasons.

Once the integration is configured, one must ensure that Insights forwards events. Under Settings > Notifications, make sure at least one behavior group includes the newly configured PagerDuty integration as an action and that it is associated with at least one event (e.g., nNew recommendations triggered by Advisor application).

Note that PagerDuty integration can be used with other actions like email notifications and/or integrations to third-party applications such as Slack, Microsoft Teams, Google Chat, Splunk, ServiceNow and Event-Driven Ansible. 

More details on Integrations & Notifications configuration can be found in the Hybrid Cloud Console product documentation.

Validating the integration end-to-end

It is time to test the integration between Red Hat Insights and PagerDuty end-to-end. You can monitor the integration status from the Settings > Integrations page in Insights and verify that events are successfully triggered from the Event Log.

Assuming Insights events are successfully forwarded via the integration, you can monitor new alerts triggered in PagerDuty for each incoming event under Incidents > Alerts.

As you can see, our intelligent grouping functionality has associated all related alerts from the same system in one incident. You can view it under Incidents > All incidents.

Each alert can be expanded and contains the data collected from the Insights event according to the Custom Event Transformer's JavaScript.

Further, a link to Insights is provided for additional details, troubleshooting and resolution.

Wrap up

This article demonstrates the configuration of PagerDuty and Insights integration through webhooks. The outcome is the automatic generation of alerts in PagerDuty and a fully detailed incident report, making it an excellent foundation for advanced filtering and handling.