SignalFx and EC2

SignalFx

SignalFx and EC2 are a match made in heaven. If you need to ensure the uptime of your EC2 instances monitoring tools like SignalFx is a must-have.

Follow along for details that will help you use SignalFx to monitor an existing EC2 instance.

If you have different EC2 & SignalFx integration requirements see this documentation.

Prerequisites

  1. An existing EC2 Instance, Amazon provides a free tier
  2. A SignalFx account, 14-day trials are available on signalfx.com

Setup

Login to your SignalFx account then click the INTEGRATIONS menu item.

Press the SFx SMARTAGENT icon then the SETUP tab.

Follow the installation instructions then navigate to the INFRASTRUCTURE tab

Clicking the Host link will take you to a view of your EC2 instance’s resource utilization.

That’s all there is to it; now you can fire alerts when bad things happen on your EC2 instance.

Related Content




Learn Signal Analog

The Basics

Learn Signal Analog by doing. Signal Analog is a tool for generating SignalFx dashboards. It’s a Nike open source project on GitHub. My team is migrating from New Relic to SignalFx, and we have opted to use Signal Analog to create our SignalFx resources.

Installation

I ran these commands to add the signal_analog package to my project.

mkdir learn-signal-analog
cd learn-signal-analog
printf 'signal_analog' > requirements.txt
pip3 install -r requirements.txt

Creating a Dashboard

Let the fun begin. I decided to start by creating a grouped dashboard with this example. The dashboards.py file was added to my learning-signal-analog directory. I then narrowed the code down to represent a single dashboard.

Up went the code

./dashboards.py --api-key <MyApiAccessToken> create

In the UI for SignalFx, the working token is labeled API Access Token.

This is what success looks like.

{
  "authorizedWriters": {
    "teams": [],
    "users": []
  },
  "created": 1590460991067,
  "creator": "EYyXD0gA0AA",
  "dashboardConfigs": [
    {
      "configId": "EY58mxKAwAA",
      "dashboardId": "EY58lP0A0AA",
      "descriptionOverride": null,
      "filtersOverride": null,
      "nameOverride": null
    }
  ],
  "dashboards": [
    "EY58lP0A0AA"
  ],
  "description": "",
  "email": null,
  "id": "EY56poZA4AA",
  "lastUpdated": 1590460992006,
  "lastUpdatedBy": "EYyXD0gA0AA",
  "name": "Learn Signal Analog",
  "teams": []
}

The dashboard is present within the dashboard group upon creation.

I then made the dashboard more useful by removing the Network chart and adding a Postgresql query time chart.

Signal Analog is an excellent Python module that will provide you an infrastructure as code option for SignalFx.

Learn Signal Analog – Beyond The Basics




Learn SignalFx

The Basics

SignalFx

Learning SignalFx can be overwhelming. There are a multitude of ways to integrate it with your applications. Choice overload can quickly set in, to combat that I like to narrow in on a fundamental problem first. I would like to monitor a PostgreSQL database. The following content will walk through my journey to do so.

Facts

Metrics and Metadata

The SignalFx agent collects metrics and sends them to their servers. The metrics are made of the following metric types.

Counters

Counters are a straightforward metric that only takes integer values. They can count things like the number of errors that have occurred.

Cumulative Counters

Cumulative counters are scoped to the lifetime of a process or an application. The number of database calls since PostgreSQL started would be a cumulative counter.

Gauges

Gauges measure values over time. The percentage of memory a PostgreSQL server is using is an example of a gauge. The database memory fluctuates, and the gauge metric captures that fluctuation over time.

Metadata

Use metadata to filter, find, and aggregate the metrics you want to chart or alert on. An example would be an environment key-value pair.

environment:prod

The environment metadata will allow you to search by the environment within SignalFx.

Charts

Charts provide visualization of the metrics you send to SignalFx. A chart that shows the percentage of memory helps you decide when it’s time to upgrade said memory.

Dashboards

Dashboards allow you to group charts so you can get the big picture. We can group them within dashboard groups. We can’t learn SignalFx without talking about dashboards.

  • Built-In – Used to provide default dashboards for integrations like PostgreSQL
  • Custom – Any custom dashboards you create
  • User – Primarily used for isolated experimentation and visualization

Detectors and Alerts

Detectors consist of events, alerts, and notifications. They can trigger alerts and notifications based on conditions. It’s possible to chain detectors by triggering additional events and notifications after a detector fires an event.

The Container

I’ve chosen to instrument a Postgresql server and the container it lives in. These are technologies I’m comfortable with and can get up and running quickly. Using familiar technologies will allow me to learn SignalFx more effectively.

I’ve created a container that installs the Postgresql server and the SignalFx collectd per the advanced installation options.

I configured the agent.yml with these options from the Postgres monitor documentation.

Grab your Organization’s access token. This can be found from this location.

Profile -> Organization Settings -> Access Tokens

I chose to store the SignalFx access token in a file. See the SignalFx remote configuration documentation for configuration options and the agent.yml for more context.

The Dashboards

Every host has its dashboard by default. Now that the agent is running, we can open Signal FX and then choose the INFRASTRUCTURE menu item.

We’ll find our host running after data makes its way to the SFX API.

In addition to the host’s dashboard, there is a PostgreSQL dashboard provided by SignalFx. The dashboard offers a view of database level resources out of the box.

Poking with Sticks

I connected to the container in another terminal with this command.

 docker exec -it learning-signalfx /bin/bash

I ran these commands to do a read-only benchmark test.

pgbench -i learnsignalfx
pgbench -c 4 -j 2 -T 600 -S learnsignalfx

The event predictably created a spike in CPU, as evidenced by the default APM dashboard.

I created a detector for CPU utilization. I then reran the benchmark, and sure enough, an alert fired.

I then stopped the benchmark. Shortly after that, I received another email letting me know everything was back to normal.

There are other fun benchmark tests you can run with the pgbench command. Enjoy trying interesting detectors, charts, and dashboards on your own.

Learn SignalFx – Beyond the Basics

It appears that books are scarce for SignalFx. These resources will help with learning more about SignalFx.