Listen to this post
Learning SignalFx can be overwhelming. There are a multitude of ways to integrate it with your applications. Choice overload can quickly set in, to combat that I like to narrow in on a fundamental problem first. I would like to monitor a PostgreSQL database. The following content will walk through my journey to do so.
- It’s an operational intelligence tool designed to observe your systems.
- Costs $15.00 per host per month or $180 per year with host-based pricing
- Up to 10 containers per host are monitored with a host-based plan.
- There are many alternative operational intelligence tools on the market.
- It has a host of integrations with cloud technologies and traditional servers.
- SignalFx provides a free trial
Metrics and Metadata
The SignalFx agent collects metrics and sends them to their servers. The metrics are made of the following metric types.
Counters are a straightforward metric that only takes integer values. They can count things like the number of errors that have occurred.
Cumulative counters are scoped to the lifetime of a process or an application. The number of database calls since PostgreSQL started would be a cumulative counter.
Gauges measure values over time. The percentage of memory a PostgreSQL server is using is an example of a gauge. The database memory fluctuates, and the gauge metric captures that fluctuation over time.
Use metadata to filter, find, and aggregate the metrics you want to chart or alert on. An example would be an environment key-value pair.
The environment metadata will allow you to search by the environment within SignalFx.
Charts provide visualization of the metrics you send to SignalFx. A chart that shows the percentage of memory helps you decide when it’s time to upgrade said memory.
Dashboards allow you to group charts so you can get the big picture. We can group them within dashboard groups. We can’t learn SignalFx without talking about dashboards.
- Built-In – Used to provide default dashboards for integrations like PostgreSQL
- Custom – Any custom dashboards you create
- User – Primarily used for isolated experimentation and visualization
Detectors and Alerts
Detectors consist of events, alerts, and notifications. They can trigger alerts and notifications based on conditions. It’s possible to chain detectors by triggering additional events and notifications after a detector fires an event.
I’ve chosen to instrument a Postgresql server and the container it lives in. These are technologies I’m comfortable with and can get up and running quickly. Using familiar technologies will allow me to learn SignalFx more effectively.
I configured the agent.yml with these options from the Postgres monitor documentation.
Grab your Organization’s access token. This can be found from this location.
Profile -> Organization Settings -> Access Tokens
Every host has its dashboard by default. Now that the agent is running, we can open Signal FX and then choose the INFRASTRUCTURE menu item.
We’ll find our host running after data makes its way to the SFX API.
In addition to the host’s dashboard, there is a PostgreSQL dashboard provided by SignalFx. The dashboard offers a view of database level resources out of the box.
Poking with Sticks
I connected to the container in another terminal with this command.
docker exec -it learning-signalfx /bin/bash
I ran these commands to do a read-only benchmark test.
pgbench -i learnsignalfx pgbench -c 4 -j 2 -T 600 -S learnsignalfx
The event predictably created a spike in CPU, as evidenced by the default APM dashboard.
I created a detector for CPU utilization. I then reran the benchmark, and sure enough, an alert fired.
I then stopped the benchmark. Shortly after that, I received another email letting me know everything was back to normal.
There are other fun benchmark tests you can run with the pgbench command. Enjoy trying interesting detectors, charts, and dashboards on your own.
Learn SignalFx – Beyond the Basics
It appears that books are scarce for SignalFx. These resources will help with learning more about SignalFx.
- Signal FX – Key Documentation
- Study Material