mtan93/cachet-url-monitor
1
0
Fork 0
mirror of https://github.com/mtan93/cachet-url-monitor.git synced 2026-03-08 05:31:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into dont_push_status

Browse Source
This commit is contained in:
mtakaki
2018-03-20 00:20:33 -07:00
committed by GitHub
parent 5e5e74938b 79bb5c5c3a
commit 1f1bab6398
17 changed files with 158 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored
View File

@@ -11,3 +11,5 @@ share/
MANIFEST MANIFEST
dist/ dist/
.idea .idea
.pytest_cache/
pip-selfcheck.json

9
Dockerfile
View File

@@ -1,6 +1,13 @@
FROM python:2-onbuild FROM python:2.7-alpine
MAINTAINER Mitsuo Takaki <mitsuotakaki@gmail.com> MAINTAINER Mitsuo Takaki <mitsuotakaki@gmail.com>
WORKDIR /usr/src/app
COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r requirements.txt
COPY cachet_url_monitor/* /usr/src/app/cachet_url_monitor/
COPY config.yml /usr/src/app/config/ COPY config.yml /usr/src/app/config/
VOLUME /usr/src/app/config/ VOLUME /usr/src/app/config/

8
README.md
View File

@@ -23,11 +23,12 @@ endpoint:
timeout: 1 # seconds timeout: 1 # seconds
expectation: expectation:
- type: HTTP_STATUS - type: HTTP_STATUS
status: 200 status_range: 200-300
- type: LATENCY - type: LATENCY
threshold: 1 threshold: 1
- type: REGEX - type: REGEX
regex: ".*<body>.*" regex: ".*<body>.*"
allowed_fails: 0
cachet: cachet:
api_url: http://status.cachethq.io/api/v1 api_url: http://status.cachethq.io/api/v1
token: my_token token: my_token
@@ -38,6 +39,7 @@ cachet:
- UPDATE_STATUS - UPDATE_STATUS
public_incidents: true public_incidents: true
frequency: 30 frequency: 30
latency_unit: ms
``` ```
- **endpoint**, the configuration about the URL that will be monitored. - **endpoint**, the configuration about the URL that will be monitored.
@@ -45,9 +47,10 @@ frequency: 30
- **method**, the HTTP method that will be used by the monitor. - **method**, the HTTP method that will be used by the monitor.
- **timeout**, how long we'll wait to consider the request failed. The unit of it is seconds. - **timeout**, how long we'll wait to consider the request failed. The unit of it is seconds.
- **expectation**, the list of expectations set for the URL. - **expectation**, the list of expectations set for the URL.
- **HTTP_STATUS**, we will verify if the response status code matches what we expect. - **HTTP_STATUS**, we will verify if the response status code falls into the expected range. Please keep in mind the range is inclusive on the first number and exclusive on the second number. If just one value is specified, it will default to only the given value, for example `200` will be converted to `200-201`.
- **LATENCY**, we measure how long the request took to get a response and fail if it's above the threshold. The unit is in seconds. - **LATENCY**, we measure how long the request took to get a response and fail if it's above the threshold. The unit is in seconds.
- **REGEX**, we verify if the response body matches the given regex. - **REGEX**, we verify if the response body matches the given regex.
- **allowed_fails**, create incident/update component status only after specified amount of failed connection trials.
- **cachet**, this is the settings for our cachet server. - **cachet**, this is the settings for our cachet server.
- **api_url**, the cachet API endpoint. - **api_url**, the cachet API endpoint.
- **token**, the API token. - **token**, the API token.
@@ -58,6 +61,7 @@ frequency: 30
- **UPDATE_STATUS**, updates the component status - **UPDATE_STATUS**, updates the component status
- **public_incidents**, boolean to decide if created incidents should be visible to everyone or only to logged in users. Important only if `CREATE_INCIDENT` or `UPDATE_STATUS` are set. - **public_incidents**, boolean to decide if created incidents should be visible to everyone or only to logged in users. Important only if `CREATE_INCIDENT` or `UPDATE_STATUS` are set.
- **frequency**, how often we'll send a request to the given URL. The unit is in seconds. - **frequency**, how often we'll send a request to the given URL. The unit is in seconds.
- **latency_unit**, the latency unit used when reporting the metrics. It will automatically convert to the specified unit. It's not mandatory and it will default to **seconds**. Available units: `ms`, `s`, `m`, `h`.
## Setting up ## Setting up

75
cachet_url_monitor/configuration.py
View File

@@ -10,6 +10,7 @@ import requests
from yaml import dump from yaml import dump
from yaml import load from yaml import load
import latency_unit
import status as st import status as st
# This is the mandatory fields that must be in the configuration file in this # This is the mandatory fields that must be in the configuration file in this
@@ -40,6 +41,16 @@ class ComponentNonexistentError(Exception):
return repr('Component with id [%d] does not exist.' % (self.component_id,)) return repr('Component with id [%d] does not exist.' % (self.component_id,))
class MetricNonexistentError(Exception):
"""Exception raised when the component does not exist."""
def __init__(self, metric_id):
self.metric_id = metric_id
def __str__(self):
return repr('Metric with id [%d] does not exist.' % (self.metric_id,))
def get_current_status(endpoint_url, component_id, headers): def get_current_status(endpoint_url, component_id, headers):
"""Retrieves the current status of the component that is being monitored. It will fail if the component does """Retrieves the current status of the component that is being monitored. It will fail if the component does
not exist or doesn't respond with the expected data. not exist or doesn't respond with the expected data.
@@ -70,6 +81,8 @@ class Configuration(object):
self.logger = logging.getLogger('cachet_url_monitor.configuration.Configuration') self.logger = logging.getLogger('cachet_url_monitor.configuration.Configuration')
self.config_file = config_file self.config_file = config_file
self.data = load(file(self.config_file, 'r')) self.data = load(file(self.config_file, 'r'))
self.current_fails = 0
self.trigger_update = True
# Exposing the configuration to confirm it's parsed as expected. # Exposing the configuration to confirm it's parsed as expected.
self.print_out() self.print_out()
@@ -84,28 +97,39 @@ class Configuration(object):
self.endpoint_url = os.environ.get('ENDPOINT_URL') or self.data['endpoint']['url'] self.endpoint_url = os.environ.get('ENDPOINT_URL') or self.data['endpoint']['url']
self.endpoint_url = normalize_url(self.endpoint_url) self.endpoint_url = normalize_url(self.endpoint_url)
self.endpoint_timeout = os.environ.get('ENDPOINT_TIMEOUT') or self.data['endpoint'].get('timeout') or 1 self.endpoint_timeout = os.environ.get('ENDPOINT_TIMEOUT') or self.data['endpoint'].get('timeout') or 1
self.allowed_fails = os.environ.get('ALLOWED_FAILS') or self.data['endpoint'].get('allowed_fails') or 0
self.api_url = os.environ.get('CACHET_API_URL') or self.data['cachet']['api_url'] self.api_url = os.environ.get('CACHET_API_URL') or self.data['cachet']['api_url']
self.component_id = os.environ.get('CACHET_COMPONENT_ID') or self.data['cachet']['component_id'] self.component_id = os.environ.get('CACHET_COMPONENT_ID') or self.data['cachet']['component_id']
self.metric_id = os.environ.get('CACHET_METRIC_ID') or self.data['cachet'].get('metric_id') self.metric_id = os.environ.get('CACHET_METRIC_ID') or self.data['cachet'].get('metric_id')
self.default_metric_value = self.get_default_metric_value()
if self.metric_id is not None:
self.default_metric_value = self.get_default_metric_value(self.metric_id)
# The latency_unit configuration is not mandatory and we fallback to seconds, by default.
self.latency_unit = os.environ.get('LATENCY_UNIT') or self.data['cachet'].get('latency_unit') or 's'
# We need the current status so we monitor the status changes. This is necessary for creating incidents. # We need the current status so we monitor the status changes. This is necessary for creating incidents.
self.status = get_current_status(self.api_url, self.component_id, self.headers) self.status = get_current_status(self.api_url, self.component_id, self.headers)
self.previous_status = self.status self.previous_status = self.status
# Get remaining settings # Get remaining settings
self.public_incidents = int(os.environ.get('CACHET_PUBLIC_INCIDENTS') or self.data['cachet']['public_incidents']) self.public_incidents = int(
os.environ.get('CACHET_PUBLIC_INCIDENTS') or self.data['cachet']['public_incidents'])
self.logger.info('Monitoring URL: %s %s' % (self.endpoint_method, self.endpoint_url)) self.logger.info('Monitoring URL: %s %s' % (self.endpoint_method, self.endpoint_url))
self.expectations = [Expectaction.create(expectation) for expectation in self.data['endpoint']['expectation']] self.expectations = [Expectaction.create(expectation) for expectation in self.data['endpoint']['expectation']]
for expectation in self.expectations: for expectation in self.expectations:
self.logger.info('Registered expectation: %s' % (expectation,)) self.logger.info('Registered expectation: %s' % (expectation,))
def get_default_metric_value(self): def get_default_metric_value(self, metric_id):
"""Returns default value for configured metric.""" """Returns default value for configured metric."""
get_metric_request = requests.get('%s/metrics/%s' % (self.api_url, self.metric_id), headers=self.headers) get_metric_request = requests.get('%s/metrics/%s' % (self.api_url, metric_id), headers=self.headers)
if get_metric_request.ok:
return get_metric_request.json()['data']['default_value'] return get_metric_request.json()['data']['default_value']
else:
raise MetricNonexistentError(metric_id)
def get_action(self): def get_action(self):
"""Retrieves the action list from the configuration. If it's empty, returns an empty list. """Retrieves the action list from the configuration. If it's empty, returns an empty list.
@@ -187,6 +211,21 @@ class Configuration(object):
del temporary_data['cachet']['token'] del temporary_data['cachet']['token']
return dump(temporary_data, default_flow_style=False) return dump(temporary_data, default_flow_style=False)
def if_trigger_update(self):
"""
Checks if update should be triggered - trigger it for all operational states
and only for non-operational ones above the configured threshold (allowed_fails).
"""
if self.status != 1:
self.current_fails = self.current_fails + 1
self.logger.info('Failure #%s with threshold set to %s' % (self.current_fails, self.allowed_fails))
if self.current_fails <= self.allowed_fails:
self.trigger_update = False
return
self.current_fails = 0
self.trigger_update = True
def push_status(self): def push_status(self):
"""Pushes the status of the component to the cachet server. It will update the component """Pushes the status of the component to the cachet server. It will update the component
status based on the previous call to evaluate(). status based on the previous call to evaluate().
@@ -195,6 +234,9 @@ class Configuration(object):
return return
self.previous_status = self.status self.previous_status = self.status
if not self.trigger_update:
return
params = {'id': self.component_id, 'status': self.status} params = {'id': self.component_id, 'status': self.status}
component_request = requests.put('%s/components/%d' % (self.api_url, self.component_id), params=params, component_request = requests.put('%s/components/%d' % (self.api_url, self.component_id), params=params,
headers=self.headers) headers=self.headers)
@@ -212,7 +254,9 @@ class Configuration(object):
In case of failed connection trial pushes the default metric value. In case of failed connection trial pushes the default metric value.
""" """
if 'metric_id' in self.data['cachet'] and hasattr(self, 'request'): if 'metric_id' in self.data['cachet'] and hasattr(self, 'request'):
value = self.default_metric_value if self.status != 1 else self.request.elapsed.total_seconds() # We convert the elapsed time from the request, in seconds, to the configured unit.
value = self.default_metric_value if self.status != 1 else latency_unit.convert_to_unit(self.latency_unit,
self.request.elapsed.total_seconds())
params = {'id': self.metric_id, 'value': value, params = {'id': self.metric_id, 'value': value,
'timestamp': self.current_timestamp} 'timestamp': self.current_timestamp}
metrics_request = requests.post('%s/metrics/%d/points' % (self.api_url, self.metric_id), params=params, metrics_request = requests.post('%s/metrics/%d/points' % (self.api_url, self.metric_id), params=params,
@@ -229,9 +273,12 @@ class Configuration(object):
"""If the component status has changed, we create a new incident (if this is the first time it becomes unstable) """If the component status has changed, we create a new incident (if this is the first time it becomes unstable)
or updates the existing incident once it becomes healthy again. or updates the existing incident once it becomes healthy again.
""" """
if not self.trigger_update:
return
if hasattr(self, 'incident_id') and self.status == st.COMPONENT_STATUS_OPERATIONAL: if hasattr(self, 'incident_id') and self.status == st.COMPONENT_STATUS_OPERATIONAL:
# If the incident already exists, it means it was unhealthy but now it's healthy again. # If the incident already exists, it means it was unhealthy but now it's healthy again.
params = {'status': 4, 'visible': self.public_incidents, 'component_id': self.component_id, 'component_status': self.status, params = {'status': 4, 'visible': self.public_incidents, 'component_id': self.component_id,
'component_status': self.status,
'notify': True} 'notify': True}
incident_request = requests.put('%s/incidents/%d' % (self.api_url, self.incident_id), params=params, incident_request = requests.put('%s/incidents/%d' % (self.api_url, self.incident_id), params=params,
@@ -292,10 +339,20 @@ class Expectaction(object):
class HttpStatus(Expectaction): class HttpStatus(Expectaction):
def __init__(self, configuration): def __init__(self, configuration):
self.status = configuration['status'] self.status_range = HttpStatus.parse_range(configuration['status_range'])
@staticmethod
def parse_range(range_string):
statuses = range_string.split("-")
if len(statuses) == 1:
# When there was no range given, we should treat the first number as a single status check.
return (int(statuses[0]), int(statuses[0]) + 1)
else:
# We shouldn't look into more than one value, as this is a range value.
return (int(statuses[0]), int(statuses[1]))
def get_status(self, response): def get_status(self, response):
if response.status_code == self.status: if response.status_code >= self.status_range[0] and response.status_code < self.status_range[1]:
return st.COMPONENT_STATUS_OPERATIONAL return st.COMPONENT_STATUS_OPERATIONAL
else: else:
return st.COMPONENT_STATUS_PARTIAL_OUTAGE return st.COMPONENT_STATUS_PARTIAL_OUTAGE
@@ -304,7 +361,7 @@ class HttpStatus(Expectaction):
return 'Unexpected HTTP status (%s)' % (response.status_code,) return 'Unexpected HTTP status (%s)' % (response.status_code,)
def __str__(self): def __str__(self):
return repr('HTTP status: %s' % (self.status,)) return repr('HTTP status range: %s' % (self.status_range,))
class Latency(Expectaction): class Latency(Expectaction):

16
cachet_url_monitor/latency_unit.py Normal file
View File

@@ -0,0 +1,16 @@
#!/usr/bin/env python
seconds_per_unit = {"ms": 1000, "milliseconds": 1000, "s": 1, "seconds": 1, "m": float(1) / 60,
"minutes": float(1) / 60, "h": float(1) / 3600, "hours": float(1) / 3600}
def convert_to_unit(time_unit, value):
"""
Will convert the given value from seconds to the given time_unit.
:param time_unit: The time unit to which the value will be converted to, from seconds.
This is a string parameter. The unit must be in the short form.
:param value: The given value that will be converted. This value must be in seconds.
:return: The converted value.
"""
return value * seconds_per_unit[time_unit]

1
cachet_url_monitor/scheduler.py
View File

@@ -25,6 +25,7 @@ class Agent(object):
""" """
self.configuration.evaluate() self.configuration.evaluate()
self.configuration.push_metrics() self.configuration.push_metrics()
self.configuration.if_trigger_update()
for decorator in self.decorators: for decorator in self.decorators:
decorator.execute(self.configuration) decorator.execute(self.configuration)

5
cachet_url_monitor/status.py
View File

@@ -1,4 +1,9 @@
#!/usr/bin/env python #!/usr/bin/env python
"""
This file defines all the different status different values.
These are all constants and are coupled to cachet's API configuration.
"""
COMPONENT_STATUS_OPERATIONAL = 1 COMPONENT_STATUS_OPERATIONAL = 1
COMPONENT_STATUS_PERFORMANCE_ISSUES = 2 COMPONENT_STATUS_PERFORMANCE_ISSUES = 2

4
config.yml
View File

@@ -4,11 +4,12 @@ endpoint:
timeout: 0.01 timeout: 0.01
expectation: expectation:
- type: HTTP_STATUS - type: HTTP_STATUS
status: 200 status_range: 200-300
- type: LATENCY - type: LATENCY
threshold: 1 threshold: 1
- type: REGEX - type: REGEX
regex: '.*(<body).*' regex: '.*(<body).*'
allowed_fails: 0
cachet: cachet:
api_url: https://demo.cachethq.io/api/v1 api_url: https://demo.cachethq.io/api/v1
token: my_token token: my_token
@@ -19,3 +20,4 @@ cachet:
- UPDATE_STATUS - UPDATE_STATUS
public_incidents: true public_incidents: true
frequency: 30 frequency: 30
latency_unit: ms

10
dev_requirements.txt
View File

@@ -1,9 +1,9 @@
PyYAML==3.11 PyYAML==3.12
codacy-coverage==1.2.18 codacy-coverage==1.2.18
ipython==4.2.0 ipython==4.2.0
mock==2.0.0 mock==2.0.0
pudb==2016.1 pudb==2016.1
pytest==2.9.1 pytest==3.4.2
pytest-cov==2.2.1 pytest-cov==2.5.1
requests==2.9.1 requests==2.18.4
schedule==0.3.2 schedule==0.5.0

6
docker-compose.yml
View File

@@ -1,6 +0,0 @@
version: '2'
services:
monitor:
build: .
volumes:
- .:/usr/src/app/

6
requirements.txt
View File

@@ -1,3 +1,3 @@
PyYAML==3.11 PyYAML==3.12
requests==2.9.1 requests==2.18.4
schedule==0.3.2 schedule==0.5.0

2
setup.cfg Normal file
View File

@@ -0,0 +1,2 @@
[metadata]
description-file = README.md

2
setup.py
View File

@@ -2,7 +2,7 @@
from distutils.core import setup from distutils.core import setup
setup(name='cachet-url-monitor', setup(name='cachet-url-monitor',
version='0.3', version='0.4',
description='Cachet URL monitor plugin', description='Cachet URL monitor plugin',
author='Mitsuo Takaki', author='Mitsuo Takaki',
author_email='mitsuotakaki@gmail.com', author_email='mitsuotakaki@gmail.com',

0
tests/__init__.py Normal file
View File

7
tests/test_configuration.py
View File

@@ -2,10 +2,11 @@
import sys import sys
import unittest import unittest
import cachet_url_monitor.status
import mock import mock
from requests import ConnectionError, HTTPError, Timeout from requests import ConnectionError, HTTPError, Timeout
import cachet_url_monitor.status
sys.modules['requests'] = mock.Mock() sys.modules['requests'] = mock.Mock()
sys.modules['logging'] = mock.Mock() sys.modules['logging'] = mock.Mock()
from cachet_url_monitor.configuration import Configuration from cachet_url_monitor.configuration import Configuration
@@ -24,7 +25,7 @@ class ConfigurationTest(unittest.TestCase):
get_return = mock.Mock() get_return = mock.Mock()
get_return.ok = True get_return.ok = True
get_return.json = mock.Mock() get_return.json = mock.Mock()
get_return.json.return_value = {'data': {'status': 1}} get_return.json.return_value = {'data': {'status': 1, 'default_value': 0.5}}
return get_return return get_return
sys.modules['requests'].get = get sys.modules['requests'].get = get
@@ -38,7 +39,7 @@ class ConfigurationTest(unittest.TestCase):
sys.modules['requests'].HTTPError = HTTPError sys.modules['requests'].HTTPError = HTTPError
def test_init(self): def test_init(self):
self.assertEqual(len(self.configuration.data), 3, 'Configuration data size is incorrect') self.assertEqual(len(self.configuration.data), 4, 'Number of root elements in config.yml is incorrect')
self.assertEquals(len(self.configuration.expectations), 3, 'Number of expectations read from file is incorrect') self.assertEquals(len(self.configuration.expectations), 3, 'Number of expectations read from file is incorrect')
self.assertDictEqual(self.configuration.headers, {'X-Cachet-Token': 'token2'}, 'Header was not set correctly') self.assertDictEqual(self.configuration.headers, {'X-Cachet-Token': 'token2'}, 'Header was not set correctly')
self.assertEquals(self.configuration.api_url, 'https://demo.cachethq.io/api/v1', self.assertEquals(self.configuration.api_url, 'https://demo.cachethq.io/api/v1',

19
tests/test_expectation.py
View File

@@ -1,8 +1,10 @@
#!/usr/bin/env python #!/usr/bin/env python
import re
import unittest import unittest
import mock import mock
import re import pytest
from cachet_url_monitor.configuration import HttpStatus, Regex from cachet_url_monitor.configuration import HttpStatus, Regex
from cachet_url_monitor.configuration import Latency from cachet_url_monitor.configuration import Latency
@@ -51,10 +53,21 @@ class LatencyTest(unittest.TestCase):
class HttpStatusTest(unittest.TestCase): class HttpStatusTest(unittest.TestCase):
def setUp(self): def setUp(self):
self.expectation = HttpStatus({'type': 'HTTP_STATUS', 'status': 200}) self.expectation = HttpStatus({'type': 'HTTP_STATUS', 'status_range': "200-300"})
def test_init(self): def test_init(self):
assert self.expectation.status == 200 assert self.expectation.status_range == (200, 300)
def test_init_with_one_status(self):
"""With only one value, we still expect a valid tuple"""
self.expectation = HttpStatus({'type': 'HTTP_STATUS', 'status_range': "200"})
assert self.expectation.status_range == (200, 201)
def test_init_with_invalid_number(self):
"""Invalid values should just fail with a ValueError, as we can't convert it to int."""
with pytest.raises(ValueError):
self.expectation = HttpStatus({'type': 'HTTP_STATUS', 'status_range': "foo"})
def test_get_status_healthy(self): def test_get_status_healthy(self):
request = mock.Mock() request = mock.Mock()

19
tests/test_latency_unit.py Normal file
View File

@@ -0,0 +1,19 @@
#!/usr/bin/env python
from cachet_url_monitor.latency_unit import convert_to_unit
def test_convert_to_unit_ms():
assert convert_to_unit("ms", 1) == 1000
def test_convert_to_unit_s():
assert convert_to_unit("s", 20) == 20
def test_convert_to_unit_m():
assert convert_to_unit("m", 3) == float(3) / 60
def test_convert_to_unit_h():
assert convert_to_unit("h", 7200) == 2