mtan93/cachet-url-monitor
1
0
Fork 0
mirror of https://github.com/mtan93/cachet-url-monitor.git synced 2026-03-08 05:31:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

Multithreading #66 (#76)

Browse Source 
* feat(multihreading): each url has it's own thread

* Fixing broken unit tests

* Improving readability when there are multiple URLs registerd and creating new action to upload metrics

* Improving error message when there's no file found

* Bumping the version

Co-authored-by: Alex Berenshtein <aberenshtein@yotpo.com>
This commit is contained in:
mtakaki
2020-01-05 10:25:06 -08:00
committed by GitHub
parent 9a73063a6f
commit a13a42d51c
7 changed files with 188 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files

61
README.md
View File

@@ -16,7 +16,8 @@ This project is available at PyPI: [https://pypi.python.org/pypi/cachet-url-moni
## Configuration ## Configuration
```yaml ```yaml
endpoint: endpoints:
- name: Google
url: http://www.google.com url: http://www.google.com
method: GET method: GET
header: header:
@@ -24,47 +25,69 @@ endpoint:
timeout: 1 # seconds timeout: 1 # seconds
expectation: expectation:
- type: HTTP_STATUS - type: HTTP_STATUS
status_range: 200-300 status_range: 200-205
incident: MAJOR
- type: LATENCY - type: LATENCY
threshold: 1 threshold: 1
- type: REGEX - type: REGEX
regex: ".*<body>.*" regex: ".*<body>.*"
allowed_fails: 0 allowed_fails: 0
cachet:
api_url: http://status.cachethq.io/api/v1
token: my_token
component_id: 1 component_id: 1
metric_id: 1 metric_id: 1
action: action:
- CREATE_INCIDENT
- UPDATE_STATUS - UPDATE_STATUS
public_incidents: true public_incidents: true
latency_unit: ms latency_unit: ms
frequency: 30 frequency: 5
- name: Amazon
url: http://www.amazon.com
method: GET
header:
SOME-HEADER: SOME-VALUE
timeout: 1 # seconds
expectation:
- type: HTTP_STATUS
status_range: 200-205
incident: MAJOR
- type: LATENCY
threshold: 1
- type: REGEX
regex: ".*<body>.*"
threshold: 10
allowed_fails: 0
component_id: 2
action:
- CREATE_INCIDENT
public_incidents: true
latency_unit: ms
frequency: 5
cachet:
api_url: http://status.cachethq.io/api/v1
token: mytoken
``` ```
- **endpoint**, the configuration about the URL that will be monitored. - **endpoints**, the configuration about the URL/Urls that will be monitored.
- **url**, the URL that is going to be monitored. - **name**, The name of the component. This is now mandatory (since 0.6.0) so we can distinguish the logs for each URL being monitored.
- **method**, the HTTP method that will be used by the monitor. - **url**, the URL that is going to be monitored. *mandatory*
- **method**, the HTTP method that will be used by the monitor. *mandatory*
- **header**, client header passed to the request. Remove if you do not want to pass a header. - **header**, client header passed to the request. Remove if you do not want to pass a header.
- **timeout**, how long we'll wait to consider the request failed. The unit of it is seconds. - **timeout**, how long we'll wait to consider the request failed. The unit of it is seconds. *mandatory*
- **expectation**, the list of expectations set for the URL. - **expectation**, the list of expectations set for the URL. *mandatory*
- **HTTP_STATUS**, we will verify if the response status code falls into the expected range. Please keep in mind the range is inclusive on the first number and exclusive on the second number. If just one value is specified, it will default to only the given value, for example `200` will be converted to `200-201`. - **HTTP_STATUS**, we will verify if the response status code falls into the expected range. Please keep in mind the range is inclusive on the first number and exclusive on the second number. If just one value is specified, it will default to only the given value, for example `200` will be converted to `200-201`.
- **LATENCY**, we measure how long the request took to get a response and fail if it's above the threshold. The unit is in seconds. - **LATENCY**, we measure how long the request took to get a response and fail if it's above the threshold. The unit is in seconds.
- **REGEX**, we verify if the response body matches the given regex. - **REGEX**, we verify if the response body matches the given regex.
- **allowed_fails**, create incident/update component status only after specified amount of failed connection trials. - **allowed_fails**, create incident/update component status only after specified amount of failed connection trials.
- **cachet**, this is the settings for our cachet server. - **component_id**, the id of the component we're monitoring. This will be used to update the status of the component. *mandatory*
- **api_url**, the cachet API endpoint.
- **token**, the API token.
- **component_id**, the id of the component we're monitoring. This will be used to update the status of the component.
- **metric_id**, this will be used to store the latency of the API. If this is not set, it will be ignored. - **metric_id**, this will be used to store the latency of the API. If this is not set, it will be ignored.
- **action**, the action to be done when one of the expectations fails. This is optional and if left blank, nothing will be done to the component. - **action**, the action to be done when one of the expectations fails. This is optional and if left blank, nothing will be done to the component.
- **CREATE_INCIDENT**, we will create an incident when the expectation fails. - **CREATE_INCIDENT**, we will create an incident when the expectation fails.
- **UPDATE_STATUS**, updates the component status - **UPDATE_STATUS**, updates the component status.
- **PUSH_METRICS**, uploads response latency metrics.
- **public_incidents**, boolean to decide if created incidents should be visible to everyone or only to logged in users. Important only if `CREATE_INCIDENT` or `UPDATE_STATUS` are set. - **public_incidents**, boolean to decide if created incidents should be visible to everyone or only to logged in users. Important only if `CREATE_INCIDENT` or `UPDATE_STATUS` are set.
- **latency_unit**, the latency unit used when reporting the metrics. It will automatically convert to the specified unit. It's not mandatory and it will default to **seconds**. Available units: `ms`, `s`, `m`, `h`. - **latency_unit**, the latency unit used when reporting the metrics. It will automatically convert to the specified unit. It's not mandatory and it will default to **seconds**. Available units: `ms`, `s`, `m`, `h`.
- **frequency**, how often we'll send a request to the given URL. The unit is in seconds. - **frequency**, how often we'll send a request to the given URL. The unit is in seconds.
- **cachet**, this is the settings for our cachet server.
- **api_url**, the cachet API endpoint. *mandatory*
- **token**, the API token. *mandatory*
Each `expectation` has their own default incident status. It can be overridden by setting the `incident` property to any of the following values: Each `expectation` has their own default incident status. It can be overridden by setting the `incident` property to any of the following values:
- `PARTIAL` - `PARTIAL`

74
cachet_url_monitor/configuration.py
View File

@@ -8,18 +8,13 @@ import time
import requests import requests
from yaml import dump from yaml import dump
from yaml import load
from yaml import FullLoader
import cachet_url_monitor.latency_unit as latency_unit import cachet_url_monitor.latency_unit as latency_unit
import cachet_url_monitor.status as st import cachet_url_monitor.status as st
# This is the mandatory fields that must be in the configuration file in this # This is the mandatory fields that must be in the configuration file in this
# same exact structure. # same exact structure.
configuration_mandatory_fields = { configuration_mandatory_fields = ['url', 'method', 'timeout', 'expectation', 'component_id', 'frequency']
'endpoint': ['url', 'method', 'timeout', 'expectation'],
'cachet': ['api_url', 'token', 'component_id'],
'frequency': []}
class ConfigurationValidationError(Exception): class ConfigurationValidationError(Exception):
@@ -78,13 +73,19 @@ class Configuration(object):
of assessing the API and pushing the results to cachet. of assessing the API and pushing the results to cachet.
""" """
def __init__(self, config_file): def __init__(self, config_file, endpoint_index):
self.logger = logging.getLogger('cachet_url_monitor.configuration.Configuration') self.endpoint_index = endpoint_index
self.config_file = config_file self.data = config_file
self.data = load(open(self.config_file, 'r'), Loader=FullLoader) self.endpoint = self.data['endpoints'][endpoint_index]
self.current_fails = 0 self.current_fails = 0
self.trigger_update = True self.trigger_update = True
if 'name' not in self.endpoint:
# We have to make this mandatory, otherwise the logs are confusing when there are multiple URLs.
raise ConfigurationValidationError('name')
self.logger = logging.getLogger(f'cachet_url_monitor.configuration.Configuration.{self.endpoint["name"]}')
# Exposing the configuration to confirm it's parsed as expected. # Exposing the configuration to confirm it's parsed as expected.
self.print_out() self.print_out()
@@ -94,33 +95,32 @@ class Configuration(object):
# We store the main information from the configuration file, so we don't keep reading from the data dictionary. # We store the main information from the configuration file, so we don't keep reading from the data dictionary.
self.headers = {'X-Cachet-Token': os.environ.get('CACHET_TOKEN') or self.data['cachet']['token']} self.headers = {'X-Cachet-Token': os.environ.get('CACHET_TOKEN') or self.data['cachet']['token']}
self.endpoint_method = os.environ.get('ENDPOINT_METHOD') or self.data['endpoint']['method'] self.endpoint_method = self.endpoint['method']
self.endpoint_url = os.environ.get('ENDPOINT_URL') or self.data['endpoint']['url'] self.endpoint_url = self.endpoint['url']
self.endpoint_url = normalize_url(self.endpoint_url) self.endpoint_url = normalize_url(self.endpoint_url)
self.endpoint_timeout = os.environ.get('ENDPOINT_TIMEOUT') or self.data['endpoint'].get('timeout') or 1 self.endpoint_timeout = self.endpoint.get('timeout') or 1
self.endpoint_header = self.data['endpoint'].get('header') or None self.endpoint_header = self.endpoint.get('header') or None
self.allowed_fails = os.environ.get('ALLOWED_FAILS') or self.data['endpoint'].get('allowed_fails') or 0 self.allowed_fails = self.endpoint.get('allowed_fails') or 0
self.api_url = os.environ.get('CACHET_API_URL') or self.data['cachet']['api_url'] self.api_url = os.environ.get('CACHET_API_URL') or self.data['cachet']['api_url']
self.component_id = os.environ.get('CACHET_COMPONENT_ID') or self.data['cachet']['component_id'] self.component_id = self.endpoint['component_id']
self.metric_id = os.environ.get('CACHET_METRIC_ID') or self.data['cachet'].get('metric_id') self.metric_id = self.endpoint.get('metric_id')
if self.metric_id is not None: if self.metric_id is not None:
self.default_metric_value = self.get_default_metric_value(self.metric_id) self.default_metric_value = self.get_default_metric_value(self.metric_id)
# The latency_unit configuration is not mandatory and we fallback to seconds, by default. # The latency_unit configuration is not mandatory and we fallback to seconds, by default.
self.latency_unit = os.environ.get('LATENCY_UNIT') or self.data['cachet'].get('latency_unit') or 's' self.latency_unit = self.data['cachet'].get('latency_unit') or 's'
# We need the current status so we monitor the status changes. This is necessary for creating incidents. # We need the current status so we monitor the status changes. This is necessary for creating incidents.
self.status = get_current_status(self.api_url, self.component_id, self.headers) self.status = get_current_status(self.api_url, self.component_id, self.headers)
self.previous_status = self.status self.previous_status = self.status
# Get remaining settings # Get remaining settings
self.public_incidents = int( self.public_incidents = int(self.endpoint['public_incidents'])
os.environ.get('CACHET_PUBLIC_INCIDENTS') or self.data['cachet']['public_incidents'])
self.logger.info('Monitoring URL: %s %s' % (self.endpoint_method, self.endpoint_url)) self.logger.info('Monitoring URL: %s %s' % (self.endpoint_method, self.endpoint_url))
self.expectations = [Expectation.create(expectation) for expectation in self.data['endpoint']['expectation']] self.expectations = [Expectation.create(expectation) for expectation in self.endpoint['expectation']]
for expectation in self.expectations: for expectation in self.expectations:
self.logger.info('Registered expectation: %s' % (expectation,)) self.logger.info('Registered expectation: %s' % (expectation,))
@@ -137,10 +137,10 @@ class Configuration(object):
"""Retrieves the action list from the configuration. If it's empty, returns an empty list. """Retrieves the action list from the configuration. If it's empty, returns an empty list.
:return: The list of actions, which can be an empty list. :return: The list of actions, which can be an empty list.
""" """
if self.data['cachet'].get('action') is None: if self.endpoint.get('action') is None:
return [] return []
else: else:
return self.data['cachet']['action'] return self.endpoint['action']
def validate(self): def validate(self):
"""Validates the configuration by verifying the mandatory fields are """Validates the configuration by verifying the mandatory fields are
@@ -148,24 +148,20 @@ class Configuration(object):
ConfigurationValidationError is raised. Otherwise nothing will happen. ConfigurationValidationError is raised. Otherwise nothing will happen.
""" """
configuration_errors = [] configuration_errors = []
for key, sub_entries in configuration_mandatory_fields.items(): for key in configuration_mandatory_fields:
if key not in self.data: if key not in self.endpoint:
configuration_errors.append(key) configuration_errors.append(key)
for sub_key in sub_entries: if 'expectation' in self.endpoint:
if sub_key not in self.data[key]: if (not isinstance(self.endpoint['expectation'], list) or
configuration_errors.append('%s.%s' % (key, sub_key)) (isinstance(self.endpoint['expectation'], list) and
len(self.endpoint['expectation']) == 0)):
if ('endpoint' in self.data and 'expectation' in
self.data['endpoint']):
if (not isinstance(self.data['endpoint']['expectation'], list) or
(isinstance(self.data['endpoint']['expectation'], list) and
len(self.data['endpoint']['expectation']) == 0)):
configuration_errors.append('endpoint.expectation') configuration_errors.append('endpoint.expectation')
if len(configuration_errors) > 0: if len(configuration_errors) > 0:
raise ConfigurationValidationError( raise ConfigurationValidationError(
f"Config file [{self.config_file}] failed validation. Missing keys: {', '.join(configuration_errors)}") 'Endpoint [%s] failed validation. Missing keys: %s' % (self.endpoint,
', '.join(configuration_errors)))
def evaluate(self): def evaluate(self):
"""Sends the request to the URL set in the configuration and executes """Sends the request to the URL set in the configuration and executes
@@ -214,6 +210,8 @@ class Configuration(object):
temporary_data = copy.deepcopy(self.data) temporary_data = copy.deepcopy(self.data)
# Removing the token so we don't leak it in the logs. # Removing the token so we don't leak it in the logs.
del temporary_data['cachet']['token'] del temporary_data['cachet']['token']
temporary_data['endpoints'] = temporary_data['endpoints'][self.endpoint_index]
return dump(temporary_data, default_flow_style=False) return dump(temporary_data, default_flow_style=False)
def if_trigger_update(self): def if_trigger_update(self):
@@ -361,6 +359,10 @@ class HttpStatus(Expectation):
@staticmethod @staticmethod
def parse_range(range_string): def parse_range(range_string):
if isinstance(range_string, int):
# This happens when there's no range and no dash character, it will be parsed as int already.
return range_string, range_string + 1
statuses = range_string.split("-") statuses = range_string.split("-")
if len(statuses) == 1: if len(statuses) == 1:
# When there was no range given, we should treat the first number as a single status check. # When there was no range given, we should treat the first number as a single status check.
@@ -382,7 +384,7 @@ class HttpStatus(Expectation):
return f'Unexpected HTTP status ({response.status_code})' return f'Unexpected HTTP status ({response.status_code})'
def __str__(self): def __str__(self):
return repr(f'HTTP status range: {self.status_range}') return repr(f'HTTP status range: [{self.status_range[0]}, {self.status_range[1]}[')
class Latency(Expectation): class Latency(Expectation):

58
cachet_url_monitor/scheduler.py
View File

@@ -1,12 +1,16 @@
#!/usr/bin/env python #!/usr/bin/env python
import logging import logging
import sys import sys
import threading
import time import time
import schedule import schedule
from yaml import load, SafeLoader
from cachet_url_monitor.configuration import Configuration from cachet_url_monitor.configuration import Configuration
cachet_mandatory_fields = ['api_url', 'token']
class Agent(object): class Agent(object):
"""Monitor agent that will be constantly verifying if the URL is healthy """Monitor agent that will be constantly verifying if the URL is healthy
@@ -32,7 +36,7 @@ class Agent(object):
def start(self): def start(self):
"""Sets up the schedule based on the configuration file.""" """Sets up the schedule based on the configuration file."""
schedule.every(self.configuration.data['frequency']).seconds.do(self.execute) schedule.every(self.configuration.endpoint['frequency']).seconds.do(self.execute)
class Decorator(object): class Decorator(object):
@@ -50,10 +54,15 @@ class CreateIncidentDecorator(Decorator):
configuration.push_incident() configuration.push_incident()
class PushMetricsDecorator(Decorator):
def execute(self, configuration):
configuration.push_metrics()
class Scheduler(object): class Scheduler(object):
def __init__(self, config_file): def __init__(self, config_file, endpoint_index):
self.logger = logging.getLogger('cachet_url_monitor.scheduler.Scheduler') self.logger = logging.getLogger('cachet_url_monitor.scheduler.Scheduler')
self.configuration = Configuration(config_file) self.configuration = Configuration(config_file, endpoint_index)
self.agent = self.get_agent() self.agent = self.get_agent()
self.stop = False self.stop = False
@@ -62,10 +71,11 @@ class Scheduler(object):
action_names = { action_names = {
'CREATE_INCIDENT': CreateIncidentDecorator, 'CREATE_INCIDENT': CreateIncidentDecorator,
'UPDATE_STATUS': UpdateStatusDecorator, 'UPDATE_STATUS': UpdateStatusDecorator,
'PUSH_METRICS': PushMetricsDecorator,
} }
actions = [] actions = []
for action in self.configuration.get_action(): for action in self.configuration.get_action():
self.logger.info('Registering action %s' % (action)) self.logger.info(f'Registering action {action}')
actions.append(action_names[action]()) actions.append(action_names[action]())
return Agent(self.configuration, decorators=actions) return Agent(self.configuration, decorators=actions)
@@ -74,7 +84,33 @@ class Scheduler(object):
self.logger.info('Starting monitor agent...') self.logger.info('Starting monitor agent...')
while not self.stop: while not self.stop:
schedule.run_pending() schedule.run_pending()
time.sleep(self.configuration.data['frequency']) time.sleep(self.configuration.endpoint['frequency'])
class NewThread(threading.Thread):
def __init__(self, scheduler):
threading.Thread.__init__(self)
self.scheduler = scheduler
def run(self):
self.scheduler.start()
def validate_config():
if 'endpoints' not in config_file.keys():
fatal_error('Endpoints is a mandatory field')
if config_file['endpoints'] is None:
fatal_error('Endpoints array can not be empty')
for key in cachet_mandatory_fields:
if key not in config_file['cachet']:
fatal_error('Missing cachet mandatory fields')
def fatal_error(message):
logging.getLogger('cachet_url_monitor.scheduler').fatal("%s", message)
sys.exit(1)
if __name__ == "__main__": if __name__ == "__main__":
@@ -87,5 +123,13 @@ if __name__ == "__main__":
logging.getLogger('cachet_url_monitor.scheduler').fatal('Missing configuration file argument') logging.getLogger('cachet_url_monitor.scheduler').fatal('Missing configuration file argument')
sys.exit(1) sys.exit(1)
scheduler = Scheduler(sys.argv[1]) try:
scheduler.start() config_file = load(open(sys.argv[1], 'r'), SafeLoader)
except FileNotFoundError:
logging.getLogger('cachet_url_monitor.scheduler').fatal(f'File not found: {sys.argv[1]}')
sys.exit(1)
validate_config()
for endpoint_index in range(len(config_file['endpoints'])):
NewThread(Scheduler(config_file, endpoint_index)).start()

13
config.yml
View File

@@ -1,4 +1,5 @@
endpoint: endpoints:
- name: swagger
url: http://localhost:8080/swagger url: http://localhost:8080/swagger
method: GET method: GET
header: header:
@@ -13,14 +14,14 @@ endpoint:
- type: REGEX - type: REGEX
regex: '.*(<body).*' regex: '.*(<body).*'
allowed_fails: 0 allowed_fails: 0
cachet: frequency: 30
api_url: https://demo.cachethq.io/api/v1
token: my_token
component_id: 1 component_id: 1
#metric_id: 1 metric_id: 1
action: action:
- CREATE_INCIDENT - CREATE_INCIDENT
- UPDATE_STATUS - UPDATE_STATUS
public_incidents: true public_incidents: true
latency_unit: ms latency_unit: ms
frequency: 30 cachet:
api_url: https://demo.cachethq.io/api/v1
token: my_token

2
setup.py
View File

@@ -3,7 +3,7 @@
from setuptools import setup from setuptools import setup
setup(name='cachet-url-monitor', setup(name='cachet-url-monitor',
version='0.5', version='0.6.0',
description='Cachet URL monitor plugin', description='Cachet URL monitor plugin',
author='Mitsuo Takaki', author='Mitsuo Takaki',
author_email='mitsuotakaki@gmail.com', author_email='mitsuotakaki@gmail.com',

5
tests/test_configuration.py
View File

@@ -4,6 +4,7 @@ import unittest
import mock import mock
from requests import ConnectionError, HTTPError, Timeout from requests import ConnectionError, HTTPError, Timeout
from yaml import load, SafeLoader
import cachet_url_monitor.status import cachet_url_monitor.status
@@ -31,13 +32,13 @@ class ConfigurationTest(unittest.TestCase):
sys.modules['requests'].get = get sys.modules['requests'].get = get
self.configuration = Configuration('config.yml') self.configuration = Configuration(load(open('config.yml', 'r'), SafeLoader), 0)
sys.modules['requests'].Timeout = Timeout sys.modules['requests'].Timeout = Timeout
sys.modules['requests'].ConnectionError = ConnectionError sys.modules['requests'].ConnectionError = ConnectionError
sys.modules['requests'].HTTPError = HTTPError sys.modules['requests'].HTTPError = HTTPError
def test_init(self): def test_init(self):
self.assertEqual(len(self.configuration.data), 3, 'Number of root elements in config.yml is incorrect') self.assertEqual(len(self.configuration.data), 2, 'Number of root elements in config.yml is incorrect')
self.assertEqual(len(self.configuration.expectations), 3, 'Number of expectations read from file is incorrect') self.assertEqual(len(self.configuration.expectations), 3, 'Number of expectations read from file is incorrect')
self.assertDictEqual(self.configuration.headers, {'X-Cachet-Token': 'token2'}, 'Header was not set correctly') self.assertDictEqual(self.configuration.headers, {'X-Cachet-Token': 'token2'}, 'Header was not set correctly')
self.assertEqual(self.configuration.api_url, 'https://demo.cachethq.io/api/v1', self.assertEqual(self.configuration.api_url, 'https://demo.cachethq.io/api/v1',

5
tests/test_scheduler.py
View File

@@ -3,6 +3,7 @@ import sys
import unittest import unittest
import mock import mock
from yaml import load, SafeLoader
sys.modules['schedule'] = mock.Mock() sys.modules['schedule'] = mock.Mock()
from cachet_url_monitor.scheduler import Agent, Scheduler from cachet_url_monitor.scheduler import Agent, Scheduler
@@ -26,7 +27,7 @@ class AgentTest(unittest.TestCase):
def test_start(self): def test_start(self):
every = sys.modules['schedule'].every every = sys.modules['schedule'].every
self.configuration.data = {'frequency': 5} self.configuration.endpoint = {'frequency': 5}
self.agent.start() self.agent.start()
@@ -45,7 +46,7 @@ class SchedulerTest(unittest.TestCase):
mock_requests.get = get mock_requests.get = get
self.scheduler = Scheduler('config.yml') self.scheduler = Scheduler(load(open('config.yml', 'r'), SafeLoader), 0)
def test_init(self): def test_init(self):
assert self.scheduler.stop == False assert self.scheduler.stop == False