Apache Airflow and Regression Monitoring

Fri, 02 Oct 2020 00:00:00 +0000

Introduction:

Airflow is a platform to programmatically author, schedule and monitor workflows or data pipelines. It was originally developed and open sourced by Airbnb, later joined Apache Software foundation’s incubation program in 2016. Workflow is a sequence of tasks defined around Directed Acyclic Graph(DAGs) – which could be started on a schedule or triggered by an event or using Command line interface. Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiate pipelines dynamically.

Components:

Metadata DB:

Stores information’s like job status and task instance status.

Scheduler:

Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. The scheduler is the brains behind setting up the workflows in airflow. The execution time begins at DAG start date and repeat every schedule interval.

Web Interface (UI):

Airflow ships with a Flask app that tracks all the defined workflows and lets you easily change, start or stop them. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues.

CLI:

Airflow has a very rich command line interface that allows to test, run, backfill, describe and clear parts of your DAGs

Concepts:

DAG:

A DAG is the container that is used to organize tasks in a way that reflects their relationship, dependencies and set their execution context and order.

Operators:

Operators are the worker that run the tasks. Workflows are defined by creating a DAG of operators. They are broadly classified into three – Sensors, Operators and Transfers. Airflow provides many prebuild operators for many common tasks and new operators can be created by inheriting BaseOperator class.

Tasks:

Once an operator is instantiated, its is referred to as a “task”. Each task is user defined and responsible for performing a specific operation in the workflow. Instantiating a task requires providing a unique task_id and DAG container. Task can be python function or external scripts that could be invoked.

Example:

In the example, we show case - how Airflow could be used to express a workflow that can be used to generate the statistics/ report as part of end-to-end regression test suit; which involves multiple systems to work together. A traditional approach would use something very basic like bunch of batch scripts w/o CRON. But the challenge is - it would very easily get tangled and developer would spend a lot of time to figure out where the log files are or what failed and why/who owns what. Airflow helps solves this problem by helping in orchestrating your processes, managing the logs and really good dashboard with visualization of what failed and much more information.

References:

Python Plugins with Topics

Sun, 13 Sep 2020 00:00:00 +0000

Source Code: PyPlugins

Any infrastructure should always have the capability to extend itself. It would be better if the functionality is added by a contributor - who is not part of the core team. And it doesn’t get in the way of core components - development, compilation and deployment.

The concept of plugin has been around there for quite a while - Visual Studio/VS Code all has plugins (aka extensions). Basic idea here is to add new functionalities - by just deploying a new dll, jar or py modules.

In this proposal using python, we have a root folder named Plugins - the infra would enumerate the Plugins directory to add functions. The functionalities register themselves with a Key - let’s call them TOPICS.

Well why Topics? I am borrowing this idea from Messaging Queue infra like ZMQ and Kafka… So, that we could create a Topic to Function mapping - and we would be able to map caller to a MQ subscriber.

Let me walk through each component:

Base class

Base class that each plugin must inherit from; this class exposes couple of items 1) List of Topics 2) Execute Method - which your plugin should implement.

class IPlugin(object):
def __init__(self):
self.description = 'UNKNOWN'
self.topics = []
def execute(self, topic, argument):
"""The method that we expect all plugins to implement. This is the
method that our framework will call
"""
raise NotImplementedError

Example Plugin : Calculate

Calculate Plugin - exposes two functionalities: add and subtract. For the client its exposed as two topics. The execute function takes in topic and the argument. Based on the topic the respective plugin could dispatch it to sub-functions within the plugin.

class CalculatorPlugin(IPlugin):
def __init__(self):
self.description = 'Calculator'
self.topics = ['Add', 'Subtract']
def execute(self, topic, args):
if topic == "Add":
return self.add(args)
raise Exception (f'Topic: {topic} has no mapping function')
def add(self, args):
count = 0
for index in range(0, len(args)):
count += args[index]
return count

Service Discovery:

The infrastructure would enumerate the plugin base_directory and try find sub class of IPlugin. Create instance of the sub_class and register to the store: MAP<topic, instance>. Infra should be able to directly call the execute API, on the instance.

class ServiceDiscovery(object):
def __init__(self, plugin_package_dir='plugins'):
self.plugin_package_base_dir = plugin_package_dir
self.plugin_topic_instance_map = {}
self.enumerate_packages()
def enumerate_packages(self, package):
"""Recursively walk the supplied package to retrieve all plugins
"""
imported_package = __import__(package, fromlist=['foo'])
for _, pluginname, ispkg in pkgutil.iter_modules(imported_package.__path__, imported_package.__name__ + '.'):
if not ispkg:
plugin_module = __import__(pluginname, fromlist=['foo'])
clsmembers = inspect.getmembers(plugin_module, inspect.isclass)
for (_, c) in clsmembers:
# Only add classes that are a sub class of Plugin, but NOT Plugin itself
if issubclass(c, IPlugin) & (c is not IPlugin):
print(f' Found plugin class: {c.__module__}.{c.__name__}')
cls_instance = c()
for topic in cls_instance.topics:
print(f' Registering Topics: {topic}')
self.plugin_topic_instance_map[topic] = cls_instance

Execution

Now that we have a Map<Topic,instance>. When a call comes in - it would have a topic and the args. Using the Map, we could get the corresponding instance and call by passing in both the topic and args. This is similar to delegate (C#) or function pointers(in C).

class ServiceDiscovery(object):
def __init__(self, plugin_package_dir='plugins'):
...
self.plugin_topic_instance_map = {}
def execute(self, topic, argument):
if topic not in self.plugin_topic_instance_map:
raise Exception (f'Topic: {topic} is not registered')
return self.plugin_topic_instance_map[topic].execute(topic, argument)

Unit Test:

Writing unit test is not optional. Well, I am a fan of TDD 😉

def test_cal_plugin_add_func(self):
# Arrange
ser_dis = ServiceDiscovery()
# Action
result = ser_dis.execute("Add", [1, 2])
# Assert
self.assertEqual(result, 3)

Deployment:

Adding a new plugin should be as simple as

Copy and paste a new directory under the Plugin base directory
Directory should have a class which implements IPlugin

Conclusion:

Building a comprehensive plugin infrastructure is non-trivial; look at Visual Studio - you could override pretty much anything, starting from adding intellisese to a new compiler tool chain. Here, we are just look at a small tip - to get started - on having a python based plugin. Always starting off any infra project with the idea of extension in mind - is good to ensure cleaner responsibility separation.

From the SOLID principle : O -> our software should be Opened for extension but closed for modifications 😍

Python | Jacob Aloysious