Python | Jacob Aloysious

Access Control - Casbin

Sun, 31 Oct 2021 00:00:00 +0000

I was trying to read about RBAB (Role-Based Access Control) and ABAC(Attribute-Based Access Control); after some reading - I realized, as a software engineer - it’s way more easier when you get to read code - So, I started to search for open source project which implements access control infrastructure, and stumbled upon Casbin.

Snip from Readme: In Casbin, an access control model is abstracted into a CONF file based on the PERM metamodel (Policy, Effect, Request, Matchers). So switching or upgrading the authorization mechanism for a project is just as simple as modifying a configuration. You can customize your own access control model by combining the available models. For example, you can get RBAC roles and ABAC attributes together inside one model and share one set of policy rules.

The interesting part here is how Casbin - has designed the whole system behind a simple yet flexible configuration file and grammars to describe predicates. Intent of this blog is - just to get you curious about Casbin. Casbin has reasonally good documentation on their site.

ProTip: Best way to understand an open source is to start with the test cases, look at each test case - review the context and assert statements.

Lets look at a simple RBAC Model Config

[request_definition]
r = sub, obj, act
[policy_definition]
p = sub, obj, act
[role_definition]
g = _, _
[policy_effect]
e = some(where (p.eft == allow))
[matchers]
m = g(r.sub, p.sub) && r.obj == p.obj && r.act == p.act

RBAC_Policy.CSV

p, alice, data1, read
p, bob, data2, write
p, data2_admin, data2, read
p, data2_admin, data2, write
g, alice, data2_admin

Basically there are four parts: Request, Policy, Matcher and Effect. Ref

Request: Defines the parameter name and order which we should provide for access control matching function. This enables the code/model to be setup when the data is read from the source.

Policy: It’s the Access stratergy. This is how our internal system is designed. It defines the name and order of the fields in the Policy rule document.

Matchers:: This is similar to a predicate function - which shall describe if the request allowed. Rules which help us to match the request and the policy.

Effect: It’s useful, when there are multiple patterns which would match and you would need to make a decision. There are defaults, here you get to override both allow and deny defaults. Refer: PolicyEffect.

Inheritance:

If A has role B, B has role C, then A has role C. This transitivity is infinite for now
Casbin doesn’t distinguish role from user in its RBAC. They are all treated as strings.

p, data2_admin, data2, read
g, alice, data2_admin

In the above example,

data2_admin has read access to data2
alice inherits/is a member of role data2_admin. alice here can be a user, resource or a role.

test_role_manager.py

 rm = get_role_manager()
#1
rm.add_link("u1", "g1")
rm.add_link("u3", "g2")
self.assertTrue(rm.has_link("u1", "g1"))
self.assertFalse(rm.has_link("u1", "g2"))
self.assertCountEqual(rm.get_roles("u1"), ["g1"])
self.assertCountEqual(rm.get_roles("u2"), ["g1"])
#2
rm.add_link("u1", "g1", "domain1")
self.assertTrue(rm.has_link("u1", "g1", "domain1"))
self.assertFalse(rm.has_link("u1", "g1", "domain2"))

What Casbin does NOT do:

Authentication (aka verify username and password when a user logs in)
manage the list of users or roles.

Reference: Casbin

Bitmap Indexed Storage

Sat, 10 Jul 2021 00:00:00 +0000

Always been facinated by the ways - data structure’s can enable better storage and faster queries.

BitMapped Indexed Storage - is useful while using Columnar Databases - where per column data is stored together.

Often the number of distinct values in a column is small compared to the total number of rows. In the below example: column sex there are only two distinct value Female or Male in the given table - the cardinality is really low.

In such cases: We can now take a column with n distinct values and turn it into n separate bitmaps: one bitmap for each distinct value, with one bit for each row. The bit is 1 if the row has that value, and 0 if not.

Picture is worth a thousand words.. here we you go..

Fantastic Query performance… As the amount of data read from disk on to the memory is less - improved Disk Throughput. As always, solution are for a specific problem. If you wanna do a range query, then this solution might not the best.

If n is very small, those bitmaps can be stored with one bit per row. But if n is bigger, there will be a lot of zeros in most of the bitmaps (sparse). In that case, the bitmaps can additionally be run-length encoded or might not be recommened.

Reference: Designing Data-Intensive Applications, by Martin Kleppmann

Feature Toggle - Nightmares

Sat, 01 May 2021 00:00:00 +0000

We were working on a custom algorithm to optimally grab images given a list of area. Every image grabbed is written into an object store with a generated hashkey. The haskkey can be used to query the recorded data - when we would like to play them back.

Software was a monolith and the algorithm was shipped with it. So, any new fixes would be shipped with the new software versions and user would be able to playback the same recorded data.

def customAlgorithm():
locations = []
# actual algo...
return locations;

A day came when we - rolled out a bug fix in the algorithm. But, we broke the backward compatiblity 😟. Users had lots of existing recorded data which they wanted to playback. Since the algorithm had changed - the generated hashkeys didn’t match the ones in the object store.

We thought its proabaly a one off scenario: so we added a feature toggle - user will have to manually (pain) figure out - which version the data was recorded on and update the toggle config (xml) and rerun.

def customAlgorithm():
locs = []
if version == '1.0':
locs = oldAlgo()
else:
locs = newAlgo()
return locs

And - Well there was another bug fix:

def customAlgorithm():
locs = []
if version == '1.0':
locs = Algo_1_0()
else if version == ''2.0':
locs = Algo_2_0()
else:
locs = latestAlgo()
return locs

And yeah the story went on for one more iteration and we had to STOP!!!! 😓 It was already becoming a nightmare to mantain the code.

It wasn’t just manually figuring out the toggle which was creating pain - but to enable code reuse, the algorthm had to be refactored multiple times - so that parts of the algorithm can be reused across versions. In addition to that the number of test suites we had to manage had gone up significantly, as there were multiple users each one using different version - and oh yeah there were version specific fixes released as patches 😟

Finally we decided to write all the metadata - related to recording - into an embedded database. And on playback we get the information from the database and not by re-running the code/algorithm.

On hindsight: the database feels like it an obvious solution, right? well the devil is in the details - in our case it about effort(read, time). The playback infrastructure based on database took ~8 weeks to build -first version. It had to handle numerous (10s) use cases. Previously, we only had one flow to be maintained i.e the flow which ran the use cases + algo shipped. But now we have to manage TWO different flows - one for recording and one for playback (x) No_Of_Use_Cases.

Its almost 3yrs since we rolled out this solution to production. When we look back it was a very good decision to build two different workflows 😍. Code is more structured and maintainable.

At times - short term goal/fix takes priority. But, its always good to step back and look at the pain points and have an item in your backlog to find a better solution. The priority should be driven based on feedback: remember the 80/20 rule

80 percent of customers only use 20 percent of the features in the software they’ve bought.

BTW: the database schema has changed a lot - with the addition of multiple use cases. Which also needs version management 😉 And evey new use case discussion now has two parts to be discussed - some problems are good to have (read, tradeoffs).

Ref: Feature Toggle

Apache Airflow and Regression Monitoring

Fri, 02 Oct 2020 00:00:00 +0000

Introduction:

Airflow is a platform to programmatically author, schedule and monitor workflows or data pipelines. It was originally developed and open sourced by Airbnb, later joined Apache Software foundation’s incubation program in 2016. Workflow is a sequence of tasks defined around Directed Acyclic Graph(DAGs) – which could be started on a schedule or triggered by an event or using Command line interface. Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiate pipelines dynamically.

Components:

Metadata DB:

Stores information’s like job status and task instance status.

Scheduler:

Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. The scheduler is the brains behind setting up the workflows in airflow. The execution time begins at DAG start date and repeat every schedule interval.

Web Interface (UI):

Airflow ships with a Flask app that tracks all the defined workflows and lets you easily change, start or stop them. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues.

CLI:

Airflow has a very rich command line interface that allows to test, run, backfill, describe and clear parts of your DAGs

Concepts:

DAG:

A DAG is the container that is used to organize tasks in a way that reflects their relationship, dependencies and set their execution context and order.

Operators:

Operators are the worker that run the tasks. Workflows are defined by creating a DAG of operators. They are broadly classified into three – Sensors, Operators and Transfers. Airflow provides many prebuild operators for many common tasks and new operators can be created by inheriting BaseOperator class.

Tasks:

Once an operator is instantiated, its is referred to as a “task”. Each task is user defined and responsible for performing a specific operation in the workflow. Instantiating a task requires providing a unique task_id and DAG container. Task can be python function or external scripts that could be invoked.

Example:

In the example, we show case - how Airflow could be used to express a workflow that can be used to generate the statistics/ report as part of end-to-end regression test suit; which involves multiple systems to work together. A traditional approach would use something very basic like bunch of batch scripts w/o CRON. But the challenge is - it would very easily get tangled and developer would spend a lot of time to figure out where the log files are or what failed and why/who owns what. Airflow helps solves this problem by helping in orchestrating your processes, managing the logs and really good dashboard with visualization of what failed and much more information.

References:

Python Plugins with Topics

Sun, 13 Sep 2020 00:00:00 +0000

Source Code: PyPlugins

Any infrastructure should always have the capability to extend itself. It would be better if the functionality is added by a contributor - who is not part of the core team. And it doesn’t get in the way of core components - development, compilation and deployment.

The concept of plugin has been around there for quite a while - Visual Studio/VS Code all has plugins (aka extensions). Basic idea here is to add new functionalities - by just deploying a new dll, jar or py modules.

In this proposal using python, we have a root folder named Plugins - the infra would enumerate the Plugins directory to add functions. The functionalities register themselves with a Key - let’s call them TOPICS.

Well why Topics? I am borrowing this idea from Messaging Queue infra like ZMQ and Kafka… So, that we could create a Topic to Function mapping - and we would be able to map caller to a MQ subscriber.

Let me walk through each component:

Base class

Base class that each plugin must inherit from; this class exposes couple of items 1) List of Topics 2) Execute Method - which your plugin should implement.

class IPlugin(object):
def __init__(self):
self.description = 'UNKNOWN'
self.topics = []
def execute(self, topic, argument):
"""The method that we expect all plugins to implement. This is the
method that our framework will call
"""
raise NotImplementedError

Example Plugin : Calculate

Calculate Plugin - exposes two functionalities: add and subtract. For the client its exposed as two topics. The execute function takes in topic and the argument. Based on the topic the respective plugin could dispatch it to sub-functions within the plugin.

class CalculatorPlugin(IPlugin):
def __init__(self):
self.description = 'Calculator'
self.topics = ['Add', 'Subtract']
def execute(self, topic, args):
if topic == "Add":
return self.add(args)
raise Exception (f'Topic: {topic} has no mapping function')
def add(self, args):
count = 0
for index in range(0, len(args)):
count += args[index]
return count

Service Discovery:

The infrastructure would enumerate the plugin base_directory and try find sub class of IPlugin. Create instance of the sub_class and register to the store: MAP<topic, instance>. Infra should be able to directly call the execute API, on the instance.

class ServiceDiscovery(object):
def __init__(self, plugin_package_dir='plugins'):
self.plugin_package_base_dir = plugin_package_dir
self.plugin_topic_instance_map = {}
self.enumerate_packages()
def enumerate_packages(self, package):
"""Recursively walk the supplied package to retrieve all plugins
"""
imported_package = __import__(package, fromlist=['foo'])
for _, pluginname, ispkg in pkgutil.iter_modules(imported_package.__path__, imported_package.__name__ + '.'):
if not ispkg:
plugin_module = __import__(pluginname, fromlist=['foo'])
clsmembers = inspect.getmembers(plugin_module, inspect.isclass)
for (_, c) in clsmembers:
# Only add classes that are a sub class of Plugin, but NOT Plugin itself
if issubclass(c, IPlugin) & (c is not IPlugin):
print(f' Found plugin class: {c.__module__}.{c.__name__}')
cls_instance = c()
for topic in cls_instance.topics:
print(f' Registering Topics: {topic}')
self.plugin_topic_instance_map[topic] = cls_instance

Execution

Now that we have a Map<Topic,instance>. When a call comes in - it would have a topic and the args. Using the Map, we could get the corresponding instance and call by passing in both the topic and args. This is similar to delegate (C#) or function pointers(in C).

class ServiceDiscovery(object):
def __init__(self, plugin_package_dir='plugins'):
...
self.plugin_topic_instance_map = {}
def execute(self, topic, argument):
if topic not in self.plugin_topic_instance_map:
raise Exception (f'Topic: {topic} is not registered')
return self.plugin_topic_instance_map[topic].execute(topic, argument)

Unit Test:

Writing unit test is not optional. Well, I am a fan of TDD 😉

def test_cal_plugin_add_func(self):
# Arrange
ser_dis = ServiceDiscovery()
# Action
result = ser_dis.execute("Add", [1, 2])
# Assert
self.assertEqual(result, 3)

Deployment:

Adding a new plugin should be as simple as

Copy and paste a new directory under the Plugin base directory
Directory should have a class which implements IPlugin

Conclusion:

Building a comprehensive plugin infrastructure is non-trivial; look at Visual Studio - you could override pretty much anything, starting from adding intellisese to a new compiler tool chain. Here, we are just look at a small tip - to get started - on having a python based plugin. Always starting off any infra project with the idea of extension in mind - is good to ensure cleaner responsibility separation.

From the SOLID principle : O -> our software should be Opened for extension but closed for modifications 😍