Tech | Jacob Aloysious

PKCE

Fri, 03 Dec 2021 00:00:00 +0000

PKCE is pronounced as ‘pixie’. PKCE - Proof Key for Code Exchange => Credits

Concepts:

What is Confidential Client?

Any backend application written in languages like .Net, Java, NodeJS can be considered as Confidential Client. As they have the ability to “store variables” in our case Secret Keys - without exposing them to the end user - as the code is running on the server side. The secret key would be used to generate the Access Token in the OAuth Flow.

What is public Client?

Any Single Page Application (SPA), where all the code is downloaded to the users browser would be considered a Public Client. Coz: the secret key cannot be securely stored as the hacker could do a “view source code” and look at the secret key which is usually in plain text/string format.
Another example are Mobile Apps - the user can download and de-compile/reverse engineer - the application and find the secret key.

Why do you need PKCE?

Once the secret key is available - the hacker can impersonate as the Client and generate the access token (note: clientID is public and secret key is private). Once the access token is available - the hacker can access lot more confidential information about the user. So the existing OAuth2.0 model is not secure for both SPA and mobile apps. To solve this challenge - PKCE was introduced.

OAuth 2.0 Normal Flow:

PKCE Flow:

What is new in PKCE?

In the PKCE flow: Instead of a constant Secret Key - Everytime an OAuth flow is initiated - A new “Secret Key” is generated and Hashed.
Hash is shared with the Authentication Server(AS) when requesting for the Temporary Code. AS would remember this Hash value.
When the client request for the Access Token - it sends the newly generated “Secret Key” - the AS would check, if the secret matches the Hash (it already remembers - #2).

Reference:

RFC7636, Oauth Hack, What’s the difference between Confidential and Public clients

Designated Authentication - OAuth2.0

Thu, 02 Dec 2021 00:00:00 +0000

OAuth2.0 and OpenID - is one of the best videos (62 mins) on this topic - Highly recommended. This blog is just a quick summary (4 mins read) of it.

What is OAuth?

OAuth is a security standard where you give one application permission to access your data from another application i.e. you authorized one application to use data from another application on your behalf, without giving them your username and password. The steps to grant permission(s) or consent is often reffered to as authorization or delegated authorization. Permissions are scoped e.g. client can be given access to read your GMail contacts but cannot Read/Write your email. OAuth is not exactly used for user identification (who loggged in), instead its used to access API’s for resources like gmail contact list.

You check-in to a hotel.
At the front desk - You give your details like Passport and Reservation
Front Desk would in turn - give you the Key Card i.e. Access Token
You can use the Key Card to access your reserved Room

Here, the Front Desk is the Authentication Server - the key card is your Access token used to acess your Room (Resources API). The API doesn’t really care who is accessing the Resource as long as they have the access token.

OAuth Terminologies:

Lets assume you are giving Yelp access to your Gmail account contacts..

Resource Owner: Represent you the owner of the gmail account; who can give permission/authorize.
Client: Represent Yelp i.e. the application which would like to perform action on behalf of you.
Authorization Server: One who validates the gmail uname/password and authenticate - in this case accounts.google.com. Resource owner will already have an account with this server (gmail).
Resource Server: Server (API) which has the actual content/resources like contact list in this case contacts.google.com. Which client would talk to get the actual data.
Authorization Grant: Once authenticated an Authorizatiion Code or temporary code is sent back to he client. So, the client has been granted relevant access to the resource owners gmail contacts. This is short lived, this is used to inturn get the access token (long lived).
Redirect URI: Also known as Callback. It is the URI to which the response of Authentication #5 should be sent back to client #2
Access Token: Token/Key used by the Client to access all the resources of the owner; based on the authorized permissions.
Response Type: The type of information the client would like to receive e.g: code (authorization code)
Scope: Granular permission the client wants acess to like Read Contact, Read Profile basic info.
Consent: Whether the client can be given permission to access the requested information.
ClientID: Unique ID of the client, which is used to talk to the authorization server.
Client Secret: Secret Key - only the client and the auth server shares. Helps Client and Resource Server to share information secretly behind the scenes.

OAuth Flow:

Prerequisite:

Client and the Authorization Server should establish a working relationship. Both the “ClientID (AppID)” and “Client Secret(AppSecret)” are generated by the Authorization Server. The client ID would be used to uniquely identify the client. Its the responsibility of the client to ensure the “ClientSecret” is kept secret and its only know by client and authorization server. As there are the only means for the Auth Server to verify the client.

Actual Flow:

In this example, Yelp would want to authenticate a user based on GMail Authentication, and access Profile/Contacts information.

Client(Yelp) would make a request to Authenticate - to “Autorization Server” (accounts.google.com); as part of the request:
1. “RedirectURI” is set to a callback URL of the client “yelp.com/callback”
2. “Respose Type” is set to “Code”, so a auth code is sent back as response to 1.a
3. “Client ID” and “Scope” (Profile/Contacts)
Auth Server pops up a dialog (mostly) - where user gets to enter the GMail credentials (uname/pass) and authenticate
Auth Server shows up a consent form - list the info access requested (like contacts, profile) - user gets to choose Yes or NO
Once concent(Yes) is given - an “Auth Code” is generated and sent back to the the client - to specifiied callback/redirectURI.
Then, the client sends the “Auth Code + ClientID + ClientSecret” to “Authorization Server” and gets back an “Access Token”
Using the “Access Token”, Client can contact the resource server (contacts.google.com) to get the actual resources (profile, contacts) of the user.

Reference: https://www.youtube.com/watch?v=aU9RsE4fcRM

WebFinger

Wed, 01 Dec 2021 00:00:00 +0000

This blog contain just snips - a detailed explanation is available in the Reference links..

Background:

Twitter started the project called Bluesky with a goal to build a social web. As part of it, ActivityPub a decentralized social networking protocol was built - which uses WebFinger in the background.

What is WebFinger:

WebFinger is used to discover information about people or other entities on the Internet that are identified by a URI using standard Hypertext Transfer Protocol (HTTP) methods over a secure transport. A WebFinger resource returns a JavaScript Object Notation (JSON) object describing the entity that is queried.

Key Items to Note:

WebFinger protocol is used to request information about an entity identified by a query target (a URI).

A WebFinger request is an HTTPS request to a WebFinger resource.

A WebFinger resource is a well-known URI

A WebFinger resource is always given a query target, which is another URI that identifies the entity whose information is sought. GET requests to a WebFinger resource convey the query target in the “resource” parameter of the WebFinger URI’s query string

The path component of a WebFinger URI MUST be the well-known path "/.well-known/webfinger”.

The WebFinger resource returns a JSON Resource Descriptor (JRD) as the resource representation to convey information about an entity on the Internet

The use of HTTPS is REQUIRED to ensure that information is not modified during transit. Clients MUST verify that the certificate used on an HTTPS connection is valid and accept a response only if the certificate is valid.

The Registration Template:

Property Identifier:
Link Type:
Description:
Reference:
Notes: [optional] The “Property Identifier” must be a URI that identifies the property being registered.

Example:

Suppose Lia wishes to authenticate with a web site she visits using OpenID Connect. She would provide the web site with her OpenID Connect identifier, say lia@example.com. The visited web site would perform a WebFinger query looking for the OpenID Connect provider. Since the site is interested in only one particular link relation, the WebFinger resource might utilize the “rel” parameter

Request:

 GET /.well-known/webfinger?
resource=acct%3Alia%40example.com&
rel=http%3A%2F%2Fopenid.net%2Fspecs%2Fconnect%2F1.0%2Fissuer
HTTP/1.1
Host: example.com

Response:

 HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/jrd+json
{
"subject" : "acct:lia@example.com",
"links" :
[
{
"rel" : "http://openid.net/specs/connect/1.0/issuer",
"href" : "https://openid.example.com"
}
]
}

Since the “rel” parameter only serves to filter the link relations returned by the resource, other name/value pairs in the response, including any aliases or properties, would be reeturned. Also, since support for the “rel” parameter is not guaranteed, the client must not assume the “links” array will contain only the requested link relation.

Take ways:

The specification is rather simple as it uses JSON as the data model for all the communication. Almost, all languages provide support for handling JSON. Validation is still not part of the spec - So both validation and handling attack should be the responsiblity for the client.

References:

WASM Cloud

Fri, 26 Nov 2021 00:00:00 +0000

What is WASM Cloud trying to solve?

Lets ask - what a typical “Development Iteration” look like?

Start of with your Napkin Drawings - conceptual vision of what is it you want to build
Initial Prototype, prove that it works
Then, Figure out how to scale

The challenge is between #2 and #3 there is more often a complete rewrite. As the author claims- Virtually nothing from prototype is written in a way it is scalable.

The goal of the project is to “Ship Napkins to Production” directly.

Draw your conceptual map of how your application should look like
Fill in the blanks with your business logic and just push it - boom!!
Don’t worry about any of the non-sense in between.
Remove all the boiler plate and let developer write only Business logic.

How is this achived?

Using the “Actor Model” and the out of the box boilerplate infrastructure. Personally, I a fan of Actor Model, as its easy to just write a function which has a single responsibility. If you have ever worked with Threads and IPC(Interprocess communication) - you would know the overheads that come in managing them. With the Actor model you have an input queue, an output queue and the function in between. Developer shall write only the function. Function would read from the input queue, process and write it to the output queue. The infrastructure should take care of running the actor, as a thread or process and also manage the input/output queue, data transfer, serialization etc. One of the popular implementation of Actor model is AKKA.

From the authors:

The creators of wasmCloud believe that we can not only change the way developers build software for the better, but make it easier to secure, deploy, maintain, observe, and upgrade that software as well–all while reducing the amount of boilerplate we have to copy and paste.

Example:

Lets look at their Hello World example.

Interesting snippets:

 #[derive(Debug, Default, Actor, HealthResponder)]
#[services(Actor, HttpServer)]
struct HelloActor {}
/// Implementation of HttpServer trait methods
#[async_trait]
impl HttpServer for HelloActor {
//Actual Impl
}

HelloActor is the name of your actor
The two lines with “#” above the actor name igenerate - at compile time - nearly all of the scaffolding needed to build an actor.
The HealthResponder term generates a function that automatically responds to health check queries from the wasmCloud host.
The #[services(…)] line declares the servicesthat your actor implements, and generates message handling code for those interfaces.
All actors implement the Actor interface.
The HttpServer entry declares that the actor will also implement that interface, and requires an implementation of that trait’s method: handle_request.

Conclusion:

I am fan of distributed computing: designing and writing scalable application. Maybe the concept of wasmCloud doing the heavy lifting sounds easy; but I belive its a very hard problem. Coz: when we actually get our hands dirty with actual application design - challenges like: managing states, versioning, schema management/registry, idempotency, trade offs w.r.t CAP theorm, heterogeneous system etc..all comes in. Also, given what Docker + Kubernetes eco system already provides - finding the right architecture to use wasmCloud would be a challenge. But this is definitely a place lot of things can be improved and room for optimization. Star’ed the project for now, will be following it closely.

Reference: WASM Cloud, GitHub

Rubber Duck Debugging

Wed, 24 Nov 2021 00:00:00 +0000

I have been in a similar situation multiple times - where in I was stuck on a problem. To get a solution, I would want to reach out for help. So, I would start to write it down - as it could post it in StackOverflow or send it as an email to my friend/colleague. While I am writing the problem down, I try to set the context and elaborate on the problem statement - after few iteration/reformatting/edits - its strange that I would have found the solution. Oops-What happened???

Stepping back and looking at it, what had happened is - while we try to explain the problem to someone (in writing) - we try to remove all the clutter and focus on the actual problem - with the intent to keep the email shorter and digestable components, so that the audience/reader is not overwhelmed. But, in the process we have a better clarity of the core (only) problem we are trying to solve. When that clarity comes up, more often we get closer to the solution.

BTW: I never knew, there was a name for it - “Rubber Duck Debugging” until I heard it from scott hanselman in one of his podcast. And there is a website rubberduckdebugging for it.

Snip from the site:

Place rubber duck on desk and inform it you are just going to go over some code with it, if that’s all right.

Explain to the duck what your code is supposed to do, and then go into detail and explain your code line by line.

At some point you will tell the duck what you are doing next and then realise that that is not in fact what you are actually doing. The duck will sit there serenely, happy in the knowledge that it has helped you on your way.

Well, I had to buy one rubber duck, as I wasn’t able to negotiate to lease one with my 7yr old daughter 😉

The next time you get stuck on a hard issue, take a moment and write down exactly whats going wrong as if you were going to explain it to someone else, you may just notice exactly where you are going wrong and even if you dont, at least you have a question all ready for submission to StackOverflow…

Software Enthusiast - Getting Started

Wed, 24 Nov 2021 00:00:00 +0000

I have been asked this question multiple times. I would like to be a software enthusiast and where do I get started. Thanks to my Friend to help put this together

Books:

Coding:

Clean Code: A Handbook of Agile Software Craftsmanship *

Unit Testing:

The Art of Unit Testing, 2nd Edition *

Design:

SOLID principles:

Refactoring:

Architecture:

Keeping up with ever evolving tech Space:

Youtube channels to follow

Twitter handles to follow:

Podcasts & blogs to follow:

Reverse ETL

Sun, 14 Nov 2021 00:00:00 +0000

While listening to Podcast stumbled upon this term of Reverse-ETL.

First lets align on a few basic terminologies:

What is a Data Warehouse (DW):

Data Warehouse centralizes and consolidates data from multiple sources like application log files and transaction applications. Orgainzation use DW to gain valuable insights from data, which can improve decision making by building a historical record of data. It can be considered the single source of truth.

What is ETL

ETL Stands for Extract Transform and Load. ETL is a data integration process that combines data from multiple data sources into a single, consistent data store that is loaded into a data warehouse or other target system.

What is Reverse ETL

Reverse ETL is the process of copying data from a central data warehouse to operational systems of record, including but not limited to SaaS tools used for growth, marketing, sales and support.

Discussion:

Hang On! this doesn’t make sense - why should someone do ETL and then Reverse it? Coz, just ETL in itself - is a HUGE infrastructure to both develop and maintain - why would I manage both. Well, as always devil is in the details.

Lets see an example from Hightouch

Reasoning(by Hightouch):

Reverse ETL is necessary because your data warehouse — the platform you bought to eliminate data silos — has ironically become a data silo. Without reverse ETL, your business’s core definitions only live in the warehouse.

More detailed information like use cases are avaialble in this blog Reverse ETL

Conclusion:

The intent here seems to bridge the knowlege gap, that comes in due to data warehouse. Say: If a business user is so used to a particular platform/tech like SalesForce, the user would prefer the analystic output/data to be available in their platform. Incase of Data Warehouse, the SalesForce business user should understand and learn the new ways to access the data and its corresponding schema in the DW; and also learn the access mechanisms like scripts, sql etc.. With Reverse ETL, it creates a comfortable/known space for the user as all the “Actionable Data” from the analytics are available in their platform. So, sure it makes sense.

Only challenge, I see as an organization is - the effort in maintaining both ETL and Reverse ETL infrastructure. Any changes in the DW schema - should be notified to both the teams and the deliverables should be synced up - to avoid any broken pipelines.

Reference: DataWarehouse

Access Control - Casbin

Sun, 31 Oct 2021 00:00:00 +0000

I was trying to read about RBAB (Role-Based Access Control) and ABAC(Attribute-Based Access Control); after some reading - I realized, as a software engineer - it’s way more easier when you get to read code - So, I started to search for open source project which implements access control infrastructure, and stumbled upon Casbin.

Snip from Readme: In Casbin, an access control model is abstracted into a CONF file based on the PERM metamodel (Policy, Effect, Request, Matchers). So switching or upgrading the authorization mechanism for a project is just as simple as modifying a configuration. You can customize your own access control model by combining the available models. For example, you can get RBAC roles and ABAC attributes together inside one model and share one set of policy rules.

The interesting part here is how Casbin - has designed the whole system behind a simple yet flexible configuration file and grammars to describe predicates. Intent of this blog is - just to get you curious about Casbin. Casbin has reasonally good documentation on their site.

ProTip: Best way to understand an open source is to start with the test cases, look at each test case - review the context and assert statements.

Lets look at a simple RBAC Model Config

[request_definition]
r = sub, obj, act
[policy_definition]
p = sub, obj, act
[role_definition]
g = _, _
[policy_effect]
e = some(where (p.eft == allow))
[matchers]
m = g(r.sub, p.sub) && r.obj == p.obj && r.act == p.act

RBAC_Policy.CSV

p, alice, data1, read
p, bob, data2, write
p, data2_admin, data2, read
p, data2_admin, data2, write
g, alice, data2_admin

Basically there are four parts: Request, Policy, Matcher and Effect. Ref

Request: Defines the parameter name and order which we should provide for access control matching function. This enables the code/model to be setup when the data is read from the source.

Policy: It’s the Access stratergy. This is how our internal system is designed. It defines the name and order of the fields in the Policy rule document.

Matchers:: This is similar to a predicate function - which shall describe if the request allowed. Rules which help us to match the request and the policy.

Effect: It’s useful, when there are multiple patterns which would match and you would need to make a decision. There are defaults, here you get to override both allow and deny defaults. Refer: PolicyEffect.

Inheritance:

If A has role B, B has role C, then A has role C. This transitivity is infinite for now
Casbin doesn’t distinguish role from user in its RBAC. They are all treated as strings.

p, data2_admin, data2, read
g, alice, data2_admin

In the above example,

data2_admin has read access to data2
alice inherits/is a member of role data2_admin. alice here can be a user, resource or a role.

test_role_manager.py

 rm = get_role_manager()
#1
rm.add_link("u1", "g1")
rm.add_link("u3", "g2")
self.assertTrue(rm.has_link("u1", "g1"))
self.assertFalse(rm.has_link("u1", "g2"))
self.assertCountEqual(rm.get_roles("u1"), ["g1"])
self.assertCountEqual(rm.get_roles("u2"), ["g1"])
#2
rm.add_link("u1", "g1", "domain1")
self.assertTrue(rm.has_link("u1", "g1", "domain1"))
self.assertFalse(rm.has_link("u1", "g1", "domain2"))

What Casbin does NOT do:

Authentication (aka verify username and password when a user logs in)
manage the list of users or roles.

Reference: Casbin

Bitmap Indexed Storage

Sat, 10 Jul 2021 00:00:00 +0000

Always been facinated by the ways - data structure’s can enable better storage and faster queries.

BitMapped Indexed Storage - is useful while using Columnar Databases - where per column data is stored together.

Often the number of distinct values in a column is small compared to the total number of rows. In the below example: column sex there are only two distinct value Female or Male in the given table - the cardinality is really low.

In such cases: We can now take a column with n distinct values and turn it into n separate bitmaps: one bitmap for each distinct value, with one bit for each row. The bit is 1 if the row has that value, and 0 if not.

Picture is worth a thousand words.. here we you go..

Fantastic Query performance… As the amount of data read from disk on to the memory is less - improved Disk Throughput. As always, solution are for a specific problem. If you wanna do a range query, then this solution might not the best.

If n is very small, those bitmaps can be stored with one bit per row. But if n is bigger, there will be a lot of zeros in most of the bitmaps (sparse). In that case, the bitmaps can additionally be run-length encoded or might not be recommened.

Reference: Designing Data-Intensive Applications, by Martin Kleppmann

Sqlite - Blunders

Sat, 05 Jun 2021 00:00:00 +0000

What is Sqlite:

SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine. SQLite is the most used database engine in the world. SQLite is built into all mobile phones and most computers and comes bundled inside countless other applications that people use every day. More Info: SQLite

My Learning:

Sqlite is a fantastic embedded database - light weight, easy to use and almost Zero Configuration. I would prefer to call it ZMQ of DataBase 😉

Blunder:

As part of our design: we had to persist a bunch of parametric information per user session (few MBs). This information is not required outside the user session. But, it would be reused when the user comes back. Session - is local running on a windows box. We decided to use sqlite as our backend persistence mechanism; as there was no installation or specific configuration required to set sqlite up.

Since there are parallel read write happening, we had to use the WAL: Write Ahead Logging journal mode.

We did couple of blunders:

Blunder #1 - Insert ONE at a time

When an action occured - we opened the db connection, inserted record and closed it.
We didn’t want to keep the connection open, throughout the session - as there were chances of software crash - which could potentially corrupt he database.
But as time went by, the number of actions became HUGE (order of millions - automated). At this point - Opening and closing the connection for every action, started to hurt us (high latency and low throughtput). Coz, the time taken to open and close a connection (~300ms) was adding up.

Solution: Microbatching: Batched about 500 actions together - open the connection once, inserted all 500 actions in one shot and closed - Transaction. Worst case, we would loose less than 500 actions, and that tradeoff was acceptable.

Blunder #2 - Mounted Drive:

The db had to be persisted on a mounted Linux drive (there are reason for this, I won’t go into details).
As the session started - the app created a sqliteDB directly on the mounted drive and started inserting.
Since we were using WAL mode - there were temporary files getting created and deleted on the mounted drive during every transaction.
As time went by - the reliablity of the drive/driver went down, and retries didn’t help keep up. The throughput significantly got impacted and multiple errors started showing up.

Solution: During the session, create the db in a local (temp) drive, and end of sesssion copy it into the mounted drive. Similarly, when session restarts - copy from mounted drive on to a new temp location and consume it; and copy it back on completion.

Never EVER - read or write an Sqlite Database from a mounted Drive. It usually works, but when it doesn’t - its a nightmare.

Blunder #3 - Lazy Load:

When a specific information was required - we opened the db connection, executed the “select” query and return the results.
Again: We didn’t want to keep the connection open, throughout the session - as there were chances of software crash - which could potentially corrupt he database.
The read was implemented in a “Lazy - On demand” fashion - so that only the minimum required information was read on demand.
Reading only the required info is an ideal design - why would we ever read more that what is required? GraphQL - is a good example of how to design minimilistic API.
But the challenge in our (specific) case: was the number of reads - it was about 600reads in less than 50 milliseconds. And each one had to open and close the connection (which takes ~50ms).
Since it was a high throughput system, the overhead of lazy read was slowing down the entire software.

Solution: PreFetch Require Information

Since we knew the sequence in which the actions are performed, we redesigned the query to read “more” records than required.
Then created an in memory cache [index/map] to hold the data and filter based on Client Query.
Read from DB would occur only if there is a cache miss.
The cache was a rolling buffer with LRU (Least Recently Used) and a watermark (memory or count), so that the RAM didn’t explode.
When a new read from DB was required, if the cache is full - we delete the oldest item and read from DB and add it to cache.
No change to the Client API. Only the Data Acess Layer changed. 😉

Conclusion:

As always on hindsight, the solution looks trivial and obvious. Since, we were using Sqlite for the first time and initially didn’t reallize the data could grow exponentially - had to learn (Adapt) the hard way. Esp, when these are running in production - you need scheme’s to manage old vs new verisons of software (read, backward compatibility).

Concurrent Dictionary and Delay - C#

Sun, 30 May 2021 00:00:00 +0000

Reference: ConcurrentDictionary

In a high throuput and low latency enviorment - we had a peice of code in the critical path - which helped in throttling. After version #2.0 of the software was released (internally) - we saw the througput has gone down significantly almost 3X - compared to previous Version #1.0 - 4mins to 11 minutes.

We looked at the history/change sets between the versions: it was huge, 100s of commits and software was running with 100s of threads - on a 2Milliion LOC software stack. So, we were unsure which peice of code nor thread - was causing the delay. We were all over the place, trying to see if its a system issue or memory trend or some fragmentation vs memory/back pressure vs excess logging etc…. And, after 3 day (yes: 3days slog) - we found its a delay caused by the ConcurrentDictionary class!!! 😳

It was an unpleasant suprise, since we didn’t expect such a huge delay cause due to a .Net Framework class.

1. Version 1.0: [4mins]

One API with its backing data store with one Dictionary.

 private Dictionary<int, User> userInfos;
public List<User> GetUsers() {
return userInfos.Values;
}

2. Version 2.0: [11 minutes]

In version #2: We changed the data store, which was holding the user info from a simple dictionary to a two level ConcurrentDictionay.

There was some obvious reason for using ConcurrentDictionay - I won’t go into the details. But, after changing the data store- we ended up blindly updating all usages to consume from the new data store.

And the API (GetUsers) implementation was updated, interface remained the same - but: it was in the critical path.

 private ConcurrentDictionary<int, ConcurrentDictionary<int, User>> userInfos;
public List<User> GetUsers(){
var users = new List<User>();
for(var userGroup in userInfos.Values){
for(var user in userGroup.Values){
users.add(user)
}
return users;
}
}

3. Version 3.0: [4 minutes]

After looking into the usages “in the critical path” - we only required the count of the active user groups.

So, we wrote another API to just return the count. With this change, we were able to bring it back the throughput back to #Version 1.0. Huff!! 😌

 private ConcurrentDictionary<int, ConcurrentDictionary<int, User>> userInfos;
public List<User> GetActiveUserGroups(){
return userInfos.Count;
}

Learning:

On highsight, the fix looks trivial and straight forward. But the challenge was the huge debugging effort to find the root cause. And little did we suspect a .Net Framework class can cause such a huge delay and slow down overall execution time. It was a great learning for us as a team - involved in debugging. BTW, we are talking about a code base of about few Million lines of code.

BTW: Concurrent dictionary exists for a reason: Usage of it as our data store was the right choice.

Represents a thread-safe collection of key/value pairs that can be accessed by multiple threads concurrently.

If a class should be thread safe, then there has to be a lock/Mutex implemented. A lock has its own tradeoff in terms of execution time (Thread Scheduling, Prempt etc..) - so think carefully on its usages.

Parallel Processing with Constraints

Sat, 15 May 2021 00:00:00 +0000

An interesting senario, I recently encountered - have tried to tear down the context to get to the solution early, and avoid nitty gritty (boring) details. Also, example showns only 2 threads.

Context:

Different Actions (like: 1,2,3) from Cutomers (like Mr.A, Miss.B..) are pushed into a stream/queue from an external component. “$” denotes end/last action from a customer. A thread can 1) Process each action 2) Generate aggregated views for each customer per session - and then write it to the sink. The processing threads are started when the session starts and a user cannot reenter the session once done.

note: all threads mentioned are background and long running.

Constraints:

Aggregation can be triggered only after all the actions are processed. i.e processing of action A1, A2, A3 should be done - before aggregation “A$” of A is triggered.

1. Single Thread

Well, this is a no brainer. We have one thread which reads (pull) from the stream/queue and process one item from the queue at a time. All constraints can be managed with a global context inside the thread.

Below is a code which runs inside the Thread.

 # {name, [actions]}
customer = {}
def read_from_stream(name, action):
if action=='$':
aggregate(name)
else:
process(name, action)
def process(name, action)
result = process_the_custom_action(action)
cusomter[name].append(result)
def aggregate(name):
action_results = cutomer[name]
view = gen_agg_view(action_results)
write_to_sink(view)
# remove customer from tracking
del customer[name]
def process_the_custom_action(action):
#Heavy lifting operation
return result

2. Pull Model - Parallel Processing

Later we realized, the function processing_the_custom_action is heavy lifting and takes more CPU. So, we were not able to keep up, with the incoming data rate. So, the memory increased drastically and triggers garbage collection and slow down actual code execution.

So, we decided to introduce multiple threads to read from the queue. Same code as above - executed in multiple threads. Each thread would pop (PULL) one item from the queue, process it and then pop the next item. Just to be explicit: one item from the queue is processed by only one of the threads, but which thread (order) gets which item is not guranteed.

This design had a major challenge in ensuring the constraints are honored.

In the above example: Aggregation for customer “A$” should be triggered only after last action “A3” is done. But “A$” and “A3” are processed in two different threads i.e. A3 by Thread #1; A$ by Thread #2. How would Thread #2 know A3 processing is done?? 😞

If we really need to go with this approach and also honor the constraints - then there should be broadcast events fired and all threads subscribed to it. Each thread should subscribe and do individual book keeping for all customers being processed across all the threads (dependency across threads, coupling..huff!!) 😰

Pros: If any of the thread is killed in between. The data loss would be minimal i.e only the customers already read by that specific thread in the past and not completed. As all NEW data would be processed by one of the threads alive.

3. Push Model - Parallel Processing

To ensure, we honor the constraint and also enable parallel processing. We introduced a load_balancer, called it a distributor module.

Distributor Module:

Would have reference to all the available processing threads.
Each thread would have a corresponding queue. So, No_Of_Threads equals No_Of_Queueus.
Each queue would have an index, which range from 0 to No_of_threads.

0 < Queue_Index < No_Of_Threads
Once an item is read - it generates a hash using the “name” of the customer. (like: SHA256)
On the HASH we apply a MOD using the No_Of_Threads available and get the queue index

Queue_Index = hash(customer_name) % No_Of_Threads
Push the item to the corresponding queue_index

With this approach, we ensured - same customer items are always processed by the same thread. So, there are no cross talks across threads or external dependency. We could increase or decrease the number of threads used (obviously: before a session starts not in-between), based on a configuration or heuristics.

 #### Distributor Module
def read_from_stream(name, action):
# Generate hashkey using customer name.
# Mod using the number of threads. No_Of_Thread == No_Of_Queues
queue_index = hash(name) % No_Of_Threads
publish(queue_index, name, action)
def publish(queue_index, name, action):
# get the queue reference using the index
queue = queues[queue_index]
# add the item into corresponding queue
queue.push(name, action)

Cons: The biggest cons: If a thread is killed for what ever reason. All the customers past and future once - which would be queue/allocated to the thread would be dropped. But in the pull model, it would have dropped only a handful sitting inside (past) the thread. Well we could handle this dynamically - like have a heart beat mechanism and update the “No_Of_Thread” and get it working - just that it would be another infra.

Conclusion:

There are pros and cons to both approaches.

More Infra == More Code == More Maintainability == More to Debug == More room for bugs to creep in.

Trying to keep it simple while keeping the tradeoffs minimal is the key. In our case; we went with the push approach - as the chances of a thread crash was minimal and also we added some defensive coding and drop an item safely if required.

Reference:

Storm Model, by Nathan Marz

Choreography vs Orchestration

Sun, 02 May 2021 00:00:00 +0000

I have been getting into this debate quite often; which approach is preferred - Choreography vs Orchestrator. As always, it depends - Here is my take on it…

Modules:

App: Like an UI module, where user trigger’s an action like:Create an account
Action #N: Specific modules, which has one specific action to be performed like: sending email, mail atm cards - once an account is created.
Coordinator: Module, which handles/triggers all the downstream actions (#n) - when an event is recived from App.

Orchestrator

Here the coordinator is the orchestrator - which has the knowledge of the business rules - like: what are the set of actions that need to be performed when a specific event occured. If you observe, the arrows are pointing from coordinator towards actions. So, the coordinator need to understand the details of each and every API (per Actions) - their base URL/REST end points. And if each of them takes in differnt params - then all this information would sit inside the coordinator module.

The cooridnator has become the central brain. And now it has tight coupling with all the Actions. And if a new action should be added, we will have changes to the coordinator module. After a point of time - this module will become the “God like” service. And every new Actions team would have to come to coordinator team (more meetings/discussion/followps) - to have their action integrated with the coordinator. Any changes to the coordinator module would become extremely hard - as there is so much coupling. And in addition failures of each action should be handled with special case - adding more and more actions specific details/code into the coordinator module.

On deployment and version management: If there are specific changes to an Actions - URI or params. Either the action module - can never depricate their existing uri and params, as it is used by coordinator module. Or, coordinator module would have to understand the versions (of action) and act accordingly. Both can get a bit too messy. And choosing one or either of them - would require some tradeoffs (debates) to be made somewhere.

BTW: When we are just starting of - coordinator might be the easiest way to get things done (read, move faster). But as we scale out - we should look for approaches which would enable better decoupling.

Pros:

Better control in coordinating the Action modules
Handling failures would be easiers - in one place
One place to look at the flow chart
Good for transaction driven and places where actions are not idempotenent

Choreography:

In this approach, lets take a broker like KAFKA as an example. When a new event is created its publised to a KAFKA queue with a specific TOPIC. There could be multiple subscribers (actions) - which would subscribe to the topic. The subscribers have their specific logics and subscibe to specific topics. At any point of time, any new actions can be added into the system without the knowledge of the App module. Look at the red arrows - there are NO arrow pointing from App towards specific Actions - so app doesn’t have any dependency on Action. We could also have custom subscribers (Actions) - which would run a pipeline for the data science team - this would enable us to generate observability data on the loads and trends.

Deploy: In this approach both app and action moudles can be indeopendently deployed and upgraded. The schema registry would have the contract details - topic vs the payloads.

Schema Registry: Every payload schema: should be published in the schema repository. We can use libraries like “Protocol Buffers” or “Apache Avro” for describing payload schemas.

Cons:

Handling errors might become very tricky. You would need an external monitoring infrastructure to track this.
If there are depenencies across actions - then you should think twice on using this pattern. Though there are ways to solve it - like: using sub topics.
Debugging can become a bit more hard. Needs distributed tracing to correlate/debug.

Conclusion:

Personal preference is to lean towards choreographed approach - as they are more loosely coupled and are more flexibble and amenable to change. Again: it all depends on your use case - if you are building a low latency system - which should catch errors and react to it really fast based on lots of buisinees logic, then orchestrator might be a better approach.

References:

Book : Building Microservices by Sam Newman

Feature Toggle - Nightmares

Sat, 01 May 2021 00:00:00 +0000

We were working on a custom algorithm to optimally grab images given a list of area. Every image grabbed is written into an object store with a generated hashkey. The haskkey can be used to query the recorded data - when we would like to play them back.

Software was a monolith and the algorithm was shipped with it. So, any new fixes would be shipped with the new software versions and user would be able to playback the same recorded data.

def customAlgorithm():
locations = []
# actual algo...
return locations;

A day came when we - rolled out a bug fix in the algorithm. But, we broke the backward compatiblity 😟. Users had lots of existing recorded data which they wanted to playback. Since the algorithm had changed - the generated hashkeys didn’t match the ones in the object store.

We thought its proabaly a one off scenario: so we added a feature toggle - user will have to manually (pain) figure out - which version the data was recorded on and update the toggle config (xml) and rerun.

def customAlgorithm():
locs = []
if version == '1.0':
locs = oldAlgo()
else:
locs = newAlgo()
return locs

And - Well there was another bug fix:

def customAlgorithm():
locs = []
if version == '1.0':
locs = Algo_1_0()
else if version == ''2.0':
locs = Algo_2_0()
else:
locs = latestAlgo()
return locs

And yeah the story went on for one more iteration and we had to STOP!!!! 😓 It was already becoming a nightmare to mantain the code.

It wasn’t just manually figuring out the toggle which was creating pain - but to enable code reuse, the algorthm had to be refactored multiple times - so that parts of the algorithm can be reused across versions. In addition to that the number of test suites we had to manage had gone up significantly, as there were multiple users each one using different version - and oh yeah there were version specific fixes released as patches 😟

Finally we decided to write all the metadata - related to recording - into an embedded database. And on playback we get the information from the database and not by re-running the code/algorithm.

On hindsight: the database feels like it an obvious solution, right? well the devil is in the details - in our case it about effort(read, time). The playback infrastructure based on database took ~8 weeks to build -first version. It had to handle numerous (10s) use cases. Previously, we only had one flow to be maintained i.e the flow which ran the use cases + algo shipped. But now we have to manage TWO different flows - one for recording and one for playback (x) No_Of_Use_Cases.

Its almost 3yrs since we rolled out this solution to production. When we look back it was a very good decision to build two different workflows 😍. Code is more structured and maintainable.

At times - short term goal/fix takes priority. But, its always good to step back and look at the pain points and have an item in your backlog to find a better solution. The priority should be driven based on feedback: remember the 80/20 rule

80 percent of customers only use 20 percent of the features in the software they’ve bought.

BTW: the database schema has changed a lot - with the addition of multiple use cases. Which also needs version management 😉 And evey new use case discussion now has two parts to be discussed - some problems are good to have (read, tradeoffs).

Ref: Feature Toggle

Side Cars

Sat, 01 May 2021 00:00:00 +0000

Idea

Add additional functionality into an existing application, with minimal or zero code changes to the exiting application. Also, the package can be consume by any software - irrespective of the language or framework (heterogeneous) it is written in.

How?

Using side cars pattern - : I/O( ZMQ) + Topic (intent) + Data( ProtoBuf) + Impl(Subscriber of topic).

DAPR framework from Microsoft Azure team – uses a similar design - runs side car. Dapr is a portable, event-driven runtime that makes it easy for developers to build resilient, micro service stateless and stateful applications that run on the cloud and edge and embraces the diversity of languages and developer frameworks.

We could use the Side Car for adding functionalities like resource monitoring, logging, log aggregation, configurations, observability (telemetry /feature trends) etc.. Once we get the sidecar installed - we could keep adding new functionality as it independently developed and deployed. The features in the sidecar could be turned on or off based on options.

Say as an organization: we decided to use cloud solutions for telemetry and internal tools decide to push data to the cloud - this solution might need a security layer - which would implement both gov and corp policies. This common implemenation of policies could be sitting inside the side cars and can be deployed on to all machine where the internal tools runs. And any update made to the policies would be transparent to the application - as the side cars would be independently deployed and maintained.

How is it different from a Microservice with REST API?

Microservice doesn’t really care about the client. But here the side car is tightly coupled and tracks everything about the main application – including say: killing the application if the resource used is above the threshold - so that the tool PC doesn’t require a restart.

Where should it be used?

Only for non-critical operation - as the latency will be high.

Reference:

MongoDB - Hybrid Schema

Sun, 14 Mar 2021 00:00:00 +0000

I recently ran into a situation where in we have a session and each Session produces a lot of messages (sync) and each message has an associated timestamp and order. This is similar to a blog post (session) and its comments (messages).

Since the messages are unbounded (not sure how many would be produced). In my initial design - I had a separate collection for Messages. Each Session would have one corresponding MessageW (wrapper) document and each MesssageW document would have an array of messages (acutal messages).

With all the available data I had - the size of the MessageW document never went above 1MB - so we were all good. But unfortunately, we had a new use case which broke this assumption 😟. The session was running for hours and there was tons of messages getting generated and yes - we hit the 16MB ceiling 😓.

Obvious solution was to create to create a new one doucment for every new message. So, that we never hit the size limit (16MB) of a mongoDB document. But, since we are looking at unbounded message - there could huge (million message docs) number of documents.

So, we choose to go with the Hybrid Approach

In the hybrid approach, we go with our initial option - where we have a MessageW (wrapper) document - which has an array of Messages. Then we add a constrain on the document i.e. the size of the array should not be more than a given arbitary number. For every new message we increment the count. If the number count crossed the constraint - then create a new MessageW document, inc the page number and also update the Session (NumberOfMsgPages). The reader would get the NumberOfMsgPages and read backwards - until he hit page 0. Voila!! 😍

Important Items to note:

Session Object : will have a NumberOfMsgPages property
MessageW Object: would have a Page and Count property.

Operators:

Since: A code is worth a thousand words. Here is the code 😊

 public void InsertMessage(Message message, ObjectId parentId)
{
var builder = Builders<Message>.Filter;
// Find MessageObject: given parentId and CurrentPageNo
// And: Constraint: Number of messages is less than 1000.
var filter = builder.Eq("parentId", parentId)
& builder.Lt("Count", 1000)
& builder.Eq("Page", CurrentMessagePageCount);
// Try and Insert into existing Message Document array And Increment the Count.
var update = Builders<Message>.Update
.AddToSet(x => x.Messages, message)
.Inc("Count", 1);
// If the constraint with "Count" matches.
// ModifiedCount would be greater than Zero
var result = MsgCollection.UpdateOne(filter, update);
if (result.IsAcknowledged)
{
if(result.ModifiedCount== 0)
{
// Increment the Page count
CurrentMessagePageCount += 1;
// Update: NumberOfMsgPages in Session Collection
var sessionFilter = Builders<MySession>.Filter.Eq("_id", parentId);
var updateSession = Builders<MySession>.Update
.Inc("NumberOfMsgPages", 1);
SessionsCollection.UpdateOne(sessionFilter, updateSession);
// Find MessageObject: given parentId and CurrentPageNo-which is obviously not found.
// So, we will use "IsUpsert=true" to add a new document.
// And: Number of messages is less than 20.
var filter2 = builder.Eq("parentId", parentId)
& builder.Eq("Page", CurrentMessagePageCount);
// Try: Insert into existing Message And Increment the Count
var update2 = Builders<Message>.Update
.AddToSet(x => x.Messages, message)
.Set("Page", CurrentMessagePageCount)
.Inc("Count", 1);
MsgCollection.UpdateOne(filter2, update2, new UpdateOptions() { IsUpsert = true });
}
}
}

A obvious reaction after reading the code is - Ok, whats the big deal here?

Lets take a code walk through: (think how would you achive this in your traditional SQL DB’s)

Code never check if the MessageW document exits before Update.
- How did it work: “$IsUpsert=true” is the magic word.
- $IsUpsert=True: MongoDB would internally insert a new document - if the document doesn’t exists.
Updating counter is happening inside MongoDB:
- $inc mongoDB operator - increments a specific field - in our case count
- Hence, Client doesn’t have to read the document to know the actual count and then increment/update.
Constraint Check:
- $lt MongoDB operator would handle the constraints.
- Update goes through if the constraint is met.
- As part of the result, we could check if any doc was modified
- Cool part - the second half of the code runs only once in 1000 inserts

How did we come up with the arbitary constraint number? well, I dont think we have a thumb rule here. Since we don’t track the actual document size - you would have to look at your data and come up with the sizing.

BTW, I am new to MongoDB and learning. If there are better approaches - feel free to write back to me. Thanks.

Reference: MongoDB Applied Design Patterns, by Rick Copeland

MongoDB Tit Bits

Sun, 07 Mar 2021 00:00:00 +0000

While I am getting started with MongoDB for my new project. Here are the few quick notes - based on my readings so far:

One Key Takeaway:

Data accessed together should be stored together

WiredTiger:

WiredTiger is the default storage engine for mongo db
It stores documents and indexes on disk
In memory cache stores some doc and frequency used index - working set
- 50% of (RAM -1 GB) Or, 256MB

Massive Arrays:

Max Document size is 16MB
Index performance on arrays decreases as array size increases
Extended Reference pattern: where we duplicate some and not all data

Index:

Each index is atleast 8KB
Index take up storage - One File for each collection and one file for each Index (WiredTiger impl)
Write performance as index needs to updated
Limit each collection to 50 Index max
Do: Add index for frequently supported queries - improves read performance
Don’t: Create unnecessary indexes - reduce performance and takes up space

Bloated Documents:

Do: Data accessed together should be stored together
Don’t: bloat your document with related data that is not accessed to gether
Data that is related to each other should NOT necessarily stored together
Remove bloat from frequently used documents - it can be inmemory wire tiger cache
Data Duplication is OK (depends!)
- Summary document and Details document - ref via links

Case Incentive Query:

$regex queries are case insensitive but not performant
Non-$regex queries are case sensitive
Collation:
- Language specific rules for MongoDB for string comparison
- Strength ranges from 1-5
- Strength 1-2 will give you case insensitive
- Query: {“first_name”: {$regex: /Jacob/i }} - case insensitive
- Query: {“first_name”: “Jacob”} - case sensitive

Accessing - Separate data together:

$lookup:
- Is used to join data from more than one collection
- Great for rarely used queries or analytical queries (batch run overnight)
- Very slow and resource intensive

Upsert:

Upsert is a combination of update and insert. Upsert performs two functions:
- Update data if there is a matching document.
- Insert a new document in case there is no document matches the query criteria.
Personally: I found this very interesting, as for my use cases - I dont need to track if the document already exist.

I understand - the blog looks a bit navive (read, nothing interseting), sry I am just getting started. BTW, I plan to write a follow blog - deep dive on my specific use cases and data modelling.

Reference:

Docker - DotNet 4.8 - Getting Started

Wed, 30 Dec 2020 00:00:00 +0000

Source Code: docker-try-net48

Goal: Develop a .Net48 application - coding would happen in the host while its run inside the docker - live

Prerequisite: Docker Desktop for windows is up and running and you are using Windows Containers

There are tons of documentation online, this is just me getting started.

Base Image:

There are two base images supported officially by Microsoft for .Net windows application.

Windows Server Core
Windows Nano Server

In this example: we are gona work on a .Net Framework 4.8 - console application. We would be using the DotNet Framework 4.8 image which internally uses servercore:ltsc2019-amd64 as base image.

Let first pull the image, into our local image cache. This would take a while as the size is huge - get yourself a coffee.

$docker pull mcr.microsoft.com/dotnet/framework/runtime:4.8

Once pulled (image is available in your local cache) - you are gona see an entry with Tag 4.8 in image list

$docker image ls

Sample App:

Here is a sample app to demostrate the usage. Clone the repository. Open the solution jacapp.sln in VS2019 and compile it. Its a simple app which starts two threads and exits after 10 seconds

One thread - print the count and the thread ID - to the console every second
Second thread - write the same count and thread ID - into an output text file every second

note: the output file would be on a volume mounted drive with the host

Dockerfile:

Let’s look at the contents of the dockerfile:

Specify Base image we plan to use - in our case we are gona use the image (4.8 Runtime) we have already downloaded (pulled)
Specify the Working Directory where all the commands would be run. Think of it as running $CD /d c:/myapp, before the EntryPoint is exectuted
Specify the Command to be executed inside the working directory. In our case its the console app, so we run the output exe jacapp.exe

Build Image:

Let’s build a docker image. Run the command from the directory which has the Dockerfile

$docker build -t testapp .

Run Image:

Items to note:

–name: name of the container image instance
-v: volume mount; here we mount the output “bin” directory with a specific directory inside the container “c:\myapp" .
-rm: remove the container instance once the program exits

$docker run --name jacinstance1 --rm -v D:\dev\dkr\docker-try-net48\jacapp\bin\Debug:C:\myapp\ testapp

Instance of the containter running:

Console output:

Contents of the file writted on the mounted drive:

Well thats it! We have our application running inside a container - Now we could edit your project (try updating the console output text), compile and directly docker run the image i.e. NO docker image re-build required!!

Code Reviews

Sat, 10 Oct 2020 00:00:00 +0000

While working in a development team; being part of the code review is not an optional thing. Depending on the size of the org/team and the # of features in parallel - I am sure, we will get multiple code review requests in a week.

I often ask myself, can I provide valuable feedbacks on every review I am part of? Most certainly NO! lets be honest here.

Then the next question is - why are you added into the review? well I could think of two reasons:

FYI: Let the team know I made this change; there could be depenencies (another developer; working on similar files)
Status update (stakeholders)

Not an active reviewer:

If you think, you don’t have the right context to contribute. It’s good to be explict and call out - Sorry, I don’t have enough context to add value. I would leave the review to the experts and recommend the author to mark you as Observer instead.

Note: code review is not a place to train; where you discuss and use it as a learning form.

Add Value:

This is critical and most important objective of a code review. Pointing out each and every spelling mistake and intendation fix - might not add much value; in which case - propose them to run it through a FxCop/style/code analyser utility.

You absolutely need to be aware of the context before providing feedback 👀. If you are not sure about the specifics, setup a meeting and request the developer to walk you through. At the end of the day - code is the ground truth (irrespective of what was aligned). Few example of real Value could be:

Pointing out specific use cases not handled
Usage of design principles (SOLID) - coupling/cohesion etc
Adding: Validation, Logging, Tracablility etc…
Handling: Errors, Exception, Leaks, Performance etc..
Code Coverage, appropriate tests (note: test should also be as clean/maintainable as SUT)

Assumptions:

This could at times come back back and bite; I have been there 😕. If you are not sure about a specfic change again: call out explicitly and confirm - if your understanding is accurate.

As authors - we do want our code to be non ambiguous. But, since there is always a context and depenency in the code - its hard to describe everything. Maybe better naming could help? well: 😏

There are only two hard things in Computer Science: cache invalidation and naming things.– Phil Karlton

No Condescending words

Thanks to Doctor McKayla for this.

Words such as “just”, “easy”, “only”, or “obvious” can come across belittling and condescending. It’s a good practice to remove those words from your feedback. Most of the time, they do not add any value….. research shows that people have a tendency to interpret written language in a negative way.

Personally, I am cautious to ensure my review comments looks like a conversation - than a direct command.

More the number of Reviewers:

I remember receiving emails like “please review my change set” sent to a DL which had 30+ developers. Do you think all 30 members would have all the required context or would you ensure all 30 will follow up and get the context and then do the reivew? Less likely! 🤔

Maybe there is a perception that more the reviewers; more the # of valuable feedback. But, what if its the other way - every one assumes its someone else responsiblity - zero feedbacks 😟.

Separate Feature Addtion and Refactoring:

Many at time its intriguing to refactor the code while we are working on a feature. We should - its always a good practise to clean up and make it better.

My only argument here is: If you have both - the feature implemetation and refactoring in the same changeset (esp on a large code base, spanning files/dependencies) - It becomes a challenge to follow the actual functional change.

Recomendation: split them into two changeset like StackedPullRequest: Personally would prefer (2) followed by (1)

Only refactoring. No feature addtion
Only feature addition

Unsure of the reviewers?

If you are already part of the team which maintains the code; no brainer. You are here since you are not sure - simplest tip Go for history. I have done this more than once; say making a change in deploy scripts maintained by operational team. Look for atleast 2 developers based on change history (again: don’t stop with one) of the file(s); secondly: make sure you look at their org structure and see if they belong to the operational team (If I get to deliver and someone else sends me a new review request 😑); finally: just to be sure add their solid line into it (why: well, you are unsure if you would get a response 😜).

Approval Enforced:

Few companies have strict code review rules like: only a pull request that has 2 or more approvals - would be merged into mainline. I feel this culture/process is awesome; as there is definetly a sens of responsibility and ownership on the part of the reviewers - at the end of the day again (would say it multiple times) “Code is the Ground Truth” (irrespective; what ever the feature/design document says).

Conclusion:

I personally belive, Code review are not optional. Anyday - another pair of eyes would give you a different prespective. While, we as developers are more focused on getting thing done, there are chances of scenarios getting overlooked - effective code review enable us to bridge this gap. This defintely requires commitment and trust between authors and reviewers.

Building a culture toward delivering quality code with code review as pivot is critical.

Check out: Doctor McKayla; there are lots of good content on this topics.

Apache Airflow and Regression Monitoring

Fri, 02 Oct 2020 00:00:00 +0000

Introduction:

Airflow is a platform to programmatically author, schedule and monitor workflows or data pipelines. It was originally developed and open sourced by Airbnb, later joined Apache Software foundation’s incubation program in 2016. Workflow is a sequence of tasks defined around Directed Acyclic Graph(DAGs) – which could be started on a schedule or triggered by an event or using Command line interface. Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiate pipelines dynamically.

Components:

Metadata DB:

Stores information’s like job status and task instance status.

Scheduler:

Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. The scheduler is the brains behind setting up the workflows in airflow. The execution time begins at DAG start date and repeat every schedule interval.

Web Interface (UI):

Airflow ships with a Flask app that tracks all the defined workflows and lets you easily change, start or stop them. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues.

CLI:

Airflow has a very rich command line interface that allows to test, run, backfill, describe and clear parts of your DAGs

Concepts:

DAG:

A DAG is the container that is used to organize tasks in a way that reflects their relationship, dependencies and set their execution context and order.

Operators:

Operators are the worker that run the tasks. Workflows are defined by creating a DAG of operators. They are broadly classified into three – Sensors, Operators and Transfers. Airflow provides many prebuild operators for many common tasks and new operators can be created by inheriting BaseOperator class.

Tasks:

Once an operator is instantiated, its is referred to as a “task”. Each task is user defined and responsible for performing a specific operation in the workflow. Instantiating a task requires providing a unique task_id and DAG container. Task can be python function or external scripts that could be invoked.

Example:

In the example, we show case - how Airflow could be used to express a workflow that can be used to generate the statistics/ report as part of end-to-end regression test suit; which involves multiple systems to work together. A traditional approach would use something very basic like bunch of batch scripts w/o CRON. But the challenge is - it would very easily get tangled and developer would spend a lot of time to figure out where the log files are or what failed and why/who owns what. Airflow helps solves this problem by helping in orchestrating your processes, managing the logs and really good dashboard with visualization of what failed and much more information.

References:

Python Plugins with Topics

Sun, 13 Sep 2020 00:00:00 +0000

Source Code: PyPlugins

Any infrastructure should always have the capability to extend itself. It would be better if the functionality is added by a contributor - who is not part of the core team. And it doesn’t get in the way of core components - development, compilation and deployment.

The concept of plugin has been around there for quite a while - Visual Studio/VS Code all has plugins (aka extensions). Basic idea here is to add new functionalities - by just deploying a new dll, jar or py modules.

In this proposal using python, we have a root folder named Plugins - the infra would enumerate the Plugins directory to add functions. The functionalities register themselves with a Key - let’s call them TOPICS.

Well why Topics? I am borrowing this idea from Messaging Queue infra like ZMQ and Kafka… So, that we could create a Topic to Function mapping - and we would be able to map caller to a MQ subscriber.

Let me walk through each component:

Base class

Base class that each plugin must inherit from; this class exposes couple of items 1) List of Topics 2) Execute Method - which your plugin should implement.

class IPlugin(object):
def __init__(self):
self.description = 'UNKNOWN'
self.topics = []
def execute(self, topic, argument):
"""The method that we expect all plugins to implement. This is the
method that our framework will call
"""
raise NotImplementedError

Example Plugin : Calculate

Calculate Plugin - exposes two functionalities: add and subtract. For the client its exposed as two topics. The execute function takes in topic and the argument. Based on the topic the respective plugin could dispatch it to sub-functions within the plugin.

class CalculatorPlugin(IPlugin):
def __init__(self):
self.description = 'Calculator'
self.topics = ['Add', 'Subtract']
def execute(self, topic, args):
if topic == "Add":
return self.add(args)
raise Exception (f'Topic: {topic} has no mapping function')
def add(self, args):
count = 0
for index in range(0, len(args)):
count += args[index]
return count

Service Discovery:

The infrastructure would enumerate the plugin base_directory and try find sub class of IPlugin. Create instance of the sub_class and register to the store: MAP<topic, instance>. Infra should be able to directly call the execute API, on the instance.

class ServiceDiscovery(object):
def __init__(self, plugin_package_dir='plugins'):
self.plugin_package_base_dir = plugin_package_dir
self.plugin_topic_instance_map = {}
self.enumerate_packages()
def enumerate_packages(self, package):
"""Recursively walk the supplied package to retrieve all plugins
"""
imported_package = __import__(package, fromlist=['foo'])
for _, pluginname, ispkg in pkgutil.iter_modules(imported_package.__path__, imported_package.__name__ + '.'):
if not ispkg:
plugin_module = __import__(pluginname, fromlist=['foo'])
clsmembers = inspect.getmembers(plugin_module, inspect.isclass)
for (_, c) in clsmembers:
# Only add classes that are a sub class of Plugin, but NOT Plugin itself
if issubclass(c, IPlugin) & (c is not IPlugin):
print(f' Found plugin class: {c.__module__}.{c.__name__}')
cls_instance = c()
for topic in cls_instance.topics:
print(f' Registering Topics: {topic}')
self.plugin_topic_instance_map[topic] = cls_instance

Execution

Now that we have a Map<Topic,instance>. When a call comes in - it would have a topic and the args. Using the Map, we could get the corresponding instance and call by passing in both the topic and args. This is similar to delegate (C#) or function pointers(in C).

class ServiceDiscovery(object):
def __init__(self, plugin_package_dir='plugins'):
...
self.plugin_topic_instance_map = {}
def execute(self, topic, argument):
if topic not in self.plugin_topic_instance_map:
raise Exception (f'Topic: {topic} is not registered')
return self.plugin_topic_instance_map[topic].execute(topic, argument)

Unit Test:

Writing unit test is not optional. Well, I am a fan of TDD 😉

def test_cal_plugin_add_func(self):
# Arrange
ser_dis = ServiceDiscovery()
# Action
result = ser_dis.execute("Add", [1, 2])
# Assert
self.assertEqual(result, 3)

Deployment:

Adding a new plugin should be as simple as

Copy and paste a new directory under the Plugin base directory
Directory should have a class which implements IPlugin

Conclusion:

Building a comprehensive plugin infrastructure is non-trivial; look at Visual Studio - you could override pretty much anything, starting from adding intellisese to a new compiler tool chain. Here, we are just look at a small tip - to get started - on having a python based plugin. Always starting off any infra project with the idea of extension in mind - is good to ensure cleaner responsibility separation.

From the SOLID principle : O -> our software should be Opened for extension but closed for modifications 😍

Version control your install directory - GIT

Sun, 05 Jul 2020 00:00:00 +0000

What problem are we trying to Solve?

Context: Desktop software installed on client computer, inside a private network and no access to internet.

Use Cases:

Shipping and applying multiple patches for a given version of the software.
Reverting a patch
Keeping track of patches installed
Check for sanity of the installation and tack manual edits (configuration files etc..)
User can apply a temporary patch on a server (i.e. common machine) and test, once done revert

How are we trying to solve?

Bring the content of the install directory under version control. We will be using GIT version control, to explain how it could be achived.

PreRequisite “git” should be installed and available in the path

Why use GIT?

Git is a distributed version control system designed to handle everything from small to very large projects with speed and efficiency.

So, with Git any local folder can be converted into a git repository by just running the init command.

D:\Foo\1.0.0>git init
Initialized empty Git repository in D:/Foo/1.0.0/.git/

Now your installed directory is version controlled. You can add files, commit, create branch(s), switch branches etc… Since Git is distributed - N0 Server nor Internet required 😄

At any point of time, you could use status to check if the install directory has any modified, new or deleted files.

D:\Foo\1.0.0>git status
On branch base_install
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: config/params.toml
Untracked files:
(use "git add <file>..." to include in what will be committed)
config/menus.toml
config/languages.toml

You could use git diff command to compare changes across two branches like base_install vs patch_1

D:\Foo\1.0.0>git diff base_install patch_1
diff --git a/Config/params.toml
index b32de60..e8e1694 100644
--- a/Config/params.toml
+++ b/Config/params.toml
@@ -29,7 +29,7 @@
- main_menu = {align = "l", show_logo = true}
+ main_menu = {align = "l", show_logo = false}

Creating a branch is a trivial operation. As a user you could

Create a new (personal) branch from current (master) branch
Apply the patch files (edit or new) -> Commit your changes to your personal branch
Run your tests
Once you are done, switch to the original (master) branch

If you observe - you don’t have to delete your patch - at any point of time when you have server time you could come back and switch to your branch…

D:\Foo\1.0.0>git checkout jac_logo_patch
Switched to branch 'jac_logo_patch'

Installer workflow and Scripts sequence…

POST INSTALL

Installer should copy this file into the root of destination directory and execute
Example: “D:\Foo\«_Build_Number_»”
Run this file post installation

Steps:

Initialize git.
Create a .gitignore file.
Add contents into .gitignore.
Create a branch base_Install.
Commit all contents into git repo with msg Base Install Commit.

@echo off
echo Initializing Git
call git init
echo Add git ignore files
call touch .gitignore
echo Add files to .git ignore for installer cmd files
call PostInstall.cmd >> .gitignore
echo Add all the files and create a initial commit named Base Install. This would take a while!!
call git add -A && git commit -m "Base Install Commit"
echo Creating a new branch base_install
call git branch base_install

PATCH INSTALLER

PRE INSTALL

Copy this file into the root of destination directory - D:\Foo\«Build_Number»”

Check if branch exists
Add all existing UNTRACKED FILES into the current branch
Add add a commit message PrePatch_<PATCH_NAME>
Add the current branch to a known file LastKnownWorkingBranch; useful for un-installation
Create a new branch PATCH_<PATCH_NAME>
Switch to new branch with PATCH_<PATCH_NAME>
Apply the patch using git am. Note: am would allow you to sign off an applied patch. This maybe useful for future reference.
Add all existing UNTRACKED FILES into new/current branch i.e. PATCH_<PATCH_NAME>
Done - we have a new branch with all patches

REM TODO: This could be an enviornment variable
set patchName=Patch_1
set branchName=Patch_%patchName%
echo Check if git installed
call git --version
if ERRORLEVEL 1 (
echo Git not installed!!!
goto :Cleanup
)
echo Check if valid git repository
call git rev-parse --is-inside-work-tree
if ERRORLEVEL 1 (
echo Not a valid git repo : %cd%
REM TODO - Should we create a new git repo? or fall back scripts to use .keep solution
goto :Cleanup
)
set commitMsg="PrePatch_%patchName%"
echo Add all un-tracked files and create a
call git add -A && git commit -m %commitMsg%
echo Add the current branch name into a temporary file - LastKnownWorkingBranch
call git symbolic-ref --short HEAD > LastKnownWorkingBranch
echo Check if branch exits
git rev-parse --verify %branchName%
if ERRORLEVEL 0 (
echo Branch exists : %branchName%
)
echo Create a new branch
git branch %branchName%
echo Checkout the new branch
git checkout %branchName%
echo Apply the .git patch
git am --signoff < %patchName%.patch
:Cleanup
echo exit

User can always go back to the base install by switching branches.

POST INSTALL

Steps

Now we are in the new branch for the Patch
And install script has already installed all the patch files
Just commit all the changes and we are good to go!!
Patch installation done!!

REM TODO: This should be an enviornment variable
set patchName=Patch_1
set branchName=Patch_%patchName%
set commitMsg="Patch_%patchName%"
echo Add all un-tracked files and create a
call git add -A && git commit -m %commitMsg%
echo Patch installed and avaialble in branch %branchName%

PATCH UN-INSTALLER

Steps

Read the lastKnownWorkingBranch from a file
git checkout lastKnownWorkingBranch
Voila thats it!!

Note: File “lastKnownWorkingBranch” file should be under .gitignore (?) so that its shared across branches and its updated only by the patch installer.

for /f "delims=" %%a in ('type LastKnownWorkingBranch') do (
set lastKnownWorkingBranch=%%a
break
)
echo lastKnownWorkingBranch: %lastKnownWorkingBranch%
echo Checkout the lastKnownWorkingBranch: %lastKnownWorkingBranch%
git checkout %lastKnownWorkingBranch%
echo Voila we have reverted the previous install

ADVANTAGES

User could compare what has changed across branches aka patches
User could always go back to base_install at any point of time
Files are never deleted
Uninstall is just switching a branch - no file deletes.
Any number of branches can be created for personal testing.

Book - My Good Reads, Technical

Sun, 14 Jun 2020 00:00:00 +0000

Podcasts

Hanselminutes, by Scott Hanselman
Data Engineering, by Tobias Macey
Talk Python To Me, by Michael Kennedy
Python Bytes, by Michael Kennedy and Brian Okken
.Net Rocks, by Carl Franklin and Richard Campbell

In Progress

MongoDB Applied Design Patterns
Designing Data-Intensive Applications, by Martin Kleppmann

Done

Big Data: Lamda Architecture, by Nathan Marz and James Warren
Docker Deep Dive, by Nigel Poulton
Chaos Engineering by Casey Rosenthal
C# in Depth, by Jon Skeet
Pro Git, by Scott Chacon and Ben Straub
Clean Code, by Robert C Martin
Clean Architecture, by Robert C Martin
Working Effectively with Legacy Code, by Michael Feathers

Tech - My Website

Wed, 20 May 2020 00:00:00 +0000

I registered my domain name on GoDaddy couple of years back (2018). I can’t really remember the motivation to get one registered. I almost forgot about this, until I got a reminder to renew.

Thanks to Scott, I realized the importance of key stroke - the nubmer of keystroke in your life time are finite. So if you worked on something interesting - why waste it in an email which you are never sure the recipient may or not read. Instead, write it in a blog and it would be there forever and could potentially help someone new in future.

On hindsight - this website should have ideally taken less than half a day to setup, but I ended up spending almost 2 days to set the first version. Well I was just getting my hands dirty.

This is a static website:

Original Content on GitHub - my private repo
Static files Generated using HUGO Extended
Themes from Academia
Static files Hosted on AWS S3
Domain DNS AWS Route S3
Continous Deployment Github Actions

Godaddy

This is a very trivial step, go to GoDaddy. Search if your expected domain name is available. If yes, just follow the steps that show up. Note: There are many providers GoDaddy is just one of them. I dont have a good reason why I chose GoDaddy or I haven’t seen a good reason to try other providers (read, lazy me).

Hugo

Hugo again has extensive documentation (bit overwhelming at times) on setting up. Every theme you donwload come with an example site, all you need to do is just start deleting content.

One item I really like is the live reload - when hugo server is running (locally) - the website would be auto deployed and page reloaded with new content as and when you save your file.

Tip: Theme I downloaded was using Hugo Extended and not the default one. It took me almost half a day to figure this out :-( debugging through css generation failures etc..

AWS Configuration

This article walks you through pretty much all the steps. If you are getting started, recomend to skip the cloundfront part: how-to-host-your-static-website-with-s3

Continous Deploy (Actions)

This was my favourite part, I am personally a CI/CD enthusiast. I wanted to deploy my website every time I push changes into my GitRepo.

I used S3 Sync Actions from marketplace. It has a few additional steps like generting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to get it configured.

Important: Secure you keys, by setting it up as secure env variable in github - so that it doesn’t show up in console log

Use my icon for the website: Every theme comes with its own default icon. Since I have referenced the theme as a GIT Submodule - as part of CD/workflow build is was always overwritten. Did a minor hack - I checked in my icon into my repo and as part of build replaced the default theme icons with mine before build step.

Below is the GitHub Action I stitched together.

 name: CD
on:
push:
branches:
- master
jobs:
deploy:
name: deploy-website
runs-on: ubuntu-latest
continue-on-error: true
steps:
- name: Checkout master
uses: actions/checkout@master
- name: Update theme
run: git submodule update --init --recursive
- name: Setup hugo #https://github.com/peaceiris/actions-hugo
uses: peaceiris/actions-hugo@v2
with:
extended: true #This website is based on Hugo-Extended
hugo-version: "0.70.0" #We could also use latest
- name: copy my icon
run: cp -f ./assets/images/icon.png ./themes/academic/assets/images/icon.png
- name: Build Website
run: hugo --minify
- name: Publish To AWS S3
uses: jakejarvis/s3-sync-action@master
with:
args: --acl public-read --follow-symlinks --delete --exclude '.git/*'
env:
AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_REGION: 'ap-south-1' # optional: defaults to us-east-1
SOURCE_DIR: 'public' # Req: hugo gen - static files into this dir