Jacob Aloysious

An AI-First Second Brain for Engineering Teams: My Talk at KLA

Sat, 26 Sep 2026 00:00:00 +0000

I had the opportunity to speak at KLA‘s Engineering Conference, an event attended by 800–1,000 engineers, about a problem that sits underneath much of our engineering work: remembering enough of the right context to make the next decision well.

KLA serves the semiconductor industry with process-control, inspection, and metrology technologies. That made it a fitting setting for a conversation about preserving context across complex engineering work.

My talk, “AI-First Second Brain for Engineering Teams,” explored how AI can help preserve that context without turning every conversation, incident, or provisional idea into permanent truth.

I had a 90-minute slot. The talk itself lasted 75 minutes, followed by more than 30 minutes of Q&A, taking the session beyond its allotted time. There was a lot of enthusiasm for the topic, and it was encouraging to see that interest continue into the discussion.

The message I wanted to leave with the audience was simple: AI can do much of the work of organizing and maintaining memory. People still need to decide what is safe to share, what deserves to be remembered, and when an old lesson applies to a new situation.

On stage at KLA’s NEOTERIX 2026 Engineering Conference in Chennai.

The work before the work

Think about a difficult engineering issue. The logs are in one place. Someone else has the physical observation. The reason for an old design choice is buried in a discussion. A workaround exists in the memory of an engineer who happens to be unavailable.

Before anyone can investigate properly, the team has to reconstruct the story.

I described this in the talk as engineering archaeology. We are searching for what happened, who understood it, which assumptions mattered, and whether the previous solution is still relevant. The information may exist, but it is not yet a usable starting point.

That reconstruction carries a memory tax. Engineers pay it when debugging an unfamiliar failure. Managers pay it when recovering an unowned commitment. Leaders pay it when trying to understand whether a team has learned something reusable or simply closed another issue.

The questions differ, but the dependency is similar: useful action waits for context. Adding more documents does not automatically reduce that delay. A larger collection can also make it harder to distinguish current guidance from an abandoned experiment.

What I mean by a second brain

For this talk, a second brain meant a maintained body of context that helps a team continue its work. It includes decisions and their rationale, incident history, runbooks, known limitations, and commitments that need follow-through.

The useful distinction is between storing information and maintaining it. A stored note can remain available long after its assumptions have expired. Maintained knowledge has a visible source, an owner, a verification state, and a reason to revisit it. It can be corrected or retired.

The recurring flow in the talk was:

Capture → Govern → Surface → Decide

Capture brings in relevant observations. Governance determines what may persist and who can use it. Retrieval surfaces the right material with its supporting evidence. A person then makes the decision with enough context to understand the consequences.

This also changes how I think about expert dependency. An experienced engineer’s judgment is valuable, but their availability should not be the only route to the team’s history. Shared memory can give another engineer the sources, reviewed guidance, and prior investigations needed to make progress.

It does not reproduce the expert. It preserves a better starting position for whoever is doing the work next.

Give AI the maintenance work

A large part of the talk concerned the division of responsibility. AI is useful for extracting candidate decisions, linking related cases, drafting summaries, noticing stale guidance, and bringing open questions back for review.

Those tasks consume attention, especially when context is scattered across messages, meetings, documents, and alerts. An assistant can prepare the material continuously, leaving people with smaller, more specific decisions to make.

In the personal workflow I described, Google Chat is the interaction surface, MyClaw is the agent runtime, and human-readable Markdown in Google Drive holds durable memory. The runtime coordinates tools, policies, and scheduled work; the underlying model is one component of that system.

The particular products matter less than the arrangement. People need somewhere to review and correct proposed memory. Durable records need to remain understandable outside a single chat session. Cases, decisions, and approved knowledge need different lifecycles.

A useful record should explain where it came from, who owns it, when it was checked, and what would make it questionable again. That gives the next reader a way to assess it instead of accepting a confident summary at face value.

Five loops that keep memory useful

I organized the maintenance work into five recurring loops. Together, they turn a collection of notes into something a team can keep using.

1. Capture decisions for review. After an architecture discussion, the assistant can extract the decision, its rationale, and the condition under which it should be revisited. A person can store, edit, or skip the proposal. Recording the reason matters: without it, a future team may preserve a decision after the original constraint has disappeared.

2. Assist with incidents. During an investigation, the assistant can surface a relevant playbook, its source, its verification state, and recent changes. The engineer chooses the next action. The assistant can preserve the checkpoint so the next handoff does not restart the investigation from scratch.

3. Draft weekly updates. Actual activity can become a draft of delivered work, ongoing work, risks, and next steps. Human review remains part of publication. This is valuable because an update should reflect what happened, including uncertainty, rather than merely sound complete.

4. Check freshness. Guidance should return for review when time passes, a dependency changes, or an original assumption stops holding. AI can identify those triggers. An owner decides whether the record remains valid, needs correction, or should be archived.

5. Propose reusable knowledge. Several cases may suggest a pattern worth preserving. The assistant can prepare a narrow proposal with evidence and limitations. A named owner reviews it before it becomes guidance for others.

These loops make maintenance a recurring responsibility. They also give the human reviewer a concrete question to answer at each step.

Discussing how AI can preserve context while keeping consequential decisions with people.

A resolved case is not automatically reusable knowledge

This was one of the most important distinctions in the talk. A fix can be correct for one incident and misleading when generalized.

A case records what happened under particular conditions. Reusable knowledge makes a broader claim: under these conditions, this diagnostic or action is worth considering again. Moving between the two requires evidence, scope, limitations, and ownership.

For example, a recovery procedure might have worked with one configuration. Before it becomes a runbook recommendation, someone needs to ask what actually explains the recovery, which prerequisites matter, and when the procedure should not be used.

The same applies to AI-generated summaries. Fluent language can hide how narrow the original evidence was. Promotion should therefore be an explicit action, with a reviewer responsible for the claim that future engineers will inherit.

Most cases can remain cases. Their history is still useful. Only selected lessons need to become shared guidance, and that guidance needs a path back to its supporting evidence.

Many cases, few promotions: reusable guidance needs verified evidence, a source trail, and an owner.

Making the boundaries concrete with a field-issue demo

I tailored the demo to KLA’s semiconductor engineering context: how AI could support an end-to-end workflow from a field issue to a tested code fix, with human approval at the consequential steps. The scenario is fictional and illustrates the proposed approach. The presenter application is a deterministic simulation backed by local artifacts and tests. It does not operate production equipment, call a live model, or deploy changes to an external system.

I see this as a practical example of forward-deployed engineering: working from a specific customer use case and shaping the AI solution around that customer’s environment. Here, the problem is moving from a semiconductor field issue to an actionable engineering investigation. The design follows the evidence available at the tool, the restrictions on sharing it, and the engineer’s responsibility for the fix. That customer context determines where AI can help and where human judgment must remain.

The automation workflow presented at KLA: raw evidence stays local, only an approved packet crosses the boundary, and an engineer reviews and ships the proposed fix.

The scenario follows evidence from a tool-side investigation to an engineering diagnosis. An on-tool agent prepares a sanitized packet while restricted raw evidence stays local. The packet must be reviewed and approved before it crosses the boundary to the engineering side.

One deliberate step is attempting the transfer too early. The system refuses it. That refusal makes the permission boundary visible: information being useful does not, by itself, authorize sharing it.

On the engineering side, the approved packet is combined with shared knowledge, code, and tests. The example concerns a settle gate that checks position while motion is still unstable. The workflow prepares a diagnostic experiment and a proposed runtime and test change. The engineer retains responsibility for reviewing and shipping the change through the normal process.

A second gate appears after the simulated resolution. The workflow refuses to promote the finding into reusable knowledge until an owner approves it. The later cold-start view shows what another engineer could inherit months afterward: a scoped diagnostic path, evidence, ownership, and review conditions.

This is the behavior I wanted the demo to make tangible. Better context can shorten the route to an investigation, while explicit gates preserve responsibility for sharing, software changes, and future reuse.

In a real implementation, those gates also require technical enforcement through credentials, isolation, network controls, validation, and audit records. A prompt alone cannot provide that boundary.

Memory about people needs its own limits

Engineering context includes people, and that requires care. Remembering an agreed follow-up can help a team. Turning an impression into a lasting judgment about a colleague can cause harm.

The manager workflow in the materials separates fragments into three outcomes: remember, keep uncertain, or reject. A professional commitment may be appropriate to retain. A conditional statement must keep its condition. A speculative personal judgment should not become durable memory.

Consider the difference between “I can review this if the release work finishes” and “I will review this.” Removing the condition changes the commitment. An assistant that compresses the first statement into the second has created an inaccurate expectation, even if its summary reads naturally.

That is why visibility and correction matter. People should be able to inspect the source of a remembered claim, correct it, and understand the boundary around its use. Useful continuity depends on restraint as much as recall.

Start with one decision that keeps losing context

My proposed starting point was a small, thirty-day pilot around one recurring problem. Choose a decision where the team repeatedly reconstructs history, then define what better continuity would look like.

A recurring incident, an architecture decision log, or a handoff process could be a suitable starting place. Give it one owner, a clear boundary around what may be stored, and a measurable outcome.

The build path in the talk moves through four stages: define the job and stop rules, wire the workflow and failure handling, ground retrieval and human review, then add ongoing ownership and operational checks.

The tests should cover the memory contract as well as answer quality. Does the workflow reject unreviewed material? Does it preserve conditional language? Can a person correct a record? Does promotion require approval? Can the system show where its guidance came from?

Measure whether the next engineer reaches a justified decision sooner, whether handoffs lose less context, or whether fewer investigations restart unnecessarily. Counting stored notes or generated summaries tells us little about those outcomes.

What I wanted engineers to take away

Speaking at KLA’s Engineering Conference was an opportunity to connect AI with a familiar engineering challenge: continuity across people, incidents, and time.

The promise of a second brain is practical. It can help a team recover the rationale behind a choice, resume an investigation, and reuse an earned lesson with its limits intact. Reaching that point requires ongoing maintenance and deliberate human ownership.

My closing question was: Where does your team pay its highest memory tax?

That is where I would start. Find the context that repeatedly disappears, preserve it with evidence and boundaries, and see whether the next decision improves.

The talk repository contains the slide deck, speaker notes, and demo materials for exploring the approach in more detail.

Bringing My Blog Back with Codex Computer Use

Sat, 26 Sep 2026 00:00:00 +0000

My blog’s source code was sitting in GitHub. My domain, jacobaloysious.in, was registered with GoDaddy. I had a Vercel account ready to use. Somewhere in the background, an old AWS setup was still involved.

The goal sounded simple: publish the blog from GitHub using Vercel and make it accessible through my own domain.

This time, I asked Codex to help do the work through Computer Use. Instead of following a deployment tutorial and switching between dashboards myself, I let it inspect the browser, configure the deployment, and trace the domain setup. I stayed involved when access or missing context required my input.

It turned into a useful exercise in both AI-assisted work and remembering how my own website was wired together.

1. Get Computer Use working

The first obstacle appeared before deployment even began. Although I was signed in to GitHub, Vercel, and GoDaddy in Chrome, Codex could not initially connect to my browser.

The Computer Use setting showed:

Disabled by your organization or unavailable in your region.

The message alone did not establish the cause. We checked the available setup guidance, but did not conclusively diagnose why the setting was disabled. I subsequently enabled Computer Use and asked Codex to continue.

Once access was working, Codex could inspect my existing signed-in Chrome tabs and interact with the account dashboards.

The first lesson: being signed in to a website and giving an assistant access to that browser are separate steps.

2. Inspect the existing Hugo project

My Hugo source lived in a private GitHub repository.

Codex inspected the project and its existing GitHub Actions workflow. That revealed several details that mattered for the new deployment:

The site used Hugo with the Academic theme.
The theme was included as a Git submodule.
The existing build pinned a specific Hugo Extended version.
A custom icon was copied into the theme before building.
The previous workflow published the generated public directory to AWS S3.

Reading the old workflow gave us a starting point for a compatible build. There was no need to combine a hosting migration with a theme or Hugo upgrade.

3. Import the repository into Vercel

In the Vercel dashboard, Codex opened Add New → Project, selected GitHub, and selected my blog repository.

Vercel already had access to the repository, so it could import the private project without making the source code public.

The project configuration was:

Setting	Value
Framework preset	Hugo
Root directory	Repository root
Source branch	The production branch
Environment variable	`HUGO_VERSION`, matching the existing build
Build output	`public` through the Hugo preset

We overrode the build command to preserve the existing theme setup. The build initialized the Git submodules, copied my custom icon into the theme, and generated the static site with the intended production URL.

A simplified example of the Hugo generation step is:

hugo --minify --baseURL https://example.com/

Use your own domain in place of example.com, and include any preparation steps required by your theme.

4. Deploy and check the actual website

The first Vercel deployment succeeded. The blog became available at its Vercel-provided address.

Codex then opened the deployed site and checked more than the deployment success message. The homepage displayed my biography, profile image, styling, and recent posts. It also opened the Leadership - Buy In | Meetings article and verified that its content loaded.

At this point, the hosting part was working. The Vercel project was connected to GitHub so future pushes to the production branch could trigger new deployments.

5. Add the custom domain in Vercel

Next, Codex added jacobaloysious.in to the project’s production environment.

We kept the address without www as the main domain. Codex also added www.jacobaloysious.in and configured a 308 permanent redirect to jacobaloysious.in.

Vercel supplied the DNS records needed for this project:

Type	Name	Destination
A	`@`	The IP address provided by Vercel
CNAME	`www`	The CNAME target provided by Vercel

The destinations are intentionally generalized here. Use the exact records displayed in your own Vercel dashboard.

6. Discover the old AWS connection

I expected the next step to be editing DNS in GoDaddy. Instead, GoDaddy showed that the domain’s DNS was managed by Amazon Route 53.

That explained why registering the domain with GoDaddy did not mean GoDaddy controlled its active DNS records.

The services had different jobs:

Service	Role in this setup
GitHub	Stored the Hugo source and content
Vercel	Built and hosted the website
GoDaddy	Registered the domain; later became its DNS provider
AWS	Previously supplied Route 53 DNS and the S3 deployment destination

Codex initially suggested updating the existing Route 53 records to preserve anything else using that DNS zone. I clarified that I no longer used the AWS setup and wanted the website routed to Vercel.

That context mattered: it allowed us to replace the old nameserver delegation instead of requiring access to AWS.

7. Move DNS management back to GoDaddy

Codex selected GoDaddy’s default nameservers and completed the confirmation flow, moving DNS management back to GoDaddy.

Once the GoDaddy DNS records became available, Codex made two changes:

Replaced the parked @ A record with the IP address supplied by Vercel.
Changed the www CNAME to the target supplied by Vercel.

GoDaddy confirmed that both records were saved successfully.

Changing DNS and retiring the previous hosting setup are separate tasks. A migration should also include a review of old deployment workflows, credentials, and billable resources.

8. Investigate Vercel’s “Invalid Configuration” warning

After saving the records, Vercel still showed Invalid Configuration for both domain names.

It was tempting to assume we had entered something incorrectly. Codex compared the saved GoDaddy records with Vercel’s required values and ran public DNS checks.

The records matched, but the public lookups were still reaching the old AWS nameservers. Those servers refused the queries, and the lookups returned SERVFAIL.

So there were two different states:

In GoDaddy: the new nameservers and Vercel records were saved.
In the public DNS checks: the old AWS delegation was still being used.

Vercel could not verify the custom domain while those lookups were failing. A browser check also failed to resolve the domain, so we could not yet verify HTTPS on the custom address.

GoDaddy advised that most DNS updates take effect within an hour, but global propagation can take up to 48 hours. Saving a DNS change is not the same as every resolver seeing it immediately.

9. Verify the domain and go live

After the DNS changes propagated, the domain was verified and the blog became accessible at jacobaloysious.in.

The completed setup:

The Hugo blog was deployed and working on its Vercel address.
GitHub was connected to Vercel for future deployments.
The main domain and www redirect were configured in Vercel.
GoDaddy had accepted the nameserver change and saved the required records.
The custom domain was verified and the website was live at jacobaloysious.in.

The earlier Invalid Configuration warning was an intermediate state during the migration. Once the DNS change took effect, the main goal was complete: readers could use my own domain to access the blog hosted on Vercel.

What I learned from using Computer Use

The most useful part was watching Codex carry context across several services. It read the old build workflow, applied those details in Vercel, discovered the Route 53 dependency in GoDaddy, and used DNS checks to explain the remaining warning.

It also exposed where my input was necessary. I had to enable browser access and explain that AWS was no longer part of the intended setup. The assistant could inspect the dashboards, but it could not infer that decision from a nameserver entry alone.

There were practical limits too. Browser pages needed time to load, some interactions needed another attempt, and a successful save did not guarantee an immediately working domain. Verification remained part of the job.

For me, this was a concrete example of AI helping with an operational task: taking an existing website, understanding enough of its history to deploy it, and working through the connections between source code, hosting, registration, and DNS.

My blog is now live at jacobaloysious.in, with the content in GitHub, hosting on Vercel, and domain registration and DNS managed through GoDaddy. The old AWS nameserver dependency has been replaced.

For future posts, the publishing flow is simple: update the content in GitHub, push to the production branch, and let Vercel build and deploy the site.

Leadership - Buy In | Meetings

Tue, 13 Feb 2024 00:00:00 +0000

Frequently, we find ourselves in leadership meetings, whether they are for making decisions or providing guidance. In such situations, I always remind myself or advise my colleagues to consider the following:

Establish the purpose and context upfront
Collaborate with the affected audience proactively
Be Open to feedbacks

Establish the Context and Purpose

Clearly express your purpose from the beginning; whether you’re seeking guidance, a decision, or simply engaging in a brainstorming session.

Strive for direct communication and don’t take offense at potentially blunt questions. These questions may be aimed at streamlining the presentation by skipping slides for which the audience already has context.

Leadership is often impatient, preferring a quicker pace than anticipated. It’s not crucial to reach the end of the slides; instead, be prepared for interruptions during the presentation.

Leadership’s time is incredibly valuable, underscoring the necessity of thorough preparation, especially with supporting data. When it comes to incorporating data into presentations, refrain from including them on slides simply because you possess them—use them as supplementary evidence.

Collaborate with the affected audience:

Certain proposals we put forth may have repercussions on other departments within the organization; for instance, implementing automation to address customer issues could potentially impact the operations team.

Expect challenging questions and feedback when delving into the territories or crossing boundaries of other groups. A more effective strategy is to collaborate with the affected group in advance, before the leadership meeting. Acknowledge the potential impact, and seek initial input on the proposed solutions. The objective is to prevent any confrontations during the leadership meeting, especially if an impacted leader feels offended by the proposal.

Be Open to feedbacks

Higher the hierarchical level, the more diverse the leader’s staff is likely to be. Depending on the specific audience, one might face differing challenges or potential criticism.

Be attentive to inquiries such as, “What problem are we exactly trying to solve?" This suggests that although the problem has been stated, there is a need for further clarification. Either the leader may not grasp the issue, or they may not perceive it as a problem. In such instances, avoid being defensive or revisiting the problem slides; instead, seek to understand the underlying intent behind the questions—simply ask.

Maintain openness—what matters most is resolving the problem. The solution doesn’t necessarily have to align with the one you initially proposed.

Covid Travel - India to SG - Dec'21

Wed, 08 Dec 2021 00:00:00 +0000

Chennai to Singapore: (Non-VTL AirIndia 0346)

Date: 06/Dec/2021 Monday

Documents Required:

Had to take a printout of all

Tickets
RT-PCR Test Result
Visa Doc
SG Arrival Card
SG Arrival Swab Test confirmation with QR code
Passport
Entry Approval

Chennai:

At the entrance - documents check
A new document from AI was given to fill in the visa details
AirIndia counter, again same set of documents check.
Complimentary Face-shield and Mask from AI; at immigration entry
Immigration - Standard - Only Passport/Visa check
You are expected to wear a mask and face shield in the flight

Singapore (SG):

No face shield required, but Mask is mandatory
Temperature check
Guided directly into immigration lobby
Immigration: Documents Check - All + Biometric and Photo
Guided to Swab Test
Verification of your Swab Test booking and barcode generated
Guided to Swab test counters - there were multiple of them (10+)
Take the Swab test
Guided: Exit the airport - Taxi stand or Arrival Pickup
Go to your SHN facility

Note:

SG airport was well organised and guided - Impressive 😍
No shopping is possible at SG airport
Toilets are accessible after immigration
At SG it took me less than 1hr
Install TraceTogether App and register/enable it after immigration

Recommendation:

Carry a Pen
Go early to the airport (4hrs) as the document check takes a lot of time.
Carry a face shield - personal the complimentary one wasn’t comfortable.
Ensure you book the SG Arrival Swab Test in advance

Links:

Swab Test Registration: https://safetravel.changiairport.com/en.html?source=ca#/

4 hours in the Airport: Me, My Tea and My Kindle 😆

Book: Born a Crime: Stories From a South African Childhood Trevor Noah

Almost empty airport - guess that was the only flight scheduled

Both Mask and Faceshield was Mandatory 😦

PKCE

Fri, 03 Dec 2021 00:00:00 +0000

Prerequisite: OAuth2.0

PKCE is pronounced as ‘pixie’. PKCE - Proof Key for Code Exchange => Credits

Concepts:

What is Confidential Client?

Any backend application written in languages like .Net, Java, NodeJS can be considered as Confidential Client. As they have the ability to “store variables” in our case Secret Keys - without exposing them to the end user - as the code is running on the server side. The secret key would be used to generate the Access Token in the OAuth Flow.

What is public Client?

Any Single Page Application (SPA), where all the code is downloaded to the users browser would be considered a Public Client. Coz: the secret key cannot be securely stored as the hacker could do a “view source code” and look at the secret key which is usually in plain text/string format.
Another example are Mobile Apps - the user can download and de-compile/reverse engineer - the application and find the secret key.

Why do you need PKCE?

Once the secret key is available - the hacker can impersonate as the Client and generate the access token (note: clientID is public and secret key is private). Once the access token is available - the hacker can access lot more confidential information about the user. So the existing OAuth2.0 model is not secure for both SPA and mobile apps. To solve this challenge - PKCE was introduced.

OAuth 2.0 Normal Flow:

PKCE Flow:

What is new in PKCE?

In the PKCE flow: Instead of a constant Secret Key - Everytime an OAuth flow is initiated - A new “Secret Key” is generated and Hashed.
Hash is shared with the Authentication Server(AS) when requesting for the Temporary Code. AS would remember this Hash value.
When the client request for the Access Token - it sends the newly generated “Secret Key” - the AS would check, if the secret matches the Hash (it already remembers - #2).

Reference:

RFC7636, Oauth Hack, What’s the difference between Confidential and Public clients

Designated Authentication - OAuth2.0

Thu, 02 Dec 2021 00:00:00 +0000

OAuth2.0 and OpenID - is one of the best videos (62 mins) on this topic - Highly recommended. This blog is just a quick summary (4 mins read) of it.

What is OAuth?

OAuth is a security standard where you give one application permission to access your data from another application i.e. you authorized one application to use data from another application on your behalf, without giving them your username and password. The steps to grant permission(s) or consent is often reffered to as authorization or delegated authorization. Permissions are scoped e.g. client can be given access to read your GMail contacts but cannot Read/Write your email. OAuth is not exactly used for user identification (who loggged in), instead its used to access API’s for resources like gmail contact list.

You check-in to a hotel.
At the front desk - You give your details like Passport and Reservation
Front Desk would in turn - give you the Key Card i.e. Access Token
You can use the Key Card to access your reserved Room

Here, the Front Desk is the Authentication Server - the key card is your Access token used to acess your Room (Resources API). The API doesn’t really care who is accessing the Resource as long as they have the access token.

OAuth Terminologies:

Lets assume you are giving Yelp access to your Gmail account contacts..

Resource Owner: Represent you the owner of the gmail account; who can give permission/authorize.
Client: Represent Yelp i.e. the application which would like to perform action on behalf of you.
Authorization Server: One who validates the gmail uname/password and authenticate - in this case accounts.google.com. Resource owner will already have an account with this server (gmail).
Resource Server: Server (API) which has the actual content/resources like contact list in this case contacts.google.com. Which client would talk to get the actual data.
Authorization Grant: Once authenticated an Authorizatiion Code or temporary code is sent back to he client. So, the client has been granted relevant access to the resource owners gmail contacts. This is short lived, this is used to inturn get the access token (long lived).
Redirect URI: Also known as Callback. It is the URI to which the response of Authentication #5 should be sent back to client #2
Access Token: Token/Key used by the Client to access all the resources of the owner; based on the authorized permissions.
Response Type: The type of information the client would like to receive e.g: code (authorization code)
Scope: Granular permission the client wants acess to like Read Contact, Read Profile basic info.
Consent: Whether the client can be given permission to access the requested information.
ClientID: Unique ID of the client, which is used to talk to the authorization server.
Client Secret: Secret Key - only the client and the auth server shares. Helps Client and Resource Server to share information secretly behind the scenes.

OAuth Flow:

Prerequisite:

Client and the Authorization Server should establish a working relationship. Both the “ClientID (AppID)” and “Client Secret(AppSecret)” are generated by the Authorization Server. The client ID would be used to uniquely identify the client. Its the responsibility of the client to ensure the “ClientSecret” is kept secret and its only know by client and authorization server. As there are the only means for the Auth Server to verify the client.

Actual Flow:

In this example, Yelp would want to authenticate a user based on GMail Authentication, and access Profile/Contacts information.

Client(Yelp) would make a request to Authenticate - to “Autorization Server” (accounts.google.com); as part of the request:
1. “RedirectURI” is set to a callback URL of the client “yelp.com/callback”
2. “Respose Type” is set to “Code”, so a auth code is sent back as response to 1.a
3. “Client ID” and “Scope” (Profile/Contacts)
Auth Server pops up a dialog (mostly) - where user gets to enter the GMail credentials (uname/pass) and authenticate
Auth Server shows up a consent form - list the info access requested (like contacts, profile) - user gets to choose Yes or NO
Once concent(Yes) is given - an “Auth Code” is generated and sent back to the the client - to specifiied callback/redirectURI.
Then, the client sends the “Auth Code + ClientID + ClientSecret” to “Authorization Server” and gets back an “Access Token”
Using the “Access Token”, Client can contact the resource server (contacts.google.com) to get the actual resources (profile, contacts) of the user.

Reference: https://www.youtube.com/watch?v=aU9RsE4fcRM

WebFinger

Wed, 01 Dec 2021 00:00:00 +0000

This blog contain just snips - a detailed explanation is available in the Reference links..

Background:

Twitter started the project called Bluesky with a goal to build a social web. As part of it, ActivityPub a decentralized social networking protocol was built - which uses WebFinger in the background.

What is WebFinger:

WebFinger is used to discover information about people or other entities on the Internet that are identified by a URI using standard Hypertext Transfer Protocol (HTTP) methods over a secure transport. A WebFinger resource returns a JavaScript Object Notation (JSON) object describing the entity that is queried.

Key Items to Note:

WebFinger protocol is used to request information about an entity identified by a query target (a URI).

A WebFinger request is an HTTPS request to a WebFinger resource.

A WebFinger resource is a well-known URI

A WebFinger resource is always given a query target, which is another URI that identifies the entity whose information is sought. GET requests to a WebFinger resource convey the query target in the “resource” parameter of the WebFinger URI’s query string

The path component of a WebFinger URI MUST be the well-known path "/.well-known/webfinger”.

The WebFinger resource returns a JSON Resource Descriptor (JRD) as the resource representation to convey information about an entity on the Internet

The use of HTTPS is REQUIRED to ensure that information is not modified during transit. Clients MUST verify that the certificate used on an HTTPS connection is valid and accept a response only if the certificate is valid.

The Registration Template:

Property Identifier:
Link Type:
Description:
Reference:
Notes: [optional] The “Property Identifier” must be a URI that identifies the property being registered.

Example:

Suppose Lia wishes to authenticate with a web site she visits using OpenID Connect. She would provide the web site with her OpenID Connect identifier, say lia@example.com. The visited web site would perform a WebFinger query looking for the OpenID Connect provider. Since the site is interested in only one particular link relation, the WebFinger resource might utilize the “rel” parameter

Request:

 GET /.well-known/webfinger?
resource=acct%3Alia%40example.com&
rel=http%3A%2F%2Fopenid.net%2Fspecs%2Fconnect%2F1.0%2Fissuer
HTTP/1.1
Host: example.com

Response:

 HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/jrd+json
{
"subject" : "acct:lia@example.com",
"links" :
[
{
"rel" : "http://openid.net/specs/connect/1.0/issuer",
"href" : "https://openid.example.com"
}
]
}

Since the “rel” parameter only serves to filter the link relations returned by the resource, other name/value pairs in the response, including any aliases or properties, would be reeturned. Also, since support for the “rel” parameter is not guaranteed, the client must not assume the “links” array will contain only the requested link relation.

Take ways:

The specification is rather simple as it uses JSON as the data model for all the communication. Almost, all languages provide support for handling JSON. Validation is still not part of the spec - So both validation and handling attack should be the responsiblity for the client.

References:

Engineering Manager - 5yrs

Sat, 27 Nov 2021 00:00:00 +0000

I have been an engineering manager since 2016 i.e. 5+ yrs, as I write this blog. More often, I have thought to myself as - what is my role as an engineering manager. This is just a brain dump and not an exhaustive list and not in any specific order.

Technical:

Solid understanding of the requirements and architecture - team deliverables
Reasonable overall breath and few focussed depth - understanding of the design
Ensure the testing stratergy (Automated and Manual) - has appropriate coverage
Quality: Follow best known practices of Software Engineering like Test Driven Development, CI/CD.

Talent: For each team member

Strength and weakness
Short term and mid term - development focus areas
Mentor or coach
Training Needs

Team:

Onboarding plan for new hires
Understanding of Cross functional teams - that your team colloborates with
Clarity on the ownership overlap between your team and cross functional team
Updated: skill and ownership matrix

Hiring:

Estimate the # of new hires - required for you team ~1yr timeframe
Clear job(s) description of all new positions
Work closely with Talent Acquition team on filtering resumes, scheduling and debriefing Sessions
Identify buddy to help the new member settle in

Roadmap:

Clear understanding of the division’s roadmap and business priorities
Project roadmap - that your team would be working on ~6 to 12months.
Project Plan/Schedule for the projects: in progress and next one in pipeline
Identifying technical debt in your functional area
Functional roadmap to pay off technical debt
New technologies that your team shall pay attension to
Technical conferences your team can attend/follow

WASM Cloud

Fri, 26 Nov 2021 00:00:00 +0000

What is WASM Cloud trying to solve?

Lets ask - what a typical “Development Iteration” look like?

Start of with your Napkin Drawings - conceptual vision of what is it you want to build
Initial Prototype, prove that it works
Then, Figure out how to scale

The challenge is between #2 and #3 there is more often a complete rewrite. As the author claims- Virtually nothing from prototype is written in a way it is scalable.

The goal of the project is to “Ship Napkins to Production” directly.

Draw your conceptual map of how your application should look like
Fill in the blanks with your business logic and just push it - boom!!
Don’t worry about any of the non-sense in between.
Remove all the boiler plate and let developer write only Business logic.

How is this achived?

Using the “Actor Model” and the out of the box boilerplate infrastructure. Personally, I a fan of Actor Model, as its easy to just write a function which has a single responsibility. If you have ever worked with Threads and IPC(Interprocess communication) - you would know the overheads that come in managing them. With the Actor model you have an input queue, an output queue and the function in between. Developer shall write only the function. Function would read from the input queue, process and write it to the output queue. The infrastructure should take care of running the actor, as a thread or process and also manage the input/output queue, data transfer, serialization etc. One of the popular implementation of Actor model is AKKA.

From the authors:

The creators of wasmCloud believe that we can not only change the way developers build software for the better, but make it easier to secure, deploy, maintain, observe, and upgrade that software as well–all while reducing the amount of boilerplate we have to copy and paste.

Example:

Lets look at their Hello World example.

Interesting snippets:

 #[derive(Debug, Default, Actor, HealthResponder)]
#[services(Actor, HttpServer)]
struct HelloActor {}
/// Implementation of HttpServer trait methods
#[async_trait]
impl HttpServer for HelloActor {
//Actual Impl
}

HelloActor is the name of your actor
The two lines with “#” above the actor name igenerate - at compile time - nearly all of the scaffolding needed to build an actor.
The HealthResponder term generates a function that automatically responds to health check queries from the wasmCloud host.
The #[services(…)] line declares the servicesthat your actor implements, and generates message handling code for those interfaces.
All actors implement the Actor interface.
The HttpServer entry declares that the actor will also implement that interface, and requires an implementation of that trait’s method: handle_request.

Conclusion:

I am fan of distributed computing: designing and writing scalable application. Maybe the concept of wasmCloud doing the heavy lifting sounds easy; but I belive its a very hard problem. Coz: when we actually get our hands dirty with actual application design - challenges like: managing states, versioning, schema management/registry, idempotency, trade offs w.r.t CAP theorm, heterogeneous system etc..all comes in. Also, given what Docker + Kubernetes eco system already provides - finding the right architecture to use wasmCloud would be a challenge. But this is definitely a place lot of things can be improved and room for optimization. Star’ed the project for now, will be following it closely.

Reference: WASM Cloud, GitHub

Rubber Duck Debugging

Wed, 24 Nov 2021 00:00:00 +0000

I have been in a similar situation multiple times - where in I was stuck on a problem. To get a solution, I would want to reach out for help. So, I would start to write it down - as it could post it in StackOverflow or send it as an email to my friend/colleague. While I am writing the problem down, I try to set the context and elaborate on the problem statement - after few iteration/reformatting/edits - its strange that I would have found the solution. Oops-What happened???

Stepping back and looking at it, what had happened is - while we try to explain the problem to someone (in writing) - we try to remove all the clutter and focus on the actual problem - with the intent to keep the email shorter and digestable components, so that the audience/reader is not overwhelmed. But, in the process we have a better clarity of the core (only) problem we are trying to solve. When that clarity comes up, more often we get closer to the solution.

BTW: I never knew, there was a name for it - “Rubber Duck Debugging” until I heard it from scott hanselman in one of his podcast. And there is a website rubberduckdebugging for it.

Snip from the site:

Place rubber duck on desk and inform it you are just going to go over some code with it, if that’s all right.

Explain to the duck what your code is supposed to do, and then go into detail and explain your code line by line.

At some point you will tell the duck what you are doing next and then realise that that is not in fact what you are actually doing. The duck will sit there serenely, happy in the knowledge that it has helped you on your way.

Well, I had to buy one rubber duck, as I wasn’t able to negotiate to lease one with my 7yr old daughter 😉

The next time you get stuck on a hard issue, take a moment and write down exactly whats going wrong as if you were going to explain it to someone else, you may just notice exactly where you are going wrong and even if you dont, at least you have a question all ready for submission to StackOverflow…

Software Enthusiast - Getting Started

Wed, 24 Nov 2021 00:00:00 +0000

I have been asked this question multiple times. I would like to be a software enthusiast and where do I get started. Thanks to my Friend to help put this together

Books:

Coding:

Clean Code: A Handbook of Agile Software Craftsmanship *

Unit Testing:

The Art of Unit Testing, 2nd Edition *

Design:

SOLID principles:

Refactoring:

Architecture:

Keeping up with ever evolving tech Space:

Youtube channels to follow

Twitter handles to follow:

Podcasts & blogs to follow:

Reverse ETL

Sun, 14 Nov 2021 00:00:00 +0000

While listening to Podcast stumbled upon this term of Reverse-ETL.

First lets align on a few basic terminologies:

What is a Data Warehouse (DW):

Data Warehouse centralizes and consolidates data from multiple sources like application log files and transaction applications. Orgainzation use DW to gain valuable insights from data, which can improve decision making by building a historical record of data. It can be considered the single source of truth.

What is ETL

ETL Stands for Extract Transform and Load. ETL is a data integration process that combines data from multiple data sources into a single, consistent data store that is loaded into a data warehouse or other target system.

What is Reverse ETL

Reverse ETL is the process of copying data from a central data warehouse to operational systems of record, including but not limited to SaaS tools used for growth, marketing, sales and support.

Discussion:

Hang On! this doesn’t make sense - why should someone do ETL and then Reverse it? Coz, just ETL in itself - is a HUGE infrastructure to both develop and maintain - why would I manage both. Well, as always devil is in the details.

Lets see an example from Hightouch

Reasoning(by Hightouch):

Reverse ETL is necessary because your data warehouse — the platform you bought to eliminate data silos — has ironically become a data silo. Without reverse ETL, your business’s core definitions only live in the warehouse.

More detailed information like use cases are avaialble in this blog Reverse ETL

Conclusion:

The intent here seems to bridge the knowlege gap, that comes in due to data warehouse. Say: If a business user is so used to a particular platform/tech like SalesForce, the user would prefer the analystic output/data to be available in their platform. Incase of Data Warehouse, the SalesForce business user should understand and learn the new ways to access the data and its corresponding schema in the DW; and also learn the access mechanisms like scripts, sql etc.. With Reverse ETL, it creates a comfortable/known space for the user as all the “Actionable Data” from the analytics are available in their platform. So, sure it makes sense.

Only challenge, I see as an organization is - the effort in maintaining both ETL and Reverse ETL infrastructure. Any changes in the DW schema - should be notified to both the teams and the deliverables should be synced up - to avoid any broken pipelines.

Reference: DataWarehouse

Access Control - Casbin

Sun, 31 Oct 2021 00:00:00 +0000

I was trying to read about RBAB (Role-Based Access Control) and ABAC(Attribute-Based Access Control); after some reading - I realized, as a software engineer - it’s way more easier when you get to read code - So, I started to search for open source project which implements access control infrastructure, and stumbled upon Casbin.

Snip from Readme: In Casbin, an access control model is abstracted into a CONF file based on the PERM metamodel (Policy, Effect, Request, Matchers). So switching or upgrading the authorization mechanism for a project is just as simple as modifying a configuration. You can customize your own access control model by combining the available models. For example, you can get RBAC roles and ABAC attributes together inside one model and share one set of policy rules.

The interesting part here is how Casbin - has designed the whole system behind a simple yet flexible configuration file and grammars to describe predicates. Intent of this blog is - just to get you curious about Casbin. Casbin has reasonally good documentation on their site.

ProTip: Best way to understand an open source is to start with the test cases, look at each test case - review the context and assert statements.

Lets look at a simple RBAC Model Config

[request_definition]
r = sub, obj, act
[policy_definition]
p = sub, obj, act
[role_definition]
g = _, _
[policy_effect]
e = some(where (p.eft == allow))
[matchers]
m = g(r.sub, p.sub) && r.obj == p.obj && r.act == p.act

RBAC_Policy.CSV

p, alice, data1, read
p, bob, data2, write
p, data2_admin, data2, read
p, data2_admin, data2, write
g, alice, data2_admin

Basically there are four parts: Request, Policy, Matcher and Effect. Ref

Request: Defines the parameter name and order which we should provide for access control matching function. This enables the code/model to be setup when the data is read from the source.

Policy: It’s the Access stratergy. This is how our internal system is designed. It defines the name and order of the fields in the Policy rule document.

Matchers:: This is similar to a predicate function - which shall describe if the request allowed. Rules which help us to match the request and the policy.

Effect: It’s useful, when there are multiple patterns which would match and you would need to make a decision. There are defaults, here you get to override both allow and deny defaults. Refer: PolicyEffect.

Inheritance:

If A has role B, B has role C, then A has role C. This transitivity is infinite for now
Casbin doesn’t distinguish role from user in its RBAC. They are all treated as strings.

p, data2_admin, data2, read
g, alice, data2_admin

In the above example,

data2_admin has read access to data2
alice inherits/is a member of role data2_admin. alice here can be a user, resource or a role.

test_role_manager.py

 rm = get_role_manager()
#1
rm.add_link("u1", "g1")
rm.add_link("u3", "g2")
self.assertTrue(rm.has_link("u1", "g1"))
self.assertFalse(rm.has_link("u1", "g2"))
self.assertCountEqual(rm.get_roles("u1"), ["g1"])
self.assertCountEqual(rm.get_roles("u2"), ["g1"])
#2
rm.add_link("u1", "g1", "domain1")
self.assertTrue(rm.has_link("u1", "g1", "domain1"))
self.assertFalse(rm.has_link("u1", "g1", "domain2"))

What Casbin does NOT do:

Authentication (aka verify username and password when a user logs in)
manage the list of users or roles.

Reference: Casbin

Premortem

Sun, 31 Oct 2021 00:00:00 +0000

Gary Klein - HBR - this article was quite intersting. I belive, this could be one of the procedures for thinking in organization - to help alleviate or provide feedback to quality of thinking.

Unlike a typical critiquing session, in which project team members are asked what might go wrong, the premortem operates on the assumption that the “patient” has died, and so asks what did go wrong. The team members’ task is to generate plausible reasons for the project’s failure.

It’s beautiful as an idea: Coz, when people are coming close to a decision, it becomes very difficult to raise doubts or raise questions. As, there is fear that person would be tagged as someone who is slowing down the whole group - when the group is about to make a decision. And may at times, that person is perceived as annoying and want to get rid of them or ignore and move on. This premortem legitimizes that sort of descent and that sort of doubt; not just legitimizes it - but also rewards it.

BTW: I dont think its going to prevent people from making mistakes. But it would certainly allow people to possible loop holes to think through before making a safer decision.

One Percent - Per Day

Sat, 09 Oct 2021 00:00:00 +0000

Born and Brought in a middle class family in India. We are often set a goal to become the number one in the class - both by family and teachers. It is all about, topping the exams, getting the highest score, getting rank #1. All you need to do is study hard(er) to become #1. Every parent has the opinion - their child can be the best and always do what ever it takes like pay for additional tution’s [though its way above they could afford], to ensure there is no roadblocks. Comparing against fellow class mates, talking about stories of students who woke up at 4.30am in 5th grade, and got rank #1. At the end of the day you get competitive and you can go as far as the best student in your class (you know). Choosing to burnout or not is not an option.

Many at times we fail to understand, what we are doing is trying to push - so that we become better than someone else. But, later in my carrer I have come to this realization that - You don’t need to compete with someone - The only person you need to compete with is yourself.

You actually, don’t really need to compete - just try to become 1% better than your yesterday self. Thanks to the book - Atomic Habits- I wish I had read this book 20yrs back.

What has changed in me is - I wake up, look at the mirror and say - what have I done yesterday to make myself 1% better. Then, what am I going to do today to make myself 1% better.

Many at times, we are so sucked into our daily lives, we fail to realize this. We are busy being busy and only at the end of the year - focal cycle - we try to retrospect and see what went well and what could have been better. But, there are two issues here 1) It’s too late feedback 2) the goal we set for the entire year - are too ambitious. But, by taking things one day at a time and having a goal of 1% improvement - which is achiveable, will enable us to be better prepared with minimum motivation.

For me 1% could be as simple as : reading 10pages of a book[tech, leadership], watching a tech podcast, writing a blog, online course, going to gym, badminton. Improvment is always both Mind or Body. As long as you cautiously spend time for these - you are learning/improving.

Note: learning is a continous process, it take time and its a never ending journey.

Happy learning!

Bitmap Indexed Storage

Sat, 10 Jul 2021 00:00:00 +0000

Always been facinated by the ways - data structure’s can enable better storage and faster queries.

BitMapped Indexed Storage - is useful while using Columnar Databases - where per column data is stored together.

Often the number of distinct values in a column is small compared to the total number of rows. In the below example: column sex there are only two distinct value Female or Male in the given table - the cardinality is really low.

In such cases: We can now take a column with n distinct values and turn it into n separate bitmaps: one bitmap for each distinct value, with one bit for each row. The bit is 1 if the row has that value, and 0 if not.

Picture is worth a thousand words.. here we you go..

Fantastic Query performance… As the amount of data read from disk on to the memory is less - improved Disk Throughput. As always, solution are for a specific problem. If you wanna do a range query, then this solution might not the best.

If n is very small, those bitmaps can be stored with one bit per row. But if n is bigger, there will be a lot of zeros in most of the bitmaps (sparse). In that case, the bitmaps can additionally be run-length encoded or might not be recommened.

Reference: Designing Data-Intensive Applications, by Martin Kleppmann

Five Ideals of Software Development

Sun, 04 Jul 2021 00:00:00 +0000

Credits: Most of the text below are “Verbatim” notes from the book The Unicorn Project, by Gene Kim.

There are times, when you read a book - it strikes you really hard/well and you would like to share it with as many people you would like. Since this blog of mine is public - I decided to put it here so that it would help someone/someday.

Here are Five Ideals:

1. Locality and Simplicity:

Design things so that we have locality in our systems and the organizations that build them. And we need simplicity in everything we do. The last place we want complexity is internally, whether its in our code, in our organization or in our process. Rate at which a software team can add new features can improve only when - we pay down technical debt as a part of daily work and a good automated test suite to catch the failures.

Remember: the “S” in the SOLID Principles - Single Responsiblitity.

2. Focus, Flow and Joy:

Its all about how our daily work feels. Is our work marked by boredom and waiting for other people to get things done on our behalf? Do we blindly work on small pieces of the whole, only seeing the outcomes of our work during a deployment when everything blows up, leading to firefighthing, punishment and burn out? Or do we work in small batches, ideally single-piece flow, getting fast and continual feedback on our work? These are the conditions that allow for focus and flow, challenge, learning, discovery, mastering our doman, and even joy.

Remember: Test Driven Development(TDD) and Continous Integration (CI)

The process of stopping a system when a defect was suspected originates back to the original Toyota System Corporation to something called Jidoka. The idea behind Jidoka is that by stopping the system you get an immediate opportunity for improvement, or find a root cause, as opposed to letting the defect move further down the line and be unresolved.

3. Improvement of Daily Work:

Reflect upon what the Toyota Andon cord teaches us about how we must elevate improvment of daily work over daily work itself.

Remember: Refactoring, by Martin Fowler

4. Psychological Safety:

Where we make it safe to talk about problems, because solving problems requires prevention, which requires honesty, and honesty requires the absence of fear. In manufacturing, psychological safety is just as important as physical safety. A culture of fear where everyone is afraid to share the bad news - will only have a negative impact.

One of the Value of KLA (my current employer) is “HFC: Honesty, Forthright and Consistency” - which I greatly admire.

5. Customer Focus:

Question ruthlessly whether something actually matters to our customers, as in, are they willing to pay us for it or is it only of value to our functional silo.

Remember the 80:20 Rule.

According to Standish’s research and more recent estimates, 20 percent of software features are often used, while 50 percent of features are hardly ever or never used. The remaining 30 percent of features are used infrequently. (For package applications, like Microsoft Office, Standish found that just five percent of features are used.) Ref

Book - Making of a Manager, by Julie Zhuo

Sun, 20 Jun 2021 00:00:00 +0000

If you are a newly promoted manager and struggling to figure out what the job is like. Don’t worry you are not alone - this book is exclusively only for you - go get a copy and keep it on your table (I have mine). Having read multiple books about management and leadership; also attending relevant sessions - nothing really stuck to me like this book. Maybe there is a reason, I could connect.

Trivia: The author - Julie Zhuo is the VP, Product Design at Facebook, having successfully worked her way up the ranks.

In this book, Julie - I would call her an accidental manager; becomes a manager at the age of 25, just because the size of her team was growing. She wasn’t prepared - she had no idea where to start and what to do and most importantly what she should NOT be doing. She juggles her way through trying to be a good manager and fighting her inner conflicts (read soul) - which was still an individual contributor(IC). And this was exactly where, I was able to connect to Julie and her book. As, I was a IC - who loved to code (time flies when I was coding) - excitement/feeling of making a feature work was awesome.

Hard Truth: The roles and responsiblities of a Manager vs IC is completely different - you cannot do both. Well if you are doing it - then you are a superhuman!

I was trying to do both for years, and it wasn’t scaling up. I was signing up for features and assiging bugs to myself - and it was hard to balance the time between the needs of the actual role(/team) vs my own code deliverables.

Well, don’t get me wrong. I dont mean, you should stop coding when you become a manager. I still code (mostly on weekends), but just that - we need to understand, that its NOT your primary responsibility. My guidance: the code you work on, should not be in the critical path and there are no depenencies (someone waiting for you to deliver).

I have been a manager for just over 5yrs as of today(Jun/2021). So, I might not have lot of experience. But this book definetly made a difference for someone like me who became a Manager being an IC.

It’s an easy to read book - you might feel like Julie is sitting on the other side of the table and talking to you - a live mentorship 😉. You can learn a lot from this book, irrespective of the team size you are leading (big or small).

Few Key Take aways:

Manager are Made not Born:

This was an interesting revelation for me 😉. This is a skill that you could really work on - if you find the roles and responsibilities motivating/interesting. Julie talks about 3 important things: 1) Purpose 2) People and 3) Process - the why, the who and the how. A great mananger constantly asks herself how she can influence these levers to improve her teams outcomes.

Why: Your team knows what success looks like and cares about achieving it. Success would be embody both - self (career) and for the product (company).
Who: Does the team have the right skill set and the motivation. Trust and coaching to their strengths.
How: Establishing guidance for principles and rationals behind decicion making.

Art of feedback:

Avoid: It’s time for our Annual Performance Review. Here is how you have been doing for the past year. Aspire: Can I share some feedback for you from that meeting this morning.

Giving feedback as often as possible, keeping it as specific as possible, clarifying what success looks/feels like and finally suggesting next steps. In case of cirtical/bad feedback - don’t do a “compliment sandwich” - as the most important focus point, which acutally matter could be missed.

Hiring Well:

Hiring is not a problem to be solved but an oppurtunity to build the future of your organization.
Hiring is Your responsiblity. No recruiter can possibly know what an ideal candidate looks like for your team. So, ensure you describe your ideal candidate as precisely as you can.

I would stop here - and recommend you to go read the book. There are lots of interesting observations and recommendations - which you could try out of the box. Read it, you can thank me later 😆

Highly Recommended!!

the-making-of-a-manager

Sqlite - Blunders

Sat, 05 Jun 2021 00:00:00 +0000

What is Sqlite:

SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine. SQLite is the most used database engine in the world. SQLite is built into all mobile phones and most computers and comes bundled inside countless other applications that people use every day. More Info: SQLite

My Learning:

Sqlite is a fantastic embedded database - light weight, easy to use and almost Zero Configuration. I would prefer to call it ZMQ of DataBase 😉

Blunder:

As part of our design: we had to persist a bunch of parametric information per user session (few MBs). This information is not required outside the user session. But, it would be reused when the user comes back. Session - is local running on a windows box. We decided to use sqlite as our backend persistence mechanism; as there was no installation or specific configuration required to set sqlite up.

Since there are parallel read write happening, we had to use the WAL: Write Ahead Logging journal mode.

We did couple of blunders:

Blunder #1 - Insert ONE at a time

When an action occured - we opened the db connection, inserted record and closed it.
We didn’t want to keep the connection open, throughout the session - as there were chances of software crash - which could potentially corrupt he database.
But as time went by, the number of actions became HUGE (order of millions - automated). At this point - Opening and closing the connection for every action, started to hurt us (high latency and low throughtput). Coz, the time taken to open and close a connection (~300ms) was adding up.

Solution: Microbatching: Batched about 500 actions together - open the connection once, inserted all 500 actions in one shot and closed - Transaction. Worst case, we would loose less than 500 actions, and that tradeoff was acceptable.

Blunder #2 - Mounted Drive:

The db had to be persisted on a mounted Linux drive (there are reason for this, I won’t go into details).
As the session started - the app created a sqliteDB directly on the mounted drive and started inserting.
Since we were using WAL mode - there were temporary files getting created and deleted on the mounted drive during every transaction.
As time went by - the reliablity of the drive/driver went down, and retries didn’t help keep up. The throughput significantly got impacted and multiple errors started showing up.

Solution: During the session, create the db in a local (temp) drive, and end of sesssion copy it into the mounted drive. Similarly, when session restarts - copy from mounted drive on to a new temp location and consume it; and copy it back on completion.

Never EVER - read or write an Sqlite Database from a mounted Drive. It usually works, but when it doesn’t - its a nightmare.

Blunder #3 - Lazy Load:

When a specific information was required - we opened the db connection, executed the “select” query and return the results.
Again: We didn’t want to keep the connection open, throughout the session - as there were chances of software crash - which could potentially corrupt he database.
The read was implemented in a “Lazy - On demand” fashion - so that only the minimum required information was read on demand.
Reading only the required info is an ideal design - why would we ever read more that what is required? GraphQL - is a good example of how to design minimilistic API.
But the challenge in our (specific) case: was the number of reads - it was about 600reads in less than 50 milliseconds. And each one had to open and close the connection (which takes ~50ms).
Since it was a high throughput system, the overhead of lazy read was slowing down the entire software.

Solution: PreFetch Require Information

Since we knew the sequence in which the actions are performed, we redesigned the query to read “more” records than required.
Then created an in memory cache [index/map] to hold the data and filter based on Client Query.
Read from DB would occur only if there is a cache miss.
The cache was a rolling buffer with LRU (Least Recently Used) and a watermark (memory or count), so that the RAM didn’t explode.
When a new read from DB was required, if the cache is full - we delete the oldest item and read from DB and add it to cache.
No change to the Client API. Only the Data Acess Layer changed. 😉

Conclusion:

As always on hindsight, the solution looks trivial and obvious. Since, we were using Sqlite for the first time and initially didn’t reallize the data could grow exponentially - had to learn (Adapt) the hard way. Esp, when these are running in production - you need scheme’s to manage old vs new verisons of software (read, backward compatibility).

Concurrent Dictionary and Delay - C#

Sun, 30 May 2021 00:00:00 +0000

Reference: ConcurrentDictionary

In a high throuput and low latency enviorment - we had a peice of code in the critical path - which helped in throttling. After version #2.0 of the software was released (internally) - we saw the througput has gone down significantly almost 3X - compared to previous Version #1.0 - 4mins to 11 minutes.

We looked at the history/change sets between the versions: it was huge, 100s of commits and software was running with 100s of threads - on a 2Milliion LOC software stack. So, we were unsure which peice of code nor thread - was causing the delay. We were all over the place, trying to see if its a system issue or memory trend or some fragmentation vs memory/back pressure vs excess logging etc…. And, after 3 day (yes: 3days slog) - we found its a delay caused by the ConcurrentDictionary class!!! 😳

It was an unpleasant suprise, since we didn’t expect such a huge delay cause due to a .Net Framework class.

1. Version 1.0: [4mins]

One API with its backing data store with one Dictionary.

 private Dictionary<int, User> userInfos;
public List<User> GetUsers() {
return userInfos.Values;
}

2. Version 2.0: [11 minutes]

In version #2: We changed the data store, which was holding the user info from a simple dictionary to a two level ConcurrentDictionay.

There was some obvious reason for using ConcurrentDictionay - I won’t go into the details. But, after changing the data store- we ended up blindly updating all usages to consume from the new data store.

And the API (GetUsers) implementation was updated, interface remained the same - but: it was in the critical path.

 private ConcurrentDictionary<int, ConcurrentDictionary<int, User>> userInfos;
public List<User> GetUsers(){
var users = new List<User>();
for(var userGroup in userInfos.Values){
for(var user in userGroup.Values){
users.add(user)
}
return users;
}
}

3. Version 3.0: [4 minutes]

After looking into the usages “in the critical path” - we only required the count of the active user groups.

So, we wrote another API to just return the count. With this change, we were able to bring it back the throughput back to #Version 1.0. Huff!! 😌

 private ConcurrentDictionary<int, ConcurrentDictionary<int, User>> userInfos;
public List<User> GetActiveUserGroups(){
return userInfos.Count;
}

Learning:

On highsight, the fix looks trivial and straight forward. But the challenge was the huge debugging effort to find the root cause. And little did we suspect a .Net Framework class can cause such a huge delay and slow down overall execution time. It was a great learning for us as a team - involved in debugging. BTW, we are talking about a code base of about few Million lines of code.

BTW: Concurrent dictionary exists for a reason: Usage of it as our data store was the right choice.

Represents a thread-safe collection of key/value pairs that can be accessed by multiple threads concurrently.

If a class should be thread safe, then there has to be a lock/Mutex implemented. A lock has its own tradeoff in terms of execution time (Thread Scheduling, Prempt etc..) - so think carefully on its usages.

Parallel Processing with Constraints

Sat, 15 May 2021 00:00:00 +0000

An interesting senario, I recently encountered - have tried to tear down the context to get to the solution early, and avoid nitty gritty (boring) details. Also, example showns only 2 threads.

Context:

Different Actions (like: 1,2,3) from Cutomers (like Mr.A, Miss.B..) are pushed into a stream/queue from an external component. “$” denotes end/last action from a customer. A thread can 1) Process each action 2) Generate aggregated views for each customer per session - and then write it to the sink. The processing threads are started when the session starts and a user cannot reenter the session once done.

note: all threads mentioned are background and long running.

Constraints:

Aggregation can be triggered only after all the actions are processed. i.e processing of action A1, A2, A3 should be done - before aggregation “A$” of A is triggered.

1. Single Thread

Well, this is a no brainer. We have one thread which reads (pull) from the stream/queue and process one item from the queue at a time. All constraints can be managed with a global context inside the thread.

Below is a code which runs inside the Thread.

 # {name, [actions]}
customer = {}
def read_from_stream(name, action):
if action=='$':
aggregate(name)
else:
process(name, action)
def process(name, action)
result = process_the_custom_action(action)
cusomter[name].append(result)
def aggregate(name):
action_results = cutomer[name]
view = gen_agg_view(action_results)
write_to_sink(view)
# remove customer from tracking
del customer[name]
def process_the_custom_action(action):
#Heavy lifting operation
return result

2. Pull Model - Parallel Processing

Later we realized, the function processing_the_custom_action is heavy lifting and takes more CPU. So, we were not able to keep up, with the incoming data rate. So, the memory increased drastically and triggers garbage collection and slow down actual code execution.

So, we decided to introduce multiple threads to read from the queue. Same code as above - executed in multiple threads. Each thread would pop (PULL) one item from the queue, process it and then pop the next item. Just to be explicit: one item from the queue is processed by only one of the threads, but which thread (order) gets which item is not guranteed.

This design had a major challenge in ensuring the constraints are honored.

In the above example: Aggregation for customer “A$” should be triggered only after last action “A3” is done. But “A$” and “A3” are processed in two different threads i.e. A3 by Thread #1; A$ by Thread #2. How would Thread #2 know A3 processing is done?? 😞

If we really need to go with this approach and also honor the constraints - then there should be broadcast events fired and all threads subscribed to it. Each thread should subscribe and do individual book keeping for all customers being processed across all the threads (dependency across threads, coupling..huff!!) 😰

Pros: If any of the thread is killed in between. The data loss would be minimal i.e only the customers already read by that specific thread in the past and not completed. As all NEW data would be processed by one of the threads alive.

3. Push Model - Parallel Processing

To ensure, we honor the constraint and also enable parallel processing. We introduced a load_balancer, called it a distributor module.

Distributor Module:

Would have reference to all the available processing threads.
Each thread would have a corresponding queue. So, No_Of_Threads equals No_Of_Queueus.
Each queue would have an index, which range from 0 to No_of_threads.

0 < Queue_Index < No_Of_Threads
Once an item is read - it generates a hash using the “name” of the customer. (like: SHA256)
On the HASH we apply a MOD using the No_Of_Threads available and get the queue index

Queue_Index = hash(customer_name) % No_Of_Threads
Push the item to the corresponding queue_index

With this approach, we ensured - same customer items are always processed by the same thread. So, there are no cross talks across threads or external dependency. We could increase or decrease the number of threads used (obviously: before a session starts not in-between), based on a configuration or heuristics.

 #### Distributor Module
def read_from_stream(name, action):
# Generate hashkey using customer name.
# Mod using the number of threads. No_Of_Thread == No_Of_Queues
queue_index = hash(name) % No_Of_Threads
publish(queue_index, name, action)
def publish(queue_index, name, action):
# get the queue reference using the index
queue = queues[queue_index]
# add the item into corresponding queue
queue.push(name, action)

Cons: The biggest cons: If a thread is killed for what ever reason. All the customers past and future once - which would be queue/allocated to the thread would be dropped. But in the pull model, it would have dropped only a handful sitting inside (past) the thread. Well we could handle this dynamically - like have a heart beat mechanism and update the “No_Of_Thread” and get it working - just that it would be another infra.

Conclusion:

There are pros and cons to both approaches.

More Infra == More Code == More Maintainability == More to Debug == More room for bugs to creep in.

Trying to keep it simple while keeping the tradeoffs minimal is the key. In our case; we went with the push approach - as the chances of a thread crash was minimal and also we added some defensive coding and drop an item safely if required.

Reference:

Storm Model, by Nathan Marz

Choreography vs Orchestration

Sun, 02 May 2021 00:00:00 +0000

I have been getting into this debate quite often; which approach is preferred - Choreography vs Orchestrator. As always, it depends - Here is my take on it…

Modules:

App: Like an UI module, where user trigger’s an action like:Create an account
Action #N: Specific modules, which has one specific action to be performed like: sending email, mail atm cards - once an account is created.
Coordinator: Module, which handles/triggers all the downstream actions (#n) - when an event is recived from App.

Orchestrator

Here the coordinator is the orchestrator - which has the knowledge of the business rules - like: what are the set of actions that need to be performed when a specific event occured. If you observe, the arrows are pointing from coordinator towards actions. So, the coordinator need to understand the details of each and every API (per Actions) - their base URL/REST end points. And if each of them takes in differnt params - then all this information would sit inside the coordinator module.

The cooridnator has become the central brain. And now it has tight coupling with all the Actions. And if a new action should be added, we will have changes to the coordinator module. After a point of time - this module will become the “God like” service. And every new Actions team would have to come to coordinator team (more meetings/discussion/followps) - to have their action integrated with the coordinator. Any changes to the coordinator module would become extremely hard - as there is so much coupling. And in addition failures of each action should be handled with special case - adding more and more actions specific details/code into the coordinator module.

On deployment and version management: If there are specific changes to an Actions - URI or params. Either the action module - can never depricate their existing uri and params, as it is used by coordinator module. Or, coordinator module would have to understand the versions (of action) and act accordingly. Both can get a bit too messy. And choosing one or either of them - would require some tradeoffs (debates) to be made somewhere.

BTW: When we are just starting of - coordinator might be the easiest way to get things done (read, move faster). But as we scale out - we should look for approaches which would enable better decoupling.

Pros:

Better control in coordinating the Action modules
Handling failures would be easiers - in one place
One place to look at the flow chart
Good for transaction driven and places where actions are not idempotenent

Choreography:

In this approach, lets take a broker like KAFKA as an example. When a new event is created its publised to a KAFKA queue with a specific TOPIC. There could be multiple subscribers (actions) - which would subscribe to the topic. The subscribers have their specific logics and subscibe to specific topics. At any point of time, any new actions can be added into the system without the knowledge of the App module. Look at the red arrows - there are NO arrow pointing from App towards specific Actions - so app doesn’t have any dependency on Action. We could also have custom subscribers (Actions) - which would run a pipeline for the data science team - this would enable us to generate observability data on the loads and trends.

Deploy: In this approach both app and action moudles can be indeopendently deployed and upgraded. The schema registry would have the contract details - topic vs the payloads.

Schema Registry: Every payload schema: should be published in the schema repository. We can use libraries like “Protocol Buffers” or “Apache Avro” for describing payload schemas.

Cons:

Handling errors might become very tricky. You would need an external monitoring infrastructure to track this.
If there are depenencies across actions - then you should think twice on using this pattern. Though there are ways to solve it - like: using sub topics.
Debugging can become a bit more hard. Needs distributed tracing to correlate/debug.

Conclusion:

Personal preference is to lean towards choreographed approach - as they are more loosely coupled and are more flexibble and amenable to change. Again: it all depends on your use case - if you are building a low latency system - which should catch errors and react to it really fast based on lots of buisinees logic, then orchestrator might be a better approach.

References:

Book : Building Microservices by Sam Newman

Feature Toggle - Nightmares

Sat, 01 May 2021 00:00:00 +0000

We were working on a custom algorithm to optimally grab images given a list of area. Every image grabbed is written into an object store with a generated hashkey. The haskkey can be used to query the recorded data - when we would like to play them back.

Software was a monolith and the algorithm was shipped with it. So, any new fixes would be shipped with the new software versions and user would be able to playback the same recorded data.

def customAlgorithm():
locations = []
# actual algo...
return locations;

A day came when we - rolled out a bug fix in the algorithm. But, we broke the backward compatiblity 😟. Users had lots of existing recorded data which they wanted to playback. Since the algorithm had changed - the generated hashkeys didn’t match the ones in the object store.

We thought its proabaly a one off scenario: so we added a feature toggle - user will have to manually (pain) figure out - which version the data was recorded on and update the toggle config (xml) and rerun.

def customAlgorithm():
locs = []
if version == '1.0':
locs = oldAlgo()
else:
locs = newAlgo()
return locs

And - Well there was another bug fix:

def customAlgorithm():
locs = []
if version == '1.0':
locs = Algo_1_0()
else if version == ''2.0':
locs = Algo_2_0()
else:
locs = latestAlgo()
return locs

And yeah the story went on for one more iteration and we had to STOP!!!! 😓 It was already becoming a nightmare to mantain the code.

It wasn’t just manually figuring out the toggle which was creating pain - but to enable code reuse, the algorthm had to be refactored multiple times - so that parts of the algorithm can be reused across versions. In addition to that the number of test suites we had to manage had gone up significantly, as there were multiple users each one using different version - and oh yeah there were version specific fixes released as patches 😟

Finally we decided to write all the metadata - related to recording - into an embedded database. And on playback we get the information from the database and not by re-running the code/algorithm.

On hindsight: the database feels like it an obvious solution, right? well the devil is in the details - in our case it about effort(read, time). The playback infrastructure based on database took ~8 weeks to build -first version. It had to handle numerous (10s) use cases. Previously, we only had one flow to be maintained i.e the flow which ran the use cases + algo shipped. But now we have to manage TWO different flows - one for recording and one for playback (x) No_Of_Use_Cases.

Its almost 3yrs since we rolled out this solution to production. When we look back it was a very good decision to build two different workflows 😍. Code is more structured and maintainable.

At times - short term goal/fix takes priority. But, its always good to step back and look at the pain points and have an item in your backlog to find a better solution. The priority should be driven based on feedback: remember the 80/20 rule

80 percent of customers only use 20 percent of the features in the software they’ve bought.

BTW: the database schema has changed a lot - with the addition of multiple use cases. Which also needs version management 😉 And evey new use case discussion now has two parts to be discussed - some problems are good to have (read, tradeoffs).

Ref: Feature Toggle

Side Cars

Sat, 01 May 2021 00:00:00 +0000

Idea

Add additional functionality into an existing application, with minimal or zero code changes to the exiting application. Also, the package can be consume by any software - irrespective of the language or framework (heterogeneous) it is written in.

How?

Using side cars pattern - : I/O( ZMQ) + Topic (intent) + Data( ProtoBuf) + Impl(Subscriber of topic).

DAPR framework from Microsoft Azure team – uses a similar design - runs side car. Dapr is a portable, event-driven runtime that makes it easy for developers to build resilient, micro service stateless and stateful applications that run on the cloud and edge and embraces the diversity of languages and developer frameworks.

We could use the Side Car for adding functionalities like resource monitoring, logging, log aggregation, configurations, observability (telemetry /feature trends) etc.. Once we get the sidecar installed - we could keep adding new functionality as it independently developed and deployed. The features in the sidecar could be turned on or off based on options.

Say as an organization: we decided to use cloud solutions for telemetry and internal tools decide to push data to the cloud - this solution might need a security layer - which would implement both gov and corp policies. This common implemenation of policies could be sitting inside the side cars and can be deployed on to all machine where the internal tools runs. And any update made to the policies would be transparent to the application - as the side cars would be independently deployed and maintained.

How is it different from a Microservice with REST API?

Microservice doesn’t really care about the client. But here the side car is tightly coupled and tracks everything about the main application – including say: killing the application if the resource used is above the threshold - so that the tool PC doesn’t require a restart.

Where should it be used?

Only for non-critical operation - as the latency will be high.

Reference:

Badminton - BG66 Ultimax String

Sun, 21 Mar 2021 00:00:00 +0000

It’s been around 3-4yrs since I started playing badminton. Have heard friends discussing about different types of badminton strings - but personally never felt a need to take a deep look at any of them. I was always using the standard - BG65 string; its so durable - I never changed them for more than a year 😉

After C19 break joined a new team of players. I am an intermediate player - but felt my game was very sluggish. I initially thought due to the C19 break - my game was completely off. But later realized that; there is a difference in the skill of the new players - they were really good. In my previous team, the rallys were really shot ~6, but here it went to almost 16 - it was also tiring.

I definitely had to do something, to not just move up my game - but atleast match the skill of the players. BTW, I am not a trained player (never been to coaching). One of my friend recommened - to try changing the string and see if it make a difference.

Below are some basic about badminton string. On a 10000 feet level: Common badminton strings’ diameters range from 0.65mm to 0.70mm. There are two types of strings - “Thick” and “Thin” ones. I won’t go into detail of each one, there are ton’s of materials out there.

Below are the specs for B66 Ultimax:

This was the only string my friend recommened - since he was an advanced player; I decided to blindly take his recomendation. This is a very thin string (0.65mm) - and a bit on the expensive side.

There are two important things to notice - Durability and Control. It has the lowest durability among Yonex string - you might have to change then every few weeks 😟. But, It has the best control/power 😍

Honestly: Within a week, I was able to see a huge difference in my game. I was able to return smashes with much better control and clears did reach the 3rd court - previously, it usually landed in the mid court and we get a killer smash return. And finally, much better control of my short serve and drives. After using it for a month - I am just loving it. I have now changed all my 3 rackets with the same string.

Finally: If you are really looking to up your game. Try out different strings, in my case B66 Ultimax worked - you might find a different one. But, keep looking and trying.

side kick: Taufik Hidayat was using BG66 Ultimax.

Next string on my list to try is Aerosonic - worlds thinnest string.

MongoDB - Hybrid Schema

Sun, 14 Mar 2021 00:00:00 +0000

I recently ran into a situation where in we have a session and each Session produces a lot of messages (sync) and each message has an associated timestamp and order. This is similar to a blog post (session) and its comments (messages).

Since the messages are unbounded (not sure how many would be produced). In my initial design - I had a separate collection for Messages. Each Session would have one corresponding MessageW (wrapper) document and each MesssageW document would have an array of messages (acutal messages).

With all the available data I had - the size of the MessageW document never went above 1MB - so we were all good. But unfortunately, we had a new use case which broke this assumption 😟. The session was running for hours and there was tons of messages getting generated and yes - we hit the 16MB ceiling 😓.

Obvious solution was to create to create a new one doucment for every new message. So, that we never hit the size limit (16MB) of a mongoDB document. But, since we are looking at unbounded message - there could huge (million message docs) number of documents.

So, we choose to go with the Hybrid Approach

In the hybrid approach, we go with our initial option - where we have a MessageW (wrapper) document - which has an array of Messages. Then we add a constrain on the document i.e. the size of the array should not be more than a given arbitary number. For every new message we increment the count. If the number count crossed the constraint - then create a new MessageW document, inc the page number and also update the Session (NumberOfMsgPages). The reader would get the NumberOfMsgPages and read backwards - until he hit page 0. Voila!! 😍

Important Items to note:

Session Object : will have a NumberOfMsgPages property
MessageW Object: would have a Page and Count property.

Operators:

Since: A code is worth a thousand words. Here is the code 😊

 public void InsertMessage(Message message, ObjectId parentId)
{
var builder = Builders<Message>.Filter;
// Find MessageObject: given parentId and CurrentPageNo
// And: Constraint: Number of messages is less than 1000.
var filter = builder.Eq("parentId", parentId)
& builder.Lt("Count", 1000)
& builder.Eq("Page", CurrentMessagePageCount);
// Try and Insert into existing Message Document array And Increment the Count.
var update = Builders<Message>.Update
.AddToSet(x => x.Messages, message)
.Inc("Count", 1);
// If the constraint with "Count" matches.
// ModifiedCount would be greater than Zero
var result = MsgCollection.UpdateOne(filter, update);
if (result.IsAcknowledged)
{
if(result.ModifiedCount== 0)
{
// Increment the Page count
CurrentMessagePageCount += 1;
// Update: NumberOfMsgPages in Session Collection
var sessionFilter = Builders<MySession>.Filter.Eq("_id", parentId);
var updateSession = Builders<MySession>.Update
.Inc("NumberOfMsgPages", 1);
SessionsCollection.UpdateOne(sessionFilter, updateSession);
// Find MessageObject: given parentId and CurrentPageNo-which is obviously not found.
// So, we will use "IsUpsert=true" to add a new document.
// And: Number of messages is less than 20.
var filter2 = builder.Eq("parentId", parentId)
& builder.Eq("Page", CurrentMessagePageCount);
// Try: Insert into existing Message And Increment the Count
var update2 = Builders<Message>.Update
.AddToSet(x => x.Messages, message)
.Set("Page", CurrentMessagePageCount)
.Inc("Count", 1);
MsgCollection.UpdateOne(filter2, update2, new UpdateOptions() { IsUpsert = true });
}
}
}

A obvious reaction after reading the code is - Ok, whats the big deal here?

Lets take a code walk through: (think how would you achive this in your traditional SQL DB’s)

Code never check if the MessageW document exits before Update.
- How did it work: “$IsUpsert=true” is the magic word.
- $IsUpsert=True: MongoDB would internally insert a new document - if the document doesn’t exists.
Updating counter is happening inside MongoDB:
- $inc mongoDB operator - increments a specific field - in our case count
- Hence, Client doesn’t have to read the document to know the actual count and then increment/update.
Constraint Check:
- $lt MongoDB operator would handle the constraints.
- Update goes through if the constraint is met.
- As part of the result, we could check if any doc was modified
- Cool part - the second half of the code runs only once in 1000 inserts

How did we come up with the arbitary constraint number? well, I dont think we have a thumb rule here. Since we don’t track the actual document size - you would have to look at your data and come up with the sizing.

BTW, I am new to MongoDB and learning. If there are better approaches - feel free to write back to me. Thanks.

Reference: MongoDB Applied Design Patterns, by Rick Copeland

MongoDB Tit Bits

Sun, 07 Mar 2021 00:00:00 +0000

While I am getting started with MongoDB for my new project. Here are the few quick notes - based on my readings so far:

One Key Takeaway:

Data accessed together should be stored together

WiredTiger:

WiredTiger is the default storage engine for mongo db
It stores documents and indexes on disk
In memory cache stores some doc and frequency used index - working set
- 50% of (RAM -1 GB) Or, 256MB

Massive Arrays:

Max Document size is 16MB
Index performance on arrays decreases as array size increases
Extended Reference pattern: where we duplicate some and not all data

Index:

Each index is atleast 8KB
Index take up storage - One File for each collection and one file for each Index (WiredTiger impl)
Write performance as index needs to updated
Limit each collection to 50 Index max
Do: Add index for frequently supported queries - improves read performance
Don’t: Create unnecessary indexes - reduce performance and takes up space

Bloated Documents:

Do: Data accessed together should be stored together
Don’t: bloat your document with related data that is not accessed to gether
Data that is related to each other should NOT necessarily stored together
Remove bloat from frequently used documents - it can be inmemory wire tiger cache
Data Duplication is OK (depends!)
- Summary document and Details document - ref via links

Case Incentive Query:

$regex queries are case insensitive but not performant
Non-$regex queries are case sensitive
Collation:
- Language specific rules for MongoDB for string comparison
- Strength ranges from 1-5
- Strength 1-2 will give you case insensitive
- Query: {“first_name”: {$regex: /Jacob/i }} - case insensitive
- Query: {“first_name”: “Jacob”} - case sensitive

Accessing - Separate data together:

$lookup:
- Is used to join data from more than one collection
- Great for rarely used queries or analytical queries (batch run overnight)
- Very slow and resource intensive

Upsert:

Upsert is a combination of update and insert. Upsert performs two functions:
- Update data if there is a matching document.
- Insert a new document in case there is no document matches the query criteria.
Personally: I found this very interesting, as for my use cases - I dont need to track if the document already exist.

I understand - the blog looks a bit navive (read, nothing interseting), sry I am just getting started. BTW, I plan to write a follow blog - deep dive on my specific use cases and data modelling.

Reference:

Goals - Yr 2021

Sun, 17 Jan 2021 00:00:00 +0000

Being in the IT industry for more than a decade:

Start of the year: “Goal setting” - you align with your manager on how the year is gona possibly look like
End of the year: “Self Assessment” - your retrospective of how the year was. Followed by “Performance Feedback” - comments from your peers and manager.

This is really good - a proven/well established process - which works!

On personal front - I have never thought about setting personal goals. Well Hang on! - what about New Yr Resolution? 😜 Oh ya; it just never worked - why? it never had a system. As a huge fan of Atomic Habits book - here is a blog from the author on Goals-Systems - which talks about the system. Below is a snip from it:

If you want better results, then forget about setting goals. Focus on your system instead - James Clear

In business - goals are usually based on SMART.

But: 3 criteria’s for MY personal goals:

It should be a Compass
Be honest about what I want
Measure the Gain not the Gap ref

Seriously!! how can you be foccussed on achieving a goal if you are not SMART?

well - I would prefer to love and embrace the Journey that the actual destination.

Now lets set some goals:

Fitness:

I started around end of last yr (2020) - but due to the vacation/Christmas season (fantastic food at home 😋) and my fracture - had to drop it (well, stop blaming - you could have done better! 😉). This year I plan to restart and give it a honest try.

System:

Signed up to Fittr
Enrolled under a Coach - Mr.HR in Fittr
- Quantified Nutrition and Workout plan
Have a buddy - Mr.P
Blocked 1hr of time on my calender every day for workout

Hopefully Fittr’s system would be able to track my progress and my coach w/ guidance. Expectation: I should be able to look back every month and see some progress. Progress could be anything - starting from loosing weight, to getting in shape, to ripped (ripped! seriously? be honest😁). As it stands today - I am in the obese category - any progress todays being healthy is good enough.

Sports:

I have always loved playing sports - like Cricket, Football and now Badminton, Cycling. But never went for coaching or had any planned schedule. But, always dreamt of playing sports every single day in my life! BTW, I don’t mean - I should have been a sportsman - I was never good at any of them 😉 - just enjoyed doing it.

Cricket requires lot of time investment and Football is too injury prone - So my only option is Badminton and Cycling.

System:

Signed up a membership - at a badminton academy close to my house.
Have a bunch of friends who play only on weekends thanks to Rozo
Combine (alternate) Fitness workout with sports - optimizing time.
Cycling once a week, with Mr.K

Honestly - just the feel of playing - is all that is required. It has always been a great refresher. Expectation: Sports 2days a week.

Journal:

Started very enthusistically on the Bullet Journal - but dropped it midway 😞.

Looking back: It was hard to manage, as I was throwing too many things at it - starting from adhoc meetings/calls to escalations. I was religiously following the - 2 minutes rule by David Allen - and the same went into the journal.

If it takes less than two minutes, then do it now - David Allen

It was a clutter - and the energy required, at the end of a long day to declutter 😟, lost motivation after a month.

System:

Update Journal before bed
email follow up: flag it and leave it in outlook.
Try use Dynalist

This might be an ongoing thing - which I would need to recalibrate throughtout the year. Expectation: Every re-calibration should be an improvement of the previous state.

Blogs:

Started this last yr(2020). Have not been consistent.

System:

Future Blog list - would be managed in Journal
Block calendar time once a week - Saturday afternoon
One blog per week
- If I missed, will add a “No Blog week” blog 😝, keeps me accountable

Books:

For a starter like me - Last year was a fantastic year for books and podcast. I should pinch myself - I read 15 books in total i.e. more than 1 book per month 😍. I wish to continue the same momentum.

System:

Read a non-fiction/non-technical book using Kindle everyday before bed - 10 pages.
Read a technical book (hard copy) - every day during bfast or lunch.
Order 2 books every month - lets push the envelope.👍
- shroff publishers and Amazon

Personal Finance:

One thing that I am pathetic is managing my self finance. Mostly I have been lazy (or) coding was more fun.

I don’t plan to make any drastic changes - but just the knowledge of investing and managing - I belive would be useful.

System:

Start reading some finance books
Gather list of getting started books

Conclusion:

Goals are good to know where to go (read Compass). I want to always look back and be happy about the progress made - than being worried about the Gap that is left to achive the goal. Again: enjoy the Journey and the destination would become immaterial.

Retrospective - Year 2020

Fri, 01 Jan 2021 00:00:00 +0000

Happy New Year - 2021

What went well 👍

This year has been more personal challenges than technical due to the Covid19.

Marathon:

Ran my first 10K Marathon, Chennai. I ran with my wife and we finished together 😍

Blog:

Started this blog - I am super excited about. Had registered this website few years back, but never got a change to blog.

WFH:

Personally I have always admired how great minds like Scott Hanselmen manage to work from home - for more than a decade - lead a team and also stay productive. Well : if you need to learn to swim, then you should first throw yourself into the water. In a way, thanks to C19 and forced WFH - I ended up setting up my home office.

But but but, as Scott says this is NOT remote work its quarantine work 😉. I don’t quite belive life would be the same (pre C19 days) again, and I am up for the challenge.

Reading:

One of the best habit I have developed this year - reading books consistently. Read ~15 books (pinch myself, record - usually its 1 or 2) this year. Ensured - I read every single day before I go to bed. Thanks to the book Atomic Habits and my bookmate best friend(Mr.P). When I look back, I feel guilty I didn’t start this early - anyway it’s never late than ever.

Home Sweet Home:

All our plans went into a array due to the lockdown - there was a 2 week mild relaxation, took a risk and we sneaked into our new house. This was a dream come true - we always wanted to move into a community esp for my daughter and here we are.

Niece (s):

Happy to welcome the two little ones (T and G) into our extended family - Stay blessed!!

Neck Pain:

6 different doctors (physio, ortho, chiro, homeopathy, ayurveda - you name it) * 1.5 yrs - no luck. Finally met a spine surgeon - took an MRI and figured out it was just LOW Vitamin D damm!! What an AntiClimax 😱. Sometimes we (including the doctor) do complicate things - all that was required was just a simple blood test - patterns can sometimes be misleading (can we use - AI? take care of the Training set).

Hackathon Judge:

I was part of the Judging panel for our first 100% virtual hackathon- pinch myself 😎. I was a participant in the last 2 hackathons at KLA - won prizes. But this time I was sitting on the other side of the table - asking question and digesting new solution and prespectives - it was such a wonderful experience.

What could have been better 👎

Meeting:

There were days I had ~14 planned meeting in a single day 😞 and an Average of ~9 meeting in a day. This might not be sustainable - I would need to figure out a better way to manage these overheads.

Journal:

Started on Journaling this year - actually tried Bullet Journal, it was nice. It gave me a good picture to reflect how the day (month) was and also plan what were my important task for the next day - but sadly I wasn’t consistent 😦. Yr 2021 - I hope to be better.

Badminton:

Wanna get back to start playing Badminton regularly - I love the game. After the game your mind is so clear - its my best form of meditation.

Fracture:

What an end to the year : a hairline fracture on my foot - casted - 4 weeks - home quarantined 😟

Fittr:

I signed up for FITTR program, but unfortunately - the fracture came in between 😞. Next year I will come back stronger.

Conclusion:

2020 was a mixed year with lots of learning both professionally and technically. Looking back - I dont have any real regrets and I hope to have an even better 2021.

Docker - DotNet 4.8 - Getting Started

Wed, 30 Dec 2020 00:00:00 +0000

Source Code: docker-try-net48

Goal: Develop a .Net48 application - coding would happen in the host while its run inside the docker - live

Prerequisite: Docker Desktop for windows is up and running and you are using Windows Containers

There are tons of documentation online, this is just me getting started.

Base Image:

There are two base images supported officially by Microsoft for .Net windows application.

Windows Server Core
Windows Nano Server

In this example: we are gona work on a .Net Framework 4.8 - console application. We would be using the DotNet Framework 4.8 image which internally uses servercore:ltsc2019-amd64 as base image.

Let first pull the image, into our local image cache. This would take a while as the size is huge - get yourself a coffee.

$docker pull mcr.microsoft.com/dotnet/framework/runtime:4.8

Once pulled (image is available in your local cache) - you are gona see an entry with Tag 4.8 in image list

$docker image ls

Sample App:

Here is a sample app to demostrate the usage. Clone the repository. Open the solution jacapp.sln in VS2019 and compile it. Its a simple app which starts two threads and exits after 10 seconds

One thread - print the count and the thread ID - to the console every second
Second thread - write the same count and thread ID - into an output text file every second

note: the output file would be on a volume mounted drive with the host

Dockerfile:

Let’s look at the contents of the dockerfile:

Specify Base image we plan to use - in our case we are gona use the image (4.8 Runtime) we have already downloaded (pulled)
Specify the Working Directory where all the commands would be run. Think of it as running $CD /d c:/myapp, before the EntryPoint is exectuted
Specify the Command to be executed inside the working directory. In our case its the console app, so we run the output exe jacapp.exe

Build Image:

Let’s build a docker image. Run the command from the directory which has the Dockerfile

$docker build -t testapp .

Run Image:

Items to note:

–name: name of the container image instance
-v: volume mount; here we mount the output “bin” directory with a specific directory inside the container “c:\myapp" .
-rm: remove the container instance once the program exits

$docker run --name jacinstance1 --rm -v D:\dev\dkr\docker-try-net48\jacapp\bin\Debug:C:\myapp\ testapp

Instance of the containter running:

Console output:

Contents of the file writted on the mounted drive:

Well thats it! We have our application running inside a container - Now we could edit your project (try updating the console output text), compile and directly docker run the image i.e. NO docker image re-build required!!

Code Reviews

Sat, 10 Oct 2020 00:00:00 +0000

While working in a development team; being part of the code review is not an optional thing. Depending on the size of the org/team and the # of features in parallel - I am sure, we will get multiple code review requests in a week.

I often ask myself, can I provide valuable feedbacks on every review I am part of? Most certainly NO! lets be honest here.

Then the next question is - why are you added into the review? well I could think of two reasons:

FYI: Let the team know I made this change; there could be depenencies (another developer; working on similar files)
Status update (stakeholders)

Not an active reviewer:

If you think, you don’t have the right context to contribute. It’s good to be explict and call out - Sorry, I don’t have enough context to add value. I would leave the review to the experts and recommend the author to mark you as Observer instead.

Note: code review is not a place to train; where you discuss and use it as a learning form.

Add Value:

This is critical and most important objective of a code review. Pointing out each and every spelling mistake and intendation fix - might not add much value; in which case - propose them to run it through a FxCop/style/code analyser utility.

You absolutely need to be aware of the context before providing feedback 👀. If you are not sure about the specifics, setup a meeting and request the developer to walk you through. At the end of the day - code is the ground truth (irrespective of what was aligned). Few example of real Value could be:

Pointing out specific use cases not handled
Usage of design principles (SOLID) - coupling/cohesion etc
Adding: Validation, Logging, Tracablility etc…
Handling: Errors, Exception, Leaks, Performance etc..
Code Coverage, appropriate tests (note: test should also be as clean/maintainable as SUT)

Assumptions:

This could at times come back back and bite; I have been there 😕. If you are not sure about a specfic change again: call out explicitly and confirm - if your understanding is accurate.

As authors - we do want our code to be non ambiguous. But, since there is always a context and depenency in the code - its hard to describe everything. Maybe better naming could help? well: 😏

There are only two hard things in Computer Science: cache invalidation and naming things.– Phil Karlton

No Condescending words

Thanks to Doctor McKayla for this.

Words such as “just”, “easy”, “only”, or “obvious” can come across belittling and condescending. It’s a good practice to remove those words from your feedback. Most of the time, they do not add any value….. research shows that people have a tendency to interpret written language in a negative way.

Personally, I am cautious to ensure my review comments looks like a conversation - than a direct command.

More the number of Reviewers:

I remember receiving emails like “please review my change set” sent to a DL which had 30+ developers. Do you think all 30 members would have all the required context or would you ensure all 30 will follow up and get the context and then do the reivew? Less likely! 🤔

Maybe there is a perception that more the reviewers; more the # of valuable feedback. But, what if its the other way - every one assumes its someone else responsiblity - zero feedbacks 😟.

Separate Feature Addtion and Refactoring:

Many at time its intriguing to refactor the code while we are working on a feature. We should - its always a good practise to clean up and make it better.

My only argument here is: If you have both - the feature implemetation and refactoring in the same changeset (esp on a large code base, spanning files/dependencies) - It becomes a challenge to follow the actual functional change.

Recomendation: split them into two changeset like StackedPullRequest: Personally would prefer (2) followed by (1)

Only refactoring. No feature addtion
Only feature addition

Unsure of the reviewers?

If you are already part of the team which maintains the code; no brainer. You are here since you are not sure - simplest tip Go for history. I have done this more than once; say making a change in deploy scripts maintained by operational team. Look for atleast 2 developers based on change history (again: don’t stop with one) of the file(s); secondly: make sure you look at their org structure and see if they belong to the operational team (If I get to deliver and someone else sends me a new review request 😑); finally: just to be sure add their solid line into it (why: well, you are unsure if you would get a response 😜).

Approval Enforced:

Few companies have strict code review rules like: only a pull request that has 2 or more approvals - would be merged into mainline. I feel this culture/process is awesome; as there is definetly a sens of responsibility and ownership on the part of the reviewers - at the end of the day again (would say it multiple times) “Code is the Ground Truth” (irrespective; what ever the feature/design document says).

Conclusion:

I personally belive, Code review are not optional. Anyday - another pair of eyes would give you a different prespective. While, we as developers are more focused on getting thing done, there are chances of scenarios getting overlooked - effective code review enable us to bridge this gap. This defintely requires commitment and trust between authors and reviewers.

Building a culture toward delivering quality code with code review as pivot is critical.

Check out: Doctor McKayla; there are lots of good content on this topics.

Apache Airflow and Regression Monitoring

Fri, 02 Oct 2020 00:00:00 +0000

Introduction:

Airflow is a platform to programmatically author, schedule and monitor workflows or data pipelines. It was originally developed and open sourced by Airbnb, later joined Apache Software foundation’s incubation program in 2016. Workflow is a sequence of tasks defined around Directed Acyclic Graph(DAGs) – which could be started on a schedule or triggered by an event or using Command line interface. Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiate pipelines dynamically.

Components:

Metadata DB:

Stores information’s like job status and task instance status.

Scheduler:

Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. The scheduler is the brains behind setting up the workflows in airflow. The execution time begins at DAG start date and repeat every schedule interval.

Web Interface (UI):

Airflow ships with a Flask app that tracks all the defined workflows and lets you easily change, start or stop them. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues.

CLI:

Airflow has a very rich command line interface that allows to test, run, backfill, describe and clear parts of your DAGs

Concepts:

DAG:

A DAG is the container that is used to organize tasks in a way that reflects their relationship, dependencies and set their execution context and order.

Operators:

Operators are the worker that run the tasks. Workflows are defined by creating a DAG of operators. They are broadly classified into three – Sensors, Operators and Transfers. Airflow provides many prebuild operators for many common tasks and new operators can be created by inheriting BaseOperator class.

Tasks:

Once an operator is instantiated, its is referred to as a “task”. Each task is user defined and responsible for performing a specific operation in the workflow. Instantiating a task requires providing a unique task_id and DAG container. Task can be python function or external scripts that could be invoked.

Example:

In the example, we show case - how Airflow could be used to express a workflow that can be used to generate the statistics/ report as part of end-to-end regression test suit; which involves multiple systems to work together. A traditional approach would use something very basic like bunch of batch scripts w/o CRON. But the challenge is - it would very easily get tangled and developer would spend a lot of time to figure out where the log files are or what failed and why/who owns what. Airflow helps solves this problem by helping in orchestrating your processes, managing the logs and really good dashboard with visualization of what failed and much more information.

References:

Python Plugins with Topics

Sun, 13 Sep 2020 00:00:00 +0000

Source Code: PyPlugins

Any infrastructure should always have the capability to extend itself. It would be better if the functionality is added by a contributor - who is not part of the core team. And it doesn’t get in the way of core components - development, compilation and deployment.

The concept of plugin has been around there for quite a while - Visual Studio/VS Code all has plugins (aka extensions). Basic idea here is to add new functionalities - by just deploying a new dll, jar or py modules.

In this proposal using python, we have a root folder named Plugins - the infra would enumerate the Plugins directory to add functions. The functionalities register themselves with a Key - let’s call them TOPICS.

Well why Topics? I am borrowing this idea from Messaging Queue infra like ZMQ and Kafka… So, that we could create a Topic to Function mapping - and we would be able to map caller to a MQ subscriber.

Let me walk through each component:

Base class

Base class that each plugin must inherit from; this class exposes couple of items 1) List of Topics 2) Execute Method - which your plugin should implement.

class IPlugin(object):
def __init__(self):
self.description = 'UNKNOWN'
self.topics = []
def execute(self, topic, argument):
"""The method that we expect all plugins to implement. This is the
method that our framework will call
"""
raise NotImplementedError

Example Plugin : Calculate

Calculate Plugin - exposes two functionalities: add and subtract. For the client its exposed as two topics. The execute function takes in topic and the argument. Based on the topic the respective plugin could dispatch it to sub-functions within the plugin.

class CalculatorPlugin(IPlugin):
def __init__(self):
self.description = 'Calculator'
self.topics = ['Add', 'Subtract']
def execute(self, topic, args):
if topic == "Add":
return self.add(args)
raise Exception (f'Topic: {topic} has no mapping function')
def add(self, args):
count = 0
for index in range(0, len(args)):
count += args[index]
return count

Service Discovery:

The infrastructure would enumerate the plugin base_directory and try find sub class of IPlugin. Create instance of the sub_class and register to the store: MAP<topic, instance>. Infra should be able to directly call the execute API, on the instance.

class ServiceDiscovery(object):
def __init__(self, plugin_package_dir='plugins'):
self.plugin_package_base_dir = plugin_package_dir
self.plugin_topic_instance_map = {}
self.enumerate_packages()
def enumerate_packages(self, package):
"""Recursively walk the supplied package to retrieve all plugins
"""
imported_package = __import__(package, fromlist=['foo'])
for _, pluginname, ispkg in pkgutil.iter_modules(imported_package.__path__, imported_package.__name__ + '.'):
if not ispkg:
plugin_module = __import__(pluginname, fromlist=['foo'])
clsmembers = inspect.getmembers(plugin_module, inspect.isclass)
for (_, c) in clsmembers:
# Only add classes that are a sub class of Plugin, but NOT Plugin itself
if issubclass(c, IPlugin) & (c is not IPlugin):
print(f' Found plugin class: {c.__module__}.{c.__name__}')
cls_instance = c()
for topic in cls_instance.topics:
print(f' Registering Topics: {topic}')
self.plugin_topic_instance_map[topic] = cls_instance

Execution

Now that we have a Map<Topic,instance>. When a call comes in - it would have a topic and the args. Using the Map, we could get the corresponding instance and call by passing in both the topic and args. This is similar to delegate (C#) or function pointers(in C).

class ServiceDiscovery(object):
def __init__(self, plugin_package_dir='plugins'):
...
self.plugin_topic_instance_map = {}
def execute(self, topic, argument):
if topic not in self.plugin_topic_instance_map:
raise Exception (f'Topic: {topic} is not registered')
return self.plugin_topic_instance_map[topic].execute(topic, argument)

Unit Test:

Writing unit test is not optional. Well, I am a fan of TDD 😉

def test_cal_plugin_add_func(self):
# Arrange
ser_dis = ServiceDiscovery()
# Action
result = ser_dis.execute("Add", [1, 2])
# Assert
self.assertEqual(result, 3)

Deployment:

Adding a new plugin should be as simple as

Copy and paste a new directory under the Plugin base directory
Directory should have a class which implements IPlugin

Conclusion:

Building a comprehensive plugin infrastructure is non-trivial; look at Visual Studio - you could override pretty much anything, starting from adding intellisese to a new compiler tool chain. Here, we are just look at a small tip - to get started - on having a python based plugin. Always starting off any infra project with the idea of extension in mind - is good to ensure cleaner responsibility separation.

From the SOLID principle : O -> our software should be Opened for extension but closed for modifications 😍

Minimalism is not Starving

Sat, 22 Aug 2020 00:00:00 +0000

Getting Started: Netflix Minimalism Documentary. There are lots of articles and blogs online on Minimalism, so I am not gona give any gyans on it. Just my two cents on its interpretation and how it has helped me in a few ways.

Let’s begin: Every time I bring up minimalism in a conversation, the initial (read, instinct) reaction has been - come ‘on I work really hard and earn money to be happy and happiness is definitely in getting the best of the things I like - Ok, I agree, the only question I have is - how many of each?

There is always this strange dilemma of need vs. want. We always want something - not because we need it, just because either it looks good or someone gave good reviews on it.

Its hard to acknowledge the fact that we all buy more than our need - unless we go through a declutter exercise. Below is the list of my declutter exercise - Not is any specific order… just the dump.

Gadgets:

Oculus VR headset and Echo Show - BTW, I didn’t buy them both - received as a gift for presenting at an engineering conference. I used the VR headset only once (yes, just once). We never felt good about using it, but neither was I reluctant to give it away. I thought someday I will use it, and after ~1.5yrs - that day never showed up and it was still eating dust. Thanks to minimalism; I had the courage to sell it and bought something I would use every day for my new house. Same with Echo Show, its on it way out.

Stationary:

Just the intent of de-clutter; i realized - I actually had a pack of pens 10nos (arg! my Frugal Instinct) + pencil(s) + sharpener(s) + rubber(s) lying on my desk - btw, they were my own stock and not my daughters 😉. Similarly, with notebooks: 3 + 1 (rough + Journal) + sticky notes etc..

Question: Do I need all of them? Obviously NO!. Now - I just have 1 notebook + 1 pen Period!! If my pen stops working or I lost my notebook - I will write it down on some digital platform (desktop or mobile -which doesn’t feel good), until I buy a new one. Thats Ok.

Headset(s):

There is a sense of guilt as I type this - I had accumulated almost 5 headsets (all working) and all for the same purpose - attend meetings. Minimalism enabled me to take a note:

OnePlus; Anker Soundbuds Slim; JBL Reflect Mini; Sony WH-1000XM3; Bose SoundSport

All 5 of them were lying on my desk. There were times; I kept juggling around with each one - as I wasn’t sure which one had charge. On hindsight - I couldn’t convince myself, what was so motivating to buy (read upgrade) a new one. It could have been some fancy ad or one bad experience.

Now I just use ONE and I have the discipline to ensure I charge it each and every day at night - so that it is ready and available for the next day. For now the remaining ones are packed together and tucked away - I might sell it soon.

Shoes:

I had 5 (2 Running and 3 Badminton) - I don’t remember the last time I had played badminton; and about Running? Stop joking. Donated 3 of them and I am left with 2nos (1 each).

Question: Why did I buy 3 Badminton shoes? Well, if you have been playing in a club - this is bound to happened (depends on your will power) -as your fellow mates might advise 1) Cushion in your shoe looks very slim, it might not take the impact and damage your knee 2) The grip doesn’t look good, you might slip and hurt yourself etc..They were right, but the point is - There is ALWAYS something better out there; and you will never feel the sense of satisfaction…

Only Keyboard:

No mouse? Seriously?- Agree, this might sound a bit weird. I started with putting my mouse in the drawer. It was pretty challenging when starting off - but it’s possible.

What did I do? Started learning and using VIM extension - like For Chrome and For VSCode. Browsing without mouse is a strange feeling, but its awesome - try it out, you might not go back to using a mouse! - also, makes you feel a bit geeky 😁. BTW, how is this related to Minimalism? Well, I had 3 mouse (fear of - it stops working and lockdown 😟)

How VIM helped me as a developer (read, coding)? I will write a separate blog. BTW, VIM is 30yr old.

Clothes:

I don’t think anyone can escape this. Buying new clothes for a festival is not optional, and we have quite a few of them + sale. A Sunday evening declutter exercise - to my surprise I was able to get rid of ~60% of my wardrobe (donated), esp. the once which I thought would fit me (read, I will slim down) someday - and it’s been years. Now my cupboard looks very familiar (read neat) - I know exactly how many clothes I have and I can actually see each one of them 😍.

I have just started off with Minimalism, not sure how far and long - I would get along - But, for now the change it has brought it… I am really liking it and esp I am NOT STARVING!!

VLog - First Bicycle Ride

Sat, 01 Aug 2020 00:00:00 +0000

Technically it’s my first bicycle ride after ~20 years. I used to drive a lot during my school days. We used to cycle almost everywhere we went - school, tuitions, meet friends, cricket in beach, movies etc..

Note: Locally we use Cycle (for Bicycle) and Bike (for motorcycle).

I never really had the motivation anytime recently to buy a Cycle - but I always had a good feeling of driving one (read, old memories).

Fortunately I got hold of a cycle through a common friend (used one) - and I have to say it was pretty cool - Scott Aspect 950 an MTB. Honestly I wasn’t aware of this brand until I bought them 😉.

Lot has changed since my cycling days in school. First thing - there were gears and lots of accessories sticking around the cycle - still trying to figure out what each one is for.

Chennai is usually know for its hot weather. We jokingly comment, Chennai has three beautiful seasons - Hot, Hotter and Hottest 😄.

You remember those Fridays, where you slog yourself out and don’t really remember at what time you went and crashed on bed. The next day when you wake up - you kind of feel blank - It was one such Saturday. Evening around 5pm (Jul/26/2020) - the Chennai weather was unusual, it was cloudy and surprisingly with pleasant breeze - while sipping on my Tea - had a strange feeling to travel with the beautiful clouds.

I decided to go for a ride. Due to C19 the roads were pretty empty (unusual for Chennai). Before I realized, I had already driven 15 kms; well 15k it’s that long - agree, but for a first timer -it was, I started to feel the strain on my knees. I forgot to carry water 😓 - was kindaa exhausted. In Chennai we have a lot of tea shops (well, they also serve bajji and I love it etc..) - and I am usually OK to go grab one, stand on the road and sip it. But, due to the pandemic, I wasn’t very comfortable.

All that I could do was to just hang in there and think how to get back to home. Well, there were no other options - I have to drive all the way back. It started drizzling and I wasn’t prepared for the rains (no rain coat) too 😟. I started to drive back home with multiple pit stops almost every 2kms - luckily it was only drizzling, reached home around ~8pm.

After I reached home, took a hot shower - ate something - and crashed into my bed and woke up only the next day ~9am. I have to say it was one of the most pleasant feeling I had in decades - 12+ hours of uninterupped sleep wow!!. I don’t remember the last time I slept so long - it was DEEP SLEEP 😴. It was an unusual feeling; there was body pain - but I felt so good.

Honestly - Just for the deep sleep - I would do this many more times. I hope, I can cycle for the rest of my life!! - Joy of cycling ❤️

I feel guilty I didn’t click pictures, maybe next time.

Version control your install directory - GIT

Sun, 05 Jul 2020 00:00:00 +0000

What problem are we trying to Solve?

Context: Desktop software installed on client computer, inside a private network and no access to internet.

Use Cases:

Shipping and applying multiple patches for a given version of the software.
Reverting a patch
Keeping track of patches installed
Check for sanity of the installation and tack manual edits (configuration files etc..)
User can apply a temporary patch on a server (i.e. common machine) and test, once done revert

How are we trying to solve?

Bring the content of the install directory under version control. We will be using GIT version control, to explain how it could be achived.

PreRequisite “git” should be installed and available in the path

Why use GIT?

Git is a distributed version control system designed to handle everything from small to very large projects with speed and efficiency.

So, with Git any local folder can be converted into a git repository by just running the init command.

D:\Foo\1.0.0>git init
Initialized empty Git repository in D:/Foo/1.0.0/.git/

Now your installed directory is version controlled. You can add files, commit, create branch(s), switch branches etc… Since Git is distributed - N0 Server nor Internet required 😄

At any point of time, you could use status to check if the install directory has any modified, new or deleted files.

D:\Foo\1.0.0>git status
On branch base_install
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: config/params.toml
Untracked files:
(use "git add <file>..." to include in what will be committed)
config/menus.toml
config/languages.toml

You could use git diff command to compare changes across two branches like base_install vs patch_1

D:\Foo\1.0.0>git diff base_install patch_1
diff --git a/Config/params.toml
index b32de60..e8e1694 100644
--- a/Config/params.toml
+++ b/Config/params.toml
@@ -29,7 +29,7 @@
- main_menu = {align = "l", show_logo = true}
+ main_menu = {align = "l", show_logo = false}

Creating a branch is a trivial operation. As a user you could

Create a new (personal) branch from current (master) branch
Apply the patch files (edit or new) -> Commit your changes to your personal branch
Run your tests
Once you are done, switch to the original (master) branch

If you observe - you don’t have to delete your patch - at any point of time when you have server time you could come back and switch to your branch…

D:\Foo\1.0.0>git checkout jac_logo_patch
Switched to branch 'jac_logo_patch'

Installer workflow and Scripts sequence…

POST INSTALL

Installer should copy this file into the root of destination directory and execute
Example: “D:\Foo\«_Build_Number_»”
Run this file post installation

Steps:

Initialize git.
Create a .gitignore file.
Add contents into .gitignore.
Create a branch base_Install.
Commit all contents into git repo with msg Base Install Commit.

@echo off
echo Initializing Git
call git init
echo Add git ignore files
call touch .gitignore
echo Add files to .git ignore for installer cmd files
call PostInstall.cmd >> .gitignore
echo Add all the files and create a initial commit named Base Install. This would take a while!!
call git add -A && git commit -m "Base Install Commit"
echo Creating a new branch base_install
call git branch base_install

PATCH INSTALLER

PRE INSTALL

Copy this file into the root of destination directory - D:\Foo\«Build_Number»”

Check if branch exists
Add all existing UNTRACKED FILES into the current branch
Add add a commit message PrePatch_<PATCH_NAME>
Add the current branch to a known file LastKnownWorkingBranch; useful for un-installation
Create a new branch PATCH_<PATCH_NAME>
Switch to new branch with PATCH_<PATCH_NAME>
Apply the patch using git am. Note: am would allow you to sign off an applied patch. This maybe useful for future reference.
Add all existing UNTRACKED FILES into new/current branch i.e. PATCH_<PATCH_NAME>
Done - we have a new branch with all patches

REM TODO: This could be an enviornment variable
set patchName=Patch_1
set branchName=Patch_%patchName%
echo Check if git installed
call git --version
if ERRORLEVEL 1 (
echo Git not installed!!!
goto :Cleanup
)
echo Check if valid git repository
call git rev-parse --is-inside-work-tree
if ERRORLEVEL 1 (
echo Not a valid git repo : %cd%
REM TODO - Should we create a new git repo? or fall back scripts to use .keep solution
goto :Cleanup
)
set commitMsg="PrePatch_%patchName%"
echo Add all un-tracked files and create a
call git add -A && git commit -m %commitMsg%
echo Add the current branch name into a temporary file - LastKnownWorkingBranch
call git symbolic-ref --short HEAD > LastKnownWorkingBranch
echo Check if branch exits
git rev-parse --verify %branchName%
if ERRORLEVEL 0 (
echo Branch exists : %branchName%
)
echo Create a new branch
git branch %branchName%
echo Checkout the new branch
git checkout %branchName%
echo Apply the .git patch
git am --signoff < %patchName%.patch
:Cleanup
echo exit

User can always go back to the base install by switching branches.

POST INSTALL

Steps

Now we are in the new branch for the Patch
And install script has already installed all the patch files
Just commit all the changes and we are good to go!!
Patch installation done!!

REM TODO: This should be an enviornment variable
set patchName=Patch_1
set branchName=Patch_%patchName%
set commitMsg="Patch_%patchName%"
echo Add all un-tracked files and create a
call git add -A && git commit -m %commitMsg%
echo Patch installed and avaialble in branch %branchName%

PATCH UN-INSTALLER

Steps

Read the lastKnownWorkingBranch from a file
git checkout lastKnownWorkingBranch
Voila thats it!!

Note: File “lastKnownWorkingBranch” file should be under .gitignore (?) so that its shared across branches and its updated only by the patch installer.

for /f "delims=" %%a in ('type LastKnownWorkingBranch') do (
set lastKnownWorkingBranch=%%a
break
)
echo lastKnownWorkingBranch: %lastKnownWorkingBranch%
echo Checkout the lastKnownWorkingBranch: %lastKnownWorkingBranch%
git checkout %lastKnownWorkingBranch%
echo Voila we have reverted the previous install

ADVANTAGES

User could compare what has changed across branches aka patches
User could always go back to base_install at any point of time
Files are never deleted
Uninstall is just switching a branch - no file deletes.
Any number of branches can be created for personal testing.

Book - Atomic Habits by James Clear

Sun, 21 Jun 2020 00:00:00 +0000

As of today (Jun'2020) - If someone asks me if there is ONE book I could recommend - without a second thought - it would be this book. Irrespective of profession or title - this book will definitely ring a bell.

The book covers what could be considered as ideas of common sense, but presents these ideas in a well-organized, clear, practical and logical way in developing good habits.

One of the first questions in the book was - “Write down your current habits". I tried and honestly there was NONE - except for the trivial ones like brush/sleep.. It was a good ice breaker for me to start thinking.

Key Takeaways:

“Habits do not restrict freedom. They create it. Building habits in the present allows you to do more of what you want in the future.”

Habit stacking:

If you have an existing habit - which is working well. Add the new habit along/right before or after the existing one. So that the environment is already set - and you don’t need additional motivation. In my case, “writing Journal as I have tea”. Tea is a must have for me, I can never skip it - so why not add a habit while I drink it 😄. Previously I always lacked the motivation to write/continue a Journal.

Habit Cues:

Setting up the environment for doing the habits. The environment should be both enabling and welcoming you to do your habits. In my case, before I sleep, I ensure - I keep the Journal book on the dining table - where I have my tea. And Kindle on my pillow, so I see it when I go to bed.

On bad habits - one of the practical way is to reduce exposure - like hide the TV remote so its hard to find 😉

“it is easier to avoid temptation than resist it.”

Compound Effect:

Very small change overtime will have a great impact. I have started realizing that as my reading time and consistency has improved.

One month after I completed this book, as I am writing this blog. I have a list of things … Start of Day:

Wakeup, Clean the dust bin at home
5 min workout (it isn’t much, but makes a difference for me)
Freshen up, Make Tea
Write Journal Book - Plan for the day (as I have tea)
Read a technical book ( Kindle)
Off to work

End of the day:

Update the Journal, review the day
Keep the Journal at the dining table (where, I have my tea)
Pray/Gratitude
Read a non-tech/non-fiction book (Kindle)
Sleep

There isn’t much - but still I feel a change - I have something that I could call as a habit - be it writing a journal or having a planned reading time. Thanks to this book - which gave me both the framework (practical ways of implementing a system) and the motivation in building habits.

“Every action you take is a vote for the type of person you wish to become. No single instance will transform your beliefs, but as the votes build up, so does the evidence of your new identity.”

I addition to having a Kindle copy - I am planning to order a hard copy - I hope it shall be a visible reminder - to add a new Good habit and get rid of the bad habit. I believe - even if the book it doesn’t completely change your behavior, it will make you more aware of yourself.

Highly Recommended!!

Book - My Good Reads, Technical

Sun, 14 Jun 2020 00:00:00 +0000

Podcasts

Hanselminutes, by Scott Hanselman
Data Engineering, by Tobias Macey
Talk Python To Me, by Michael Kennedy
Python Bytes, by Michael Kennedy and Brian Okken
.Net Rocks, by Carl Franklin and Richard Campbell

In Progress

MongoDB Applied Design Patterns
Designing Data-Intensive Applications, by Martin Kleppmann

Done

Big Data: Lamda Architecture, by Nathan Marz and James Warren
Docker Deep Dive, by Nigel Poulton
Chaos Engineering by Casey Rosenthal
C# in Depth, by Jon Skeet
Pro Git, by Scott Chacon and Ben Straub
Clean Code, by Robert C Martin
Clean Architecture, by Robert C Martin
Working Effectively with Legacy Code, by Michael Feathers

Review - Forest App

Tue, 09 Jun 2020 00:00:00 +0000

I have been recently fascinated by this idea of getting into the flow - to get work done.

Since I am working from home (due to Covid19); personally it’s quite challenging to manage interruptions and get focussed time - to get things done - so that I get to feel productive over the work week.

I started to look around for help and stumbled upon - Pomodoro Technique.

The Pomodoro Technique is a time management method developed by Francesco Cirillo in the late 1980s. The technique uses a timer to break down work into intervals, traditionally 25 minutes in length, separated by short breaks. Each interval is known as a pomodoro, from the Italian word for ‘tomato’, after the tomato-shaped kitchen timer that Cirillo used as a university student

First: I tried to use my phone alarm - set to 25mins from now. It didn’t work - lost motivation to set up an alarm each time.

Next: Downloaded an app which supports Pomodoro Technique; it was better - but still I wasn’t completely in - I continue to get pulled into looking at my phone/emails.

Finally: As part of one of the self-help videos - I was introduced to this app FOREST - Stay focussed, be Present. First time I started using it - fell in love with it 😍.

Idea behind Forest app - is pretty simple:

Whenever you want to stay focussed, plant a tree.
Your tree will grow while you focus on your work.
Leaving the app halfway will cause your tree to Die.

Over the day-weeks-month you could see how many trees you have planted and how your forest looks like. And when you review your progress, its a visual treat to see the number of trees you have planted in your forest 😆

This app gives you - both a feeling that you are growing something (visual stimulation) and at the same time getting work done (stay productive).

I personally prefer apps - which does only one thing and does it really well - this is one such app..

Nowadays: When I am planting a tree (read, focussed time) - when I get some notification - the urge for me to check it out is NO longer there. Coz; if I need to check the notification - I would have to quit the app and if I quit the app my tree, which I have been planting all this while - would die.

A simple but a beautiful idea. Taking us closer to our goal of getting into the flow and getting things done - In addition giving a visual treat.

I am growing a new tree as I write this.. Highly recommended! 😄

Diary - 4 Movies per weekend - TO - One in 4 Weekends

Sun, 24 May 2020 00:00:00 +0000

Its week #7 in lockdown - Work From Home (WFH) due to COVID19 at Chennai, India.

First two weeks was just chaotic to be honest… Week #3, on a Wednesday - I was about to have lunch and was feeling incredibly exhausted (read burn out) 😓. Since it was already half the week, took a moment to pause and look at how productive I was in this week. To my dismay, I couldn’t add even one single item onto my Done list.

I knew something was not right, wanted to dig deep - I looked at my calendar and realized there wasn’t even a single 1hr free slot. I counted the number of meeting I have attended only this week till now - it was 25! Planned meeting, this is excluding adhoc phone calls and IM chat(s) discussion/queries.

I decided to go back a little and see how my past 2 weeks have been, and it wasn’t any better. Though the team was delivering and it was business as usual - personally felt I was low on my actual contribution/work.

In the mean time - Scott had tweeted this

 Qurantine_Work != Remote_Work

I could easily relate to him - there were lot of additional chores I was running into esp with a 5yr old kid at home. If the worlds best productivity guru is talking about burn out - it actually feels good to know “I am not alone and its Ok”.

Since these are abnormal times, there wasn’t much I could do to fix my week day plans - decided to hang in there and go with the flow - for now!!

So, I decided to take a look at how I was doing over weekends - “Binge Watching!!" ➿. As a movie buff, I always thought watching movies (more…) was relaxing, but it wasn’t. It was just adding more strain - as I wasn’t spending enough time with family and supporting house hold chores. And when the weekend actually ended I was feeling more exhausted.

I needed support - So, I went online and started looking for productivity tips. And I stumbled up this one adhoc tip, which hit the nail - Light Bulb Moment!!

Always watch movies (tv shows) - only when you have someone to accompany you

Personally, this was possibly the best tip - I decided, from now onwards I am never gona watch any movie alone - It was an extremely hard decision.

Four weeks later, when I look back - I have watched just ONE movie (with my wife). Personally, feels much better - I feel I have lots of time over the weekend. I am able to spend time more effectively - writing a blog (like this), call up friends/ family and even try a hand on cooking (making tea 😉).

At the end of the day - I still get to work.. I am blessed. No Lie. ❤️ 💻 😌

For now … Stay home, Stay Safe - and remember - This too will pass !!

Book - My Good Reads, Non-Technical

Sat, 23 May 2020 00:00:00 +0000

In Progress

Homo Deus: A History of Tomorrow, by Yuval Noah Harari
Bad Blood
The Coaching Habit
Practical Ways to Manage Yourself

Done

2021:

Backlog: GRIT, The Diary of a Young Girl

2020:

Peopleware: Productive Projects and Teams, by Tom DeMarco, Timothy R. Lister
Art of Thinking Clearly, by Rolf Dobelli
Steal Like An Artist, by Austin Kleon
Make Time, by Jake Knapp & John Zeratsky
The Making of a Manager, by Julie Zhuo
Atomic Habits, by James Clear
Sapiens, by Yuval Noah Harari
Show Your Work, by Austin Kleon
Anything You Want, by Derek Sivers
Tuesdays with Morrie, by Mitch Albom
When Breath Becomes Air, by Dr. Paul Kalanithi
Ikigai, by Hector Garcia and Francesc Miralles
The Joy of Leadership, by Tal Ben-Shahar, Angus Ridgway
Conquering the Chaos, by Ravi Venkatesan

2019:

Man’s Search For Meaning, by Vicktor E. Frankl
The Alchemist, by Coelho Paulo
Crucial Conversations, by Kerry Patterson, Joseph Grenny, Ron McMillan,

Older:

Who Moved My Cheese, by Spencer Johnson
The Present, spencer Johnson
The Situational Leader, by Paul Hersey

Backlog

Bad Blood: Secrets and Lies in a Silicon Valley Startup by John Carreyrou
21 Lessons for the 21st Century, by Yuval Noah Harari)
The Rosie Project, by Graeme Simsion
Elastic: Flexible Thinking in a Time of Change, by Leonard Mlodinow
The Unfair Advantage, by Ash Ali, Hasan Kubba
The Theory of Everything: The Origin and Fate of the Universe, by Stephen Hawking
A Brief History of Time by Stephen Hawking
The Motivation Hacker, by Nick Winter
7 Habits of Highly Effective People, by Stephen R.Covey
Pragmatic Thinking and Learning, by Andy Hunt
The Happiness Equation, by Neil Pasricha
How to Stop Worrying and Start Living, by Dale Carnegie
How to Win Friends and Influence People, by Dale Carnegie
Radical Candor, by Kim Malone Scott
Managing Humans, by Michael Lopp
The Five Dysfunctions of a Team, by Patrick Lencioni
High Output Management, by Andrew S. Grove

Started and Shelved 😞

Range: Why Generalists Triumph in a Specialized World, by David Epstein
The Book of Life, by Jiddu Krishnamurti
Leonardo da Vinci Autobiogrphy, by Walter Isaacson
The 4-Hour Work Week, by Timothy Ferriss
A Gentleman in Moscow, by Towles Amor
Factfulness, Rosling Hans
I am Malala, by Malala Yousafzai, Christina Lamb
Thinking, Fast and Slow, by Daniel Kahneman
The Monk Who Sold His Ferrari, by Robin Sharma
The Leader Who Had No Title, by Robin Sharma
The Doctor and the Saint, by Arundhati Roy
Inner Engineering: A Yogi’s Guide to Joy, by Sadhguru

Tech - My Website

Wed, 20 May 2020 00:00:00 +0000

I registered my domain name on GoDaddy couple of years back (2018). I can’t really remember the motivation to get one registered. I almost forgot about this, until I got a reminder to renew.

Thanks to Scott, I realized the importance of key stroke - the nubmer of keystroke in your life time are finite. So if you worked on something interesting - why waste it in an email which you are never sure the recipient may or not read. Instead, write it in a blog and it would be there forever and could potentially help someone new in future.

On hindsight - this website should have ideally taken less than half a day to setup, but I ended up spending almost 2 days to set the first version. Well I was just getting my hands dirty.

This is a static website:

Original Content on GitHub - my private repo
Static files Generated using HUGO Extended
Themes from Academia
Static files Hosted on AWS S3
Domain DNS AWS Route S3
Continous Deployment Github Actions

Godaddy

This is a very trivial step, go to GoDaddy. Search if your expected domain name is available. If yes, just follow the steps that show up. Note: There are many providers GoDaddy is just one of them. I dont have a good reason why I chose GoDaddy or I haven’t seen a good reason to try other providers (read, lazy me).

Hugo

Hugo again has extensive documentation (bit overwhelming at times) on setting up. Every theme you donwload come with an example site, all you need to do is just start deleting content.

One item I really like is the live reload - when hugo server is running (locally) - the website would be auto deployed and page reloaded with new content as and when you save your file.

Tip: Theme I downloaded was using Hugo Extended and not the default one. It took me almost half a day to figure this out :-( debugging through css generation failures etc..

AWS Configuration

This article walks you through pretty much all the steps. If you are getting started, recomend to skip the cloundfront part: how-to-host-your-static-website-with-s3

Continous Deploy (Actions)

This was my favourite part, I am personally a CI/CD enthusiast. I wanted to deploy my website every time I push changes into my GitRepo.

I used S3 Sync Actions from marketplace. It has a few additional steps like generting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to get it configured.

Important: Secure you keys, by setting it up as secure env variable in github - so that it doesn’t show up in console log

Use my icon for the website: Every theme comes with its own default icon. Since I have referenced the theme as a GIT Submodule - as part of CD/workflow build is was always overwritten. Did a minor hack - I checked in my icon into my repo and as part of build replaced the default theme icons with mine before build step.

Below is the GitHub Action I stitched together.

 name: CD
on:
push:
branches:
- master
jobs:
deploy:
name: deploy-website
runs-on: ubuntu-latest
continue-on-error: true
steps:
- name: Checkout master
uses: actions/checkout@master
- name: Update theme
run: git submodule update --init --recursive
- name: Setup hugo #https://github.com/peaceiris/actions-hugo
uses: peaceiris/actions-hugo@v2
with:
extended: true #This website is based on Hugo-Extended
hugo-version: "0.70.0" #We could also use latest
- name: copy my icon
run: cp -f ./assets/images/icon.png ./themes/academic/assets/images/icon.png
- name: Build Website
run: hugo --minify
- name: Publish To AWS S3
uses: jakejarvis/s3-sync-action@master
with:
args: --acl public-read --follow-symlinks --delete --exclude '.git/*'
env:
AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_REGION: 'ap-south-1' # optional: defaults to us-east-1
SOURCE_DIR: 'public' # Req: hugo gen - static files into this dir

Book - Anything you want by Derek Sivers

Sun, 17 May 2020 00:00:00 +0000

For the second time, I completed reading a book in a day. Agree, its small and meant to be completed in 1hr :).

In this book Derek Sivers shares his experience of starting, running and donating a million dollar company. Interestingly he started his “CD Baby” without realizing he has started a business. Yeah, its true - he just wanted to help himself and then his musician friends to try sell their independent work. It gained popularity initially - just by word of mouth - friends telling their friends; and eventually turned into a million dollar business…

There are lots of details in this book Derek walks us through - my key take aways:

Dont punish everyone for one person:

Don’t bring in rules just because one of your hundred(s) customer or employee did a single mistake.

Delegate or Die:

I could personally relate to this and its something I sincerly want to follow i.e. it should be business as usual irrespective of - I am avaiable or not in office

Delegate, but dont abdicate:

You need to be careful what and how much you are delegating esp when you have the responsiblity and ownership. Have a process checkpoint like review and ensure critical decisions are in control.

Programmer Himself:

This was my favourite part. Derek is a musician by profession. He started this website as it was a necessity just for himself. Later when the business grew, he wanted more than plain HTML on his website. Since he couldn’t afford to hire a programmer - he went to the bookstore and got a book on PHP and MYSQL programming. Until the day he sold his company - he was the sole software programmer of the company’s website… A business purely based on an online website making millions of dollars WoW!!!!. One of the points he mentioned - he liked doing it as it was creative, but he only had to learn just enough to get the things done - #LightBulb moment for me :)

Interseting Quotes

Anything you hate to do, someone else loves. So find that person and let he or she do it

Making money was never an objective for “CD Baby” and that was the reason it grew faster than he thought.

Do things which make you happy, you might do it better than you would like to !

Highly Recommended!

Book - Show Your Work by Austin Kleon

Sat, 16 May 2020 00:00:00 +0000

For the first time, I completed reading a book in a day. Though I like physical books, since the entire city is currently in lockdown due to COVID19, non essential services are not available. Thanks to Amazon Kindle, I am kindaa liking it.

I stumbled upon this 3-Books that Changed my Life - what actually caught me - author said I read couple of these books in less than an hour each.

Though the size of this book was relatively small, I should give credits to the author Austin Kleon, for keeping it interesting all through; esp. the doodles – it was easy to connect. His ideas are simple but incredibly profound, easy to grasp and put to use. I got a feeling, he is sitting right across the table and conversing with me.

I always thought one should not show your work until is finished. After reading this book, I realize more than the final product the process of doing it is equally important, as it makes you happy!!

Give what you have. To someone, it may be better than you dare to thing - Henry Wadsoworth.

My key take away - We don’t really have to be terrified of being revealed as amateurs; its perfectly fine (read normal).

Author talks about Amateurs and Curiosity - The day you presume, you know lots (read, mastery) - then the curiosity might fade away and complacency might step in. Being an amateur - you take chances, exxperiment and follow your whims - and in the process keep learning .

That’s all any of us are: amateurs. We don’t live long enough to be anything else - Charlie Chaplin

Honestly, I feel good and motivated. I will probably start writing about the things that I find interesting esp. the once I am working on. Thank you Mr Kleon, you have definitely made a huge impact.

Highly Recommended! I already sent it to few of my close friends :)

Welcome

Sun, 10 May 2020 00:00:00 +0000

If you know me and you are here, welcome back! If you dont know me and you are here, I am plesantly Surprised - thank you!

As the world is experiencing the pandemic (COVID-19), as a developer I am trying to get myself occupied with something which make me feel good.

Jacob Aloysious

An AI-First Second Brain for Engineering Teams: My Talk at KLA

The work before the work

What I mean by a second brain

Give AI the maintenance work

Five loops that keep memory useful

A resolved case is not automatically reusable knowledge

Making the boundaries concrete with a field-issue demo

Memory about people needs its own limits

Start with one decision that keeps losing context

What I wanted engineers to take away

Bringing My Blog Back with Codex Computer Use

1. Get Computer Use working

2. Inspect the existing Hugo project

3. Import the repository into Vercel

4. Deploy and check the actual website

5. Add the custom domain in Vercel

6. Discover the old AWS connection

7. Move DNS management back to GoDaddy

8. Investigate Vercel’s “Invalid Configuration” warning

9. Verify the domain and go live

What I learned from using Computer Use

Leadership - Buy In | Meetings

Establish the Context and Purpose

Collaborate with the affected audience:

Be Open to feedbacks

Covid Travel - India to SG - Dec'21

Documents Required:

Chennai:

Singapore (SG):

Note:

Recommendation:

Links:

4 hours in the Airport: Me, My Tea and My Kindle 😆

Almost empty airport - guess that was the only flight scheduled

Both Mask and Faceshield was Mandatory 😦

PKCE

Concepts:

What is Confidential Client?

What is public Client?

Why do you need PKCE?

OAuth 2.0 Normal Flow:

PKCE Flow:

What is new in PKCE?

Reference:

Designated Authentication - OAuth2.0

What is OAuth?

OAuth Terminologies:

OAuth Flow:

Prerequisite:

Actual Flow:

WebFinger

Background:

What is WebFinger:

Key Items to Note:

Example:

Take ways:

References:

Engineering Manager - 5yrs

Technical:

Talent: For each team member

Team:

Hiring:

Roadmap:

WASM Cloud

What is WASM Cloud trying to solve?

How is this achived?

Example:

Conclusion:

Rubber Duck Debugging

Software Enthusiast - Getting Started

Books:

Coding:

Unit Testing:

Design:

SOLID principles:

Refactoring:

Architecture:

Keeping up with ever evolving tech Space:

Youtube channels to follow

Documents Required: