Concurrent Dictionary and Delay - C#

Sun, 30 May 2021 00:00:00 +0000

In a high throuput and low latency enviorment - we had a peice of code in the critical path - which helped in throttling. After version #2.0 of the software was released (internally) - we saw the througput has gone down significantly almost 3X - compared to previous Version #1.0 - 4mins to 11 minutes.

We looked at the history/change sets between the versions: it was huge, 100s of commits and software was running with 100s of threads - on a 2Milliion LOC software stack. So, we were unsure which peice of code nor thread - was causing the delay. We were all over the place, trying to see if its a system issue or memory trend or some fragmentation vs memory/back pressure vs excess logging etc…. And, after 3 day (yes: 3days slog) - we found its a delay caused by the ConcurrentDictionary class!!! 😳

It was an unpleasant suprise, since we didn’t expect such a huge delay cause due to a .Net Framework class.

1. Version 1.0: [4mins]

One API with its backing data store with one Dictionary.

 private Dictionary<int, User> userInfos;
public List<User> GetUsers() {
return userInfos.Values;
}

2. Version 2.0: [11 minutes]

In version #2: We changed the data store, which was holding the user info from a simple dictionary to a two level ConcurrentDictionay.

There was some obvious reason for using ConcurrentDictionay - I won’t go into the details. But, after changing the data store- we ended up blindly updating all usages to consume from the new data store.

And the API (GetUsers) implementation was updated, interface remained the same - but: it was in the critical path.

 private ConcurrentDictionary<int, ConcurrentDictionary<int, User>> userInfos;
public List<User> GetUsers(){
var users = new List<User>();
for(var userGroup in userInfos.Values){
for(var user in userGroup.Values){
users.add(user)
}
return users;
}
}

3. Version 3.0: [4 minutes]

After looking into the usages “in the critical path” - we only required the count of the active user groups.

So, we wrote another API to just return the count. With this change, we were able to bring it back the throughput back to #Version 1.0. Huff!! 😌

 private ConcurrentDictionary<int, ConcurrentDictionary<int, User>> userInfos;
public List<User> GetActiveUserGroups(){
return userInfos.Count;
}

Learning:

On highsight, the fix looks trivial and straight forward. But the challenge was the huge debugging effort to find the root cause. And little did we suspect a .Net Framework class can cause such a huge delay and slow down overall execution time. It was a great learning for us as a team - involved in debugging. BTW, we are talking about a code base of about few Million lines of code.

BTW: Concurrent dictionary exists for a reason: Usage of it as our data store was the right choice.

Represents a thread-safe collection of key/value pairs that can be accessed by multiple threads concurrently.

If a class should be thread safe, then there has to be a lock/Mutex implemented. A lock has its own tradeoff in terms of execution time (Thread Scheduling, Prempt etc..) - so think carefully on its usages.

MongoDB - Hybrid Schema

Sun, 14 Mar 2021 00:00:00 +0000

I recently ran into a situation where in we have a session and each Session produces a lot of messages (sync) and each message has an associated timestamp and order. This is similar to a blog post (session) and its comments (messages).

Since the messages are unbounded (not sure how many would be produced). In my initial design - I had a separate collection for Messages. Each Session would have one corresponding MessageW (wrapper) document and each MesssageW document would have an array of messages (acutal messages).

With all the available data I had - the size of the MessageW document never went above 1MB - so we were all good. But unfortunately, we had a new use case which broke this assumption 😟. The session was running for hours and there was tons of messages getting generated and yes - we hit the 16MB ceiling 😓.

Obvious solution was to create to create a new one doucment for every new message. So, that we never hit the size limit (16MB) of a mongoDB document. But, since we are looking at unbounded message - there could huge (million message docs) number of documents.

So, we choose to go with the Hybrid Approach

In the hybrid approach, we go with our initial option - where we have a MessageW (wrapper) document - which has an array of Messages. Then we add a constrain on the document i.e. the size of the array should not be more than a given arbitary number. For every new message we increment the count. If the number count crossed the constraint - then create a new MessageW document, inc the page number and also update the Session (NumberOfMsgPages). The reader would get the NumberOfMsgPages and read backwards - until he hit page 0. Voila!! 😍

Important Items to note:

Session Object : will have a NumberOfMsgPages property
MessageW Object: would have a Page and Count property.

Operators:

Since: A code is worth a thousand words. Here is the code 😊

 public void InsertMessage(Message message, ObjectId parentId)
{
var builder = Builders<Message>.Filter;
// Find MessageObject: given parentId and CurrentPageNo
// And: Constraint: Number of messages is less than 1000.
var filter = builder.Eq("parentId", parentId)
& builder.Lt("Count", 1000)
& builder.Eq("Page", CurrentMessagePageCount);
// Try and Insert into existing Message Document array And Increment the Count.
var update = Builders<Message>.Update
.AddToSet(x => x.Messages, message)
.Inc("Count", 1);
// If the constraint with "Count" matches.
// ModifiedCount would be greater than Zero
var result = MsgCollection.UpdateOne(filter, update);
if (result.IsAcknowledged)
{
if(result.ModifiedCount== 0)
{
// Increment the Page count
CurrentMessagePageCount += 1;
// Update: NumberOfMsgPages in Session Collection
var sessionFilter = Builders<MySession>.Filter.Eq("_id", parentId);
var updateSession = Builders<MySession>.Update
.Inc("NumberOfMsgPages", 1);
SessionsCollection.UpdateOne(sessionFilter, updateSession);
// Find MessageObject: given parentId and CurrentPageNo-which is obviously not found.
// So, we will use "IsUpsert=true" to add a new document.
// And: Number of messages is less than 20.
var filter2 = builder.Eq("parentId", parentId)
& builder.Eq("Page", CurrentMessagePageCount);
// Try: Insert into existing Message And Increment the Count
var update2 = Builders<Message>.Update
.AddToSet(x => x.Messages, message)
.Set("Page", CurrentMessagePageCount)
.Inc("Count", 1);
MsgCollection.UpdateOne(filter2, update2, new UpdateOptions() { IsUpsert = true });
}
}
}

A obvious reaction after reading the code is - Ok, whats the big deal here?

Lets take a code walk through: (think how would you achive this in your traditional SQL DB’s)

Code never check if the MessageW document exits before Update.
- How did it work: “$IsUpsert=true” is the magic word.
- $IsUpsert=True: MongoDB would internally insert a new document - if the document doesn’t exists.
Updating counter is happening inside MongoDB:
- $inc mongoDB operator - increments a specific field - in our case count
- Hence, Client doesn’t have to read the document to know the actual count and then increment/update.
Constraint Check:
- $lt MongoDB operator would handle the constraints.
- Update goes through if the constraint is met.
- As part of the result, we could check if any doc was modified
- Cool part - the second half of the code runs only once in 1000 inserts

How did we come up with the arbitary constraint number? well, I dont think we have a thumb rule here. Since we don’t track the actual document size - you would have to look at your data and come up with the sizing.

BTW, I am new to MongoDB and learning. If there are better approaches - feel free to write back to me. Thanks.