When outsourcing goes bad, part 6 of 16

The Old Team Returns

Jul 10, 2024

Around this time it was announced that DevModeMax had re-hired 4 of the 25 engineers who had been let go from Luganesk. I was curious about a few things. Had they been unemployed for 7 months? Did that indicate a tech recession? Were they only available because of the recession? Or maybe they were not as good as the other engineers, and that is why they were still available? Did they get re-hired at the same salary?

Why not re-hire them at AndersonRiskAssessment? That they were welcomed back to the team was a clear admission that it had been a mistake to fire them, so why not hire them back to their old positions, with their old salary, at AndersonRiskAssessment? Or was this whole effort just a way to cut the salaries of the engineers?

About 2 weeks after they started, the 5 of us were called into a meeting with a fellow named Naseeruddin, who was described as an enterprise software architect at DevModeMax. Deepika was initially in this meeting, but she dropped out after 5 minutes, after she had introduced us. She indicated that McCulley was the most senior of the 4 engineers who had been re-hired, and so he could answer all questions about the Luganesk system.

A side note: this was not the enterprise software architect that Mack had mentioned. This meeting left me uneasy for several reasons, among them the potential duplicity of effort at the level of architecture. If Arwin had hired an enterprise software architect to work with him directly, down in Atlanta, then why did we also need an enterprise software architect from DevModeMax?

Naseeruddin: So, my friends, you work for a company called... [reading something on screen] ... AndersonRiskAssessment? You work for a company called AndersonRiskAssessment?
McCulley: That's correct.
Naseeruddin: Very good. I am part of the modernization program at DevModeMax. I'm going to help AndersonRiskAssessment modernize. I need you to tell me about the problems in the software. I've been told that the software at AndersonRiskAssessment is very buggy. We need to fix this. Where are the most bugs?
McCulley: Well, I don't know that it's very buggy. We have some problems with insurance quotes that don't get processed sometimes.
Naseeruddin: Okay, very interesting. I'm writing that down. Now tell me, why don't the quotes get processed correctly?
McCulley: Well, uh, no one knows. We had some people investigating this, last year, but they were let go.
Naseeruddin: Okay, interesting, so we need to do the research ourselves? That is unfortunate. But walk me through it, where do the quotes get stuck?
McCulley: So, the quotes come in as JSON, and we initially stick them into MongoDB and then...
Naseeruddin: Why MongoDB? Why don't we use Microsoft SQL Server?
McCulley: Uh, I guess because at first it's just JSON, and we're looking for a place to put it? And MongoDB is just JSON? And we had the problem that many of our partners, the insurance companies, they were changing their APIs all the time, without warning, so we had to deal with constantly changing schemas. So we needed to just put a blob of JSON somewhere, before we attempted to validate the schema.
Naseeruddin: But we could use SQL Server, yes? That might fix some of the problems?
McCulley: I guess?
Naseeruddin: Very interesting. I'm writing this down. Okay, now continue, what happens next?
McCulley: Uh, so then we pull it off MongoDB, we validate the schema, and we do a few different things based on that validation. If the schema has changed and we no longer recognize it, then we try to other ways of querying that external partner, otherwise we send a notification to the product team, who then follows up with that partner and talks to them directly about the change. But if the schema is valid, then we put it on RabbitMQ, which is the first step in the process of real processing.
Naseeruddin: But why do we use RabbitMQ? Couldn't we use something like Microsoft Power Automate?
McCulley: Uh, I've never worked with Power Automate.
Naseeruddin: Okay, interesting, well, please continue.
McCulley: We have a variety of microservices that can process a quote based on the type of insurance it is. So, for instance, is this general liability? Or fire insurance? Or maybe this is insurance for cyber security? We have a different microservice for each of these.
Naseeruddin: And these are all Microsoft Dot Net applications?
McCulley: Uh, no, these are almost all written in NodeJS. Actually, Typescript. Our older code was written in PHP, but for the last 5 years we've written new stuff using NodeJS.
Naseeruddin: Fascinating. I can see why the code is so buggy! I'm writing all of this down.
McCulley: We have fairly good test coverage, and we have a suite of Playwright tests to protect us when we deploy.
Naseeruddin: Incredible. And yet the code is still so buggy? I'm writing this down.
McCulley: I don't think the code is especially buggy.
Naseeruddin: No?
McCulley: I think we have a few bugs, which we have wanted to fix for awhile, but the system still functions.
Naseeruddin: Why do you say the system functions?
McCulley: More than 99% of the quotes are processed correctly.
Naseeruddin: What do you mean when you say processed correctly?
McCulley: They end up in ConceptOne, which is the industry standard system, used throughout the insurance industry, to manage insurance policies, and then the financial data gets sent over to Netsuite.
Naseeruddin (clearly disappointed): More than 99% of the quotes are processed correctly?
McCulley: Actually, more than 99.9% of the quotes are processed correctly.
Naseeruddin: Oh.

At this point, Naseeruddin stopped writing things down. He asked a few more questions, then he ended the meeting early.

We never heard from him again.

What I concluded, after this was over, was that DevModeMax was eager to open up new revenue streams with AndersonRiskAssessment. Which I thought was outrageous, given that they were failing at their main job. But they wanted more money, so they were looking for an excuse to make the pitch: "Hey, your software is buggy. If we convert everything over to Microsoft SQL Server, all of your problems will be solved."

Another concern I had was the way that DevModeMax could now weaponize AndersonRiskAssessment's engineers to sabotage AndersonRiskAssessment. McCulley now worked for DevModeMax, so his loyalty was supposed to be to DevModeMax, not AndersonRiskAssessment. In this particular case, that I had witnessed, McCulley had remained loyal to AndersonRiskAssessment, or at least to the old team at Luganesk, but in the future some of these engineers might be more willing to share secrets about AndersonRiskAssessment’s weaknesses.

The crucial point is that DevModeMax has a set of concerns that are different from AndersonRiskAssessment's concerns. DevModeMax does not necessarily worry about the long-term health of AndersonRiskAssessment, DevModeMax simply wants to maximize what revenue it can get from AndersonRiskAssessment.

I'm also curious who told Naseeruddin that the Luganesk software was hopelessly buggy? Did Arwin also believe such a thing?

But having said all that, it is also true that re-hiring some of the engineers who had been fired was the fastest way to restore competence, and momentum, to the Luganesk project. This was an intelligent move.

Read the whole series:

1. But what do these glib little bullet points mean?

2. When the CTO does not trust their own team

3. Everyone is under pressure, everyone is too busy to help

4. They lie. They lie flagrantly. They lie all of the time, about everything.

5. That place is a total sweatshop!

6. The Old Team Returns

7. I am very, very proud of you. The work you are doing is amazing.

8. I blame you. You suck. You are the problem.

9. We just got $10,000 dollars!!!!

10. The Taj Mahal was built with blood

11. Like a punch to the face