When outsourcing goes bad, part 3 of 16

Everyone is under pressure, everyone is too busy to help

Jul 10, 2024

A month later, I started working at DevModeMax. The first week was mostly orientations, and obligatory HR training videos. I did not speak to anyone at AndersonRiskAssessment, I was entirely absorbed by the corporate bureaucracy of DevModeMax.

By the second week, I was trying to install the Luganesk software on my machine, but I ran into some problems. I reached out to my manager at DevModeMax, a woman named Deepika. She was non-technical so she could not help me directly, but she sent out a mass email telling some of the other software developers to help me, and they in turn pointed me to documentation in Jira that was supposed to answer my questions. This let me install the virtual machine (VM) and the old PHP monolith app they had, which was called Darwin.

Then I attempted to do some work. I accepted my first ticket from Jira, thinking I would learn the system by doing grunt level work, at least for the first 90 days, and then once I knew the system better I could start to help train others.

I ran into another problem: the ticket I accepted required me to set up 7 other apps: QuoteDirector, QuoteIngester, QuoteStatus, and QuoteProcessor, and also the EventQueue and the Monster app and the Godzilla app. Each of these had to be installed manually. I followed the instructions but I was still missing some basic configuration information. In particular, many internal API calls were being made with empty URLs, which were supposed to be set somewhere, but I could not find out where.

At this point Deepika contacted me on Microsoft Teams and she asked me if I was done with the ticket that I had accepted. I told her that I had not even started yet, because I was still setting up my system. She told me that from now on, when I needed help, I should contact her, and she would find someone who could help me. I agreed to this. She then forwarded my email to 6 other software developers, asking them to help me. But the day ended and I did not hear from any of them.

Early the next morning I reached out to Akshay, who I knew was in India. I knew the team in India worked from 2 AM, east coast time, to about noon (which is roughly 11:30 AM to 9:30 PM their time. An oddity of time in India is that they keep time 30 minutes off the hour).

I chatted with Akshay via Microsoft Teams:

Me: Hello, Akshay. I am trying to get all of the apps to run together. I think I am missing some configuration information. Can you help me work through this?
Akshay: Yes, sure. Soon.

An hour went by.

Me: Hello Akshay. Will you have some time today?
Akshay: Yes, of course. Very soon now.

Another hour went by.

Me: Hello, Akshay. I hate to bother you again. Will you have time today? I know your day ends soon.
Akshay: I'm afraid, not today. Can we meet tomorrow?
Me: Yes, I can meet at your time. I think you start your work day when it is 2 AM in New York City? I will go to bed early, and get up early, and we can meet at 4 AM?
Akshay: Yes, that should be very good. I will have plenty of time then.
Me: Okay, good, I'll add that to the calendar in Microsoft Teams.

At 4 AM the next morning:

Me: Hello, Akshay. Do you have time to talk now?
Akshay: Very soon, Lawrence.
Me: Are you dealing with something?
Akshay: Yes. Several deals are stuck in RabbitMQ. They need to be re-queued.
Me: Our system failed to process them?
Akshay: We will talk soon.

Two hours went by.

Me: Do you have any time soon Akshay?
Akshay: Yes, okay, I have 10 minutes.
Me: Okay, let's video chat.

So then we connected via video and were able to talk to each other directly.

Me: Busy morning?
Akshay: Everything is good. But we have problems to fix.
Me: What was wrong with RabbitMQ?
Akshay: This happens every morning. Some deals get stuck.
Me: Every morning? Why hasn't someone fixed this permanently?
Akshay: I don't know.
Me: Is someone supposed to fix this?
Akshay: I don't know.
Me: What exactly is the problem?
Akshay: Customers send in deals that need to be processed, but our system doesn't read the deals correctly, so they sit in RabbitMQ.
Me: Why doesn't our system processes these deals?
Akshay: I don't know.
Me: Okay, thank you for that. Well, listen, I need to get this system working on my machine.
Akshay: You need to install the Virtual Machine.
Me: I have installed the VM. And I logged in. And the PHP app, Darwin, is running great. No problems there. But I also need to run the QuoteDirector, QuoteIngester, QuoteStatus, and QuoteProcessor app, and also the EventQueue and the Monster app and the Godzilla app. So far I've only managed to get the QuoteDirector, QuoteIngester, and QuoteStatus apps working.
Akshay: Have you set up the .env files?
Me: Yes, which I got out of the Secrets on Bitbucket, but I still seem to be missing configuration variables.
Akshay: Did you split the configuration into the correct files?
Me: What are you talking about?
Akshay: A lot of the configuration information is stored in Secrets as a single block of variables, but you actually need to split it into multiple files on your machine.
Me: Uh, how would I know that? I don't think I saw that documented anywhere?
Akshay: I don't think it is documented, but you can look for the triple blank lines. Anywhere there is a tripple blank line in the configuration, you are supposed to split it there.
Me: Uh, okay. And then do what with it? I was putting everything into a top level .env file. Where else would I put the information?
Akshay: Lawrence, I am very sorry, but we are having an emergency. Can I call you back in 15 minutes?
Me: What kind of emergency?
Akshay: I need to go. We've lost some information. We erased. We have erased.
Me: You erased?
Akshay: In 15 minutes.
Me: Okay, no problem. I understand. Let's talk in 15 minutes.

I did not hear from Akshay for the rest of the day. I later learned that someone at DevModeMax had erased everything in RabbitMQ, when they only meant to delete the quote-requests that had been stuck but then re-queued. The data was entirely lost. This included roughly $6 million worth of possible deals.

The next day Akshay said he could help but he never found the time. I asked Deepika if someone else could help me and she forwarded my email to the 6 software developers as before. I contacted them one by one and all of them said they would find some time tomorrow.

The next day went by but everyone was too busy to help.

There was a holiday, a 3 day weekend, and then I was at the start of my 4th week at DevModeMax.

Deepika contacted me on Microsoft Teams and asked me if I was done with the ticket I had accepted and I explained that I had not yet started because my system did not work. She complained that I needed to be more transparent about the problems that I was facing. I felt that I had been transparent, but also agreed that I would send her twice-a-day updates from that point forward. She then asked Akshay to help me and then Akshay said he would be free in 1 hour to help me.

When it got to be noon, I realized the team in India was gone for the day so I would not hear from Akshay. If I was going to get any help I would need to reach out to someone on the American side, so I reached out to Stratton, one of the 5 surviving American engineers. Stratton and I started a video chat and then, over the course of two hours, Stratton helped me configure the Monster app and the Godzilla app.

The next day we met again, via video, and we finished setting up my machine, so I finally had all of the apps running and talking to each other. This was essential if I was going to get any work done.

Me: Stratton, I have a question. Why do I need to configure all of this by hand? You guys have a VM. I have worked at companies where the VM is fully configured, so all I have to do is install it and, presto, I am ready to work. But here, I just lost weeks trying to get all of this configured.
Stratton: That's a good question. First of all, we had that 6 years ago. Once upon a time, the Darwin monolith app was our whole system. So you could install the VM and, presto, you were ready to work. But then we decided to move to a microservices architecture, so now we have all of these other apps that you need to install, and we never got around to updating the VM, so the installation process has become a lot more chaotic.
Me: But why don't you fix this? How much time would it take to reconfigure the VM so it comes with every app installed and configured?
Stratton: Right, we had someone working on this last summer. Kathryn. She was a great engineer. She did some good work on this, to give us a modern setup. But she was one of the people they let go in October, before she had finished. And we've had no one else who could work on it.
Me: That's a shame about Kathryn. But what about the project? Do you think anyone will ever modernize the VM? It's inefficient to have engineers waste 2 or 3 weeks trying to get setup.
Stratton: Sure. We'd like to see someone finish the work that Kathryn started. We had initially been told that someone from DevModeMax could do it, as soon as they learned our system, but so far no one has learned our system well enough that they could take on that task. But I want to ask you, are you exaggerating about wasting 2 or 3 weeks? You haven't really been stuck for 2 weeks, have you?
Me: This is my 4th week working for DevModeMax. The first week was just corporate orientation. And then I setup Darwin and that went okay. But then I tried to set up all of the apps, and I've had non-stop problems for more than 2 weeks now.
Stratton: Why didn't you ask for help from your co-workers at DevModeMax? I think we all assumed you were getting whatever help you needed from DevModeMax.
Me: I have asked my manager a few times, and she's put me in touch with people, but they seem busy.
Stratton: Oh, okay, sure. I guess that is true. I was worried about that.
Me: You were worried about them being busy?
Stratton: Yes, I have the impression a lot of them are given more work than they can handle. And so then they don't have time to help anyone.
Me: Okay, yes, true, but what is weird is that they can define "busy" however they wish. If they defined "onboard new employees" as part of their official tasks, then they could be “busy” helping me.
Stratton: The problem is with the story points. They've become obsessed with Jira story points. We have had several arguments about this.
Me: Why are they focused on story points?
Stratton: I don't know. It's driving me crazy. Apparently they've told their engineers that everyone needs to do 10 story points a week. And nothing else matters. Which leads to problems like this. The engineers at DevModeMax have no incentive to help each other because they don't get story points when they help someone else.
Me: How did you handle the work before? I mean a year ago, before DevModeMax?
Stratton: We used story points only to give ourselves a rough idea of how long it would take to build a new feature. We never used story points as some kind of individual metric. We'd commit to a certain number of story points for a 2 week sprint, but the story points were always the responsibility of the whole team, they were never the responsibility of any individual. We did that to encourage team work. And we used to do a lot of pair programming. It was a good way for us to teach each other about the different parts of the system.
Me: It sounds like you had healthy team dynamics.
Stratton: I thought it was great. And I thought we were doing good work. And we were fast.
Me: I was going to ask about that. I'm guessing the leadership at AndersonRiskAssessment must be disappointed with the way things are going?
Stratton: Oh, it's a total fiasco. Just a total disaster. DevModeMax has been with us for 7 months now, and they've done less work than what we used to accomplish in a single month. But we are hoping you can help us with this. We are hoping you'll be able to improve the team.
Me: I'll do what I can.