Full of contempt, she refused to talk to the engineers
The project was over-budget and way behind schedule exactly because she avoided developing a close relationship with the team she was relying on.
In 2016, to help them streamline internal processes on the tech side, I started consulting with Open Verse (a company with 150 employees which sold both paper books and ebooks) and had this conversation with Robin, the COO:
Robin: How long would it take to create a new report telling us which books have given us the greatest revenue per dollar of marketing?
Me: I’m not sure. I’m still learning about your internal CMS. But so far you’ve relied on MegaStars to build this system? Have you spoken to them?
Robin: I sent an email to Amelia, who is our account manager with them.
Me: Did she respond?
Robin: I think she tried to upsell me on something expensive.
Me: She’s part of their sales team?
Robin: No, I think she’s part of customer success, but you know, all of them are trained to get more money out of us.
Me: Did the prices seem reasonable?
Robin: It’s always more expensive than it should be. We fired the last firm for just that reason. Galactic Technology? Yes, that was their name. They charged us too much so I fired them. Then we found MegaStars. They were reasonable at first, but nowadays they are trying to charge us too much. I think I might fire them.
Me: Maybe, but it can take a lot of effort to find a new firm that is a definite improvement. I could start looking for a new firm, if you are certain that MegaStars has failed to meet your needs, and you cannot get them back on track. Should I invest the time finding a replacement?
Robin: No, I guess not. Can you follow up with Amelia?
Me: I could, but the more people who join the conversation, the slower the conversation will get. Direct communication would speed things up. Maybe you should ask to speak directly to an engineer?
Robin: Oh, dear, no, no, no.
Me: Uh…?
Robin: You’re not serious?
Me: Uh… well… if you’d like a fast answer to your question, the best way to proceed might be to establish a direct line of communication with whoever is going to do the work.
Robin: I’m very busy with my own work. I pay others exactly so they will do their work.
Me: Uh… okay…. but do you need this report as part of the work you are doing?
Robin: Yes, it’s urgent, that’s why I’m asking you.
Me: Okay, but, if you don’t mind me saying so, if you feel this report is urgent for your work, then isn’t creating this report part of your overall work?
Robin: For God’s sake, I’m the COO. Do you understand what that means?
Me: I hope so.
Robin: I’ve got many, many responsibilities.
Me: Of course. I wasn’t doubting that.
Robin: The COO can’t waste time trying to track down some engineer and talk to them about… uh… God knows what.
Me: Okay, so… you’d like me to track this down?
Robin: Why don’t you?
Me: I’ll go and apply maximum pressure to MegaStars.
Robin: Very good. I think that would be best.
But all in all, I didn’t think this was a good idea. For what she needed, there were already too many people in the process, and adding me into the process would slow things down, at least until I had learned a lot more about the company and its needs. I would be one more person in the process, one more person asking questions, one more person needing answers, one more person who could misinterpret a request. All the same, I asked for a meeting with Amelia. Since they were out in Ohio and we were in New York, we had a video chat.
Amelia: Hi, Lawrence, it’s great to meet you. How can we help you?
Me: Hi, Amelia, I’m glad to meet you. I need to ask you about a request from Robin. I think you already know about this. You know she wants you to create a new kind of report that will allow us to see how much revenue we get per dollar of marketing invested in each book?
Amelia: Lawrence, I’m so glad that I’ve got a chance to talk to you about this. We got the request but it’s extremely vague. I wrote back to Robin to ask her for more details but we never heard back from her.
Me: So, you’ve made no progress on this request?
Amelia: I’m sorry to say that we haven’t even started. We don’t know enough to start.
Me: Well, what is the problem?
Amelia: For instance, what does she mean when she talks about marketing dollars? Does she mean the money spent on Google Ad Words? Or on Facebook? Or in print media? Or on Amazon? Or on Goodreads? Or what?
Me: I’m not sure but I’m going to go out on a limb and suggest that she meant total marketing spend. Just add up all the money spent on all marketing efforts on each book and divide that by the revenue we get on each book.
Amelia: Right, that’s what I thought. But we don’t have access to that information on our side. I wrote to Robin about this but I never heard back from her.
Me: Okay, you mean you need access to our Facebook account, and our Google Ad Word account, and the Amazon account, and all the rest?
Amelia: Yes, that would be a start.
Me: Okay, no problem, I can help you with that. Give me a week and I’ll gather up all of the account data and get that to you. And once I’ve got that to you, how much time will you need to implement this feature?
Amelia: Well, that was the other thing that I wanted to talk about with Robin. You see, this would be a great opportunity for us to build a data lake for Open Verse.
Me: I don’t think that’s in-scope right now.
Amelia: But it is in-scope, it’s automatically in-scope.
Me: No, no, no. I’m sorry, but no. We want something faster than that.
Amelia: Okay, but what does this project involve? I mean, really? Have you thought about the details? You want us to get all of your Facebook data, all of your Google Ad Words data, all of your Amazon data, all of your print data? And your entire catalog? And all of your sales data? That’s an enormous amount of data. And it is heterogeneous, it lacks a common pattern.
Me: It is a lot of data, but does it need a data lake?
Amelia: Well, as a practical matter, where would you like to see it? I think your company uses Looker to view reports. Looker only allows queries to pull from one data source, so if you wanted to get a combined look at a campaign across multiple tools, then we should consolidate that data first. If we push it all into something like Redshift, then Redshift can be your data lake, and Looker can just pull everything out of Redshift. And, again, the main issue isn’t even the amount of data, but it is heterogeneous, it lacks a common pattern.
Me: Oh, I see. It is heterogeneous, so we need a place where we can normalize it, so we can run reports against it.
Amelia: Exactly.
Me: Is that the only way to do it?
Amelia: Oh, there is a million ways to do this, but that might be the easiest. And anyway, this is historical data that will expand with time. You already have almost ten years of data. And eventually you’ll have many more data sources than what you have now. So this would be a good architecture for future expansion.
Me: Oh, I see. Right, this might be simplest approach when we think about where we will be a year from now, or two years from now. And anyway, you need a place where you can put all this data.
Amelia: Exactly, we need a place to put it. So, we were thinking, the easiest way for us to enact this request was to build our own, internal data lake, and we will put your data there. But I’m thinking, wait, does that make sense? We build a private data lake full of your data? Won’t you want access to that data lake?
Me: Yes, of course. If it’s our data, then we would want access to the data lake.
Amelia: That’s what I’m thinking.
Me: Since you have to do it anyway.
Amelia: Exactly. It’s the way we would do it anyway, and so if we do it, then we might as well make it available to you.
Me: Okay, let me talk to Robin about this. I’m not sure she understood the scale of the request.
Amelia: I want to emphasize that I’m not trying to upsell you.
Me: I understand.
Amelia: It’s just that we’d have to build a data lake no matter what, so we might as well build a good one and make it available to you.
Me: I understand. Let me get back to you.
I then tried to follow up with Robin, but she was traveling for a week and when she got back she said she was “very busy” for a week and so it was almost 3 weeks before I could finally talk to her.
Robin: Alright, Lawrence, what is it you wanted to talk about?
Lawrence: I wanted to follow up about the report you wanted MegaStars to work on.
Robin: I’ve asked them for several reports.
Me: That’s an interesting fact that I wish I’d known before. But right now I mean the report that can tell us our revenue per marketing dollar on each book.
Robin: Yes, have they completed the work? It’s been a couple months now since I requested it.
Lawrence: No, no, no, they have not even started working on it.
Robin: What? Why haven’t they started on it?
Lawrence: Well, you haven’t approved a budget yet nor is there any agreement on the scope of work.
Robin: It’s a simple request! They should have finished it by now!
Lawrence: Okay, let me make sure you know exactly what this request entails.
Robin: This is unacceptable! I want you tell them they have one week to get this done or they will be fired! I will end our contract with them!
Lawrence: Okay, well, let’s think about this. I spoke with Amelia. She says she wrote to you and asked you some follow up questions about this request. Did you respond to her email?
Robin: I don’t recall any email.
Lawrence: Okay, well, look, am I correct in saying that you want a report that will show you every marketing dollar spent on every book, per dollar of revenue received?
Robin: Yes. And why are they treating that like a big deal?
Lawrence: To fulfill this request, they will need to gather up all of our marketing data, and that includes all of our Facebook data, Google data, Amazon data, and so on. It’s a huge amount of data. Then they will need to put it somewhere. They will be building a data lake to fulfill this request. And if they are going to build a data lake internally, they thought it would make sense to build the data lake for us, so that we can use it to build our own reports in the future.
Robin: No, no, no. This is nonsense. They are trying to upsell us.
Lawrence: I don’t think they are trying to upsell us.
Robin: It sounds like they got to you. Did you fall for their sales tricks?
Lawrence: I did not talk to any of their sales representatives, I simply spoke to Amelia.
Robin: Well, I think she’s part of their sales process. She tricked you.
Lawrence: Uh… okay, look. Let’s think about this. It’s a huge amount of data. Where do you think they are going to put it?
Robin: That’s their problem. It’s not our problem. That’s why we pay them, so that we don’t have to worry about these problems. If they don’t like our money then we can take our money somewhere else. But if they want our money, then they need to solve this problem for us.
Lawrence: They are going to build this report for us, and to do that they are going to build a data lake, and they are going to connect that data lake to some business intelligence tool. I suspect they’ll push the data into Snowflake or Redshift and then make it available in Looker. You already use Looker, because it makes it easy to create new reports, so long as the data is there in the source of data, but first we have to get all of the data into one source of data so Looker can use it. You said you wanted them to build multiple reports for us? Building a data lake and connecting it to an easy-to-use business intelligence took might actually be the cheapest way to go, once everything is considered.
Robin: I’m worried they are going to find ways to squeeze us for more and more money.
Lawrence: We can get them to commit to a fixed price. If the project is bigger than they estimate, they can take the loss.
Robin: If we refuse to fund their data lake, what other options do we have?
Lawrence: I assume they could build the report as an ad-hoc one-off project.
Robin: Is that the cheapest option?
Lawrence: Probably for one report, but if you want several reports, then the price will add up for each report.
Robin: Go get estimates for all options, the one-off ad-hoc report versus the data lake.
Lawrence: Okay.
There is an old cliche that says New York City is full of old media dinosaurs who lack the New Culture that makes Silicon Valley tech giants so innovative. For the most part, I’ve found this cliche to be as false as any other, but there are moments when I find myself dealing with what feels like an extremely out-of-date culture. This was one of those moments.
I spoke to Amelia. I asked for estimates. She spoke to her engineers. A week later they sent me some estimates. A week after that, I again spoke with Robin. Building the full data lake was about 20% more expensive than building the report in the cheapest way, but it would give us the option of creating our own reports in the future, without needing help from MegaStars. By the time we had asked for a dozen new reports, the data lake would be the cheapest option. After a long effort, I convinced Robin to authorize the full data lake.
The actual work took 4 weeks, whereas the discussion beforehand had taken 4 months. There was no good reason for the process to have been so slow. My time costs something, Amelia’s time costs something, and even Robin’s time costs something (a lot, actually) so simply having so many conversations automatically increased the cost of the project.
Every project cost too much and was way behind schedule
The whole company was dependent on the internal content management system (the CMS). It was built from scratch using the Ruby on Rails framework. Robin had overseen its development. Various teams had been working on it for 7 years. Robin had so far spent $3 million on it. The staff absolutely hated it because it was so extremely slow and buggy.
Why were things so bad?
Here was the central dynamic of the situation: Robin felt she held the power because she could terminate the relationship with MegaStars at any time, when in fact, the opposite was true. All of the problems in the relationship were because she could end the relationship at any time and she leaned on that fact as her main way of getting compliance. What she didn’t realize was that MegaStars was unwilling to commit to the long-term health of Open Verse while Robin was constantly threatening to fire them.
In fact, MegaStars was only the latest outsourcing firm that Robin had a toxic relationship with. There had been many companies before, and she had fired them all, sometimes after a few years, but sometimes after just a few weeks.
Why is this high churn rate a problem?
Any code base will develop some "tech debt," that is, mistakes which need to be carefully refactored out of the code. When you have full-time software developers on your staff, or a long-term, trusting relationship with an outside team, they often feel a sense of ownership regarding the code they are creating. Like great craftspeople, they take pride in their work. They will want to resolve any ugliness in the code as soon as possible.
Tech debt tends to accumulate when software developers are facing a deadline. The developers get sloppy and cut some corners. After the deadline has been met, then they can go back and clean up the code. After all, when they have a long-term commitment to the code, they want it to be of high quality.
When you work with an outside agency, they typically can't or won't go back and clean up the code, because the customer is not willing to pay $500 an hour for that work. Some of the better agencies try to include the clean-up work in the overall price, but then those agencies seem expensive — and so they then get undercut by other agencies that are willing to do the absolute minimum, even if that means writing poor-quality code full of errors.
More one-on-one meetings would have helped
Please note, this is not a rant about out-sourcing. I’ve seen companies have great results while working with an outside agency. The real issue is this: if your company depends on an outside relationship, then that relationship needs to be a close, long-term, trusting relationship.
There were several factors that caused things to get so bad at Open Verse:
The CEO was an industry legend, but rather elderly, so she pushed most of her responsibilities onto her COO. Robin was therefore spread thin with too many responsibilities. Robin was the one who was basically running the company.
Robin was very slow to realize how much the organization depended on the CMS. She herself didn’t use it, so perhaps she didn’t realize how painful it was for staff to have to wait 60 seconds for a page to render.
Robin thought her power, regarding MegaStars, lay in the fact that she could fire them. In fact, her constant threats were a source of weakness in the relationship.
When I say a long-term trusting relationship is needed regarding any technology or role that is essential to the survival of the business, some people mistakenly think I mean that you should trust other people blindly. However, what I mean is the opposite of blind trust: fire anyone you don’t trust, and fire them quickly, get rid of them, cut ties. Dump the dead wood. If anything, entrepreneurs are too slow to fire people. Keep the folks who you’ve learned you can trust.
If Robin had doubts about MegaStars, then she should have fired them quickly and found someone else she could trust. Her mistake was in constantly threatening to fire them, without ever actually firing them. Her constant threats left the relationship ambiguous and distrustful, precisely the thing you need to avoid when dealing with the people (whether internal or external) who create the software that the whole company depends on. What she needed to do was find someone she could trust, and then commit to building a close and long-term relationship with that person or team.
To repeat the main theme of this weblog: respectful leadership pays dividends in all circumstances. But it especially pays dividends when we are talking about the relationship between top leadership and those teams on whom the top leadership are utterly dependent. Respectful Leadership does not mean that you have to spend your time listening to everyone in the organization — in a large enough organization that is not even possible. But it does mean that when you are dependent on someone’s work, you commit to working with them in good faith, and you make time to hear their concerns and their suggestions.
Sadly, I was unable to convince Robin that she should change her habits. In 2018 the Board Of Directors fired her. She had wasted 9 years and millions of dollars building a CMS that still lacked crucial features that the staff needed, and so she had done great harm to the organization.