You’ve decided it’s time to set up a data stack for your company, but you’re unsure what’s the best way of doing so. Should you buy all the components and put everything together, build your data stack from scratch, or hand it off to a third-party expert? It’s an important decision that impacts how successful your implementation will be.
“I talked to people who had been through the ringer a bit more and asked them how you set up a data stack properly. One thing that stuck out to me was the importance of being deliberate about how you structure your data. So that way, people will speak a common language, use the same data sources, and use them correctly,” said Matt Marcus, Co-Founder and Chief Product Officer at Modern Treasury. Luckily, lots of companies have been here before, like Modern Treasury and Zeplin.
Here’s what you need to know about each option. These pros and cons will help you think through your choices and make the right decision for your company.
Buying
|
Pros |
Cons |
|
|
Pros
Fastest implementation
Buying all the components for a data stack and assembling it yourself is the fastest option of all three. Assembly usually takes a few weeks. Out-of-the-box data stacks, like Mozart Data, are also an option now. They give you all the components you need and only take an hour to set up.
Cheapest option
Buying your data stack is also going to be the cheapest. You’re only paying for the technology and the internal resources that you spend on setup. And given that it doesn’t take long to buy and assemble a data stack, the amount that you spend on internal employees is going to be much less than if those employees built your data stack from scratch. “I think a lot of these tools are going to in the long run be cheaper than headcount, especially as we see engineers are not cheap these days,” said Jessica Larson, Data Engineer at Pinterest.
Better quality than building
It’s not just faster and cheaper to buy, it’s also better. This is because you’re buying from a company whose core competency is building the technology for a data stack. They have lots of in-house experts, and they’ve devoted more time to understanding and building the technology than you probably ever will.
Cons
Won’t address needs or problems that are unique to you
The main downside of buying is if there’s something very specific and unique about your needs, you probably won’t be able to find a solution that addresses it. For example, you might need specific data connectors that are core to your business and those connectors aren’t supported by vendors. The best solutions available solve universal problems really well, but they usually don’t address edge cases or niche capabilities.
Might buy the wrong tools
When you don’t have expertise in data infrastructure, there’s a chance you might end up buying the wrong tool. The market for data tools is crowded and lots of companies have similar value props, so it’s easy to make the wrong choice. And if you do, that can have long-term consequences, such as inaccurate data and frequent errors.
Building
|
Pros |
Con |
|
|
Pros
Solves your unique needs or problems
If there’s something you need from your data stack that’s unique to you, you’ll only be able to get that by building. Maybe you need real-time data or cutting-edge capabilities. Building gives you full control over your data stack and the ability to provide the unique capabilities you need.
While you can give vendors product feedback, they’re never building just for you. They won’t create something that can only be used by you or a handful of customers because that doesn’t provide value to the majority of their customers and it can’t be easily monetized.
Flexibility
Another benefit of building is you have the flexibility to shape your data stack as your needs change. You can determine what changes are made, how the technology evolves, and what gets prioritized. You’ll always be able to ensure your needs are being met by your data stack.
Develop internal expertise
By building, you’ll also develop in-house expertise on your data stack. You’ll know everything about your tools and systems, which means you can fix issues and you won’t be dependent on consultants or vendors to help you.
Can provide enormous value at a massive scale
If your company is incredibly large, like the size of Amazon or Google, and the value you deliver is tied to your data infrastructure, then the specifics of your data stack can make a big difference. This is when building your data stack makes a large impact on your business. Also, when your company is at this scale, it’s difficult to buy a solution that can handle your volume of data.
Cons
Slowest implementation
Building your data stack takes months or even years. It’s the slowest option, and you might not even succeed. Some companies who have attempted to build their data stack end up failing and having to buy it instead. And the entire time you’re trying to build your own solution, you’re slowing down data analysis and missing critical insights that can help your business grow.
Expensive
Full-time engineers are expensive. When you combine that with how long it takes to build, the total cost of your data stack is going to be a lot more than if you had bought your data stack. If you’re hiring new engineers to build your data stack, they’ll still be on your payroll once the project is finished so there needs to be enough maintenance work or new projects to justify that.
Lower quality than buying
Building usually results in a lower quality solution. Other companies have already invested way more hours and engineering, product, and design resources into building a data stack or the components of a data stack than you will. They also have a better product because they deeply understand the core problems, and they’re able to take feedback from a wide variety of customers and use it to improve the product for everyone.
Outsourcing
|
Pros |
Cons |
|
|
Pros
Expertise
By working with a consultant or agency, you’ve essentially bought the expertise and technical skills that your company is lacking for as long as you need it. They’ll tell you what you don’t know and do the work that’s needed to get your data stack up and running.
Relatively fast
Working with a consultant to buy and assemble your data stack is a pretty quick way to get what you need. However, it won’t be as quick as buying a data stack yourself because there’s an onboarding process for consultants. They have to take some time to understand your data, business, people involved in the project, and who the decision-makers are first before they can make any recommendations.
Cons
Expensive
Consultants are expensive because the general consulting model is to charge a large markup that accounts for the time they spend marketing themselves, getting context about their clients, and getting set up for a project. Also, the knowledge they acquire on a job is no longer useful as soon as their work with a client ends. The result is you might end up paying two to 10 times more than the cost of hiring someone internally to do the work.
Incentives aren’t aligned
One of the problems with outsourcing is the consultant’s incentives aren’t necessarily aligned with your incentives. They often have your best interests at heart, but at the end of the day, their incentive is to extend the contract so they can continue working. This makes it easy to take shortcuts, which can result in sloppy implementation or bad code.
The work you initially hired them for can also stretch into a larger project or longer timeline. And when you don’t have expertise, you’re not able to tell whether that’s really necessary or not. It’s more difficult to accurately scope the project and know when their proposal isn’t the right decision for your business.
Don’t develop internal expertise
Hiring a consultant works well when you don’t need to develop internal expertise. When it comes to your data infrastructure though, having a consultant implement such a big, critical part of your underlying technology can create problems after they finish their contract with you.
Since you’ve been relying on the contractor to supply the expertise needed and know how your data stack works, you haven’t been building expertise internally. If you run into problems with your data stack or have questions later on, you’ll have to go back to the contractor.
Might hire the wrong consultant
Just as you can buy the wrong tool, you can also outsource to the wrong consultant. If you don’t understand data infrastructure well enough, you can find the wrong person to give you advice.
Now that you know the benefits and pitfalls of buying, building, and outsourcing your data stack, you’re one step closer to making an informed decision. To get additional help with your decision, watch our on-demand recording on how data leaders at Pinterest and Cloud9 Esports have made the same decision before.
