Data-driven is a nice buzzword. We run around our organizations shouting that we need to be data-driven and try to wade through all our data to find the nuggets of gold we’ve been promised. We convince ourselves, as technologists, that we have big data, massive streams of data on par with Uber and we need the latest open source projects to handle this.
Businesses have empowered engineering teams to drive data projects. At the same time, I like many in the industry, had been guilty of focusing on technology in order to further my career, worried we would fall behind the rest of the market unless we adopted the latest open source.
But having spent a lot of time using both open source and proprietary software, I have recognized one point time and time again: Without proven tooling, successfully applying governance and visibility around open source technologies at an enterprise level is very hard.
In the land of the blind
Gartner stated that 60% of big data projects fail, then quickly revised this up to 85%. That’s really bad but no surprise to me. Projects which start with a business idea become consumed by technologists using technology, just for the sake of technology, who lose focus of the business objective.
To stay focused on business outcomes and to increase project success, visibility, safety, and security are key.
I recently bought a new car. One of the criteria I used to select my car was visibility. I want to be able to see all the dashboard information available and have it easily accessible. I also want good safety and security. Why should working with data be different? In my time, while working with fast data I would never have put a solution together that didn’t also have these features. Yet many happily build around projects that have these massive gaps.
For data projects, visibility comes in the form of identifying and giving the right users access to explore the payload and metadata, as well as how and where that data is being used and processed with appropriate control. Visibility, combined with control, will make or break the success of a project.
Another important consideration is to recognize the value of both technical and business expertise. I have seen extremely knowledgeable business domain experts sidelined in favor of highly paid “big data experts” (me included) across many industries. They were seen as more valuable as the technologies deployed were so complex. With this approach, they forgot that without domain expertise, the project would never be able to stay focused on the business objective.
We all stand together
Where I have seen success is when powerful open source technologies have been used while giving business users with domain knowledge the ability to self-serve their data access. A ubiquitous language, such as SQL, makes it possible for a wider array of users to serve themselves and get visibility into the data platform and data applications. Business experts who were able to discover, explore, visualize and build using data in an accessible way advanced the organization’s goals and optimized data in ways I couldn’t because they were the domain knowledge experts. The best role for a technologist is to be a technology partner to the business and enabler of the business goals.
Without building integrated data teams of business analysts and technologists, we will continue to see this high project failure rate —which isn’t acceptable in any other industry. Imagine 85% of, say, construction projects being abandoned?
Time for Data Intensity
The surest way to find our gold nuggets in a data-driven world is to make data actionable by the business. Use a data mesh architecture that is composed of commoditized best-of-breed technologies that eliminate the need to create complicated solutions from the ground up. Decentralized technology coupled with a DataOps approach, that provides governed and secure access to data, gives organizations what they need to deliver faster and be more successful with their data projects.