The idea of the data mesh framework emphasizes things like decentralization, domain-specific ownership, and treating data as a product in order to address the complexities and challenges of these modern data ecosystems.”
Fern Halper, VP Research and Senior Research Director for Advanced Analytics, TDWI
The concept of the data mesh architecture is widely recognized and often sought after by data teams across a variety of industries. Given its range of promising benefits – including enhanced data democratization and business-driving data products – this should come as no surprise. But with the technical, structural, and organizational overhauls required to put a data mesh in place, this evolving architectural paradigm can easily start to feel like it’s out of reach.
This is especially true if your organization operates in a highly regulated industry. The more strictly regulated a company is, the more hesitant they typically are to adopt ecosystem-changing concepts like the data mesh, regardless of the potential benefits. For organizations subject to these stringent regulatory demands, is there any feasible way to implement and benefit from a data mesh architecture?
Answering this question was the focus of TDWI’s recent webinar Data Mesh for Highly Regulated Industries: One Size Does Not Fit All, featuring TDWI VP Research and Senior Research Director for Advanced Analytics Fern Halper, Databricks Solutions Architect Pavithra Rao, and Immuta Co-Founder and CTO Steve Touw. In this blog, we’ll explore insights from their conversation, dig deeper into data mesh challenges, and break down the feasibility of data mesh implementation for highly regulated organizations.
The Potential of the Data Mesh
First, it’s important to understand why the data mesh is so appealing. The modern growth of data volumes, use cases, and cloud data platforms has ultimately made the process of data management increasingly difficult to achieve at scale. According to Halper, the concept of the data mesh emerged as a logical solution to these challenges.
“The idea of the data mesh framework emphasizes things like decentralization, domain-specific ownership, and treating data as a product in order to address the complexities and challenges of these modern data ecosystems,” said Halper.
This is where data mesh’s potential might be enticing for your data team. The concept is built on four core principles:
- Domain-Centric Ownership: Domain ownership and maintenance are assigned to the teams who work most closely with the respective data.
- Data-as-a-Product: Data products are designed to meet the specific needs of different domain and consumer teams, ensuring purpose and upkeep.
- Self-Service Platforms: Consistent domain-agnostic access and security measures create a structure that is clearly defined and enables ease of use.
- Federated Computational Governance: The combination of centralized and decentralized governance capabilities ensures consistent data protection and compliance.
In theory, these principles enable you to create a data ecosystem that supports self-service data use across teams and use cases while maintaining oversight and security.
The benefits of this model have not gone unnoticed. In one 2023 survey conducted by TDWI, researchers found that over half (55%) of total respondents agreed that the data mesh was their best option for modern data management. This demonstrates a tangible shift towards embracing the data mesh, cementing it as more than just another buzzworthy new concept. However, when examining the data by industry, TDWI found that respondents’ confidence in data mesh dropped significantly in more regulated industries like healthcare, insurance, telecommunications, and banking – from over half (55%) to just over a third (35%).
What might be keeping these teams from giving data mesh a shot?
Challenges for Highly Regulated Industries
To find out, Halper asked webinar attendees what is holding their company back from data mesh adoption. The most common challenges included:
Regulatory Compliance & Governance
Over a third (35%) of the webinar attendees noted data governance concerns as a barrier to data mesh adoption. Businesses in fields like healthcare and life sciences and financial services are often subject to a larger number of regulatory compliance standards than other industries like retail or entertainment. This requires a greater effort in regulated industries to ensure all data use is compliant with all applicable legal and regulatory frameworks.
The best way to address this complexity and enforce compliance is through effective data governance. Still, this can be challenging enough in centralized data ecosystems. Introducing a decentralized and distributed model into the mix could easily give highly regulated data security and privacy teams a massive headache.
Platform & Technology Complexity
Similarly, 35% of the attendees shared that data mesh is too big of a change both technically and organizationally for their team to consider. This is not uncommon, as the concept of data mesh requires a large-scale reworking of both the technological and user-oriented aspects of a data ecosystem.
A change of this magnitude can be difficult for teams with established platforms and practices. Given the data mesh’s relative novelty, there’s no standard guide for implementation. What’s more, not every team will be starting from the same point – your existing framework could be cloud-based, hybrid, or still entirely on-premises. Coupled with constantly evolving data platforms and tools, finding the right place to start is daunting.
[Read More] There might not be a standard guide for implementing a data mesh, but we’ve put together a blueprint to help:Data Security for Data Mesh Architectures
Organizational Buy-In & Maturity
Lastly, 40% of the attendees claimed that their current organizational structure doesn’t lend itself to a data mesh implementation. From its genesis, the data mesh has been just as much a people-centered paradigm as it is an architectural one. In her original definition of the concept, data mesh creator Zhamak Dehghani noted that “Data mesh is a decentralized sociotechnical approach to share, access, and manage analytical data in complex and large-scale environments within or across organizations.”
Data mesh is a decentralized sociotechnical approach to share, access, and manage analytical data in complex and large-scale environments within or across organizations."
Zhamak Dehghani, CEO & Founder, Nextdata
This “sociotechnical” consensus can be difficult to achieve for modern teams, especially in highly regulated environments. Often, data teams have a concrete understanding of the technical side of data access and use, while legal, business, and compliance teams have the best knowledge about regulatory needs and business use cases. Unfortunately, these distinct areas of expertise can easily be siloed, and connecting technical, legal, and business stakeholders to support a data mesh architecture can be quite challenging.
Adopting Data Mesh Principles in Highly Regulated Industries
With a myriad of challenges standing in the way, how can teams in highly regulated industries implement a data mesh framework that balances business results with compliance?
The short answer: step by step! You can steadily, intentionally, and compliantly ramp your organization up to a data mesh model by focusing on the following practices:
Take a Governance-First Approach
Ensuring regulatory compliance through proper governance is a massive requirement for highly regulated organizations. The pull towards data mesh is influenced by desires for ease of access, but a self-service model must be built upon a core of holistic data governance.
“The cloud data security dilemma…becomes more acute in a data mesh world, because the data access decisions are no longer just managed by IT,” said Steve Touw. “Part of the data mesh goal is to delegate some of that policy management down to the data product owners without breaking your policy rules, which becomes complex.”
The cloud data security dilemma becomes more acute in a data mesh world, because the data access decisions are no longer just managed by IT.”
Steve Touw, Co-Founder & CTO, Immuta
To strike a balance between access and security in a distributed model, you should begin by determining the compliance-driven governance policies that must be enforced across your data ecosystem.
By creating and maintaining consistent data governance across platforms, you can start to decentralize data storage and access across various project- or use-case-specific domains. Building these domains upon global governance policies allows you to then develop local domain-specific policies for any additional compliance needs.
Leverage Dynamic Platforms
Platform choice is also crucial to the success of your data mesh architecture. You need to be sure that the platforms you choose are dynamic and scalable, not rigid or outdated. They’ll need to support cloud-based data storage and analysis in order to facilitate the number of distributed domains you’d like to establish.
This, according to Rao, is where the Databricks Lakehouse Platform can play a pivotal enablement role.
“A Lakehouse is not something that’s built in a silo, it’s an open foundation for all ecosystems,” she said. “You can bring any data integrations, orchestrations, or business intelligence [tools], which can be purpose built for something very specific.”
A Lakehouse is not something that's built in a silo, it's an open foundation for all ecosystems, you can bring any data integrations, orchestrations, or business intelligence [tools], which can be purpose built for something very specific."
Pavithra Rao, Solutions Architect, Databricks
Ultimately, Databricks provides the necessary flexibility to set up the right kinds of scalable and accessible data mesh domains.
With scalable storage enabling your ecosystem, your team will need a tool that ensures the application of predetermined global and local governance policies.
“You want to be able to orchestrate [policies] at a global level, but also empower data product owners to augment these global policies,” said Touw. With the Immuta Data Security Platform, your team can automatically discover sensitive data and write regulation-based policies that apply automatically across platforms at query time. What’s more, all activity is monitored and tracked, allowing your team to prove compliance at any time.
Align & Enable Teams
To ensure that policy creation and platform selection are not done in a silo, your organization needs to align stakeholders across a variety of internal teams. This is essential to the “sociotechnical” nature of the data mesh. A successful implementation – especially in a highly regulated environment – requires a hefty amount of teamwork from IT, legal, compliance, engineering, and any other teams that expect to make use of new domains.
Rao spoke of one Databricks customer creating a domain-agnostic “focus group” comprising internal stakeholders who have been able to guide compliance efforts across their data mesh. She noted that while this model “might slow down things a little bit,” putting a cross-functional focus on development helps to “make sure that the entire organization, and each data domain, is compliant with their governance policies.”
By taking the initiative to create policies informed by business and compliance teams and implemented by technical teams, your organization can build a robust foundation for a distributed, secure, and compliant self-service data mesh ecosystem. Closing out his presentation, Touw shared that with this model “your data consumers can get faster access to data products rather than waiting around for humans in the loop to approve things,” leading to both operational efficiencies and business outcomes – without sacrificing compliance.
Is Data Mesh Feasible for My Organization?
By assessing the common implementation challenges, starting with a governance-forward mindset, leveraging dynamic data platforms, and aligning internal stakeholders, even the most highly regulated organizations can take steps towards a secure and compliant data mesh implementation.
To dig deeper into the process of data mesh implementation, check out our eBook Data Security for Data Mesh Architectures. For more insights from TDWI, Databricks, and Immuta, you can watch the full Data Mesh for Highly Regulated Industries webinar here.
Data Mesh for Highly Regulated Industries
Watch the full on-demand webinar with TDWI, Databricks, and Immuta.
