The 10 Commandments of an Internal Data Marketplace

Like most data initiatives, data marketplaces are meant to deliver value. But unlike those other initiatives, data marketplaces have an edge – they increase the potential of the entire business by clearly defining the roles, rules, and capabilities that lead to success.

To truly maximize the value that data marketplaces offer, you need to be in the driver’s seat and know where you’re going. Often, companies struggle when distractions get in the way or processes haven’t been defined. Here, we’ll lay out the 10 commandments to follow when implementing a data marketplace, so you can hit the ground running and get value from your data products more quickly.

Be Clear About Goals

For the purposes of this blog, we’ll focus on internal data marketplaces. Unlike external marketplaces, which prioritize monetizing data products and/or sharing them with third parties, internal marketplaces aim to make data products available, accessible, and usable throughout your organization. This is highly efficient since it reduces redundancy in data product development and publishing, allowing stakeholders across lines of business to leverage and collaborate on existing data products.

A data marketplace acts as a formal handoff between data engineering and data consumers, where the data engineering team’s work can be formally published, discovered, and provisioned to consumers. As we work with enterprise customers we see the goals of catalogs and marketplaces blurred quite frequently. It’s important that you are very clear about your goals, which requires you to be clear about who you are serving:

1. Data Catalogs are for builders
2. Data Marketplaces are for consumers

If you align to that framing, it is much easier to come to clear conclusions on the critical capabilities of your data marketplace. But when the lines between the two are blurred, both personas suffer and the critical capabilities become unclear.

Take Miro, for example. The digital collaboration platform was able to optimize performance, but still faced issues operationalizing data products:

“Despite these advancements, we continued to encounter challenges in aligning on quality with all data consumers across the organization. This realization led us to adopt a new approach, where all our data products, contracts, and expectations are accessible and created through a holistic metadata strategy.

….

Shifting focus from technical intermediaries to final stakeholders, we prioritize delivering relevant data products and their associated documentation. This approach ensures that users access precisely what they need without navigating through technical complexities.

‘Instead of +10 ARR intermediate tables/events/dashboards, I want to discover the revenue data product and all its documentation’.”

While not referenced explicitly as a “marketplace,” Miro’s holistic metadata approach separates the sausage from how the sausage is made: data catalogs are for builders, data marketplaces are for consumers.

The 10 Commandments of an Internal Consumer Data Marketplace

Now that we have our goals straight, the following critical components of a consumer marketplace are undeniable; commandments, if you will:

1. Publishing: A formal process for publishing data products to the marketplace. This is the official “handshake” where the data engineering teams formally release their hard work for consumption.
2. Publishing Criteria: We do not specifically address what a data product is or how it should be managed in this article. That is because the way it is managed, such as quality, updates, and change management, are not directly done in a marketplace – that is the job of the data catalog and data engineers. However, clearly defining and enforcing criteria that the data products must meet (sometimes termed data contracts), and what is worthy of being published to the marketplace (and continuously monitored), and potentially having an approval flow to publish it, is a marketplace capability.
3. Ownership: Assignment of who owns published data products from a value perspective, but also from an approval and compliance perspective.
4. Holistic Metadata: Be careful here, using Miro’s early attempts as a cautionary tale: holistic metadata does not mean “kitchen sink” metadata. It refers to publishing metadata that is meaningful to the consumer; in other words, meaningful roll-ups of metadata. A good analogy is knowing that your pizza was made with organic ingredients, not needing to know the 16 steps that got the harvesting of the yeast into the final raised dough.
5. Discovery: Powerful discovery capabilities that allow the consumers to find what they need quickly. With the advent of GenAI, users should no longer have to search for specific terms, but rather describe their use cases and be presented with results.
6. Request and Approve Flows: Not all data products should be available to all consumers. Data products should have stewards responsible for access request determinations, and approvals should have customizable expirations. This should also not be limited to tables and objects, but also have the ability to remove masks from columns or reduce row filtering upon request.
7. Auto-Provisioning: What good is an approval for access if the consumer can’t actually query the data? This could be the most critical capability of all – without it, nothing else matters. Ticketing systems where administrators eventually run grants are not good enough because they erode trust and create data consumer frustration.
8. Compliance Audit: Periodic access reviews and recertifications are critical, as well as at-a-glance reporting of which consumers have access to what data and why.
9. Data Product Feedback: Data engineers deliver value to the business through data products. To maximize that value, they need a way to gather feedback on data products, as well as recommendations for future data products from the consumers.
10. Return on Investment (ROI) Monitoring: How can you measure data product value? How can you know that the data being shared across your organization is actually driving business value? How can you be sure speed to insights are actually happening holistically? ROI monitoring helps answer these questions. It should be customizable to fit your goals, but based on core inputs such as user query audit, provided by the marketplace, to drive the ROI calculations.

Value Created by Data Marketplaces

So, you followed the 10 commandments of a marketplace. Now what can you expect?

Your data engineering team(s) will feel more empowered to deliver value to the business because they clearly understand:

• what to build,
• how to deliver it,
• how to protect it,
• how to delegate business ownership,
• and how to measure their efforts.

And also because they no longer need to:

• deal with managing access requests through tickets,
• maintain irrelevant metadata on non-data products in a catalog,
• and deal with compliance audits.

You data consumers will feel greater autonomy and satisfaction because they are able to:

• have a formal and user-friendly environment to find and access data,
• more easily define their priorities and goals, since data is more trusted, discoverable, and accessible,
• and provide feedback on what they want, and receive what they need.

And also because they no longer need to:

• find data through ad hoc means,
• wait for weeks to receive access,
• …and then find out it’s not what they needed in the first place,
• and have no way to provide that feedback.

The business will meet its objectives because:

• data products are meeting business needs,
• the data engineering team is motivated and can acquire top tier talent,
• and the data consumers are driving objectives more quickly and easily.

As you can see, a data marketplace is at the core of powering data-driven business initiatives that deliver results. Without it, you are missing the formal “handshake” between the builders and the consumers. Following these 10 commandments for your marketplace will empower both sides of that handshake – and ultimately lead to better data products and better business outcomes.