Duplicates reporting: the 3rd method to un-double your market data

If you manage market data contracts or expenses at a bank or other financial institution, you have probably encountered a familiar problem: duplicate data. It is not always easy to detect and it is rarely intentional, but it costs your organisation money, adds complexity, and weakens control.

At times, the complexity of market data flows, often described as market data spaghetti, means that no one in the firm knows precisely who receives which market data, from which data vendor, or why. Without that insight, accidental double purchases soon become routine. This is where BIQH can help.

The BIQH Market Data Platform is designed to untangle your market data spaghetti. In this article we concentrate on the third possibility in that journey, duplicates reporting. Before we examine it, let us briefly explain how we reach that stage.

The three methods to un-double your market data

1. Insight during implementation

The first opportunity to un-double your market data arises during step one of the BIQH implementation approach: consolidating your data needs.

When a firm connects to the BIQH Market Data Platform, we begin by creating a clear overview of your current market data structure. We map your data flows, from original sources through to downstream applications.

It is common to discover that different teams request the same data, often from different data vendors and often without realising it. By identifying and consolidating these requests, we eliminate unnecessary duplication at its source. This not only simplifies data flows but also delivers immediate cost reductions, particularly with data vendors who use consumption-based pricing models.

Want to know more about our three-step implementation approach? Read it on this page.

2. Waterfalling as a request strategy

The second possibility to un-double your market data is waterfalling, a method to optimise how you request data from data vendors.

Rather than sending data requests to all data vendors at once, waterfalling prioritises a primary data vendor. Only missing data is retrieved from fallback vendors.

This method reduces overlap, but it is not without limitations.. Duplicates may still occur, for example when fallback logic differs between business units or when historical contract choices continue unnoticed.

Want to explore waterfalling or waterfalling strategies in more detail? Read our article.

3. Duplicates reporting as a governance tool

Unlike the first two methods, which focus on operational measures, the third method is about governance. BIQH has developed a module called Duplicates Reporting, designed for vendor managers, contract owners, and anyone responsible for the bigger picture.

With this module, you gain continuous insight into:

• which data points are duplicated across vendors
• where and how those duplicates enter the organisation
• whether the duplication is justified or avoidable

By comparing equivalent data from different sources, the module identifies overlaps. For example, it may detect that pricing data for the same instrument is delivered twice: once as intraday and once as end-of-day. In other cases, the same type of data might come from two different data vendors. Sometimes this is intentional, but often it is not.

The module shows how often duplication occurs, from which data vendors, and to which downstream applications the data is sent. This enables vendor managers to assess the situation and take action where needed.

By surfacing these overlaps, you can:

• justify or eliminate duplicates
• improve data vendor negotiations
• strengthen your data governance
• build transparency for audits and compliance

A real-world case: uncovering unnecessary overlap

A major European bank used the BIQH duplicate reporting module to investigate its data vendor landscape, with a focus on ESG data. Over the past years, ESG had been introduced as a new data domain within the bank.

The tool revealed that the same ESG data was being purchased multiple times, both from different data vendors and from the same data vendor via separate contracts or request flows.

The bank’s internal review confirmed that several duplications were unnecessary. As a result, the bank streamlined its internal processes and data vendor relationships, improving efficiency and achieving measurable cost savings.

Why it matters

Duplicate data is more than a budgeting issue. It weakens your data model, increases compliance risk, and suggests a lack of control. But not all duplication is bad.

What matters is knowing where and why it happens. That is why the BIQH Market Data Platform includes complete logging of all incoming and outgoing data, ensuring full data lineage, traceability and auditability.

That is the value of duplicate reporting: it gives you the facts to decide what is strategic, and what is simply waste.

In summary

Duplicate data leads to more than only increasing costs. It adds complexity, makes responsibilities unclear, and weakens your control over market data vendor contracts and internal data flows. BIQH addresses the issue through three practical possibilities: gaining insight during implementation, optimising request flows with waterfalling, and maintaining oversight through duplicate reporting.

The duplicates reporting module is not a standalone tool. It is fully integrated into the BIQH Market Data Platform.

To learn more download the BIQH Market Data platform factsheet or get in touch with our team.