Untangling the Market Data Spaghetti: Understanding and Addressing the Complexity

In the complex world of market data, financial institutions often face what we at BIQH like to call “The Market Data Spaghetti.” This term describes the complicated system of data flows between market data vendors and various downstream systems and applications within financial institutions. The complexity can be overwhelming, creating significant challenges for those trying to make sense of it all. But understanding this complexity is the first step toward untangling it and finding effective solutions.

What is Market Data?

Market data covers all data related to financial markets, ranging from pricing information to fundamental analysis, ESG data, news, and more. It includes the rich stream of information generated from trading activities. Picture a busy trading floor: every bid, ask, and trade becomes a data point for financial institutions to gather and analyze. This pricing data is just one piece of the puzzle; market data covers a much broader spectrum. Below, we detail the most essential types of market data.

Pricing and Trading Data

Pricing data includes bid and ask prices, last traded prices, volumes, and opening and closing prices. Additionally, order book data provides detailed information on buy and sell orders, including prices and quantities at various levels. This data helps traders understand the supply and demand dynamics in the market.

Time Series Data

Time Series data, like historical data offers a full tick history of every transaction over a specific period. This data is crucial for backtesting trading strategies, conducting technical analysis, and understanding long-term trends.

Reference Data

Reference data provides the necessary context and identifiers for financial instruments. It includes security identifiers (e.g., ISIN, CUSIP), company data, sector classifications, and corporate actions, ensuring consistency and accuracy across markets.

Static Data

Static data refers to information that does not change frequently, such as coupon rates for bonds, dividend schedules for equities, and maturity dates. It is essential for portfolio management, accounting, and long-term investment strategies.

ESG Data

ESG (Environmental, Social, and Governance) data includes metrics related to a company’s environmental impact, social responsibility, and governance practices. It helps investors evaluate companies’ sustainability and ethical performance. For an in-depth exploration of common ESG data management challenges, read more here about the seven key issues.

News Data

News data includes real-time news articles, press releases, and other media that impact market sentiment and decision-making. It helps traders and investors stay informed about events that could affect their portfolios.

All this data must be collected, processed, and utilized effectively. This is where the challenge lies.

How Market Data Streams Typically Operate

Market data is delivered through various channels and formats to different downstream systems and applications, each with specific requirements. Real-time data provides real-time updates, crucial for fast trading and instant decision-making. Delayed data, frequently used to save costs, includes a 15- or 20-minute delay. This is adequate for many applications that don’t require immediate action. Snapshot data offers a static view of the market at a specific point in time, while streaming data delivers a continuous flow of updates.

Each type of data stream comes from multiple vendors and exchanges, each with its own format and update frequency. This diversity requires extensive data standardization and merging, which can be a challenging task. Imagine trying to combine data from different sources into a single, cohesive system. It’s easy to see how quickly complexities can develop.

Why the Market Data Spaghetti Exists

Over time, the market data landscape has evolved from simple, localized data sources to a complex, global web of information flows, as illustrated in the picture below. Several factors lead to this complicated network.

A major cause is the huge growth in data sources, especially driven by regulations such as MiFID, SFDR, etc. Financial institutions now use data from numerous data vendors, third parties, and internal sources. Each comes with unique identifiers, data models, and data formats.

Regulatory requirements increase the complexity, needing multiple data sets to comply with data quality rules. Technological improvements are moving data delivery from on-premise to cloud-based solutions. This shift introduces new challenges related to speed, scalability, and data volume.

Legacy systems worsen the problem. Many organizations still rely on outdated infrastructure not meant for modern data complexities. Dependence on specific vendors and symbology adds to the complexity. Different vendors can use exclusive symbols and codes, making it difficult to switch vendors or integrate data from multiple sources without significant reprogramming. This challenge is further compounded by the industry’s limited capacity for change and a general shortage of market data expertise.

The Problem with Market Data Spaghetti

The complex nature of market data flows presents several serious challenges for financial institutions. These challenges are compounded by the need to integrate market data into existing and new systems. Operational inefficiency is a major issue. Managing multiple data sources manually is time-consuming and prone to errors, leading to inefficiencies and frustrations. Different departments often even end up purchasing or requesting the same data multiple times, leading to avoidable costs. High costs are another concern, with unnecessary data subscriptions and the need for custom IT infrastructure and development to handle diverse data formats increasing operational expenses.

Compliance risks are ever-present. Firstly, ensuring all data usage complies with market data licenses and regulatory requirements is a complex and ongoing task. Secondly, data quality issues arise from inconsistent formats and incomplete integration, which can result in inaccurate or incomplete data affecting decision-making processes.

Conclusion

The complexity of market data flows—the “market data spaghetti”—poses significant challenges for financial institutions. Untangling this web requires a deep understanding of the data landscape, solid data governance, and innovative, flexible solutions to streamline market data management processes.

This is where BIQH steps in. We provide a complete solution crafted to tackle these challenges effectively. By leveraging BIQH’s expertise and technology, financial institutions can transform their tangled market data spaghetti into a streamlined, manageable, and highly efficient infrastructure. Our solution helps organizations manage market data efficiently, reduce costs, and ensure compliance.

If you’re interested in our offerings or have any questions, download our factsheet or reach out to us. Together, we can simplify complex market data structures and lead your organization toward efficient market data management.