Why Your Data Quality Dashboard Isn't Telling You What You Think It Is

Last week, your data quality dashboard reported 98% accuracy across all critical systems. Green lights across the board. Validation checks passed.

Your data governance team presented the metrics to leadership with confidence.

This week, finance discovered a £2.3 million discrepancy in your reserve calculations.

Both can't be true.

If your data quality metrics look perfect, why doesn't anyone actually trust the data?

The Green Light Illusion

Organisations invest heavily in data quality dashboards. Rows of green ticks, high percentage scores, passed validation checks. It creates a comforting sense of security: our data is fine.

The metrics prove it.

Yet business users still maintain shadow Excel systems. Finance still performs manual reconciliations before releasing numbers. Actuaries still spend hours checking calculations that should be automated.

The disconnect between what the dashboard says and what people actually trust reveals a fundamental challenge.

These dashboards aren't measuring what matters most.

Most data quality tools check technical correctness:

• Is this number within an acceptable range?

• Is there a value in this field?

• Does it look like a date?

These checks are necessary, but on their own -  they're insufficient. They measure whether data conforms to basic rules, not whether it's actually correct.

A postcode can pass every validation check whilst being the customer's old address. A premium amount can reconcile perfectly to the source system, whilst the source system itself has been receiving incorrect feeds for months.

Your dashboard shows green. Your decisions are based on questionable data.

What A Typical Data Quality Monitoring Dashboard Actually Tells You

Here's what those reassuring metrics typically measure:

• Completeness checks verify fields aren't empty. They don't verify the values are right. A claims system might show 100% completeness on incident dates, but if half of them are policy inception dates copied across by data entry staff taking shortcuts, your loss development patterns are fundamentally wrong.

• Format validation confirms data looks correct. Date fields contain dates, numeric fields contain numbers, rates have the % symbol. But format compliance tells you nothing about accuracy. Every single value could be technically valid whilst being completely incorrect.

• Range checks catch obvious outliers. They'll flag a claim amount of £50 million on a household insurance policy. They won't catch systematic errors that fall within acceptable ranges - like rating factors applied incorrectly but producing premiums that look reasonable.

• Null checks and duplicate detection find missing data and repeated records. Valuable, certainly. But they don't validate business logic, transformation rules or the calculations that actually drive decisions.

The problem isn't that these checks are wrong. It's that they measure what's technically simple to automate rather than what's business critical.

Technical teams build dashboards that measure technical correctness because it's objective and automatable. Business teams don't trust the data because reports don't reconcile and numbers don't make sense.

The dashboard says everything's fine. The actuary knows it isn't.

A Different Approach: From Technical Compliance to Business Confidence

Effective data oversight isn't about achieving high scores on technical metrics. It's about creating transparency across the entire data journey - from source to report  - giving your organisation genuine confidence that decisions are based on accurate, reliable data.

This requires a fundamentally different approach, though implementing it at scale presents real challenges.

The framework has three core steps:

1. Understand the Complete Journey

Traditional data lineage tools show you the technical flow: system A connects to system B, transformation C applies business rule D.

This is necessary but insufficient.

What you need instead: End-to-end process mapping that captures both technical architecture and operational reality.

For a catastrophe model, this means tracing:

• Extraction from policy administration systems (which tables?, which filters?, by whom?)

• Address standardisation & cleansing (what transformations occur? what gets dropped?)

• Export to Excel for manual geocoding corrections (which properties? what judgements?)

• Manual interventions (address lookups, postcode assignments, exposure adjustments)

• Upload to the cat modelling platform (what validation happens? what gets rejected?)

• Model output aggregation into risk reports (what further calculations happen here?)

The critical difference: You're not just documenting systems and queries. You're documenting the Excel steps, the manual interventions, the reconciliations that happen "offline" and the adjustments that get made along the way.

In practice, this looks like:

• Interviewing the analysts who actually prepare the exposure data each quarter

• Documenting the undocumented workarounds and manual geocoding checks

• Mapping the institutional knowledge in people's heads, not systems

Why traditional lineage tools miss this: They can't see the Excel step where your property underwriter manually assigns flood zone classifications for ambiguous addresses.

They don't capture the quarterly email exchange where the cat modelling team queries specific geocodes and asks for corrections. They miss the manual address validation spreadsheet that everyone relies on, but no one has formally documented.

2. Identify Inherent Risks at Each Step

Generic data quality frameworks list standard risks: incomplete data, duplicate records, referential integrity failures. These are real, but they're not specific enough to guide meaningful control design.

What you need instead: Systematic identification of the specific failure modes that would materially corrupt your critical decisions.

For a catastrophe model flow, ask at each step:

• During extraction: Are we pulling all active commercial property policies? What if policies written in the last 24 hours haven't synced yet? What if renewal endorsements update addresses after we've extracted?

• During address cleansing: What if the standardisation tool silently drops unit numbers or building names? What if “Manchester” gets geocoded to “Greater Manchester” rather than city centre?

• During manual Excel corrections: What if someone updates the geocode lookup table but doesn't refresh the formulae? What if copy-paste operations overwrite flood zone assignments?

• During model upload: What if properties with incomplete addresses get silently excluded rather than flagged? How many high-value properties are we losing in translation?

The critical difference: You're identifying risks that are specific to your data, your processes, and your business context -  not generic risks

3. Ensure Data Controls Are Adequately Designed

Most organisations have controls, but are they matched to the actual risks?

What You Need: Controls explicitly designed for your identified risks, operating at the point where corruption can occur.

For each specific risk identified, design a control using this logic:

• Risk: [Specific failure mode]

• Impact if undetected: [What decision gets corrupted and by how much]

• Control type: [Preventive, detective, or corrective]

• Control narrative: [What specifically happens, when, and by whom]

• Evidence: [What artefact proves the control operated]

Here’s an example based on the risk of ambiguous address information on the system of record:

The critical difference: These aren't generic "data validation checks." They're specific interventions designed to catch the specific errors that would materially corrupt specific decisions.

The Hard Truth

This approach is more expensive and more difficult than deploying a traditional data quality dashboard. It requires:

• Deep understanding of your business processes, not just technical architecture

• Continuous engagement with the people who actually work with the data

• Controls that can't always be fully automated

But the cost of not addressing this challenge is:

• Continuing to make decisions based on data that nobody quite trusts

• Discovering material errors after they've impacted business results

• Maintaining expensive workarounds and shadow systems

The question isn't whether your data quality dashboard shows green lights. It's whether you can trust your data enough to act on it with confidence.

Begin with one critical data flow. Map the entire journey - technical architecture and process reality. Identify the specific risks at each step. Then evaluate whether your current controls would catch those errors before they matter.

You may find that your existing controls are more robust than the dashboard suggests. Or you may discover gaps that explain why people don't trust data that technically passes all validation checks.

Either way, you'll have moved from measuring what's easy to understanding what's important.

That's where genuine data confidence begins.

*Originally posted on Navin Ahuja's Data Risk Fortnightly LinkedIn newsletter.