DCAS — data correlation and entity resolution
A high-throughput correlation engine that unifies heterogeneous datasets, resolves identities across dirty records, and streams match results live.
Outcome
The problem
Analysts and intelligence teams routinely work against fragmented data silos: internal databases, customer registries, transaction logs, external communication records. Each dataset follows different schema conventions, lacks standardised formatting — inconsistent phone numbers, partial names, typos, alias variations — and runs to millions of rows. Cross-referencing records by hand across separate spreadsheets, or firing naive batch queries at each database in turn, produces severe bottlenecks, missed connections and high false-positive rates in exactly the investigations where time matters most.
Approach
We designed DCAS as a high-throughput entity resolution and correlation platform, pairing a Python correlation engine with MongoDB's flexible schema storage and a Next.js 16 web interface.
The engine applies a two-tier matching strategy: an exact normalisation layer for structured identifiers — national IDs, standardised phone numbers, email hashes — followed by a token-sort fuzzy pipeline built on RapidFuzz that scores name variations and partial aliases against fine-grained confidence thresholds. Correlation runs execute on asynchronous workers that stream live progress and verified match batches straight to the browser over Server-Sent Events, so a run across millions of records never blocks the interface an analyst is watching it from.
Outcome
Days of manual spreadsheet lookups and brittle one-off scripts collapsed into an automated correlation workflow with visible progress. Analysts can watch a run resolve linkages across massive datasets in real time, then export verified match clusters and audit-ready reports directly to stakeholders.
What the client said
“A real client quote goes here, kept short. Two sentences beats a paragraph, and a specific complaint that you fixed beats generic praise. This block is not rendered at all once placeholders are switched off and no testimonial has been added.”