Real-Time ML
Real-Time Fraud Detection Engine
A graph neural network scoring 2.3M transactions a day in under 800 ms. 99.6% precision and $47M/yr of ring fraud the rules never saw.
$ cd /services/data-engineering agent.ready · data engineering
Data Science & Data Engineering
> Data Platforms Built : 8 +
> Records Processed Daily : 100 M+
> Industry Verticals : 5 +
From raw event streams to the forecast in this quarter’s board deck.
We build the Iceberg lakehouse, the dbt or SQLMesh models,
the Dagster / Prefect orchestration, the feature stores,
and then the Chronos / TimesFM forecasts on top —
same team, one repo, one on-call. Trace any dashboard
number back to the source within a week, with OpenLineage.
What we actually build
Pipelines that move raw events into curated tables on time, every time. Streaming CDC on Kafka / Kinesis / Debezium where latency matters, good old batch where it doesn’t. dbt or SQLMesh layers with column-level contracts between them, an Iceberg / Delta lakehouse queryable from DuckDB, Trino, StarRocks or ClickHouse, a feature store your ML team can actually use, access controls your security team signs off on.
On top of that: forecasts on Chronos / TimesFM 2.0 / Moirai / TimeGPT baselines before any bespoke XGBoost, Bayesian MMM (Robyn, Meridian, Orbit, PyMC v5) for marketing, churn, attribution and anomaly models — the ones that show up on Monday standups. Experiment tracking, retraining triggers, conformal prediction on the alerts, and an OpenLineage-backed quality dashboard so you notice accuracy slipping before your CFO does.
How a data project runs
-
Source mapping first — We write down every producer, its SLA, its volume, its contract. OpenLineage and Marquez go in early. Half of the work is knowing what you already have.
-
Thin layers, tight contracts — Incremental dbt or SQLMesh transforms with Soda / Great Expectations 1.x tests between layers. No 2000-line queries that nobody dares touch.
-
Models on a short leash — Hypothesis, offline eval, held-out test, A/B with a business metric — Chronos / TimesFM as the foundation-model baseline, conformal intervals on the predictions. If it doesn't beat the baseline, we kill it.
-
Boring in production — Freshness alerts, lineage tracking through OpenLineage, retraining triggers, and a Dagster / Prefect 3 dashboard the on-call actually keeps open. Nothing heroic, lots of reliable.
What We Offer
Lakehouse & Streaming
- + Apache Iceberg / Delta on S3, GCS or ADLS
- + Databricks, Spark, DuckDB, Trino, StarRocks, ClickHouse
- + Kafka, Kinesis, Debezium CDC and event-bus integrations
Metrics & BI Experience
- + Cube / dbt Semantic Layer / MetricFlow governed dimensions
- + Executive & operational dashboards (Looker, Mode, Metabase)
- + Embedded analytics with multi-tenant caching
Data Science &
ML Models
- + Forecasting on Chronos / TimesFM / Moirai / TimeGPT baselines
- + Bayesian MMM (Robyn, Meridian, Orbit, PyMC v5)
- + Conformal prediction, MLflow, feature stores
Governance, Quality
& Observability
- + dbt / SQLMesh tests, Soda + Great Expectations 1.x contracts
- + PII classification, RBAC & row-level policies
- + OpenLineage + Marquez lineage, freshness & volume alerts
Migration &
Platform Modernisation
- + Teradata, Hadoop & legacy ETL to Snowflake / Databricks / Iceberg
- + Stored-proc to dbt or SQLMesh refactor with dual-run parity
- + Cutover playbooks with rollback checkpoints
How the data
actually gets there
01 Audit &
Audit &
contracts
We interview every stakeholder, inventory every source, write down what each KPI actually means, and stand up OpenLineage on day one. The doc alone usually saves an argument.
02 Pipelines &
Pipelines &
modelling
Dagster or Prefect 3 orchestration up, dbt or SQLMesh layers down, Soda / Great Expectations tests in between. Lineage tracked from event to dashboard row.
03 Dashboards &
Dashboards &
rollout
Prototype, validate the numbers against finance through reconciliation queries, train the humans who will live with it. No big-bang reveals.
04 Watch &
Watch &
keep honest
Freshness alerts, cost dashboards, usage analytics, conformal-interval thresholds on the model alerts. We keep iterating schema and docs — data debt compounds quickly if nobody is watching.
Is your team making calls on numbers they don't fully trust? Send us the brief
A few case studies where this work shows up.
We’ve shipped this before.
Five data projects where the numbers moved and somebody signed the invoice happily.
The questions people actually ask.
If your question isn’t here, email us. We read everything that comes in.
Which tools do you actually reach for?
Dagster, Prefect 3 or Airflow for orchestration; dbt, SQLMesh or Spark for modelling; Snowflake, BigQuery, Databricks or Iceberg-on-S3 for the warehouse/lakehouse; DuckDB / Trino / StarRocks / ClickHouse for query. We fit into whatever your finance team is already paying for — we don’t flip tooling on reflex.
What we typically ship:- 1. Orchestration and observability you can actually read
- 2. Iceberg / Delta lakehouse design with cost modelling
- 3. BI tool integration (Looker, Mode, Metabase, Hex, Evidence — whatever)
How do you keep the data honest?
Soda + Great Expectations 1.x assertion tests on every layer, dbt / SQLMesh column-level contracts, OpenLineage tracking, anomaly monitoring on the volumes and distributions, reconciliation jobs against the source of truth, and golden queries owned by the people who care most if they drift.
What we typically ship:- 1. Column-level tests and dbt / SQLMesh contracts
- 2. Freshness, volume and OpenLineage alerts
- 3. Reconciliation dashboards finance will trust
Can you migrate us off a legacy pipeline?
Yes, and we do it without reporting gaps. Dependencies mapped first, dual-run outputs side-by-side for a week, then cutover with a rollback that actually works. Nobody is holding their breath.
What we typically ship:- 1. Dependency map of the legacy graph
- 2. Dual-write and numeric comparison harness
- 3. Cutover playbook with rollback checkpoints
How do you handle privacy and compliance?
PII is classified before it moves, masked or tokenised where it lands, and access is enforced with RBAC and row-level policies. Every flow is documented cleanly enough to hand to an auditor.
What we typically ship:- 1. PII classification and tagging
- 2. Least-privilege access and role mapping
- 3. Audit-ready lineage and DPIA docs
Do you ship embedded analytics too?
Yes — Cube / dbt Semantic Layer / MetricFlow APIs, in-product charts, ClickHouse / DuckDB caches tuned for multi-tenancy, and role-aware views so customer A never sees customer B’s numbers. Fast and safe, in that order.
What we typically ship:- 1. Metric APIs (Cube, MetricFlow) + SDK hooks
- 2. Caching strategies that survive spiky tenants
- 3. Tenant isolation reviewed with your security team
How long does this usually take?
One domain, end-to-end, in four to eight weeks. The second and third domain are faster because you already have the pipelines, contracts and runbook patterns from the first.
What we typically ship:- 1. Roadmap organised by business domain
- 2. Milestone demos every two weeks
- 3. Prioritised backlog you can steer from