LLM Safety
Hallucination Detection & Content Safety
A two-model consensus loop that catches factual errors before readers do. 30K+ pieces a day, 92.4% F1.
$ cd /services/ai-ml agent.ready · ai ml
AI & Machine Learning Engineering
> AI Projects Shipped : 50 +
> Average to Production : 6 wk
LLMs break in weird ways once real users show up.
We build the Ragas / Promptfoo evals, the GraphRAG grounding,
the Llama Guard / ShieldGemma guardrails, and the
Langfuse / Arize Phoenix observability that catches it
— then the vLLM, SGLang and TensorRT-LLM serving
that keeps catching it at 3 a.m. on a Sunday.
What we actually build
Model design, training loops, eval harnesses (Ragas, Promptfoo, DeepEval, Inspect AI), inference on vLLM / SGLang / TensorRT-LLM with continuous batching and speculative decoding, MCP tool servers, drift monitors and Langfuse / Arize Phoenix tracing — the lot. If it starts with "can we use AI for this?" and ends with a pager rotation, we have probably done it.
We obsess about the unglamorous bits: labeling quality, cost per call after an AI gateway (Portkey, LiteLLM, Cloudflare AI Gateway, Bedrock Gateway) routes the request, what happens when the retriever returns nothing, what your ops team does when a model misfires at 11pm. Those are the things that decide whether an AI feature survives its first quarter.
How we run an AI project
-
Frame the problem honestly — If an 80-line Python script would solve it, we'll tell you. If a small fine-tune beats a frontier model, we'll tell you that too. Then we scope what ML actually adds.
-
Build small, evaluate often — Reproducible pipelines, held-out sets, Ragas / Promptfoo / Inspect AI suites on day one, BIRD-SQL or HaluEval baselines where they fit. No magic runs.
-
Ship with a seatbelt on — Llama Guard 4, ShieldGemma, NeMo Guardrails or Bedrock Guardrails on the input and output, drift alerts on Langfuse / Arize Phoenix, rollback triggers, and a rehearsed failure mode before we let it near a user.
-
Keep it sharp after launch — Flags, A/Bs, human feedback loops, a labelled regression set on every prompt change, and a quarterly review. Models rot if you don't look at them.
What We Offer
LLM, Agent & MCP Systems
- + Instruction tuning, GraphRAG / HippoRAG grounding
- + MCP servers + OpenAI Agents SDK / PydanticAI / LangGraph
- + Reasoning-model verifiers, hallucination checks
Classical ML & Forecasting
- + XGBoost / LightGBM with Chronos / TimesFM baselines
- + Demand, revenue & probabilistic forecasting
- + Feature stores, conformal prediction at scale
Computer Vision
& NLP
- + SAM 2.1, DINOv3, Mask2Former, Florence-2 pipelines
- + LayoutLMv3, Donut, ColPali for document AI
- + Multimodal VLMs served on FP8 / FP4 GPU stacks
Data Labeling
& Annotation
- + Image, text, audio, video & document annotation
- + Labeling guidelines, QA & inter-rater checks
- + Human-in-the-loop & active-learning loops
MLOps &
Production Readiness
- + vLLM / SGLang / TensorRT-LLM serving + AI gateway
- + Langfuse / Arize Phoenix tracing, Ragas / Promptfoo evals
- + Llama Guard 4 / ShieldGemma guardrails, cost dashboards
How a model
goes from idea to prod
01 Scope &
Scope &
data check
A week. We look at your data, pick the metric that actually matters, and write down what "done" means — eval set, latency budget, cost ceiling — before any training starts.
02 Train &
Train &
evaluate
Held-out sets, error analysis, a shared eval dashboard on Ragas / Promptfoo / Inspect AI / BrainTrust. If a Chronos / GPT-class baseline beats the bespoke model, we say so.
03 Harden &
Harden &
deploy
Inference on vLLM / SGLang / TensorRT-LLM behind an AI gateway, Llama Guard 4 / ShieldGemma guardrails, rollback plan, runbook. We rehearse the first bad response before a user ever sees one.
04 Watch &
Watch &
improve
Langfuse / Arize Phoenix tracing, drift alerts, weekly review, and a priority queue that lines up the next experiment. Models stay sharp, not just launched.
Got a model that's stuck in notebooks? Send us the brief
A few case studies where this work shows up.
We’ve shipped this before.
A few places the AI work actually shipped — and what moved as a result.
The questions people actually ask.
If your question isn’t here, email us. We read everything that comes in.
What does your AI & ML work actually include?
Data prep, training, eval, deployment and the on-call pager rotation if you want it. LLM features, agentic systems on MCP, reasoning-model verifiers, classic ML — whichever fits the problem, not whichever is trendy this quarter.
What that usually looks like:- 1. End-to-end build or fine-tune (LoRA, QLoRA, DPO)
- 2. Retrieval (GraphRAG, ColBERTv2, ColPali) + eval harnesses
- 3. MLOps: vLLM / SGLang serving, AI gateway, observability
How do you actually stop hallucinations?
You can’t zero them out, but you can cage them. We use grounded retrieval (GraphRAG / HippoRAG), structured outputs, automated eval suites (Ragas, Promptfoo), guardrails (Llama Guard 4, ShieldGemma, NeMo Guardrails) and one piece of human review where it counts — so every answer is traceable to a source or held back.
What that usually looks like:- 1. Citation checks + reasoning-model verifier on every claim
- 2. Ragas / Promptfoo regression tests on every prompt change
- 3. Runtime guardrails (Llama Guard 4 / ShieldGemma) with safe fallbacks
Will you work with our existing cloud and data stack?
Yes — we've shipped on AWS, GCP and Azure, on top of Databricks, Snowflake, Redshift and most of the vector DB ecosystem (pgvector / pgvectorscale, Qdrant, Weaviate, Vespa, Turbopuffer, Pinecone, Milvus). We fit into what you run; we don’t force a migration to justify the invoice.
What that usually looks like:- 1. AWS Bedrock, Vertex AI or Azure AI Foundry — whichever you’re on
- 2. Warehouse, feature store and vector DB choices that fit your cost envelope
- 3. AI gateway (Portkey / LiteLLM) routing across your existing model bills
How quickly do we see something in production?
Most teams see a thin slice in production inside six weeks. Not a demo, not a Friday-afternoon Streamlit — a real endpoint serving real traffic behind a flag, with Langfuse traces and a Ragas eval set running on every deploy.
What that usually looks like:- 1. Baseline and Ragas / Promptfoo eval harness in the first two weeks
- 2. Shadowed model behind a feature flag by week four
- 3. Launch checklist and rehearsed rollback before any user traffic
What observability do you set up?
Langfuse, Arize Phoenix, LangSmith or Datadog LLM Observability for prompts and outputs (where it’s legal), latency and cost on a visible dashboard via the AI gateway, drift alerts that wake the right person, and a quality regression test that runs on every deploy. The monitoring is something your team can actually read, not a black box we keep.
What that usually looks like:- 1. Langfuse / Arize Phoenix traces + drift and data-quality checks
- 2. Latency + cost dashboards tied to SLOs, per-tenant where it matters
- 3. Alert routing your on-call actually responds to
Can you keep maintaining the model after launch?
Only if you want us to. Most clients sign a light retainer for retraining, eval updates and roadmap. A few take the handover and run it in-house — we write the docs with that exit in mind either way.
What that usually looks like:- 1. Retraining playbook with triggers (drift, distribution, business KPI)
- 2. Quarterly accuracy, cost-per-call and guardrail-bypass review
- 3. On-call or business-hours support options