LLM Safety
Hallucination Detection & Content Safety
A two-model consensus loop that catches factual errors before readers do. 30K+ pieces a day, 92.4% F1.
$ cd /services/ai-ml agent.ready · ai ml
AI & Machine Learning Engineering
LLMs break in odd ways once real users arrive. We build the parts that catch it: evals that run on every change, grounding that cites its sources, guardrails on the way in and out, and tracing you can read. Then we serve the model so it stays fast and affordable after launch.
- AI Projects Shipped
- 50 +
- Average to Production
- 6 wk
What we build
Model design, training loops, eval harnesses (Ragas, Promptfoo, DeepEval, Inspect AI), inference on vLLM / SGLang / TensorRT-LLM with continuous batching and speculative decoding, MCP tool servers, drift monitors and Langfuse / Arize Phoenix tracing and the rest of it. If it starts with "can we use AI for this?" and ends with a pager rotation, we have probably done it.
We obsess about the unglamorous bits: labeling quality, cost per call after an AI gateway (Portkey, LiteLLM, Cloudflare AI Gateway, Bedrock Gateway) routes the request, what happens when the retriever returns nothing, what your ops team does when a model misfires at 11pm. Those are the things that decide whether an AI feature survives its first quarter.
How we run an AI project
-
Frame the problem honestly — If an 80-line Python script would solve it, we'll tell you. If a small fine-tune beats a frontier model, we'll tell you that too. Then we scope what ML adds.
-
Build small, evaluate often — Reproducible pipelines, held-out sets, Ragas / Promptfoo / Inspect AI suites on day one, BIRD-SQL or HaluEval baselines where they fit. No magic runs.
-
Ship with a seatbelt on — Llama Guard 4, ShieldGemma, NeMo Guardrails or Bedrock Guardrails on the input and output, drift alerts on Langfuse / Arize Phoenix, rollback triggers, and a rehearsed failure mode before we let it near a user.
-
Keep it sharp after launch — Flags, A/Bs, human feedback loops, a labelled regression set on every prompt change, and a quarterly review. Models rot if you don't look at them.
What we build for you
S-01
LLM, Agent & MCP Systems
- + Instruction tuning, GraphRAG / HippoRAG grounding
- + MCP servers + OpenAI Agents SDK / PydanticAI / LangGraph
- + Reasoning-model verifiers, hallucination checks
S-02
Classical ML & Forecasting
- + XGBoost / LightGBM with Chronos / TimesFM baselines
- + Demand, revenue & probabilistic forecasting
- + Feature stores, conformal prediction at scale
S-03 Computer Vision
Computer Vision
& NLP
- + SAM 2.1, DINOv3, Mask2Former, Florence-2 pipelines
- + LayoutLMv3, Donut, ColPali for document AI
- + Multimodal VLMs served on FP8 / FP4 GPU stacks
S-04 Data Labeling
Data Labeling
& Annotation
- + Image, text, audio, video & document annotation
- + Labeling guidelines, QA & inter-rater checks
- + Human-in-the-loop & active-learning loops
S-05 MLOps &
MLOps &
Production Readiness
- + vLLM / SGLang / TensorRT-LLM serving + AI gateway
- + Langfuse / Arize Phoenix tracing, Ragas / Promptfoo evals
- + Llama Guard 4 / ShieldGemma guardrails, cost dashboards
How a model
goes from idea to prod
01 Scope &
Scope &
data check
A week. We look at your data, pick the metric that matters, and write down what "done" means (eval set, latency budget, cost ceiling) before any training starts.
02 Train &
Train &
evaluate
Held-out sets, error analysis, a shared eval dashboard on Ragas / Promptfoo / Inspect AI / BrainTrust. If a Chronos / GPT-class baseline beats the bespoke model, we say so.
03 Harden &
Harden &
deploy
Inference on vLLM / SGLang / TensorRT-LLM behind an AI gateway, Llama Guard 4 / ShieldGemma guardrails, rollback plan, runbook. We rehearse the first bad response before a user ever sees one.
04 Watch &
Watch &
improve
Langfuse / Arize Phoenix tracing, drift alerts, weekly review, and a priority queue that lines up the next experiment. Models stay sharp, not just launched.
A few case studies where this work shows up.
We’ve shipped this before.
A few places the AI work shipped, and what moved as a result.
The questions people actually ask.
If your question isn’t here, email us . We read everything that comes in.
What does your AI & ML work include?
Data prep, training, eval, deployment and the on-call pager rotation if you want it. LLM features, agentic systems on MCP, reasoning-model verifiers, classic ML. Whichever fits the problem, not whichever is trendy this quarter.
What that usually looks like:- 1. End-to-end build or fine-tune (LoRA, QLoRA, DPO)
- 2. Retrieval (GraphRAG, ColBERTv2, ColPali) + eval harnesses
- 3. MLOps: vLLM / SGLang serving, AI gateway, observability
How do you stop hallucinations?
You can’t zero them out, but you can cage them. We use grounded retrieval (GraphRAG / HippoRAG), structured outputs, automated eval suites (Ragas, Promptfoo), guardrails (Llama Guard 4, ShieldGemma, NeMo Guardrails) and one piece of human review where it counts — so every answer is traceable to a source or held back.
What that usually looks like:- 1. Citation checks + reasoning-model verifier on every claim
- 2. Ragas / Promptfoo regression tests on every prompt change
- 3. Runtime guardrails (Llama Guard 4 / ShieldGemma) with safe fallbacks
Will you work with our existing cloud and data stack?
Yes. We've shipped on AWS, GCP and Azure, on top of Databricks, Snowflake, Redshift and most of the vector DB ecosystem (pgvector / pgvectorscale, Qdrant, Weaviate, Vespa, Turbopuffer, Pinecone, Milvus). We fit into what you run; we don’t force a migration to justify the invoice.
What that usually looks like:- 1. AWS Bedrock, Vertex AI or Azure AI Foundry — whichever you’re on
- 2. Warehouse, feature store and vector DB choices that fit your cost envelope
- 3. AI gateway (Portkey / LiteLLM) routing across your existing model bills
How quickly do we see something in production?
Most teams see a thin slice in production inside six weeks. Not a demo or a Friday-afternoon Streamlit app: a real endpoint serving real traffic behind a flag, with Langfuse traces and a Ragas eval set running on every deploy.
What that usually looks like:- 1. Baseline and Ragas / Promptfoo eval harness in the first two weeks
- 2. Shadowed model behind a feature flag by week four
- 3. Launch checklist and rehearsed rollback before any user traffic
What observability do you set up?
Langfuse, Arize Phoenix, LangSmith or Datadog LLM Observability for prompts and outputs (where it’s legal), latency and cost on a visible dashboard via the AI gateway, drift alerts that wake the right person, and a quality regression test that runs on every deploy. The monitoring is something your team can read, not a black box we keep.
What that usually looks like:- 1. Langfuse / Arize Phoenix traces + drift and data-quality checks
- 2. Latency + cost dashboards tied to SLOs, per-tenant where it matters
- 3. Alert routing your on-call responds to
Can you keep maintaining the model after launch?
Only if you want us to. Most clients sign a light retainer for retraining, eval updates and roadmap. A few take the handover and run it in-house. We write the docs with that exit in mind either way.
What that usually looks like:- 1. Retraining playbook with triggers (drift, distribution, business KPI)
- 2. Quarterly accuracy, cost-per-call and guardrail-bypass review
- 3. On-call or business-hours support options