Manas Vardhan
making LLM systems measurable
About
- M.S. Computer Science at USC, graduating December 2026.
- Two years at JP Morgan Chase building data and ML infrastructure.
- TMLR-accepted research; now on agent memory, evaluation, and abstention.
- Author of 7 PyPI packages for LLM guardrails, tracing, and cost control.
- Writing The Agent Stack on agent runtimes and evaluation.
- Contributor to HuggingFace Transformers, lm-evaluation-harness, Axolotl, and MLflow.
Experience
FRIDAY.AI
Current Employer- Permissioned workflow agents running inside customer infrastructure on approved open models.
- Connector ingestion across Slack, GitHub, Jira, Notion, Gmail, Drive, and Asana.
- Evaluation-gated releases: production failures become replayable tests.
- Cut inference cost by routing to smaller specialist models with caching.
- Governance throughout: allowlisted tools, approval gates, and audit trails.
- Agent Orchestration
- RAG
- Model Routing
- Evaluation
- FastAPI
- MCP
- vLLM
University of Southern California
- Multi-physics pretraining across PDE families to replace expensive FEM runs.
- Scaled training with PyTorch DDP and deterministic multi-GPU seeding.
- Cut data generation from 20 minutes to 15 seconds per sample (80x).
- Matched FEM within 2.1% L2 error, inference from minutes to milliseconds.
- PyTorch
- DDP
- Distributed Training
- Scientific ML
JP Morgan Chase & Co.
- Shipped 6+ production services used by 40+ internal teams.
- Built PySpark, Kafka, and Airflow pipelines ingesting 15TB per day.
- Modeled curated tables on Snowflake and Databricks Delta Lake.
- AWS infrastructure as code via Terraform; cut deploy time 45%.
- Fine-tuned T5-Large for code summarization; cut review time 20%.
- MiniLM triage service: 8K+ daily tickets, 14 classes, 92% accuracy.
- Cut infrastructure incidents 30% with observability and auto-remediation.
- Python
- PySpark
- Kafka
- Airflow
- Snowflake
- Databricks
- Terraform
- AWS
- SQL
Projects(8)
reMem
Belief-consolidation memory for AI agents.
- Distills conversations into durable, updatable beliefs
- Sub-100ms recall over 10K+ entries via FAISS and reranking
- Beats Mem0 on its own harness: 77.1% vs. 73.2%
- Temporal reasoning 40.0% vs. 21.1%
- 66–72% on PrefEval against a 45–49% baseline
The abstention track for reMem: when to say "I don't know".
- Kept separate so the accuracy loop cannot train abstention away
- Signal from belief confidence, evidence count, and supersession
- Beats a scalar similarity threshold, more so on hard negatives
- AURC, risk-coverage, and McNemar implemented and tested
FRIDAY.AI
Enterprise agent platform running inside customer infrastructure.
- Permissioned agents on approved open models, zero retention
- Ingestion across Slack, GitHub, Jira, Notion, Gmail, and Drive
- Evaluation-gated releases on quality, latency, and cost
- Allowlisted tools, approval gates, and audit trails
LLM & Agent Infrastructure Suite
Seven published packages for running LLM systems in production.
llm-shelter: PII redaction and prompt-injection detectionagent-trace-replay: deterministic replay and OTel exportai-agent-sentry: failure classification and reliability scoringllm-cost-guardian,bench-my-llm: cost metering and benchmarksllm-promptdiff,mcp-server-forge: prompt CI and MCP scaffolding
Alignment & RL Measurement
Probe suite auditing reward-model bias across six axes.
- 4,800 paired probes across 5 open reward models
- +0.31 reward per 100 tokens at constant quality
- Sycophancy wins 68% of pairs against correct disagreement
- RLHF cost: HumanEval −6.3pp against +31pp harmlessness
- Failure mode predicted at 87% within the first 15% of steps
Proactive Intent Prediction
Predicting user needs before an explicit query.
- Screen context and history through a head over LLM embeddings
- Screenshot OCR and layout detection with MCP app metadata
- Evaluated on acceptance rate and LLM-judged relevance
Editable 3D Scenes
SIGGRAPH demo for editing 3D scenes frozen after generation.
- Localized latent edits over TripoSR codes, re-decoding only masks
- Insert, move, relight, and swap materials from text or a click
- Edit latency 94s → 1.8s (52x), geometry held at SSIM 0.97
- 83% of edits rated at or above regeneration by 12 evaluators
mlx-air
Layer-streaming inference for memory-constrained Apple Silicon.
- Loads, runs, and evicts transformer layers sequentially from disk
- Peak memory 35GB → 1.5GB: a 70B 4-bit model on an 8GB Air
- Unified memory makes weights usable the moment they land