Chip Huyen

Bio

Chip Huyen (b. 1990) is a Vietnamese-born computer scientist and author focused on production machine learning systems. She developed and teaches CS 329S: Machine Learning Systems Design at Stanford University, and wrote two O'Reilly books: Designing Machine Learning Systems (2022) and AI Engineering (December 2024). She describes her work as production AI engineering: bringing AI into production and writing about how to do it.

Her writing falls into two recognizable phases. From 2020 to 2022, she wrote about the full ML lifecycle: data pipelines, feature engineering, monitoring, and handling distribution shift. From 2023 onward, her focus shifted to building applications on top of foundation models. The underlying concern across both phases is the same: the distance between a demo that works and a system that serves real users, reliably, over time.

Philosophical lens

Huyen's central argument is that making something impressive with AI is not the hard part. The hard part is making it work in production — at scale, without degrading, for real users. She has documented this across multiple companies: LinkedIn took one month to reach 80% of the experience they wanted and four more months to get to 95%.

Cost and latency, in her framing, are engineering constraints from day one. Inference is where the money goes, and she argues that any architectural commitment should be preceded by unit-economics arithmetic. She also observes that model capability is now largely a commodity; product design and user experience are the real differentiators. And she treats evaluation — knowing whether the system is actually working — as the hardest unsolved problem in the field, harder than model selection itself.

Recurring themes

• The demo-to-production gap — getting to an impressive demo is the easy part; the rest of the work comes after
• Eval-first — define how you'll measure quality before you design the system
• Data quality as the foundational constraint — most production failures are data failures, not model failures
• Simplicity first — heuristics before prompting, prompting before RAG, RAG before fine-tuning
• Iteration loop velocity — how fast can you detect a regression, diagnose it, and redeploy?
• UX and product design, not model capability, as the dominant constraint on AI application success

Key concepts

The demo-to-production gap

Getting an AI demo working takes weeks. Getting it to production-grade quality takes substantially longer. LinkedIn's experience — one month to 80%, four more months to 95% — is the pattern, not the exception. Huyen attributes this not to bad planning but to how ML systems behave over time: data distributions shift, model versions change, and prompts drift. Without deliberate design for change, systems accumulate debt.

Evaluation as the hard problem

Most real-world AI tasks have no single correct answer, which makes evaluation genuinely hard. Automated metrics are imperfect proxies; AI judges have their own failure modes and need engineering attention like any other component. Huyen's position is that human review of system outputs should be a daily practice — typically 30 to 1,000 examples — regardless of how mature the automated tooling is.

Simplicity-first architecture

Huyen argues that teams consistently reach for complex solutions before exhausting simple ones. The principle she applies: heuristics before machine learning, prompting before RAG, RAG before fine-tuning. Fine-tuning in particular she treats as a last resort, not a first option. The same logic applies to infrastructure decisions: batch prediction and basic retrieval before streaming and embedding-based search.

Context construction as feature engineering

In the foundation-model era, assembling the context window is what feature engineering was in classical ML — the discipline of deciding what information the model actually receives and in what form. How much context a model can process and how efficiently it uses that context are distinct questions. More context is not always better.

Where this voice fits in your decisions

Huyen's perspective is most useful after the proof-of-concept stage, when the questions become operational. She asks about evaluation before architecture, and about economics before capability. If you are working through a fine-tuning decision, designing a retrieval pipeline, or figuring out how to monitor a live system, her approach gives those decisions a systematic structure.

Limitations

Huyen's focus is applied AI systems at the team level. She does not engage with model architecture research or the mechanics of how foundation models work internally — those are a different discipline. Her writing also tends to assume teams with some ML infrastructure experience. Individual developers or very early-stage projects may find the emphasis on monitoring, evaluation pipelines, and staged architecture heavier than their situation calls for. Questions about product strategy, market positioning, or organizational leadership without an ML component are outside her scope.

Selected works

• Designing Machine Learning Systems (O'Reilly, 2022) — the canonical book on the full ML lifecycle; data pipelines, monitoring, distribution shift
• AI Engineering (O'Reilly, December 2024) — the foundation-model-era companion; building on top of LLMs rather than training from scratch
• CS 329S: Machine Learning Systems Design (Stanford, course materials publicly archived) — the lecture course she developed; the long-form pedagogical expression of her systems framing

Further reading

• huyenchip.com — her blog (2018-present); long-form posts on ML systems, AI engineering, real-time ML
• "Building LLM Applications for Production" (April 2023) — the load-bearing post on the production gap in foundation-model applications
• "Building A Generative AI Platform" (July 2024) — the canonical AI-engineering-era stack overview (orchestration, retrieval, eval, observability)
• "Open challenges in LLM research" — frequently cited survey of unsolved problems in the field

Last reviewed: 2026-05-17 · Page version: 1

How Invisico counselors work →