⚙️

AI Infrastructure

The open source tools and platforms that power AI development. From training to deployment, inference to monitoring.

Building blocks of open AI

AI infrastructure is the plumbing that makes everything else possible. It’s the tools for training, fine-tuning, serving, monitoring, and deploying AI systems — and the open source ecosystem here is remarkably strong.

Key categories

Inference engines

vLLM — High-throughput LLM serving TGI (Hugging Face) — Text Generation Inference SGLang — Fast serving with structured generation Triton — NVIDIA’s inference server

Training & fine-tuning

Axolotl — Easy fine-tuning with multiple methods Unsloth — Fast, memory-efficient fine-tuning TRL — Transformer Reinforcement Learning torchtune — PyTorch-native fine-tuning

Orchestration & pipelines

LangChain / LangGraph — LLM application framework LlamaIndex — Data framework for LLM applications Haystack — End-to-end NLP/LLM framework

Vector stores & retrieval

Qdrant — High-performance vector database Chroma — Open source embedding database Weaviate — Vector search engine Milvus — Scalable vector database

Evaluation & monitoring

LangSmith — LLM observability (partially open) Phoenix (Arize) — ML observability Promptfoo — LLM output testing

What to watch

  • Inference optimization making larger models practical on smaller hardware
  • Unified serving platforms that handle multiple model types
  • Evaluation becoming a first-class concern (not an afterthought)
  • Cost of self-hosting continuing to drop