NVIDIA RTX 6000 PRO 96GB VRAM: Best ML side projects for a C++ dev
A developer just got a 96GB GPU and needs project ideas to learn ML infrastructure.
Deep Dive
A C++ developer with three years of experience and some free-time PyTorch practice (admitting to being not very good at it) just received a brand new GPU from a colleague. They are asking for cool side project ideas to learn machine learning and inference/infrastructure, noting they have no current preferences and are completely new to the space.
Key Points
- 96GB VRAM can run quantized Llama 3 70B or Falcon 180B locally for inference experiments.
- Ideal projects include building an inference server with vLLM or implementing a RAG pipeline with ChromaDB.
- C++ skills are directly applicable to optimizing custom CUDA kernels or using TensorRT for deployment.
Why It Matters
Massive VRAM GPUs let solo devs prototype production-grade ML infra without cloud costs.