GPU Setup & Cloud
Training on CPU is fine for learning. Training for real needs a GPU.
Learning Objectives
- Verify local GPU availability using
nvidia-smiand PyTorch's CUDA API - Configure Google Colab with a T4 GPU for free cloud-based experiments
- Benchmark matrix multiplication on CPU vs GPU and measure the speedup
- Estimate the largest model that fits in your VRAM using the fp16 rule of thumb
The Problem
Most lessons in phases 1-3 run fine on CPU. But once you start training CNNs, transformers, or LLMs (phases 4+), you need GPU acceleration. A training run that takes 8 hours on CPU takes 10 minutes on GPU.
You have three options: local GPU, cloud GPU, or Google Colab (free).
The Concept
Your options:
1. Local NVIDIA GPU
Cost: $0 (you already have it)
Setup: Install CUDA + cuDNN
Best for: Regular use, large datasets
2. Google Colab (free tier)
Cost: $0
Setup: None
Best for: Quick experiments, no GPU at home
3. Cloud GPU (Lambda, RunPod, Vast.ai)
Cost: $0.20-2.00/hr
Setup: SSH + install
Best for: Serious training, large modelsBuild It
Option 1: Local NVIDIA GPU
Check if you have one:
nvidia-smiInstall PyTorch with CUDA:
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"CUDA version: {torch.version.cuda}")
if torch.cuda.is_available():
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"Memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")Option 2: Google Colab
- Go to colab.research.google.com
- Runtime > Change runtime type > T4 GPU
- Run
!nvidia-smito verify
Upload notebooks from this course directly to Colab.
Option 3: Cloud GPU
For Lambda Labs, RunPod, or Vast.ai:
ssh user@your-gpu-instance
pip install torch torchvision torchaudio
python -c "import torch; print(torch.cuda.get_device_name(0))"No GPU? No problem.
Most lessons work on CPU. The ones that need GPU will say so and include Colab links.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using: {device}")Build It: GPU vs CPU benchmark
import torch
import time
size = 5000
a_cpu = torch.randn(size, size)
b_cpu = torch.randn(size, size)
start = time.time()
c_cpu = a_cpu @ b_cpu
cpu_time = time.time() - start
print(f"CPU: {cpu_time:.3f}s")
if torch.cuda.is_available():
a_gpu = a_cpu.to("cuda")
b_gpu = b_cpu.to("cuda")
torch.cuda.synchronize()
start = time.time()
c_gpu = a_gpu @ b_gpu
torch.cuda.synchronize()
gpu_time = time.time() - start
print(f"GPU: {gpu_time:.3f}s")
print(f"Speedup: {cpu_time / gpu_time:.0f}x")Exercises
- Run the benchmark above and compare CPU vs GPU times
- If you don't have a GPU, run it on Google Colab and compare
- Check how much GPU memory you have and estimate the largest model you can fit (rule of thumb: 2 bytes per parameter for fp16)
Hints
Exercise 1. Run the code from the Build It section as it is, then write down the CPU time, the GPU time, and the speedup line. Run it a second time; the first GPU run includes start-up cost, so the second numbers are the ones to compare.
Exercise 2. In Colab, select Runtime > Change runtime type > GPU, then paste the benchmark into a cell. Run !nvidia-smi in another cell first; it must show a GPU name before the benchmark result means anything.
Exercise 3. Read the total memory with torch.cuda.get_device_properties(0).total_memory / 1e9as in code/gpu_check.py. Divide the memory in bytes by 2 to get the parameter count for fp16, then keep a margin, because activations and the CUDA context also use memory.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| CUDA | "GPU programming" | NVIDIA's parallel computing platform that lets you run code on the GPU |
| VRAM | "GPU memory" | Video RAM on the GPU, separate from system RAM. Limits model size. |
| fp16 | "Half precision" | 16-bit floating point, uses half the memory of fp32 with minimal accuracy loss |
| Tensor Core | "Fast matrix hardware" | Specialized GPU cores for matrix multiplication, 4-8x faster than regular cores |
Check what you learned
5 questions. Answer each one to see why.
Why is a GPU faster than a CPU for training neural networks?
The full code
Every file for this lesson. Copy it, run it, then change it.
import time
def check_gpu():
try:
import torch
except ImportError:
print("PyTorch not installed. Run: pip install torch")
return
print("=== GPU Check ===\n")
print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
if not torch.cuda.is_available():
print("\nNo GPU detected. That's fine for most lessons.")
print("For GPU-heavy lessons, use Google Colab (free).")
return
print(f"CUDA version: {torch.version.cuda}")
print(f"GPU: {torch.cuda.get_device_name(0)}")
props = torch.cuda.get_device_properties(0)
print(f"Memory: {props.total_memory / 1e9:.1f} GB")
print(f"Compute capability: {props.major}.{props.minor}")
print("\n=== CPU vs GPU Benchmark ===\n")
size = 4000
a = torch.randn(size, size)
b = torch.randn(size, size)
start = time.time()
_ = a @ b
cpu_time = time.time() - start
print(f"CPU matrix multiply ({size}x{size}): {cpu_time:.3f}s")
a_gpu = a.to("cuda")
b_gpu = b.to("cuda")
torch.cuda.synchronize()
start = time.time()
_ = a_gpu @ b_gpu
torch.cuda.synchronize()
gpu_time = time.time() - start
print(f"GPU matrix multiply ({size}x{size}): {gpu_time:.3f}s")
print(f"Speedup: {cpu_time / gpu_time:.0f}x")
vram_gb = props.total_memory / 1e9
params_fp16 = vram_gb * 1e9 / 2
params_billions = params_fp16 / 1e9
print(f"\nEstimated max model size (fp16): ~{params_billions:.0f}B parameters")
if __name__ == "__main__":
check_gpu()