Jessie A Ellis
Mar 25, 2026 17:19
New NVIDIA benchmarks present Multi-Occasion GPU partitioning achieves 1.00 req/s per GPU versus 0.76 for time-slicing in manufacturing AI workloads.
NVIDIA has launched benchmark information exhibiting its Multi-Occasion GPU (MIG) expertise delivers 33% larger throughput effectivity than software-based time-slicing for AI inference workloads—a discovering that would reshape how enterprises allocate compute sources for manufacturing AI deployments.
The exams, carried out on NVIDIA A100 Tensor Core GPUs in a Kubernetes surroundings, demonstrated MIG reaching roughly 1.00 requests per second per GPU in comparison with 0.76 req/s for time-slicing configurations. Each approaches maintained 100% success charges with no failures throughout testing.
The GPU Fragmentation Downside
Most manufacturing AI pipelines undergo from a mismatch between mannequin necessities and {hardware} allocation. Light-weight fashions for automated speech recognition or text-to-speech may want solely 10 GB of VRAM however occupy a complete GPU underneath commonplace Kubernetes scheduling. NVIDIA’s information exhibits GPU compute utilization typically hovers between 0-10% for these help fashions.
The corporate examined three configurations utilizing a voice-to-voice AI pipeline: a baseline with devoted GPUs for every mannequin, time-slicing the place ASR and TTS share a GPU by software program scheduling, and MIG the place {hardware} bodily partitions the GPU into remoted situations with devoted reminiscence and streaming multiprocessors.
{Hardware} Isolation Wins on Throughput
Below heavy load with 50 concurrent customers over 375 seconds of sustained interplay, MIG’s {hardware} partitioning eradicated useful resource rivalry fully. Time-slicing confirmed quicker particular person job completion for bursty workloads—144.7ms imply TTS latency versus MIG’s 168.2ms—however that 23.5ms distinction turns into negligible when the LLM bottleneck accounts for roughly 9 seconds of whole processing time.
The vital benefit: MIG’s fault isolation prevents reminiscence overflow in a single course of from crashing others sharing the cardboard. Time-slicing’s shared execution context means a deadly error propagates throughout all processes, doubtlessly triggering a GPU reset.
Manufacturing Implications
NVIDIA recommends MIG because the default for manufacturing environments prioritizing throughput and reliability, whereas time-slicing fits improvement, CI/CD pipelines, and proof-of-concept work the place minimizing {hardware} footprint issues greater than peak efficiency.
For organizations working combined AI workloads, consolidating help fashions onto partitioned GPUs frees complete playing cards for LLM situations—the precise compute bottleneck in most generative AI purposes. The corporate has printed implementation guides and YAML manifests for Kubernetes deployments by its NIM Operator framework.
Picture supply: Shutterstock