Generate & export
Export a deterministic model to ONNX, then create a model repository with a matching version directory and configuration.
python -m inferbench.cli export
python -m inferbench.cli triton-repo --out model_repositoryHANDS-ON INFERENCE LAB/TensorRT/Triton
Benchmark honestly.
Deploy confidently.
A practical path from model shapes and batching to measured results and Triton serving. Explore real ResNet-18 results from an Apple M1, then follow the NVIDIA deployment runbook—without mistaking them for GPU measurements.
OPEN SOURCE · APPLE M1 SAMPLE · NOT NVIDIA GPU RESULTS
TRITON INFERENCE SERVER · LESSON 01
Model configuration builder
Generate a starter config.pbtxt, then change batch settings to see how the model contract changes.
GENERATED CONFIGREADY
Loading the worked example…WHY IT MATTERSTriton adds the leading batch dimension when
max_batch_size is greater than zero. Input dimensions describe one sample.LESSON 02 · MEASURED RESULTS ONLY
Benchmark report analyzer
Load an inferbench report or a flat vLLM-style LLM summary. The Space parses results; it does not run a benchmark.
REAL SAMPLE
ResNet-18 · Apple M1 · macOS · batch 1 · 100 iterations
VIEW SOURCE ↗CUDA, TensorRT, and Triton were unavailable on the recorded host. These are not NVIDIA GPU results.
SCHEMA DEMO
LLM metrics example · synthetic, not measured
Illustrates TTFT, TPOT, ITL, request latency and token throughput fields. Use the button below to load this format example.
READ THE EVIDENCEUnavailable backends are not zero-speed results. Read every number with its host, batch size, iteration count, and accuracy drift.
LESSON 03 · FROM CONFIG TO SERVING
NVIDIA deployment runbook
A compact path from an ONNX model to Triton. Validate every engine on the target GPU before publishing performance claims.
Serve with Triton
Run the NVIDIA Triton Server container with GPU access and mount the generated model repository.
docker run --gpus=all --rm \
-p8000-8002:8000-8002 \
-v "$PWD/model_repository:/models" \
nvcr.io/nvidia/tritonserver:24.08-py3 \
tritonserver --model-repository=/modelsMeasure & compare
Record the host, batch size, warm-up, iterations, latency percentiles, throughput, and accuracy drift for each available backend.
python -m inferbench.cli bench \
--batch 32 --iterations 200HARDWARE NOTEThis Space runs in the visitor’s browser and uses no Hugging Face GPU. Use an NVIDIA GPU host for TensorRT and Triton measurements, then inspect the resulting JSON here.
Triton model configuration docs ↗Hugging Face Docker Spaces ↗Source repository ↗