Model optimisation

Kernels and a stack tuned to your model

General-purpose kernels and default settings are written to work for every model on every GPU. Tuned to your model's shapes and precision and to your hardware, the same GPUs can do much more.

• Kernels written for the modelCustom and fused GPU kernels for your model's shapes and precision, in Triton, CUDA with CUTLASS, or ROCm for AMD GPUs. Fusing steps keeps data on the chip instead of sending it back through memory.
• The right attention and matrix kernelsEngines ship several implementations of attention and matrix multiplication. We choose and tune them for your model, context lengths and GPUs.
• Compilation and graph captureCompiling the model fuses operations, and capturing CUDA graphs removes the cost of launching kernels one at a time, which matters most when few requests run at once.
• A serving stack set for your modelParallelism, batch limits, memory and the engine's other settings are tuned to the model, the hardware and how it's called, instead of left at their defaults.

A smaller model that does the job

Quantization and smaller models cut memory and compute per request. The question is always whether the answers hold up for your use. The figure is schematic, not to scale.

• QuantizationWeights can be stored at lower precision, including 8-bit or 4-bit formats. Runtime and hardware compatibility, memory use and quality measured on your test data determine which format is appropriate.
• Smaller or distilled modelsA smaller model, or one trained to mimic a larger one, can be enough for the task. We compare them with the original on test data that reflects real requests, against the same requirements.

More from the same hardware

Caching and co-location don't change the model, only how it runs.

• CachingResponses and intermediate results that recur don't need to be computed again. How often requests repeat and how long a response stays valid decide where the cache sits and when to compute again.
• Co-locationSeveral models share one GPU when none of them needs all of it. We place them from measured memory use and load patterns, and keep them isolated from each other.

One change at a time, measured against your requirements

Each change is measured before and after on the same load and test data, so a faster service doesn't give worse answers.

• BaselineLatency, median and p95 Throughput per replica GPU, CPU and memory use Cost per request Quality on your test data Result A baseline to compare against
• RequirementsRequired response time and throughput Minimum acceptable quality Test data that reflects real requests Result Measurable requirements, set before anything changes
• ChangeOne change at a time The same load and the same test data Measured before and after Result Measurements from before and after the change
• DecisionThe change is kept if it meets the requirements Otherwise it's rolled back The next change is measured against the same requirements Result A decision based on measurements

Around optimization

• The same metrics the service was built withLatency, throughput, resource use, cost per request and quality on your test data.
• What fits in GPU memoryHow model size, context and concurrent requests decide what a GPU can handle.
• An optimized model is a new versionIt's versioned, evaluated and monitored over time, like any other version.

The measurements stay with you

Test data, measurement scripts and results are yours, so you can measure the next model or the next change the same way.

• Run the measurement against a new model version
• Compare the results with the requirements and the baseline
• Roll back a change that misses the requirements
• Test suiteTest data and measurement scripts, so the measurement can be repeated.
• Measurement reportThe baseline and every change, before and after, against the same requirements.
• Decision logWhich changes were kept, which were rolled back, and why.
• Configuration as codeKernels, runtime settings, quantization and GPU allocation in your repository.

Related services